A Deep Learning Approach for Digital Color Reconstruction of Van Gogh’s Paintings Using Unpaired Areas Under the Frame

Factors such as dirt accumulation, aging of paint materials and chemical reactions with the surrounding atmosphere lead to alteration and degradation of paintings. As a result, colors in the paintings can change. Van Gogh’s paintings are no exception to this process, and virtual reconstruction of the original colors is a very challenging problem. In this work we propose a novel approach for color reconstruction that does not require any pre-existing digital reconstructions, physical reproductions or artificial aging experiments, and relies purely on the data within the painting. We exploit the fact that areas of the painting located under the frame are typically well-preserved (mainly due to the protection from light exposure and dirt accumulation offered by the frame) and contain colors which are relatively close to their original look. Inspired by the recent advances in machine learning techniques for unpaired image-to-image translation, a practical weakly supervised approach for digital color reconstruction is formulated. Moreover, its performance is demonstrated for paintings by Vincent van Gogh. To our knowledge, this is the first time that a method for color reconstruction that relies purely on the data available within one painting is described in literature.