Artificial Intelligence to the rescue

Artificial Intelligence has found its way into many facets of our everyday lives. No longer is it unusual to hear how AI enhances the speed, precision, and effectiveness of human efforts. Recently, the field was used in the realm of Fine Art to help restore a mutilated Rembrandt painting.

In 1715, an unfortunate accident caused pieces of Rembrandt’s famous “Night Watch” to snip off. One of the artist’s most famous works, “Militia Company of District II under the Command of Captain Frans Banninck Cocq” was painted by the artist in 1642. It is often considered a perfect representation of Holland’s Golden Age of painting. The case of this damaged painting has been enormously distressing for many years.

The disfigurement occurred after the artist’s death, when it was moved from the original location at Arquebusiers Guild Hall to Amsterdam’s City Hall. City officials wanted to hang the painting between two doors which were too narrow for it to fit through. Rather than finding another way to get the painting inside, the uninterested staff cut large panels off the sides and sections from the top and bottom. Without a home, those fragments were soon permanently lost.

Art historians always knew that the original painting was bigger and more complete because of a smaller copy that is attributed to Gerrit Lundens. After two years of painstaking research that includes using a battery of high-tech scanners, x-rays, and digital photography, they compiled enough data to be able to compare a AI restored image to Lundens’ copy. In such, the professionals were able to recreate the original and print the missing pieces to perfection.

The Rijksmuseum has owned “The Night Watch” since the museum opened in 1885 and the Rembrandt is considered one of the most well-known paintings in the museum’s collection. In 2019, “Operation Night Watch” was initiated… a multi-million dollar, multi-year effort to restore the painting.

The painting’s 26th restoration primary goal was to restore the painting to its original size. Luckily, Erst van den Wetering, a prominent Rembrandt scholar, requested to be a part of the project. He wrote to the museum suggesting that composition would be altered. The Rijksmuseum called on Rob Erdmann, a senior scientist on staff, to lead the project. Erdmann used a preserved section of the original painting, the 17th-century copy praised by Lundens, and Artificial Intelligence.

AI was used because the scholars hoped to leave all subjectivity out of the restoration equation. As told to ARTnews, “Our attempt here is to make the best guess without the hand of an artist into what the missing parts of the Night Watch looked like.” And continuing, “So this an attempt to use the latest technology to use the Lundens to help us visualise what the missing parts of the Night Watch looked like. The first problem that the neural network helped us solve was to make semantic correspondences between the two. I can use a second neural network to help me make a plan, a so-called flow field that would basically show how to stretch and compress and rotate the Lundens image so that it’s perfectly aligned with the Night Watch image. But now I have another problem, and that is, Lundens is a different painter than Rembrandt. So, then comes a third neural network. And in this third neural network, I use this alignment between the Lundens and the Night Watch to basically send the neural network to art school. So there is a little tile from the Lundens and then the same tile from the Night Watch. And so this is basically a quiz that we’re giving a neural network. We say, all right, paint this like Rembrandt would paint it. And by performing training with neural networks, we can use hundreds of thousands of iterations of this process until the network gets better and better, so that it can take something painted in the style, scale and palette of Lundens and approximate what it might look like if Rembrandt did that.”

The painting has been through a plethora of trials and tribulations. In 1911, it was stabbed by a knife; it was hidden in a bunker during the invasion of Nazi Germany into the Netherland. In 1975, it was stabbed by another knifeman, and finally, it was splashed with acid in 1990.

AI was vital in solving some problem that arose during this complicated process. The first was that Lundens’ copy was one-fifth the size of the original Rembrandt. The second problem was that the artists painted in different styles. Thus, how could the missing pieces be restored. Erdmann designed three separate “networks.” These are specific machine learning technology that trains a computer to learn how to do specific tasks to solve particular problems.

Network #1 was responsible for specifying the common details. All in all, more than 10,000 details are common between the original and the copy. Once these details are found, Network #2 comes into play by manipulating the pieces in a delicate fashion to perfectly align the sections. The final stage entailed the 3rd Network; during this process, the painting was essentially “split” into thousands of tiles and both the original and the copy were placed side-by-side. AI was then able to create an “approximation” of those tiles in “Rebrandt’s style.” The program was run millions of times.

Lastly, the nearly identical reproduction was printed onto canvas and lightly varnished to display the appropriate patina. At that point, the reproduced panels were attached to the painting’s frame over the fragmented original. In order to honor the original, the reconstructed panels do not touch the original painting. Those panels were removed three months after their placement. This is probably a controversial topic; but there is no denying that as time goes on, Artificial Intelligence will be used more and more often in the art world.