What Makes Unforgettable Images So Memorable?

One of the surreal things about living in the age of blossoming artificial intelligence is that the definition of humanity keeps narrowing. That is, as machines learn more and more, they continue to edge into spaces that previously seemed dominated by humans and humans alone. The list of things that only a human can do is growing smaller, with no end in sight.

One recent example of this phenomenon comes from a team of researchers at M.I.T., who used a computer brain to establish a new procedure for measuring human memory. What they came up with, the researchers say, is an unprecedented way for machines to predict the images that humans will find memorable.

This is remarkable for a few reasons, not least of which is the tenuousness of the very concept of memorability.

“Intuitively, the question of an artificial system successfully predicting human visual memory seems out of reach,” wrote Aditya Khosla, Akhil Raju, Antonio Torralba, and Aude Oliva in a paper accompanying their work. “Unlike visual classification, images that are memorable, or forgettable, do not even look alike: an elephant, a kitchen, an abstract painting, a face, and a billboard can all share the same level of memorability, but no visual-recognition algorithms would cluster these images together.”

How do you begin to teach a machine to analyze an image for how memorable it will be to humans? First you need to find out, from the humans themselves, what images are most memorable. So the researchers had people them play a game in which they were presented with a series of images, some of which were eventually repeated. The dataset was huge: It included some 60,000 diverse images—sunsets, clocks, ballerinas, selfies, dogs, trees, etc. Participants were told to press a key if they recognized an image they’d already been shown.

Then, the researchers trained a computer, using a model they call MemNet, to classify those same images 1,000 different ways, based on what humans found memorable. Here’s where it gets really crazy: “By visualizing the learned representation of the layers of MemNet, we discover the emergent representations, or diagnostic objects, that explain what makes an image memorable or forgettable.”

In other words, once they knew the machine could accurately predict what humans found memorable, they were able to figure out why.

Using a demo of this model, for instance, I was able to learn that this image of me is pretty memorable. It scores 0.85 on a possible scale from 0 (least memorable) to 1 (most memorable), with the red section representing the most memorable aspect of the photo:

This doesn’t have much to do with my face, per se. Which is probably a good thing: Researchers found that images that evoke disgust, anger, and fear tend to be more memorable than those that evoke awe or contentment. Overall, negative images are more memorable than positive ones—with the exception of images that people find amusing, which also score high for memorability.

The reason the photo of me scored so high on the memorability scale, Khosla told me, probably has more to do with the framing of the photo, since the computer model was not trained to be face-centric. (He and his colleagues are working on building a face-centric model that will distinguish memorability based on facial features.) Using the current model, the most memorable images have salient focal points, and are often close-ups.

This guy, for example, was almost as memorable as my face:

This dog was pretty memorable, too:

But this townscape wasn’t very memorable at all:

On the other hand, these eggs with little faces painted on them were super memorable:

One intriguing finding was that memorability appears correlated with popularity. Images in the dataset that came from Flickr, and were popular on that site, ranked higher in memorability than images that were less popular. This link suggests people may eventually be able to design images so they’re optimized for sharing across social networks. It also raises a new and complicated question about the nature of memorability. What would happen if people saw only memorable images? “Would they start forgetting everything and only remember the most memorable of the lot?” Khosla said. “Our initial experiments suggest that this might not be the case. We found that, for at least a small set of images, the average number of images remembered by people increased when we showed them more memorable images.”

All of which suggests super-memorable images don’t crowd out other memorable images, but instead have a stoking effect on memorability. “This implies that we could potentially increase the amount people remember by showing them more memorable images,” Khosla said.

The potential applications here are huge.

Learning materials could be modified to make it easier for students to remember lesson plans, presidential candidates and advertising firms could create images algorithmically guaranteed to stick in people’s minds. The M.I.T. researchers sum it up simply, but the implications are complex and profound: “Our work shows that predicting human cognitive abilities is within reach for the field of computer vision.”