Ishan Misra on Self-Supervised Learning, the "Dark Matter" of Machine Intelligence

High-Level Overview

Lex Fridman talks with Ishan Misra, a research scientist at Facebook AI Research (FAIR) who works on self-supervised learning for computer vision: teaching AI systems to understand images and video with minimal help from human labelers. Lex frames the dream in his introduction: leave a robot watching YouTube videos all night and come back in the morning to a much smarter robot. The conversation starts from the blog post Ishan co-wrote with Yann LeCun, "Self-Supervised Learning: The Dark Matter of Intelligence," and works through why human labeling can't scale, why vision is harder than language, how contrastive and non-contrastive methods work, and what Ishan's own projects (SwAV, SEER, DINO, VISSL and audio-visual learning) have shown.

The tone is technical but friendly and often playful. Lex challenges listeners to count how many times the word "banana" comes up, since it becomes the running example for nearly every idea. The conversation moves from definitions to research details, then widens into autonomous driving, the limits of deep learning, emotion and consciousness, and ends with career advice for researchers and students.

The Discussion Explores:

• Why supervised learning doesn't scale and what "self-supervised" really means
• Categories, similarity, and why a perfect taxonomy of objects is hopeless
• Why Ishan thinks vision is harder than language
• Contrastive learning, energy-based models, and the problem of "collapse"
• Data augmentation as the hidden source of human knowledge in self-supervised vision
• SwAV, SEER and training a billion-parameter model on a billion uncurated images
• Multimodal learning with audio and video, and active learning
• Autonomous driving timelines, the limits of deep learning, and advice for researchers

Deep Dive into Main Ideas

Why Human Labels Can't Scale

Ishan begins with the basics. In supervised learning, humans label data ("this image contains a banana"), and the model learns to mimic those labels. Semi-supervised learning adds a large pool of unlabeled data to a smaller labeled set. Self-supervised learning goes further: the data itself is the source of supervision. That, he explains, is why he and LeCun prefer the term to "unsupervised," which technically covers everything that isn't supervised.

The problem with labels is scale. ImageNet, the dataset credited with helping launch deep learning, has about 22,000 concepts and 14 million images and took roughly 22 human years to annotate, yet Ishan says 14 million "is nothing really" next to the hundreds of millions of images uploaded to social media. Humans are also inconsistent labelers: what separates a "dining table" from a "table"? He recalls spending a week of his PhD drawing boxes around cars in video, only to be told one was a pickup truck, which made him wonder what he was even annotating.

The tricks that unlock self-supervision are simple but powerful. In language, you mask a word ("the cat jumped over the dog") and ask the model to predict it. In video, you show the first nine seconds and ask what happens in the tenth, which teaches object permanence and gravity for free. In images, the most popular trick is to take different crops of the same picture and teach the network that their features should be related.

Similarity Instead of Perfect Categories

Lex asks whether a perfect taxonomy of objects is possible. Ishan thinks it is "hopeless in some way": for any list of categories, you can blend two concepts and make a new one, the way the croissant and doughnut produced a cronut. Things are compositional, and names often aren't needed at all. Animals get through their day without syntactic language, and Ishan could open an unfamiliar door knob because it resembled other door knobs, not a hair dryer.

Similarity, and just as importantly dissimilarity, can take us very far, he argues, though it depends on context: an elephant and a lion are similar as four-legged land animals and very different as herbivore and carnivore. Lex suggests understanding might be placing a thing in a network of knowledge and adding edges, and they agree self-supervised learning should be the "cake" in LeCun's analogy, with supervised categorization as the icing.

Why Vision Is Harder Than Language

Ishan says plainly that computer vision is harder. Many animals are intelligent without language, which suggests something more fundamental about vision. Language is a human-built structure: masking works because the model only has to choose among roughly 50,000 words. Masking a patch of an image means predicting a combinatorially huge set of pixel values, which quickly becomes intractable. That's why vision methods instead compare feature vectors from two crops and ask that they be close together.

Language also benefits from the distributional hypothesis, the idea that words in the same context have similar meanings, and it is captured almost losslessly when written down. Images pass through lighting, sensor noise and a complicated processing pipeline. Lex pushes back that language is itself a projection of thought, and they settle on a nuance: getting impressive results is easier in language, but both fields face a wall before human-level understanding. They also discuss transformers, whose self-attention lets every element attend to every other element, giving the wide context a local window in a convolutional network misses.

Contrastive Learning, Energy and Collapse

Contrastive learning builds an embedding space by pulling related samples ("positives) together and pushing unrelated ones (negatives) apart: a cat and a dog together, a banana away. Ishan recalls finally cornering Yann LeCun in a conference room to ask what energy-based models" meant; LeCun patiently explained at a whiteboard that contrastive methods, GANs and variational autoencoders can all be described as minimizing or maximizing an energy function, where related things sit in low-energy dips.

The weakness of contrastive learning is that it needs many good negatives, which is hard to scale. Non-contrastive methods avoid that. Clustering puts two crops of an image in the same cluster; self-distillation trains a student network to match a teacher network. The main enemy of all these approaches is collapse, where the network learns the same features for every image, or as Lex puts it, everything is a banana. Ishan's team also used decorrelation between features to prevent collapse, inspired by Horace Barlow's neuroscience.

Data Augmentation: Where Human Knowledge Sneaks Back In

Ishan calls data augmentation "key to self-supervised learning" in vision. Current methods crop, blur, rotate, and jitter an image's colors, brightness and contrast, then require the features from two perturbed versions to match. The irony, he says, is that deep learning purists want the network to discover patterns from raw pixels, yet researchers hand-pick augmentations that encode human assumptions, such as a changed color still being the same apple. In a sense the supervision has been sneaked in at the input.

He'd like augmentation to become part of the learning process and aware of the image's content: turning a banana purple or red isn't realistic, and some geometric augmentations make no sense for medical images of the human body. Lex dreams of wildly imaginative augmentation, like elves with bananas hiding behind a table, and Ishan notes the chicken-and-egg problem: great augmentation requires understanding the scene. Asked to choose between unlimited natural data and a great augmentation algorithm, Ishan picks the algorithm, since today's methods still need something to say which two things are similar.

SwAV, SEER and Learning in the Wild

SwAV is a clustering method that performs clustering online as the network trains, with a fixed number of clusters (for example 3,000) and an equal-partition constraint that forces all clusters to be used, which prevents collapse. It uses soft assignments, so one sample can partly belong to several clusters.

SEER builds on SwAV to test whether self-supervised learning is just overfit to ImageNet. Ishan admits the research community kind of cheats: it strips ImageNet's labels and pretends the data is unlabeled, even though its images are curated, centered and object-focused. SEER instead trained a model of about a billion parameters on about a billion random internet images, three orders of magnitude more than ImageNet's million, without filtering them by category. The images still carry photographers' framing bias and skew toward parts of the world with more internet access. The model was a RegNet, a network family designed to be efficient in memory, not just floating-point operations. Ishan believes data, augmentation and the training algorithm matter far more than architecture. He also describes VISSL, FAIR's PyTorch library that standardizes how self-supervised methods are implemented and benchmarked.

Multimodal and Active Learning

In a CVPR 2021 best-paper candidate, Ishan's team trained one network on video and another on its audio, requiring their features to agree. The result surprised him: without labels, the video network learned to recognize human actions from the 400-category Kinetics dataset and, when shown a silent video of someone strumming a guitar and given a guitar sound, focused on the guitar. It could even pick out a speaking celebrity's mouth. He thinks most things can be learned from one modality, but an extra one helps; hearing someone cut is how you tell an apple from an onion.

Ishan likes active learning, describing an earlier paper in which an agent learned by asking the questions it would learn the most from. The key challenge is modeling what the model doesn't know. A simple heuristic: after labeling one banana, look next at images that are neither near-duplicates nor totally different. Lex likens Tesla's data engine, which sends back near-failure edge cases for labeling, to active learning, and Ishan agrees that catching edge cases through prediction surprises is a smart use of self-supervised predictive models.

Driving, Limits of Deep Learning and What Intelligence Needs

Ishan, noting he's not a driving expert, guesses five to ten years for solved autonomous driving, and was impressed seeing a Tesla's vision-only system handle rain. Lex's range is five years optimistic and 30 pessimistic; he suspects human behavior and society make driving a human-robot interaction problem, possibly requiring general intelligence. Ishan is skeptical of training in simulation generally: it's expensive, lighting is hard to simulate, and the mix of human and robot drivers will keep changing.

On limits, Ishan names data efficiency (learning from one or two examples), the lack of guarantees (a cat classifier failing isn't treated as a bug the way a broken sort would be), weak reasoning compared with snap recognition, and catastrophic forgetting. He finds it odd that emotion plays no role in machine learning, since surprise comes from a mismatch between prediction and observation, and thinks AI will eventually need embodiment and some self-awareness about its role. His favorite idea in the field: from the "dumb idea" of making two crops' features match, DINO models discovered object boundaries, such as which pixels belong to a running dog, without ever being taught.

Advice for Researchers and Students

Ishan's paper advice: pick a problem you can make progress on in the time you have and that genuinely excites you, push one simple idea rather than cramming in many, and start writing early, since writing exposes holes in your thinking. Some researchers he knows write the introduction before running any experiments. For beginners he recommends Python, prefers PyTorch because its imperative style is easier to debug, and says the open-source community quickly ports code between frameworks.

His broader advice is to get your hands dirty and spend hours debugging before searching for answers, to be hungry ("you shouldn't want it you should need it"), and to grow a thick skin, since in research experiments fail almost every day. He recalls Pittsburgh's saying that "snow builds character" and Edison's many failed filaments. On the meaning of life, he admits he has no idea and doesn't think AI will answer it; the fact that humans have different, evolving objective functions is a feature, not a bug.

Conclusion

Ishan Misra makes a clear case that human labeling can't be the path to machine intelligence at scale, and that the real signal is hidden in the data itself: in sequences, in different views of the same image, and in sound paired with video. He's also candid about the field's blind spots: vision methods lean on hand-designed augmentations that quietly reintroduce human knowledge, benchmarks have been gamed by treating curated ImageNet as "unlabeled," and deep learning still struggles with data efficiency, guarantees and reasoning. The most striking result is how much structure, such as object boundaries, emerges from simple ideas like matching two crops. For anyone curious how modern computer vision learns to see, and what a working researcher thinks about writing papers and handling constant failure, the full conversation is worth the time.

This is a summary of "#206 – Ishan Misra: Self-Supervised Deep Learning in Computer Vision" from Lex Fridman Podcast, giving you the GIST so you can decide if you want the REST. Listen to the full episode.