Dawn Song on Adversarial Machine Learning, Privacy and Computer Security

High-Level Overview

In this episode of the Lex Fridman Podcast, Lex talks with Dawn Song, a professor of computer science at UC Berkeley whose work sits where computer security meets machine learning. They cover a lot: whether software will ever be free of vulnerabilities, why attackers increasingly go after people, how machine learning models can be fooled during both training and use, the stop signs from her lab that now sit in a London museum, privacy attacks that pull Social Security numbers out of a language model, the idea that people should own their data, blockchain and her startup Oasis Labs, and program synthesis, which means teaching computers to write code. It ends with her path from physics in China to computer science in the US and a long exchange about the meaning of life.

The tone is curious and technical but easy to follow. Lex keeps asking Dawn to slow down and explain terms, and she answers with clear examples. There are light moments too: Lex asks to be misclassified as Putin instead of Trump "because I'm Russian." The ending is reflective.

The conversation starts with security basics, moves through current attacks and defenses, then turns to data economics and AI research, and finishes with her personal story and philosophy.

The Discussion Explores:

• Why perfectly secure software is practically impossible, and what formal verification can and cannot promise
• How attacks are moving "up the stack" toward humans, and how AI chatbots might help protect people from social engineering
• Adversarial machine learning: altering inputs while a model is in use and poisoning training data with hidden backdoors
• Physical-world attacks, including stickered stop signs, and what they mean for Tesla Autopilot and other self-driving systems
• Defenses built on consistency checks across image patches, audio and video, and why attackers are currently winning
• Privacy attacks that extract sensitive training data from models, and differential privacy as a defense
• Data ownership as a property right, the tension between usefulness and privacy, and blockchain-based tools for a "responsible data economy"
• Program synthesis as a path to intelligent machines, plus Dawn's personal journey and her view on the meaning of life

Deep Dive into Main Ideas

Will Software Always Have Vulnerabilities?

Lex opens with a broad question: will software systems always have security holes? Dawn says it is very hard to write bug-free code, partly because "vulnerability" is such a broad idea and the kinds of attacks keep changing. She walks through classic memory-safety bugs such as buffer overflows, where malicious input lets an attacker change a program's state and take over what it does. She also describes side-channel attacks, where an attacker works out secrets just by watching a program's outputs or behavior.

On the hopeful side, she says we are entering an era of formally verified systems. Researchers have spent decades building tools that analyze and verify programs, and there are now verified microkernels, compilers, file systems and crypto libraries. Most of these techniques work statically: they prove properties by analyzing the code without running it. Testing and fuzzing, by contrast, only cover the situations you happen to run. But Dawn adds a caution. A system can be proven free of one class of vulnerability and still fall to another. Security is unusual because you are trying to prove that no attack exists, which is not even a well-defined statement. She sums it up with a joke from the field: "security is job security."

Humans Are the Weakest Link

Dawn has worked on network, software, binary, web and mobile security, and she has watched attacks shift away from systems and toward people. As systems get harder to break, attackers go after what security researchers call the weakest link, which is often the human. Social engineering and phishing trick people into giving up passwords or even wiring money, and she notes this has happened at companies with very strong security. Deepfakes and fake news are likewise aimed at manipulating people's opinions and perceptions. Machines can be patched and hardened, but as she puts it, "we don't have a way to do a software upgrade or do a hardware change for humans." This is the type of attack she worries about most.

Her group's answer is to use AI to protect people. One project uses language-processing and chatbot techniques. A bot watches a conversation between a user and someone contacting them and looks for signs of an attack, such as a supposed relative stranded in London with a stolen wallet asking for money. The bot could then run a challenge-and-response test, asking questions only the real relative would know. Later it might even keep an attacker talking to learn more about them. Lex calls it a little lawyer that protects you and asks who would control it. A bot that goes with you across Facebook, Twitter and every other service would know a lot about you, which is fine if you are the one controlling it. Dawn thinks it could work either as a service users choose or as one platforms provide, once the language technology is strong enough.

Adversarial Machine Learning: Fooling Models During Use and During Training

Dawn defines adversarial machine learning as attacks that aim to trick a learning system into making the wrong decision. While a model is in use (the "inference" stage), an attacker adds tiny changes to an input, often invisible to the human eye. A model that classified the original image correctly then gets the altered one completely wrong. In a targeted attack, the model gives the specific wrong answer the attacker wants.

During training, attackers can poison the data. Her group has shown backdoor attacks in which a model behaves normally on almost all inputs but gives the attacker's chosen answer on specific "trigger" inputs. These attacks are stealthy. Even people reviewing the training data struggle to spot them, and it is almost impossible to tell from the model that it was trained wrong. In a facial recognition study, she explains, a very small number of poisoned data points was enough. More strikingly, the team added training images of people wearing a certain kind of glasses, labeled as one target person. The model learned to link the glasses themselves with that person, so anyone wearing them would be recognized as that person. They tested this in the physical world with real glasses. They also showed the glasses could be blended into an image's pixels so faintly that people inspecting the images could not tell.

Stop Signs in a Museum and the Physical World

Lex turns to Dawn's 2018 paper on robust physical-world attacks on deep learning visual classification. The stop signs from that work are now exhibits at the Science Museum in London, which she calls a very rare occasion for research artifacts. The question was whether adversarial examples exist outside the digital world. Could a physically altered stop sign make a car's image classifier read it as, say, a speed limit sign? And would it keep working from different distances, angles and lighting? The unfortunate answer, she says, is yes.

Moving to the physical world brings many constraints. The changes have to sit on the sign itself rather than anywhere in the image. They have to be large enough to survive the camera, since you cannot edit what the camera sees directly. And they have to survive printing: the team bought real stop signs and put printed stickers on them. Asked whether this is art or science, Dawn says it is mostly science. They build all these constraints into the loss function they optimize, then do extra real-world testing at the end.

What does it reveal about neural networks? Dawn thinks it shows we are at a very early stage of building robust, generalizable machine learning, and that "we are not learning the right things." Human vision takes in much richer, more nuanced information than a label like "person" or "camera." She believes richer representations would make learning systems both more generalizable and more resilient.

Defenses: Consistency Checks and Who Is Winning

Dawn says thousands of papers have now been written on this topic. Most are about attacks, but there are also many hundreds on defenses. She calls much of the defense work, such as adversarial training, a kind of patchwork with limited effectiveness. There is no strong, general defense yet.

Her own promising direction uses natural constraints in the task. A 2018 paper looked at semantic segmentation, where a system labels every pixel of an image. Her team first showed that attackers could make segmentation systems produce almost any pattern, even a Hello Kitty shape or the letters ICCV. For defense they used spatial consistency. If you take two random, overlapping patches of an image and segment each one, the labels in the overlap should agree. That holds well for normal images. But it is very hard for an attacker to fool the system in a way that stays consistent across randomly chosen patches, and the method proved very effective against state-of-the-art attacks. The same idea works as temporal consistency in audio such as speech, and the two can be combined for video.

Still, when Lex asks who is winning, Dawn says it is clearly the attackers. Attacks come in white-box and black-box forms, and black-box attacks do not even require knowing the model's architecture or parameters. To companies who believe their real-world systems are safe, she says: "I wouldn't call that a hope. I think it's more of a wishful thinking." Her students and collaborators showed they could copy a model like Google Translate just by querying its API. They trained an imitation model that performed similarly, created adversarial examples on the copy, and found some of them transferred to the real system. In one English-to-German example, changing "six" to "seven in a sentence about freezing weather changed the translation from six Fahrenheit to 21 Celsius. Lex called it incredible and terrifying." Earlier work showed similar black-box attacks against cloud vision APIs.

Self-Driving Cars and Sensor Attacks

Lex presses on self-driving cars. He notes that Elon Musk told him adversarial attacks on Tesla are not a real problem because they are very hard to pull off in the real world. Dawn separates two questions: whether such an attack is feasible, and whether someone will actually carry one out. On feasibility she says "I'm 100% confident that the attack is possible," pointing to research showing that a few stickers on the road could fool a Tesla. Lex pushes back that many demonstrations are lab-like. He compares them to cracking a safe with unlimited time rather than stealing the crown jewels.

Dawn replies that you do not even need sophisticated attacks to see the risk. There have already been real incidents where vehicles made wrong decisions with no attacker at all, because learning systems do not generalize well even in normal conditions. From a passenger's point of view, it hardly matters whether a failure is random misbehavior or a targeted attack. Her recommended defense is to use many different sensors and combine their readings, with consistency checks. The more sensor inputs a car uses and the better they are combined, the harder it is to attack. Lex adds that this means multiple cameras plus radar and even ultrasonic sensors.

Privacy Attacks and Differential Privacy

Dawn frames security in terms of two properties. Integrity asks whether the system gives the right answer. Confidentiality asks whether attackers can steal sensitive information. Adversarial examples attack integrity, while privacy is about confidentiality. Training data is often very sensitive, from financial and health records to sensor data collected in the real world. Neural networks have high capacity, so they can memorize a lot of it.

With white-box access, attackers can use a model's parameters to learn about its training data, even whether a specific person's data was used. But access to queries alone can be enough. In work with Google researchers, her team trained a language model on the Enron email dataset, which naturally contains Social Security and credit card numbers. They showed that new attacks, using only queries, could pull out those original numbers. The good news: training a differentially private language model instead kept similar usefulness while making their attacks no longer effective. Differential privacy works by adding noise during training, for example to the model's gradient updates.

Owning Your Data

Dawn's larger vision is about who owns data. She compares it to physical property. Establishing and enforcing property rights has been a major driver of economic growth, and she mentions research suggesting that in many developing countries the problem is less a lack of capital than a lack of property rights and their enforcement. Today more and more of what defines a person lives in the digital world, from political views and music tastes to financial and health information. For the most part, she says, that data is effectively owned by internet companies. Clarifying ownership is the first step, because the owner gets to decide how the data is used.

Lex puts on his "grumpy old grandpa hat": would owning your data end the free internet of Facebook, YouTube and Twitter? Dawn does not think so. Users who enjoy the free services could still agree to let their data be used in certain ways. Others could pay for a private version, much as people already pay subscriptions to avoid ads. She wants a more constructive, nuanced conversation instead of two sides fighting, because there is a real technical tension between usefulness and privacy: a good recommendation system needs your data. Her group builds technology to balance the two, and she believes technology plus regulation can make it a win-win. Lex adds his own appreciation for what these platforms have built, especially Facebook's role in bringing people's real identities into the digital world.

Blockchain, Consensus and Oasis Labs

Dawn describes "blockchain" as an overloaded word, like AI. Usually it means a distributed ledger kept by a community of computers (nodes). Individual nodes may not be trustworthy, but the system works as long as enough of them behave properly. It has to stop malicious nodes from changing transaction records or spending the same money twice. How hard it is to attack depends on the consensus mechanism, such as proof of work or proof of stake. For each mechanism you can show in theory what share of nodes an attacker would need to compromise. Other kinds of attacks, though, make a complete estimate difficult. She also names availability as a third security property alongside integrity and confidentiality.

On confidentiality, public ledgers offer none by default: anyone can see every transaction. Tools like zero-knowledge proofs and secure computing can hide who is sending money to whom and how much, and can allow confidential smart contracts. At her startup, Oasis Labs, she is building what she calls a "platform for responsible data economy." It combines privacy-preserving computation with a ledger that keeps a permanent record of users' ownership of their data, the rules they set for its use, and how it has actually been used.

Program Synthesis: Teaching Computers to Write Code

When Dawn moved her focus from security to AI, program synthesis and adversarial machine learning became her two main areas. Her motivation was building intelligent machines, and she calls program synthesis "the perfect playground for building intelligent machines." Since humans can write programs, she sees no reason computers cannot. In limited, well-defined domains she thinks it is already ready for real-world use. At a conference, the CEO of a startup told her that one of her papers, on translating natural language into SQL database queries, had become a key product for his company.

She names three open challenges. The first is complexity: her group went from simple "if this then that" programs to SQL queries to programs with loops. Lex jokes that once you can synthesize recursion, "it's all over." The second is generalization to inputs the system has never seen. Her earlier work on learning recursive neural programs showed that with recursion, perfect generalization is possible for certain tasks. The third is adaptation. Today a model is trained for one particular task, but the goal, as in deep reinforcement learning, is to learn from past tasks in order to solve new ones. She calls this especially important for program synthesis.

From Physics in China to Computer Science in America

Dawn studied physics as an undergraduate at Tsinghua and spent a year in the physics PhD program at Cornell before switching to computer science at Carnegie Mellon. She loved physics and "thought physics is the language of nature." She remembers sitting in the library and realizing that from a few simple laws she could derive so much: "It's almost like I can derive the rest of the world." But theoretical physics meant writing Fortran simulations, experimental physics meant spending most of your time fixing expensive equipment, and many senior students were heading to Wall Street. She had taught herself C programming, and what drew her to computer science was how quickly you can turn an idea into something that runs. What she gave up was elegance. Computer systems are built by people, often for historical reasons, so you cannot derive them from a few simple laws.

She recalls that when she came to the US there was little access to email, let alone the web, and far less knowledge of Western culture than today's students have. Her turning point came in college. After teaching herself C, she built a Go game and a 3D Tetris and realized she could simply try an idea and make it real. Asked about the US and China in AI, she is optimistic because "science has no border." Academic research is open: papers and code are public for anyone, anywhere.

The Meaning of Life

Dawn says the meaning of life is probably the question that has followed her most. After a long search, including looking to voices and factors outside herself, she concluded that "it's you yourself that defines the meaning of life." She also asks whether the question is well defined, and whether knowing the answer actually makes life better or happier. Lex offers creation as his own answer. Dawn agrees creation matters to her, and adds growth, trying to be a better self every day as part of an evolving world. Lex notes that the growth seems to matter more than the destination. Dawn ends with a concrete example: she moved from security to AI because she believed her purpose was to build intelligent machines. So the question can help you focus, even if you can never be sure of the answer.

Conclusion

Dawn Song paints a sobering but constructive picture. Software will never be perfectly secure. Attackers are shifting toward people, and toward machine learning models that can be fooled with stickers, poisoned data or a single changed word. For now the attackers are clearly winning. Her answers point in consistent directions: richer representations and consistency checks for robust AI, many combined sensors for self-driving cars, differential privacy for training data, and clear data ownership backed by privacy-preserving technology for a fairer data economy. Add her optimism about program synthesis, her journey from physics to computer science, and a thoughtful reflection on meaning, and this is a rich episode for anyone curious about how secure and trustworthy AI systems really are.

This is a summary of "#95 – Dawn Song: Adversarial Machine Learning and Computer Security" from Lex Fridman Podcast, giving you the GIST so you can decide if you want the REST. Listen to the full episode.