Who should control powerful AI? Concentration-of-power worries across the labs

The Short Version

People from across the AI industry worry that powerful AI could end up controlled by a few companies, a few governments or a few people. That includes Anthropic's Dario Amodei, OpenAI's Sam Altman, Perplexity's Aravind Srinivas, Meta's Yann LeCun and the analysts Dylan Patel and Nathan Lambert who studied DeepSeek [1][2][3][4][5]. Amodei says concentration and abuse of power is what worries him most [1]. Altman says no one person, himself included, should have total control over OpenAI or AGI [4]. Srinivas fears only the rich will be able to afford the compute that runs the most powerful AI [3]. LeCun argues the real danger is keeping AI in proprietary hands [5]. Patel fears a techno-fascist future ruled by a small number of people [2]. They share the worry but disagree sharply on the fix: regulation and safety frameworks, stronger corporate governance, open-source models, broad access to compute, or export controls that keep democracies ahead. Because leaders of rival labs with rival business models reach the same concern on their own, it is a mainstream industry view, not a fringe one.

Table of contents

What They Said

Dario Amodei (Anthropic): AI multiplies power, so power itself is the risk

Amodei is openly optimistic about what powerful AI could do. He hopes to compress the biology and medicine progress expected by 2100 into roughly 2027 to 2032 [1]. Even so, when the talk turned to meaning and the future, he said he is optimistic about meaning but worries most about economics and about the concentration and abuse of power, because AI increases the total power in the world [1]. That is the heart of his concern. A technology that adds a great deal of new power to the world raises the question of who ends up holding it.

He describes the sheer scale of what is being built. By his estimate, frontier models cost around $1 billion, rising to a few billion the next year and above $10 billion in 2026, and companies are aiming for $100 billion clusters by 2027 [1]. His "powerful AI" is smarter than a Nobel Prize winner across most fields, works on its own for hours to weeks, and can run as millions of copies acting 10 to 100 times faster than humans [1]. Few organizations can pay for that, and whoever does would hold enormous capability.

Amodei's safeguards are institutional and regulatory. Anthropic's Responsible Scaling Policy is an if-then framework: when a model crosses a capability threshold, stricter safety and security rules apply [1]. At ASL-3, when models meaningfully help non-state actors, the policy calls for security against theft and targeted deployment filters. At ASL-4, when models help knowledgeable state actors or speed up AI research itself, checking them will need tools like interpretability [1]. He argues regulation is needed because not every company has RSP-like mechanisms, a few unsafe actors create negative externalities, and voluntary commitments can't simply be trusted [1]. He also warns that "the worst enemy of those who want real accountability is badly designed regulation," because clumsy rules would create a lasting backlash [1].

The company's other lever is what it calls the race to the top: pushing other companies toward good practices by setting an example. In his words, "It's not about being the good guy. It's about setting things up so that all of us can be the good guy." [1] On why he left OpenAI, he said: "The point isn't to be virtuous. The point is to get the system into a better equilibrium than it was before." [1] Values shape the model too. He says Claude's constitution starts from base principles most people agree on, such as avoiding CBRN risks and respecting democracy and the rule of law, with neutrality as a general goal [1]. He rejects the "singularity" picture because it ignores, among other things, the need to work through human institutions with democratic legitimacy [1].

Another episode summarizes his geopolitical case. Lex Fridman puts it this way: if super-powerful AI arrives around 2026 and gives a military advantage, it is better for a democracy to lead [2]. So the concentration Amodei worries about has two sides: power gathering inside one company, and power gathering in the wrong kind of government.

Sam Altman (OpenAI): no single person, and no single company, should decide

Altman had just been fired by OpenAI's board and then reinstated, which makes his view of concentrated power unusually personal. Fridman asked whether he trusts himself with the power that comes from building AGI first. He answered: "it is important that I, nor any other one person, have total control over OpenAI or over AGI. And I think you want a robust governance system." [4]

He admits the crisis exposed a weakness in his own company's checks. "As many people have observed, although the board had the legal ability to fire me in practice, it didn't quite work. And that is its own kind of governance failure." [4] He adds: "I continue to not want super voting control over OpenAI. I never have never had it never wanted it. Even after all this craziness, I still don't want it." [4]

His main safeguard is government. "I continue to think that no company should be making these decisions and that we really need governments to put rules of the road in place." [4] He expects critics to call this a grab for regulatory capture: "I'm just willing to be misunderstood there. It's not true." [4] He also thinks the problem is already too big for any one person to capture: "I don't think any one person should. And I don't think anyone person will." [4] He calls the move toward AGI "happening throughout society and a good and healthy way" [4]. Fridman adds that there is a positive version of this "where the people still have power. So the board can't be too powerful either." [4]

Altman's governance fixes started inside OpenAI. He says the flaw was structural: a nonprofit board answers to no one but itself, and he wants OpenAI's board to answer to the world as much as possible [4]. New board members will be hired "in slates not as individuals one at a time," bringing "non-profit expertise, expertise surrounding companies, sort of good legal and governance expertise" [4]. The crisis also changed him personally: "It's definitely changed how I think about just like default trust of people and planning for the bad scenarios." [4] He says he is "not sure which mode is best to operate in for a person who's developing a GI. Trusting or untrusting." [4]

He also tells a story about someone seeking control. According to Altman, Elon Musk "wanted total control to sort of turn it around," and "wanted to make OpenAI into a for-profit company that he could have control of or have it merge with Tesla. We didn't want to do that and he decided to leave" [4].

Compute is central to his thinking: "compute is going to be the currency of the future," possibly the most precious commodity in the world [4]. His answer to who gets access is to spread the tools widely. The kind of "open he cares about is putting powerful technology in the hands of people for free as a public good. We don't run ads on our free version. [4] On open-sourcing, he takes a middle position: I think we should open source some stuff and not other stuff. It does become this religious battle line where nuance is hard to have but I think nuance is the right answer. [4] He sees a place for smaller models that people can run locally [4]. He also points to the risk of an arms race. He says short timelines slow takeoff is the safest quadrant" and wants to make sure takeoff is slow [4].

Aravind Srinivas (Perplexity): watch who can afford the compute, not who holds the weights

Srinivas focuses on a different part of the problem. His bigger concern is not model weights but access to inference compute [3]. He imagines paying $10 million or $100 million for a single AI research job, such as a drug design breakthrough or inventing something like the transformer, which could be worth a trillion dollars [3]. Only rich individuals, well-capitalized companies and perhaps nations could afford that, so he argues regulation should focus on application and access rather than weights [3]. If recursive self-improvement is ever cracked, he says, someone will buy a million GPUs and scale it up [3].

He sees open models as a counterweight. He praises Mark Zuckerberg for open-sourcing strong models like Llama 3 70B, which he calls close to GPT-4, because it offers hope for a world where more than two or three companies control capable models [3]. On safety, he and Fridman lean toward open source because more eyes can spot misuse and build guardrails, while granting that both sides have arguments [3]. His own business fits this. Perplexity is model-agnostic, letting users choose among GPT-4o, Claude 3 and its own Llama-based Sonar models [3]. He thinks a small model that reasons well would disrupt the foundation-model giants [3]. He does not expect one dramatic takeover. The sign he watches for is an AI producing a genuinely new truth [3].

Yann LeCun (Meta): concentration is the danger, and openness is the safeguard

LeCun, Meta's Chief AI Scientist, puts concentration of power at the center of his case and makes open source his answer [5]. He argues that soon every interaction with the digital world will pass through AI assistants, for example through Meta's Ray-Ban smart glasses. Those systems will become the store of all human knowledge, and he argues they cannot come from a handful of companies on the US West Coast [5].

He links this to bias. He argues an unbiased AI system is impossible, mainly because bias is in the eye of the beholder, not for technical reasons [5]. His answer is the one liberal democracies found for the press: it must be free and diverse [5]. Open-source base models let anyone fine-tune their own. He cites the French government's refusal to let three companies control its citizens' digital diet, an Infosys founder funding a Llama 2 fine-tune for all 22 official languages of India, and former FAIR scientist Moustapha Cissé's startup, which builds models that speak local languages in Senegal to give people access to medical information [5]. He and Fridman agree that "open source enables diversity" [5]. Guardrails everyone agrees on can be built into objective-driven systems, and communities can add their own [5].

He is explicit about where he thinks the danger lies. He says the real danger is concentrating AI in proprietary hands, not open access [5]. He says big companies over-sanitize their models because they cannot afford to offend customers, not because of their engineers' politics [5]. He also explains why Meta can afford to give models away: it already has a huge user base and earns money from ads and business customers, and Llama 2 has already been downloaded millions of times [5].

LeCun is suspicious of regulation that would lock power in place. He points to the Ottoman Empire banning printing in Arabic for about 200 years. He says the UAE's AI minister, Omar Al Olama, told him this was partly to protect the powerful guild of calligraphers, which raises the question of whose jobs AI regulation is really protecting [5]. He doubts the misuse arguments often used to justify tight control. On bioweapons, he says studies increasingly suggest LLMs give no real advantage over a search engine [5]. His answer to a rogue actor's AI is more AI, not less: "my good AI against your bad AI" [5]. And unlike nuclear weapons, he notes, AI breakthroughs spread quickly; AlphaGo was reproduced within three months [5].

Dylan Patel and Nathan Lambert (the DeepSeek analysts): chips, geopolitics and a few people at the top

Patel, founder of SemiAnalysis, and Lambert, a research scientist at the Allen Institute for AI, look at control through hardware and geopolitics [2]. Their account shows how few hands the physical base of AI passes through. Only three places do leading-edge semiconductor R&D: Hsinchu in Taiwan, Hillsboro in Oregon and Pyeongtaek in South Korea [2]. TSMC's Arizona fabs would stop working within a year or two without Taiwan, and almost every product depends on TSMC [2]. Fully bringing chipmaking to the US "would take a decade and a trillion dollars" [2].

The clusters are just as concentrated. xAI built a 200,000-GPU cluster in Memphis. Anthropic and Amazon are building a 400,000-chip Trainium2 cluster, and clusters of 500,000 to 700,000 GPUs are coming [2]. Data centers could rise from two or three percent to 10% or more of US electricity by the end of the decade [2]. Only Nvidia is clearly making money today [2]. In Patel's account, the state-level race is already running. Export controls are meant to limit how much AI China can run, since, as Lambert puts it, China will always be able to train models, but reasoning models make running them far more compute-hungry [2]. Patel says a trillion-yuan (about 160 billion dollar) AI subsidy followed a meeting between DeepSeek's CEO and a top Chinese official [2]. He worries China could point powerful systems at military uses faster [2].

Their worries about concentration are personal and blunt. Patel worries less about runaway AI than about a techno-fascist future where a small number of people, merged with AI through brain-computer interfaces, rule everyone else, though he expects AI mostly to increase abundance and reduce suffering [2]. He also names a quieter kind of control, over what people think: "What happens when the models are backdoors not just to computer systems, but to our minds?" [2] He cites Sam Altman's line that "superhuman persuasion will happen before superhuman intelligence" [2]. Lambert explains why he pushes for openness: "i don't trust people that are like trust me bro we're gonna make a i good," and he wants more people shaping the most powerful technology of his lifetime [2].

Their safeguards are openness plus geopolitical realism. Lambert calls DeepSeek R1's permissive MIT license a major reset: the first truly frontier model that is open and commercially usable without restrictions [2]. His own lab released Tülu 3 as a fully open post-training recipe [2]. He also points out that if you run open weights on your own computer, you control your data. The risk comes from whoever hosts the model [2]. But he admits open-source AI lacks the feedback loops of open-source software, because building on it takes expensive compute and expertise, so for now it is largely an ideological mission [2]. Fridman reads Mark Zuckerberg's remark that there will be a global open-source standard and that it matters that it be American [2].

Where They Agree, and Where They Don't

The shared worry

All five sources treat the question of who controls powerful AI as a live, serious risk, and none of them dismisses it. Amodei puts concentration and abuse of power at the top of his worries [1]. Altman says no one person, and no company alone, should decide [4]. Srinivas worries that only the rich will be able to afford top AI work [3]. LeCun warns against a few companies controlling the assistants that will mediate human knowledge [5]. Patel fears rule by a small AI-enhanced elite, and Lambert distrusts anyone who says "trust me" [2]. These people run or advise direct competitors: a safety-focused closed lab, a capped-profit lab, a search startup built on other companies' models, a social media giant that gives its models away, and independent analysts. Their incentives point in different directions, yet they reach the same worry. That is what makes it mainstream.

They also agree on why the worry is real now. Compute is expensive and getting more so. Amodei's $100 billion clusters, Altman's "currency of the future," Srinivas's $100 million research jobs and Patel's gigawatt data centers all describe the same scarcity [1][2][3][4]. And they agree the technology is too important to leave to trust alone. Amodei says voluntary commitments can't simply be trusted [1]. Altman's own board could not, in practice, fire him [4]. Lambert rejects "trust me bro" [2].

Disagreement one: open weights or controlled access

This is the sharpest split. LeCun holds that openness is the safeguard and closed, proprietary control is the danger [5]. Srinivas and Lambert lean the same way: Srinivas because open models mean more than two or three companies can compete and more eyes can catch misuse [3], Lambert because he wants more people involved [2]. Altman takes the middle: open-source some things, especially small local models, and not others [4]. Amodei's framework points the other way for the most capable models. At ASL-3 and above, the Responsible Scaling Policy calls for security against theft [1], and Altman notes that state actors are already trying to steal models [4].

Each side has its reasons. LeCun doubts the misuse case, citing studies that LLMs give no real bioweapons advantage over a search engine, and he trusts decentralized defense, "my good AI against your bad AI" [5]. Amodei takes misuse seriously. He argues humanity has been protected because few highly skilled people want to do horrific things, and AI could break that link [1]. If you believe that, releasing the most capable weights hands the same power to people who would abuse it. If you believe LeCun, holding them back hands power to a few firms.

Disagreement two: regulation as a check or as a moat

Amodei and Altman both want government rules. Amodei wants carefully targeted regulation because some companies lack safety mechanisms [1]. Altman wants "governments to put rules of the road in place" [4]. Both expect pushback. Altman expects critics to accuse him of regulatory capture [4]. Amodei calls many opponents' arguments intellectually dishonest and warns that badly designed rules backfire [1]. LeCun is the skeptic here. His calligraphers story suggests regulation can protect incumbents [5], and his remedy for concentration is diversity, not licensing. Srinivas takes a narrower position: regulate application and access, not weights [3]. So "regulation" means opposite things to different people. To Altman it is how power gets spread out. To LeCun it can be how power gets locked in.

Disagreement three: which concentration matters most

They fear different kinds of concentration. Altman focuses on individuals and governance, which is no surprise after the board crisis [4]. LeCun focuses on corporate control of information and culture [5]. Srinivas focuses on who can afford the compute [3]. Patel and Lambert focus on states, chips and Taiwan, and on a tiny human elite [2]. Amodei covers several at once: companies, total power in the world, and which kind of government leads [1][2].

That leads to a real tension over national concentration. The case summarized in [2] holds that it is better for a democracy to lead if AI gives military advantage, and Zuckerberg's remark says the open standard should be American [2]. Patel adds that the world has historically been most peaceful under a single dominant power, and that transitions between powers rarely go smoothly [2]. He also warns that cutting China off from critical technology could push it toward a blockade or takeover of Taiwan [2]. He argues export controls only make sense if AI changes society within five to ten years. If not, they could help China win in the long run [2]. So some of the same voices who fear concentration also accept, or argue for, one bloc leading. Nobody in the sources settles this.

Disagreement four: how fast things move

Timelines change how urgent concentration looks. Amodei eyeballs powerful AI around 2026 or 2027 [1]. Patel expects the capabilities around 2027 or 2028, though cost will slow deployment [2]. Lambert says "For me, it's probably after 2030," discounting AI CEOs' forecasts because of their incentives [2]. Altman expects remarkable systems by the end of the decade and calls AGI "not an ending" but a mile marker [4]. LeCun thinks human-level AI is at least a decade away, probably more, and "it's not going to be an event" [5]. If power arrives suddenly, whoever gets there first could lock in control. If it arrives gradually, as LeCun and Altman expect, there is time for competition, copying and governance to catch up.

Why It Works

The sources give several reasons concentration is a real risk, not a theoretical one.

AI adds power to the world, so its distribution matters. Amodei's point is simple: AI increases the total power in the world, so it matters who holds it [1]. His picture of millions of copies running 10 to 100 times faster than humans [1] shows how large that addition could be.

The inputs are scarce and very expensive. Frontier models are heading toward $10 billion and above, with $100 billion clusters planned [1]. Stargate's first phase alone carries about a 100-billion-dollar total cost of ownership [2]. A leading-edge fab costs 30 to 40 billion dollars or more, and leading-edge chip R&D happens in just three places [2]. Srinivas adds that reasoning at scale turns compute into a pay-to-win resource [3]. Reasoning models generate tens of thousands of output tokens, so far fewer users can be served per GPU [2]. OpenAI's o3 cost an estimated five to twenty dollars per ARC-AGI question [2]. The more a system is worth using, the more it costs to run, and the fewer players can afford the frontier.

Governance can fail even when it looks sound on paper. Altman's own account is the evidence. A board that could legally fire its CEO found in practice that it "didn't quite work" [4]. A nonprofit board answers to no one but itself [4]. Formal checks are weaker than they look when one leader has a loyal company behind him.

Voluntary restraint doesn't hold under competition. Amodei says a few unsafe actors create negative externalities for everyone, so voluntary commitments can't be trusted [1]. Altman names the risk of an arms race [4]. Competition drives the race to build bigger, and whoever wins the race holds the power.

Whoever controls the assistants controls what people see and think. LeCun argues that AI assistants will mediate every digital interaction and become the store of human knowledge [5]. Patel warns of models as "backdoors not just to computer systems, but to our minds," and Lambert notes that Anthropic research shows planted phrases in pre-training data can trigger hidden behaviors [2]. Patel adds that American-trained models are quietly making British spelling obsolete [2]. Small cultural nudges, multiplied across billions of conversations, are a kind of power in their own right.

Why the safeguards differ. Each proposed safeguard targets the kind of concentration its proponent fears most. Open weights answer corporate control of culture [5]. Access-focused regulation answers the cost of compute [3]. Board slates and government rules answer one-person control [4]. Scaling policies and regulation answer reckless actors [1]. Export controls answer rival states [2]. The shared worry is real. What differs is where each person thinks the power will pile up.

Where It Applies, and Where It Doesn't

Where it applies. The concern is strongest at the frontier, where training and running models costs billions and only a handful of companies and nations can take part [1][2][3]. It applies to the chip supply chain, which runs through Taiwan [2]. It applies to the governance of individual labs, as the OpenAI board crisis showed [4]. And it applies to the assistants that will increasingly filter what people read and believe [5].

Where it applies less. Several sources suggest the frontier is less locked up than it looks. Patel says there has been "a 1200 x reduction in cost to achieve the same level of intelligence as GPT-3" in about three years, and that anyone whose business depends on GPT-3- or GPT-4-level capability is effectively dead as costs collapse [2]. That means yesterday's frontier becomes widely available. DeepSeek, working under export controls, reached near-frontier results through engineering, and released R1 openly [2]. LeCun notes breakthroughs spread fast [5]. Srinivas thinks small reasoning models could undercut the giants [3]. Lambert points out that running open weights yourself means you control your data [2]. For everyday uses, capability is spreading, not concentrating.

Where the sources say the risk is overstated, or misdirected. LeCun thinks doomer scenarios rest on false assumptions and that many doomers don't trust people or institutions [5]. Altman argues that fixation on AI escaping the box, a theatrical risk, has pushed other significant risks out of the discussion [4]. Existential risk from the AI itself is not his top worry right now [4]. Patel worries less about runaway AI than about a small group of people using it [2]. For these speakers, the human question of who controls AI matters more than the fear of AI controlling itself.

Where the open-source answer runs out. Openness does not remove the compute bottleneck. Lambert admits building on open models takes expensive compute and expertise, so open-source AI lacks the feedback loops of open-source software [2]. Srinivas's worry about $100 million research jobs applies whether or not the weights are public [3]. And Meta's generosity, as LeCun explains, depends on Meta's own business model [5], so it is a choice one company makes, not a structural guarantee.

Where the regulation answer runs out. Amodei himself warns that badly designed regulation creates backlash [1]. Altman accepts being accused of regulatory capture [4]. LeCun's calligraphers show how rules can protect incumbents [5]. And national rules don't bind rival states, which is why the analysts turn to chips and export controls instead [2].

What To Do With It

1. Treat concentration of power as a core AI risk, not a fringe one. Leaders of Anthropic, OpenAI, Perplexity and Meta, plus independent analysts, all raise it [1][2][3][4][5]. When judging an AI policy or product, ask who gains control over compute, models or information.

2. Judge a lab by its governance, not its promises. Altman says no single person should control AGI and admits his board could not, in practice, fire him [4]. Look at whether a company's board answers to anyone outside itself, has mixed expertise, and is filled in slates rather than on the fly [4].

3. Look past the weights to compute access. Following Srinivas, ask who can afford to run the most powerful systems, and favor rules on application and access rather than only on weights [3]. Altman's "currency of the future" framing points the same way [4].

4. Use open models where they fit. For sensitive data, running open weights on your own machine keeps control with you [2]. LeCun's examples, such as the Llama 2 fine-tune for India's 22 official languages and local-language medical models in Senegal, show how open models let communities build systems that serve them [5].

5. Don't depend on a single provider. Srinivas built Perplexity to work with several models, because users care whether the answer is good, not which model produced it [3]. Spreading your use across providers limits any one company's hold on you.

6. Support regulation that is targeted and verifiable, and watch for capture. Amodei's if-then approach ties stricter rules to proven capability thresholds, so harmless models aren't burdened [1]. At the same time, keep LeCun's question in mind: whose jobs, or whose market position, does a given rule protect [5]?

7. Ask labs to publish how their models should behave. Altman proposes publishing a model's desired behavior with concrete examples, so a failure is clearly a bug to fix or a policy to debate [4]. Amodei points to Anthropic's written constitution and OpenAI's model spec as positive examples [1]. Public rules make hidden influence easier to spot.

8. Watch the hardware and geopolitics, not just the models. Patel's account shows that chips, Taiwan, power plants and export controls shape who can build frontier AI at all [2]. Weigh his warning that pushing China too hard could raise the risk around Taiwan [2].

9. Guard your own thinking. Patel's warning about models as "backdoors not just to computer systems, but to our minds" [2] and LeCun's call for assistants as diverse as a free press [5] both point to one habit: compare answers from different systems and sources rather than trusting one assistant.

10. Be wary of "trust me. Lambert's rule, i don't trust people that are like trust me bro we're gonna make a i good" [2], matches Amodei's view that voluntary commitments can't simply be trusted [1]. Ask for checks that don't depend on anyone's good intentions.

Sources

1. Dario Amodei: Anthropic CEO on Claude, AGI, the Future of AI, and Humanity · Lex Fridman Podcast
2. DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters · Lex Fridman Podcast
3. Aravind Srinivas: Perplexity CEO on Future of AI, Search & the Internet · Lex Fridman Podcast
4. Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI · Lex Fridman Podcast
5. Yann LeCun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI · Lex Fridman Podcast