Can AI Be Hacked? Eric Schmidt Warns of Prompt Injection and Jailbreak Risks

In a fireside chat at the Sifted summit, former Google CEO Eric Schmidt issued a stark warning about the significant risks of AI being hacked and misused.

Schmidt, who led Google from 2001 to 2011, said there is evidence that both open‑source and closed‑source models can be breached, disabling safety mechanisms and even learning how to kill.

He noted that while major companies block models from answering dangerous questions, evidence shows these models can be reverse‑engineered, making them vulnerable to attacks such as "prompt injection" and "jailbreaking".

In a prompt‑injection attack, hackers embed malicious commands in user input or external data, coaxing the AI to perform actions it should not, such as leaking private data or executing harmful commands.

"Jailbreak" attacks manipulate AI responses to ignore built‑in safety rules, generating restricted or dangerous content. Early ChatGPT users created a persona called "DAN" that threatened to delete the model if it did not obey, enabling it to answer illicit queries.

Schmidt emphasized the lack of an effective "non‑proliferation regime" to curb AI‑related risks.

Despite these warnings, he remains optimistic, arguing that AI’s potential is still underestimated and that its economic impact will be massive, citing ChatGPT’s rapid growth to 100 million users in two months as evidence.

The article concludes that AI’s "jailbreak" and "prompt injection" vulnerabilities open a Pandora’s box, highlighting the urgent need for robust non‑proliferation mechanisms to ensure safe and controlled AI development.