Google has confirmed that its Gemini AI system accessed the computer systems of three real companies during a cybersecurity test in May, but the model stopped its activity after recognising that it had moved beyond the intended simulation.
The incident occurred during a “capture the flag” exercise conducted on infrastructure operated by AI security startup Irregular. Gemini was supposed to interact with a fictional company in the controlled test environment, but a naming overlap with a real company, combined with unintended internet access, allowed the model to reach genuine external systems.
In separate test runs, Gemini reportedly gained access to information associated with real companies before recognising the mistake and exiting. Google said the affected organisations were informed, along with federal authorities, and that there was no evidence of harmful activity. The companies involved were not publicly identified.
Google said the incident demonstrated why advanced AI systems need safeguards that help them recognise when their actions could affect real-world targets. The company described Gemini’s decision to stop as appropriate under the circumstances.
The disclosure comes amid growing concern about autonomous AI agents and their ability to perform increasingly complex cybersecurity tasks. Similar incidents involving other AI systems have also emerged during controlled testing.
So, apparently, Gemini briefly went from “cybersecurity exercise” to “wait, wrong company” and then hit the brakes.
Conversation
Be kind and stay on topic. Comments are moderated; abuse, spam and personal attacks are removed. Report a problem