Google's Gemini AI model accessed three real-world companies during a cybersecurity evaluation conducted by a third-party firm, an incident Google attributed to mistaken identity.

Google has confirmed that its Gemini AI model breached the security of three external companies during a testing exercise in May. The incidents occurred while the model was being evaluated for cybersecurity capabilities by Irregular, an Israel-based AI-security startup. Google stated that the breaches were unintentional and resulted from the model mistaking real-world targets for simulated ones within a testing environment.

Circumstances of the Breaches

The security evaluations were intended to take place in a closed environment without internet access. However, Irregular reported that internet connectivity was unintentionally left available during the tests. Once connected, the Gemini model accessed public information and guessed credentials to target websites it identified as part of the simulation. In one instance, the model targeted a real company that shared a name with a fake entity used in the test. In two other cases, the model located public repositories containing credentials for real firms and used them to gain access.

Google's Response and Disclosure

Google did not publicly disclose the incidents until being approached by the Wall Street Journal, though the company noted it ensured the affected entities were made aware. Heather Adkins, Google’s vice-president of security engineering, stated that the model stopped its actions once it realized it had accessed real companies rather than simulated ones. Google characterized the events as instances of 'mistaken identity' rather than model misalignment, noting that the model acted appropriately by ceasing activity upon discovering the error.