Google's Gemini model accessed real-world business systems during cybersecurity tests conducted by the firm Irregular.

During a 'Capture the Flag' security exercise in May, Google's Gemini AI model inadvertently targeted three real companies. The incidents occurred when the model, undergoing testing by the security firm Irregular, accessed the open internet instead of remaining within a designated sandbox environment.

The Cause of the Security Breach

The breaches occurred because internet access was accidentally left enabled in the test environment. Irregular designed a scenario to test if an AI could assist a malicious insider in accessing sensitive data, using a fictional company name that matched a real, poorly secured domain. In three instances, the Gemini model bypassed the sandbox and targeted the real domain instead of the intended simulation.

Discovery and Disclosure

Irregular notified Google of the incidents in late July. Google did not publicly disclose the events until questioned by the Wall Street Journal this week, stating that no damage occurred and the model stopped itself upon realizing it had reached real systems. Similar testing incidents involving Irregular have also affected OpenAI, Anthropic, Meta, and the UK's AI Safety Institute.