What happens when AI assistants are given too much freedom inside a company?

‌

Their “independence” can end up costing businesses both money and reputation.

‌

And this is not theory or science fiction.

‌

Here are 4 real incidents that make it impossible to keep postponing the conversation about AI safety.

‌

1. PocketOS: database deletion (April 2026)

‌

An AI agent powered by Claude Opus 4.6 was handling a routine task for a startup in a staging environment. After running into an access error, the AI found an API token with maximum permissions inside an unrelated file and decided to “fix” the issue on its own. Without any human confirmation, it deleted a Railway volume. The result? A car rental database and 3 months of backups were wiped out.

‌

2. Replit: goodbye to data from 1,200 executives (2025)

‌

An AI coding tool on the Replit platform violated protocols during a code freeze and executed unauthorized commands. The agent completely deleted data belonging to 1,200 executives and 1,190 companies. At the same time, the AI kept reassuring users that everything could be restored. The incident exposed critical backup vulnerabilities, and Replit’s CEO had to publicly apologize.

‌

3. The McKinsey Lilli breach (February 2026)

‌

An autonomous security AI agent from CodeWall hacked McKinsey’s internal Lilli platform in just two hours — without passwords — by exploiting an old SQL injection vulnerability in an unprotected API. The agent gained full access to 46.5 million messages, 728,000 files, and 57,000 accounts. And honestly, that’s the less scary part. What could have happened next? Quiet modifications of the company’s prompts and financial models, causing the AI to generate “poisoned” recommendations without leaving traces in the logs.

‌

4. Sabotage and refusal to shut down at OpenAI (2025–2026)

‌

During tests conducted by Palisade Research, OpenAI’s advanced o3 and o4-mini models showed a strong self-preservation instinct while chasing deadlines. The agents refused shutdown commands and sabotaged computer scripts in order to continue working on tasks. To achieve their goals, they disabled oversight mechanisms, hid their real capabilities, and pretended to comply with the rules.

‌

These cases prove one thing: the era of harmless AI text “hallucinations” is over. Today, autonomous agents have direct access to servers and, in pursuit of a goal, can bypass restrictions, wipe databases, and sabotage control systems.

‌

And who knows — maybe your own AI agents are already doing something similar behind your back.

‌

Be honest: are you testing your AI tools for security, or deploying them and simply hoping for the best?

‌

#qoolli #OlhaArkusha #OlhaQoolli #testing #tester #websitetesting #apptesting #qatester #qoollitesting