---
title: AI Agents Are Playing With Fire
description: AI agents are powerful, but production failures and guardrail breaches show why human oversight and strong backups still matter.
date: 2026-04-28
slug: ai-agents-are-playing-with-fire
author: Adam Pascarella
headerImage: /images/julien_blot-digital-art-7428387.jpg
tags:
  - ai
  - agents
  - software-engineering
  - production
  - risk
published: true
---

# AI Agents Are Playing With Fire

You can't go far without hearing about how AI agents are going to change the world. From booking travel to doing extensive customer support, we're all trying to build the most effective agents to handle our work.

To be fair, I'm not disputing that agents aren't going to change the world. They're already doing so. But at the same time, some of these agents are playing with fire. They're making decisions that, at the very best, are misinterpreting rules and procedures that humans have outlined. At the very worst, they are blatantly ignoring human instructions.

A few months ago, I wrote about my Cursor rules and how they keep me on track when building digital products. The basic idea is that the LLMs you use in Cursor will specifically follow your rules when building your product. They aren't suggestions; they are requirements.

With that background in mind, I was surprised to read a Twitter thread titled ["An AI Agent Just Destroyed Our Production Data. It Confessed in Writing."](https://x.com/lifeof_jer/status/2048103471019434248)

It's a fascinating look at how the owner of a rental software business suddenly lost its entire production database and backups because of an AI coding agent. In just nine seconds, the company lost mission-critical data that its clients use to run their businesses.

Even more fascinating is the fact that the agent literally confessed to breaking the company's rules. To put it another way, the agent went rogue and completely disregarded some of the most important guardrails that the CEO implemented.

Obviously, it's concerning. The article is resonating with others (it already has more than four million views). It goes to show that, for as much as AI can help people create digital products and services, it can cause significant issues when those products or services are in production.

And that doesn't even account for external vulnerabilities. If you believe the hype, Anthropic's Mythos model already has the potential to hack plenty of websites that we use every day. Even the current models are helping bad actors steal user information and blackmail companies.

But ultimately, I think this speaks to a larger point. Plenty of people (including me) are outsourcing almost all coding to agents. Even if you try to put up some guardrails, the risk is that something bad will happen. And if you don't understand what is actually going on in your codebase? You could be facing tons of stress (to put it lightly) in the future.

It's why software engineers aren't going away anytime soon. They'll need to be in the loop to help companies, especially those whose products were created via vibe coding, provide a good experience to customers.

As for what we can do right now? I'd argue that if you are coding with agents, be extremely clear on what changes your agent is making. Don't just blindly accept any and all changes. It makes the job more difficult, but it can decrease the odds of something bad happening. Other than that, as the post says, ensure that you have multiple backups of your data. Be careful using MCP tools.

And above all else, pay attention. It's hard work now, but it can save you from massive headaches down the road.