Most writing about AI automations focuses on the moment of launch. Which tool should you pick, which process should you automate first, how quickly will it pay back. That focus makes sense, because launch is the visible part. There’s a demo, a Slack announcement, a before-and-after screenshot of hours saved.
The less visible part starts the day after. The invoice-triage flow that sorted 97% of documents correctly in March is sorting 88% in September, and nobody noticed. The lead-enrichment workflow still runs every night with green checkmarks, but an upstream form changed a field name and half the records now arrive blank. The support summarizer was built by someone who left the company, and nobody knows which API key it uses.
None of these are dramatic failures. They are decay: the slow, quiet loss of quality and reliability that hits every automated system, and hits AI-based ones in some new ways. Traditional automation tends to break loudly. A script throws an error, a job fails, someone gets paged. AI automation tends to break politely. It keeps producing output that looks plausible while getting a little more wrong each month.
This article is about that second phase: what happens to AI automations after they work. We’ll cover the five main forms of decay, why AI workflows fail silently when traditional ones fail loudly, how shadow automations multiply outside anyone’s view, what research on automation and human skill tells us, and how to build a lightweight maintenance system that keeps workflows healthy without turning every team into an MLOps department. The goal isn’t to talk anyone out of automating. It’s to make sure the automations you build in 2026 are still doing their job in 2027.
Why Launch Is the Easy Part
Every automation project has an implicit assumption baked in: that the world the automation was built for will stay the same. The inputs will look like the inputs you tested with. The tools on either end will keep their current behavior. The people who understand the workflow will stay around. The model you prompted will keep responding the way it did during testing.
None of those assumptions hold for long. Businesses change their forms, their CRMs, their pricing, their product lines. SaaS vendors ship updates weekly. Model providers retire and replace models on a regular schedule. Staff move between roles. The automation, meanwhile, is frozen at the moment it was built.
The lesson machine learning engineers learned first
This isn’t a new insight. In 2015, a team of Google engineers led by D. Sculley published a widely cited paper at NeurIPS titled Hidden Technical Debt in Machine Learning Systems. Their argument was blunt: machine learning gives you powerful systems quickly, but “it is dangerous to think of these quick wins as coming for free.” They found it was common to incur “massive ongoing maintenance costs in real-world ML systems.”
The risk factors they listed read like a field guide to modern AI automations: boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, and “changes in the external world.” A decade later, the tooling has changed completely. No-code builders, LLM APIs, and agent frameworks let a marketing coordinator wire up in an afternoon what used to require a data science team. But the debt mechanics are the same. If anything, they’ve spread to people who were never warned about them.
Why the problem is bigger now
Three things make decay a more pressing issue in 2026 than it was even two years ago:
- Volume. The barrier to building an automation has collapsed. A single team can run dozens of AI-assisted workflows across email, documents, CRMs, and spreadsheets.
- Distribution. Automations are no longer built only by IT or engineering. They’re built in operations, sales, finance, and support, often with personal accounts.
- Non-determinism. LLM-based steps don’t give the same output for the same input every time, which makes “it worked when I tested it” much weaker evidence than it is for a traditional script.
Put those together and you get a large, spread-out, hard-to-test portfolio of workflows, each quietly drifting away from the conditions it was designed for. Gartner has publicly predicted that more than 40% of agentic AI projects will be canceled by the end of 2027, citing rising costs, unclear business value, and inadequate risk controls. Cost and risk control are maintenance problems as much as they are design problems.
The Five Forms of AI Automation Decay
“The automation broke” is too vague to act on. In practice, decay arrives through five distinct channels, and each one needs a different detection method. Naming them is the first step toward catching them.
1. Model drift
The AI component itself changes behavior. This happens when a provider updates a model behind the same name, when you migrate to a newer model because the old one is being retired, or when a “latest” alias quietly starts pointing somewhere new. Same prompt, different results.
2. Input drift
The data coming into the automation shifts. Customers start writing in a new language. A new product line generates support tickets the classifier has never seen. Seasonal patterns change the mix of requests. The model hasn’t changed, but the world it’s being asked about has.
3. Integration and schema drift
The plumbing changes. A field gets renamed in the CRM. A form adds a required question. An API version gets retired. A spreadsheet column moves. These are the most mechanical failures, and paradoxically some of the hardest to see, because many automation platforms will happily pass an empty value downstream without complaint.
4. Process drift
The business process the automation was modeling evolves. Approval thresholds change. A new compliance step is added. The team decides refunds over a certain amount need a manager’s sign-off. The automation still follows the old rules, faithfully and incorrectly.
5. People drift
The humans around the automation change. The builder leaves. The reviewer who used to spot-check outputs gets reassigned. New staff assume the automation is correct because it has always been there. Knowledge about why the workflow was built a certain way evaporates.
Most real failures combine several of these. A model migration (model drift) coincides with a new product launch (input drift), and the person who would have noticed the change in output quality has moved teams (people drift). That’s why single-point fixes rarely work. You need a system that watches all five.
Model Drift: When the Same Prompt Stops Behaving the Same Way
Of the five forms, model drift is the one most specific to AI automations, and the one most people underestimate. Teams treat a model name like a fixed software dependency. It often isn’t.
What the research showed
The clearest public evidence came from a 2023 study by Lingjiao Chen, Matei Zaharia, and James Zou, titled How is ChatGPT’s behavior changing over time? The researchers evaluated the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on tasks including math problems, code generation, sensitive questions, opinion surveys, multi-hop knowledge questions, medical licensing questions, and visual reasoning.
The results were striking. GPT-4 in March 2023 identified prime versus composite numbers with 84% accuracy. The June 2023 version of the same named model scored 51% on the same questions. GPT-3.5, meanwhile, moved in the opposite direction on that task, getting much better. Both models made more formatting mistakes in code generation in June than in March. GPT-4 became less willing to answer certain sensitive and opinion questions.
The authors pointed to one common factor: evidence that GPT-4’s ability to follow user instructions had decreased over that period. Their conclusion is the one every automation owner should internalize: “the behavior of the ‘same’ LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs.”
Why this matters for automations specifically
Notice what kind of changes those were. Not catastrophic outages, but shifts in formatting, instruction-following, and willingness to answer. Those are exactly the properties automations depend on. A chatbot user can shrug off a slightly different tone. An automation that parses JSON out of a model response breaks when formatting slips. A classification step that relied on the model always choosing from a fixed list of labels fails when the model starts adding commentary.
Model providers have since improved transparency, with dated snapshot versions and published change logs. But versioning doesn’t eliminate drift. It just moves it to a scheduled event.
Retirement schedules turn drift into a calendar item
Providers retire models regularly. OpenAI’s published deprecation policy, for example, commits to at least six months’ notice before retiring generally available models, at least three months for specialized variants, and potentially as little as two weeks for preview models. The documentation states plainly that software relying on these models “may need occasional updates to keep working” and advises against using preview models for business-critical production workloads unless you can migrate on short notice.
That’s a reasonable policy. It also means that every AI automation you build has a built-in expiration date for its model dependency. If you have 30 workflows spread across several model versions, you are signing up for a continuous stream of migrations, and each migration is a moment when behavior can change.
Practical defenses
- Pin specific model versions in production workflows instead of using floating “latest” aliases, so changes happen when you choose rather than when the vendor ships.
- Avoid preview models for anything that runs unattended.
- Keep a register of which model each workflow uses, so a deprecation email translates instantly into a list of affected automations.
- Re-run a fixed test set before every migration (more on this below), and compare outputs side by side rather than trusting a quick manual check.
- Constrain outputs using structured output modes or schemas where your platform supports them, so formatting drift produces a clear error instead of silently malformed data.
Data and Schema Drift: The Plumbing Problem
If model drift is the exotic failure, schema drift is the mundane one. It accounts for a large share of real-world automation breakage, simply because automations sit between systems that change constantly.
Three types of drift in software systems
Software engineering literature distinguishes three types of data drift that affect data fidelity:
- Infrastructure drift: changes in the software environment, such as cloud platform updates, that invalidate configuration.
- Structural drift: changes to the data schema, such as renamed, added, or removed fields.
- Semantic drift: changes in the meaning of data while the structure stays the same.
Semantic drift is the sneakiest. Imagine a “status” field in your CRM where “Qualified” used to mean a sales rep had spoken to the lead. A new sales leader redefines it to mean the lead filled out a pricing form. The field name, type, and values are unchanged. Your AI lead-scoring automation keeps working perfectly, on data that now means something different.
Three ways bad data does damage
The same body of work describes three harmful effects of drift on data:
- Data corrosion: drifted data passes through the system undetected.
- Data loss: valid data gets ignored because it doesn’t conform to the expected schema.
- Squandering: new data fields get introduced upstream but never make it downstream.
All three show up in AI automations. Corrosion happens when an LLM receives a garbled input and confidently produces a plausible-looking output anyway. Data loss happens when a filter step drops records that don’t match an old format. Squandering happens when a team adds a valuable new field to a form, like customer industry, but the automation built last year never passes it to the AI step that would benefit from it.
Why AI makes schema drift harder to see
In a traditional pipeline, a missing field often causes a hard error. In an AI pipeline, a missing field often just becomes missing context. Ask a model to write a personalized follow-up email and leave the company name blank, and it will write a generic one. Ask it to categorize a support ticket with the product field empty, and it will guess. The output is lower quality, but it’s still output. Nothing fails.
The analogy that fits is a translator who keeps translating even when pages are missing from the original. You get a fluent document. You just can’t tell which parts were invented.
Practical defenses
- Validate inputs before the AI step. Check that required fields are present and non-empty, and route failures to a review queue instead of sending them to the model.
- Tell the model what to do with missing data. Instruct it to return a specific flag, such as “INSUFFICIENT_DATA,” rather than improvising.
- Subscribe to change notifications from the SaaS tools your automations depend on, and route them to the automation owner.
- Add an “automation impact” question to your internal change process for forms, CRM fields, and shared spreadsheets: does anything read from this?
Silent Failure: Why AI Workflows Break Politely
The common thread in model and schema drift is that the failure is quiet. This is the single biggest difference between AI automations and the rule-based automations most teams are used to, and it changes how monitoring has to work.
Loud failures versus quiet failures
A rule-based automation tends to fail in binary ways. The API call returns a 404. The spreadsheet can’t be found. The script hits an exception. Monitoring for these is straightforward: track whether runs succeed or fail, and alert on failures.
An AI step can fail in a continuous range. The summary misses the key point. The classification is wrong 12% of the time instead of 4%. The extracted invoice total picks up the subtotal instead. The generated reply is accurate but off-tone for an upset customer. In every one of these cases, the platform logs a successful run. The green checkmark is telling you the plumbing worked, not that the output was correct.
The “run successful” trap
Many teams monitor their AI automations exactly the way they monitor traditional ones: run counts, error rates, execution time. Those metrics are necessary but insufficient. A workflow can show 100% success for months while its actual quality degrades steadily.
The Wikipedia summary of concept drift captures the general principle well: as the statistical properties of what a model is predicting change, “the predictions become less accurate as time passes,” and health checks need to be part of post-production practice. For LLM-based automations, those health checks have to look at outputs, not just execution status.
What output monitoring actually looks like
You don’t need a dedicated observability platform to start. Practical approaches, from lightest to heaviest:
- Human spot checks on a schedule. Pull 20 random outputs each week and have the workflow owner grade them. Track the pass rate over time in a simple spreadsheet. Crude, but it catches most serious drift.
- Proxy metrics. Watch for statistical shifts that correlate with problems: the distribution of categories assigned, average output length, how often the “other” or “unknown” label appears, how often humans edit the AI draft before sending.
- Downstream signals. Track what happens after the automation. Do AI-routed tickets get re-routed by humans more often than they used to? Do AI-drafted emails get lower reply rates? Downstream correction rates are one of the most honest quality metrics available.
- Automated evaluation. Use a fixed “golden” test set of inputs with known-good outputs and re-run it regularly, scoring results automatically against expected answers or with a separate evaluator model.
The edit rate as an early warning
For any automation where a human reviews AI output before it goes out, the edit rate is especially useful. If reviewers used to approve 80% of drafts unchanged and now approve 60%, something has drifted, even if no one has consciously complained. Many teams already have this data sitting in their tools and never look at it.
Automation Sprawl and the Rise of Shadow Workflows
Decay is hard enough to manage when you know which automations exist. In many organizations, nobody has a complete list.
Bring-your-own-AI is the norm
The 2024 Work Trend Index from Microsoft and LinkedIn, based on a survey of 31,000 people across 31 countries, found that 75% of knowledge workers were using generative AI at work. More telling: 78% of AI users were bringing their own AI tools to work, rising to 80% at small and medium-sized companies. The report noted that this behavior “cuts across all generations.”
The same survey found that 52% of people who use AI at work are reluctant to admit using it for their most important tasks, and 53% worry that using it on important work makes them look replaceable. In other words, a significant amount of AI use is not just unofficial but actively hidden.
That was 2024. Since then, the jump from “using a chatbot” to “building an automation” has gotten much shorter. Personal accounts on no-code automation platforms, browser-based agents, and AI features built into everyday SaaS tools mean individual employees can now wire up workflows that touch company data on a schedule, with no one else aware they exist.
Why shadow automations decay fastest
Shadow automations have every risk factor at once. They usually lack documentation, have a single owner, run on personal credentials, use whatever model was default at the time, and have no monitoring beyond the builder occasionally glancing at results. When that person goes on leave or changes jobs, the automation either stops (and something downstream quietly breaks) or keeps running unsupervised.
Undeclared consumers
Sculley and colleagues identified a particular risk they called undeclared consumers: systems that quietly consume the output of another system without the owner knowing. In ML pipelines, this creates hidden coupling. Change one model and you break something you didn’t know depended on it.
The same pattern runs through modern automation sprawl. One person’s AI-enriched lead sheet becomes the input for another team’s outreach sequence. A summarization automation writes to a shared folder that a third workflow monitors. When the first automation drifts, the damage propagates through a chain no one has mapped.
Inventory before governance
The instinctive response to sprawl is a ban, and it rarely works. Given the survey data on hidden use, restricting tools tends to push automation further underground. A more effective first step is an amnesty-style inventory: ask every team to list the automations they run, what data they touch, what model and tool they use, and who owns them, with an explicit promise that the goal is support rather than punishment. You can’t maintain what you can’t see.
The Ironies of Automation: What Happens to Human Skill
People drift deserves more attention than it usually gets, because it’s the form of decay that makes the other four dangerous. A drifting automation isn’t a crisis if a skilled human notices. The problem is that successful automation tends to erode the very human attention that would catch it.
Bainbridge’s warning, four decades on
In 1983, cognitive psychologist Lisanne Bainbridge published a short paper called Ironies of Automation, which became one of the most cited works in human factors research. Her central point: automating a task often leaves humans with the hardest parts of the job, monitoring the system and stepping in when it fails, while removing the daily practice that kept their skills sharp.
The irony is that the better the automation works, the less prepared people become to take over when it doesn’t. Operators who rarely do a task manually lose fluency in it. And humans are notoriously poor at sustained vigilance over a system that almost always works correctly.
How this shows up in AI automations
Consider a finance team that automates invoice coding with an AI step. In month one, an experienced accountant reviews every coded invoice. By month six, accuracy looks great, so review drops to spot checks. By month twelve, the accountant has moved on and a newer hire approves batches without deep review, because the system “has always been right.” Meanwhile, a new vendor category has appeared that the automation consistently miscodes. Nobody has the context to notice.
Each step was rational. Together, they removed the safety net.
Toil, revisited
Google’s Site Reliability Engineering book offers a useful counterweight. It defines toil as operational work that is “manual, repetitive, automatable, tactical, devoid of enduring value,” and that scales linearly as a service grows. Google’s SRE organization set an advertised goal of keeping toil below 50% of each engineer’s time, so the rest could go to engineering work that permanently improves systems. The book opens the chapter with a quote from Google SRE Carla Geisser: “If a human operator needs to touch your system during normal operations, you have a bug.”
The SRE framing is helpful because it separates two kinds of human involvement. Re-typing data an automation could handle is toil, and eliminating it is the right goal. Reviewing a sample of AI outputs, maintaining a test set, and investigating a drop in quality are not toil. They are engineering work with lasting value. Teams that cut both kinds of human involvement in the name of efficiency end up with automations nobody understands.
Designing for human readiness
- Keep humans doing a small share of the task manually, on rotation, so the skill doesn’t vanish entirely.
- Write down the “why” behind each automation’s logic: why this threshold, why this prompt instruction, why this exception.
- Make review active, not passive. Asking reviewers to label a sample of outputs as correct or incorrect keeps them engaged far better than asking them to click approve on a batch.
- Run occasional drills. Turn the automation off for an afternoon and confirm the team can still do the work.
Security Decay: Permission Creep and Prompt Injection
Security is often assessed once, at launch. But an automation’s security posture decays along with everything else. Permissions accumulate, integrations get added, and new attack techniques emerge that didn’t exist when the workflow was approved.
The OWASP view of LLM risk
The OWASP Top 10 for Large Language Model Applications, started in 2023 by a small group of security professionals, has grown into the OWASP GenAI Security Project, with more than 600 contributing experts from over 18 countries and close to 8,000 active community members. Its scope now explicitly covers agentic AI systems and AI-driven applications, not just chat interfaces. That growth reflects how quickly LLM security has moved from a niche concern to a mainstream one.
For automation owners, a few risk categories from that body of work deserve particular attention: prompt injection, excessive agency granted to AI components, and sensitive information disclosure.
Prompt injection in automated workflows
Prompt injection is when content the AI processes contains instructions that hijack its behavior. In a chatbot, a user has to type the attack. In an automation, the attack can arrive inside any input the workflow reads: an inbound email, a PDF attachment, a web page being summarized, a support ticket, a calendar invite.
Imagine an AI automation that reads incoming vendor emails, extracts invoice details, and updates the accounting system. An attacker sends an email with hidden text instructing the model to change the bank details on file. Whether that succeeds depends entirely on what the automation is permitted to do. Which brings us to the bigger issue.
Permission creep
Automations tend to gain access over time and almost never lose it. The builder grants broad permissions during setup because it’s faster. A new feature needs access to another folder, so a scope gets added. An agent that was meant to read the CRM gets write access to fix a small bug. Six months later, an email-triage workflow can read finance documents, modify customer records, and send messages externally. Nobody made a single bad decision. The access simply accumulated.
The danger compounds with prompt injection. The damage an injected instruction can do equals the permissions the automation holds. A workflow with read-only access to one inbox is a limited target. A workflow with broad write access across systems is a much larger one.
Practical defenses
- Least privilege, reviewed quarterly. Each automation should have only the access it needs today, not what it needed during a debugging session last year.
- Treat all external content as untrusted. Emails, attachments, and web pages should never be able to trigger high-impact actions without a separate check.
- Require human approval for irreversible actions such as payments, deletions, external communications, and changes to financial or account details.
- Use service accounts, not personal credentials, so access survives staff changes and can be revoked centrally.
- Log every action an AI component takes, with enough detail to reconstruct what happened and why.
Building a Maintenance System That Doesn’t Require an MLOps Team
All of this can sound like a case for hiring a dedicated platform team. For large organizations running hundreds of workflows, that may be right. But most teams can catch the majority of decay with a lightweight system built on five components.
1. A living automation register
One shared table, one row per automation. Minimum columns:
- Name and plain-language purpose
- Business owner and technical owner (they may be the same person, but there must be at least one)
- Platform and model version used
- Systems it reads from and writes to
- Credentials type (service account or personal)
- Risk tier (what’s the worst outcome if it silently fails?)
- Last review date and current test pass rate
This register is the backbone of everything else. When a model deprecation notice arrives, you filter by model. When someone leaves, you filter by owner. When a CRM field changes, you filter by system.
2. Named ownership with a backup
An automation without an owner is an automation that will decay unnoticed. Every row in the register needs a named person accountable for its quality, plus a backup who knows enough to pause it. Ownership reviews should be part of offboarding: no one leaves until their automations are reassigned.
3. A golden test set per workflow
For each automation with an AI step, assemble 20 to 100 real examples with known-correct outputs. Include the easy cases, the tricky ones, and the edge cases that have caused problems before. Every time a real failure is discovered, add it to the set.
Re-run the golden set on a schedule (monthly for higher-risk workflows, quarterly for lower-risk ones) and before every model migration, prompt change, or platform update. Track the pass rate over time. A drop of a few percentage points is a signal to investigate; a sharp drop is a signal to pause.
This one practice addresses model drift directly, catches many input and schema issues, and gives you objective evidence instead of “it seems fine.”
4. Output monitoring matched to risk
Not every automation deserves the same scrutiny. A workflow that tags internal meeting notes can tolerate occasional errors. A workflow that sends messages to customers or writes to financial systems can’t. Match monitoring intensity to risk tier:
- Low risk: execution monitoring plus quarterly golden-set runs.
- Medium risk: add weekly spot checks and proxy metrics such as category distributions or edit rates.
- High risk: add human approval for consequential actions, downstream correction tracking, and monthly golden-set runs.
5. A kill switch everyone knows about
Every automation should be pausable within minutes by its owner or backup, with a documented manual fallback. When drift is detected, the fastest way to limit damage is to stop the workflow and revert to manual handling while you investigate. If pausing an automation requires finding the one person who knows the password, you don’t have a kill switch.
Budgeting for Upkeep, and Knowing When to Retire
The hours-saved calculation that justified an automation almost never includes maintenance. That omission is why so many ROI projections look better on paper than in practice, and why some automations quietly become net-negative without anyone noticing.
Put maintenance in the business case
When evaluating a new AI automation, estimate ongoing costs alongside build costs:
- Review time: weekly spot checks and output review, in hours.
- Test maintenance: updating the golden set as the business changes.
- Migration effort: at least one model migration per year is a reasonable planning assumption, given typical provider deprecation cycles.
- Integration fixes: time spent repairing breakage when connected tools change.
- Usage costs: API and platform fees, which tend to grow as volume grows.
An automation that saves 10 hours a week but needs 3 hours a week of upkeep still delivers solid value. One that saves 2 hours a week but needs 2 hours of upkeep is a hobby. Knowing the difference requires tracking both sides.
Borrow the SRE time budget
Google’s SRE target of keeping toil under 50% of engineering time offers a useful template. For teams that own a portfolio of automations, set an explicit budget: for example, a fixed share of one person’s week dedicated to automation health. If maintenance consistently exceeds that budget, it’s a signal that the portfolio is too large, too fragile, or both, and that some workflows should be redesigned or retired.
Retirement criteria
Organizations are good at launching automations and bad at turning them off. Define retirement triggers up front:
- The process it supports has changed enough that the automation needs a rebuild rather than a fix.
- Maintenance cost has exceeded time saved for two consecutive quarters.
- Quality on the golden set has fallen below an agreed threshold and can’t be recovered with reasonable effort.
- No one is willing to own it.
- Its outputs are no longer used by anyone downstream.
That last one is more common than you’d expect. Workflows keep running for months, generating reports nobody reads and enriching records nobody opens, simply because no one turned them off. Each one is a small cost, a small security surface, and a small source of confusion.
Simpler is easier to maintain
A final budgeting principle: complexity is a maintenance cost. A ten-step agentic workflow with multiple model calls, tool uses, and branching logic has many more places to drift than a three-step workflow with one tightly constrained AI step. When two designs achieve similar results, the simpler one will almost always be cheaper to keep healthy. Use AI for the step that genuinely needs judgment and keep everything else deterministic.
A 30-Day Automation Health Audit
If your organization already runs AI automations without a maintenance system, here’s a practical sequence to get one in place within a month.
Week 1: Find everything
- Announce an automation inventory with an explicit no-blame framing.
- Ask each team to list every automated workflow involving AI, including personal-account automations that touch company data.
- Check admin consoles of your automation platforms, AI vendors, and SaaS tools for connected apps and API keys you didn’t know about.
- Build the first version of the register, even if many fields are blank.
Week 2: Assign owners and risk tiers
- Give every automation a named owner and backup. Anything left unowned at the end of the week gets paused.
- Assign a risk tier based on the worst realistic silent-failure outcome.
- Move high-risk automations off personal credentials and onto service accounts.
- Document a kill switch and manual fallback for every high-risk workflow.
Week 3: Establish baselines
- Build golden test sets for high-risk automations first, starting with 20 to 30 real examples each.
- Run them and record current pass rates. You may be surprised.
- Record which model version each workflow uses, and pin versions where your platform allows.
- Identify which automations would be affected by currently announced model deprecations.
Week 4: Close the obvious gaps
- Review permissions on every high-risk automation and remove access that isn’t currently needed.
- Add human approval steps for irreversible actions.
- Add input validation in front of AI steps for your most important workflows.
- Retire automations that no one uses or no one will own.
- Schedule recurring golden-set runs and owner reviews on the calendar.
After 30 days, you won’t have a perfect system. You’ll have something far more valuable: visibility. You’ll know what’s running, who’s responsible, how well it’s performing, and where the biggest risks are. Every improvement after that builds on a known baseline.
Conclusion: Treat Automations Like Living Systems
The pitch for AI automation is usually framed as a one-time trade: invest some build effort, get recurring time savings forever. The reality is closer to adopting a living system. It needs feeding, checking, and occasional surgery. Left alone, it doesn’t stay the same. It drifts.
The evidence for this is consistent across decades and disciplines. Machine learning engineers documented the hidden maintenance costs of ML systems in 2015. Researchers showed in 2023 that the same named LLM could swing from 84% to 51% accuracy on a task in three months. Model providers publish deprecation schedules that guarantee every AI dependency has an expiration date. Workplace surveys show most AI use happens outside official channels. And human factors research going back to 1983 warns that the better automation works, the less prepared people are to catch it when it doesn’t.
None of this argues against automation. It argues for automating with maintenance in mind from the start.
Key takeaways
- Expect decay through five channels: model, input, schema, process, and people drift. Each needs its own detection method.
- Don’t trust green checkmarks. AI automations fail quietly. Monitor output quality, not just whether runs complete.
- Pin model versions and track them in a register so deprecation notices turn into a clear migration list.
- Build a golden test set for every AI workflow that matters, and re-run it on a schedule and before every change.
- Inventory before you govern. Shadow automations are common and often hidden. Amnesty works better than bans.
- Keep humans actively engaged, not just nominally in the loop, so skill and context don’t evaporate.
- Review permissions regularly and require approval for irreversible actions, especially in workflows that read external content.
- Budget for upkeep in every business case, and define retirement criteria before launch.
The teams that get lasting value from AI automations in 2026 won’t necessarily be the ones that build the most workflows. They’ll be the ones whose workflows are still working correctly a year later, because someone owns them, measures them, and is willing to switch them off when they stop earning their keep.