{"id":413,"date":"2026-10-16T15:37:05","date_gmt":"2026-10-16T15:37:05","guid":{"rendered":"https:\/\/www.algofuse.ai\/blog\/your-ai-automation-works-today-who-keeps-it-working-in-six-months\/"},"modified":"2026-11-13T21:30:14","modified_gmt":"2026-11-13T21:30:14","slug":"your-ai-automation-works-today-who-keeps-it-working-in-six-months","status":"publish","type":"post","link":"https:\/\/www.algofuse.ai\/blog\/your-ai-automation-works-today-who-keeps-it-working-in-six-months\/","title":{"rendered":"Your AI Automation Works Today. Who Keeps It Working in Six Months?"},"content":{"rendered":"<p>Building an AI automation has never been easier. Someone on your operations team links an inbox to a language model, sends the output to a CRM, posts a summary in Slack, and by Friday afternoon a process that used to eat four hours a week runs on its own. Everyone is pleased, the demo goes well, and the workflow goes live.<\/p>\n<p>Then six months go by. The model behind it has been quietly updated or scheduled for retirement. The refund policy it quotes changed in March. The person who built it moved to another team. Nobody can say whether the outputs are still right, because the dashboard shows nothing but green checkmarks.<\/p>\n<p>This is the <strong>day-two problem<\/strong> of AI automations, and it gets far less attention than it deserves. Most writing on the subject covers which processes to automate, how to measure ROI, or how to get a pilot into production. Much less covers what happens <em>after<\/em> production: the slow, often invisible wear that turns a working automation into a liability.<\/p>\n<p>This article is about that stretch of time. We&#8217;ll go through the five main ways AI automations break after launch, each backed by documented cases and published research. Then we&#8217;ll lay out the practical systems that keep them reliable: an automation registry, evaluation sets that work like regression tests, monitoring that checks meaning and not just uptime, a model migration runbook, and a realistic way to budget for upkeep.<\/p>\n<p>If you already run more than a handful of AI workflows, or plan to, this is the part of the work that decides whether they still save you time in a year or start quietly costing you.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/szukdzugaodusagltwla.supabase.co\/storage\/v1\/object\/public\/marketing-media\/f71482aa-ece0-4f48-be89-4a95e0933103\/8c6e03ed-90c2-42d8-b2e3-b8da444fe61e\/image\/1792164453094.png\" alt=\"Illustration of an automated workflow machine connecting email, an LLM, a CRM and Slack, with an engineer inspecting a cracked gear under the headline Launch is day one, maintenance is every day after\" \/><\/p>\n<h2>The Day-Two Problem: Why Launch Is the Cheap Part<\/h2>\n<p>In 2015, a team of Google engineers led by D. Sculley published a paper at NeurIPS called <em>Hidden Technical Debt in Machine Learning Systems<\/em>. Its main argument still holds a decade later: machine learning makes it quick to ship useful systems, but &#8220;it is dangerous to think of these quick wins as coming for free.&#8221; The authors found that &#8220;it is common to incur massive ongoing maintenance costs in real-world ML systems.&#8221;<\/p>\n<p>The risk factors they listed read like a description of today&#8217;s AI automation stacks: boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, and &#8220;changes in the external world.&#8221; Each of these is now part of the no-code and low-code workflows that business teams build every day.<\/p>\n<h3>Why AI automations have more debt than traditional automations<\/h3>\n<p>A traditional rule-based automation, such as &#8220;when a form is submitted, create a row in the spreadsheet,&#8221; is deterministic. If it worked yesterday and nothing upstream changed, it works today. When it breaks, it usually breaks loudly: an API returns an error, a field goes missing, a task fails.<\/p>\n<p>AI automations are different in three ways:<\/p>\n<ul>\n<li><strong>The core component is non-deterministic.<\/strong> The same input can produce slightly different outputs, which makes &#8220;it worked in testing&#8221; a weaker guarantee.<\/li>\n<li><strong>The core component is rented, not owned.<\/strong> The model sits behind a vendor&#8217;s API, and the vendor updates, deprecates, and retires models on its own schedule.<\/li>\n<li><strong>Failures are often semantic, not technical.<\/strong> An LLM step rarely throws an error. It returns fluent, plausible text that happens to be wrong.<\/li>\n<\/ul>\n<h3>The shape of the cost curve<\/h3>\n<p>Most teams budget for AI automation like a one-time project: discovery, build, test, launch. In reality, the cost curve looks more like owning a vehicle than buying a tool. The purchase price is visible and small. Fuel, servicing, insurance, and the occasional large repair arrive later and add up.<\/p>\n<p>That doesn&#8217;t mean AI automations aren&#8217;t worth building. Many clearly are. It means the decision to build one should come with a decision about who will maintain it, how, and at what cost. The rest of this article is about making that second decision well.<\/p>\n<h2>Failure Mode #1: The Model Under Your Workflow Changes<\/h2>\n<p>The most common assumption in AI automation is that &#8220;the model&#8221; is a fixed component. You tested your prompt against it, it gave good results, so it will keep giving good results. Published research and vendors&#8217; own documentation both say otherwise.<\/p>\n<h3>Same name, different behavior<\/h3>\n<p>In 2023, researchers Lingjiao Chen, Matei Zaharia, and James Zou of Stanford and UC Berkeley published a study titled <em>How is ChatGPT&#8217;s behavior changing over time?<\/em> They compared the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on a range of tasks.<\/p>\n<p>The results were striking. GPT-4 in March 2023 identified prime versus composite numbers with 84% accuracy. The June 2023 version of the same model scored 51% on the same questions. GPT-3.5 moved the opposite way on that task, getting better over the same period. Both models made more formatting mistakes in code generation in June than in March.<\/p>\n<p>The authors found that a decline in GPT-4&#8217;s ability to follow user instructions was &#8220;one common factor behind the many behavior drifts,&#8221; and concluded that &#8220;the behavior of the &#8216;same&#8217; LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs.&#8221;<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/szukdzugaodusagltwla.supabase.co\/storage\/v1\/object\/public\/marketing-media\/f71482aa-ece0-4f48-be89-4a95e0933103\/8c6e03ed-90c2-42d8-b2e3-b8da444fe61e\/image\/1792164531725.png\" alt=\"Infographic comparing GPT-4 March 2023 at 84 percent accuracy and GPT-4 June 2023 at 51 percent accuracy on prime number identification, from Chen, Zaharia and Zou 2023\" \/><\/p>\n<p>For automation builders, the formatting finding matters most. Many workflows depend on the model returning output in an exact shape, such as JSON with specific keys, a category label from a fixed list, or a date in a particular format. A small drift in instruction-following can break the parsing step downstream, or worse, pass malformed data along without anyone noticing.<\/p>\n<h3>Models also retire on a schedule<\/h3>\n<p>Even if you pin a specific model version to avoid silent updates, that version has an end date. Both major API providers publish deprecation policies, and both state that software built on their models will need updates to keep working.<\/p>\n<p>OpenAI&#8217;s deprecations documentation says the company provides at least six months&#8217; notice before retiring generally available models, at least three months for specialized variants, and much shorter notice, &#8220;such as 2 weeks,&#8221; for preview models. It explicitly advises against using preview models &#8220;for business-critical production workloads unless you can migrate on short notice.&#8221; The same page lists GPT-4o, GPT-4o mini, GPT-4.1, and GPT-4.1 mini snapshots for removal from the API on January 14, 2027.<\/p>\n<p>Anthropic&#8217;s model deprecation page commits to at least 60 days&#8217; notice before retiring publicly released models. It also notes that partner platforms such as Amazon Bedrock and Google Cloud &#8220;set their own retirement schedules,&#8221; so the same model can have different end dates depending on where you access it. Older models like Claude 3.7 Sonnet have already been retired, and requests to retired models simply fail.<\/p>\n<h3>What this means in practice<\/h3>\n<ul>\n<li><strong>Every AI step in every workflow has an expiry date<\/strong>, whether or not anyone has written it down.<\/li>\n<li><strong>Floating aliases<\/strong> (model names that always point to &#8220;latest&#8221;) trade retirement risk for drift risk. You won&#8217;t hit a shutdown date, but behavior can change under you without notice.<\/li>\n<li><strong>Pinned versions<\/strong> trade drift risk for retirement risk. Behavior stays stable until the day you&#8217;re forced to migrate.<\/li>\n<li><strong>Neither option removes the need to re-test.<\/strong> It only changes when re-testing happens.<\/li>\n<\/ul>\n<p>A team running 40 automations across three model providers will, realistically, face model migrations several times a year. Without preparation, each one is a scramble.<\/p>\n<h2>Failure Mode #2: The World Changes Around the Automation<\/h2>\n<p>The Sculley paper listed &#8220;changes in the external world&#8221; as its own risk factor, and with good reason. An AI automation encodes a snapshot of how your business worked when it was built: your policies, your product catalog, your pricing, your customer base, your tone. The business keeps moving. The automation doesn&#8217;t, unless someone updates it.<\/p>\n<h3>The Air Canada precedent<\/h3>\n<p>The best-known example of this gap is the 2024 ruling in <em>Moffatt v. Air Canada<\/em>. As BBC Travel reported, Air Canada&#8217;s website chatbot told passenger Jake Moffatt he could book a full-fare flight to his grandmother&#8217;s funeral and apply for a bereavement discount afterward. That wasn&#8217;t the airline&#8217;s actual policy, which required the request before travel.<\/p>\n<p>When Moffatt applied for the discount, Air Canada refused and argued that the chatbot was &#8220;a separate legal entity that is responsible for its own actions.&#8221; The British Columbia Civil Resolution Tribunal rejected that argument and ordered the airline to pay $812.02 in damages and fees. Tribunal member Christopher Rivers wrote: &#8220;It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.&#8221;<\/p>\n<p>The dollar amount was small. The principle was not. As Gabor Lukacs of the Air Passenger Rights advocacy group told the BBC: &#8220;If you are handing over part of your business to AI, you are responsible for what it does.&#8221;<\/p>\n<h3>Policy drift is a maintenance problem, not just a model problem<\/h3>\n<p>It&#8217;s tempting to read the Air Canada case as a hallucination story. From a maintenance point of view, the more useful lesson is about <strong>keeping sources in sync<\/strong>. Whenever an automation answers questions about policies, prices, terms, eligibility, or product specs, there are two copies of the truth: the official one and the one the automation is working from. Every policy change creates a window where the two disagree.<\/p>\n<p>Common sources of this kind of drift include:<\/p>\n<ul>\n<li>Return, refund, or warranty terms updated on the website but not in the knowledge base the automation retrieves from<\/li>\n<li>Prices or SKUs changed in the catalog while a prompt still contains hard-coded examples<\/li>\n<li>New product lines that the classification step has no category for, so it forces them into the nearest wrong one<\/li>\n<li>Regulatory changes that make previously acceptable wording non-compliant<\/li>\n<li>Org changes that send escalations to a person or channel that no longer exists<\/li>\n<\/ul>\n<h3>When the input population shifts<\/h3>\n<p>The world can also change through who is sending inputs. McDonald&#8217;s ended its AI drive-thru ordering test with IBM in more than 100 restaurants in 2024. According to two sources who spoke to CNBC, one of the system&#8217;s challenges was interpreting different accents and dialects, which affected order accuracy. McDonald&#8217;s declined to comment on accuracy.<\/p>\n<p>The broader lesson applies well beyond voice ordering. An automation tuned on one population of inputs, such as one region&#8217;s customers, one language, one email template, or one vendor&#8217;s invoice format, can degrade when that population changes. Expansion into a new market, a new supplier, or a new customer segment is a trigger to re-test, even if nobody touched the workflow.<\/p>\n<h2>Failure Mode #3: Silent Failures and the &#8220;200 OK&#8221; Problem<\/h2>\n<p>Traditional automation platforms are built to catch technical failures. A step times out, an authentication token expires, an API returns an error code, and the platform flags the run as failed and sends an alert. That&#8217;s useful, but it covers only a small share of how AI automations actually go wrong.<\/p>\n<h3>Technically successful, substantively wrong<\/h3>\n<p>An LLM step that misreads an invoice total, picks the wrong support category, invents a policy detail, or writes a summary that leaves out the one important sentence will, in almost every case, return a successful response. The API call worked. The text was generated. The next step received it. From the platform&#8217;s point of view, the run succeeded.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/szukdzugaodusagltwla.supabase.co\/storage\/v1\/object\/public\/marketing-media\/f71482aa-ece0-4f48-be89-4a95e0933103\/8c6e03ed-90c2-42d8-b2e3-b8da444fe61e\/image\/1792164611490.png\" alt=\"Automation monitoring dashboard showing all runs as successful while a magnifying glass reveals an output quoting an outdated refund policy, with the callout Green dashboard, wrong answer\" \/><\/p>\n<p>This is the &#8220;200 OK problem&#8221;: the HTTP status code says success while the content is wrong. It&#8217;s why so many teams are confident in automations that have been drifting for weeks. Their monitoring answers &#8220;did it run?&#8221; when the question that matters is &#8220;was it right?&#8221;<\/p>\n<h3>How silent failures compound<\/h3>\n<p>Silent failures are dangerous less because of any single wrong output than because errors spread downstream. Consider a typical chain:<\/p>\n<ol>\n<li>An LLM extracts fields from inbound purchase orders.<\/li>\n<li>The extracted data creates records in an ERP.<\/li>\n<li>Those records drive inventory reorder logic.<\/li>\n<li>Reorder data feeds a weekly forecast that leadership reviews.<\/li>\n<\/ol>\n<p>If step one starts misreading a unit-of-measure field on 3% of orders after a supplier changes its template, nothing alerts. The error passes through the ERP, distorts reorder quantities, and appears weeks later as an unexplained inventory variance. By then, tracing it back to the extraction step is detective work.<\/p>\n<p>This is what the Sculley paper called <strong>entanglement<\/strong>, often summarized as &#8220;changing anything changes everything.&#8221; In AI workflows, it means the cost of a semantic error grows with every step it travels through before someone notices.<\/p>\n<h3>Signals that you have silent failures right now<\/h3>\n<ul>\n<li>Downstream teams have built their own &#8220;fix-up&#8221; processes to clean automation output<\/li>\n<li>People say they &#8220;double-check&#8221; automated outputs but nobody tracks what they find<\/li>\n<li>Error rates are reported only as platform task failures, not as content accuracy<\/li>\n<li>Nobody can say what share of last month&#8217;s outputs were correct<\/li>\n<li>Complaints arrive as anecdotes (&#8220;the summaries seem worse lately&#8221;) rather than as data<\/li>\n<\/ul>\n<p>If two or more of these sound familiar, your automations likely have silent failures you haven&#8217;t measured.<\/p>\n<h2>Failure Mode #4: The Security Surface Grows With Every Tool<\/h2>\n<p>The first wave of AI automations mostly generated text that a human then read. The current wave gives models tools: they can read inboxes, query databases, browse the web, call APIs, send messages, and update records. Every added tool makes the automation more useful and widens its attack surface.<\/p>\n<h3>The lethal trifecta<\/h3>\n<p>Developer and researcher Simon Willison has offered one of the clearest framings of this risk, which he calls the <strong>&#8220;lethal trifecta&#8221;<\/strong> for AI agents. An automation becomes exploitable when it combines three capabilities:<\/p>\n<ol>\n<li><strong>Access to private data<\/strong>, which, as Willison notes, is &#8220;one of the most common purposes of tools in the first place&#8221;<\/li>\n<li><strong>Exposure to untrusted content<\/strong>, meaning any way that text or images controlled by an attacker could reach the model<\/li>\n<li><strong>The ability to communicate externally<\/strong> in a way that could be used to send data out<\/li>\n<\/ol>\n<p><img decoding=\"async\" src=\"https:\/\/szukdzugaodusagltwla.supabase.co\/storage\/v1\/object\/public\/marketing-media\/f71482aa-ece0-4f48-be89-4a95e0933103\/8c6e03ed-90c2-42d8-b2e3-b8da444fe61e\/image\/1792164721100.png\" alt=\"Venn diagram of the lethal trifecta for AI agents: access to private data, exposure to untrusted content, and ability to communicate externally, concept by Simon Willison\" \/><\/p>\n<p>The root cause, in Willison&#8217;s words: &#8220;LLMs follow instructions in content.&#8221; They &#8220;are unable to reliably distinguish the importance of instructions based on where they came from. Everything eventually gets glued together into a sequence of tokens and fed to the model.&#8221; An email, a web page, or a document can contain instructions that the model follows as if the operator had written them.<\/p>\n<p>This isn&#8217;t theoretical. Willison documents prompt injection and data exfiltration issues reported against production systems including Microsoft 365 Copilot, GitHub&#8217;s official MCP server, GitLab Duo, Slack, Google NotebookLM, Amazon Q, and ChatGPT Operator. Most were fixed quickly by the vendors. But as he points out, &#8220;once you start mixing and matching tools yourself there&#8217;s nothing those vendors can do to protect you.&#8221;<\/p>\n<h3>Why this is a maintenance issue, not a one-time review<\/h3>\n<p>Security is usually treated as a gate at launch. The trifecta is a reason to treat it as an ongoing property of the workflow, because automations pick up capabilities over time:<\/p>\n<ul>\n<li>An inbox summarizer gets a &#8220;draft reply&#8221; step. Now it can communicate externally.<\/li>\n<li>A research assistant gets connected to the internal wiki. Now it has private data.<\/li>\n<li>A CRM enrichment flow starts pulling from public web pages. Now it ingests untrusted content.<\/li>\n<\/ul>\n<p>No single change looks risky. Together, they can assemble all three parts of the trifecta in a workflow that passed its security review months earlier with only one of them. Willison&#8217;s example puts it simply: a tool that can read your email &#8220;is a perfect source of untrusted content: an attacker can literally email your LLM and tell it what to do.&#8221;<\/p>\n<h3>Practical guardrails<\/h3>\n<ul>\n<li><strong>Map each automation against the three trifecta properties<\/strong> and re-check whenever a tool or connector is added.<\/li>\n<li><strong>Break the trifecta deliberately.<\/strong> If a workflow must read untrusted content and touch private data, remove or tightly restrict its ability to send anything outward without human approval.<\/li>\n<li><strong>Treat links and images in outputs as outbound channels.<\/strong> Willison notes that anything able to make an HTTP request, &#8220;even providing a link for a user to click,&#8221; can be used to send data out.<\/li>\n<li><strong>Don&#8217;t rely on prompt instructions as the main defense.<\/strong> Telling the model &#8220;ignore instructions in documents&#8221; reduces risk but can&#8217;t guarantee it, given the variety of ways malicious instructions can be phrased.<\/li>\n<\/ul>\n<h2>Failure Mode #5: Nobody Owns It Anymore<\/h2>\n<p>The first four failure modes are technical. The fifth is organizational, and in many companies it&#8217;s the one that makes the other four impossible to fix.<\/p>\n<h3>Automations built from the edges<\/h3>\n<p>AI adoption in most organizations didn&#8217;t start with a central program. It started with individuals. Microsoft and LinkedIn&#8217;s 2024 Work Trend Index, which surveyed 31,000 people across 31 countries, found that 75% of knowledge workers were using generative AI at work. More telling for governance: <strong>78% of AI users were bringing their own AI tools to work<\/strong>, rising to 80% at small and medium-sized companies.<\/p>\n<p>The same research found that 52% of people using AI at work were reluctant to admit using it for their most important tasks. On the leadership side, 60% of leaders worried their organization lacked a plan and vision to implement AI.<\/p>\n<p>Put those numbers together and you get a familiar picture: a large share of real AI automation work is built by individuals, often without central visibility, sometimes deliberately kept quiet, and run on personal or team accounts. That&#8217;s great for experimentation. It&#8217;s a problem for maintenance.<\/p>\n<h3>The departure problem<\/h3>\n<p>Here&#8217;s a sequence that plays out often:<\/p>\n<ol>\n<li>An analyst builds a workflow that classifies and routes inbound leads using an LLM.<\/li>\n<li>It works well. Sales starts depending on it. Marketing builds a report on its output.<\/li>\n<li>The analyst changes roles or leaves the company.<\/li>\n<li>Months later, the model is deprecated or the lead form changes, and the workflow starts misrouting.<\/li>\n<li>Nobody knows who built it, where the prompt lives, what account pays for the API, or which teams depend on the output.<\/li>\n<\/ol>\n<p>The Sculley paper called the last part of this <strong>&#8220;undeclared consumers&#8221;<\/strong>: systems that quietly depend on another system&#8217;s output without the owner knowing. In AI automation terms, every spreadsheet, dashboard, or downstream workflow that reads an automation&#8217;s output is a consumer. If they&#8217;re undeclared, changing or retiring the automation breaks things nobody expected.<\/p>\n<h3>Ownership has to be explicit<\/h3>\n<p>The fix isn&#8217;t to ban employee-built automations. The Work Trend Index data suggests that would simply push more of the work out of sight. The fix is to make ownership a condition of running in production. Every automation that touches customers, money, or shared data should have a named owner, a named backup, and a record of what it does. That record is the registry, which is where we turn next.<\/p>\n<h2>The Fix, Part 1: Build an Automation Registry<\/h2>\n<p>You can&#8217;t maintain what you can&#8217;t see. The first concrete step for any organization running more than a few AI automations is a single inventory of them. It doesn&#8217;t need special software. A well-kept spreadsheet or database table works fine at first. What matters is the fields and the discipline.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/szukdzugaodusagltwla.supabase.co\/storage\/v1\/object\/public\/marketing-media\/f71482aa-ece0-4f48-be89-4a95e0933103\/8c6e03ed-90c2-42d8-b2e3-b8da444fe61e\/image\/1792164807023.png\" alt=\"Automation registry displayed on a wall screen with columns for workflow, owner, model version, data touched, risk tier, last eval and retirement date, with rows flagged owner left company\" \/><\/p>\n<h3>Minimum fields for each automation<\/h3>\n<ul>\n<li><strong>Name and purpose:<\/strong> one sentence on what the workflow does and why it exists<\/li>\n<li><strong>Owner and backup:<\/strong> named people, not teams<\/li>\n<li><strong>Platform and location:<\/strong> where it runs (Zapier, Make, n8n, custom code, an agent platform) and which account<\/li>\n<li><strong>Models used:<\/strong> provider, exact model identifier, and whether it&#8217;s pinned or a floating alias<\/li>\n<li><strong>Known retirement dates:<\/strong> pulled from the provider&#8217;s deprecation page<\/li>\n<li><strong>Data touched:<\/strong> what it reads and what it writes, flagged for personal, financial, or confidential data<\/li>\n<li><strong>Trifecta status:<\/strong> which of the three properties (private data, untrusted input, external communication) it has<\/li>\n<li><strong>Downstream consumers:<\/strong> every system, report, or team that depends on its output<\/li>\n<li><strong>Risk tier:<\/strong> a simple low, medium, or high rating based on customer exposure, financial impact, and data sensitivity<\/li>\n<li><strong>Last evaluated:<\/strong> the date of the most recent accuracy check<\/li>\n<\/ul>\n<h3>How to populate it without a witch hunt<\/h3>\n<p>Given how much AI use happens out of view, a registry built through mandates and audits tends to miss most of what exists. A few approaches work better:<\/p>\n<ul>\n<li><strong>Make it an amnesty, not an investigation.<\/strong> Ask people to register what they&#8217;ve built, and make clear the goal is to support those workflows, not shut them down.<\/li>\n<li><strong>Offer something in return.<\/strong> Registered automations get access to company-paid API keys, shared monitoring, and help with model migrations. Unregistered ones don&#8217;t.<\/li>\n<li><strong>Use billing and usage data.<\/strong> Provider consoles can usually export usage by API key and model. Anthropic&#8217;s documentation, for example, describes exporting a usage CSV broken down by API key and model specifically to find workloads on deprecated models.<\/li>\n<li><strong>Check platform admin views.<\/strong> Most automation platforms on team plans let administrators list active workflows and their owners.<\/li>\n<\/ul>\n<h3>Keeping it alive<\/h3>\n<p>A registry that&#8217;s accurate on the day it&#8217;s built and stale three months later is worse than none, because it creates false confidence. Tie updates to events that already happen: when an employee changes roles, their registered automations need a new owner before offboarding is complete. When a new connector is added to a workflow, the trifecta field gets re-checked. When a provider publishes a deprecation, every row using that model gets a retirement date.<\/p>\n<h2>The Fix, Part 2: Evals as Regression Tests<\/h2>\n<p>Software engineers don&#8217;t ship code changes without running tests. AI automations need the equivalent, and the field has settled on a name for it: <strong>evals<\/strong>, short for evaluations. An eval is a fixed set of inputs with known-good expected outputs that you run the automation against to check whether it still behaves correctly.<\/p>\n<h3>Building a golden set<\/h3>\n<p>The core of an eval is a &#8220;golden set&#8221;: a curated collection of real examples with correct answers. For most business automations, building one is less work than people expect.<\/p>\n<ol>\n<li><strong>Pull 50 to 200 real inputs<\/strong> from the automation&#8217;s history. Include ordinary cases and the awkward ones: unusual formats, edge cases, past failures.<\/li>\n<li><strong>Record the correct output for each<\/strong>, as decided by someone who knows the process well. For classification, that&#8217;s the right label. For extraction, the right field values. For generation, a set of criteria the output must meet.<\/li>\n<li><strong>Write down how to score each one.<\/strong> Exact match works for labels and structured fields. For free text, use a checklist: does it mention the correct policy, avoid forbidden claims, stay under a length limit, include the required disclaimer?<\/li>\n<li><strong>Store the golden set next to the automation&#8217;s record in the registry<\/strong> so whoever owns it next can find it.<\/li>\n<\/ol>\n<h3>When to run evals<\/h3>\n<p>The value of an eval set comes from running it at the right moments:<\/p>\n<ul>\n<li><strong>Before any model change<\/strong>, whether it&#8217;s a planned upgrade or a forced migration<\/li>\n<li><strong>Before any prompt change<\/strong>, however small. A one-word edit can shift behavior on edge cases.<\/li>\n<li><strong>On a fixed schedule<\/strong>, weekly or monthly depending on risk tier, to catch drift in floating-alias models. The Chen, Zaharia, and Zou findings are the reason this matters.<\/li>\n<li><strong>After upstream changes<\/strong>, such as a new form, a new supplier template, a new product line, or a policy update<\/li>\n<\/ul>\n<h3>Growing the set over time<\/h3>\n<p>Every production failure should become a new eval case. When someone finds the automation misclassified a ticket or misread an invoice, the fix isn&#8217;t complete until that example is in the golden set with its correct answer. Over months, the set comes to reflect the real failure patterns of your specific workflow, which is far more useful than any generic benchmark.<\/p>\n<p>One caution: if you use an LLM to grade other LLM outputs (a common approach for free-text tasks), that grader is a model too, with its own drift and retirement schedule. Spot-check its judgments against human review regularly, and list it in the registry as a dependency.<\/p>\n<h2>The Fix, Part 3: Monitor Meaning, Not Just Uptime<\/h2>\n<p>Evals catch problems at known checkpoints. Monitoring catches them in between. The goal is to close the gap exposed by the &#8220;200 OK problem,&#8221; so that your alerts fire when outputs go wrong, not only when runs fail.<\/p>\n<h3>Cheap signals that catch a lot<\/h3>\n<p>Sophisticated observability platforms exist, but a few simple checks catch a surprising share of semantic failures:<\/p>\n<ul>\n<li><strong>Schema validation:<\/strong> if the model is supposed to return JSON with specific fields, validate every response and alert when the failure rate rises. This catches the formatting drift the 2023 study documented.<\/li>\n<li><strong>Distribution tracking:<\/strong> for classification steps, track the share of outputs in each category day by day. If &#8220;Other&#8221; jumps from 4% to 15%, something changed: either the inputs or the model.<\/li>\n<li><strong>Length and refusal tracking:<\/strong> sudden changes in average output length, or a rise in refusal phrases (&#8220;I can&#8217;t help with that&#8221;), are early warnings of model behavior changes.<\/li>\n<li><strong>Confidence and fallback rates:<\/strong> if your workflow routes uncertain cases to humans, track how often that happens. A rising fallback rate is a leading indicator.<\/li>\n<li><strong>Downstream correction rates:<\/strong> if humans edit automation output before using it, log the edits. The edit rate is one of the most honest accuracy measures you&#8217;ll get.<\/li>\n<\/ul>\n<h3>Human sampling, sized to risk<\/h3>\n<p>Automated checks can&#8217;t confirm that a summary captured the right point or a reply quoted the right policy. For that, you need people looking at a sample of real outputs on a regular cadence. The key is to size the sample to the risk tier instead of reviewing everything or nothing:<\/p>\n<ul>\n<li><strong>High risk<\/strong> (customer-facing, financial, regulated): review a meaningful sample every week, and consider human approval before anything is sent externally<\/li>\n<li><strong>Medium risk<\/strong> (internal decisions, shared data): review a smaller sample every two to four weeks<\/li>\n<li><strong>Low risk<\/strong> (personal productivity, drafts a human always edits): spot-check during scheduled evals<\/li>\n<\/ul>\n<p>Record what reviewers find in a structured way: correct, minor issue, major issue, and why. Over time, this turns &#8220;the summaries seem worse lately&#8221; into a trend line that can trigger action.<\/p>\n<h3>Watch the inputs, too<\/h3>\n<p>Many failures start with input changes, not model changes. Monitor for shifts in what&#8217;s going in: new email senders, new document formats, inputs in new languages, a sudden spike in volume from a single source. The McDonald&#8217;s case is a reminder that accuracy depends on how closely real inputs match the conditions the system was built for.<\/p>\n<h2>The Fix, Part 4: A Model Migration Runbook<\/h2>\n<p>Given the deprecation policies described earlier, model migrations aren&#8217;t emergencies. They&#8217;re routine events with advance notice. The organizations that handle them smoothly treat them like any other scheduled maintenance, with a written procedure.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/szukdzugaodusagltwla.supabase.co\/storage\/v1\/object\/public\/marketing-media\/f71482aa-ece0-4f48-be89-4a95e0933103\/8c6e03ed-90c2-42d8-b2e3-b8da444fe61e\/image\/1792164876542.png\" alt=\"Model migration runbook timeline from deprecation notice to shutdown date with steps for inventory, golden-set evals, shadow traffic, staged cutover and retirement, plus notice windows for preview models, Anthropic and OpenAI\" \/><\/p>\n<h3>Step 1: Know the moment a notice is published<\/h3>\n<p>Providers email active customers and update their documentation, but emails go to whoever owns the account, which may not be whoever owns the automation. Assign someone to check provider deprecation pages on a regular schedule. OpenAI&#8217;s documentation also notes that its List models endpoint includes a <code>shutdown_date<\/code> field, so the check can be automated and tied to your registry.<\/p>\n<h3>Step 2: Find every affected workflow<\/h3>\n<p>This is where the registry pays for itself. Filter by model identifier and you have the list. Without a registry, this step means searching through automation platforms, code repositories, and usage exports, and usually still missing something. Remember that the same model may have different retirement dates on partner clouds like Bedrock or Google Cloud.<\/p>\n<h3>Step 3: Evaluate the replacement against your golden sets<\/h3>\n<p>Run each affected automation&#8217;s eval set against the recommended replacement model. Don&#8217;t assume a newer model is better on <em>your<\/em> task. The Chen study showed that changes between versions can move performance in either direction depending on the task. Compare results side by side and look closely at formatting, edge cases, and anything the old prompt was tuned to work around.<\/p>\n<h3>Step 4: Adjust prompts and re-test<\/h3>\n<p>Newer models often respond differently to the same instructions. Prompts packed with workarounds for an older model&#8217;s quirks may need simplifying. Make changes, re-run evals, and repeat until results match or beat the baseline.<\/p>\n<h3>Step 5: Shadow, then stage<\/h3>\n<p>For high-risk automations, run the new model in parallel with the old one on live inputs for a period, without acting on its outputs, and compare. Then cut over in stages, starting with a portion of traffic, before moving everything.<\/p>\n<h3>Step 6: Update the record<\/h3>\n<p>Update the registry with the new model identifier, its known retirement date, the eval results, and the migration date. Remove old configurations so nothing falls back to the retired model by accident.<\/p>\n<h3>Mind the notice windows<\/h3>\n<p>The notice periods shape how much time you have. A six-month window for a generally available OpenAI model is comfortable. A 60-day minimum from Anthropic is workable with a runbook. A two-week window for a preview model is not, which is why OpenAI&#8217;s own guidance is to keep preview models out of business-critical workflows. A sound rule: <strong>never put a preview model into a high-risk tier automation<\/strong>.<\/p>\n<h2>Budgeting for Maintenance: Making the Cost Visible<\/h2>\n<p>Everything above takes time. The most common reason it doesn&#8217;t happen is that nobody budgeted for it. AI automation projects are approved on the strength of hours saved, and the maintenance cost is left out, so it ends up taken from whatever the owner was supposed to be doing next.<\/p>\n<h3>A simple maintenance cost model<\/h3>\n<p>Rather than relying on industry averages that may not fit your situation, estimate maintenance cost per automation from its own parts. For each workflow in the registry, estimate annual hours for:<\/p>\n<ul>\n<li><strong>Scheduled evals:<\/strong> time per eval run multiplied by runs per year<\/li>\n<li><strong>Human sampling:<\/strong> review time per week multiplied by 52, sized to the risk tier<\/li>\n<li><strong>Model migrations:<\/strong> expected migrations per year (check provider deprecation history for the models you use) multiplied by hours per migration<\/li>\n<li><strong>Source-of-truth updates:<\/strong> how often the policies, catalogs, or templates it relies on change, multiplied by update time<\/li>\n<li><strong>Incident handling:<\/strong> a buffer for investigating and fixing the failures that get through anyway<\/li>\n<li><strong>Security re-review:<\/strong> time to re-check trifecta exposure when tools are added<\/li>\n<\/ul>\n<p>Add API and platform costs, and you have a real annual cost of ownership. Compare it to the annual hours saved. Some automations will look even better than expected. Others, especially low-volume workflows that touch high-risk data, may turn out to cost more to maintain safely than they save.<\/p>\n<h3>Use the numbers to prune<\/h3>\n<p>One of the most useful results of this exercise is permission to retire automations. In most organizations&#8217; registries, a meaningful share of workflows run on low volume, have no clear owner, and save little time. Shutting them down reduces the surface area for every failure mode in this article. Fewer, better-maintained automations nearly always beat a sprawling collection that nobody can vouch for.<\/p>\n<h3>Assign real capacity<\/h3>\n<p>Once the cost is visible, assign it. That could mean a percentage of each owner&#8217;s time formally set aside for upkeep, a small central team that runs evals and migrations for everyone, or both. The Work Trend Index found that 59% of leaders worried about quantifying AI&#8217;s productivity gains. A maintenance budget built from the registry gives them the other half of that equation: what it actually costs to keep those gains coming.<\/p>\n<h2>Conclusion: A Day-Two Checklist for AI Automations<\/h2>\n<p>AI automations rarely fail on launch day. They fail later and quietly. The model underneath changes or retires. The policies they quote go out of date. Outputs drift while dashboards stay green. New tools assemble a security risk nobody approved. The person who understood the workflow moves on. None of these problems are exotic. They&#8217;re documented in research going back to Google&#8217;s 2015 work on machine learning technical debt, in vendors&#8217; own deprecation policies, in tribunal rulings, and in a steady stream of security disclosures.<\/p>\n<p>The good news is that the fixes are ordinary operational discipline, not research projects. Here&#8217;s where to start this month:<\/p>\n<ol>\n<li><strong>Build the registry.<\/strong> List every AI automation that touches customers, money, or shared data, with a named owner, backup, exact model identifier, and downstream consumers.<\/li>\n<li><strong>Add retirement dates.<\/strong> Check every model in the registry against the provider&#8217;s deprecation page and record known end dates. Flag anything running on a preview model in a high-risk workflow.<\/li>\n<li><strong>Map trifecta exposure.<\/strong> For each automation, mark whether it has private data access, untrusted input, and external communication. Anything with all three gets reviewed first.<\/li>\n<li><strong>Create golden sets for your top five.<\/strong> Start with the highest-risk automations: 50 to 200 real examples with correct answers, run on a schedule and before every change.<\/li>\n<li><strong>Add one semantic monitor per workflow.<\/strong> Schema validation, category distribution, or correction rate. Pick whichever fits, and alert on changes.<\/li>\n<li><strong>Set a human sampling cadence by risk tier<\/strong> and record findings in a structured way.<\/li>\n<li><strong>Write the migration runbook once<\/strong> and use it every time a deprecation notice arrives.<\/li>\n<li><strong>Cost out maintenance<\/strong> for each automation, compare it to time saved, and retire the ones that don&#8217;t earn their upkeep.<\/li>\n<\/ol>\n<p>The Air Canada tribunal summed up the principle behind all of this in one line: it makes no difference whether information comes from a static page or a chatbot. Your organization owns what its automations say and do, on day one and on day four hundred. Building the automation is the easy part. Keeping it right is the actual work, and the teams that plan for it from the start get the full value of what they&#8217;ve built.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>AI automations rarely fail at launch. They fail months later through model drift, retirements, silent errors and missing owners. Here&#8217;s how to maintain them.<\/p>\n","protected":false},"author":1,"featured_media":412,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[510],"tags":[93,136,477,484,479,154],"class_list":["post-413","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-automation","tag-ai-automation","tag-ai-governance","tag-llm-monitoring","tag-model-deprecation","tag-prompt-injection","tag-workflow-automation"],"_links":{"self":[{"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/posts\/413","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/comments?post=413"}],"version-history":[{"count":1,"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/posts\/413\/revisions"}],"predecessor-version":[{"id":561,"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/posts\/413\/revisions\/561"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/media\/412"}],"wp:attachment":[{"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/media?parent=413"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/categories?post=413"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.algofuse.ai\/blog\/wp-json\/wp\/v2\/tags?post=413"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}