Why every AI automation needs a regular checkup

When you build an AI automation, you pick a model: which company's AI, which size, at what price. It's easy to treat that as a decision you make once. It isn't.

In one week this fall, across two of my own apps:

• I moved my board game app off a model OpenAI is shutting down in December.
• OpenAI and Anthropic both released new models that cost less than the ones they replaced.
• A different kind of AI model made it worth rebuilding an automation that was working fine.
• The table I use to calculate AI costs had drifted out of date in two places.

None of that shows up in the automation itself. It keeps running until the day its model is switched off, and in the meantime it may cost several times what it needs to. Most business owners will never see the retirement notice or the price announcement. Someone has to be watching, and testing.

When a model is retired, measure the replacement

Roll Play, the board game collection app I'm building, ran its chat, the game summaries it writes, and the titles it gives each conversation on GPT-5 nano. OpenAI shuts that model down on December 11, 2026. Switching took one afternoon, and the obvious replacement turned out to be the wrong one.

The obvious plan was the next model in the same line, GPT-5.4 nano. But both of the AI companies I use had just released new models, so I priced the field:

Model	Input	Output	Where it stood
GPT-5 nano	$0.05	$0.40	Being retired
GPT-5.4 nano	$0.20	$1.25	The obvious successor
GPT-6 Luna	$0.10	$0.50	OpenAI's newest small model
Claude Haiku 4.5	$1.00	$5.00	Anthropic's small model, a year old
Claude Sonnet 5.5	$2.00	$10.00	Anthropic's newest mid-size model
Claude Haiku 5.5	$0.10	$0.50	Released October 7, after I'd switched

Prices are per million tokens, from each provider's list prices in October 2026.

Nothing was going to stay as cheap as the model being retired. The obvious successor cost four times as much for input and about three times as much for output. The year-old Haiku cost ten times what the brand-new Luna did. Sonnet is new and very capable, but priced for harder jobs than this one. Of what was out in September, Luna was the sweet spot: brand new, and cheaper than everything except the model going away.

Price only matters if the answers hold up. Roll Play has a grader: a set of real chat questions, each with an answer I've checked, that runs against any model and scores the results. When I built it in May, it had scored GPT-5.4 nano below the model it was supposed to replace, 85.5% to 89%. This time I ran the retiring model and Luna side by side:

	GPT-5 nano	GPT-6 Luna
Answers that passed	82%	91%
Accuracy score (F1, 0 to 1)	0.88	0.95
Time per call	5.0 seconds	3.5 seconds

Luna is better and faster, and it cost well under everything else that was out in September. It still costs more than nano did, but I could only see that it was the right trade with the numbers in front of me. Without the grader, I'd have taken the obvious upgrade and paid more for a worse app.

The grader needed work first, and that was a lesson in itself. I hadn't run it since May, and the app had changed under it: the grader no longer even compiled, and it didn't send the model the same instructions the live app does. A test that doesn't match production measures the wrong thing. Fixing it is likely part of why the old model scored 82% here, against 89% in May.

Once the numbers were in, I didn't wait for the shutdown date. The old model came out of the code the next day.

A few weeks later, Anthropic released Claude Haiku 5.5 at exactly Luna's price. So the decision I made in September already has a new candidate. With the grader working, checking it is an afternoon's work, not a guess.

Keeping up cuts costs too

Retirement forces a switch. Most switches aren't forced, and those are the ones that save money.

The same weekend, I updated the models in Wombot, the agent that runs my scheduled automations. Nothing there was being retired. Its default model was Claude Haiku 4.5, the year-old model from the table above, at ten times Luna's price. That was one of seven model changes I made to Wombot that day, as both OpenAI and Anthropic released new models at lower prices.

Price per token is only half the story, since models differ in how many tokens they use for the same work. So the number to watch is cost per run. My daily to-do job had moved off Haiku to the previous version of Luna earlier in September. Same job, same work: its cost fell from about 2.2 cents a run to about 0.7 cents, less than a third of what it was.

At my volume, twice a day, that's pocket change: about $16 a year down to about $5. But the ratio is what carries over. The same switch on an automation that runs 1,000 times a day would be the difference between about $8,000 a year and about $2,400.

Sometimes the better move is a rebuild

The bigger gain came from questioning the job itself.

For months, that daily job was one AI agent doing everything I described in letting AI triage my to-do list: tagging tasks, filing them into projects, and rewriting vague titles. It worked. But new options had arrived since I built it, including Jev, a decision model that answers yes-or-no and pick-one questions for a fraction of a cent. Most of what the job did was exactly those kinds of questions: does this task need the Errand tag? Which project does it belong in?

So I rebuilt it. Jev now makes the tagging and filing decisions, and Luna only does the parts that need writing, like rewriting a title or answering a question I've left in a task's notes. Before switching, I measured the new version against my own past choices:

• Filing: it picked the exact project I had for 72% of tasks, and the right area for 89%.
• Headings within a project: it matched mine for 46 of 50 open tasks.
• Tags: I tuned each tag's threshold against tasks I had tagged myself.

The new job does more than the old one did, and this month its runs have averaged about two-tenths of a cent. They aren't the same job, so that's not a clean comparison, but the direction is hard to miss.

None of this was broken. The old job would have kept running fine. It just wasn't the best way to do the work anymore.

Prices drift, so the bill does too

Updating Wombot's models meant updating its price table too, the list it uses to work out what each run costs. Comparing it with the providers' current pricing pages turned up two places where it had drifted from what they actually charge:

• Web searches weren't in it. Both providers charge a penny per search, on top of the model itself. One test answer that used a single search cost $0.0135, and Wombot logged it as $0.0033. With a cheap model, the search is most of the bill.
• Writing to the provider's cache had a new price. OpenAI now charges 1.25 times the normal input rate for it, and Wombot still had it at the plain rate.

That's drift, not a bug. Vendors add charges, change rates, and price new models differently, and a cost table that was right when it was written slowly stops being right. If your cost numbers come from your own math rather than the vendor's invoice, someone has to check that math every time a model or a price changes.

Support is more than keeping the lights on

AI automation isn't build-once software. In one week I dealt with a model retirement, new releases from two vendors at lower prices, a job worth rebuilding around a different kind of model, and a price table that had drifted. That's not unusual. It's what this field looks like right now.

Handling the retirement is the minimum: it keeps the automation running. The value is in everything after that: rerunning the tests when a better model comes out, catching that a job could cost a tenth of what it does, noticing when an automation built six months ago should be rebuilt with what's available today.

It takes a few habits for every automation:

1. A test set. A few dozen real inputs with answers you trust, so any model can be scored in minutes. Kept in step with how the automation actually works.
2. A cost per run. Logged every time, so a price change shows up as a number, not a surprise on the invoice.
3. A price check. Your cost math compared with the vendor's pricing page whenever a model or price changes.
4. Someone reading the vendor's notices. A retirement email starts the clock.
5. A regular review. Every few months, run the test set against what's new, and ask whether the job itself should change.

Some places this matters in a business:

• A support assistant: when its model is retired, a test set of past questions and approved answers shows whether the replacement still gets your policies right.
• Invoice or form extraction: a sample of documents with the right values filled in shows whether a cheaper model still reads them correctly.
• Lead or email sorting: a set of messages you've already sorted by hand shows whether a decision model can sort them for a fraction of what a chat model costs.
• Anything billed per run: a cost log turns "the new model is half the price" into something you can confirm.

In each case a person decides whether to switch. The test set and the cost log make it a decision based on numbers, not a vendor email.