How much should AI do on its own? Let its confidence decide

The first question most business owners ask me about AI is some version of "Will it send something in my name that I wouldn't?"

It's the right question, and the answer isn't yes or no. An automation doesn't have to choose between doing everything on its own and doing nothing without permission. There are three settings, and a good system uses all of them:

• Full auto, when the system is confident. A routine question with a clear answer doesn't need a person.
• A person checks, when it isn't confident, or when the system is new. Until you've watched it make good calls for a while, you approve before anything goes out.
• A hybrid. The automation handles the easy part right away, and a person follows up on the rest.

The trick is that the system has to know how sure it is, and tell you. Here's how that works in an automation I run for my own apps.

Replying to App Store reviews

People review my apps on the App Store, and a good reply matters. It can turn a one-star review around, and it shows everyone reading that someone is listening. But checking for new reviews, deciding which ones need an answer, and writing a decent reply is exactly the kind of small chore that slips.

So I built a workflow in n8n that runs every 12 hours:

1. It pulls any reviews posted since the last run.
2. A decision model reads each review and answers three questions: how serious is the problem (on a five-point scale), what is it mainly about (a bug, pricing, a feature request, praise, or something else), and how likely is it that a reply would help.
3. Reviews that probably need a reply go to Claude, which drafts one in the reviewer's own language, with an English translation for me.
4. I get an email with the review, the analysis, and the draft. I approve it, edit it, or reject it.
5. Approved replies post to the App Store automatically.

The approval email for a German review, from a test run on my older reviews. The model calls it a feature request with 82% confidence and a 90% chance a reply would help, and the draft comes in English for me and German for the reviewer.

The email is the part that matters here. It doesn't just show me a draft. It shows me how sure the system is: the category and the model's confidence in it, the severity, and the chance that a reply is needed at all. When a review is labeled "praise" with high confidence and the draft is a two-line thank-you, I can approve it in seconds. When it's a low-confidence call on an angry review, I read more closely.

Why every reply waits for me, for now

Right now, nothing posts without my approval. That's not because the drafts are bad. It's because of where this system is in its life:

• It's new. I built it at the end of September. Until I've seen it make the same good call many times, I want to see every one.
• The replies are public, and they go out in my name. A wrong reply to a customer is worse than a late one.
• My volume is low. Approving a handful of replies a week costs me almost nothing.

A business with hundreds of reviews a month would weigh that differently, and that's the point. Where the line sits is a business decision, not a technical one.

Moving the line as trust grows

Because the system reports its confidence, the line can move one category at a time instead of all at once. The next step I'd take is simple: a high-confidence "praise" review with a short thank-you could post on its own. Bug reports, refund demands, and anything the model is unsure about would still come to me.

That's how I'd roll this out for a client, too. Start with a person approving everything. Watch where the system is reliably right. Then hand it those cases, and keep a person on the rest.

The hybrid: answer now, follow up anyway

Some jobs don't fit cleanly into "auto" or "approve." Think about a business inbox that fills up with the same handful of questions: hours, pricing, how to book a party. A hybrid automation can answer the common questions right away from an approved list of answers, so the customer hears back in minutes instead of days. Then it flags the message for a person, who can follow up if the automatic answer missed something.

My own email assistant works this way: it replies to every email right away, copies me on the reply, and I take the conversation from there.

The customer gets speed. The business keeps the personal touch. And nothing that needs judgment goes out without someone looking at it.

Where this fits in your business

The same pattern works for any stream of messages that needs sorting and a response: Google and Yelp reviews, contact forms, support email, new lead inquiries. In every case, the questions are the same:

• How sure is the system, and does it tell you?
• Which cases are safe to hand over completely?
• Who checks the rest, and how easy is it for them to approve?

Automation isn't all or nothing. Start with a person in the loop, let the system earn its independence one category at a time, and keep a person wherever it matters.