Noticed this pattern reviewing how people actually use AI tools for writing transformation queries. First few outputs get checked carefully, run against a sample, compared to expected results. After enough of those come back correct, the checking quietly stops. Not a decision anyone makes on purpose, it just fades, because checking something that's been right nine times in a row feels like wasted effort in the moment.

The problem is that correctness on the first nine doesn't predict correctness on the tenth. Nothing about the model improved or built trust in a way that actually reduces its error rate on the next query, it's still working from the same context window, same limitations, same chance of misreading an edge case in the schema. What changed is the human's willingness to look, not the model's actual reliability.

This shows up worse on queries that produce plausible wrong numbers instead of obvious failures. A query that returns zero rows gets noticed immediately. A query that silently double-counts something due to a join issue produces a number that looks completely reasonable, and by the point someone's stopped spot-checking, that's exactly the kind of error that gets through.

Don't have a clean fix for this beyond forcing some kind of check that doesn't rely on remembering to be suspicious, a fixed row-count sanity check that runs regardless of how many previous queries were correct, something that doesn't degrade as trust builds the way manual vigilance does.