It increasingly feels like AI is becoming the new snake oil, with companies being sold ambitious promises by consultants about what AI can supposedly automate.
The problem becomes much clearer when you actually test these systems internally. Even some of the most expensive and capable models, such as Opus, are not capable of achieving 100% accuracy on many business tasks... particularly tasks where the output must be completely correct.
Of course, there is always the disclaimer that “AI-generated responses may contain mistakes.” But I think the significance of that disclaimer is being heavily glossed over when we start talking about applying AI to real business processes. The obvious question is: who takes responsibility when the AI gets something wrong?
When I raise this with consultants, the response is usually along the lines of: “AI will never be 100% accurate. The goal isn't to replace employees, but to enhance their productivity and free them up to focus on higher-value tasks.”
But I'm not convinced that this argument always makes sense.
Suppose an employee is paid R20,000 a month to perform a process correctly and ensure that certain requirements are followed. We introduce an AI system costing another R2,000 a month to automate that process. However, because the AI isn't 100% reliable, the employee still has to review and verify its output.
At that point, what have we actually automated?
If verification takes a significant amount of time, and the employee remains fully responsible for identifying and correcting the AI's mistakes, there is a strong incentive for them to simply perform the task manually. With a manual process, they understand exactly how the decision was reached and can confidently explain or defend it if questioned later.
This is where I think the conversation around AI often becomes disconnected from the realities of business processes. It's not enough for AI to be “mostly accurate” if the human still carries 100% of the accountability.
For AI to genuinely replace or materially reduce the work involved in a process, the organisation needs to be comfortable with the residual error rate or have sufficiently strong controls around the AI that the human verification burden is genuinely reduced.
Otherwise, we're not really eliminating the work. We're potentially just adding another layer: AI generates the output, an employee checks it, and the organisation still carries the liability when something slips through.
And if that's the model, I think businesses need to be much more honest about what they are actually buying when they invest in AI automation.