An automated system where it's impossible to know what it did, when, with what data, and why, isn't being operated; it's being supported. The difference becomes apparent the day something goes wrong, and at that point, there's no way to reconstruct what happened: the necessary data wasn't saved because no one decided to save it.
This is probably the most widespread design flaw in the AI-component systems we audit, and the cheapest to avoid if addressed early on.
Monitoring means verifying that the system is alive: it responds, doesn't produce errors, and latency is acceptable. It is necessary, but not sufficient.
Observation allows you to answer questions you didn't anticipate. Why did this specific case resolve itself this way? Which version of the model was involved last Tuesday? How many times did the system have to retry before responding? What percentage of the agent's suggestions are being corrected before approval?.
The distinction is especially important with probabilistic components, because an AI system can be perfectly healthy in terms of infrastructure—zero errors, low latency, full availability—and still be producing progressively worse results. A traditional monitoring panel doesn't detect that. It shows everything in green.
We have developed it in more detail in LLM observability, the metric that nobody is measuring
|
Layer |
What is recorded |
What is it for? |
|
Entrance |
What data and what context did the system receive? |
Reproduce the case and detect anomalous data |
|
Decision |
What did the model produce, with which version and with what configuration? |
Explain the result and detect drift |
|
Action |
What was actually run on which system |
Audit, reversal and liability |
|
Validation |
Who reviewed it, what changed, and when? |
Measure actual quality and ensure human supervision |
|
Cost |
Consumption by execution and by business case |
Controlling unit cost at scale |
The validation layer is the most frequently overlooked yet most valuable. Recording what the person corrected before approval transforms human oversight into a data flow about system quality. Without this record, the only available signal is the customer complaint, which arrives late and biased.
The cost layer is the second most overlooked. An AI system has a variable cost per execution, which is unusual in enterprise software. If this cost isn't tracked by business case, it's impossible to know if the automated process is profitable, and that question always comes up, usually during the following quarter's budget review.
Technical indicators—latency, tokens, API error rate—serve the operational team. Only four are relevant to a steering committee:
Acceptance rate unchanged. Of the proposals generated by the system, what percentage are approved as is? This is the most honest quality indicator available because it is based on real-world use, not laboratory testing.
Human scaling rate. What percentage of cases can the system not resolve? This should have a target value: if it's too high, the system isn't contributing; if it's zero, it's probably resolving cases it shouldn't.
Cost per case resolved. The total cost divided by the number of cases actually closed, not by the number of attempts. This is the number that is compared to the cost of the previous process.
Fault detection time. How long does it take for an organization to realize that the system is producing incorrect results? If no one has measured this, the real answer is usually "when a customer complains.".
These four metrics can be presented on one slide and answer the only question that matters to a board: Does this work, how much does it cost, and how long does it take us to find out if it stops working?
Until recently, traceability was defended on engineering grounds. Since 2025, there has been a second argument, and it is more difficult to refute in a committee.
The European AI Regulation requires, for certain categories of systems, event logging, technical documentation, and demonstrable human oversight. The NIST AI Risk Management Framework structures risk management around the ability to measure and monitor. The ISO/IEC 42001 standard requires evidence of ongoing control to certify an AI management system.
All three frameworks require the same thing, albeit with different vocabularies: the ability to demonstrate what the system did. A system that didn't record its actions cannot demonstrate them, and adding this information later requires modifying the architecture.
It's important to say this clearly because it changes the order of priorities: Observability has ceased to be an operations task and has become a design requirement..
There's no need to rewrite it. The usual path has three steps and can be executed without stopping the operation:
The mistake to avoid is trying to implement everything at once. A system with two well-registered layers is infinitely more manageable than one with five partially registered layers.
Before considering an automation project complete, it is advisable to carry out a concrete test: Choose a random case from three weeks ago and ask them to explain exactly what happened.
If the team can reconstruct it—what data was input, which version of the model was used, what was executed, and who validated it—the system is operational. If the response begins with "we'd have to look at the logs, although I don't know if we keep those," the system is in production but not finished.
It is the ability to answer unanticipated questions about the system's behavior: why a specific case was resolved in a certain way, which version of the model was involved, how many retries there were, or what proportion of proposals are being corrected. It differs from monitoring, which only verifies that the system is available and responsive.
Five layers: input (received data and context), decision (model output, version and configuration), action (what was executed and on which system), validation (what the reviewer corrected), and cost per execution and per business case.
Four: unchanged acceptance rate, escalation rate to human, cost per resolved case, and time to failure detection. Technical metrics like latency or token consumption serve the operations team, not the business decision.
Yes. The European AI Regulation requires event logging, technical documentation, and demonstrable human oversight for certain categories of systems. The NIST AI RMF structures risk management around the ability to measure and monitor, and ISO/IEC 42001 requires evidence of ongoing control for certification.
Yes, without rewriting it. The effective order is to first implement input and action—which allow reconstructing an incident—then add the human validation record and define the alert thresholds before having the data, not after.
Because infrastructure indicators (availability, latency, errors) can be green while the quality of the results degrades. The quality of a probabilistic component is not measured with traditional monitoring tools, but with performance metrics based on actual usage.
|
Could you reconstruct today what your AI system did three weeks ago? If the answer is unclear, it's best to review it before a client or auditor asks the question. We'll analyze your case in two hours, with no obligation. Let's talk → |
Español Español 
English
Français
العربية
Português
Deutsch
Italiano
