You ask your AI agent, "Is the payments service healthy?" It answers, "It's degraded. I opened a ticket."
That is all you see: a question going in and an answer coming out. Which model calls it made, which tools it used, what it decided along the way, how long each step took, what it cost, and what failed are all hidden. For a demo, that is fine. For an agent that touches production systems, it is a problem.
We recently ran an internal training on agent observability as part of our AI Talks series. Our data team built the same setup twice, once on AWS and once on Databricks, and compared the results. Here is what we learned.
Think about tracking a package. It gets ordered, sits in a warehouse, goes out for delivery, and arrives. At any point you can see where it is and how long each leg took.
A trace does the same for a request to your agent. It is the full path from question to answer. Each step inside it is called a span, and a span records what ran, when it started, how long it took, and what went in and out.
Here is a real trace from our training. The question was "Is payments-api healthy?"
get_service_status: 0.36 screate_ticket: 0.21 sThe model spent 88% of that time thinking. The tools were fast. Without the trace, you would probably guess the opposite and start optimizing the wrong thing.
A trace gives you four things you cannot get from the input and output alone. You see where the time went, step by step. You see what it cost, because tokens per step are the unit model providers bill by. When something breaks, you see the exact step that failed and its error message. And you see what the agent decided: which tools it picked, with what data, and what came back.
OpenTelemetry is an open standard for telemetry data, backed by the Cloud Native Computing Foundation. It is free and vendor neutral.
Because it is a standard, you instrument your agent once and send traces wherever you want. Changing where the traces go does not mean rewriting the agent. AWS packages OpenTelemetry as ADOT (the AWS Distro for OpenTelemetry), and Databricks, Azure Monitor, and most observability tools can read it.
Monitoring tells you the agent ran. Evaluation tells you whether it answered well. You need both, and the second one is harder.
The common approach is LLM-as-a-judge. A second AI model reads the trace and scores the answer against criteria you define. As our demo showed, the criteria matter more than the judge.
Every agent observability setup we have built follows the same four steps:
AWS and Databricks handle the first two steps the same way. They differ in steps three and four.
AWS uses a lot of names for this. In plain terms:
A trace travels from the agent (ADOT records each span), through an optional collector that samples, filters, and masks data, into X-Ray and CloudWatch, and finally into Omni, where you explore, investigate, and evaluate.
We built an "operations assistant" agent on an Amazon Bedrock model. It answers questions like "Is payments-api healthy?" using three simulated tools: one that checks a service's status and always works, one that searches incidents and is slow and fails one time in four on purpose, and one that opens a ticket for degraded or down services.
Each test run asked six questions and recorded 75 spans. Traces showed up in CloudWatch about five minutes later, and the agent itself contained zero lines of tracing code.
The most useful result came from this question: "Should I worry about anything right now? Check all known services."
The trace had 19 spans, took 7.83 seconds, used 4,706 tokens, and recorded one tool error. Here is what the agent did. It checked payments-api and found it degraded. It checked auth-service and found it healthy. It asked about two services that do not exist. Its incident search timed out. And it never checked card-issuing.
Card-issuing was down, with 1,290 errors in the previous hour.
The agent did not crash, and nothing looked wrong from the outside. It gave a confident answer that missed the one outage that mattered most.
We ran the trace through the built-in evaluators:
Only helpfulness caught the gap, and even it gave a passing-looking score. Correctness and tool use were too lenient. A generic judge has no way of knowing that skipping a service that is down is the worst mistake a payments company's assistant can make. It can also only judge what is in the trace.
AWS lets you add custom evaluators in AgentCore, and they show up in Omni next to the built-in ones. They come in two kinds.
An LLM-as-a-judge evaluator uses a model, your instructions, and a scoring scale. For our case, the instruction was: "For a general status check, every known service must be checked. Skipping one is a FAIL." This kind works best for judgment calls such as tone, clarity, or whether the answer addressed the question.
A code-based evaluator is a small AWS Lambda function that reads the spans. Ours counted the services checked: 2 of 4, so FAIL. For objective rules like this one, code is exact and cheap, and it returns the same result every time.
For the Databricks case we set ourselves a specific question: if your agent traces already sit in Amazon S3, how do you analyze and evaluate them in Databricks? That is a common situation when agents run on another platform, or when traces were collected before anyone planned to study them.
This was one scenario, chosen on purpose. Databricks can also handle the whole flow on its own, with agents traced straight into MLflow, and you do not need S3 in the middle to use it.
In our exercise, the route looked like this. The agent recorded spans with OpenTelemetry, the trace files landed in S3, Unity Catalog gave Databricks secure read-only access to them, MLflow 3 provided the trace viewer and SQL, and evaluation ran on top with scorers, custom judges, and Genie. The agent that produced the traces never had to change.
Here is what we did:
make_judge, ranging from judges that look only at the final answer to judges that read the full trace.We also applied this pattern to AI code review on one of our projects. Review comments from Copilot, SonarCloud, and pipeline policies were already stored in Databricks, along with how each one ended up: resolved or "won't fix."
An LLM judge with a rubric now scores each AI comment on whether it is valid, how severe it is, and whether a developer can act on it. Every judgment is traced in MLflow. To check the judge itself, we compared its scores with a Tech Lead's on about 50 comments and adjusted it until they matched.
The output is an informational PR review block that does not block merges and did not need a new platform or API. The team can now measure how precise the AI reviewer is (valid findings divided by findings judged, per project and repository) and how often its suggestions get accepted, compared with the judge's view. Every score comes with its reason, the rubric version, and the evidence, so it can be audited.
make_judge.You do not have to pick one. If you mostly work in AWS, CloudWatch and Omni are native to your account and you do not need to adopt a new platform. If your data lives in Databricks, MLflow 3 lets you judge agent quality right next to your business data, whether you trace agents there directly or bring in traces you already store in S3. If you need both, one OpenTelemetry setup can send the same traces to two places: CloudWatch for live operations and Databricks for deeper analysis.
Generic judges are lenient. If a business rule matters, write an evaluator for it, and if the rule is objective, a few lines of code usually beat a model.
Traces carry more than you expect. Ours included prompts, responses, and even access key IDs. Filter sensitive data in the collector before anything gets stored.
Cost is a design decision. Sampling, retention, and indexing are what keep the bill low, so decide on them early.
New tools have rough edges. The Omni setup failed twice before it worked for us. Run a pilot before you commit.
At Celerik we build AI agents and the data infrastructure around them for companies that need those agents to work in production. We build observability in from day one, so your team can see why an agent made a decision before you rely on it for real ones.
If you are running agents and are not sure what they are doing between the question and the answer, let's talk.
