AI Agent Observability: Metrics, Traces, and Logs That Actually Matter

June 30, 2026 38 min read

observability monitoring agent-governance proxy opentelemetry

Most AI agent monitoring captures the wrong things. Here is what to instrument across metrics, traces, and logs, and the signals that actually tell you something.

The agent that failed quietly for three weeks

A support team ran a customer-facing agent that answered billing questions. The agent had a dashboard. The dashboard showed request count, average latency, and a green uptime indicator. For three weeks every chart stayed green. The agent was up. The agent was fast. The agent was answering.

The agent was also wrong about forty percent of the time.

It had a knowledge-base search tool and a billing-lookup tool. Somewhere in week one, a prompt template change shifted how the agent phrased its search queries. The knowledge-base tool kept returning results, just the wrong results, and the tool call kept returning HTTP 200. The agent took those wrong documents, reasoned over them confidently, and produced answers that sounded right and were not. No error was thrown. No latency spike appeared. No alert fired. The dashboard had no signal that could have caught it, because the dashboard measured a web service and the thing running was an agent.

The team found out from a spike in support escalations, which is the worst possible monitoring system: angry customers. When they went back to investigate, the data they needed did not exist. They had request counts. They did not have the tool-call results, the retrieved documents, the search queries, or the reasoning the agent did between them. The information that would have explained the failure had never been recorded.

This is the agent observability gap. Teams instrument agents the way they instrument microservices, get a dashboard that looks complete, and miss the failure modes that are specific to agents. Agents fail in ways HTTP services do not. A service that returns 200 did its job. An agent that returns 200 might have done the wrong job perfectly. Observability built for the first case is blind to the second.

This post is about what to instrument instead. The three pillars of observability still apply: metrics, traces, and logs. But each pillar means something different when the thing you are observing is an agent, and the default instrumentation captures the service-shaped version of each pillar rather than the agent-shaped one. We will go through all three, name the signals that matter, name the ones that are noise, and cover how OpenTelemetry’s GenAI conventions and a governance proxy make the agent-shaped version practical to collect.

What does observability mean for an AI agent?

Observability is the property of a system that lets you understand its internal state from the outside, using the data it emits. For a traditional service, the internal state worth understanding is mechanical: is it up, how fast is it, how often does it error. For an agent, the internal state worth understanding includes a harder layer: did it pick the right tool, did it interpret the result correctly, did it pursue the goal it was given, and did any of that change over time. The mechanical layer still matters. It is no longer sufficient.

Agent observability differs from service observability in three structural ways.

The unit of analysis is the session, not the request. A traditional APM tool treats each HTTP call as the thing to measure. An agent task is not one call. It is a loop: the model decides, a tool runs, the result comes back, the model decides again, and the loop continues until the agent believes the task is done. A single user request can produce twenty model calls and forty tool calls. Measuring any one of those calls in isolation tells you almost nothing about whether the task succeeded. The session, the full loop from user request to final answer, is the only unit at which “did it work” has a meaning.

Failures are frequently silent. A microservice failure is usually loud: an exception, a 500, a timeout, a stack trace. An agent failure is often a clean HTTP 200 with the wrong content inside it. The model selected a tool that was not the right tool. The tool returned a result the model misread. The model hallucinated a parameter that happened to be syntactically valid. None of those trip a 5xx alert. The OneUptime team writing about the agent observability gap named three of these soft failure modes directly: wrong tool selection, incomplete or unexpected tool returns, and parameter hallucination. All three return 200. All three need semantic inspection, not status-code inspection, to catch.

The system is non-deterministic. The same input to the same agent can produce a different sequence of tool calls on two consecutive runs, because a language model sits in the loop. Traditional observability can lean on the assumption that identical inputs produce identical execution paths. Agent observability cannot. This is why per-session tracing is not a nice-to-have. When every run is potentially unique, the trace of the specific run that failed is sometimes the only artifact that explains the failure.

The three pillars divide the work cleanly. Metrics are cheap, aggregate, and always on; they tell you that something changed. Traces are per-session and structured; they tell you where in the loop the change happened. Logs are per-event and detailed; they tell you exactly what the content was. A team that has all three can move from “escalations are up” to “the search queries changed on June 3rd and started returning the wrong docs” in minutes. A team that has only the first pillar, the service-shaped dashboard, finds out from the customers.

What metrics matter for AI agents?

Metrics are the aggregate, numeric view of agent health. They are the cheapest pillar to collect and the right place to put alerts. The mistake teams make is collecting the metrics a web service needs and stopping there. Request count and uptime are real metrics. They are not the metrics that catch agent failures. Here is the set that does.

Token usage per request, split by input and output. Every model call consumes input tokens and produces output tokens, and the two have different prices and different meanings. Input tokens climbing usually means context is growing: a longer system prompt, accumulated conversation history, larger retrieved documents. Output tokens climbing means the model is generating more, which can mean it is reasoning longer or looping. The OpenTelemetry GenAI metrics spec defines gen_ai.client.token.usage as a histogram for exactly this, with a gen_ai.token.type attribute to separate input from output. Collect it as a histogram, not an average. Averages hide the long tail, and the long tail is where the expensive requests live.

Cost, computed twice: per request and per task. Cost per request is token usage times the model’s per-token price, and it is straightforward. Cost per task is the sum across every model call in a session, and it is the number that actually matters, because the unit of work is the task. An agent that costs two cents per model call and makes sixty calls to finish one task costs more than a dollar twenty per task. The per-call metric looks fine. The per-task metric is the one that shows up on the bill. Track both, and treat per-task cost as a primary metric, not a derived one.

Latency percentiles, measured at p50, p95, and p99, for both model calls and tool calls. Average latency is close to useless for agents because the distribution is wide and skewed. The OpenTelemetry spec captures this with gen_ai.client.operation.duration, a histogram with bucket boundaries that run from ten milliseconds to over eighty seconds, which tells you how wide the real distribution is. What you want from latency metrics is the breakdown: how much time was model inference, how much was tool execution, how much was your own orchestration code. A task that takes thirty seconds is fine if twenty-eight of them were a slow third-party API and a problem if twenty-eight of them were your retry logic spinning.

Tool error rate, per tool, not aggregated. An aggregate tool error rate of two percent can hide one tool failing thirty percent of the time and nine tools failing at zero. Break it out by tool name. A tool whose error rate climbs is either broken, being called with bad parameters, or being called in a situation it was not built for, and all three are worth knowing before the customers do.

Cache hit rate. If you cache model responses or tool results, the hit rate is a direct cost lever and a direct latency lever. A hit rate that drops suddenly means something upstream changed the inputs enough that previously cacheable requests no longer match, which is often the first visible symptom of a prompt template change. Anthropic and OpenAI both expose prompt-caching token counts in their usage objects; surface them as a metric.

Tool calls per task, and loop or retry rate. The number of tool calls it takes an agent to finish a task is a health signal. If the median task used to take eight tool calls and now takes nineteen, something changed: the model is less sure, the tools are returning worse results, or the agent is caught in a partial loop. Loop rate, the fraction of sessions where the agent repeated a near-identical call several times, is the metric that catches the runaway-cost class of incident. An agent stuck in a loop produces a flat request-count chart and a vertical cost chart.

Metric	What it catches	Alert when
Cost per task	Runaway spend, looping, context bloat	Above budget ceiling, or 2x week-over-week
Tool error rate per tool	A broken or misused tool	Any single tool above 5% sustained
Latency p95, per phase	Slow tools, slow orchestration, regressions	p95 above task SLA
Loop / retry rate	Stuck agents, the runaway-cost incident	Above 1% of sessions
Tool calls per task	Degraded tool quality, less certain model	Median moves 50%+ from baseline
Cache hit rate	Prompt or input drift, cost regression	Sudden drop of 20+ points
Token usage, input split out	Context window bloat	Steady upward trend with flat task volume

The discipline with alerts is to alert on the agent-shaped metrics, not the service-shaped ones. An alert on raw request count tells you traffic changed, which you usually already know. An alert on cost per task or loop rate tells you the agent’s behavior changed, which is the thing you actually need to hear about. We covered the cost side of this in detail in detecting token count manipulation; the short version is that cost metrics are only trustworthy if you collect them at a layer the agent does not control.

What should an agent trace capture?

A trace is the structured record of one session. Metrics told you something is wrong across the population of sessions. The trace tells you what happened inside one specific session, step by step, with timing and parentage. For agents, the trace is the single most valuable artifact, because when a non-deterministic system fails on one run, the trace of that run is sometimes the only thing that explains it.

The structure that works is a tree of spans. The root span is the session: one user request, one final answer, everything in between. Under the root sit the steps of the agent loop, each as a child span. The OpenTelemetry GenAI agent spans convention, currently in Development status, defines the span types directly: an invoke_agent span for an agent invocation, an execute_tool span for each tool call, an invoke_workflow span when several agent invocations are coordinated together, and a create_agent span for agent setup. Span names follow patterns like invoke_agent {gen_ai.agent.name}. The convention is experimental and the attribute set will move before it stabilizes, but the span shape is stable enough to build on now, and building on it means your traces speak the same vocabulary as every other tool that adopts the spec.

Four span types carry the weight in an agent trace.

Model call spans. Each call to a language model is a span. It carries the model name and version, the input and output token counts, the latency, the temperature and other sampling parameters, and the finish reason. The finish reason matters more than people expect: a model call that ended because it hit the max-token limit rather than completing naturally is a span worth flagging, because a truncated model response is a common upstream cause of a downstream agent failure.

Tool call spans. Each tool invocation is a span, and it is the span type that catches the most failures. It carries the tool name, the input parameters, the return value, the latency, and the status. The status is where the silent failures surface: a tool call that returned HTTP 200 but produced an empty result, or a result that does not match the schema the agent expected, is visible in the span even though it never tripped an error. The OneUptime analysis of soft agent failures is essentially a list of things that are invisible at the metric layer and visible at the tool-span layer.

Sub-agent spawn spans. When an agent spawns a sub-agent, the sub-agent’s entire trace should nest under a span in the parent’s trace. This is what keeps a multi-agent system debuggable. Without nesting, a multi-agent system produces a pile of disconnected traces and no way to see which parent decision led to which child’s work. With nesting, the orchestrator’s trace contains the full tree, and you can read a five-agent task as one document.

Retry spans. Every retry is its own span, never a silent re-run folded into the original. If the agent called a tool, got an error, and called it again, that is two spans, and the trace should show both. Retries folded invisibly into a single span are how a trace lies to you: the session looks like it took one tool call when it took four, and the cost and latency do not add up.

The trace tree is what turns “the session failed” into “the billing lookup failed, the retry recovered, the sub-agent ran on stale policy data, and the final answer was built on it.” Each span is small. The value is in the structure: parentage shows which decision caused which action, and timing shows where the wall-clock time actually went. Adopt the OpenTelemetry trace shape rather than inventing your own; every backend that speaks the spec, and the major ones increasingly do, can then read your traces without custom glue.

What belongs in agent logs?

Logs are the per-event detail layer. The trace told you the tool call failed; the log tells you the tool was called with account_id: null and returned error: account not found. Logs are where the actual content lives, and content is exactly what agent debugging and agent evaluation both need. They are also the pillar with the sharpest tradeoff, because the content that makes logs useful is frequently the content you are not allowed to store.

Here is what belongs in agent logs.

Prompt and response content, conditionally. The full prompt sent to the model and the full response it returned are the highest-value log data you can keep. They are what lets you replay a failure, run an evaluation, or understand why the model decided what it decided. They are also the data most likely to contain personal information, regulated data, or secrets. The OpenTelemetry GenAI conventions treat prompt and response content as opt-in for exactly this reason: it is recorded only when explicitly enabled. The right posture is to log content by default in development, log it with redaction in production, and never log it raw on a path that handles regulated data. “When not to” is a real answer here. A medical-records agent should not have full prompt content sitting in a log aggregator, and the absence of that content is a deliberate, defensible choice.

Tool inputs and outputs. The parameters passed to each tool and the values returned. This is the log data that catches parameter hallucination and schema mismatch, the silent failures the metric layer cannot see. Tool I/O is usually lower-risk than prompt content, but not always; a tool that reads customer records returns customer records, and that output needs the same redaction discipline as a prompt.

Policy decisions. If the agent runs behind a policy engine, every allow, deny, modify, and escalate verdict belongs in the log, with the policy version, the input the engine saw, and the reason for the verdict. This is the log data that answers “why was this blocked” and “why was this allowed,” and it is the data that compliance audits actually want. We covered the regulatory side of this in auditing agent activity for SOC 2 and the EU AI Act; the logging requirement and the audit requirement are the same requirement seen from two angles.

Errors with full context. An error log that says “tool call failed” is nearly worthless. An error log that says “tool call billing.lookup failed, parameters were account_id null and region eu-west, the tool returned a 404, this was retry one of two, the session is conv_8841” is something you can act on. Log the error, the inputs that produced it, the retry state, and the trace and session IDs that connect it back to the rest of the picture.

Model and version, on every call. Which model answered, which version of it, which prompt template revision. This is small data and it is the data that makes “answers got worse after Tuesday” diagnosable. A model provider silently routing you to a new snapshot, or your own template changing, is invisible unless the model and version are on every log line.

Retrieval scores and hits. If the agent uses retrieval, log what was retrieved and the similarity scores. Bad retrieval is one of the most common root causes of confidently wrong answers, and it is only visible if the retrieved chunks and their scores were written down. The support agent in the opening of this post failed precisely here, and the reason the team could not diagnose it for three weeks is that this log data did not exist.

The hard rule for agent logs is that every log line carries the trace ID and session ID. A log that cannot be joined back to its trace is an orphan. The point of three pillars is that they connect: the metric alert points you at a time window, the trace points you at a session, the session points you at its log lines, and the log lines point you at the content. Break the join and you have three disconnected datasets instead of one observability system.

What signals are just noise?

Not every number is a signal. Agent observability has its own collection of vanity metrics and over-instrumentation habits, and they cost real money in storage and real attention in alert fatigue. An honest post about what to instrument has to be equally clear about what to stop instrumenting.

Raw request count, as a headline metric, is close to noise. It tells you traffic volume, which you usually know from other sources, and it tells you nothing about whether the agent did its job. A flat request-count chart sat above the failing support agent for three weeks. Request count is fine as context next to a real metric. As a primary dashboard tile it is a comfort blanket.

Uptime, in the binary sense, is misleading for agents. A traditional service is either serving or not, and uptime captures that honestly. An agent process can be one hundred percent “up” and one hundred percent wrong. Uptime measured as process availability is a real metric for the infrastructure under the agent. Presented as agent health it is a false signal, because the failure mode that matters most, the confident wrong answer, leaves uptime at one hundred percent.

Aggregate averages hide the cases you need. Average latency, average cost, average token count: each one smooths away the long tail, and the long tail is where the expensive and slow requests live. A p50 that looks healthy can sit next to a p99 that is ten times worse. Collect histograms and percentiles. An average is a summary of a summary, and by the time a problem moves the average, it has been hurting the tail for a while.

Over-logging is its own failure. Logging every token of every intermediate reasoning step for every request, in production, at scale, produces a log volume that costs more than the model calls and is too large to search when you actually need it. The reasoning content is valuable, and it is valuable as a sampled artifact and as a development-environment default, not as a hundred-percent production firehose. Sample it. Keep full content for the sessions that errored or got flagged, sample the rest, and you keep the debugging value without the bill.

Alerts on metrics nobody acts on are worse than no alert. An alert that fires, gets acknowledged, and never causes an action is training your team to ignore alerts, including the next real one. Every alert should map to a decision. If the cost-per-task alert fires, someone investigates spend. If the loop-rate alert fires, someone looks for a stuck agent. If an alert fires and the honest response is “yes, that happens, ignore it,” the alert threshold is wrong or the alert should not exist.

The principle underneath all of this: instrument for the questions you will actually ask. The questions are “is it costing too much,” “is it failing,” “is it slow,” “did this specific session work,” and “why did it answer that.” Every signal that helps answer one of those questions earns its storage. Every signal that just fills a dashboard is noise wearing a metric’s clothes.

How does OpenTelemetry apply to AI agents?

OpenTelemetry is the open standard for collecting metrics, traces, and logs, and it is a graduated CNCF project that backends across the industry already support. For most of its life it had no opinion about language models. That changed with the GenAI semantic conventions, a set of conventions that define how to represent model calls, agent invocations, tool executions, token usage, and operation latency in OpenTelemetry’s data model.

The value of a convention is shared vocabulary. Without it, every team names its spans and attributes differently, every observability backend needs custom glue to read every team’s data, and nothing is portable. With it, a span named invoke_agent with a gen_ai.agent.name attribute means the same thing everywhere, and any backend that speaks the convention can render your agent traces without bespoke integration work. This is the same payoff policy-as-code gets from a standard policy language, and the same payoff HTTP got from being HTTP.

The GenAI conventions cover the three pillars directly. On metrics, gen_ai.client.token.usage is the token histogram and gen_ai.client.operation.duration is the latency histogram, both with defined bucket boundaries so that two systems measuring the same thing produce comparable distributions. On traces, the agent spans convention defines invoke_agent, execute_tool, invoke_workflow, and create_agent, with attributes for the agent name, the provider, the conversation ID, and token counts. On logs, the conventions specify how to attach prompt and response content as opt-in event data, with the explicit understanding that content is sensitive and recorded only when enabled.

The honest caveat is maturity. The GenAI conventions are in Development status. The span shapes are stable enough to build on, but specific attribute names will change before the spec stabilizes, and instrumentation libraries gate the newest version behind the OTEL_SEMCONV_STABILITY_OPT_IN environment variable so that a convention change does not silently break your pipeline. Building on a Development-status spec is a real bet. It is a bet worth making, because the alternative is a bespoke schema you maintain alone and migrate later anyway, and the convention is clearly where the ecosystem is converging.

The practical question is where the instrumentation runs: inside the agent process with an SDK, or at a proxy on the network layer the agent’s traffic already passes through. The next section is about that choice, because for agent observability it changes what you can actually see.

Proxy-based vs SDK-based observability

There are two places to collect agent observability data. The SDK approach instruments inside the agent process: you import a library, wrap your model client and tool calls, and the wrapper emits metrics, traces, and logs. The proxy approach instruments at the network layer: the agent’s traffic flows through a proxy, and the proxy records every model call and every tool call as it passes. We made the architectural case for proxies in the governance context in proxy vs SDK governance. For observability the tradeoff is related but not identical, so it is worth walking through on its own terms.

The SDK approach sees the most. It runs inside the agent, so it has access to the model’s intermediate reasoning, the agent’s internal state between steps, the variable values, the framework’s own events. If you want the richest possible view of what the agent was thinking, that view exists only inside the process, and only an SDK can reach it. The cost is coverage. An SDK is a per-language, per-framework library. If your agent spawns a sub-agent in another language, or makes a direct HTTP call that bypasses the wrapped client, or a dependency upgrade breaks the wrapper’s instrumentation, those calls produce no telemetry. The SDK sees deeply and it sees only what flows through it.

The proxy approach sees the least per call but misses nothing. It runs outside the agent at the network boundary, so it cannot see the model’s reasoning or the agent’s internal variables. What it can see is every single model call and every single tool call, in every language, from every process, including the ones an SDK would miss, because all of them are network requests and all network requests go through the proxy. The proxy also produces cost and token metrics the agent cannot tamper with, because they are measured at a layer the agent does not control, which is the property that makes proxy-collected cost data trustworthy.

Dimension	SDK observability	Proxy observability
Depth of detail	High: reasoning, internal state	Lower: request and response only
Coverage	Only instrumented code paths	Every network call, every language
Catches direct HTTP and subagents	No	Yes
Survives a dependency upgrade	Sometimes breaks silently	Unaffected, operates at HTTP layer
Cost data integrity	Measured inside the agent	Measured outside the agent, tamper-resistant
Setup	Per language, per framework	One config, language-agnostic
Best for	Deep debugging, eval, reasoning analysis	Cost, audit, complete coverage, enforcement

The honest framing is that this is not a competition, it is a division of labor, and the same one we recommend for governance. The proxy is the layer of record: complete coverage of every call, trustworthy cost and token metrics, an audit-grade log no agent process can edit. The SDK is the depth layer: the reasoning traces and internal state you cannot get from outside. We build a governance proxy, so the proxy half is the half we have a commercial interest in, and we will say plainly that a serious production setup uses both. The proxy gives you the metrics and logs you can stake an audit on. The SDK gives you the deep traces you debug hard failures with. Run the proxy for coverage and integrity, add the SDK where you need depth, and join the two on shared trace IDs.

What are the limits of observability?

Observability is necessary for running agents in production. It is not sufficient, and a post that has spent six thousand words on what to instrument owes you a clear account of what instrumenting does not buy you.

Observability shows what the agent did, not why the model decided it. You can capture every model call, every token, every tool result, and every span, and you still cannot see inside the model’s decision. The trace shows that the model called the search tool with a particular query. It does not show why that query and not another. The reasoning a model exposes, the chain-of-thought text, is itself a generated output, not a faithful transcript of the computation that produced the decision. Observability gives you the actions and the visible reasoning. It does not give you the actual mechanism. Anyone who tells you their observability tool explains model decisions is overselling.

Observability is detection, not prevention. Every signal in this post is recorded after the action happened. The metric fires after the cost was spent. The trace is written after the session ran. The log captures the wrong answer after it was sent to the customer. This is the right design, and we made the same argument about token discrepancy detection: detection systems that try to block in the hot path create false-positive outages and get switched off. But it means observability cannot stop a bad action. Stopping things is the job of a separate layer, a policy engine that enforces in front of the action rather than recording behind it. Observability and enforcement compose. Neither replaces the other.

Observability cannot tell you the answer was wrong. It can tell you the agent returned a 200, took eight tool calls, cost twelve cents, and finished in nine seconds. None of those numbers know whether the answer was correct. Correctness is a semantic property, and reading it requires evaluation: a human reviewing transcripts, an automated grader scoring outputs against a rubric, or a model judging another model’s work. Evaluation consumes the logs and traces that observability produces. It is a distinct discipline built on top, and the failing support agent is the proof: every observability signal was green, and only an evaluation of the actual answers would have caught the forty-percent error rate.

Observability has a cost, and the cost scales with the agent. A high-volume agent emitting full traces and full-content logs for every session produces telemetry that can rival the model spend itself. The answer is sampling and tiering: full fidelity for errored and flagged sessions, sampled fidelity for the healthy majority, short retention for high-volume low-value data and long retention for the audit-grade log. Observability is not free, and pretending it is leads to a telemetry bill that surprises a team the same way a model bill does.

The honest claim is the modest one. Observability is how you find out something went wrong and reconstruct what happened. It is the foundation the other disciplines stand on: evaluation needs the logs, enforcement needs the audit trail, debugging needs the traces. It does not, by itself, make agents correct, and it does not, by itself, make them safe. What it does is end the three-week blind spot. With it, the support agent’s failure is a question you can answer in an afternoon instead of a mystery you learn about from your customers.

FAQ

What is the difference between LLM observability and agent observability?

LLM observability measures individual model calls: tokens, latency, cost, and the prompt and response of one call. Agent observability measures the session, the full multi-step loop of model calls, tool calls, retries, and sub-agent spawns that make up one task. The distinction matters because most agent failures are not visible at the single-call level. A tool that returned the wrong result, an agent that picked the wrong tool, a loop that repeated a call forty times: each of those is a property of the session, not of any one model call. LLM observability is a component of agent observability. It is not a substitute for it. If your monitoring treats each model call as the unit of analysis, it is LLM observability, and it will miss the cross-step failures that are specific to agents.

Should I log full prompts and responses in production?

It depends on the data the agent handles, and the honest answer is often no. Full prompt and response content is the highest-value debugging and evaluation data you can keep, and it is also the content most likely to contain personal data, regulated data, or secrets. The workable posture is tiered: log full content by default in development, log it with redaction applied in production, and do not log it raw at all on paths that handle regulated data such as health or financial records. The OpenTelemetry GenAI conventions make content opt-in for this exact reason. If you cannot redact reliably, log the metadata, the token counts, the tool names, the timing, the trace IDs, and skip the content. Metadata-only logs are still useful. A data-protection incident caused by your own log aggregator is not.

What should I alert on for an AI agent?

Alert on the agent-shaped metrics, not the service-shaped ones. The set that earns an alert: cost per task crossing a budget ceiling, which catches runaway spend; loop or retry rate above roughly one percent of sessions, which catches stuck agents; tool error rate for any single tool above about five percent sustained, which catches a broken or misused tool; and p95 latency above your task SLA, which catches real slowdowns. Do not alert on raw request count or binary uptime; they tell you about traffic and process state, not about whether the agent is doing its job. The test for any alert is whether a human will take an action when it fires. An alert that gets acknowledged and ignored is training your team to ignore the next real one.

Does OpenTelemetry support AI agents yet?

Partially, and the partial support is usable. OpenTelemetry has GenAI semantic conventions covering metrics, traces, and logs for model calls and agent operations, including defined span types like invoke_agent and execute_tool and metrics like gen_ai.client.token.usage. The conventions are in Development status, which means the span shapes are stable enough to build on but specific attribute names will change before the spec stabilizes. Instrumentation libraries gate the newest conventions behind the OTEL_SEMCONV_STABILITY_OPT_IN environment variable so a spec change does not silently break your pipeline. Building on a Development-status spec is a real bet, and it is the right one, because the convention is where the ecosystem is converging and the alternative is a private schema you migrate later anyway.

Can observability tell me if my agent gave a wrong answer?

No. Observability tells you what the agent did: which tools it called, how many tokens it used, how long it took, whether anything threw an error. None of that knows whether the final answer was correct. Correctness is a semantic property, and measuring it requires evaluation, a separate discipline where a human reviews transcripts, an automated grader scores outputs against a rubric, or one model judges another’s work. Evaluation runs on top of the data observability produces; it consumes the logs and traces. The failing support agent in this post is the clearest illustration: every observability metric was green for three weeks while the agent was wrong forty percent of the time, and only an evaluation of the actual answers would have caught it. Instrument for observability, then build evaluation on top of it. They are different jobs.