Agentic observability: what leaders should require before agents reach production

1Agents are distributed systems that make decisions

A conventional service does what its code says. An agent decides what to do next. A single customer request might cause an orchestrator to start two agents; one of them plans, calls a model, calls a policy-lookup tool through an MCP server, calls the model again, and hands off. The other retries a slow database call before it answers. None of that sequence existed before the request arrived, and the next request will produce a different one.

That has three consequences for anyone accountable for the program:

• Cost is emergent. It depends on how many model calls and tool calls the agent chose to make, not on a fixed per-request price.
• Failure is partial. An agent can return a confident answer after a tool call silently failed. Nothing errors and nothing alerts, yet the output is wrong.
• Accountability needs a record. When an agent takes an action on a customer's behalf, someone will ask what it saw, what it decided and which systems it touched.

The answer to all three is the same engineering artifact: a trace. A trace records the run as a tree of timed operations, which are called spans, each with a name and attributes. Distributed tracing is mature for ordinary services. What was missing for agents was an agreed way to name the spans and attributes, so that traces from different frameworks, vendors and teams mean the same thing.

2The shared vocabulary now exists

The OpenTelemetry project maintains semantic conventions for generative AI [OTEL-GENAI]. They define, for any GenAI system:

• Spans for model operations (chat, text completion, embeddings, retrieval, memory) and for agent operations: creating an agent, invoking an agent, invoking a workflow, planning, and executing a tool [SPANS] [AGENT].
• Metrics for operation duration, agent and workflow duration, tool duration, the number of model and tool calls per agent invocation, and token usage including cached and reasoning tokens [METRICS] [TOKENS].
• Events for evaluation results and for exceptions [EVENTS] [EXC].
• Protocol coverage for the Model Context Protocol, including how trace context crosses from an MCP client into an MCP server [MCP].
• Provider mappings for Anthropic, AWS Bedrock, Azure AI Inference and OpenAI.

The strategic value is not any single attribute. It is that a framework, a model vendor, an MCP server and your own platform can all emit telemetry that lands in the same backend and joins into one tree. That is what turns observability from a per-team project into a program-wide capability, and it is what keeps you from being locked in to one vendor's console.

Every document in the set is marked Development. Names and requirement levels can still change. Section 6 covers how to adopt now without being caught out later.

3What one agent run looks like

The exhibit below is a single run of a hypothetical claims-triage workflow, drawn the way a tracing tool would show it. Each row is a span. Indentation is the parent–child relationship and the bar is when the span ran.

span name 0 s4.2 s8.4 s duration

1. invoke_workflow claims-triage8.40 s
2. invoke_agent intake-agent2.80 s
3. plan intake-agent0.73 s
4. chat example-model-large0.69 s
5. execute_tool lookup_policy0.60 s
6. chat example-model-large1.25 s
7. invoke_agent fraud-check-agent5.30 s
8. chat example-model-small0.65 s
9. execute_tool query_ledgererror.type = timeout3.00 s
10. execute_tool query_ledger0.60 s
11. chat example-model-small0.80 s

workflow agent / plan model inference tool execution error

Read top to bottom, the trace tells the story a leader needs. One workflow ran two agents. The intake agent planned and then looked up a policy. The fraud check spent more than a third of the whole run waiting on a ledger query that timed out and was retried. Every model call carries the tokens it consumed, so the cost of this run can be added up span by span and charged to the workflow and the agents responsible.

A few conventions make this tree trustworthy across teams:

• Remote and in-process agents are distinguished. An agent invoked over a remote service (the conventions cite OpenAI Assistants and AWS Bedrock Agents) is a CLIENT span. An agent running inside your process (LangChain or CrewAI, for example) is an INTERNAL span [AGENT].
• Workflows are reported only at real entry points. invoke_workflow is for user-facing orchestrations that coordinate several agents or GenAI calls, not for a runner that a framework starts behind the scenes [AGENT].
• Planning is reported only when it can be identified reliably. A plan span marks the phase where an agent decides its strategy. The model call that produces the plan is its child, and the work that follows sits beside it under the agent [AGENT].
• MCP calls join the same trace. Instrumentations propagate trace context inside the MCP request's params._meta, so a tool server owned by another team appears in your tree. Where an outer GenAI instrumentation already traces the tool call, MCP adds its attributes to that span instead of creating a duplicate [MCP].

4From executive questions to signals

Executives do not need to learn attribute names, but they should know that each of their recurring questions has a specific, standard answer. If your platform team cannot produce these, the gap is in instrumentation, not in reporting.

The question	The standard signal
What does a completed task cost, and which agent spent it?	Token counts on each inference span, including cached and reasoning tokens, rolled up through the trace to the agent and workflow. gen_ai.usage.input_tokensgen_ai.usage.output_tokensgen_ai.client.inference.usage.reasoning.output_tokens
Is an agent doing more work than it should?	Distributions of model calls and tool calls per agent invocation. A long tail usually means loops, retries or a bad prompt. gen_ai.invoke_agent.inference_callsgen_ai.invoke_agent.tool_calls
Where does the time go?	Durations at every level, from the whole workflow down to time to first token from the model server. gen_ai.invoke_workflow.durationgen_ai.invoke_agent.durationgen_ai.execute_tool.durationgen_ai.server.time_to_first_token
Was the answer any good?	An evaluation event attached to the span it judges, with a named metric, a score and an explanation. gen_ai.evaluation.resultgen_ai.evaluation.score.valuegen_ai.evaluation.score.label
What failed, and did the user notice?	Span status and a low-cardinality error type on the failing span, plus an exception event for client-side failures such as rate limits and timeouts. error.typegen_ai.client.operation.exception
Which agent, which version, which conversation, which model?	Stable identity on every agent span, the provider, and the model requested versus the model that actually answered. gen_ai.agent.namegen_ai.agent.versiongen_ai.conversation.idgen_ai.provider.namegen_ai.response.model
What did it touch?	A span for every tool execution, including tools reached through MCP and agent skills, with the tool's name and type. gen_ai.tool.namegen_ai.tool.typegen_ai.skill.name

5Content capture is a governance decision

The most consequential setting in agent telemetry is whether to record content: the system instructions, the messages sent to the model, the model's output, tool definitions, tool-call arguments and results, retrieval queries and memory records. The conventions treat all of these as Opt-In attributes, and they warn that message content is likely to include sensitive information, including personal data [SPANS].

Engineers will reasonably want content turned on, because it makes debugging far easier. Legal, privacy and security functions will reasonably want it off. The conventions give you both paths. You can keep content out of telemetry, capture it under explicit configuration, or send it to separate storage through an upload hook that is independent of the capture flags [SPANS]. Which path each environment takes is a policy choice with an owner. It should not be a default someone accepted in a pull request.

6Requirements for leadership

These are written so that they can be copied into a platform standard, a program charter or a vendor contract. Each one links to the section above that motivates it and to the source in the conventions.

The key words MUST, MUST NOT and SHOULD are to be read as described in RFC 2119 and RFC 8174 when, and only when, they appear in capitals.

1. R1

   AI programs MUST emit agent and model telemetry using the OpenTelemetry GenAI semantic conventions. Contracts for agent platforms, frameworks and model gateways MUST require it, and MUST NOT accept telemetry available only in a vendor's own console.

   §2 · [OTEL-GENAI]

2. R2

   Every user-facing agent entry point MUST produce a single connected trace from workflow to individual tool call, including across MCP servers operated by other teams.

   §3 · [AGENT] · [MCP]

3. R3

   Every agent in production MUST report a stable name and version, so that cost, quality and incidents can be attributed to a specific agent and a specific release.

   §4 · [AGENT]

4. R4

   Cost reporting MUST be derived from token usage on inference spans, including cached and reasoning tokens, and MUST be attributable per agent and per workflow rather than only per API key or account.

   §4 · [TOKENS]

5. R5

   Programs SHOULD alert on the distribution of model calls and tool calls per agent invocation, not only on latency and error rate, so that runaway loops are detected before they appear on an invoice.

   §4 · [METRICS]

6. R6

   Quality SHOULD be measured with evaluation events attached to the spans they judge, using named evaluation metrics that are reviewed with the business owner of the workflow.

   §4 · [EVENTS]

7. R7

   Capture of prompts, outputs, system instructions and tool arguments MUST be disabled unless a written policy enables it for a named environment, with an accountable owner, a retention period and access controls.

   §5 · [SPANS]

8. R8

   Because the conventions are in Development status, programs MUST record which version of the conventions each service emits and SHOULD plan upgrades as normal platform work, so that dashboards and cost reports do not break when names change.

   §2 · [OTEL-GENAI]

7A first ninety days

1. Days 1–30: one workflow, end to end. Choose the agent workflow closest to production that has a real business owner. Instrument it with the conventions, keep content capture off, and confirm that a single request produces one connected trace that includes the tool calls (R2, R3, R7).

2. Days 31–60: the four numbers. Publish cost per completed task, the distribution of model and tool calls per invocation, tool error rate, and end-to-end duration for that workflow. Add one evaluation metric agreed with the business owner (R4, R5, R6).

3. Days 61–90: make it the standard. Write R1 to R8 into the platform standard and procurement templates, pin the convention version, and review a handful of real traces with the leadership team. Looking at traces together builds a shared, concrete sense of how the agents actually behave faster than any summary dashboard.

8What is still open

Being specific about maturity is part of the guidance:

• All of the GenAI conventions are in Development status, so names, requirement levels and span structure can change.
• Events, which carry evaluation results and exceptions, are not yet available in every OpenTelemetry language SDK [EVENTS].
• Structured content attributes may need to be serialized to JSON on spans in some languages until complex attribute values are supported everywhere [SPANS].
• Coverage depends on each framework's instrumentation. For example, planning spans appear only where an instrumentation can tell planning apart from an ordinary model call [AGENT].

None of this is a reason to wait. The cost of instrumenting against a moving standard is a planned upgrade. The cost of running agents without a shared trace is finding out what they did only after a customer, an auditor or an invoice tells you.

9References

• [OTEL-GENAI]OpenTelemetry. Semantic conventions for generative AI systems. Status: Development.
• [AGENT]OpenTelemetry. Semantic conventions for GenAI agent and framework spans.
• [SPANS]OpenTelemetry. Semantic conventions for GenAI model spans, including “Recording content on attributes”.
• [METRICS]OpenTelemetry. Semantic conventions for GenAI metrics.
• [TOKENS]OpenTelemetry. Semantic conventions for GenAI inference token metrics.
• [EVENTS]OpenTelemetry. Semantic conventions for GenAI events.
• [EXC]OpenTelemetry. Semantic conventions for GenAI exceptions.
• [MCP]OpenTelemetry. Semantic conventions for Model Context Protocol.
• [RFC2119]Bradner, S. Key words for use in RFCs to Indicate Requirement Levels, and RFC 8174.