Agent Observability

Observability is the ability to infer the internal state of a system from external signals (logs, metrics, traces). A trace is a single execution of a task; each LLM call, tool call, search is a span with input, output, time, tokens, errors; parent-child relationships form a tree. Standards: OpenTelemetry + OpenInference. The most valuable application is feeding data back into evaluation assets: failed production cases → de-personalization → new evaluation dataset examples and regression tests. Observability is about “seeing”, evaluation is about cementing that into verifiable standards.

Related: [From Benchmark to Improvement], [Three Levels of Evaluation System]