Agents don't fail with clean error messages. They wander, pick the wrong tool, or give a confident answer that is quietly wrong, and ordinary logs rarely explain why. Two practices help you stay in control: evaluation, which measures how good your agent's behavior is, and observability, which lets you see what the agent actually did, step by step.
The first two lessons cover evaluation: how to combine quick qualitative checks, structured human review, benchmarks, code-based evals, and LLM-as-a-judge, and how to evaluate not just the final answer but the tools and decisions along the way. The last lesson is a hands-on look at tracing with LangSmith: inspecting each step of a run and watching latency, success rates, and token costs in a monitoring dashboard.
As you watch, think about how the two fit together. Traces give you the evidence of what happened, and evals turn that evidence into a score you can track as you change prompts, tools, or models. No single method is enough on its own, so the goal is a mix you can trust and afford to run repeatedly.
A practical end-to-end framework for evaluation: how quick qualitative checks, structured human review, and benchmarks work together, and where human judgment matters most.
Video 34minA closer look at the main ways to evaluate an agent (code-based evals, LLM-as-a-judge, and human annotations), with a hands-on demo of tracing and evaluating a travel agent in Phoenix.
Video 7minAdd tracing to an LLM app with LangSmith: configure it with environment variables, inspect each step of a run, and use the monitoring dashboard to track latency, success rates, and token costs.
Video 11min