Evaluating Agentic Systems

Your agent can return a plausible answer and still take the wrong path to get there. It might call the wrong tool, call the right tool three times, ignore a tool error, or invent data after a failed lookup. The final response reads fine. The trajectory is a mess.

In this lesson we put a stock research agent under evaluation. We score both the final answer and the trajectory — the sequence of tool calls that produced it.

A tool-calling agent has failure modes a plain RAG pipeline does not:

• It picks the wrong tool (search news when asked for a P/E ratio)
• It picks the right tool but calls it redundantly (look up the same ticker three times)
• It returns a fluent answer that ignores what the tools actually returned

The same mlflow.genai.evaluate() machinery from the RAG evaluation tutorial accepts agent-specific scorers. We mix two trajectory scorers with two answer-quality scorers:

• ToolCallCorrectness¹ - Did the agent call the tools the task required, with the right arguments?
• ToolCallEfficiency² - Did it call them without redundancy or unnecessary detours?
• RelevanceToQuery³ - Does the final answer address the user's question?
• Completeness⁴ - Does the answer cover all the facts the question asks for?

All four are LLM-as-judge scorers. Each returns a pass/fail verdict and a rationale. The split is the whole point:

Use trajectory scores to debug tool use. Use answer scores to debug final communication. A production agent needs both layers.