Comparison Accuracy Evaluation

Exmaple showing how to evaluate agent accuracy on comparison tasks.

1. Create a Python file

   accuracy_comparison.py

   from typing import Optional
   
   from agno.agent import Agent
   from agno.eval.accuracy import AccuracyEval, AccuracyResult
   from agno.models.openai import OpenAIResponses
   from agno.tools.calculator import CalculatorTools
   
   evaluation = AccuracyEval(
       name="Comparison Evaluation",
       model=OpenAIResponses(id="gpt-5.2"),
       agent=Agent(
           model=OpenAIResponses(id="gpt-5.2"),
           tools=[CalculatorTools()],
           instructions="You must use the calculator tools for comparisons.",
       ),
       input="9.11 and 9.9 -- which is bigger?",
       expected_output="9.9",
       additional_guidelines="Its ok for the output to include additional text or information relevant to the comparison.",
   )
   
   result: Optional[AccuracyResult] = evaluation.run(print_results=True)
   assert result is not None and result.avg_score >= 8
   
2. Set up your virtual environment

   Mac

   uv venv --python 3.12
   source .venv/bin/activate
   
3. Install dependencies

   uv pip install -U openai agno
   
4. Export your OpenAI API key

   Mac/Linux

     export OPENAI_API_KEY="your_openai_api_key_here"
   
5. Run Agent

   python accuracy_comparison.py