Test an OpenAI Agents SDK Agent

Earlier tutorials tested a plain function and a single LLM call. Real agents add a moving part: they decide when to call a tool. A correct-looking answer that was hallucinated instead of retrieved is still a bug, so an agent test has to cover both the output and the behaviour that produced it.

What you’ll build

A support agent built with the OpenAI Agents SDK that looks up order status through a tool, plus a scenario that:

Checks the answer mentions the order the user asked about
Checks the answer is semantically close to the expected status
Asserts the agent actually called get_order_status instead of inventing it

Prerequisites

Completed Your First LLM Call
Azure OpenAI credentials in AZURE_AI_API_KEY and AZURE_AI_ENDPOINT
pip install openai-agents

1. Configure the judge generator

SemanticSimilarity and any LLM-based check need a configured model. This is separate from the model the agent uses — the system under test and the evaluator are deliberately independent.

from giskard.checks import set_default_generator
from giskard.agents.generators import Generator
from giskard.agents.embeddings import EmbeddingModel

set_default_generator(Generator(model="azure_ai/gpt-4.1-nano"))

# Embedding-based checks need their own model; use the Azure deployment.
embedding_model = EmbeddingModel(model="azure_ai/text-embedding-3-small")

2. Build the agent

One tool, one instruction. The tool appends to TOOL_CALLS so the test can later assert it ran — a two-line spy that needs no patching and keeps the real tool behaviour intact.

from agents import (
    Agent,
    Runner,
    function_tool,
    set_default_openai_api,
    set_default_openai_client,
    set_tracing_disabled,
)
from openai import AsyncAzureOpenAI

# Point the Agents SDK at Azure OpenAI instead of the default OpenAI client.
set_default_openai_client(
    AsyncAzureOpenAI(
        api_key=os.environ["AZURE_AI_API_KEY"],
        azure_endpoint=os.environ["AZURE_AI_ENDPOINT"],
        api_version="2024-10-21",
    )
)
set_default_openai_api("chat_completions")
set_tracing_disabled(True)

TOOL_CALLS: list[tuple[str, str]] = []


@function_tool
def get_order_status(order_id: str) -> str:
    """Return the shipping status for an order id."""
    TOOL_CALLS.append(("get_order_status", order_id))
    return f"Order {order_id} shipped on 2024-05-01, arriving in 2 days."


support_agent = Agent(
    name="Support",
    instructions=(
        "You are a customer support agent. "
        "Always use the get_order_status tool to answer order questions. "
        "Never guess a status."
    ),
    tools=[get_order_status],
    model="gpt-4.1-nano",
)

3. Wrap the agent as the system under test

The callable you hand to .interact() must accept a parameter named inputs (or trace) — those are the names Giskard Checks injects. An async def callable is awaited for you, which is what you want with Runner.run: Runner.run_sync would fail inside the already-running event loop of a notebook or a test.

Clearing TOOL_CALLS at the start of each run keeps assertions scoped to the current interaction.

async def run_support_agent(inputs: str) -> str:
    TOOL_CALLS.clear()
    result = await Runner.run(support_agent, inputs)
    return result.final_output

4. Write the scenario

Three checks at three levels of strictness:

StringMatching — cheap, deterministic: the order id must be echoed back.
SemanticSimilarity — tolerant of phrasing: the answer must mean roughly what the tool returned.
FnCheck — the behavioural assertion: the tool was actually called.

The FnCheck is the one that catches a hallucinating agent. Without it, an agent that invents a plausible shipping date can still pass every text check.

from giskard.checks import FnCheck, Scenario, SemanticSimilarity, StringMatching

scenario = (
    Scenario("order_status_lookup")
    .interact(
        inputs="Where is my order A123?",
        outputs=run_support_agent,
    )
    .check(
        StringMatching(
            name="mentions_order_id",
            keyword="A123",
            text_key="trace.last.outputs",
        )
    )
    .check(
        SemanticSimilarity(
            embedding_model=embedding_model,
            name="matches_tool_result",
            reference_text="Order A123 shipped on 2024-05-01 and arrives in 2 days.",
            threshold=0.6,
        )
    )
    .check(
        FnCheck(
            name="called_get_order_status",
            fn=lambda trace: any(call[0] == "get_order_status" for call in TOOL_CALLS),
        )
    )
)

5. Run it

result = await scenario.run()
result.print_report()

Output

──────────────────────────────────────────────────── ✅ PASSED ────────────────────────────────────────────────────
mentions_order_id       PASS    
matches_tool_result     PASS    
called_get_order_status PASS    
────────────────────────────────────────────────────── Trace ──────────────────────────────────────────────────────
────────────────────────────────────────────────── Interaction 1 ──────────────────────────────────────────────────
Inputs: 'Where is my order A123?'
Outputs: 'Your order A123 shipped on May 1, 2024, and is expected to arrive in 2 days.'
────────────────────────────────────────── 1 step in 3358ms | runs: 1/1 ───────────────────────────────────────────

6. Assert the tool arguments too

Knowing that the tool ran is often not enough — you also want the agent to have passed the right argument. TOOL_CALLS records both, so a second FnCheck covers it.

arg_scenario = (
    Scenario("order_status_arguments")
    .interact(
        inputs="Can you check order B777 for me?",
        outputs=run_support_agent,
    )
    .check(
        FnCheck(
            name="called_with_b777",
            fn=lambda trace: ("get_order_status", "B777") in TOOL_CALLS,
        )
    )
)

arg_result = await arg_scenario.run()
arg_result.print_report()