Scenario

​

overview

​

constructor

Scenario(
    name: str,
    prompt: str,
    evaluations: List[Evaluation] = []
)

​

parameters

• name (str): the name of the scenario
• prompt (str): the system prompt that defines what the agent should do in this scenario
• evaluations (List[Evaluation], optional): list of evaluation criteria to assess the scenario. defaults to an empty list.

​

example usage

scenario = Scenario(
    name="order_donut",
    prompt="order a dozen donuts with sprinkles and a coffee",
    evaluations=[
        Evaluation(name="order_success", prompt="the order was successful"),
        Evaluation(name="price_confirmed", prompt="the agent confirmed the price of the order"),
    ]
)

​

notes

• the scenario prompt should clearly define the task or interaction you want to test
• each scenario can have multiple evaluation criteria to assess different aspects of the interaction
• scenarios are used in combination with agents to create tests