​
prepare the environment

first, we recommend using venv to setup the Python environment.

python -m venv .venv

and activate it with:

source .venv/bin/activate

​
install the package

pip install fixa-dev

​
set up your environment variables in a .env file

.env
OPENAI_API_KEY=your_openai_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
CARTESIA_API_KEY=your_cartesia_api_key
TWILIO_ACCOUNT_SID=your_twilio_account_sid
TWILIO_AUTH_TOKEN=your_twilio_auth_token
NGROK_AUTH_TOKEN=your_ngrok_auth_token

​
create a test script:

make sure to replace TWILIO_PHONE_NUMBER and PHONE_NUMBER_TO_CALL with your own numbers.

if you have a paid ngrok plan, be sure to add your ngrok domain to the listener variable.

main.py
from fixa import Test, Agent, Scenario, Evaluation, TestRunner
from fixa.evaluators import LocalEvaluator
from dotenv import load_dotenv
import ngrok, os, asyncio

load_dotenv(override=True)

TWILIO_PHONE_NUMBER = "+15554443333" # the twilio phone number to initiate calls from (must be a number you own)
PHONE_NUMBER_TO_CALL = "+15554443333" # the phone number of your agent

async def main():
    # define test agent to call your voice agent
    agent = Agent(
        name="jessica",
        prompt="you are a young woman named jessica who says 'like' a lot",
    )

    # define a scenario to test
    scenario = Scenario(
        name="order_donut",
        prompt="order a dozen donuts with sprinkles and a coffee",
        # define evaluations to evaluate the scenario after it finishes running
        evaluations=[
            Evaluation(name="order_success", prompt="the order was successful"),
            Evaluation(name="price_confirmed", prompt="the agent confirmed the price of the order"),
        ],
    )

    # start an ngrok server so twilio can access your local websocket endpoint
    port = 8765
    # uncomment the line below if you have a paid ngrok plan (need to specify a domain)
    listener = await ngrok.forward(port, authtoken=os.getenv("NGROK_AUTH_TOKEN")) # type: ignore (needed or else python will complain)
    # listener = await ngrok.forward(port, authtoken=os.getenv("NGROK_AUTH_TOKEN"), domain="your_domain.ngrok.dev") # type: ignore (needed or else python will complain)

    # initialize a test runner
    test_runner = TestRunner(
        port=port,
        ngrok_url=listener.url(),
        twilio_phone_number=TWILIO_PHONE_NUMBER,
        evaluator=LocalEvaluator(),
    )

    # add tests to the test runner
    test = Test(scenario=scenario, agent=agent)
    test_runner.add_test(test)

    # run the tests!
    test_results = await test_runner.run_tests(
        phone_number=PHONE_NUMBER_TO_CALL,
        type=TestRunner.OUTBOUND,
    )

if __name__ == "__main__":
    asyncio.run(main())

​
run the test

python main.py

​
view the results

example output in the console

✨ All tests completed!

📊 Test Results:
==================================================

🎯 order_donut (jessica)
🔊 Recording URL: https://api.twilio.com/XXX
-- ✅ order_success: The order was successfully placed and confirmed by the user and the assistant.
-- ❌ price_confirmed: The price of the order was not mentioned or confirmed during the conversation.

==================================================

for more details, check out the repo, and the examples.