overview
the TestRunner
class is responsible for executing tests by managing the test server, making calls, and collecting results. it coordinates the entire testing process.
constructor
TestRunner(
port: int,
ngrok_url: str,
twilio_phone_number: str,
evaluator: BaseEvaluator | None = None
)
parameters
port
(int): the port to run the test server on
ngrok_url
(str): the ngrok url for external access to the test server
twilio_phone_number
(str): the phone number to use as the “from” number for outbound test calls
evaluator
(BaseEvaluator | None, optional): the evaluator to use for assessing test results. defaults to none.
constants
INBOUND
: Constant for inbound call type
OUTBOUND
: Constant for outbound call type
methods
add_test()
adds a test to the test runner.
def add_test(self, test: Test)
run_tests()
runs all added tests and returns the results.
async def run_tests(
phone_number: str,
type: str = OUTBOUND
) -> List[TestResult]
parameters
phone_number
(str): the phone number to call or receive calls from
type
(str, optional): the type of calls to make (INBOUND or OUTBOUND). defaults to OUTBOUND.
returns
- List[TestResult]: the results of all executed tests
example usage
# initialize test runner
test_runner = TestRunner(
port=8765,
ngrok_url=listener.url(),
twilio_phone_number=TWILIO_PHONE_NUMBER,
evaluator=LocalEvaluator()
)
# add tests
test_runner.add_test(test1)
test_runner.add_test(test2)
# run tests
test_results = await test_runner.run_tests(
phone_number=PHONE_NUMBER_TO_CALL,
type=TestRunner.OUTBOUND
)
features
- manages a fastapi server for handling calls
- coordinates with twilio for making/receiving calls
- executes multiple tests in parallel
- collects and evaluates test results
- supports both inbound and outbound testing
notes
- requires environment variables for various api keys (twilio, openai, etc.)
- uses ngrok for exposing the local server to the internet
- can run multiple tests simultaneously
- supports different types of evaluators (local/cloud)
- handles test execution, result collection, and cleanup
Responses are generated using AI and may contain mistakes.