Skip to content

Testing

The project uses three test layers.

Test taxonomy

  • Unit: isolated logic and utility behavior (tests/unit)
  • Integration: provider/API behavior and combinations (tests/integration)
  • E2E: full workflows through high-level abstractions (tests/e2e)

Test runners (preferred)

Use the provided runner scripts as the primary interface for test execution:

  • tests/unit/run_tests.py
  • tests/integration/run_tests.py
  • tests/e2e/run_tests.py
  • tests/run_all_tests.py

These wrappers already expose file/class/test targeting and reporting options, and are preferred over calling raw pytest selectors directly.

Important execution constraint:

  • Unit tests can be run either via tests/unit/run_tests.py or direct pytest.
  • Integration and E2E suites should be run through their run_tests.py wrappers; direct pytest invocation for those suites is not the supported path for this project.

Run tests by suite

From repository root:

python tests/unit/run_tests.py
python tests/integration/run_tests.py
python tests/e2e/run_tests.py

Target a specific file/class/test

Examples:

python tests/unit/run_tests.py --file test_agent.py
python tests/unit/run_tests.py --file test_agent.py --class TestAgentConfiguration
python tests/unit/run_tests.py --file test_agent.py --test test_basic_config

python tests/integration/run_tests.py --file test_openai_function_calling.py
python tests/integration/run_tests.py --file test_openai_function_calling.py --class TestOpenAIFunctionCallingWithLocalMCP
python tests/integration/run_tests.py --file test_openai_function_calling.py --test test_with_local_mcp_calculator

python tests/e2e/run_tests.py --file test_agent_e2e.py
python tests/e2e/run_tests.py --file test_tool_formats.py --class TestAllToolFormats
python tests/e2e/run_tests.py --file test_tool_formats.py --test test_format_1_plain_function

Useful runner options

python tests/unit/run_tests.py --list
python tests/unit/run_tests.py --verbose
python tests/unit/run_tests.py --html

python tests/integration/run_tests.py --markers "not slow"
python tests/e2e/run_tests.py --markers "e2e"

Run all test suites

Use the master runner to execute unit + integration + e2e suites in one command:

python tests/run_all_tests.py

Common variants:

python tests/run_all_tests.py --unit-only
python tests/run_all_tests.py --integration-only
python tests/run_all_tests.py --e2e-only
python tests/run_all_tests.py --skip-slow
python tests/run_all_tests.py --html
python tests/run_all_tests.py --verbose

Notes

  • Unit tests are expected to run without provider API keys.
  • Integration and e2e tests depend on provider credentials and may incur API cost.
  • Runner scripts generate output artifacts under each suite's output/ directory, and tests/run_all_tests.py writes combined artifacts under tests/combined_output/.
  • Two integration files are opt-in: their candidate rows go through the same MODEL_OVERRIDE_* filter as everything else, so a cost filter that does not name them skips them with a stated reason. test_anthropic_preserved_thinking_live.py runs the preserved-thinking gate on claude-sonnet-5-5, claude-opus-5-5 and/or claude-fable-5-1 (MODEL_OVERRIDE_ANTHROPIC=claude-sonnet-5-5, or several comma-separated), and test_realtime_live.py needs the realtime rows named for it. The repository's test guide has the exact commands.

Building the docs locally

The site is MkDocs with the Material theme. Install the docs dependencies, serve the site locally, and run a strict build (any warning fails it) before publishing:

pip install -r docs/requirements.txt
mkdocs serve
mkdocs build --strict

MkDocs local customisation and deploy flow

Figure: local docs development workflow.

Recipe mapping

Recipes in docs/recipes are sourced from test files and include explicit test references.

Start with: