This document describes the testing infrastructure for the StackOne AI SDK, including the pytest-based test framework, snapshot testing for tool definitions, example validation, and coverage reporting. For information about the CI/CD pipeline that executes these tests, see CI/CD Pipeline. For code quality standards including pre-commit hooks, see Code Quality Standards.
The StackOne AI SDK uses pytest as its primary testing framework, with specialized testing strategies for different components of the system. The test suite is organized into unit tests for tool definitions, integration tests for API execution, and validation tests for examples.
Test Execution Flow and Directory Structure
Sources: justfile1-44 CLAUDE.md79-83 .github/workflows/ci.yaml1-101
The SDK provides multiple test execution commands through the justfile, each targeting different test scopes and output formats.
| Command | Purpose | Pytest Arguments | Output |
|---|---|---|---|
just test | Run all tests | pytest | Terminal only |
just coverage | Run tests with coverage | pytest --cov --cov-report=term --cov-report=json --cov-report=html | Terminal + JSON + HTML |
just test-tools | Run tool-specific tests | pytest tests | Terminal only |
just test-examples | Run example validation | pytest examples | Terminal only |
All test commands use uv run to execute pytest within the isolated environment managed by uv, ensuring consistent dependency versions across local development and CI environments.
Sources: justfile13-27
The CI pipeline executes the test suite across a matrix of Python versions to ensure compatibility with all supported Python releases.
CI Test Matrix Execution
The test matrix runs on every push and pull request, executing the full test suite (lint → type check → tests) for each Python version. This ensures that the SDK maintains compatibility across its supported Python version range (≥3.10).
Sources: .github/workflows/ci.yaml36-60
Snapshot testing is used to validate the OpenAPI-to-tool conversion process. When tools are generated from OpenAPI specifications, their resulting definitions are compared against stored snapshots to detect unintended changes.
Snapshot Testing Workflow for Tool Definitions
When OpenAPI specifications change, the snapshot tests will fail, alerting developers to review the changes. If the changes are intentional, developers can update the snapshots using pytest's snapshot update flag. This provides a safety mechanism to detect unintended changes in tool definitions that could break existing integrations.
Sources: CLAUDE.md79-83
The examples/ directory contains executable example scripts that demonstrate SDK usage patterns. These examples are also run as tests to ensure they remain valid as the SDK evolves.
Example Validation Architecture
The just test-examples command executes all scripts in the examples/ directory as pytest test cases. This ensures that documentation examples remain accurate and functional as the codebase evolves. The .claude/rules/examples-standards rule file defines the requirements for example organization and validation.
Sources: justfile26-27 CLAUDE.md79-83 CLAUDE.md28
The SDK includes asynchronous tool execution capabilities, which are tested using pytest-asyncio. This plugin allows pytest to discover and execute async test functions.
| Test Type | Decorator | Purpose |
|---|---|---|
| Sync Tests | (none) | Standard synchronous tool operations |
| Async Tests | @pytest.mark.asyncio | Asynchronous tool execution and API calls |
The pytest configuration in pyproject.toml includes pytest-asyncio as a test dependency, enabling async test discovery and execution without additional configuration.
Sources: CLAUDE.md82
The SDK provides comprehensive code coverage reporting with multiple output formats for different use cases.
Coverage Reporting Pipeline
Coverage is calculated using pytest --cov, which instruments the code during test execution. The JSON output is processed by the jaywcjlove/coverage-badges-cli action to generate a coverage badge SVG. Both the badge and the full HTML coverage report are deployed to GitHub Pages, making coverage metrics publicly accessible.
| Output Format | Use Case | Location |
|---|---|---|
| Terminal | Local development feedback | stdout |
| JSON | CI integration, badge generation | coverage/coverage.json |
| HTML | Detailed coverage inspection | coverage/htmlcov/index.html |
| Badge SVG | README and documentation | coverage/badges.svg |
Sources: justfile18-19 .github/workflows/ci.yaml61-89
The pytest configuration is embedded in the project's pyproject.toml file, defining test discovery patterns, output options, and plugin configurations.
Pytest Configuration Structure
The pyproject.toml configuration defines test paths to include both the tests/ and examples/ directories, ensuring that example scripts are validated alongside unit tests. Coverage configuration specifies source paths for instrumentation and exclusion patterns for generated or vendored code.
Sources: CLAUDE.md79-83 justfile14-27
To execute tests in a local development environment:
just install to sync Python dependencies via uvjust test for the full test suitejust coverage to generate coverage reportsjust test-tools or just test-examples for targeted testingThe local test execution uses the same uv run pytest commands as CI, ensuring consistency between local development and automated testing environments. All test commands respect the pytest configuration in pyproject.toml, including test discovery paths, plugin settings, and output formatting.