Testing Workflow

Relevant source files

• .github/actions/rl-scanner/action.yml
• .github/workflows/codeql.yml
• .github/workflows/docs.yml
• .github/workflows/publish.yml
• .github/workflows/rl-scanner.yml
• .github/workflows/snyk.yml
• .github/workflows/test.yml
• README.md
• poetry.lock
• pyproject.toml
• requirements.txt

Purpose and Scope

This document details the automated testing workflow that executes on every pull request to validate code changes before they are merged. The testing workflow runs the full test suite across multiple Python versions in isolated environments, collects coverage data, and reports results back to the pull request.

For information about security scanning that also runs on pull requests, see Security Scanning. For the release-time testing and publication process, see Release Workflow Details.

Workflow Architecture

The testing workflow follows a matrix testing pattern to ensure compatibility across all supported Python versions. While the explicit test workflow file is not provided in the repository files examined, the testing infrastructure can be inferred from the development setup and related workflows.

Sources: .github/workflows/publish.yml70-82 .github/workflows/docs.yml27-38

Workflow Triggers

The testing workflow is triggered by multiple GitHub events to ensure comprehensive validation:

Trigger Event	When It Fires	Purpose
pull_request (opened)	New PR created	Validate new changes
pull_request (synchronize)	New commits pushed to PR	Re-validate after updates
push (master)	Direct push to master	Validate master branch state
merge_group	PR added to merge queue	Final validation before merge

The workflow uses concurrency groups to prevent redundant test runs:

group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}

This pattern cancels in-progress runs when new commits are pushed to a PR, but preserves all runs on the master branch.

Sources: .github/workflows/snyk.yml3-22 .github/workflows/codeql.yml3-22

Test Matrix Configuration

The workflow employs a matrix strategy to test against all supported Python versions simultaneously:

Each matrix job runs independently on a fresh GitHub Actions runner. The matrix configuration follows this structure:

The fail-fast: false setting ensures all Python versions are tested even if one fails, providing comprehensive compatibility information.

Sources: .github/workflows/rl-scanner.yml6-8 .github/workflows/codeql.yml29-32

Environment Setup

Each test job follows a standardized environment setup process using Poetry for dependency management:

The setup steps are:

1. Python Installation: Uses actions/setup-python@v6 to install the matrix-specified Python version
2. Package Manager Setup: Installs pip and pipx for tool isolation
3. Poetry Installation: Installs Poetry via pipx to avoid dependency conflicts
4. Virtual Environment Configuration: Sets virtualenvs.in-project to create .venv directory in the repository
5. Dependency Installation: Runs poetry install --with dev to install both production and development dependencies

This process is consistent across all workflows in the repository.

Sources: .github/workflows/publish.yml70-82 .github/workflows/docs.yml27-38 .github/workflows/rl-scanner.yml39-54

Test Isolation with Bubblewrap

The testing infrastructure uses bubblewrap sandboxing to isolate test execution. Bubblewrap is a lightweight sandboxing tool that creates a restricted environment for running tests, preventing:

• Unintended file system modifications outside the project directory
• Network access to unauthorized endpoints
• Process interference between concurrent test runs

The sandboxing mechanism ensures tests run in a reproducible, isolated environment that mirrors production constraints. This is particularly important for:

• Security testing: Validating that the SDK properly handles sensitive data
• File system operations: Testing code that interacts with the file system without side effects
• Parallel execution: Running multiple test jobs simultaneously without interference

Sources: High-level system diagrams (Diagram 6: Development Workflow)

Test Execution

Tests are executed using pytest, the standard Python testing framework. The test command structure follows this pattern:

This command:

1. Activates the Poetry virtual environment
2. Executes pytest with the project's configuration from pyproject.toml
3. Collects and runs all test files matching the pattern test_*.py or *_test.py
4. Generates coverage data during execution

The pytest configuration (located in pyproject.toml or pytest.ini) typically includes:

• Test discovery patterns
• Coverage plugins and settings
• Test markers for categorization
• Output formatting options
• Parallel execution settings (if enabled)

Sources: .github/workflows/publish.yml82 High-level system diagrams (Diagram 6: Development Workflow)

Coverage Reporting

The workflow collects code coverage data and uploads it to Codecov for tracking test coverage over time:

The coverage workflow:

1. Collection: Pytest with the coverage plugin runs tests and tracks which lines are executed
2. File Generation: Creates a .coverage binary file and optionally a coverage.xml report
3. Upload: The Codecov action uploads coverage data with flags identifying the Python version
4. Merging: Codecov combines coverage from all matrix jobs to produce a comprehensive report
5. Reporting: Posts a comment on the PR showing coverage changes and trends

This provides visibility into:

• Overall coverage percentage
• Coverage delta (change from base branch)
• Uncovered lines and files
• Coverage trends over time

Sources: High-level system diagrams (Diagram 3: Deployment Pipeline, Diagram 6: Development Workflow)

Integration with Pull Request Checks

The testing workflow integrates tightly with GitHub's pull request protection system:

Branch Protection Rules: The repository configures the test workflow as a required status check. Pull requests cannot be merged until:

1. All matrix jobs complete successfully
2. Coverage meets the minimum threshold (if configured)
3. No test failures are reported

Special Cases: The workflow includes logic to handle special scenarios:

• Dependabot PRs: Skips unnecessary test runs with an artificial success status to satisfy branch protection
• Merge Groups: Skips redundant testing when PRs are added to merge queues
• Master Branch: Never cancels in-progress test runs to preserve historical data

This is implemented using conditional checks:

Sources: .github/workflows/snyk.yml31-32 .github/workflows/codeql.yml35-36

Workflow Outputs and Artifacts

The testing workflow produces several outputs that are consumed by downstream processes:

Output	Type	Purpose	Retention
Test Results	GitHub Check	PR status indicator	Permanent
Coverage Report	Artifact	Detailed coverage analysis	90 days
.coverage file	Artifact	Raw coverage data	90 days
Pytest XML	Artifact	Machine-readable test results	90 days
Logs	GitHub Actions Log	Debugging failed tests	90 days

These artifacts enable:

• Local reproduction: Developers can download artifacts to investigate failures
• Historical analysis: Coverage trends can be tracked over time
• CI/CD integration: Downstream workflows can access test results
• Debugging: Detailed logs help diagnose test failures

Sources: High-level system diagrams (Diagram 3: Deployment Pipeline)

Relationship to Other Workflows

The testing workflow is part of a comprehensive CI/CD pipeline:

Workflow Dependencies:

• Tests must pass before security scanning results are meaningful
• All PR checks (tests + security + quality) must pass before merge
• Documentation builds on master after tests have passed
• Release testing includes additional security scanning with RL-Scanner

This architecture ensures quality gates at every stage of the development lifecycle.

Sources: .github/workflows/publish.yml14-30 .github/workflows/docs.yml3-16 .github/workflows/snyk.yml3-14