FlyPython
Sign inStart a challenge

Resource playbooks

Leave with a concrete artifact, not another reading list.

Start with the playbook that matches your current problem. Each one names the steps, expected output, and first-party guide; maintained official references follow as supporting material.

Start here

Twelve first-party playbooks

Pick one path, complete the checklist, and keep the named output as evidence of what you decided or verified.

01
Python foundations

Start a Python project you can verify

Problem

A script works once, but its environment and failure behavior are not reproducible.

Full playbook — best for, checklist, verification, sources

Best for

A first project, a small script that needs hardening, or a clean-room verification pass.

Not for

A full Python syntax course or a separate beginner project template; this path deliberately reuses the published agent-loop example.

FlyPython decision and trade-off

Reuse one shared, tested artifact before inventing a new sample. You gain a real failure suite, but the example uses agent terminology that beginners may need to look up.

Checklist

  1. Create an empty project directory and a virtual environment with Python 3.12 or newer.
  2. Record the supported Python version and the command that runs the checks.
  3. Download the published agent-loop source and its 13-test suite from the guide.
  4. Run python3 -m unittest -v from the example directory and keep the pass or failure output.
  5. Write one input rule, one output rule, and one invalid-input test for your next function.

Supported environment

Python 3.12 or newer with the standard library. The shared suite was last run with Python 3.14.7 on macOS; after the two example files are downloaded, the test run needs no network or package install.

Verification

Code checkRun the shared 13-test agent-loop suite. This is the same suite used by the AI agents playbook, not an independent foundation example.

python3 -B -m unittest -v

Last reviewed · verification recorded

Next stepOpen the guide, create the environment, run the shared suite, then apply the same contract pattern to one function of your own.

02
Web & APIs

Write an API boundary before choosing a framework

Problem

A Python function needs an HTTP interface, but its validation, failures, and dependency limits are still implicit.

Full playbook — best for, checklist, verification, sources

Best for

One new endpoint, a service rewrite, or a pre-implementation review of an API change.

Not for

A finished server implementation or deployment tutorial. This playbook produces a contract and test matrix, not running API code.

FlyPython decision and trade-off

Define the HTTP contract before selecting a framework. That delays the first route handler, but keeps status semantics, dependency limits, and release evidence independent of one library.

Checklist

  1. Name one method and route, then write the smallest valid request and successful response.
  2. List required fields, types, allowed values, size limits, and the status for each rejected request.
  3. Set connection and operation deadlines for every database, HTTP, model, or queue dependency.
  4. Write a test matrix for valid input, every validation boundary, dependency timeout, malformed dependency data, and an idempotent repeat.
  5. Define release evidence: hostname, version marker, healthy response, controlled error, security headers, and one matching request ID in logs.

Supported environment

A framework-neutral design exercise for JSON-over-HTTP services. Basic Python functions and HTTP concepts are assumed; no server runtime is included.

Verification

Documentation reviewCheck status semantics against HTTP Semantics RFC 9110 and request validation against the FastAPI request-body documentation.

Last reviewed · verification recorded

Next stepOpen the guide and fill the endpoint contract for one route before choosing a framework or writing a handler.

03
Automation

Design a retryable automation run

Problem

A repeated task can restart or partially fail, but there is no durable definition of what completed.

Full playbook — best for, checklist, verification, sources

Best for

File processing, browser workflows, data pipelines, scheduled jobs, and API-driven operations.

Not for

A vendor-specific integration or ready-to-run scheduler. This playbook designs recovery behavior; it does not include an automation implementation.

FlyPython decision and trade-off

Specify durability before optimizing throughput. The extra state and verification work costs time up front, but makes a retry distinguishable from a duplicate side effect.

Checklist

  1. Assign a stable run ID and record immutable inputs, the final output location, timestamps, and named states.
  2. Separate discovery, decision, and mutation so the intended changes can be inspected before execution.
  3. Choose an atomic-write or idempotency strategy, and record the last successful checkpoint.
  4. Classify retryable failures, then set attempt, operation, and total time limits.
  5. Define a business-level output check and test first run, exact rerun, interrupted resume, transient recovery, and permanent failure.

Supported environment

A framework-neutral design exercise for Python 3.12+ workflows. No scheduler, browser session, external provider, or runnable workflow is included.

Verification

Documentation reviewReview the file and operational guidance against Python pathlib and tempfile documentation, then apply it to one named workflow.

Last reviewed · verification recorded

Next stepOpen the guide and write the run contract for one repeated task; do not automate it until completion and recovery rules are explicit.

04
AI agents

Run an agent loop before adding a model

Problem

A task may need model judgment, but the action contract and failure boundaries have not been tested independently.

Full playbook — best for, checklist, verification, sources

Best for

Developers evaluating whether an agent belongs in a workflow or where a provider adapter should sit.

Not for

A production agent framework, model-quality evaluation, or approval example for mutating tools. The published planner is deterministic and its tools are read-only.

FlyPython decision and trade-off

Test a deterministic planner before adding a provider. This isolates the runtime contract and failure boundaries, but it does not measure model judgment, latency, or cost.

Checklist

  1. Download the published deterministic agent-loop source and tests; no API key is required.
  2. Run agent_loop.py and inspect the ordered action-name trace ending in finish.
  3. Run python3 -m unittest -v and stop if any of the 13 tests fail.
  4. Identify which decision genuinely needs model judgment while keeping validation, tools, deadlines, and completion rules in code.
  5. Add one narrow read-only tool and its failure tests before connecting a provider.

Supported environment

Python 3.12 or newer with the standard library. The suite was last run with Python 3.14.7 on macOS; after the two example files are downloaded, the test run needs no model account, API key, network, or package install.

Verification

Code checkRun the 13 published tests covering expected classifications, action validation, tool validation, step limits, and time limits.

python3 -B -m unittest -v

Last reviewed · verification recorded

Next stepOpen the guide, run the example and all 13 tests, then decide whether a provider belongs only at the planner decision boundary.

05
Python foundations

Fix a Python bug with a regression test

Problem

A bug report describes pain, but the behavior, cause, and proof of the fix are not recorded anywhere reproducible.

Full playbook — best for, checklist, verification, sources

Best for

Any reported defect you intend to fix today, especially one you will hand to a coding agent.

Not for

Feature work, refactors, or incidents that need hotfix coordination rather than one bounded fix.

FlyPython decision and trade-off

Prove the bug with a failing test before touching code. It feels slower, but without it neither you nor the agent can show the regression stays fixed.

Checklist

  1. Record the observed behavior, expected behavior, smallest failing input, and affected user path in a written task contract.
  2. Add one test that fails for the reported reason; if it does not fail before the change, it is not yet regression evidence.
  3. Ask the coding agent to inspect the call path and propose the smallest plausible cause; do not authorize an unrelated refactor.
  4. Change only the behavior the contract requires and preserve public error forms unless the contract changes them.
  5. Run the new test, the nearest suite, then the full deterministic suite; review the diff for widened scope and new side effects.
  6. Re-run the original user path and record the commands and results.

Supported environment

Any Python project with a runnable test command; the companion product-slug example practices the loop with the standard library alone.

Verification

Code checkRun the product-slug example verifier that practices this loop: a failing starter, a bounded fix, and four tests.

python examples/product-slug/verify.py solution

Last reviewed · verification recorded

Official sources

Next stepPractice the loop on the product-slug example, then apply it to one real defect with its own regression test.

06
Python foundations

Upgrade Python dependencies in a bounded change

Problem

Dependency upgrades drift into big-bang events: unrelated packages move together and nobody can say what one upgrade changed.

Full playbook — best for, checklist, verification, sources

Best for

Planned upgrades of one package or a small set, and security-driven bumps with a deadline.

Not for

Untangling years of accumulated drift or migrating Python major versions in the same change.

FlyPython decision and trade-off

Upgrade in bounded changes with lockfile evidence. Slower per week, but every change keeps a reason, a diff, and a rollback.

Checklist

  1. Define the package range, the reason, supported Python versions, and the rollback before touching the lockfile.
  2. Read upstream release notes and security advisories; identify breaking or deprecated behavior first.
  3. Update only the intended direct dependencies and review every transitive change.
  4. Run formatting, types, tests, and build, then a real runtime smoke check.
  5. Review dependency provenance, install scripts, licenses, artifact size, and newly requested permissions.
  6. Record the resolved versions and user-visible impact; keep unrelated upgrades out of the same change.

Supported environment

Any dependency-locked Python project; uv and pip-tools workflows both fit.

Verification

Documentation reviewReview the upgrade workflow against the uv documentation and the Python packaging user guide.

Last reviewed · verification recorded

Next stepPick one outdated direct dependency, run the checklist end to end, and record the evidence before starting the next.

07
Web & APIs

Add or change a Python API by contract first

Problem

A new endpoint gets implemented before anyone writes down its contract, so validation, errors, and compatibility rules stay implicit.

Full playbook — best for, checklist, verification, sources

Best for

One new route or a change to an existing public API, before implementation starts.

Not for

Internal refactors with no observable contract change, or API programs spanning many services.

FlyPython decision and trade-off

Write the contract first, including failures and compatibility. It delays the first handler but keeps status semantics and migration paths deliberate.

Checklist

  1. Write request, response, status, authentication, idempotency, and error contracts before selecting implementation details.
  2. Separate domain logic from transport code and validate data at the boundary.
  3. Add contract tests for one success, each meaningful failure, permissions, and malformed input; test generated schemas when clients depend on them.
  4. Decide whether the change is additive, deprecating, or breaking, and document a migration path for every breaking change.
  5. Apply explicit timeouts and cancellation to downstream calls; never expose provider errors or secrets to clients.
  6. Verify the API through the same network boundary a real client uses, then inspect logs and state changes.

Supported environment

Framework-neutral design and test steps for JSON-over-HTTP services; FastAPI, Django, and plain ASGI apps all fit.

Verification

Documentation reviewCheck status and error semantics against HTTP Semantics RFC 9110 and request validation against the FastAPI request-body documentation.

Last reviewed · verification recorded

Next stepFill the contract for one route, then implement the handler against its contract tests.

08
Web & APIs

Integrate an external API behind a typed boundary

Problem

Third-party API calls leak through the codebase: credentials near source, no timeouts, blind retries, and tests that only pass with live traffic.

Full playbook — best for, checklist, verification, sources

Best for

Adding the first client for a provider, or consolidating scattered calls behind one typed interface.

Not for

Providers with no testable contract or cost model, and one-off scripts where the boundary outweighs reuse.

FlyPython decision and trade-off

Wrap the provider behind a small typed interface with deterministic fakes. The interface costs time up front, but keeps credentials, retries, and failure modes in one auditable place.

Checklist

  1. Document the provider, endpoint, data sent, credential owner, cost limit, rate limit, and allowed side effects.
  2. Put provider code behind a small typed interface and validate every response; model output is untrusted data too.
  3. Keep credentials outside source control and redact them from logs and errors.
  4. Set connect and response timeouts; retry only transient, idempotent work with bounded backoff and never retry an irreversible action blindly.
  5. Test with a deterministic fake transport covering timeout, malformed data, rate limits, and partial failure; keep live tests opt-in.
  6. Verify cost, latency, logs, and the user-visible fallback in a controlled environment before production traffic.

Supported environment

Framework-neutral steps for Python 3.12+ HTTP clients; HTTPX and requests both fit the interface pattern.

Verification

Documentation reviewReview timeout, retry, and transport guidance against the HTTPX documentation and response validation against the Pydantic documentation.

Last reviewed · verification recorded

Next stepDocument one provider's endpoint, cost, and rate limits, then wrap its client behind the interface before adding the next call site.

09
Automation

Manage database schema migrations reversibly

Problem

Schema changes ship entangled with application code, so a failed deploy leaves the database and the running version out of sync.

Full playbook — best for, checklist, verification, sources

Best for

Services introducing non-additive schema changes, and teams moving raw SQL scripts to a migration tool.

Not for

Greenfield databases with no live data, or stores without a migration story.

FlyPython decision and trade-off

Separate migrations from deploys and expand before you contract. The two-phase work takes longer, but every step stays reversible while the site serves traffic.

Checklist

  1. Separate schema migrations from application deployments whenever a change is not purely additive.
  2. Expand and contract: add nullable columns or tables first, write to both old and new fields, backfill in batches, then drop old columns.
  3. Review autogenerated migration SQL for destructive operations — dropped columns, table locks, unindexed constraints.
  4. Implement and test both upgrade and downgrade paths against a clean local database before submitting.
  5. Apply explicit statement timeouts during migration runs so production tables cannot lock indefinitely.
  6. Verify application behavior with both old and new schema versions active for zero-downtime rollout safety.

Supported environment

Python services using Alembic-style migrations against a relational database with a disposable local test database.

Verification

Documentation reviewReview the migration workflow against the Alembic documentation for autogeneration and upgrade/downgrade guidance.

Last reviewed · verification recorded

Next stepTake one pending schema change, split it into expand and contract migrations, and test both directions locally.

10
Automation

Set up production structured logging

Problem

Logs are unstructured print output: no request correlation, inconsistent fields, and secrets interleaved with useful events.

Full playbook — best for, checklist, verification, sources

Best for

Services and scheduled jobs moving from print or ad-hoc logging to searchable operational records.

Not for

Notebooks and one-off scripts where nobody will query the output later.

FlyPython decision and trade-off

Standardize JSON logs with propagated trace identifiers and redaction. The schema discipline costs setup time, but turns logs from text into queryable evidence.

Checklist

  1. Choose JSON output for production and readable text for local development; drop print statements and string concatenation.
  2. Propagate one request or trace identifier through every request, job, and downstream call using contextvars.
  3. Standardize the event schema: ISO 8601 timestamp, level, logger name, event name, trace identifiers, and structured payload fields.
  4. Redact sensitive fields automatically — passwords, tokens, authorization headers, API keys, and personal data.
  5. Capture full exception tracebacks only at service boundaries instead of every nested catch block.
  6. Test log output by asserting structured fields, not brittle substring matches.

Supported environment

Python 3.12+ services and jobs; the standard-library logging module plus contextvars covers propagation without extra dependencies.

Verification

Documentation reviewReview identifier propagation against the Python contextvars documentation and handler configuration against the logging documentation.

Last reviewed · verification recorded

Next stepDefine your event schema and redaction list first, then wire one request path end to end.

11
Automation

Ship a Python release you can trace and roll back

Problem

Releases are assembled from memory: the commit is ambiguous, artifacts are rebuilt per environment, and verification means the command exited zero.

Full playbook — best for, checklist, verification, sources

Best for

Package releases and service deployments that need a reviewable path from commit to reachable artifact.

Not for

Exploratory work, or platforms where a release pipeline already enforces these gates.

FlyPython decision and trade-off

Release one exact commit once, then verify the real user path. The discipline removes works-on-my-machine releases and makes rollback a documented step.

Checklist

  1. Select the exact commit and confirm tests, version, changelog, migrations, configuration, and compatibility from it.
  2. Build artifacts once in a clean environment and inspect their contents.
  3. Publish or deploy with least-privilege credentials; record the artifact digest, deployment identity, and configuration version.
  4. Verify installation or the production user path, not merely command success.
  5. Check health, logs, data changes, and critical integrations.
  6. If acceptance fails, stop the rollout and use the documented rollback; announce availability only after the artifact is verified reachable.

Supported environment

Python packages and deployable services; building with uv or twine and deploying through an existing pipeline both fit.

Verification

Documentation reviewReview the packaging and publication steps against the Python packaging user guide.

Last reviewed · verification recorded

Next stepRun the checklist for your next release and record the digest, deploy identity, and verification results in the release note.

12
AI agents

Write deterministic evals for LLMs and agents

Problem

Prompt and model changes ship on vibes: no golden dataset, no before-and-after numbers, and flaky judgments standing in for regression tests.

Full playbook — best for, checklist, verification, sources

Best for

Teams shipping prompt or model changes to production, and agent builders who need tool-calling to stay correct.

Not for

One-off model experiments, or quality programs that need human review panels rather than CI gates.

FlyPython decision and trade-off

Assert deterministic boundaries first and use semantic judges sparingly. Schema and tool-contract assertions catch most regressions cheaply on every change.

Checklist

  1. Curate a versioned golden dataset of real inputs covering happy paths, adversarial prompts, ambiguous cases, and known regressions.
  2. Define deterministic boundary assertions before semantic judges: JSON Schema conformity, required fields, disallowed tokens.
  3. Test tool-calling parameters against strict type contracts and assert tool selections without hallucinated arguments.
  4. Separate cheap local unit tests from live evaluations; use recorded fixtures in CI and run live evals on a schedule.
  5. Record pass rates, token counts, and latency before and after any prompt or model migration; never ship prompt changes without the comparison.
  6. Guard against flaky evals with tolerance thresholds and isolated temperature and seed parameters.

Supported environment

Python 3.12+ with pytest; JSON Schema validation of model output and recorded fixtures need no live model in CI.

Verification

Documentation reviewReview schema assertion guidance against the Python jsonschema documentation and the suite structure against the pytest documentation.

Last reviewed · verification recorded

Next stepCurate ten real inputs with expected boundaries, assert the schema and tool contracts in CI, then add the before-and-after comparison.

Supporting material

Secondary official references

Use these maintainer-owned sources to resolve implementation details after the matching FlyPython playbook has made the next step clear.

Python foundationsOfficial referencefoundation

The Python Tutorial ↗

A durable starting point for programmers learning Python syntax and core concepts.

Python Software FoundationChecked 2026-08-31
Python foundationsOfficial referenceintermediate

Python Packaging User Guide ↗

Explains modern project packaging, dependency management, publishing, and virtual environments.

PyPAChecked 2026-08-31
Python foundationsOfficial referenceintermediate

pytest documentation ↗

The practical testing reference for small scripts, libraries, services, and agent workflows.

pytest projectChecked 2026-08-31
Python foundationsOfficial referencefoundation

uv documentation ↗

A fast, modern workflow for Python versions, environments, dependencies, and project commands.

AstralChecked 2026-08-31
Web & APIsOfficial referenceintermediate

FastAPI documentation ↗

A direct route from typed Python functions to tested APIs with validation and generated documentation.

TiangoloChecked 2026-08-31
Web & APIsOfficial referencefoundation

Django documentation: First steps ↗

The official path through Django projects, models, views, templates, forms, tests, and reusable apps.

Django Software FoundationChecked 2026-09-06
Web & APIsOfficial referenceintermediate

Pydantic documentation ↗

The primary guide to validating untrusted data and expressing typed contracts at Python system boundaries.

PydanticChecked 2026-08-31
Web & APIsOfficial referenceintermediate

HTTPX documentation ↗

The official reference for synchronous and asynchronous HTTP clients, timeouts, streaming, and transport control.

EncodeChecked 2026-08-31
AutomationOfficial referencefoundation

pathlib — Object-oriented filesystem paths ↗

The standard-library reference for readable, cross-platform file and directory automation.

Python Software FoundationChecked 2026-08-31
AutomationOfficial referenceintermediate

subprocess — Subprocess management ↗

The standard-library contract for launching processes, capturing output, handling failures, and avoiding unsafe shell usage.

Python Software FoundationChecked 2026-08-31
AutomationOfficial referenceintermediate

Playwright for Python ↗

The official Python guide to reliable browser automation, locators, assertions, traces, and isolated contexts.

MicrosoftChecked 2026-08-31
AutomationOfficial referencefoundation

pandas getting started guides ↗

The primary entry point for tabular data loading, cleaning, transformation, analysis, and export.

pandas development teamChecked 2026-08-31
AI agentsOfficial referenceintermediate

OpenAI Agents SDK ↗

The official Python toolkit for agents, tools, handoffs, guardrails, sessions, tracing, and orchestration.

OpenAIChecked 2026-08-31
AI agentsOfficial referenceintermediate

Pydantic AI ↗

Typed agent development built around validated inputs, structured output, tools, testing, and model portability.

PydanticChecked 2026-08-31
AI agentsOfficial referenceintermediate

Model Context Protocol ↗

The open protocol and reference documentation for connecting AI applications to tools, data, and reusable context.

AnthropicChecked 2026-08-31
AI agentsOfficial referenceintermediate

Instructor documentation ↗

Structured outputs and validation for LLMs powered by Pydantic, enabling predictable tool calling.

Instructor teamChecked 2026-09-02

No playbooks or references match that search. Try a broader term or another track.