notcrawl 🗞️ — Your Notion memory, on disk

notcrawl mirrors a Notion workspace into local SQLite and normalized Markdown. It is for people and agents that need to search, query, diff, or share workspace history without depending on the Notion UI.

SQLite is the canonical archive. Markdown is the durable human-readable export.

Homebrew is the smallest install path on macOS and Linux:

brew install openclaw/tap/notcrawl

GitHub Releases also provide signed and notarized macOS archives, Linux archives, and .deb and .rpm packages. Download the appropriate file from the latest release.

With Notion Desktop installed and opened at least once:

notcrawl sync --source desktop
notcrawl search "launch plan"
notcrawl export-md

The sync reads a snapshot of Notion's local cache. Search uses the SQLite FTS5 index, and export-md writes the normalized archive under ~/.notcrawl/pages.

Run notcrawl doctor if the desktop cache is not found.

The official API is the stable remote integration:

export NOTION_TOKEN="secret_..."
notcrawl sync --source api

For a slow or stalled sync, add --verbose to that invocation:

notcrawl sync --source api --verbose

Verbose diagnostics go to stderr and report source phases, elapsed time, and counts. Official API sync also reports request attempts, endpoint classes, numeric HTTP statuses, and retry delays. No credentials, headers, request/response bodies, raw URLs, cursors, page identifiers, or upstream error text are logged. Warnings become counts and failures use a fixed message in verbose mode. Normal stdout is unchanged and may still contain local paths; without --verbose, the existing progress, warnings, and errors are unchanged. Desktop and MCP sync report source phases and counts, not per-request API traces.

Notion MCP can repair known incomplete pages, fetch a page by ID or URL, or run a bounded workspace search:

notcrawl sync --source notion-mcp --page PAGE_ID
notcrawl sync --source notion-mcp --query "launch plan" --limit 25

Without --page or --query, the MCP source retries known Desktop pages with missing cached content and API pages with incomplete block sync. It does not enumerate the entire workspace. Empty connector responses leave archived content unchanged and remain eligible for retry. The transport uses Codex authentication through the experimental ChatGPT apps gateway.

notcrawl tui opens a three-pane terminal browser for workspaces, teamspaces, pages, and databases. It supports keyboard and mouse navigation, filtering, sorting, local refresh, opening or copying the selected Notion URL, and local/remote state in the footer.

Database rows can be exported separately from the Markdown archive:

notcrawl databases
notcrawl export-db --database DATABASE_ID --format csv --output roadmap.csv
notcrawl export-db --all --dir exports/csv

The main commands are:

Run notcrawl --help for the full command summary.

sql accepts one SELECT, WITH, or PRAGMA statement and opens the archive read-only. Leading comments, multiline queries, and quoted punctuation are supported. It never creates or migrates the archive; run sync first to prepare a new archive or upgrade an older schema.

Git share mode publishes compressed JSONL table snapshots and normalized Markdown. Another machine can subscribe and search the archive without Notion credentials.

publish --tag NAME creates an immutable checkpoint. subscribe and update merge snapshots without deleting local-only rows by default; --restore requests exact replacement, and --retain-revisions saves replaced local payloads. Snapshot imports preserve signed 64-bit integer literals exactly, including ordering values above 2^53; out-of-range integer literals fail the import without changing the archive. Decimal and exponent notation retain floating-point behavior.

An archive with no Markdown pages can still be published and subscribed to.

Secrets are not included in Markdown or git share snapshots. Structured signed-file URL credentials are removed from these outgoing projections; ordinary link parameters and private local raw recovery payloads are retained.

export-md and publish stop with an error if protected provider JSON is malformed or a signed file URL cannot be safely sanitized. They do not skip the record or remove its raw payload from the local archive.

Before investigating, stop archive writers and make a consistent, private SQLite backup, including any uncheckpointed WAL data. Keep the original archive and backup unchanged. Use read-only queries on the backup to inspect JSON validity and record counts; do not print or share raw payloads, signed URLs, or credentials in diagnostics or issue reports.

If the source is still available, use a separate configuration with new db_path, cache_dir, markdown_dir, and [share].repo_path locations to reimport it. Select the intended source explicitly. Desktop cache coverage is limited, and a normal resync is not guaranteed to repair historical records. Preserve the original archive for records the source can no longer supply.

A failed export can leave partial or stale files in the Markdown or share directory; output changes are not rolled back atomically. Do not manually commit, push, or distribute that output. Retry into separate output locations after resolving the source problem, and require a successful export before using the result.

notcrawl init writes ~/.notcrawl/config.toml. The default data paths are:

See config.example.toml for every setting.

To limit future Desktop ingestion to selected workspaces, set space_ids under [notion.desktop] in your config:

[notion.desktop]
enabled = true
space_ids = ["01234567-89ab-cdef-0123-456789abcdef"]

An empty list (the default) includes all cached workspaces. IDs match regardless of case or UUID hyphens. Find known IDs with notcrawl sql "SELECT id, name FROM spaces" in an existing archive. The filter applies to sync --source desktop, tap, and the Desktop phase of sync. Rows without a workspace ID are skipped when a list is set.

The list filters workspace content, not the complete archive: existing rows and their Markdown remain, shared user metadata is still imported, and cache snapshots still contain the complete local Notion database. API and MCP sync are unaffected. For a new Desktop-only archive, use separate db_path, cache_dir, and markdown_dir settings and disable the API and MCP sources.

Interactive terminal runs check for a newer release once per day. notcrawl check-update checks immediately; set NOTCRAWL_NO_UPDATE_CHECK=1 or CRAWLKIT_NO_UPDATE_CHECK=1 to disable the passive check.

Desktop mode is read-only. It snapshots Notion's local SQLite database before reading it and never writes to Notion application storage. Cache coverage is opportunistic, so missing rows are not treated as deletions; explicit Notion tombstones still retire records. Markdown marks pages whose bodies were not cached so the API or MCP source can fill them later.

API mode uses the official Notion API and stores raw payloads alongside normalized rows so exports can improve without another crawl.

Notion MCP mode is read-only and targeted. It reads the Codex bearer credential at request time, never stores it, resolves only the connected Notion search and fetch tools, and strips signed URL credentials before persisting connector Markdown. Credentials are sent only to the configured HTTPS ChatGPT apps gateway. The gateway and Codex auth-file format are experimental contracts.

notcrawl uses crawlkit for config paths, SQLite helpers, snapshot packing and import, git-backed sharing, output formatting, status payloads, and the terminal explorer. Notion API and Desktop parsing, schemas, Markdown rendering, and FTS content remain in this repository.

See SPEC.md for the data model and archive contracts. Maintainers can find release packaging and verification in docs/distribution.md.

Go 1.27.1 or newer is required.

make build
make test
make check

make check runs the dependency, formatting, vet, dead-code, test, smoke, release-config, and snapshot gates used by CI. See CONTRIBUTING.md before sending a change.

MIT. See LICENSE.