crawlkit 🧰 — one kit, many crawlers

crawlkit is the shared Go library for local-first crawler archives. It gives crawler authors provider-neutral building blocks for config paths, SQLite stores, snapshots, backups, synchronization, search, terminal interfaces, and automation.

Provider APIs, authentication, schemas, privacy filters, and user-facing command contracts stay in the downstream crawl apps.

crawlkit requires Go 1.27.0 or newer.

Development builds prefer Go 1.27.1 through the toolchain directive in go.mod.

Add the package you need to a Go module. For the quick start below:

go get github.com/openclaw/crawlkit/store@latest

Install the optional archive controller:

go install github.com/openclaw/crawlkit/cmd/crawlctl@latest

The latest module version is available through the Go module proxy. Signed CLI archives and the release process are documented in Publishing Crawlkit.

This example opens an in-memory SQLite store with crawlkit's connection defaults, applies a schema, writes a row, and reads it back:

package main

import (
	"context"
	"fmt"

	"github.com/openclaw/crawlkit/store"
)

func main() {
	ctx := context.Background()
	db, err := store.Open(ctx, store.Options{
		Path:   ":memory:",
		Schema: `create table items (title text not null)`,
	})
	if err != nil {
		panic(err)
	}
	defer db.Close()

	if _, err := db.DB().ExecContext(ctx, `insert into items values (?)`, "first crawl"); err != nil {
		panic(err)
	}
	rows, err := db.Query(ctx, `select title from items`)
	if err != nil {
		panic(err)
	}
	fmt.Println(rows.Values[0]["title"])
}

Run it from a module that depends on crawlkit:

go run .

first crawl

See the package guide for the complete inventory and Go package reference for exported APIs.

crawlctl discovers installed crawl apps through their machine-readable metadata, runs configured refresh jobs under a held OS lock, and records JSONL run history.

The lock file is persistent: do not remove or replace it to unlock the runner. Closing the owning handle or exiting releases the lock. Keep its directory private; on Windows, file access follows the directory's ACLs. Unsupported OS locking fails closed.

Stop all old runners before upgrading from PID-file locking. Legacy or ambiguous lock contents require an explicit stopped-runner migration: verify all old runners are stopped, archive that legacy file once, then start the new runner. There is no mixed old/new runner safety or automatic stale-PID reclamation.

Default discovery checks gitcrawl, discrawl, notcrawl, wacrawl, telecrawl, slacrawl, graincrawl, imsgcrawl, photoscrawl, and weicrawl. Use --app to select binaries explicitly. Discovery does not automatically schedule source-specific commands such as Photos' import_apple.

If a write is interrupted, history reads ignore a truncated final JSON value and the next run repairs that tail before appending. Valid final records without a trailing newline are retained; complete corrupt records still report an error.

History reads accept the full records written by the runner, including long command arguments. Explicit configuration paths work without a home directory; default and ~/ paths still require one. Saving a controller configuration applies private file permissions before replacing existing content.

Scheduling uses launchd on macOS, systemd user units on Linux, Task Scheduler on Windows, and cron rendering as the portable fallback.

Systemd plans preserve literal argument text, including quotes, percent signs, and dollar signs. Executable paths must follow systemd rules: quotes, backslashes, and control characters are rejected before installation. Dry-run previews can still render plans for another operating system.

See Background workers for content-triggered processing alongside ingestion.

crawlkit accepts shared mechanics only when they are provider-neutral, reusable by at least two apps, and preserve each app's database and CLI contracts. The ownership map tracks what belongs here and what remains in GitHub-, Discord-, Slack-, Notion-, and other provider-specific applications.

The remote package owns the Go client and v1 wire contract for hosted archives. Worker deployment, D1 schema, authentication policy, and secrets live outside this module; see the remote contract and Cloudflare archive design.

Tests and examples use temporary or in-memory data. They do not access app runtime stores such as ~/.config/gitcrawl, ~/.slacrawl, ~/.discrawl, or ~/.notcrawl.

Snapshot imports reject absolute and parent-traversing shard paths while preserving literal root and shard names, including current-directory roots. This is a lexical check: snapshots must still come from a trusted filesystem tree because shard reads follow symlinks.

Pass a plain filesystem path to store.Open unless SQLite driver parameters are intentional. A caller-supplied file: URI keeps its query parameters, which can override crawlkit's default pragmas or fail connection validation.

make check

This runs module tidiness, formatting, vet, dead-code and vulnerability checks, unit tests, and race tests with GOWORK=off. See CONTRIBUTING.md for the compatibility rules.

Keep modernc.org/libc at the exact version required by the selected modernc.org/sqlite module. SQLite v1.58.0 requires libc v1.75.6; its runtime dependency must be updated together with SQLite, not independently.

MIT