Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/build-expt.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@ jobs:
uses: actions/setup-go@v4
with:
go-version: ^1.19
- name: Install cross-compiler for linux/arm64
run: sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
- name: Run Goreleaser
uses: goreleaser/goreleaser-action@v5
with:
Expand Down
198 changes: 112 additions & 86 deletions .github/workflows/build.yml

Large diffs are not rendered by default.

18 changes: 2 additions & 16 deletions .github/workflows/openai-plugin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,24 +37,10 @@ jobs:
- uses: actions/setup-go@v6
with:
go-version-file: go.mod
- name: Configure Windows C toolchain
if: runner.os == 'Windows'
shell: pwsh
run: |
$mingwPath = 'C:\mingw64'
if (-not (Test-Path $mingwPath)) {
$mingwPath = 'C:\ProgramData\chocolatey\lib\mingw\tools\install\mingw64'
}
if (-not (Test-Path $mingwPath)) {
throw 'MinGW toolchain not found'
}
"$mingwPath\bin" | Out-File $env:GITHUB_PATH -Append
"CC=$mingwPath\bin\x86_64-w64-mingw32-gcc" | Out-File $env:GITHUB_ENV -Append
"CXX=$mingwPath\bin\x86_64-w64-mingw32-g++" | Out-File $env:GITHUB_ENV -Append
- name: Build StackQL
env:
CGO_ENABLED: '1'
run: go build --tags sqlite_stackql -o ${{ matrix.binary }} ./stackql
CGO_ENABLED: '0'
run: go build -o ${{ matrix.binary }} ./stackql
- name: Validate package
run: ${{ matrix.python }} packaging/openai-plugin/scripts/validate.py
- name: Test stdio connector
Expand Down
26 changes: 25 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Authoritative references: [developer guide](/docs/developer_guide.md), [test sum
Common commands (run from repo root):

- Build: `python cicd/python/build.py --build` (output at `./build/stackql`).
- Unit tests: `python cicd/python/build.py --test` (CI uses `go test -timeout 1200s --tags "sqlite_stackql" -v ./...`).
- Unit tests: `python cicd/python/build.py --test` (CI uses `go test -timeout 1200s -v ./...`).
- Robot tests: `python cicd/python/build.py --robot-test` (requires the binary from the build step).
- Lint: `golangci-lint run` (CI pins the version in [`.github/workflows/lint.yml`](/.github/workflows/lint.yml); config in [`.golangci.yml`](/.golangci.yml)).

Expand Down Expand Up @@ -58,6 +58,30 @@ StackQL supports flexible configuration management, including authentication sec

---

## SQLite backend

- The embedded backend is `modernc.org/sqlite` (pure Go, no cgo), pinned >= v1.57.0
(caller-constructed `Driver` function registration). It is consumed via
[`any-sdk`](https://github.com/stackql/any-sdk) `public/sqlengine`, which registers
the driver under the name `stackql-sqlite`.
- DSN construction happens only via the central `BuildDSN` in any-sdk `public/sqlengine`
(legacy mattn-style params such as `_busy_timeout=5000` are translated to modernc
`_pragma=busy_timeout(5000)` syntax there); driver errors are inspected only via the
error predicates in that package (`isBusy`, `isConstraintViolation`); the custom
extension functions (`split_part`, `regexp_like`, `regexp_substr`, `regexp_replace`,
`json_equal`, `aws_policy_equal`) live only in any-sdk `public/sqlfuncs` and are
registered on the constructed driver, never globally.
- Never reintroduce `mattn/go-sqlite3`, cgo, the `sqlite_stackql` build tag, or
package-global function registration. All builds and release artifacts are
`CGO_ENABLED=0`.
- Bumps of `modernc.org/sqlite` merge only after the full robot suite passes
(transpile regenerations occasionally regress).
- Rollback anchor: the tag `v0.11.707-final-cgo` marks the last cgo commit on `main`
(the `stackql-go-sqlite3` fork plus the `sqlite_stackql` build tag). To roll back
the pure Go migration, branch from that tag.

---

## StackQL Resource Key Encoding Quirk

### Hierarchical Keys and URL Encoding
Expand Down
4 changes: 4 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ Before beginning implementation work, please raise an issue and receive a consen

Please raise pull requests (PRs) from a fork of this repository, using the provided PR template. The richer the detail / evidence supplied, the easier the review process. A usefully named feature branch for the PR source is appreciated.

## Dependency notes

- Bumps of `modernc.org/sqlite` (the pure Go embedded SQLite backend, consumed via `any-sdk`) merge only after the full robot test suite passes; its transpile regenerations occasionally regress.

Finally, in respect of all your efforts; we thank you and ask that you please give us a star while you are at it.

Many Thanks,
Expand Down
7 changes: 3 additions & 4 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -46,20 +46,19 @@ ENV SRC_DIR=/work/stackql/src
ENV BUILD_DIR=/work/stackql/build

RUN cd ${SRC_DIR} \
&& go test --tags "sqlite_stackql" ./... \
&& go build -ldflags "-X github.com/stackql/stackql/internal/stackql/cmd.BuildMajorVersion=$BUILDMAJORVERSION \
&& CGO_ENABLED=0 go test ./... \
&& CGO_ENABLED=0 go build -ldflags "-X github.com/stackql/stackql/internal/stackql/cmd.BuildMajorVersion=$BUILDMAJORVERSION \
-X github.com/stackql/stackql/internal/stackql/cmd.BuildMinorVersion=$BUILDMINORVERSION \
-X github.com/stackql/stackql/internal/stackql/cmd.BuildPatchVersion=$BUILDPATCHVERSION \
-X github.com/stackql/stackql/internal/stackql/cmd.BuildCommitSHA=$BUILDCOMMITSHA \
-X github.com/stackql/stackql/internal/stackql/cmd.BuildShortCommitSHA=$BUILDSHORTCOMMITSHA \
-X \"github.com/stackql/stackql/internal/stackql/cmd.BuildDate=$BUILDDATE\" \
-X \"stackql/internal/stackql/planbuilder.PlanCacheEnabled=$PLANCACHEENABLED\" \
-X github.com/stackql/stackql/internal/stackql/cmd.BuildPlatform=$BUILDPLATFORM" \
--tags "sqlite_stackql" \
-o ${BUILD_DIR}/stackql ./stackql

RUN cd ${SRC_DIR} \
&& go build \
&& CGO_ENABLED=0 go build \
-o ${BUILD_DIR}/stackql_mcp_client ./mcp_client/cmd

# bookworm, not bullseye: bullseye LTS ended 2026-08-31 and its security suite
Expand Down
8 changes: 4 additions & 4 deletions cicd/python/build.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ def build_stackql(verbose :bool) -> int:
os.environ['BUILDMAJORVERSION'] = os.environ.get('BUILDMAJORVERSION', '1')
os.environ['BUILDMINORVERSION'] = os.environ.get('BUILDMINORVERSION', '1')
os.environ['BUILDPATCHVERSION'] = os.environ.get('BUILDPATCHVERSION', '1')
os.environ['CGO_ENABLED'] = os.environ.get('CGO_ENABLED', '1')
os.environ['CGO_ENABLED'] = os.environ.get('CGO_ENABLED', '0')
return subprocess.call(
f'go build {"-x -v" if verbose else ""} --tags "sqlite_stackql" -ldflags "-X github.com/stackql/stackql/internal/stackql/cmd.BuildMajorVersion={os.environ.get("BUILDMAJORVERSION")} '
f'go build {"-x -v" if verbose else ""} -ldflags "-X github.com/stackql/stackql/internal/stackql/cmd.BuildMajorVersion={os.environ.get("BUILDMAJORVERSION")} '
f'-X github.com/stackql/stackql/internal/stackql/cmd.BuildMinorVersion={os.environ.get("BUILDMINORVERSION")} '
f'-X github.com/stackql/stackql/internal/stackql/cmd.BuildPatchVersion={os.environ.get("BUILDPATCHVERSION")} '
f'-X github.com/stackql/stackql/internal/stackql/cmd.BuildCommitSHA={os.environ.get("BUILDCOMMITSHA", "")} '
Expand All @@ -31,7 +31,7 @@ def build_stackql_mcp_client(verbose :bool) -> int:
os.environ['BUILDMAJORVERSION'] = os.environ.get('BUILDMAJORVERSION', '1')
os.environ['BUILDMINORVERSION'] = os.environ.get('BUILDMINORVERSION', '1')
os.environ['BUILDPATCHVERSION'] = os.environ.get('BUILDPATCHVERSION', '1')
os.environ['CGO_ENABLED'] = os.environ.get('CGO_ENABLED', '1')
os.environ['CGO_ENABLED'] = os.environ.get('CGO_ENABLED', '0')
return subprocess.call(
'go build '
f'{"-x -v" if verbose else ""} '
Expand All @@ -42,7 +42,7 @@ def build_stackql_mcp_client(verbose :bool) -> int:

def unit_test_stackql(verbose :bool) -> int:
return subprocess.call(
f'go test -timeout 1200s {"-v" if verbose else ""} --tags "sqlite_stackql" ./...',
f'go test -timeout 1200s {"-v" if verbose else ""} ./...',
shell=True
)

Expand Down
9 changes: 4 additions & 5 deletions docs/developer_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,7 @@ As of now, `stackql` handles `xml` SERDE through the core, and does not route th
## Building locally

```bash
env CGO_ENABLED=1 PLANCACHEENABLED=false go build \
--tags "sqlite_stackql" \
env CGO_ENABLED=0 PLANCACHEENABLED=false go build \
-ldflags "-X github.com/stackql/stackql/internal/stackql/cmd.BuildMajorVersion=${BUILDMAJORVERSION:-1} \
-X github.com/stackql/stackql/internal/stackql/cmd.BuildMinorVersion=${BUILDMINORVERSION:-1} \
-X github.com/stackql/stackql/internal/stackql/cmd.BuildPatchVersion=${BUILDPATCHVERSION:-1} \
Expand All @@ -71,7 +70,7 @@ We have upgraded to golang `>= 1.25.3` in order to take advantage of [time simul
To run all unit tests:

```bash
go test -timeout 1200s --tags "sqlite_stackql" ./...
go test -timeout 1200s ./...
```

### Robot tests
Expand Down Expand Up @@ -186,7 +185,7 @@ Requirements are [detailed in the root README](/README.md#system-requirements).

Local testing of the application:

1. Run `go test --tags "sqlite_stackql" ./...` tests.
1. Run `go test ./...` tests.
2. Build the executable [as per the root README](/README.md#build)
3. Perform registry rewrites as needed for mocking `python3 test/python/stackql_test_tooling/registry_rewrite.py --srcdir "$(pwd)/test/registry/src" --destdir "$(pwd)/test/registry-mocked/src"`.
3. Run robot tests:
Expand All @@ -205,7 +204,7 @@ Test coverage is sparse. Regressions are mitigated by `go test` integration tes
#### Point in time gotest coverage

If not already done, then install 'cover' with `go get golang.org/x/tools/cmd/cover`.
Then: `go test --tags "sqlite_stackql" -cover ../...`.
Then: `go test -cover ../...`.

### Functional and Integration testing

Expand Down
2 changes: 1 addition & 1 deletion docs/diagrams/plantuml/hld-components.puml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ node "any-sdk" as AnySdk
node "stackql-parser" as StackQLParser
node "psql-wire" as PsqlWire
node "stackql-provider-registry" as ProviderRegistry
node "stackql-go-sqlite3" as StackqlSqlite
node "modernc.org/sqlite" as StackqlSqlite
node "postgres" as Postgres
node "postgres clients\neg: libpq" as PostgresClient
node "AI agents\neg: Claude Code" as AIAgent
Expand Down
2 changes: 1 addition & 1 deletion docs/licenses/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Some deep links to original licenses are supplied for convenience:

| Name | Source | Local copy if present | Original License |
| ----- | ----- | ------ | ------ |
| go-sqlite3 | https://github.com/mattn/go-sqlite3 | [go-sqlite3_license](/docs/licenses/go-sqlite3_license) | https://github.com/mattn/go-sqlite3/blob/master/LICENSE |
| modernc.org/sqlite | https://gitlab.com/cznic/sqlite | [modernc-sqlite_license](/docs/licenses/modernc-sqlite_license) | https://gitlab.com/cznic/sqlite/-/blob/master/LICENSE |
| psql-wire | https://github.com/jeroenrinzema/psql-wire | [psql-wire_license](/docs/licenses/psql-wire_license) | https://github.com/jeroenrinzema/psql-wire/blob/ccdd1b552fc805192177fb5263a67b735603406e/LICENSE |
| readline | https://github.com/chzyer/readline | [readline_license](/docs/licenses/readline_license) | https://github.com/chzyer/readline/blob/master/LICENSE |
| [vitess](https://vitess.io) | https://github.com/vitessio/vitess | [vitess_license](/docs/licenses/vitess_license) | https://github.com/vitessio/vitess/blob/main/LICENSE |
Expand Down
21 changes: 0 additions & 21 deletions docs/licenses/go-sqlite3_license

This file was deleted.

26 changes: 26 additions & 0 deletions docs/licenses/modernc-sqlite_license
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
Copyright (c) 2017 The Sqlite Authors. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software without
specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
2 changes: 1 addition & 1 deletion docs/sql_backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,5 +53,5 @@ List of drivers:

### Data Source Name (DSN) strings

- [SQLite as per golang](https://github.com/mattn/go-sqlite3#dsn-examples).
- [SQLite as per golang (modernc.org/sqlite)](https://pkg.go.dev/modernc.org/sqlite#Driver.Open).
- [Postgres URI](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING).
10 changes: 10 additions & 0 deletions docs/stackql release process.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
## StackQL release process

### Release notes: pure Go SQLite backend (first release after the modernc migration)

Include the following in the notes of the first release cut after the modernc.org/sqlite migration:

> The embedded SQLite backend is now [`modernc.org/sqlite`](https://gitlab.com/cznic/sqlite) (pure Go). All binaries are built with `CGO_ENABLED=0` and are statically linked; no C toolchain is involved anywhere in the build.
> The custom extension functions (`split_part`, `regexp_like`, `regexp_substr`, `regexp_replace`, `json_equal`, `aws_policy_equal`) are ported to pure Go in [`any-sdk`](https://github.com/stackql/any-sdk); documented behaviour divergences (RE2 regexp limits) are listed in any-sdk `public/sqlfuncs/DIVERGENCES.md`.
> Library embedders now inherit the `modernc.org/sqlite` dependency tree instead of `mattn/go-sqlite3`/cgo.
> External loadable SQLite extensions (`.load`) are no longer supported.
> Rollback anchor: the tag `v0.11.707-final-cgo` marks the last cgo commit on `main`.

1. Download Artifacts from Latest Build

Download the following artifacts from the latest build on the `main` branch including:
Expand Down
Loading