Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 31 additions & 18 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,40 +4,53 @@ These guidelines help contributors, human and otherwise, understand and work eff

We also encourage reading [`docs/developer_guide.md`](/docs/developer_guide.md) for further useful information. For deeper understanding of the "brains" of `stackql`, it is worthwhile to consult [the `AGENTS.md` file of `any-sdk`](https://github.com/stackql/any-sdk/blob/main/AGENTS.md) and linked documents from there.

We have upgraded to golang `>= 1.25.3` in order to take advantage of [time simultion and other capabilites in `testing/synctest`](https://go.dev/blog/testing-time).
We have upgraded to golang `>= 1.25.3` in order to take advantage of [time simulation and other capabilities in `testing/synctest`](https://go.dev/blog/testing-time).


## Project Structure & Module Organization

- Entrypoint: [`stackql/main.go`](/stackql/main.go).
- Ideally, foregin system semantics are dealt with in the `any-sdk` repository.
- Ideally, foreign system semantics are dealt with in the `any-sdk` repository.
- Loose adherence to popular idioms:
- App internals in [`internal`](/internal).
- Re-usable modules in [`pkg`](/pkg).
— The MCP server function is built upon the golang MCP SDK.
- The MCP server function is built upon the golang MCP SDK.
- CICD: please see [the github actions workflows](/.github/workflows).
- Docs: `README.md`, this `AGENTS.md`.

## Build, Test, and Development Commands

Please refer to [the developer guide](/docs/developer_guide.md), [the testing summary document](/test/README.md), [the robot testing document](/test/robot/README.md), and possibly most useful of all, [the doco explaining testing with mocks](/test/python/stackql_test_tooling/flask/README.md). For CI in the wild, please see [`.github/workflows/build.yml`](/.github/workflows/build.yml).
Authoritative references: [developer guide](/docs/developer_guide.md), [test summary](/test/README.md), [robot tests](/test/robot/README.md), [mock testing](/test/python/stackql_test_tooling/flask/README.md), and [CI workflow](/.github/workflows/build.yml).

Common commands (run from repo root):

- Build: `python cicd/python/build.py --build` (output at `./build/stackql`).
- Unit tests: `python cicd/python/build.py --test` (CI uses `go test -timeout 1200s --tags "sqlite_stackql" -v ./...`).
- Robot tests: `python cicd/python/build.py --robot-test` (requires the binary from the build step).
- Lint: `golangci-lint run` (CI pins the version in [`.github/workflows/lint.yml`](/.github/workflows/lint.yml); config in [`.golangci.yml`](/.golangci.yml)).

## Coding Style & Naming Conventions

- Publish and program to abstractions.
- Program to abstractions; concrete types and foreign-system semantics belong behind interfaces, ideally in `any-sdk`.
- `gofmt`/`goimports` formatting; `golangci-lint` must pass (see [`.golangci.yml`](/.golangci.yml)).
- Go identifier conventions: exported `CamelCase`, unexported `camelCase`, acronyms uppercase (`HTTPClient`, not `HttpClient`). Package names short and lowercase.

## Testing Guidelines

- Black box regression tests are effectively mandatory. The canaonical ones reside in [`test/robot`](/test/robot/README.md).
- Black-box regression tests are effectively mandatory for behaviour changes. The canonical ones reside in [`test/robot`](/test/robot/README.md) and run against mocks defined in [`test/python/stackql_test_tooling/flask`](/test/python/stackql_test_tooling/flask/README.md).
- Add a new robot scenario when you introduce or fix user-visible query behaviour; unit tests alone are not sufficient evidence.

## Tools & Resources

- Please inspect using the API.

- Inspect provider surface via the running server: `SHOW PROVIDERS;`, `SHOW SERVICES IN <provider>;`, `DESCRIBE <resource>;`.
- Provider definitions live in the [`stackql-provider-registry`](https://github.com/stackql/stackql-provider-registry); the request-execution brain lives in [`any-sdk`](https://github.com/stackql/any-sdk).

## Commit & Pull Request Guidelines

- Fork and pull model for general public; we **strongly** welcome public contributions, comment and issues.
- Fork-and-pull for external contributors; public contributions, issues, and comments are welcome.
- PR title doubles as the squash-merge commit subject (see recent `main` history for the prevailing style: short hyphenated slug, optional `(#NNN)` appended on merge).
- Keep PRs focused; include a brief summary and a test plan (robot scenario added/modified, or a justification when not applicable).
- All required checks (build, lint, robot) must be green before merge.

## Security & Configuration Tips

Expand All @@ -49,31 +62,31 @@ StackQL supports flexible configuration management, including authentication sec

### Hierarchical Keys and URL Encoding

StackQL resource methods (especially for GitHub refs/tags) require exact matches for hierarchical keys, such as `refs/tags/x1.3`. However, due to dependency library behavior (e.g., gorilla/mux in Go), forward slashes in parameters may be interpreted as path delimiters, causing query failures or incorrect resource matching.
StackQL resource methods require exact matches for hierarchical keys, such as `refs/tags/x1.3`. Historically, due to dependency library behavior (e.g., gorilla/mux in Go), forward slashes in parameters were interpreted as path delimiters, causing query failures or incorrect resource matching.

**Current status (since PR #648, `slash-path-parameters`):** A literal `/` is supported inside a path parameter **only** when the surrounding URL template has no consecutive path parameters. In that case the slash can be passed verbatim. The robot test `Google Cloud Asset IAM Policies Select With Slash In Path Param` covers this.

**Workaround:** Always URL encode forward slashes in resource keys when constructing queries. For example:
**When the workaround is still required:** If the slash-bearing parameter is adjacent to another path parameter in the URL template (e.g. consecutive `{p1}/{p2}` slots, as seen in many GitHub routes), the slash must still be URL-encoded as `%2F`. For example, against an endpoint shaped like `/repos/{owner}/{repo}/git/refs/{ref}` where `{ref}` follows other path parameters:

- **Incorrect:**
- **May fail:**
```sql
SELECT ref FROM github.git.refs WHERE owner = 'stackql' AND repo = 'mcp-stackql' AND ref = 'refs/tags/x2.0';
```
- **Correct:**
- **Safe fallback:**
```sql
SELECT ref FROM github.git.refs WHERE owner = 'stackql' AND repo = 'mcp-stackql' AND ref = 'tags%2Fx2.0';
```

This ensures the backend treats the parameter as a literal string, not a path.
If you are unsure whether a given route has consecutive path parameters, encoding the slash is always safe.

### Guidance for LLMs and Contributors

- Always encode slashes in hierarchical resource keys for StackQL queries.
- Prefer passing the literal value first; if the query fails or returns the wrong resource, fall back to URL-encoding any embedded slashes.
- Do not use wildcards or inequalities for `WHERE` clauses that map to parameters (eg: HTTP path parameters); in such cases, only exact matches are supported.
- Document this quirk in scripts, tests, and code comments to avoid confusion.
- Example for LLM prompt:
> "When querying StackQL resources with keys containing slashes, always URL encode the slash."

### Why This Is Necessary

Many RESTful routing libraries (like gorilla/mux) treat slashes as path separators. Encoding slashes prevents misinterpretation and ensures correct resource access.
Many RESTful routing libraries (like gorilla/mux) treat slashes as path separators. PR #648 relaxes this for the single-parameter case, but the consecutive-parameter case remains ambiguous to the router, so encoding is still needed there.

Refer to this section whenever you encounter issues with resource keys containing slashes or hierarchical identifiers.
11 changes: 11 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# CLAUDE.md

Project memory for Claude Code working on `stackql`.

The canonical contributor guide is imported below. Do not duplicate its content here - edit [AGENTS.md](/AGENTS.md) instead.

@AGENTS.md

## Claude Code specifics

Add entries here only when they are Claude Code-specific and not appropriate for `AGENTS.md` (e.g. tool allowlists, hook notes, harness quirks). Everything else belongs in `AGENTS.md` or the docs it links to.
4 changes: 2 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ require (
github.com/spf13/cobra v1.4.0
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.10.1
github.com/stackql/any-sdk v0.5.2-beta04
github.com/stackql/any-sdk v0.5.2-beta05
github.com/stackql/go-suffix-map v0.0.1-alpha01
github.com/stackql/psql-wire v0.1.2-beta01
github.com/stackql/stackql-parser v0.0.16-alpha01
github.com/stackql/stackql-parser v0.0.16-alpha02
github.com/stretchr/testify v1.10.0
golang.org/x/mod v0.25.0
golang.org/x/sync v0.15.0
Expand Down
8 changes: 4 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -467,8 +467,8 @@ github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.10.1 h1:nuJZuYpG7gTj/XqiUwg8bA0cp1+M2mC3J4g5luUYBKk=
github.com/spf13/viper v1.10.1/go.mod h1:IGlFPqhNAPKRxohIzWpI5QEy4kuI7tcl5WvR+8qy1rU=
github.com/stackql/any-sdk v0.5.2-beta04 h1:GnXiDQCWF2VdSHbtw1b4EmaT/GtUdVq+Cr622p2p/98=
github.com/stackql/any-sdk v0.5.2-beta04/go.mod h1:ZRTGrcDKFLaC+5yWo4OqXVs1HTNxgYM9nQsQDlq0Fe0=
github.com/stackql/any-sdk v0.5.2-beta05 h1:32dRW2L1KjKNpGg8nVcQJTYVWsQdqdhPbVM6JSOgcMc=
github.com/stackql/any-sdk v0.5.2-beta05/go.mod h1:ZRTGrcDKFLaC+5yWo4OqXVs1HTNxgYM9nQsQDlq0Fe0=
github.com/stackql/go-suffix-map v0.0.1-alpha01 h1:TDUDS8bySu41Oo9p0eniUeCm43mnRM6zFEd6j6VUaz8=
github.com/stackql/go-suffix-map v0.0.1-alpha01/go.mod h1:QAi+SKukOyf4dBtWy8UMy+hsXXV+yyEE4vmBkji2V7g=
github.com/stackql/psql-wire v0.1.2-beta01 h1:+0xeLLam3nbI1iv84pWJYGYw5VO5S/+eICRbdu6+dj8=
Expand All @@ -477,8 +477,8 @@ github.com/stackql/readline v0.0.2-alpha05 h1:ID4QzGdplFBsrSnTuz8pvKzWw96JbrJg8f
github.com/stackql/readline v0.0.2-alpha05/go.mod h1:OFAYOdXk/X4+5GYiDXFfaGrk+bCN6Qv0SYY5HNzD2E0=
github.com/stackql/stackql-go-sqlite3 v1.0.5-stackql h1:cgy71EPxlbURHAjAn+gwNmDgjhWdm6V3zGY5QvB78eY=
github.com/stackql/stackql-go-sqlite3 v1.0.5-stackql/go.mod h1:HemqCrcMK2xyhMMMt6oZ7ERDtoSmyyDsw5LBBcTZ+Rk=
github.com/stackql/stackql-parser v0.0.16-alpha01 h1:R3jYy9O8TGn0b7UlpyEOssXUkI8CuNFTWgjo5BOh+J0=
github.com/stackql/stackql-parser v0.0.16-alpha01/go.mod h1:iyB47SvRS+Fvpn7joF7mHAkeiWSq83TbUhglRmLzPLQ=
github.com/stackql/stackql-parser v0.0.16-alpha02 h1:kU3THgY23jS39Uv8b0xnFp329gWWWsKbsyvbFWeGrL8=
github.com/stackql/stackql-parser v0.0.16-alpha02/go.mod h1:iyB47SvRS+Fvpn7joF7mHAkeiWSq83TbUhglRmLzPLQ=
github.com/stackql/stackql-provider-registry v0.0.2-alpha01 h1:fPz6MhfXaMC/VPNJnjUo0sZyjTYhlifzaUWZ9Yj8NVE=
github.com/stackql/stackql-provider-registry v0.0.2-alpha01/go.mod h1:87rVxnS2aRASK20lBQgoYA0o7FSJTZBGGRaWFR7IDm4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
Expand Down
4 changes: 4 additions & 0 deletions internal/stackql/astanalysis/earlyanalysis/ast_expand.go
Original file line number Diff line number Diff line change
Expand Up @@ -776,6 +776,10 @@ func (v *indirectExpandAstVisitor) Visit(node sqlparser.SQLNode) error {
return err
}

case *sqlparser.DescribeMethod:
// DescribeMethod is resolved directly by the plan builder and does not
// participate in the table-metadata hierarchy expansion.

case *sqlparser.OtherAdmin:
buf.WriteString("otheradmin")

Expand Down
2 changes: 2 additions & 0 deletions internal/stackql/astvisit/assign_leftover_references.go
Original file line number Diff line number Diff line change
Expand Up @@ -403,6 +403,8 @@ func (v *standardLeftoverReferencesAstVisitor) Visit(node sqlparser.SQLNode) err

case *sqlparser.DescribeTable:

case *sqlparser.DescribeMethod:

case *sqlparser.OtherAdmin:

case sqlparser.Comments:
Expand Down
4 changes: 4 additions & 0 deletions internal/stackql/astvisit/fragment_rewriting.go
Original file line number Diff line number Diff line change
Expand Up @@ -583,6 +583,10 @@ func (v *standardFragmentRewriteAstVisitor) Visit(node sqlparser.SQLNode) error
buf.WriteString("describetable")
v.rewrittenQuery = buf.String()

case *sqlparser.DescribeMethod:
buf.WriteString("describemethod")
v.rewrittenQuery = buf.String()

case *sqlparser.OtherAdmin:
buf.WriteString("otheradmin")
v.rewrittenQuery = buf.String()
Expand Down
4 changes: 4 additions & 0 deletions internal/stackql/astvisit/from_rewrite.go
Original file line number Diff line number Diff line change
Expand Up @@ -564,6 +564,10 @@ func (v *standardFromRewriteAstVisitor) Visit(node sqlparser.SQLNode) error {
buf.WriteString("describetable")
v.rewrittenQuery = buf.String()

case *sqlparser.DescribeMethod:
buf.WriteString("describemethod")
v.rewrittenQuery = buf.String()

case *sqlparser.OtherAdmin:
buf.WriteString("otheradmin")
v.rewrittenQuery = buf.String()
Expand Down
2 changes: 2 additions & 0 deletions internal/stackql/astvisit/internally_routable_typing.go
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,8 @@ func (v *standardInternallyRoutableTypingAstVisitor) Visit(node sqlparser.SQLNod

case *sqlparser.DescribeTable:

case *sqlparser.DescribeMethod:

case *sqlparser.OtherAdmin:

case sqlparser.Comments:
Expand Down
3 changes: 3 additions & 0 deletions internal/stackql/astvisit/param_extract.go
Original file line number Diff line number Diff line change
Expand Up @@ -541,6 +541,9 @@ func (v *standardParserParamAstVisitor) Visit(node sqlparser.SQLNode) error {
case *sqlparser.DescribeTable:
buf.WriteString("describetable")

case *sqlparser.DescribeMethod:
buf.WriteString("describemethod")

case *sqlparser.OtherAdmin:
buf.WriteString("otheradmin")

Expand Down
3 changes: 3 additions & 0 deletions internal/stackql/astvisit/placeholder_param_extract.go
Original file line number Diff line number Diff line change
Expand Up @@ -558,6 +558,9 @@ func (v *standardParserPlaceholderParamAstVisitor) Visit(node sqlparser.SQLNode)
case *sqlparser.DescribeTable:
buf.WriteString("describetable")

case *sqlparser.DescribeMethod:
buf.WriteString("describemethod")

case *sqlparser.OtherAdmin:
buf.WriteString("otheradmin")

Expand Down
4 changes: 4 additions & 0 deletions internal/stackql/astvisit/provider_string_extract.go
Original file line number Diff line number Diff line change
Expand Up @@ -585,6 +585,10 @@ func (v *standardProviderStringAstVisitor) Visit(node sqlparser.SQLNode) error {
buf.WriteString("describetable")
// v.rewrittenQuery = buf.String()

case *sqlparser.DescribeMethod:
buf.WriteString("describemethod")
// v.rewrittenQuery = buf.String()

case *sqlparser.OtherAdmin:
buf.WriteString("otheradmin")
// v.rewrittenQuery = buf.String()
Expand Down
2 changes: 2 additions & 0 deletions internal/stackql/astvisit/query_rewriting.go
Original file line number Diff line number Diff line change
Expand Up @@ -614,6 +614,8 @@ func (v *standardQueryRewriteAstVisitor) Visit(node sqlparser.SQLNode) error {

case *sqlparser.DescribeTable:

case *sqlparser.DescribeMethod:

case *sqlparser.OtherAdmin:

case sqlparser.Comments:
Expand Down
2 changes: 2 additions & 0 deletions internal/stackql/astvisit/table_alias_pairing.go
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,8 @@ func (v *standardParserTableAliasPairingAstVisitor) Visit(node sqlparser.SQLNode

case *sqlparser.DescribeTable:

case *sqlparser.DescribeMethod:

case *sqlparser.OtherAdmin:

case sqlparser.Comments:
Expand Down
2 changes: 2 additions & 0 deletions internal/stackql/astvisit/table_extract.go
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,8 @@ func (v *standardParserTableExtractAstVisitor) Visit(node sqlparser.SQLNode) err

case *sqlparser.DescribeTable:

case *sqlparser.DescribeMethod:

case *sqlparser.OtherAdmin:

case sqlparser.Comments:
Expand Down
81 changes: 81 additions & 0 deletions internal/stackql/buildinfo/buildinfo.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
// Package buildinfo holds build-time identifiers populated at link time.
// Keeping these in a leaf package lets both the CLI command layer and the
// runtime primitives reference them without creating an import cycle.
//
// The public surface is intentionally minimal: a BuildInfo interface, a
// constructor, and a process-wide accessor. The implementation is a private
// struct that is constructed exactly once, at program start, from cmd/root.go.
package buildinfo

import (
"fmt"
"sync"
)

// BuildInfo provides read-only access to build-time identifiers.
type BuildInfo interface {
GetMajorVersion() string
GetMinorVersion() string
GetPatchVersion() string
GetCommitSHA() string
GetShortCommitSHA() string
GetDate() string
GetPlatform() string
GetSemVersion() string
}

type buildInfo struct {
majorVersion string
minorVersion string
patchVersion string
commitSHA string
shortCommitSHA string
date string
platform string
semVersion string
}

// NewBuildInfo constructs an immutable BuildInfo from the given identifiers.
// SemVersion is derived from the major/minor/patch trio.
func NewBuildInfo(major, minor, patch, commitSHA, shortCommitSHA, date, platform string) BuildInfo {
return &buildInfo{
majorVersion: major,
minorVersion: minor,
patchVersion: patch,
commitSHA: commitSHA,
shortCommitSHA: shortCommitSHA,
date: date,
platform: platform,
semVersion: fmt.Sprintf("%s.%s.%s", major, minor, patch),
}
}

func (b *buildInfo) GetMajorVersion() string { return b.majorVersion }
func (b *buildInfo) GetMinorVersion() string { return b.minorVersion }
func (b *buildInfo) GetPatchVersion() string { return b.patchVersion }
func (b *buildInfo) GetCommitSHA() string { return b.commitSHA }
func (b *buildInfo) GetShortCommitSHA() string { return b.shortCommitSHA }
func (b *buildInfo) GetDate() string { return b.date }
func (b *buildInfo) GetPlatform() string { return b.platform }
func (b *buildInfo) GetSemVersion() string { return b.semVersion }

//nolint:gochecknoglobals // process-wide singleton; written exactly once via Init.
var (
singleton = NewBuildInfo("", "", "", "", "", "", "")
once sync.Once
)

// Init publishes the process-wide BuildInfo. It is intended to be called once
// from cmd/root.go's init(), after the -ldflags-populated string variables in
// the cmd package have been observed. Subsequent calls are no-ops.
func Init(major, minor, patch, commitSHA, shortCommitSHA, date, platform string) {
once.Do(func() {
singleton = NewBuildInfo(major, minor, patch, commitSHA, shortCommitSHA, date, platform)
})
}

// Get returns the process-wide BuildInfo. Before Init is called it returns an
// instance with empty strings.
func Get() BuildInfo {
return singleton
}
7 changes: 7 additions & 0 deletions internal/stackql/cmd/mcp.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (
"github.com/stackql/any-sdk/pkg/dto"
"github.com/stackql/any-sdk/pkg/logging"
"github.com/stackql/stackql/internal/stackql/acid/tsm_physio"
"github.com/stackql/stackql/internal/stackql/buildinfo"
"github.com/stackql/stackql/internal/stackql/entryutil"
"github.com/stackql/stackql/internal/stackql/handler"
"github.com/stackql/stackql/internal/stackql/iqlerror"
Expand Down Expand Up @@ -74,6 +75,8 @@ func runMCPServer(handlerCtx handler.HandlerContext) {
if config.Server.IsReadOnly != nil {
isReadOnly = *config.Server.IsReadOnly
}
bi := buildinfo.Get()
transport := config.Server.Transport
var backend mcp_server.Backend
var backendErr error
if mcpServerType == "reverse_proxy" {
Expand All @@ -91,6 +94,8 @@ func runMCPServer(handlerCtx handler.HandlerContext) {
db,
handlerCtx,
logging.GetLogger(),
bi,
transport,
)
iqlerror.PrintErrorAndExitOneIfError(backendErr)
} else {
Expand All @@ -103,6 +108,8 @@ func runMCPServer(handlerCtx handler.HandlerContext) {
orchestrator,
handlerCtx,
logging.GetLogger(),
bi,
transport,
)
iqlerror.PrintErrorAndExitOneIfError(backendErr)
iqlerror.PrintErrorAndExitOneIfNil(backend, "mcp backend is unexpectedly nil")
Expand Down
Loading
Loading