feat: implement secret scanner, redaction utility, and update docs #1313 - #1321
Conversation
|
@DakshtaMethwani is attempting to deploy a commit to the shashankss1205's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
👋 Thanks for contributing to CodeGraphContext! Since this PR was opened, |
|
Triage update: secret scanning/redaction before graph storage (#1313) is still a real gap we want closed. This branch now conflicts with main's indexing pipeline; @DakshtaMethwani would you be able to rebase it onto current main? One ask for the update: keep the scanner opt-out-able via config and make sure redaction happens before any backend write, not after. Happy to review quickly once it's up to date! 🙏 |
writer.py lives at codegraphcontext/tools/indexing/persistence/, so `from ...cli.config_manager` resolves to codegraphcontext.tools.cli, which does not exist. Every add_file_to_graph call raised ModuleNotFoundError, breaking indexing entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shashankss1205
left a comment
There was a problem hiding this comment.
Nice feature — verified end-to-end on a fixture with an AWS key, a GitHub PAT and a Postgres connection string:
REDACT_SECRETS=true → AWS_KEY/GITHUB_TOKEN/DB_URL = "[REDACTED]", NORMAL preserved
REDACT_SECRETS unset → all values stored verbatim (backward compatible)
I resolved the config_manager.py conflict (purely additive on both sides — your REDACT_SECRETS entries alongside main's OPENAI_MODEL/ANTHROPIC_MODEL/CGC_API_KEY) and pushed it to your branch.
I also had to push one fix: writer.py used from ...cli.config_manager, but writer.py sits at codegraphcontext/tools/indexing/persistence/, so three dots resolve to codegraphcontext.tools.cli — which doesn't exist. Every add_file_to_graph call raised ModuleNotFoundError, so indexing failed completely with the feature in place. Four dots (....cli.config_manager) is correct. Worth remembering that the writer path had no test exercising it, which is why this got through — a small test around add_file_to_graph with REDACT_SECRETS=true would have caught it.
Full unit suite green afterwards (1166 passed). Merging — thanks for the contribution, and reach out if you'd like to pick up more security work.
Pull Request Description
Overview
This PR addresses the security vulnerability outlined in Issue #1313. Previously, when CodeGraphContext (CGC) indexed a codebase, string literals containing plain-text secrets (API keys, tokens, passwords) were stored verbatim in the graph database (KuzuDB/Neo4j/FalkorDB) as node metadata. This posed a leakage risk if
.cgcbundles were shared or if an LLM accessed the graph via MCP.This implementation introduces runtime regex/entropy scanning, a warning mechanism, and an optional redaction system.
Proposed Changes
sk-,ghp_,Bearer,AWSkeys).CGC_REDACT_SECRETS. When active, any string matching a secret pattern is replaced with[REDACTED]prior to DB persistence.⚙️ Configuration
CGC_REDACT_SECRETSis set via environment variable and defaults tofalse, preserving existing behavior. SetCGC_REDACT_SECRETS=trueto enable redaction.Before / After
Before (redaction off — default):
{ "function_name": "fetchData", "api_key": "sk-abc123def456ghi789" } After (CGC_REDACT_SECRETS=true): { "function_name": "fetchData", "api_key": "[REDACTED]" } Key Files Modified - src/codegraphcontext/tools/graph_builder.py - Hooked into the node property assembly pipeline to scan string literal values before they are assigned to function/class metadata. - src/codegraphcontext/tools/indexing/persistence/writer.py - Integrated the CGC_REDACT_SECRETS config check to scrub values right before writing properties to the active graph database backend. - docs/ - Updated security practices and added a clear note to the .cgc export workflow documentation regarding shared bundles. Testing - Unit tests added for pattern matching regex (e.g., sk-, ghp_, Bearer, AWS key formats). - Unit tests added for high-entropy string detection. - Manually verified CGC_REDACT_SECRETS=true masks values via database node inspection. - Verified documentation builds correctly with updated security notices. Checklist - Closes #1313 - Added unit tests for pattern matching regex and high-entropy detection strings - Verified CGC_REDACT_SECRETS=true successfully masks values in database node inspections - Documentation builds correctly with updated security notices - No breaking changes — redaction is opt-in via environment variable