Python: convert data extensions to JSON by nickrolfe · Pull Request #23076 · github/codeql

nickrolfe · 2026-11-10T13:28:10Z

CodeQL CLI 2.27.3 supports JSON data extensions (aka MaD models), and can load them significantly faster than YAML. This PR converts those in Python's library pack (first commit), along with $TEST.ext.yml files (second commit). The conversions were automated by an LLM-generated Python script (which I will add in a separate PR).

Although JSON is preferred going forward, python/ql/lib/qlpack.yml now has patterns that will pick up both .model.yml and .model.json files, so that any YAML files that do get checked in won't get silently dropped.

To verify successful conversion in the first commit, I cherry-picked it onto #23075, which has a temporary test that enumerates all tuples from data extensions. The test still passes after conversion.

For the second commit, I manually inspected the diffs.

Copilot

🟡 Changes recommended

The conversion still has an active generator-output mismatch and lacks DCA validation of downstream alert effects, so correctness and impact remain unresolved.

2 open issues

What changed in this PR

This pull request migrates Python CodeQL library and test data extensions from YAML to JSON, preserving the existing model data while enabling faster loading with CodeQL CLI 2.27.3.

Changes:

Converts Python framework, threat-model, and test extensions to JSON.
Updates pack discovery patterns to support both JSON and YAML extensions.
Updates identical-file metadata for the converted test extensions.

File	Description
python/ql/test/query-tests/Security/CWE-089-SqlInjection-local-threat-model/SqlInjection.ext.yml	Removes the YAML SQL injection threat-model test extension.
python/ql/test/query-tests/Security/CWE-089-SqlInjection-local-threat-model/SqlInjection.ext.json	Adds the equivalent JSON SQL injection threat-model test extension.
python/ql/test/modelling/FrameworkModeEndpoints.ext.yml	Removes the YAML framework endpoint modelling extension.
python/ql/test/modelling/FrameworkModeEndpoints.ext.json	Adds the equivalent JSON framework endpoint modelling extension.
python/ql/test/library-tests/frameworks/data/warnings.ext.yml	Removes the YAML framework warning-model test extension.
python/ql/test/library-tests/frameworks/data/warnings.ext.json	Adds the equivalent JSON framework warning-model test extension.
python/ql/test/library-tests/frameworks/data/test.ext.yml	Removes the YAML framework data test extension.
python/ql/test/library-tests/frameworks/data/test.ext.json	Adds the equivalent JSON framework data test extension.
python/ql/test/library-tests/dataflow/model-summaries/NormalDataflowTest.ext.yml	Removes the YAML normal dataflow model-summary extension.
python/ql/test/library-tests/dataflow/model-summaries/NormalDataflowTest.ext.json	Adds the equivalent JSON normal dataflow model-summary extension.
python/ql/test/library-tests/dataflow/model-summaries/InlineTaintTest.ext.yml	Removes the YAML inline-taint model-summary extension.
python/ql/test/library-tests/dataflow/model-summaries/InlineTaintTest.ext.json	Adds the equivalent JSON inline-taint model-summary extension.
python/ql/lib/semmle/python/frameworks/xml.model.yml	Removes the YAML XML framework model.
python/ql/lib/semmle/python/frameworks/xml.model.json	Adds the JSON XML framework model.
python/ql/lib/semmle/python/frameworks/Stdlib.model.yml	Removes the YAML standard-library models.
python/ql/lib/semmle/python/frameworks/Stdlib.model.json	Adds the JSON standard-library models.
python/ql/lib/semmle/python/frameworks/openai.model.yml	Removes the YAML OpenAI models.
python/ql/lib/semmle/python/frameworks/openai.model.json	Adds the JSON OpenAI models.
python/ql/lib/semmle/python/frameworks/lxml.model.yml	Removes the YAML lxml model.
python/ql/lib/semmle/python/frameworks/lxml.model.json	Adds the JSON lxml model.
python/ql/lib/semmle/python/frameworks/Gevent.model.yml	Removes the YAML Gevent model.
python/ql/lib/semmle/python/frameworks/Gevent.model.json	Adds the JSON Gevent model.
python/ql/lib/semmle/python/frameworks/Eventlet.model.yml	Removes the YAML Eventlet model.
python/ql/lib/semmle/python/frameworks/Eventlet.model.json	Adds the JSON Eventlet model.
python/ql/lib/semmle/python/frameworks/Django.model.yml	Removes the YAML Django model.
python/ql/lib/semmle/python/frameworks/Django.model.json	Adds the JSON Django model.
python/ql/lib/semmle/python/frameworks/data/internal/empty.model.yml	Removes the YAML empty internal model definitions.
python/ql/lib/semmle/python/frameworks/data/internal/empty.model.json	Adds the JSON empty internal model definitions.
python/ql/lib/semmle/python/frameworks/builtins.model.yml	Removes the generated YAML built-in exception model.
python/ql/lib/semmle/python/frameworks/builtins.model.json	Adds the generated JSON built-in exception model.
python/ql/lib/semmle/python/frameworks/Azure.Storage.model.yml	Removes the YAML Azure Storage models.
python/ql/lib/semmle/python/frameworks/Azure.Storage.model.json	Adds the JSON Azure Storage models.
python/ql/lib/semmle/python/frameworks/Azure.Keyvault.model.yml	Removes the YAML Azure Key Vault models.
python/ql/lib/semmle/python/frameworks/Azure.Keyvault.model.json	Adds the JSON Azure Key Vault models.
python/ql/lib/semmle/python/frameworks/Asyncpg.model.yml	Removes the YAML Asyncpg models.
python/ql/lib/semmle/python/frameworks/Asyncpg.model.json	Adds the JSON Asyncpg models.
python/ql/lib/semmle/python/frameworks/AntiSSRF.model.yml	Removes the YAML AntiSSRF model.
python/ql/lib/semmle/python/frameworks/AntiSSRF.model.json	Adds the JSON AntiSSRF model.
python/ql/lib/semmle/python/frameworks/anthropic.model.yml	Removes the YAML Anthropic models.
python/ql/lib/semmle/python/frameworks/anthropic.model.json	Adds the JSON Anthropic models.
python/ql/lib/semmle/python/frameworks/agent.model.yml	Removes the YAML agent model.
python/ql/lib/semmle/python/frameworks/agent.model.json	Adds the JSON agent model.
python/ql/lib/qlpack.yml	Adds JSON data-extension discovery patterns while retaining YAML compatibility.
python/ql/lib/ext/default-threat-models-fixup.model.yml	Removes the YAML default threat-model fixup.
python/ql/lib/ext/default-threat-models-fixup.model.json	Adds the JSON default threat-model fixup.
config/identical-files.json	Updates identical-file metadata to reference the converted JSON test extensions.

_{Review effort: Lite (auto)}

+  - semmle/python/frameworks/**/*.model.json
+  - ext/*.model.json


@@ -0,0 +1,257 @@
+// process-builtin-exceptions 0.0.1; Python 3.11.0rc2


owen-mc

Copilot brought up a good point about python/ql/src/meta/ClassHierarchy/process-builtin-exceptions.py (I think).

nickrolfe · 2026-11-12T11:00:09Z

I've pushed a commit to update process-builtin-exceptions.py. I've tested it manually, along with the split-model-files.py and join-model-files.py scripts, and they all seem to be working.

Python: convert data extensions to JSON

2bde17c

github-actions Bot added the Python label Nov 10, 2026

Python: convert test data extensions to JSON

497b803

nickrolfe force-pushed the nickrolfe/json-data-extensions-python branch from 6d8f897 to 497b803 Compare November 10, 2026 13:58

nickrolfe marked this pull request as ready for review November 10, 2026 14:37

nickrolfe requested review from a team as code owners November 10, 2026 14:38

Copilot AI auto review requested due to automatic review settings November 10, 2026 14:38

Copilot started reviewing on behalf of nickrolfe November 10, 2026 14:38

View session

nickrolfe added the no-change-note-required This PR does not need a change note label Nov 10, 2026

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread python/ql/lib/qlpack.yml

Comment on lines +21 to +22

- semmle/python/frameworks/**/*.model.json

- ext/*.model.json

Comment thread python/ql/lib/semmle/python/frameworks/builtins.model.json

@@ -0,0 +1,257 @@

// process-builtin-exceptions 0.0.1; Python 3.11.0rc2

nickrolfe mentioned this pull request Nov 11, 2026

MaD: add script to convert YAML data extensions to JSON #23102

Open

owen-mc requested changes Nov 11, 2026

View reviewed changes

Python: update model processing scripts to use JSON

88fd4d9

Navigation Menu

Repository navigation

Python: convert data extensions to JSON - #23076

Python: convert data extensions to JSON#23076
nickrolfe wants to merge 3 commits into
mainfrom
nickrolfe/json-data-extensions-python

nickrolfe commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

owen-mc left a comment

Uh oh!

nickrolfe commented Nov 12, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		- semmle/python/frameworks/*/.model.json
		- ext/*.model.json

		@@ -0,0 +1,257 @@
		// process-builtin-exceptions 0.0.1; Python 3.11.0rc2

Conversation

nickrolfe commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

owen-mc left a comment

Choose a reason for hiding this comment

Uh oh!

nickrolfe commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nickrolfe commented Nov 12, 2026 •

edited

Loading