DATA AUTOPSY

Machine learning models rarely fail in production because of bad algorithms. They fail from silent tabular defects that inflate validation scores before collapsing upon deployment:

• Post-Event Target Leakage: Features recorded downstream of the target outcome (e.g. readmission disposition codes) that give models 99% validation accuracy that vanishes in live environments.
• Contaminated Splits: Undetected duplicate rows leaking across train/test partitions, causing models to memorize records rather than generalize.
• Disguised Identifiers: High-cardinality keys and surrogate IDs passed as features, leading tree-based models to overfit on noise.
• Masked Class Imbalance: Severe 30:1 class skew obscured by naive uncalibrated accuracy metrics.
• Misclassified Outliers: Legitimate physical extremes (e.g. server network spikes) incorrectly scrubbed as data corruption.

flowchart LR
    subgraph Traditional["Traditional Tabular Profilers"]
        direction LR
        A1["Raw Tabular CSV"] --> B1["Shallow Histograms"]
        B1 --> C1["Uncalibrated Metrics"]
        C1 --> D1["Silent Production Collapse"]
    end

    subgraph Forensic["Data Autopsy Workstation"]
        direction LR
        A2["Raw Tabular CSV"] --> B2["10 Deterministic Engines"]
        B2 --> C2["0-Indexed Row Audit Trail"]
        C2 --> D2["Scikit-Learn Pipeline Ablation"]
        D2 --> E2["Verifiable Mathematical Proof"]
    end

• 10 Deterministic Forensic Engines: 100% mathematically reproducible anomaly detection using Tukey IQR, modified Z-scores, Pearson correlations, ANOVA effect sizes, and mutual information. Zero hallucinations.
• Verifiable Mathematical Evidence: Every single alert links directly to exact original 0-indexed row locations and verbatim observed sample tuples.
• Leakage Pipeline Ablation: Automated Scikit-Learn evaluation trains two isolated pipelines (baseline vs ablated) with preprocessing fitted strictly on training splits to prove real-world metric drop.
• Non-Mutating Column Exclusion: Interactively drop columns from subsequent forensic analysis without modifying the source dataset on disk.
• Sovereign Local AI (Ollama): Optional plain-English explanations using local open-weight qwen2.5:1.5b or qwen2.5:0.5b. No data ever leaves your computer. Includes 100% offline deterministic fallback.
• Exportable Forensic Dossiers: Download comprehensive audit reports in standardized JSON and Markdown formats for compliance and code review.

Instant cryptographic verification (SHA-256), payload profiling, runtime benchmarks, and severity-tiered finding summaries.

Deep column structural analysis: inferred statistical types, exact missingness counts, unique ratios, memory consumption, and interactive Plotly distribution charts.

Deterministically ranked findings sorted by composite severity and sample-calibrated empirical support. Filter by severity tier or check category.

Detects exact target copies (LEAK-001), inverted binary targets (LEAK-005), and categorical encodings (LEAK-004). Trains dual Scikit-Learn pipelines to quantify accuracy collapse when suspicious columns are ablated.

Forensic evidence viewer showing exact 0-indexed row numbers, raw values, and mathematical detection thresholds directly from the source table.

Sovereign AI interpretations generated locally on your workstation. Strictly explains verified findings without running calculations, inventing numbers, or sending telemetry to external clouds.

One-click download of audit dossiers in Markdown and JSON formats, complete with metadata, baseline scores, and mathematical limitation boundaries.

Data Autopsy executes 10 reproducible checks with explicit mathematical boundaries:

• Python: 3.11 or higher
• RAM: Minimum 4 GB (8 GB recommended)
• OS: macOS (Apple Silicon or Intel), Linux, or Windows

# Clone the repository and enter directory
cd data-autopsy

# Create and activate virtual environment
python3.11 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

streamlit run app.py

Open http://localhost:8501 in your browser.

Generate all 6 synthetic demo datasets deterministically (seed=42):

python scripts/generate_demo_datasets.py

Data Autopsy features strict hardware protection: heavy 7B/14B+ models are blocked to preserve host memory. Only lightweight, responsive models are allowed:

# 1. Install Ollama from https://ollama.com
# 2. Pull the default lightweight model
ollama pull qwen2.5:1.5b

# 3. Start Ollama
ollama run qwen2.5:1.5b

To use the ultra-lightweight 0.5B model instead:

ollama pull qwen2.5:0.5b
export QWEN_MODEL=qwen2.5:0.5b

Note: The application is 100% functional without Ollama. When offline, deterministic rule-based forensic explanations are displayed automatically. Models are never downloaded without explicit user initiation.

• Zero Cloud Exfiltration: All statistical computation, model training, and chart rendering execute locally within your Python runtime.
• No Remote Telemetry: No user analytics, tracking cookies, or dataset rows are sent over the network.
• Bounded Local Prompts: When the local AI is enabled, only compact mathematical summaries (rule ID, affected columns, observed metrics) are shared with localhost:11434. Raw data rows are never transmitted.

Run the complete test suite (73 automated tests):

pytest -v

Execute specific test modules:

# Verify demonstration datasets
pytest -v tests/test_demo_datasets.py

# Verify target leakage & pipeline ablation
pytest -v tests/test_leakage_investigation.py

# Verify evidence integrity & deterministic ranking
pytest -v tests/test_evidence_and_ranking.py

============================== 73 passed in 1.41s ==============================

flowchart TD
    subgraph Ingest["1. Untrusted Ingestion Layer"]
        RAW["Raw CSV / TSV File"] --> SNIFF["Encoding & Delimiter Sniffer"]
        SNIFF --> SEC["Safety Bounds & File Size Check"]
        SEC --> HASH["SHA-256 Cryptographic Hash"]
    end

    subgraph Core["2. Deterministic Forensic Core"]
        HASH --> D1["Null Topology & Duplicates (INT, DUP)"]
        HASH --> D2["Distributions & Outliers (STAT, OUT)"]
        HASH --> D3["Cardinality & Encodings (CARD, CAT)"]
        HASH --> D4["Target Leakage & Imbalance (LEAK, IMB, CORR)"]
    end

    subgraph Validation["3. Evidence Verification & Ablation"]
        D4 --> ABL["Scikit-Learn Pipeline Baseline Ablation"]
        D1 & D2 & D3 & D4 --> EVID["0-Indexed Row-Level Evidence Inspector"]
        D1 & D2 & D3 & D4 --> RANK["Deterministic Severity Hierarchy"]
    end

    subgraph UserInterface["4. Local Presentation & Export"]
        RANK --> UI["Streamlit 7-Tab Workstation"]
        RANK --> AI["Sovereign Ollama Qwen 2.5 Explainer"]
        EVID & ABL & RANK --> DOSSIER["Exportable Forensic Dossiers (JSON & Markdown)"]
    end

.
├── app.py                             # Streamlit forensic workstation interface
├── config.py                          # Application configuration & safety bounds
├── requirements.txt                   # Stable pinned dependencies (Python 3.11+)
├── README.md                          # Comprehensive system manual & visual tour
├── demo/
│   └── DEMO_SCRIPT.md                 # 90-second competition demonstration script
├── docs/
│   └── images/                        # Workstation screenshots (Tabs 1-7)
├── engine/
│   ├── loader.py                      # Safe CSV parser, SHA-256 hasher & encoding sniffer
│   ├── models.py                      # Finding, Evidence, Profile & Report data models
│   ├── detectors.py                   # 10 deterministic anomaly detectors
│   ├── profiler.py                    # Column & dataset structural profiler
│   ├── ranking.py                     # Deterministic composite ranking hierarchy
│   ├── evaluator.py                   # Scikit-learn pipeline baseline & leakage ablation
│   ├── validator.py                   # Mathematical audit verification against raw data
│   ├── limitations.py                 # Forensic scope & boundary catalog
│   └── utils.py                       # Hashing, byte formatters & statistical helpers
├── ai/
│   └── explainer.py                   # Local Ollama Qwen 2.5 explainer with offline fallback
├── reports/
│   └── exporter.py                    # JSON and Markdown forensic dossier exporters
├── sample_data/                       # 6 reproducible demo datasets + 2 baseline fixtures
├── scripts/
│   ├── generate_demo_datasets.py      # Deterministic data generator (seed=42)
│   └── generate_readme_screenshots.py # Automated headless CDP screenshot generator
└── tests/                             # 8 test suites with 73 test cases

• Code of Conduct: Please review our Code of Conduct before participating in this project.
• Contributing: Read our Contributing Guidelines to set up your environment, write detectors, and submit pull requests.
• Security Policy: View our Security Policy for vulnerability disclosure guidelines and sovereign privacy guarantees.

This project is licensed under the MIT License — see the LICENSE file for details.