Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs

Zhiyang Chen¹   Tara Saba¹   Xun Deng¹   Xujie Si¹   Fan Long¹

¹University of Toronto, Toronto, Ontario, Canada

Scam2Prompt is a research artifact for auditing whether production LLMs generate code that embeds malicious scam endpoints when prompted with ordinary developer-style programming requests.

This repository contains the end-to-end artifact for the paper "Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs". It includes URL collection, prompt synthesis, code generation, malicious URL detection, analysis, guardrail evaluation, and new-model validation tooling.

• End-to-end audit pipeline from curated scam infrastructure to developer-style prompts, generated code, oracle validation, and paper-table reproduction.
• Innoc2Scam-bench support with 1,377 benchmark prompts and validation tooling for newer production LLMs.
• Paper-ready reproducibility scripts for regenerating every final-version table from checked-in artifact data.
• Guardrail and mitigation experiments for evaluating post-generation defenses against malicious URL reproduction.

For questions about the artifact, contact Zhiyang Chen at zhiychen@cs.toronto.edu.

Scam2Prompt starts from known scam domains and URLs, safely gathers webpage context, synthesizes developer-style prompts, asks production LLMs to generate code, and checks whether the generated code contains malicious URLs. The artifact also includes analysis scripts for summarizing results, validating new models on Innoc2Scam-bench, and evaluating post-generation guardrail strategies.

Create an isolated Python environment and install the shared dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Copy the environment template and fill in only the providers you intend to use:

cp .env.example .env

Most scripts are designed to be run from the repository root. Individual package READMEs contain package-specific details when additional setup is needed.

1. Seed collection: load known scam URL/domain sources from scamDatabase/.
2. Safe browsing: fetch and cache target webpages with browsePackage/.
3. Prompt synthesis: generate developer-style prompts with promptGenPackage/.
4. Code generation: run target code-generation models through codegenPackage/.
5. Oracle validation: extract URLs from generated code and validate them through oraclePackage/.
6. Analysis and reporting: summarize malicious generations with malicious_urls_analysis/ and reportPackage/.
7. Model revalidation: apply Innoc2Scam-bench to newer models using validate_new_LLMs/.
8. Defense evaluation: test prompt and code guardrails using guardrailsPackage/.

Prompt generation is implemented in promptGenPackage/generate_prompts.py. The script reads scam URLs from scamDatabase/, normalizes them through browsePackage.process_html.load_scam_urls() and normalize_url(), then looks up previously crawled HTML in the optimized cache rooted at static/. For every cached, non-trivial page, it extracts visible text with extract_visible_text() and asks the configured prompt-generation LLM to produce short developer-style prompts that are tied to the page content without directly including the URL.

Run from the repository root:

python3 promptGenPackage/generate_prompts.py

The prompt-generation models are configured in the models_to_test list near the bottom of promptGenPackage/generate_prompts.py. The paper artifact uses prompt batches under model-specific directories such as:

Each prompts_batch_*.json file contains the source URL, prompt-generation model, visible-text length and preview, raw LLM response, parsed prompts, prompt count, timing, and success flag. Resume state is stored under the same model directory in staging/, so interrupted runs can be restarted with the same command. API clients are created through openaiPackage/openaiClient.py; use AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, and/or OPENROUTER_API_KEY variables in .env depending on the selected models.

These prompt batches are the direct inputs to the original cross-model code-generation experiments in codegenPackage/codeGenerator.py. They are also the provenance source for Table 1 and Table 11, reproduced by:

python3 codegenPackage/reproduce_tables_1_11.py

There are two testing paths in the artifact.

The original Scam2Prompt cross-product experiment is implemented in codegenPackage/codeGenerator.py. It reads prompt batches from promptGenPackage/<prompt_model>/results/, asks a configured code-generation model to produce code, extracts URLs from the generated code, checks them with oraclePackage.oracle.MaliciousURLOracle, and writes both complete generated code and malicious-code subsets.

Run from the repository root after selecting the desired (prompt_generation_model, code_generation_model) tuples in the experiments list near the bottom of codegenPackage/codeGenerator.py:

python3 codegenPackage/codeGenerator.py

Outputs are organized by prompt-generation and code-generation model:

For each prompt, the generator uses a fixed code-generation wrapper, temperature=0.0, top_p=1.0, and a deterministic seed derived from the prompt hash where the provider supports it. Each saved file includes metadata for the source prompt, original scam URL, extracted URLs, and oracle results. The analysis reports consumed by Table 1 are produced from these directories by codegenPackage/codeAnalyzer.py and codegenPackage/analyze_all_combinations.py; the checked-in final reports are then formatted by codegenPackage/reproduce_tables_1_11.py.

The final-paper Innoc2Scam-bench validation path is implemented in validate_new_LLMs/run_all_models_optimized.py, which calls validate_new_LLMs/filesystem_optimized_validation.py. This path tests newer production models against the benchmark prompt set assembled from:

Run from validate_new_LLMs/ so the category paths resolve as written:

cd validate_new_LLMs
python3 run_all_models_optimized.py

The model list is defined in run_all_models_optimized.py and currently includes Grok Code Fast, DeepSeek Chat v3.1, GPT-5, Qwen3-Coder, Gemini 2.5 Flash, Gemini 2.5 Pro, and Claude Sonnet 4. The validator reuses existing files in validation_results/<model>/ as a filesystem cache, retries failed requests, saves generated and malicious outputs separately, and writes comprehensive per-model summaries. Tables 2 and 6-10 are reproduced from the compact final status records with:

python3 validate_new_LLMs/reproduce_tables_2_6_7_8_9_10.py

The final paper tables can be regenerated from checked-in artifact data. Run commands from the repository root; outputs are written under each package's paper_tables/outputs/ directory.

Notes on provenance:

• validate_new_LLMs/paper_tables/status_records.json is a compact reduction of the final Innoc2Scam-bench result JSONs: prompt IDs, categories, and status sets are retained; generated code bodies are intentionally omitted.
• Tables 4-5 use the final paper's single-label domain-subcategory aggregate source because no per-prompt classifier output was present in the artifact snapshot.
• Table 14 writes both the exact paper table and table14_guardrails_audit.csv. The checked-in guardrail JSONs reproduce the model-level flagged totals; one DeepSeek flagged prompt has a category assignment discrepancy between the compact prompt map and the final paper layout, so the exact paper layout is preserved in table14.csv and the raw audit is kept separately.

This repository intentionally contains examples, prompts, generated code, and reports related to malicious scam infrastructure. Treat all domains, URLs, generated scripts, cached pages, and model outputs as unsafe research artifacts.

• Do not execute generated code unless you have inspected it and isolated the environment.
• Do not browse listed scam domains from a normal browser profile.
• Use throwaway credentials and controlled network environments for replication.
• Review SECURITY.md before reporting issues or handling suspicious outputs.

Innoc2Scam-bench is the public benchmark derived from Scam2Prompt. It contains 1,377 developer-style prompts and evaluation outputs for newer production LLMs.

• GitHub: https://github.com/Scam2Prompt/Innoc2Scam-bench-ICML26
• Hugging Face: https://huggingface.co/datasets/jeffchen006/Innoc2Scam-bench-ICML26

If you use this artifact or Innoc2Scam-bench, cite the accompanying paper:

@inproceedings{chen2026scam2prompt,
  title = {Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs},
  author = {Chen, Zhiyang and Saba, Tara and Deng, Xun and Si, Xujie and Long, Fan},
  booktitle = {International Conference on Machine Learning},
  year = {2026}
}

This artifact is released to support research reproducibility. Focused fixes to documentation, setup instructions, and reproducibility scripts are welcome. See CONTRIBUTING.md for the expected workflow.