LLMs.txt Generator

LLMs.txt Generator is a Python-based tool that automates the creation of AI-friendly documentation files (llms.txt and llms-full.txt). It crawls a given website, extracts key documentation pages, converts them to Markdown, and generates structured index files to improve retrieval by Large Language Models (LLMs).

• Automated Crawling: Recursively scans a website and extracts relevant documentation pages.
• Markdown Conversion: Converts HTML content into clean, AI-readable Markdown format.
• Structured Documentation Indexing: Generates llms.txt and llms-full.txt for AI-friendly search and retrieval.
• Configurable Parameters: Allows setting ignore paths, request delays, and custom output directories.

# Clone the repository
git clone https://github.com/yourusername/llms-txt-generator.git
cd llms-txt-generator

# Install dependencies
pip install -r requirements.txt

python llms_generator.py <URL> [-o OUTPUT_DIR] [--ignore PATHS] [--delay SECONDS]

python llms_generator.py https://example.com/docs -o ./output --ignore /private /admin --delay 1.5

• llms.txt: A structured index of key documentation pages.
• llms-full.txt: The full documentation in Markdown format.
• markdown/: Directory containing converted Markdown files for individual pages.

# example.com
> AI-friendly documentation generated by LLMs.txt Generator

## Core Documentation
- [Getting Started](https://example.com/getting-started.md): Introduction to the platform.
- [API Reference](https://example.com/api-reference.md): Detailed API documentation.

## Optional
- [Blog](https://example.com/blog.md): Additional insights and articles.

Contributions are welcome! To contribute:

1. Fork the repository.
2. Create a feature branch.
3. Commit your changes.
4. Submit a pull request.

This project is licensed under the MIT License.

• Inspired by Jeremy Howard’s LLMs.txt proposal.
• Uses BeautifulSoup and markdownify for web scraping and conversion.

🚀 Enhance AI-driven searchability of your docs with LLMs.txt Generator!