Show HN: Llms.txt Generator – Turn websites into a text file to feed to any LLM https://bit.ly/3ZiXxWk

Show HN: Llms.txt Generator – Turn websites into a text file to feed to any LLM Hey HN! It’s Eric from Firecrawl ( https://bit.ly/4eI9IjK ). I just launched llms.txt Generator, a tool that transforms any website into a clean, structured text file optimized for feeding to LLMs. You can learn more about the standard at https://bit.ly/3Ziwc6J . Here’s how it works under the hood: 1. We use Firecrawl, our open-source scraper, to fetch the full site, handling JavaScript-heavy pages and complex structures. 2. The markdown content is parsed and then the title and description are extracted using GPT-4o-mini. 3. The everything is combined and the result is a lightweight llms.txt file that you can paste into any LLM. Let me know what you think! https://bit.ly/3V3thw8 November 21, 2024 at 08:33PM