Engineering a Scalable llms.txt Generator with Semantic Clustering
This technical article details the engineering architecture behind a scalable generator for llms.txt files, designed to create structured hierarchies of Markdown documents rather than flat summary indexes. The system processes websites through a five-stage pipeline: crawling, embedding, clustering, summarizing, and file generation. During the crawling phase, content is extracted while handling failures gracefully without stopping the pipeline. The embedding stage utilizes Google's Gemini model to convert pages into vectors, employing Redis caching to optimize costs and speed by avoiding redundant API calls. A key innovation is the use of K-Means clustering with cosine similarity, implemented directly in TypeScript without external machine learning libraries. This approach groups pages by semantic meaning rather than URL structure, ensuring that topically related content is merged into coherent documents regardless of its location on the site. The author emphasizes the importance of using cosine similarity for high-dimensional vectors and highlights how each stage operates at different speeds with distinct failure modes, providing a robust solution for automated documentation generation at scale.
Wire timeline
Engineering a Scalable llms.txt Generator with Semantic Clustering
This technical article details the engineering architecture behind a scalable generator for llms.txt files, designed to create structured hierarchies of Markdown documents rather than flat summary indexes. The system processes websites through a five-stage pipeline: crawling, embedding, clustering, summarizing, and file generation. During the crawling phase, content is extracted while handling failures gracefully without stopping the pipeline. The embedding stage utilizes Google's Gemini model to convert pages into vectors, employing Redis caching to optimize costs and speed by avoiding redundant API calls. A key innovation is the use of K-Means clustering with cosine similarity, implemented directly in TypeScript without external machine learning libraries. This approach groups pages by semantic meaning rather than URL structure, ensuring that topically related content is merged into coherent documents regardless of its location on the site. The author emphasizes the importance of using cosine similarity for high-dimensional vectors and highlights how each stage operates at different speeds with distinct failure modes, providing a robust solution for automated documentation generation at scale.
DEV Community