A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
paulpierre/markdown-crawler has 455 GitHub stars.
It currently ranks #29917 daily, #30340 weekly, #32043 monthly by momentum on Trending Repos; its best-ever rank is #5054.