Find trending repositories by name or description.
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Shattered Pixel Dungeon is an open-source traditional roguelike dungeon crawler with randomized levels and enemies, and hundreds of items to collect and use. It's based on the source code of Pixel Dungeon, by Watabou.
Elegant Scraper and Crawler Framework for Golang
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
Crawler (Bot) searching for leaks on paste sites.
Apache Nutch is an extensible and scalable web crawler
Eases DOM navigation for HTML and XML documents
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
A scalable, mature and versatile web crawler based on Apache Storm
brozzler - distributed browser-based web crawler
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
A high performance web crawler / scraper in Elixir.
Google search results crawler, get google search results that you need
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
A lightweight web crawler framework.（Java爬虫框架）
Generate and download e-books from online sources.
Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (ISML=international saimoe; 2022 ISML is last ISML)
新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频
State-of-the-art web crawler 🔱
Tracks changes to Telegram's sites, beta apps and servers
A very simple news crawler with a funny name
Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀
SpiderSuite (web security crawler) releases, wiki and roadmap
A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!
A local Apple Documentation crawler and MCP server. Written in Swift.
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.