Search

Scrapy, a fast high-level web crawling & scraping framework for Python.

A next-generation crawling and spidering framework.

Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.

Blackfire Player is a powerful Web Crawling, Web Testing, and Web Scraper application. It provides a nice DSL to crawl HTTP services, assert responses, and extract data from HTML/XML/JSON responses.

A simple, fast and reliable Coursera crawling & downloading tool

One-stop Proxies Crawling and Aggregation Platform

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

Golang Crawling and scraping framework

DataHarvest is a tool for data search 🔍, crawling 🕷, and cleaning 🧽.

Official Search1API MCP server for web search, news, crawling, sitemaps, and trends—hosted with OAuth 2.1 or local via npm.

Exa MCP for web search and web crawling!

EmailCrawl - is a professional-grade OSINT (Open Source Intelligence) tool designed for advanced email address extraction through comprehensive web crawling. Built for cybersecurity professionals, penetration testers, and intelligence analysts.

Firecrawl plugin for Claude Code - web scraping and crawling

Open-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key. Firecrawl-compatible API planned.

Browser automation, web crawling, and iOS + Android device control for AI agents. Zig-native, token-efficient CDP snapshots, HAR recording, native adb wire-protocol client, and a standalone fetcher.