Search

The headless browser for AI agents and web scraping

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

Scrapy, a fast high-level web crawling & scraping framework for Python.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.

An API that returns the number of GitHub contributions by scraping a user's GitHub profile

Selenium that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.

Git scraping of AT Protocol/Bluesky instances

Scraping the TikTok discovery web API every 15 minutes using Github Actions to view changes

🕸 Turn almost any website into an RSS feed and serve it instantly. Automatic scraping. Start fast with Docker.

Take the hassle out of web scraping

Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.

A PHP library/toolkit designed to handle all of your web scraping needs under a MIT or LGPL license. Also has web server and WebSocket server classes for building custom servers.

Browser automation, web scraping, E2E-testing, and stealth in one SDK. SeleniumBase CDP Mode can evade bot-detection and pass CAPTCHAs.

📚🇵🇱 A free, open-source utility for scraping data from the Synergia Librus (http://synergia.librus.pl/) electronic gradebook. Developed and maintained by the community, it allows easy access to grades, attendance, messages, and more - completely free of charge.

pagodo (Passive Google Dork) - Automate Google Hacking Database scraping and searching

📘 Get all Warframe items directly from Warframe's API. No more messy wikia scraping.

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

Universal Reddit Scraper - A comprehensive Reddit scraping/archival command-line tool.

Fast, flexible, sync/async, Python 3.10+ screen scraping client specifically for network devices

Dorks Eye Google Hacking Dork Scraping and Searching Script. Dorks Eye is a script I made in python 3. With this tool, you can easily find Google Dorks. Dork Eye collects potentially vulnerable web pages and applications on the Internet or other awesome info that is picked up by Google's search bots. Author: Jolanda de Koff

Anti-detect browser for web scraping and automation. Engine-level fingerprint masking for Chromium and Firefox. Self-hosted, Docker-ready. Integrates with Selenium, Playwright, and Puppeteer via SDKs in Python, JavaScript, and C#.

Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

Adaptive Python web scraping toolkit + MCP server for AI agents. Self-healing selectors that survive site changes, TLS-fingerprint stealth to bypass anti-bot filters, CSS/XPath parsing, and 24 built-in scrapers, clean, structured, LLM-ready data from any URL.

Scraping Trends Keywords from Google ( All Country)

estela, an elastic web scraping cluster 🕸

A simple program for scraping Cloudflare clearance (cf_clearance) cookies from websites issuing Cloudflare challenges to visitors