Skip to content

Add crawlberg to Web Scraping#3238

Merged
JinyangWang27 merged 1 commit into
vinta:masterfrom
Goldziher:add-crawlberg
Jul 5, 2026
Merged

Add crawlberg to Web Scraping#3238
JinyangWang27 merged 1 commit into
vinta:masterfrom
Goldziher:add-crawlberg

Conversation

@Goldziher

Copy link
Copy Markdown
Contributor

Add crawlberg

Category: Web Scraping → Frameworks

Single project, per the one-project-per-PR guideline.

Why it qualifies:

  • 120+ GitHub stars, MIT-licensed, and actively maintained (commits within the last few days).
  • Production-ready release on PyPI with a clear README containing installation and usage examples.
  • Distinct value: A Rust-core crawling engine (not a wrapper around requests/Playwright) with a headless-browser fallback and native robots.txt/sitemap handling, offering a different performance profile from the existing pure-Python entries.

It is a Python-first library (installable from PyPI) with a Rust core for performance — the same architecture as the already-listed xberg entry.

@JinyangWang27 JinyangWang27 merged commit 0c992c5 into vinta:master Jul 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants