🕷️ deepscrape

• 🕷️ deepscrape
  • The AI‑powered web scraping & crawl‑orchestration API for AI agents 🤖
  • Table of Contents
  • 📸 Product preview
  • 🤔 What is deepscrape?
  • ✨ Highlights
  • 🧩 What it does for you
  • 🧱 Tech stack
  • 🏗️ Architecture
    • Repository layout
    • Request lifecycle (SSR)
    • Functions domain map
  • ⚡ Prerequisites
  • 🚀 Getting started
  • 🌍 Environment variables
    • Root .env (app)
    • functions/.env.local (backend)
  • 🧰 Available scripts
    • App (root)
    • Functions (functions/)
  • 🧪 Testing
  • 🚢 Deployment
  • 🛟 Troubleshooting
  • 🤝 Contributing
  • 📄 License

_{🦾 Web data at lightning speed — the scraping API for AI agents.}

_{🔎 A live research agent that browses the web and answers with structured data.}

deepscrape is what runs on deepscrape.dev — a complete web‑extraction platform, not just a scraper. Point it at any site and it:

1. 🖥️ Renders the page like a real browser — handles JavaScript‑heavy SPAs.
2. 🛡️ Defeats proxies & bot protection — rotates proxies and navigates anti‑bot walls.
3. 🧹 Extracts clean, structured data — messy HTML becomes the fields, rows and JSON you want.
4. 🤖 Hands it to the AI you prefer — Claude, GPT, Groq, Google, Jina, Fireworks, OpenRouter, Crawl4AI and any OpenAI‑compatible endpoint.

This repository is the full source: the Angular app, the SSR server that renders it, the API layer that talks to AI + crawl providers, and the Firebase backend.

• Scrapes the hard stuff 🪨 — real browser rendering, proxy rotation and anti‑bot handling, so pages that block plain HTTP clients still come through.
• Extracts clean data 🧼 — structured extraction turns messy HTML into the fields, rows and JSON you actually want.
• Works with any AI 🔌 — a multi‑provider platform. Route scraping results to Anthropic (Claude), OpenAI (GPT), Groq, Google, Jina, Fireworks, OpenRouter, Crawl4AI — or any OpenAI‑compatible endpoint. No lock‑in.
• Answers, not just pages 💬 — ask in plain language and a live research agent searches, navigates and reads for you, returning structured answers to your dashboard.
• Your infrastructure, your way 🏗️ — deploy your own crawls with CrawlPack machines, watch everything from the dashboard, and manage billing, plans and API keys in one place.
• Fast & reliable by default ⚡ — Angular 20 zoneless + SSR with incremental hydration, lazy routes, and prerendered marketing pages for instant first paint and SEO.

deepscrape/
├── src/app/                  # Angular app (standalone, zoneless)
│   ├── routes/               # main · service · user route files (lazy)
│   ├── pages/                # home, landpage, admin, billing, user…
│   ├── layout/               # chrome, landpage sections (hero, code-demo…)
│   └── core/                 # services, guards, shared components
├── functions/                # Firebase backend (BFF + ~60 functions)
│   └── src/
│       ├── app/              # Express app, auth, stripe
│       ├── config/           # Firebase / env / secret config
│       ├── domain/           # business logic (billing, analytics, global)
│       ├── gfunctions/       # callable & scheduled functions
│       ├── handlers/         # route handlers
│       ├── infrastructure/   # external clients
│       ├── index.ts          # function exports
│       └── server.ts         # Express BFF (deepscrape onRequest)
├── api/                      # Elysia SyncAI API
├── public/                   # static hosting assets (icons, 404…)
├── src/styles.scss           # global design tokens / Tailwind
└── angular.json · firebase.json · package.json

flowchart LR
  U[Browser] --> H[Firebase Hosting]
  H -->|static / prerendered| F[dist browser]
  H -->|dynamic route rewrite| FN[deepscrape function]
  FN -->|engine render| E[Angular SSR runtime]
  FN -->|API calls| BFF[Express BFF / proxies]
  BFF --> AI[(Anthropic/OpenAI/Groq/Crawl4AI)]
  FN --> DB[(Firestore)]

• Prerendered routes (marketing pages) ship full static HTML from Hosting.
• Dynamic routes (admin/**, settings/**, …) are engine‑rendered at request time by the deepscrape function via @angular/ssr (CommonEngine) when the SSR runtime is packaged — with a graceful CSR fallback. Disable with DEEPSCRAPE_DYNAMIC_SSR=false.
• Hydration uses incremental hydration + event replay for fast, interactive first paint.

Business logic lives in domain/, entry points in gfunctions/, and only entry points are exported from functions/src/index.ts. New functions follow: gfunctions/<name>.ts → export in index.ts → business logic in domain/<domain>/.

• Bun (recommended) or Node.js 20+
• Firebase CLI (firebase-tools)
• npm (for functions/)
• A Firebase project + (for live features) Stripe, Upstash Redis, and AI‑provider API keys
• Java (only for local Firebase Firestore emulator on some OSes)

# 1. Clone
git clone https://github.com/deepscrape/deepscrape.git
cd deepscrape

# 2. Install dependencies (Bun for the app, npm for functions)
bun install
cd functions && npm install && cd ..

# 3. Configure environment (dotenvx — .env files are NOT committed)
cp .env.example .env                  # add your keys
cp functions/.env.example functions/.env.local

# 4. Generate environment.ts + start the dev server
bun run prbuild
bun run dev

Open http://localhost:4200 🎉

Want the full stack (auth, Firestore, hosting) locally with Firebase emulators?

bun run serve

Production build & deploy:

bun run build     # production build (prerenders static routes)
bun run deploy    # build + deploy hosting & functions

💡 src/environments/environment.ts is generated — never hand‑edit it. Change .env / prod_gen.ts, then run bun run prebuild.

🔒 Secrets are never committed. Keep them in dotenv files locally and in Google Secret Manager for production function deploys.

bun run test       # app unit tests (Karma/Jasmine)
bun run test:ci    # headless CI run
cd functions && npm run test   # backend tests (tsx)

Tests live next to their code (*.spec.ts). CI runs the app suite headless and lints functions before every deploy.

Stack: Firebase Hosting (static + prerendered app) · Firebase Functions (Gen2, Node 22) · Secret Manager · CI via GitHub Actions.

# 1. Build the app (prerenders static routes)
bun run build

# 2. Build functions — packages the Angular SSR runtime into functions/lib
cd functions && npm run build && cd ..

# 3. Deploy
firebase deploy --only hosting           # static / prerendered app
firebase deploy --only functions:deepscrape   # SSR BFF (engine render)

CI/CD — GitHub Actions workflows:

• firebase-hosting-merge.yml → builds & deploys hosting + functions on merge to main/next.
• firebase-hosting-pull-request.yml → per‑PR hosting preview channels (like the staging previews) + functions analysis.
• test.yml → runs the Angular test suite.

Release process — semantic-release runs on push to main/next. It writes CHANGELOG.md, bumps package.json, and opens a chore/release-assets-* PR. Commit with git cz (commitlint enforces conventional commits).

⚠️ Production function deploy requires billing enabled on the project and the FUNCTIONS_ENV_JSON secret accessible in Secret Manager — otherwise the deploy fails with 403 requires billing.

1. Branch from main (or target next for pre‑releases).
2. Commit with git cz — conventional commits (commitlint + husky enforce it).
3. Open a PR. CI runs tests, lint, and (on merge) semantic‑release:
   • feat → minor, fix/perf/refactor → patch, BREAKING CHANGE: → major.
4. Keep commit bodies ≤ 100 chars/line (commitlint).

Code style: Angular 20 standalone + zoneless + inject() + OnPush; lazy‑load every page route; never hand‑edit generated environment.ts.

MIT © Deepscrape

Made with 🖤 · Fast, reliable AI web extraction — deepscrape.dev