README | 中文文档

This project crawls proxy ip resources from the Internet.What we wish is to provide a anonymous ip proxy pool with highly availability and low latency for distributed spiders.

• Distributed crawlers with high performance, powered by scrapy and redis
• Large-scale of proxy ip resources
• HA design for both crawlers and schedulers
• Flexible architecture with task routing
• Support HTTP/HTTPS and Socks5 proxy
• MIT LICENSE.Feel free to do whatever you want

• Install Python3 and Redis Server

• Change redis args of the project config/settings.py according to redis conf,such as REDIS_HOST,REDIS_PASSWORD

• Install scrapy-splash and change SPLASH_URL in config/settings.py

• Install dependencies

  pip install -r requirements.txt

• Start scrapy worker,including ip proxy crawler and validator

  python crawler_booter.py --usage crawler

  python crawler_booter.py --usage validator

• Start task scheduler,including crawler task scheduler and validator task scheduler

  python scheduler_booter.py --usage crawler

  python scheduler_booter.py --usage validator

• Install Docker

• Install docker-compose

  pip install -U docker-compose

• ChangeSPLASH_URLandREDIS_HOSTin settings.py

• Start all the containers using docker-compose

  docker-compose up

• This project is highly dependent on redis,if you want to replace redis with another mq or database, just do it at your own risk
• If there is no Great Fire Wall at your country,setproxy_mode=0 in both gfw_spider.py and ajax_gfw_spider.py. If you don't want to crawl some websites, set enable=0 in rules.py
• Issues and PRs are welcome
• Just star it if it's useful to you

Thanks to all the contributors of the following projects.

dungproxy

proxyspider

ProxyPool

proxy_pool

ProxyPool

IPProxyTool

IPProxyPool

proxy_list

proxy_pool