Proxy Server Crawler
When we are doing security tests, we always change our IP address to bypass some security strategies. The easiest way to change IP is using a proxy.
Some websites can provide proxy IPs, but none of them can ensure the healthy of those proxy hosts. It’s a horrible thing to check them one by one by hand when you wanna got one. So we can crawl these websites and test every proxy IP automatically.
The project is hosted at https://github.com/xelzmm/proxy_server_crawler.
Introduction
Proxy Server Crawler is a tool used to crawl public proxy servers from proxy websites. When crawled a proxy server(ip::port::type), it will test the functionality of the server automatically.
Currently supported websites:
- http://www.66ip.cn
- http://www.cz88.net
- http://www.cn-proxy.com
- http://www.haodailiip.com
- http://www.kuaidaili.com
- http://www.proxylists.net
- http://www.qiaodm.net
- http://www.socks-proxy.net
- http://www.xroxy.com
- http://www.xicidaili.com
Currently supported testing(for http proxy)
- ssl support
- post support
- speed (tested with 10 frequently used sites)
- type(high/anonymous/transparent)
Requirements
- Python >= 2.7
- Scrapy 1.3.0 (not tested for lower version)
- node (for some sites, you need node to bypass waf based on javascript)
Usage
1 2 |
|
[log]
1 2 3 4 5 6 7 8 9 10 |
|
License
The MIT License (MIT)