started to write guide about creating collectors

Author: DevAlone

README.rst

@@ -1,5 +1,5 @@
 proxy_py README
-========
+===============
 
 proxy_py is a program which collects proxies, saves them in a database
 and makes periodically checks.

check_from_stdin.py

@@ -0,0 +1,52 @@
+"""
+just a helper script for testing proxies
+"""
+from proxy_py import settings
+from models import Proxy
+from checkers.base_checker import BaseChecker
+
+import asyncio
+import proxy_utils
+import sys
+import re
+
+
+proxy_find_regex = \
+    r"([0-9]{1,3})[^0-9]+([0-9]{1,3})[^0-9]+([0-9]{1,3})[^0-9]+([0-9]{1,3})"\
+    r"[^0-9]+([0-9]{1,5})"
+semaphore = asyncio.BoundedSemaphore(settings.NUMBER_OF_CONCURRENT_TASKS)
+tasks = []
+
+
+async def check_task(ip, port):
+    async with semaphore:
+        for raw_protocol in range(len(Proxy.PROTOCOLS)):
+            proxy_url = '{}://{}:{}'.format(
+                Proxy.PROTOCOLS[raw_protocol],
+                ip,
+                port
+            )
+            check_result, _ = await proxy_utils.check_proxy(proxy_url)
+            if check_result:
+                break
+        # if check_result:
+        #     print('proxy {} works'.format(proxy_url))
+        print('+' if check_result else '-', end='')
+
+
+async def main():
+    for line in sys.stdin:
+        line = line.strip()
+        groups = re.search(proxy_find_regex, line).groups()
+        ip = '.'.join(groups[:4])
+        port = groups[4]
+
+        tasks.append(asyncio.ensure_future(check_task(ip, port)))
+
+    await asyncio.gather(*tasks)
+    print()
+    BaseChecker.clean()
+
+
+if __name__ == '__main__':
+    asyncio.get_event_loop().run_until_complete(main())

collectors/collector.py renamed to collectors/abstract_collector.py

@@ -1,5 +1,5 @@
 from proxy_py import settings
-from collectors.collector import AbstractCollector
+from collectors.abstract_collector import AbstractCollector
 
 
 # TODO: save pages to collector state

collectors/pages_collector.py

@@ -1,4 +1,4 @@
-from collectors.collector import AbstractCollector
+from collectors.abstract_collector import AbstractCollector
 
 import async_requests
 import json

collectors/web/net/checkerproxy/collector.py

@@ -1,4 +1,4 @@
-from collectors.collector import AbstractCollector
+from collectors.abstract_collector import AbstractCollector
 import lxml.html
 import lxml.etree
 

collectors/web/net/free_proxy_list/collector.py

@@ -1,7 +1,8 @@
 proxy_py Guides
-========
+===============
 
 .. toctree::
-   :maxdepth: 4
+    :maxdepth: 4
 
+    how_to_create_collector.rst
 

docs/source/guides/guides.rst

@@ -0,0 +1,30 @@
+proxy_py How to create collector
+================================
+
+Collector is a class which is used to parse proxies from web page or another source. All collectors are inherited from `collectors.abstract_collector.AbstractCollector`, also there is `collectors.pages_collector.PagesCollector` which is used for paginated sources. It's better to understand through examples.
+
+Simple collector
+****************
+
+Let's start with the simplest collector we can imagine, it will be from the page http://www.89ip.cn/ti.html as you can see, it sends form as GET request to this url http://www.89ip.cn/tqdl.html?num=9999&address=&kill_address=&port=&kill_port=&isp= 
+
+Firstly we can try to check that these proxies are really good. Just copy and paste list of proxies to file say /tmp/proxies and run this command inside virtual environment
+
+.. code-block:: bash
+
+    cat /tmp/proxies | python3 check_from_stdin.py
+
+You're gonna get something like this:
+
+`++++++++++++++++++++++-+++++-+++++++++++++++++++++++++++-++++++-++++-+++++++++++++++++++++++++++++++--+++++++-+++++++-++-+-+++-+++++++++-+++++++++++++++++++++--++--+-++++++++++++++++-+++--+++-+-+++++++++++++++++--++++++++++++-+++++-+++-++++++++-+++++-+-+++++++-++-+--++++-+++-++++++++++-++++--+++++++-+++++++-++--+++++-+-+++++++++++++++++++++-++-+++-+++--++++--+++-+++++++-+++++++-+++++++++++++++---+++++-+++++++++-+++++-+-++++++++++++-+--+++--+-+-+-++-+++++-+++--++++++-+++++++++++--+-+++-+-++++--+++++--+++++++++-+-+-++++-+-++++++++++++++-++-++++++--+--++++-+-++--++--+++++-++-+++-++++--++--+---------+--+--++--------+++-++-+--++++++++++++++++-+++++++++-+++++++--+--+--+-+-+++---++------------------+--+----------+-+-+--++-+----------+-------+--+------+----+-+--+--++----+--+-++++++-++-+++`
+
++ means working proxy with at leat one protocol, - means not working, the result above is perfect, so many good proxies.
+
+Note: working means proxy respond with timeout set in settings, if you increase it you'll get more proxies.
+
+Paginated collector
+*******************
+
+So, let's create a simple paginated collector. 
+
+