update readme

Author: DevAlone

README.rst

@@ -17,43 +17,10 @@ You can donate here -> https://www.patreon.com/join/2313433
 
 Thank you :)
 
-How to build?
-*************
-
-1 Clone this repository
-
-.. code-block:: bash
-
-    git clone https://github.com/DevAlone/proxy_py.git
-
-2 Install requirements
-
-.. code-block:: bash
-
-    cd proxy_py
-    pip3 install -r requirements.txt
-
-3 Create settings file
-
-.. code-block:: bash
-
-    cp config_examples/settings.py proxy_py/settings.py
-
-4 Install postgresql and change database configuration in settings.py file
-
-5 (Optional) Configure alembic
-
-6 Run your application
-
-.. code-block:: bash
-    python3 main.py
-
-7 Enjoy!
-
-I'm too lazy. Can I just use it?
-********************************
+How to install?
+***************
 
-Yep, here is a prepared docker image
+There is a prepared docker image.
 
 1 Install docker. If you're using ubuntu:
 
@@ -218,7 +185,18 @@ Example using aiohttp library:
 How to interact with API?
 *************************
 
-Read more about API here -> https://github.com/DevAlone/proxy_py/tree/master/docs/API.md
+Read more about API here -> https://proxy-py.readthedocs.io/en/latest/api_overview.html
+
+What about WEB interface?
+*************************
+
+There is lib.ru inspired web interface which consists of these pages(with slash at the end):
+
+- http://localhost:55555/i/get/proxy/
+- http://localhost:55555/i/get/proxy_count_item/
+- http://localhost:55555/i/get/number_of_proxies_to_process/
+- http://localhost:55555/i/get/number_of_proxies_to_process/
+- http://localhost:55555/i/get/collector_state/
 
 How to contribute?
 ******************
@@ -242,78 +220,35 @@ anything, just run
 
 inside virtual environment in proxy_py project directory.
 
-How to deploy on production using supervisor, nginx and postgresql in 9 steps?
-******************************************************************************
-
-1 Install supervisor, nginx and postgresql
-
-.. code-block:: bash
-
-    root@server:~$ apt install supervisor nginx postgresql
-
-2 Clone this repository
-
-.. code-block:: bash
-
-    user@server:~/$ git clone https://github.com/DevAlone/proxy_py.git
+How to build from scratch?
+**************************
 
-3 Create virtual environment and install requirements on it
-
-.. code-block:: bash
-
-    user@server:~/proxy_py$ python3 -m venv env
-    user@server:~/proxy_py$ pip3 install -r requirements.txt
-
-4 Copy settings.py example:
+1 Clone this repository
 
 .. code-block:: bash
 
-    proxy_py@server:~/proxy_py$ cp config_examples/settings.py ./
+    git clone https://github.com/DevAlone/proxy_py.git
 
-5 create unprivileged user in postgresql database
-and change database authentication data in settings.py
+2 Install requirements
 
 .. code-block:: bash
 
-    proxy_py@server:~/proxy_py$ vim proxy_py/settings.py
-
-.. code-block:: python
-
-    DATABASE_CONNECTION_KWARGS = {
-        'database': 'YOUR_POSTGRES_DATABASE',
-        'user': 'YOUR_POSTGRES_USER',
-        'password': 'YOUR_POSTGRES_PASSWORD',
-        # number of simultaneous connections
-        # 'max_connections': 20,
-    }
+    cd proxy_py
+    pip3 install -r requirements.txt
 
-6 Copy supervisor config example and change it for your case
+3 Create settings file
 
 .. code-block:: bash
 
-    cp /home/proxy_py/proxy_py/config_examples/proxy_py.supervisor.conf /etc/supervisor/conf.d/proxy_py.conf
-    vim /etc/supervisor/conf.d/proxy_py.conf
-
-7 Copy nginx config example, enable it and change if you need
+    cp config_examples/settings.py proxy_py/settings.py
 
-.. code-block:: bash
+4 Install postgresql and change database configuration in settings.py file
 
-    cp /home/proxy_py/proxy_py/config_examples/proxy_py.nginx.conf /etc/nginx/sites-available/proxy_py
-    ln -s /etc/nginx/sites-available/proxy_py /etc/nginx/sites-enabled/
-    vim /etc/nginx/sites-available/proxy_py
+5 (Optional) Configure alembic
 
-8 Restart supervisor and Nginx
+6 Run your application
 
 .. code-block:: bash
+    python3 main.py
 
-    supervisorctl reread
-    supervisorctl update
-    /etc/init.d/nginx configtest
-    /etc/init.d/nginx restart
-
-9 Enjoy using it on your server!
-
-What is it depend on?
-*********************
-
-See *requirements.txt*
+7 Enjoy!

docs/source/api_v1_overview.md renamed to docs/source/api_overview.md

@@ -1,4 +1,4 @@
-# proxy_py API
+# proxy_py API v1
 
 proxy_py expects HTTP POST requests with JSON as a body, so you need
 to add header `Content-Type: application/json` and send correct
@@ -7,16 +7,16 @@ JSON document.
 Example of correct request:
 ```json
 {
-  "model": "proxy",
-  "method": "get"
+  "method": "get",
+  "model": "proxy"
 }
 ```
 
 Response is also HTTP with JSON and status code depending on whether
 error happened or not.
 
 * 200 if there wasn't error
-* 400 if you sent bad request
+* 400 if you sent a bad request
 * 500 if there was an error during execution your request or in some
 other cases
 
@@ -35,32 +35,33 @@ Now it's only supported to work with `proxy` model.
 
 ### get method
 
-`get` method supports following keys:
-* `order_by` (string) - specifies ordering fields as comma separated
+`get` method supports the following keys:
+* `order_by` (string) - specifies ordering fields as comma separated ("response_time" if not provided)
 value.
 
-Examples:
+Explanation:
 
 `"uptime"` just sorts proxies by uptime field ascending.
 
 Note: `uptime` is the timestamp from which proxy is working,
 NOT proxy's working time
 
-To sort descending use `-` before field name.
+To sort descending use `-`(minus) before the field name.
 
 `"-response_time"` returns proxies with maximum response_time first
 (in microseconds)
 
-It's possible to sort using multiple fields
+It's also possible to sort using multiple fields
 
 `"number_of_bad_checks, response_time"` returns proxies with minimum
 `number_of_bad_checks` first, if there are proxies with the same
 `number_of_bad_checks`, sorts them by `response_time`
 
-* `limit` (integer) -  specifying how many proxies to return
-* `offset` (integer) - specifying how many proxies to skip
+* `limit` (integer) -  specifies how many proxies to return (1024 if not provided)
+* `offset` (integer) - specifies how many proxies to skip (0 if not provided)
 
 Example of `get` request:
+
 ```json
 
 {
@@ -100,7 +101,9 @@ Response
 }
 ```
 
-* `count` (integer) - total number of proxies for that request
+Response fiels:
+
+* `count` (integer) - total number of proxies for that request(how many you can fetch increasing offset)
 * `data` (array) - list of proxies
 * `has_more` (boolean) - value indicating whether you can increase
 offset to get more proxies or not
@@ -132,4 +135,18 @@ Response:
 
 ### count method
 
-Same as get, but not returns data
+Same as get, but doesn't return data
+
+# proxy_py API v2
+
+Second version of API has only 2 methods so far 
+
+```bash
+curl https://localhost:55555/api/v2/get_proxy_for_id --data '{"id": "ID"}'
+```            
+```bash
+curl https://localhost:55555/api/v2/get_proxies_for_id --data '{"id": "ID", "number": 2}'
+```
+
+get_proxy_for_id should return the best proxy for a given ID avoiding overlapping with other IDs, but so far it just returns a random one ignoring ID at all.
+get_proxies_for_id is the same, but also has a parameter `number` to specify the number of proxies to return.

docs/source/conf.py

@@ -35,7 +35,6 @@
     'sphinx.ext.doctest',
     'sphinx.ext.todo',
     'sphinx.ext.coverage',
-    'sphinx.ext.pngmath',
     'sphinx.ext.ifconfig',
     'sphinx.ext.viewcode',
 ]

docs/source/index.rst

@@ -12,7 +12,7 @@ Contents:
    :maxdepth: 2
 
    readme_link
-   api_v1_overview.md
+   api_overview.md
 
    guides/guides
 

docs/source/modules/check_from_stdin.rst

@@ -0,0 +1,7 @@
+check\_from\_stdin module
+=========================
+
+.. automodule:: check_from_stdin
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/modules/checkers.google_com_checker.rst

@@ -0,0 +1,7 @@
+checkers.google\_com\_checker module
+====================================
+
+.. automodule:: checkers.google_com_checker
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/modules/checkers.rst

@@ -8,6 +8,7 @@ Submodules
 
    checkers.base_checker
    checkers.d3d_info_checker
+   checkers.google_com_checker
    checkers.ipinfo_io_checker
 
 Module contents

docs/source/modules/collectors.abstract_collector.rst

@@ -0,0 +1,7 @@
+collectors.abstract\_collector module
+=====================================
+
+.. automodule:: collectors.abstract_collector
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/modules/collectors.rst

@@ -1,25 +1,12 @@
 collectors package
 ==================
 
-Subpackages
------------
-
-.. toctree::
-
-    collectors.checkerproxy_net
-    collectors.free_proxy_list_net
-    collectors.freeproxylists_net
-    collectors.gatherproxy_com
-    collectors.nordvpn_com
-    collectors.premproxy_com
-    collectors.proxy_list_org
-
 Submodules
 ----------
 
 .. toctree::
 
-   collectors.collector
+   collectors.abstract_collector
    collectors.pages_collector
 
 Module contents

docs/source/modules/http_client.rst

@@ -0,0 +1,7 @@
+http\_client module
+===================
+
+.. automodule:: http_client
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/modules/materialized_view_updater.rst

@@ -0,0 +1,7 @@
+materialized\_view\_updater module
+==================================
+
+.. automodule:: materialized_view_updater
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/modules/modules.rst

@@ -5,15 +5,20 @@ proxy_py Modules
    :maxdepth: 4
 
    async_requests
+   check_from_stdin
    checkers
    collectors
    collectors_list
-   dump_db
-   fill_db
+   http_client
    main
+   materialized_view_updater
    models
+   parsers
    processor
    proxy_py
    proxy_utils
+   proxy_validator
    server
    setup
+   statistics
+   tests

docs/source/modules/parsers.regex_parser.rst

@@ -0,0 +1,7 @@
+parsers.regex\_parser module
+============================
+
+.. automodule:: parsers.regex_parser
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/modules/parsers.rst

@@ -0,0 +1,17 @@
+parsers package
+===============
+
+Submodules
+----------
+
+.. toctree::
+
+   parsers.regex_parser
+
+Module contents
+---------------
+
+.. automodule:: parsers
+    :members:
+    :undoc-members:
+    :show-inheritance: