Skip to content

Commit 2a62a0e

Browse files
DevAloneDevAlone
committed
docker files
1 parent 5a75dfb commit 2a62a0e

File tree

4 files changed

+132
-110
lines changed

4 files changed

+132
-110
lines changed

‎Dockerfile

Lines changed: 24 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,36 @@
11
FROM python:3.7-slim
22

3-
RUN apt update && apt install -y wget > /dev/null && rm -rf /var/lib/apt/lists/* \
4-
&& rm /bin/sh && ln -s /bin/bash /bin/sh \
5-
&& groupadd -r user && useradd --create-home --no-log-init -r -g user user \
6-
&& mkdir /proxy_py && chown user:user /proxy_py
3+
RUN apt-get update \
4+
&& apt-get install -y wget \
5+
&& rm -rf /var/lib/apt/lists/* \
6+
&& rm /bin/sh \
7+
&& ln -s /bin/bash /bin/sh \
8+
&& groupadd -r user \
9+
&& useradd --create-home --no-log-init -r -g user user \
10+
&& mkdir /proxy_py \
11+
&& chown user:user /proxy_py
712

813
WORKDIR /proxy_py
914
USER user
1015
ARG VERSION=master
1116

12-
RUN wget https://github.com/DevAlone/proxy_py/archive/$VERSION.tar.gz -O sources.tar.gz 2> /dev/null \
17+
RUN echo "Downloading proxy_py sources..." \
18+
&& wget https://github.com/DevAlone/proxy_py/archive/$VERSION.tar.gz -O sources.tar.gz 2> /dev/null \
1319
&& tar -xf sources.tar.gz && rm sources.tar.gz \
1420
&& mv proxy_py-*/.[!.]* ./ && mv proxy_py-*/* ./ \
1521
&& rmdir proxy_py-* \
16-
&& python3 -m venv env
17-
18-
RUN cp config_examples/settings.py proxy_py/settings.py
19-
RUN source ./env/bin/activate && pip3 install -r requirements.txt --no-cache-dir
22+
&& python3 -m venv env \
23+
&& echo "Creating IP:Location database..." \
24+
&& mkdir /tmp/proxy_py_9910549a_7d41_4102_9e9d_15d39418a5cb \
25+
&& cd /tmp/proxy_py_9910549a_7d41_4102_9e9d_15d39418a5cb \
26+
&& wget "https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz" 2> /dev/null \
27+
&& tar xf GeoLite2-City.tar.gz \
28+
&& mv GeoLite2-City_*/GeoLite2-City.mmdb ./ \
29+
&& rm -r GeoLite2-City_* \
30+
&& cd /proxy_py \
31+
&& cp config_examples/settings.py proxy_py/settings.py \
32+
&& echo "Installing dependencies..." \
33+
&& source ./env/bin/activate \
34+
&& pip3 install -r requirements.txt --no-cache-dir
2035

2136
EXPOSE 55555
22-
CMD ./run.sh

‎README.rst

Lines changed: 106 additions & 100 deletions
Original file line numberDiff line numberDiff line change
@@ -22,37 +22,43 @@ How to install?
2222

2323
There is a prepared docker image.
2424

25-
1 Install docker. If you're using ubuntu:
25+
1 Install docker and docker compose. If you're using ubuntu:
2626

2727
.. code-block:: bash
2828
29-
sudo apt install docker.io
29+
sudo apt install docker.io docker-compose
30+
31+
2 Download docker compose config:
32+
33+
.. code-block:: bash
34+
35+
wget "https://raw.githubusercontent.com/DevAlone/proxy_py/master/docker-compose.yml"
3036
3137
2 Create a container
3238

3339
.. code-block:: bash
3440
35-
docker create -p 55555:55555 --name proxy_py devalone/proxy_py:v2.2
41+
docker-compose build
3642
3743
3 Run
3844

3945
.. code-block::
4046
41-
docker start proxy_py
47+
docker-compose up
4248
4349
It will give you a server on address localhost:55555
4450

4551
To see running containers use
4652

4753
.. code-block:: bash
4854
49-
docker ps
55+
docker-compose ps
5056
5157
To stop proxy_py use
5258

5359
.. code-block:: bash
5460
55-
docker stop proxy_py
61+
docker-compose stop
5662
5763
How to get proxies?
5864
*******************
@@ -65,11 +71,11 @@ on address `http://127.0.0.1:55555/api/v1/`
6571

6672
.. code-block:: json
6773
68-
{
69-
"model": "proxy",
70-
"method": "get",
71-
"order_by": "response_time, uptime"
72-
}
74+
{
75+
"model": "proxy",
76+
"method": "get",
77+
"order_by": "response_time, uptime"
78+
}
7379

7480
Note: order_by makes the result sorted
7581
by one or more fields(separated by comma).
@@ -80,25 +86,25 @@ It's gonna return you the json response like this:
8086

8187
.. code-block:: json
8288
83-
{
84-
"count": 1,
85-
"data": [{
86-
"address": "http://127.0.0.1:8080",
87-
"auth_data": "",
88-
"bad_proxy": false,
89-
"domain": "127.0.0.1",
90-
"last_check_time": 1509466165,
91-
"number_of_bad_checks": 0,
92-
"port": 8080,
93-
"protocol": "http",
94-
"response_time": 461691,
95-
"uptime": 1509460949
96-
}
97-
],
98-
"has_more": false,
99-
"status": "ok",
100-
"status_code": 200
101-
}
89+
{
90+
"count": 1,
91+
"data": [{
92+
"address": "http://127.0.0.1:8080",
93+
"auth_data": "",
94+
"bad_proxy": false,
95+
"domain": "127.0.0.1",
96+
"last_check_time": 1509466165,
97+
"number_of_bad_checks": 0,
98+
"port": 8080,
99+
"protocol": "http",
100+
"response_time": 461691,
101+
"uptime": 1509460949
102+
}
103+
],
104+
"has_more": false,
105+
"status": "ok",
106+
"status_code": 200
107+
}
102108

103109
Note: All fields except *protocol*, *domain*, *port*, *auth_data*,
104110
*checking_period* and *address* CAN be null
@@ -107,80 +113,80 @@ Or error if something went wrong:
107113

108114
.. code-block:: json
109115
110-
{
111-
"error_message": "You should specify \"model\"",
112-
"status": "error",
113-
"status_code": 400
114-
}
116+
{
117+
"error_message": "You should specify \"model\"",
118+
"status": "error",
119+
"status_code": 400
120+
}
115121

116122
Note: status_code is also duplicated in HTTP status code
117123

118124
Example using curl:
119125

120126
.. code-block:: bash
121127
122-
curl -X POST http://127.0.0.1:55555/api/v1/ -H "Content-Type: application/json" --data '{"model": "proxy", "method": "get"}'
128+
curl -X POST http://127.0.0.1:55555/api/v1/ -H "Content-Type: application/json" --data '{"model": "proxy", "method": "get"}'
123129

124130
Example using httpie:
125131

126132
.. code-block:: bash
127133
128-
http POST http://127.0.0.1:55555/api/v1/ model=proxy method=get
134+
http POST http://127.0.0.1:55555/api/v1/ model=proxy method=get
129135

130136
Example using python's *requests* library:
131137

132138
.. code-block:: python
133139
134-
import requests
135-
import json
140+
import requests
141+
import json
136142

137143

138-
def get_proxies():
139-
result = []
140-
json_data = {
141-
"model": "proxy",
142-
"method": "get",
143-
}
144-
url = "http://127.0.0.1:55555/api/v1/"
144+
def get_proxies():
145+
result = []
146+
json_data = {
147+
"model": "proxy",
148+
"method": "get",
149+
}
150+
url = "http://127.0.0.1:55555/api/v1/"
145151

146-
response = requests.post(url, json=json_data)
147-
if response.status_code == 200:
148-
response = json.loads(response.text)
149-
for proxy in response["data"]:
150-
result.append(proxy["address"])
151-
else:
152-
# check error here
153-
pass
152+
response = requests.post(url, json=json_data)
153+
if response.status_code == 200:
154+
response = json.loads(response.text)
155+
for proxy in response["data"]:
156+
result.append(proxy["address"])
157+
else:
158+
# check error here
159+
pass
154160

155-
return result
161+
return result
156162

157163
Example using aiohttp library:
158164

159165
.. code-block:: python
160166
161-
import aiohttp
167+
import aiohttp
162168

163169

164-
async def get_proxies():
165-
result = []
166-
json_data = {
167-
"model": "proxy",
168-
"method": "get",
169-
}
170+
async def get_proxies():
171+
result = []
172+
json_data = {
173+
"model": "proxy",
174+
"method": "get",
175+
}
170176

171-
url = "http://127.0.0.1:55555/api/v1/"
177+
url = "http://127.0.0.1:55555/api/v1/"
172178

173-
async with aiohttp.ClientSession() as session:
174-
async with session.post(url, json=json_data) as response:
175-
if response.status == 200:
176-
response = json.loads(await response.text())
177-
for proxy in response["data"]:
178-
result.append(proxy["address"])
179-
else:
180-
# check error here
181-
pass
179+
async with aiohttp.ClientSession() as session:
180+
async with session.post(url, json=json_data) as response:
181+
if response.status == 200:
182+
response = json.loads(await response.text())
183+
for proxy in response["data"]:
184+
result.append(proxy["address"])
185+
else:
186+
# check error here
187+
pass
182188

183-
return result
189+
return result
184190

185191
How to interact with API?
186192
*************************
@@ -193,53 +199,53 @@ What about WEB interface?
193199
There is lib.ru inspired web interface which consists of these pages(with slash at the end):
194200

195201
- http://localhost:55555/i/get/proxy/
196-
- http://localhost:55555/i/get/proxy_count_item/
197-
- http://localhost:55555/i/get/number_of_proxies_to_process/
198-
- http://localhost:55555/i/get/collector_state/
202+
- http://localhost:55555/i/get/proxy_count_item/
203+
- http://localhost:55555/i/get/number_of_proxies_to_process/
204+
- http://localhost:55555/i/get/collector_state/
199205

200-
How to contribute?
201-
******************
206+
How to contribute?
207+
******************
202208

203-
Just fork, do your changes(implement new collector, fix a bug
204-
or whatever you want) and create pull request.
209+
Just fork, do your changes(implement new collector, fix a bug
210+
or whatever you want) and create pull request.
205211

206-
Here are some useful guides:
212+
Here are some useful guides:
207213

208-
`How to create a collector <https://proxy-py.readthedocs.io/en/latest/guides/how_to_create_collector.html>`_
214+
`How to create a collector <https://proxy-py.readthedocs.io/en/latest/guides/how_to_create_collector.html>`_
209215

210-
How to test it?
211-
***************
216+
How to test it?
217+
***************
212218

213-
If you've made changes to the code and want to check that you didn't break
214-
anything, just run
219+
If you've made changes to the code and want to check that you didn't break
220+
anything, just run
215221

216-
.. code-block:: bash
222+
.. code-block:: bash
217223
218-
py.test
224+
py.test
219225

220-
inside virtual environment in proxy_py project directory.
226+
inside virtual environment in proxy_py project directory.
221227

222-
How to build from scratch?
223-
**************************
228+
How to build from scratch?
229+
**************************
224230

225-
1 Clone this repository
231+
1 Clone this repository
226232

227-
.. code-block:: bash
233+
.. code-block:: bash
228234
229-
git clone https://github.com/DevAlone/proxy_py.git
235+
git clone https://github.com/DevAlone/proxy_py.git
230236

231237
2 Install requirements
232238

233239
.. code-block:: bash
234240
235-
cd proxy_py
236-
pip3 install -r requirements.txt
241+
cd proxy_py
242+
pip3 install -r requirements.txt
237243

238244
3 Create settings file
239245

240246
.. code-block:: bash
241247
242-
cp config_examples/settings.py proxy_py/settings.py
248+
cp config_examples/settings.py proxy_py/settings.py
243249

244250
4 Install postgresql and change database configuration in settings.py file
245251

@@ -248,7 +254,7 @@ How to build from scratch?
248254
6 Run your application
249255

250256
.. code-block:: bash
251-
python3 main.py
257+
python3 main.py
252258

253259
7 Enjoy!
254260

@@ -257,5 +263,5 @@ Mirrors
257263
*******
258264

259265
* https://github.com/DevAlone/proxy_py
260-
* https://gitlab.com/DevAlone/proxy_py
261-
* https://bitbucket.org/d3dev/proxy_py
266+
* https://gitlab.com/DevAlone/proxy_py
267+
* https://bitbucket.org/d3dev/proxy_py

‎TODO

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,4 @@ write number of asyncio tasks to DB to be able to draw more grahps(everybody lov
1616
fix issue with domains like this 08.8.8.8
1717
refresh materialized view periodically
1818
send request to proxy multiple times
19+
consider moving to PyPy

‎docker-compose.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ services:
1313
- ./init_db.sql:/docker-entrypoint-initdb.d/init_db.sql
1414
core:
1515
image: "devalone/proxy_py:latest"
16+
command: ./run.sh
1617
restart: always
1718
ports:
1819
- "55555:55555"

0 commit comments

Comments
 (0)