File tree Expand file tree Collapse file tree 1 file changed +14
-0
lines changed Expand file tree Collapse file tree 1 file changed +14
-0
lines changed Original file line number Diff line number Diff line change @@ -48,6 +48,19 @@ motivation.
48
48
python main.py --online
49
49
```
50
50
51
+ ---
52
+
53
+ ** Important:**
54
+
55
+ - The online pipeline will run until you stop it manually, or it reaches the maximum number of sites.
56
+ - You can adapt the configuration in the ` main.py ` . The crawler has alot of options to configure.
57
+ - The online pipeline will start a lot of threads, so it can be quite resource-intensive. You can limit the number of
58
+ - You need a lot of RAM (~ 20 GB of RAM) for the offline pipeline.
59
+ threads in the ` main.py ` file.
60
+ - Have fun crawling the web!
61
+
62
+ ---
63
+
51
64
### Start the server:
52
65
53
66
``` shell
@@ -75,6 +88,7 @@ You can see a list of all available routes by navigating to <http://localhost:80
75
88
- The server will only work if you have crawled some pages before.
76
89
- For the summarization you will need a strong CPU and a lot of RAM, as the summarization is done on the fly and can be
77
90
quite resource-intensive.
91
+
78
92
---
79
93
80
94
## Known Issues
You can’t perform that action at this time.
0 commit comments