infra/hc-spider
16
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

explain how to enter the mentioned venv

Browse Source 
there was a line missing how to activate the virtual environment. In the described way it would've affected or harmed the system-wide installation
This commit is contained in:
Nils Magnus 2024-03-23 23:52:11 +00:00
parent 16fc233881
commit 7e6d4ee581
1 changed files with 42 additions and 42 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

84
README.md
View File

@ -1,42 +1,42 @@
# Help Center Spider # Help Center Spider
## About ## About
This is a spider tool with which you can visit all links on https://docs.otc.t-systems.com to find urls that are not correct. This is a spider tool with which you can visit all links on https://docs.otc.t-systems.com to find urls that are not correct.
## Requirements ## Requirements
After you cloned the repository you need to prepare an environment to run the tool. You can easily do this with After you cloned the repository you need to prepare an environment to run the tool. You can easily do this with
python virtual environment: python virtual environment:
``` ```
$ cd <local_folder>/ $ cd <local_folder>/
$ python -m venv venv/ $ python -m venv venv/
$ python -m pip install -r requirements.txt $ source venv/bin/activate
``` (venv)$ python -m pip install -r requirements.txt
```
## Configuration
In _config.json_ you can define a couple items: ## Configuration
In _config.json_ you can define a couple items:
- _watchdog_file_: if you run the tool in the background and want to stop it properly (not using `kill`),
just send an exit message into the watchdog file: `echo exit > watchdog.fifo` - _watchdog_file_: if you run the tool in the background and want to stop it properly (not using `kill`),
- _timer_runtime_: maximum runtime limit in seconds just send an exit message into the watchdog file: `echo exit > watchdog.fifo`
- _log_dir_: logging folder - _timer_runtime_: maximum runtime limit in seconds
- _logging_interval_: frequency of dumping log files - _log_dir_: logging folder
- _workers_: number of workers (background processes) you want to run. If you set to 0 it will count from the number of cores (_number_of_cores_ - 1) - _logging_interval_: frequency of dumping log files
- _starting_point_: base url where to start - _workers_: number of workers (background processes) you want to run. If you set to 0 it will count from the number of cores (_number_of_cores_ - 1)
- _starting_point_: base url where to start
## How to run
There are two ways to do it ## How to run
There are two ways to do it
### In foreground
``` ### In foreground
$ source venv/bin/activate ```
$ python main.py $ source venv/bin/activate
``` $ python main.py
```
### In background
``` ### In background
$ source venv/bin/activate ```
$ nohup python main.py > log/hc_spider.log 2> log/hc_spider.err <&- & $ source venv/bin/activate
``` $ nohup python main.py > log/hc_spider.log 2> log/hc_spider.err <&- &
```
In case you running the tool in background you can stop the execution with `$ echo exit > <watchdog_file>`
In case you running the tool in background you can stop the execution with `$ echo exit > <watchdog_file>`