infra/hc-spider
16
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

explain how to enter the mentioned venv

Browse Source 
there was a line missing how to activate the virtual environment. In the described way it would've affected or harmed the system-wide installation
This commit is contained in:
Nils Magnus 2024-03-23 23:52:11 +00:00
parent 16fc233881
commit 7e6d4ee581
1 changed files with 42 additions and 42 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

84
README.md
View File

@ -1,42 +1,42 @@
# Help Center Spider
## About
This is a spider tool with which you can visit all links on https://docs.otc.t-systems.com to find urls that are not correct.
## Requirements
After you cloned the repository you need to prepare an environment to run the tool. You can easily do this with
python virtual environment:
```
$ cd <local_folder>/
$ python -m venv venv/
$ python -m pip install -r requirements.txt
```
## Configuration
In _config.json_ you can define a couple items:
- _watchdog_file_: if you run the tool in the background and want to stop it properly (not using `kill`),
just send an exit message into the watchdog file: `echo exit > watchdog.fifo`
- _timer_runtime_: maximum runtime limit in seconds
- _log_dir_: logging folder
- _logging_interval_: frequency of dumping log files
- _workers_: number of workers (background processes) you want to run. If you set to 0 it will count from the number of cores (_number_of_cores_ - 1)
- _starting_point_: base url where to start
## How to run
There are two ways to do it
### In foreground
```
$ source venv/bin/activate
$ python main.py
```
### In background
```
$ source venv/bin/activate
$ nohup python main.py > log/hc_spider.log 2> log/hc_spider.err <&- &
```
In case you running the tool in background you can stop the execution with `$ echo exit > <watchdog_file>`
# Help Center Spider
## About
This is a spider tool with which you can visit all links on https://docs.otc.t-systems.com to find urls that are not correct.
## Requirements
After you cloned the repository you need to prepare an environment to run the tool. You can easily do this with
python virtual environment:
```
$ cd <local_folder>/
$ python -m venv venv/
$ source venv/bin/activate
(venv)$ python -m pip install -r requirements.txt
```
## Configuration
In _config.json_ you can define a couple items:
- _watchdog_file_: if you run the tool in the background and want to stop it properly (not using `kill`),
just send an exit message into the watchdog file: `echo exit > watchdog.fifo`
- _timer_runtime_: maximum runtime limit in seconds
- _log_dir_: logging folder
- _logging_interval_: frequency of dumping log files
- _workers_: number of workers (background processes) you want to run. If you set to 0 it will count from the number of cores (_number_of_cores_ - 1)
- _starting_point_: base url where to start
## How to run
There are two ways to do it
### In foreground
```
$ source venv/bin/activate
$ python main.py
```
### In background
```
$ source venv/bin/activate
$ nohup python main.py > log/hc_spider.log 2> log/hc_spider.err <&- &
```
In case you running the tool in background you can stop the execution with `$ echo exit > <watchdog_file>`