22 lines
388 B
Markdown
22 lines
388 B
Markdown
# Concurrent web scraper
|
|
|
|
## Requirements
|
|
|
|
This crawler requires at least Python 3.5 in order to utilise the async/await keywords from `asyncio`.
|
|
|
|
Install required modules:
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
Run:
|
|
|
|
```bash
|
|
python crawler.py -u https://urltocrawl.com [-c 100]
|
|
```
|
|
|
|
## Results
|
|
|
|
The resulting sitemap will be output to the root of this directory as `sitemap.html`
|