22 lines
362 B
Markdown
22 lines
362 B
Markdown
# Concurrent web scraper
|
|
|
|
## Requirements
|
|
|
|
This crawler was written in 3.7.0 to take advantage of the latest `asyncio` features.
|
|
|
|
Install required modules:
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
Run:
|
|
|
|
```bash
|
|
python crawler.py -u https://urltocrawl.com
|
|
```
|
|
|
|
## Results
|
|
|
|
The resulting sitemap will be output in the root of this directory as `sitemap.html`
|