updated readme
This commit is contained in:
20
README.md
20
README.md
@@ -1 +1,21 @@
|
|||||||
# Concurrent web scraper
|
# Concurrent web scraper
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
This crawler was written in 3.7.0 to take advantage of the latest `asyncio` features.
|
||||||
|
|
||||||
|
Install required modules:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python crawler.py -u https://urltocrawl.com
|
||||||
|
```
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
The resulting sitemap will be output in the root of this directory as `sitemap.html`
|
||||||
|
|||||||
Reference in New Issue
Block a user