Concurrent web scraper

Requirements

This crawler was written in 3.7.0 to take advantage of the latest asyncio features.

Install required modules:

pip install -r requirements.txt

Run:

python crawler.py -u https://urltocrawl.com

The resulting sitemap will be output in the root of this directory as sitemap.html