Concurrent web scraper

Requirements

This crawler requires at least Python 3.5 in order to utilise the async/await keywords from asyncio.

Install required modules:

pip install -r requirements.txt

Run:

python crawler.py -u https://urltocrawl.com [-c 100]

The resulting sitemap will be output to the root of this directory as sitemap.html