# Concurrent web scraper

## Requirements

This crawler requires at least Python 3.5 in order to utilise the async/await keywords from `asyncio`.

Install required modules:

```bash
pip install -r requirements.txt
```

Run:

```bash
python crawler.py -u https://urltocrawl.com [-c 100]
```

## Results

The resulting sitemap will be output to the root of this directory as `sitemap.html`