# Concurrent web scraper

## Requirements

This crawler was written in 3.7.0 to take advantage of the latest `asyncio` features.

Install required modules:

```bash
pip install -r requirements.txt
```

Run:

```bash
python crawler.py -u https://urltocrawl.com
```

## Results

The resulting sitemap will be output in the root of this directory as `sitemap.html`