Simon Weald fdd84a8786 manually retrieve robots.txt to ensure we can set the user-agent

2018-09-07 12:40:12 +01:00

..

__init__.py

blank __init__.py

2018-08-28 22:29:11 +01:00

helpers.py

manually retrieve robots.txt to ensure we can set the user-agent

2018-09-07 12:40:12 +01:00