Simon Weald 738ab8e441 adjust robots handling to deal with 404s and enforce a user agent which allows us to initially obtain the user agent

2018-09-09 09:57:16 +01:00

..

__init__.py

blank __init__.py

2018-08-28 22:29:11 +01:00

helpers.py

adjust robots handling to deal with 404s and enforce a user agent which allows us to initially obtain the user agent

2018-09-09 09:57:16 +01:00