Simon Weald 738ab8e441 adjust robots handling to deal with 404s and enforce a user agent which allows us to initially obtain the user agent

2018-09-09 09:57:16 +01:00

5.1 KiB

Raw Blame History

View Raw