implement parsing of robots.txt

2018-09-05 18:56:20 +01:00
parent f2c294ebdb
commit a3ec9451e3
3 changed files with 41 additions and 12 deletions
--- a/notes.md
+++ b/notes.md
@@ -9,4 +9,4 @@
  * better url checking to get bare domain
  * ~~remove base url from initial urls with and without trailing slash~~
  * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls
-  * implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)
+  * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~