added new ideas to implement

2018-09-04 15:40:11 +01:00
parent 1b9b207a28
commit f2c294ebdb
1 changed files with 2 additions and 0 deletions
--- a/notes.md
+++ b/notes.md
@@ -8,3 +8,5 @@
  * ignore any links which aren't to pages
  * better url checking to get bare domain
  * ~~remove base url from initial urls with and without trailing slash~~
+  * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls
+  * implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)