From f2c294ebdb135ec5e653fcb6dee5ce1e8e093afb Mon Sep 17 00:00:00 2001 From: Simon Weald Date: Tue, 4 Sep 2018 15:40:11 +0100 Subject: [PATCH] added new ideas to implement --- notes.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/notes.md b/notes.md index 1575f85..3225f8a 100644 --- a/notes.md +++ b/notes.md @@ -8,3 +8,5 @@ * ignore any links which aren't to pages * better url checking to get bare domain * ~~remove base url from initial urls with and without trailing slash~~ + * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls + * implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)