From ce1f2745c9dacf74f7a2e94eb5024984fb57be16 Mon Sep 17 00:00:00 2001 From: Simon Weald Date: Thu, 6 Sep 2018 17:30:28 +0100 Subject: [PATCH] update thoughts --- notes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/notes.md b/notes.md index 36d72d5..4b133db 100644 --- a/notes.md +++ b/notes.md @@ -7,6 +7,7 @@ * ~~`WebPage.parse_urls()` needs to compare startswith to base url~~ * ~~ignore any links which aren't to pages~~ * better url checking to get bare domain + * remove trailing slash from any discovered url * ~~remove base url from initial urls with and without trailing slash~~ * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~