From d686ae0bc41d49f2f261e782eb88817f08c2c319 Mon Sep 17 00:00:00 2001 From: Simon Weald Date: Sun, 9 Sep 2018 10:21:45 +0100 Subject: [PATCH] update with changes --- notes.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/notes.md b/notes.md index fed82e5..97db6d2 100644 --- a/notes.md +++ b/notes.md @@ -6,11 +6,11 @@ * ~~return false once the set is empty~~ * ~~`WebPage.parse_urls()` needs to compare startswith to base url~~ * ~~ignore any links which aren't to pages~~ - * better url checking to get bare domain - * remove trailing slash from any discovered url + * ~~better url checking to get bare domain~~ #wontfix + * ~~remove trailing slash from any discovered url~~ * ~~investigate lxml parser~~ * ~~remove base url from initial urls with and without trailing slash~~ - * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls + * ~~investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls~~ #wontfix * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~ * investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request) * implement some kind of progress display