From 69f5788745b57f5d165cd9cdd6b58f4762c28839 Mon Sep 17 00:00:00 2001 From: Simon Weald Date: Sun, 9 Sep 2018 10:16:22 +0100 Subject: [PATCH] update notes --- notes.md | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/notes.md b/notes.md index ad9be45..fed82e5 100644 --- a/notes.md +++ b/notes.md @@ -8,16 +8,9 @@ * ~~ignore any links which aren't to pages~~ * better url checking to get bare domain * remove trailing slash from any discovered url - * investigate lxml parser + * ~~investigate lxml parser~~ * ~~remove base url from initial urls with and without trailing slash~~ * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~ * investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request) - - -``` -text/html; charset=utf-8 -application/xhtml+xml -'WebPage' object has no attribute 'source' -'WebPage' object has no attribute 'discovered_hrefs' -``` \ No newline at end of file + * implement some kind of progress display