From 17fa9f93f9cb66526441a7c23e0d4d36687efe29 Mon Sep 17 00:00:00 2001 From: Simon Weald Date: Sun, 9 Sep 2018 10:52:37 +0100 Subject: [PATCH] tick off gzip encoding --- notes.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/notes.md b/notes.md index cb50d6f..d40faad 100644 --- a/notes.md +++ b/notes.md @@ -12,6 +12,7 @@ * ~~remove base url from initial urls with and without trailing slash~~ * ~~investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls~~ #wontfix * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~ - * investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request) + * ~~investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request)~~ * implement some kind of progress display - * async \ No newline at end of file + * async + * better exception handling