From ab0ab0a0108516026cfdcd52449d4be38a075f60 Mon Sep 17 00:00:00 2001 From: Simon Weald Date: Fri, 7 Sep 2018 11:50:53 +0100 Subject: [PATCH] add more thoughts --- notes.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/notes.md b/notes.md index 5519084..ad9be45 100644 --- a/notes.md +++ b/notes.md @@ -12,4 +12,12 @@ * ~~remove base url from initial urls with and without trailing slash~~ * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~ - * investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request) \ No newline at end of file + * investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request) + + +``` +text/html; charset=utf-8 +application/xhtml+xml +'WebPage' object has no attribute 'source' +'WebPage' object has no attribute 'discovered_hrefs' +``` \ No newline at end of file