misc/web-scraper
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

update with changes

Browse Source
This commit is contained in:
Simon Weald
2018-09-09 10:21:45 +01:00
parent 69f5788745
commit d686ae0bc4
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
notes.md
View File

@@ -6,11 +6,11 @@
* ~~return false once the set is empty~~ * ~~return false once the set is empty~~
* ~~`WebPage.parse_urls()` needs to compare startswith to base url~~ * ~~`WebPage.parse_urls()` needs to compare startswith to base url~~
* ~~ignore any links which aren't to pages~~ * ~~ignore any links which aren't to pages~~
* better url checking to get bare domain * ~~better url checking to get bare domain~~ #wontfix
* remove trailing slash from any discovered url * ~~remove trailing slash from any discovered url~~
* ~~investigate lxml parser~~ * ~~investigate lxml parser~~
* ~~remove base url from initial urls with and without trailing slash~~ * ~~remove base url from initial urls with and without trailing slash~~
* investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls * ~~investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls~~ #wontfix
* ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~ * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~
* investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request) * investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request)
* implement some kind of progress display * implement some kind of progress display