misc/web-scraper
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

update notes

Browse Source
This commit is contained in:
Simon Weald
2018-09-09 10:16:22 +01:00
parent b5d644a223
commit 69f5788745
1 changed files with 2 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
notes.md
View File

@@ -8,16 +8,9 @@
* ~~ignore any links which aren't to pages~~ * ~~ignore any links which aren't to pages~~
* better url checking to get bare domain * better url checking to get bare domain
* remove trailing slash from any discovered url * remove trailing slash from any discovered url
* investigate lxml parser * ~~investigate lxml parser~~
* ~~remove base url from initial urls with and without trailing slash~~ * ~~remove base url from initial urls with and without trailing slash~~
* investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls
* ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~ * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~
* investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request) * investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request)
* implement some kind of progress display
```
text/html; charset=utf-8
application/xhtml+xml
'WebPage' object has no attribute 'source'
'WebPage' object has no attribute 'discovered_hrefs'
```