misc/web-scraper
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Compare commits

base: misc:e70bdc9ca191e7672e412bd8869d1fbece4b02aa
Branches Tags
misc:master misc:asyncio
...
compare: misc:164239b3437ee23c0f4b3b9bd6d9b6714b05a74d
Branches Tags
misc:asyncio misc:master

2 Commits
e70bdc9ca1 ... 164239b343

Author SHA1 Message Date
Simon Weald
164239b343 more thoughts 2018-09-06 17:31:12 +01:00
Simon Weald
ce1f2745c9 update thoughts 2018-09-06 17:30:28 +01:00
1 changed files with 2 additions and 0 deletions
Expand all files Collapse all files

2
notes.md
View File

@@ -7,6 +7,8 @@
* ~~`WebPage.parse_urls()` needs to compare startswith to base url~~ * ~~`WebPage.parse_urls()` needs to compare startswith to base url~~
* ~~ignore any links which aren't to pages~~ * ~~ignore any links which aren't to pages~~
* better url checking to get bare domain * better url checking to get bare domain
* remove trailing slash from any discovered url
* investigate lxml parser
* ~~remove base url from initial urls with and without trailing slash~~ * ~~remove base url from initial urls with and without trailing slash~~
* investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls
* ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~ * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~