more thoughts

update thoughts
2018-09-06 17:31:12 +01:00 · 2018-09-06 17:30:28 +01:00
1 changed files with 2 additions and 0 deletions
--- a/notes.md
+++ b/notes.md
@@ -7,6 +7,8 @@
  * ~~`WebPage.parse_urls()` needs to compare startswith to base url~~
  * ~~ignore any links which aren't to pages~~
  * better url checking to get bare domain
+  * remove trailing slash from any discovered url
+  * investigate lxml parser
  * ~~remove base url from initial urls with and without trailing slash~~
  * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls
  * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~
Author	SHA1	Message	Date
Simon Weald	164239b343	more thoughts	2018-09-06 17:31:12 +01:00
Simon Weald	ce1f2745c9	update thoughts	2018-09-06 17:30:28 +01:00