diff --git a/notes.md b/notes.md index 4b133db..9adf763 100644 --- a/notes.md +++ b/notes.md @@ -8,6 +8,7 @@ * ~~ignore any links which aren't to pages~~ * better url checking to get bare domain * remove trailing slash from any discovered url + * investigate lxml parser * ~~remove base url from initial urls with and without trailing slash~~ * investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~