display count of crawled/uncrawled URLs whilst running

2018-09-09 22:35:55 +01:00
parent 9e754a5584
commit a523154848
2 changed files with 6 additions and 3 deletions
--- a/notes.md
+++ b/notes.md
@@ -13,6 +13,6 @@
  * ~~investigate using [tldextract](https://github.com/john-kurkowski/tldextract) to match urls~~ #wontfix
  * ~~implement parsing of [robots.txt](http://docs.w3cub.com/python~3.6/library/urllib.robotparser/)~~
  * ~~investigate [gzip encoding](https://stackoverflow.com/questions/36383227/avoid-downloading-images-using-beautifulsoup-and-urllib-request)~~
-  * implement some kind of progress display
+  * ~~implement some kind of progress display~~
  * async
  * better exception handling