add scratchpad for notes

2018-08-28 22:34:05 +01:00
parent 5d94991167
commit fb096b4468
1 changed files with 9 additions and 0 deletions
--- a/notes.md
+++ b/notes.md
@@ -0,0 +1,9 @@
 ## Thoughts
 ###### for each URL, do the following:
 * mark it as crawled
 * get page content
   * if that fails, mark the link as invalid
 * find all links in the content
   * check each link for dupes
   * add to pool or discard