add scratchpad for notes

2018-08-28 22:34:05 +01:00
parent 5d94991167
commit fb096b4468
1 changed files with 9 additions and 0 deletions
--- a/notes.md
+++ b/notes.md
@@ -0,0 +1,9 @@
+## Thoughts
+
+###### for each URL, do the following:
+ * mark it as crawled
+ * get page content
+   * if that fails, mark the link as invalid
+ * find all links in the content
+   * check each link for dupes
+   * add to pool or discard