add talking points

notes.md

@@ -18,39 +18,9 @@
   * better exception handling
   * randomise output filename
 
-### Async bits
+### talking points
 
-in `__main__`:
+ - token bucket algo to enforce n requests per second
-
+   - read up on bucket algo types
-```python
+ - re-structuring AsyncCrawler to be more testable
-loop = asyncio.get_event_loop()
+ - use exponential backoff algo?
-try:
-    loop.run_until_complete(main())
-finally:
-    loop.close()
-```
-
-  * initialises loop and runs it to completion
-  * needs to handle errors (try/except/finally)
-
-```python
-async def run(args=None):
-    tasks = []
-
-    for url in pool:
-        tasks.append(url)
-    # for i in range(10):
-        # tasks.append(asyncio.ensure_future(myCoroutine(i)))
-
-    # gather completed tasks
-    await asyncio.gather(*tasks)
-```
-
-Getting the contents of the page needs to be async too
-
-```python
-async def get_source():
-    blah
-    blah
-    await urlopen(url)
-```