simon

0 Followers · 0 Following

• Joined on 2018-08-15

Repositories

2

Projects Packages Public Activity Starred Repositories

simon pushed to asyncio at misc/web-scraper 2018-09-16 08:21:50 +00:00

f1855f5add re-order imports because I'm fussy

336517e84a more documentation and add back some required imports

7bc9fe0679 improved documentation and remove unneeded set

Compare 3 commits »

simon pushed to asyncio at misc/web-scraper 2018-09-15 20:48:52 +00:00

6548f55416 improve documentation

0244435fea remove unecessary imports

Compare 2 commits »

simon pushed to asyncio at misc/web-scraper 2018-09-15 20:30:04 +00:00

d6964672b6 commit of working async crawler

simon pushed to asyncio at misc/web-scraper 2018-09-14 15:06:18 +00:00

3808f72f73 correct semaphore usage

simon pushed to asyncio at misc/web-scraper 2018-09-14 15:02:24 +00:00

7ebe4855b8 remove unecessary classes2

simon pushed to asyncio at misc/web-scraper 2018-09-14 15:01:20 +00:00

db986b0eba async crawler in a mostly-working state

simon pushed to asyncio at misc/web-scraper 2018-09-12 21:54:20 +00:00

36e1f7693f initial foray into asynchronous crawling

8698c21fda return from WebPage to indicate whether a link was actually crawlable and only actually crawl it if it was

Compare 2 commits »

simon pushed to master at misc/web-scraper 2018-09-12 07:03:30 +00:00

8698c21fda return from WebPage to indicate whether a link was actually crawlable and only actually crawl it if it was

simon pushed to asyncio at misc/web-scraper 2018-09-11 12:42:43 +00:00

simon pushed to master at misc/web-scraper 2018-09-11 12:42:19 +00:00

273cf56a3b add some basic tests

1af26f50f2 added a docstring

Compare 2 commits »

simon pushed to master at misc/web-scraper 2018-09-10 20:29:49 +00:00

c40c5cea50 add async info

simon pushed to master at misc/web-scraper 2018-09-10 19:56:14 +00:00

a6224f9b6a updated readme

simon pushed to master at misc/web-scraper 2018-09-10 10:58:59 +00:00

b64711973f add new thoughts

simon pushed to master at misc/web-scraper 2018-09-09 21:49:58 +00:00

9e125dfae0 added comments and docstrings

simon pushed to master at misc/web-scraper 2018-09-09 21:40:43 +00:00

f16f82fdfb improved completion message

simon pushed to master at misc/web-scraper 2018-09-09 21:35:59 +00:00

a523154848 display count of crawled/uncrawled URLs whilst running

simon pushed to master at misc/web-scraper 2018-09-09 10:21:47 +00:00

9e754a5584 improve handling of gzip/deflated data detection

simon pushed to master at misc/web-scraper 2018-09-09 09:53:10 +00:00

1b005570ee implement gzip compression requests and handling

17fa9f93f9 tick off gzip encoding

Compare 2 commits »

simon pushed to master at misc/web-scraper 2018-09-09 09:22:22 +00:00

1e51e10db2 update with changes

simon pushed to master at misc/web-scraper 2018-09-09 09:22:04 +00:00

225fd8b3ea update with changes