Commit Graph

Select branches

Hide Pull Requests

asyncio

master

Mono Color

• 5f7d66912f add test files asyncio Simon Weald 2018-09-19 08:39:05 +01:00
• d4cd93e3d4 update docs Simon Weald 2018-09-19 08:38:49 +01:00
• f5f6afd1a4 correct tests with new arg names Simon Weald 2018-09-19 08:37:55 +01:00
• 679b1b7b53 rename all instances of base_url to rooturl, add more documentation Simon Weald 2018-09-18 18:24:15 +01:00
• 32d7f1e54b add talking points Simon Weald 2018-09-18 18:23:12 +01:00
• f6265f18a7 initial test for AsyncCrawler Simon Weald 2018-09-18 18:22:55 +01:00
• 9a4e9ddfc7 add test for missing robots.txt Simon Weald 2018-09-18 10:53:13 +01:00
• 51f988e1bc added more tests Simon Weald 2018-09-17 21:44:20 +01:00
• 73c21e5bd3 small improvements to docs and variables Simon Weald 2018-09-17 21:44:04 +01:00
• eb2395d461 minor change to README Simon Weald 2018-09-17 08:11:26 +01:00
• c53f62b55d add most changes suggested by pycodestyle Simon Weald 2018-09-16 16:10:38 +01:00
• 75d3756bbc fix errors discovered by pycyodestyle Simon Weald 2018-09-16 16:04:07 +01:00
• 5262c23281 add flags to README Simon Weald 2018-09-16 15:58:17 +01:00
• 524f6a45cd improve documentation Simon Weald 2018-09-16 15:53:47 +01:00
• a926090bed update requirements Simon Weald 2018-09-16 15:44:30 +01:00
• 91cd988f52 more comments and progress output Simon Weald 2018-09-16 15:26:49 +01:00
• f1855f5add re-order imports because I'm fussy Simon Weald 2018-09-16 09:06:30 +01:00
• 336517e84a more documentation and add back some required imports Simon Weald 2018-09-16 09:00:43 +01:00
• 7bc9fe0679 improved documentation and remove unneeded set Simon Weald 2018-09-16 08:56:44 +01:00
• 6548f55416 improve documentation Simon Weald 2018-09-15 21:48:50 +01:00
• 0244435fea remove unecessary imports Simon Weald 2018-09-15 21:38:51 +01:00
• d6964672b6 commit of working async crawler Simon Weald 2018-09-15 21:30:02 +01:00
• 3808f72f73 correct semaphore usage Simon Weald 2018-09-14 16:06:17 +01:00
• 7ebe4855b8 remove unecessary classes2 Simon Weald 2018-09-14 16:02:20 +01:00
• db986b0eba async crawler in a mostly-working state Simon Weald 2018-09-14 16:01:12 +01:00
• 36e1f7693f initial foray into asynchronous crawling Simon Weald 2018-09-12 22:54:12 +01:00
• 8698c21fda return from WebPage to indicate whether a link was actually crawlable and only actually crawl it if it was master Simon Weald 2018-09-12 08:00:08 +01:00
• 273cf56a3b add some basic tests Simon Weald 2018-09-11 13:42:15 +01:00
• 1af26f50f2 added a docstring Simon Weald 2018-09-11 13:42:02 +01:00
• c40c5cea50 add async info Simon Weald 2018-09-10 21:29:46 +01:00
• a6224f9b6a updated readme Simon Weald 2018-09-10 20:56:12 +01:00
• b64711973f add new thoughts Simon Weald 2018-09-10 11:58:58 +01:00
• 9e125dfae0 added comments and docstrings Simon Weald 2018-09-09 22:49:55 +01:00
• f16f82fdfb improved completion message Simon Weald 2018-09-09 22:40:42 +01:00
• a523154848 display count of crawled/uncrawled URLs whilst running Simon Weald 2018-09-09 22:35:55 +01:00
• 9e754a5584 improve handling of gzip/deflated data detection Simon Weald 2018-09-09 11:21:46 +01:00
• 1b005570ee implement gzip compression requests and handling Simon Weald 2018-09-09 10:53:09 +01:00
• 17fa9f93f9 tick off gzip encoding Simon Weald 2018-09-09 10:52:37 +01:00
• 1e51e10db2 update with changes Simon Weald 2018-09-09 10:22:18 +01:00
• 225fd8b3ea update with changes Simon Weald 2018-09-09 10:22:03 +01:00
• d686ae0bc4 update with changes Simon Weald 2018-09-09 10:21:45 +01:00
• 69f5788745 update notes Simon Weald 2018-09-09 10:16:22 +01:00
• b5d644a223 various minor improvements to exception handling Simon Weald 2018-09-09 10:16:03 +01:00
• 6508156aa4 use lxml as the parser and only find links on a page if we've got the source Simon Weald 2018-09-09 10:06:25 +01:00
• 738ab8e441 adjust robots handling to deal with 404s and enforce a user agent which allows us to initially obtain the user agent Simon Weald 2018-09-09 09:57:16 +01:00
• fdd84a8786 manually retrieve robots.txt to ensure we can set the user-agent Simon Weald 2018-09-07 12:40:12 +01:00
• ab0ab0a010 add more thoughts Simon Weald 2018-09-07 11:50:53 +01:00
• 6a1259aa7d update plans to add gzip encoding Simon Weald 2018-09-06 17:33:10 +01:00
• 164239b343 more thoughts Simon Weald 2018-09-06 17:31:12 +01:00
• ce1f2745c9 update thoughts Simon Weald 2018-09-06 17:30:28 +01:00
• e70bdc9ca1 update requirements.txt Simon Weald 2018-09-06 17:25:30 +01:00
• d1c1e17f4f report runtime of script in generated sitemap Simon Weald 2018-09-06 17:20:59 +01:00
• 816a727d79 ignore generated file Simon Weald 2018-09-06 17:08:56 +01:00
• 84ab27a75e render results as HTML Simon Weald 2018-09-06 17:08:26 +01:00
• 6d9103c154 improved content-type detection Simon Weald 2018-09-06 17:08:12 +01:00
• e57a86c60a only attempt to read html Simon Weald 2018-09-06 16:30:11 +01:00
• a3ec9451e3 implement parsing of robots.txt Simon Weald 2018-09-05 18:56:20 +01:00
• f2c294ebdb added new ideas to implement Simon Weald 2018-09-04 15:40:11 +01:00
• 1b9b207a28 attempt to remove base url with trailing slash (if discovered) Simon Weald 2018-09-04 13:57:52 +01:00
• 6abe7d68e0 updated notes Simon Weald 2018-09-04 12:51:59 +01:00
• 7d919039b6 removed unecessary modules Simon Weald 2018-09-04 10:14:27 +01:00
• 0726bcccb0 removed original file Simon Weald 2018-09-04 09:21:55 +01:00
• 05e907ecec too many changes to make a sensible commit message Simon Weald 2018-09-04 09:21:26 +01:00
• abc628106d added a docstring to the WebPage object Simon Weald 2018-08-31 19:18:00 +01:00
• c436016e0c remove unecessary function Simon Weald 2018-08-31 19:16:08 +01:00
• 03554fde80 add docstrings Simon Weald 2018-08-31 19:15:35 +01:00
• 759f965e95 use more explicit names, use urljoin to combine urls Simon Weald 2018-08-31 19:12:58 +01:00
• 0517e5bc56 crawler now initialises and populates crawled pool with urls it finds Simon Weald 2018-08-31 19:02:21 +01:00
• 1b18aa83eb corrected some small errors and added runner function Simon Weald 2018-08-31 19:01:35 +01:00
• 5e0d9fd568 initial commit of crawler skeleton Simon Weald 2018-08-31 18:26:49 +01:00
• 915def3a5d rework url sanitiser to use urllib modules, move WebPage object to helpers Simon Weald 2018-08-31 18:26:25 +01:00
• 453331d69d simplified url qualifier Simon Weald 2018-08-29 22:27:26 +01:00
• 2b812da26a simplify UrlPoolManager to use a set instead of a dict Simon Weald 2018-08-29 21:49:15 +01:00
• fb096b4468 add scratchpad for notes Simon Weald 2018-08-28 22:34:05 +01:00
• 5d94991167 start making the scraper an object Simon Weald 2018-08-28 22:29:36 +01:00
• 482d23dd4f blank __init__.py Simon Weald 2018-08-28 22:29:11 +01:00
• 452de87f35 change name of pool management object to be more clear Simon Weald 2018-08-28 22:28:49 +01:00
• 73cb883151 add a list manager object Simon Weald 2018-08-28 22:28:16 +01:00
• 5c933fc5c9 initial commit of single-page scraper Simon Weald 2018-08-28 18:29:34 +01:00
• 25f8c4c686 remove testing url with requests and assume that the user is correct Simon Weald 2018-08-28 17:22:52 +01:00
• 0d0438670c adjusted title Simon Weald 2018-08-28 09:12:48 +01:00
• 8a1fd39dc4 added pycache dirs Simon Weald 2018-08-27 19:38:13 +01:00
• 79b10798a3 initial commit of utils Simon Weald 2018-08-27 19:37:41 +01:00
• fb6b976391 initial commit of utils tests Simon Weald 2018-08-27 19:36:43 +01:00
• a04de7f4de changed venv name Simon Weald 2018-08-27 14:28:20 +01:00
• 665ec1d7a7 add readme Simon Weald 2018-08-23 16:05:24 +01:00
• 65fc332925 ignore venv and vscode dirs Simon Weald 2018-08-23 16:03:46 +01:00
• c6ce63838f bare script file Simon Weald 2018-08-23 16:02:09 +01:00
• c383fb7ee9 initial requirements file Simon Weald 2018-08-23 15:59:18 +01:00
• 01a16a998c initial gitignore Simon Weald 2018-08-23 15:47:30 +01:00