 simon
|
51f988e1bc
|
added more tests
|
2018-09-17 21:44:20 +01:00 |
|
|
simon
|
73c21e5bd3
|
small improvements to docs and variables
|
2018-09-17 21:44:04 +01:00 |
|
|
simon
|
eb2395d461
|
minor change to README
|
2018-09-17 08:11:26 +01:00 |
|
|
simon
|
c53f62b55d
|
add most changes suggested by pycodestyle
|
2018-09-16 16:10:38 +01:00 |
|
|
simon
|
75d3756bbc
|
fix errors discovered by pycyodestyle
|
2018-09-16 16:04:07 +01:00 |
|
|
simon
|
5262c23281
|
add flags to README
|
2018-09-16 15:58:17 +01:00 |
|
|
simon
|
524f6a45cd
|
improve documentation
|
2018-09-16 15:53:47 +01:00 |
|
|
simon
|
a926090bed
|
update requirements
|
2018-09-16 15:44:30 +01:00 |
|
|
simon
|
91cd988f52
|
more comments and progress output
|
2018-09-16 15:26:49 +01:00 |
|
|
simon
|
f1855f5add
|
re-order imports because I'm fussy
|
2018-09-16 09:06:30 +01:00 |
|
|
simon
|
336517e84a
|
more documentation and add back some required imports
|
2018-09-16 09:00:43 +01:00 |
|
|
simon
|
7bc9fe0679
|
improved documentation and remove unneeded set
|
2018-09-16 08:56:44 +01:00 |
|
|
simon
|
6548f55416
|
improve documentation
|
2018-09-15 21:48:50 +01:00 |
|
|
simon
|
0244435fea
|
remove unecessary imports
|
2018-09-15 21:38:51 +01:00 |
|
|
simon
|
d6964672b6
|
commit of working async crawler
|
2018-09-15 21:30:02 +01:00 |
|
|
simon
|
3808f72f73
|
correct semaphore usage
|
2018-09-14 16:06:17 +01:00 |
|
|
simon
|
7ebe4855b8
|
remove unecessary classes2
|
2018-09-14 16:02:20 +01:00 |
|
|
simon
|
db986b0eba
|
async crawler in a mostly-working state
|
2018-09-14 16:01:12 +01:00 |
|
|
simon
|
36e1f7693f
|
initial foray into asynchronous crawling
|
2018-09-12 22:54:12 +01:00 |
|
|
simon
|
8698c21fda
|
return from WebPage to indicate whether a link was actually crawlable and only actually crawl it if it was
|
2018-09-12 08:03:08 +01:00 |
|
|
simon
|
273cf56a3b
|
add some basic tests
|
2018-09-11 13:42:15 +01:00 |
|
|
simon
|
1af26f50f2
|
added a docstring
|
2018-09-11 13:42:02 +01:00 |
|
|
simon
|
c40c5cea50
|
add async info
|
2018-09-10 21:29:46 +01:00 |
|
|
simon
|
a6224f9b6a
|
updated readme
|
2018-09-10 20:56:12 +01:00 |
|
|
simon
|
b64711973f
|
add new thoughts
|
2018-09-10 11:58:58 +01:00 |
|
|
simon
|
9e125dfae0
|
added comments and docstrings
|
2018-09-09 22:49:55 +01:00 |
|
|
simon
|
f16f82fdfb
|
improved completion message
|
2018-09-09 22:40:42 +01:00 |
|
|
simon
|
a523154848
|
display count of crawled/uncrawled URLs whilst running
|
2018-09-09 22:35:55 +01:00 |
|
|
simon
|
9e754a5584
|
improve handling of gzip/deflated data detection
|
2018-09-09 11:21:46 +01:00 |
|
|
simon
|
1b005570ee
|
implement gzip compression requests and handling
|
2018-09-09 10:53:09 +01:00 |
|
|
simon
|
17fa9f93f9
|
tick off gzip encoding
|
2018-09-09 10:52:37 +01:00 |
|
|
simon
|
1e51e10db2
|
update with changes
|
2018-09-09 10:22:18 +01:00 |
|
|
simon
|
225fd8b3ea
|
update with changes
|
2018-09-09 10:22:03 +01:00 |
|
|
simon
|
d686ae0bc4
|
update with changes
|
2018-09-09 10:21:45 +01:00 |
|
|
simon
|
69f5788745
|
update notes
|
2018-09-09 10:16:22 +01:00 |
|
|
simon
|
b5d644a223
|
various minor improvements to exception handling
|
2018-09-09 10:16:03 +01:00 |
|
|
simon
|
6508156aa4
|
use lxml as the parser and only find links on a page if we've got the source
|
2018-09-09 10:06:25 +01:00 |
|
|
simon
|
738ab8e441
|
adjust robots handling to deal with 404s and enforce a user agent which allows us to initially obtain the user agent
|
2018-09-09 09:57:16 +01:00 |
|
|
simon
|
fdd84a8786
|
manually retrieve robots.txt to ensure we can set the user-agent
|
2018-09-07 12:40:12 +01:00 |
|
|
simon
|
ab0ab0a010
|
add more thoughts
|
2018-09-07 11:50:53 +01:00 |
|
|
simon
|
6a1259aa7d
|
update plans to add gzip encoding
|
2018-09-06 17:33:10 +01:00 |
|
|
simon
|
164239b343
|
more thoughts
|
2018-09-06 17:31:12 +01:00 |
|
|
simon
|
ce1f2745c9
|
update thoughts
|
2018-09-06 17:30:28 +01:00 |
|
|
simon
|
e70bdc9ca1
|
update requirements.txt
|
2018-09-06 17:25:30 +01:00 |
|
|
simon
|
d1c1e17f4f
|
report runtime of script in generated sitemap
|
2018-09-06 17:20:59 +01:00 |
|
|
simon
|
816a727d79
|
ignore generated file
|
2018-09-06 17:08:56 +01:00 |
|
|
simon
|
84ab27a75e
|
render results as HTML
|
2018-09-06 17:08:26 +01:00 |
|
|
simon
|
6d9103c154
|
improved content-type detection
|
2018-09-06 17:08:12 +01:00 |
|
|
simon
|
e57a86c60a
|
only attempt to read html
|
2018-09-06 16:30:11 +01:00 |
|
|
simon
|
a3ec9451e3
|
implement parsing of robots.txt
|
2018-09-05 18:56:20 +01:00 |
|
|
simon
|
f2c294ebdb
|
added new ideas to implement
|
2018-09-04 15:40:11 +01:00 |
|
|
simon
|
1b9b207a28
|
attempt to remove base url with trailing slash (if discovered)
|
2018-09-04 13:57:52 +01:00 |
|
|
simon
|
6abe7d68e0
|
updated notes
|
2018-09-04 12:51:59 +01:00 |
|
|
simon
|
7d919039b6
|
removed unecessary modules
|
2018-09-04 10:14:27 +01:00 |
|
|
simon
|
0726bcccb0
|
removed original file
|
2018-09-04 09:21:55 +01:00 |
|
|
simon
|
05e907ecec
|
too many changes to make a sensible commit message
|
2018-09-04 09:21:26 +01:00 |
|
|
simon
|
abc628106d
|
added a docstring to the WebPage object
|
2018-08-31 19:18:00 +01:00 |
|
|
simon
|
c436016e0c
|
remove unecessary function
|
2018-08-31 19:16:08 +01:00 |
|
|
simon
|
03554fde80
|
add docstrings
|
2018-08-31 19:15:35 +01:00 |
|
|
simon
|
759f965e95
|
use more explicit names, use urljoin to combine urls
|
2018-08-31 19:12:58 +01:00 |
|
|
simon
|
0517e5bc56
|
crawler now initialises and populates crawled pool with urls it finds
|
2018-08-31 19:02:21 +01:00 |
|
|
simon
|
1b18aa83eb
|
corrected some small errors and added runner function
|
2018-08-31 19:01:35 +01:00 |
|
|
simon
|
5e0d9fd568
|
initial commit of crawler skeleton
|
2018-08-31 18:26:49 +01:00 |
|
|
simon
|
915def3a5d
|
rework url sanitiser to use urllib modules, move WebPage object to helpers
|
2018-08-31 18:26:25 +01:00 |
|
|
simon
|
453331d69d
|
simplified url qualifier
|
2018-08-29 22:27:26 +01:00 |
|
|
simon
|
2b812da26a
|
simplify UrlPoolManager to use a set instead of a dict
|
2018-08-29 21:49:15 +01:00 |
|
|
simon
|
fb096b4468
|
add scratchpad for notes
|
2018-08-28 22:34:05 +01:00 |
|
|
simon
|
5d94991167
|
start making the scraper an object
|
2018-08-28 22:29:36 +01:00 |
|
|
simon
|
482d23dd4f
|
blank __init__.py
|
2018-08-28 22:29:11 +01:00 |
|
|
simon
|
452de87f35
|
change name of pool management object to be more clear
|
2018-08-28 22:28:49 +01:00 |
|
|
simon
|
73cb883151
|
add a list manager object
|
2018-08-28 22:28:16 +01:00 |
|
|
simon
|
5c933fc5c9
|
initial commit of single-page scraper
|
2018-08-28 18:29:34 +01:00 |
|
|
simon
|
25f8c4c686
|
remove testing url with requests and assume that the user is correct
|
2018-08-28 17:22:52 +01:00 |
|
|
simon
|
0d0438670c
|
adjusted title
|
2018-08-28 09:12:48 +01:00 |
|
|
simon
|
8a1fd39dc4
|
added pycache dirs
|
2018-08-27 19:38:13 +01:00 |
|
|
simon
|
79b10798a3
|
initial commit of utils
|
2018-08-27 19:37:41 +01:00 |
|
|
simon
|
fb6b976391
|
initial commit of utils tests
|
2018-08-27 19:36:43 +01:00 |
|
|
simon
|
a04de7f4de
|
changed venv name
|
2018-08-27 14:28:20 +01:00 |
|
|
simon
|
665ec1d7a7
|
add readme
|
2018-08-23 16:05:24 +01:00 |
|
|
simon
|
65fc332925
|
ignore venv and vscode dirs
|
2018-08-23 16:03:46 +01:00 |
|
|
simon
|
c6ce63838f
|
bare script file
|
2018-08-23 16:02:09 +01:00 |
|
|
simon
|
c383fb7ee9
|
initial requirements file
|
2018-08-23 15:59:18 +01:00 |
|
|
simon
|
01a16a998c
|
initial gitignore
|
2018-08-23 15:47:30 +01:00 |
|