|
|
8698c21fda
|
return from WebPage to indicate whether a link was actually crawlable and only actually crawl it if it was
|
2018-09-12 08:03:08 +01:00 |
|
|
|
f16f82fdfb
|
improved completion message
|
2018-09-09 22:40:42 +01:00 |
|
|
|
a523154848
|
display count of crawled/uncrawled URLs whilst running
|
2018-09-09 22:35:55 +01:00 |
|
|
|
d1c1e17f4f
|
report runtime of script in generated sitemap
|
2018-09-06 17:20:59 +01:00 |
|
|
|
84ab27a75e
|
render results as HTML
|
2018-09-06 17:08:26 +01:00 |
|
|
|
a3ec9451e3
|
implement parsing of robots.txt
|
2018-09-05 18:56:20 +01:00 |
|
|
|
1b9b207a28
|
attempt to remove base url with trailing slash (if discovered)
|
2018-09-04 13:57:52 +01:00 |
|
|
|
05e907ecec
|
too many changes to make a sensible commit message
|
2018-09-04 09:21:26 +01:00 |
|
|
|
c436016e0c
|
remove unecessary function
|
2018-08-31 19:16:08 +01:00 |
|
|
|
0517e5bc56
|
crawler now initialises and populates crawled pool with urls it finds
|
2018-08-31 19:02:21 +01:00 |
|
|
|
5e0d9fd568
|
initial commit of crawler skeleton
|
2018-08-31 18:26:49 +01:00 |
|
