|
|
a3ec9451e3
|
implement parsing of robots.txt
|
2018-09-05 18:56:20 +01:00 |
|
|
|
f2c294ebdb
|
added new ideas to implement
|
2018-09-04 15:40:11 +01:00 |
|
|
|
1b9b207a28
|
attempt to remove base url with trailing slash (if discovered)
|
2018-09-04 13:57:52 +01:00 |
|
|
|
6abe7d68e0
|
updated notes
|
2018-09-04 12:51:59 +01:00 |
|
|
|
fb096b4468
|
add scratchpad for notes
|
2018-08-28 22:34:05 +01:00 |
|
