|
|
164239b343
|
more thoughts
|
2018-09-06 17:31:12 +01:00 |
|
|
|
ce1f2745c9
|
update thoughts
|
2018-09-06 17:30:28 +01:00 |
|
|
|
e57a86c60a
|
only attempt to read html
|
2018-09-06 16:30:11 +01:00 |
|
|
|
a3ec9451e3
|
implement parsing of robots.txt
|
2018-09-05 18:56:20 +01:00 |
|
|
|
f2c294ebdb
|
added new ideas to implement
|
2018-09-04 15:40:11 +01:00 |
|
|
|
1b9b207a28
|
attempt to remove base url with trailing slash (if discovered)
|
2018-09-04 13:57:52 +01:00 |
|
|
|
6abe7d68e0
|
updated notes
|
2018-09-04 12:51:59 +01:00 |
|
|
|
fb096b4468
|
add scratchpad for notes
|
2018-08-28 22:34:05 +01:00 |
|
