 owner
|
e9fd738f1f
|
Title and description extraction now only look in the head section of the HTML document
|
2024-12-23 14:09:51 +01:00 |
|
|
owner
|
580d687f62
|
Add user-agent
|
2024-12-21 21:57:01 +01:00 |
|
|
owner
|
5fe35192c6
|
Misc changes such as performance tuning, added support for more than 64 threads for the scanner.
|
2024-12-04 13:20:45 +01:00 |
|
|
owner
|
3034e66126
|
Changed GET to HEAD for checking robots txt.
|
2024-11-29 11:02:51 +01:00 |
|
|
owner
|
f02203c14c
|
Fixed duplicate insertions into the filtered database.
|
2024-11-27 15:53:00 +01:00 |
|
|
owner
|
be07cdb13e
|
Create runtime variables and remove MessagePack.
|
2024-11-26 19:40:27 +01:00 |
|
|
owner
|
f3c6338a6a
|
Initial commit
|
2024-11-24 12:40:51 +01:00 |
|