3 Commits

Author	SHA1	Message	Date
Rasmus	3034e66126	Changed GET to HEAD for checking robots txt.	2024-11-29 11:02:51 +01:00
Rasmus	f02203c14c	Fixed duplicate insertions into the filtered database.	2024-11-27 15:53:00 +01:00
Rasmus	f3c6338a6a	Initial commit	2024-11-24 12:40:51 +01:00