Make The HttpHelper Use HEAD Method For Robots TXT #20

New Issue

owner · 2024-11-28T19:22:26Z

owner commented

2024-11-28 19:22:26 +00:00

		Backend/Helper/HttpClientHelper.cs
		Line 58 in 701ffff27e
	
				            response = await client.GetAsync("/robots.txt");

There is no reason for us to download the whole robots.txt to know that it exists. We should just do a HEAD method instead of GET, so it goes a little faster to complete this step in the filtering process. Plus it should help with the GC as well.

https://git.rbwr.dk/owner/RSE/src/commit/701ffff27e54926c6fd6042f90c1f9ad01c955b4/Backend/Helper/HttpClientHelper.cs#L58 There is no reason for us to download the whole robots.txt to know that it exists. We should just do a HEAD method instead of GET, so it goes a little faster to complete this step in the filtering process. Plus it should help with the GC as well.

owner added this to the Minimize GC Pressure milestone 2024-11-28 19:22:26 +00:00

owner added the

enhancement

label 2024-11-28 19:22:26 +00:00

owner self-assigned this 2024-11-28 19:22:26 +00:00

owner added reference main

2024-11-29 08:22:50 +00:00

owner closed this issue

2024-11-29 10:03:24 +00:00

Sign in to join this conversation.