For a web developer one of the most frustrating things should be finding your site is crawling because of the bots or search engine crawlers doing their job so hard. In this site's case, they (bots, crawlers) gets intense suddenly, and sometimes they don't want to give up for a few hours. I've rate limiting code for crawlers, but seems like it doesn't work in every case.
Why bots can bring your site down?
In my case, it's probably because the "valid" URLs are plenty, even if their content is almost same. For example take this URL:
http://defkey.com/fr/some-program-raccourcis
Bots are smart enough to replace "fr" part with some other language and try to get many pages with it. I want to return English content, if translated one isn't available, and that simply makes every possible language URL "available". Similarly, they can do it with query strings. That's simply disaster when they clog it with a few requests per second.
For Cloudflare users, the solution is...
Cloudflare's "under attack" mode immediately mitigates this by putting a captcha for site entry, although it'll probably hurt your regular visitor counts. So only keep it on until you're sure bots are gone or you've taken another measure to get rid of the bot "attack".
How to turn on Cloudflare's Under Attack mode programatically?
Here is my code that worked, hopefully save you from long minutes of fiddling:
using Microsoft.Extensions.Configuration;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
public interface ICloudFlareService
{
Task<CloudFlareSecurityLevel> GetSecurityLevel();
Task<bool> SetSecurityLevel(CloudFlareSecurityLevel mode);
}
public class CloudFlareService(IConfiguration configuration, IHttpClientFactory httpClientFactory) : ICloudFlareService
{
private readonly string _zoneId = configuration["CloudFlare:ZoneId"];
private readonly string _apiToken = configuration["CloudFlare:ApiToken"];
/// <summary>
/// CloudFlare security levels. Don't rename these values, they are used in the CloudFlare API.
/// </summary>
public enum CloudFlareSecurityLevel
{
unknown,
off,
essentially_off,
low,
medium,
high,
under_attack,
}
public async Task<bool> SetSecurityLevel(CloudFlareSecurityLevel mode)
{
string apiUriBase = $"https://api.cloudflare.com/client/v4/zones/{_zoneId}/settings/security_level";
try
{
var client = httpClientFactory.CreateClient();
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", _apiToken);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var payload = new
{
value = mode.ToString()
};
var content = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");
var response = await client.PatchAsync(apiUriBase, content);
var result = await response.Content.ReadAsStringAsync();
if (response.IsSuccessStatusCode)
{
return true;
}
else
{
string message = $"Failed to set CloudFlare security level to {mode}. Status Code: {response.StatusCode}, Response: {result}";
return false;
}
}
catch (Exception ex)
{
string message = $"Error setting CloudFlare security level to {mode}. Error: {ex.Message}";
return false;
}
}
public async Task<CloudFlareSecurityLevel> GetSecurityLevel()
{
string apiUriBase = $"https://api.cloudflare.com/client/v4/zones/{_zoneId}/settings/security_level";
try
{
var client = httpClientFactory.CreateClient();
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", _apiToken);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = await client.GetAsync(apiUriBase);
var result = await response.Content.ReadAsStringAsync();
if (response.IsSuccessStatusCode)
{
var jsonDocument = JsonDocument.Parse(result);
var securityLevel = jsonDocument.RootElement
.GetProperty("result")
.GetProperty("value")
.GetString();
if (Enum.TryParse(securityLevel, out CloudFlareSecurityLevel parsedLevel))
{
return parsedLevel;
}
else
{
string message = $"Unexpected security level value: {securityLevel}";
return CloudFlareSecurityLevel.unknown;
}
}
else
{
string message = $"Failed to get CloudFlare security level. Status Code: {response.StatusCode}, Response: {result}";
return CloudFlareSecurityLevel.unknown;
}
}
catch (Exception ex)
{
string message = $"Error getting CloudFlare security level. Error: {ex.Message}";
return CloudFlareSecurityLevel.unknown;
}
}
}
Remember to add your ZoneId and ApiToken into "configuration". In my case it's the Secrets Manager in local, and environment variables in production. You can put them directly in your code, but that's probably not a good idea for most people.
Don't forget to register ICloudFlareService in your Startup.cs's ConfigureServices method:
services.AddScoped<ICloudFlareService, CloudFlareService>();
You can now call it like in this example, providing that you injected ICloudFlareService into your class:
// Check if already in under attack mode
var currentSecurityLevel = await cloudFlareService.GetSecurityLevel();
if (currentSecurityLevel == CloudFlareService.CloudFlareSecurityLevel.under_attack)
{
//Already in under attack mode. Skipping check.
}
else
{
await cloudFlareService.SetSecurityLevel(CloudFlareService.CloudFlareSecurityLevel.under_attack);
}Conclusion
While this doesn't automate the toggling of Security Level in Cloudflare, it makes the setting reachable to your code. From there, you can apply your logic (I implemented the automation rather easily, by creating a background task, and checking a "current requests" number. So that's possible, if you're determined to automate it). I recommend you to send yourself an email or notification when you set this mode is on, to remind you it's on.
