How to defend your sites from AI bots — David Mytton, Arcjet
Source: YouTube · AI Engineer · published Jul 30, 2025 · 20:13
AI-driven bots are significantly increasing automated traffic to websites, exacerbating cost and performance issues. This trend, especially with AI crawlers like OpenAI's GPTBOT, is making it harder to distinguish between beneficial and malicious traffic. 2:30
Key Takeaways:
• 50% of web traffic is now automated, with gaming sites seeing up to 60%, and AI crawlers are worsening this trend 0:42.
• AI crawlers like GPTBOT account for 24% of Diaspora’s traffic and 35% of Wikipedia’s, significantly increasing bandwidth costs 2:56.
• Traditional defenses like robots.txt and user-agent strings are insufficient, as malicious bots can spoof identities 8:08.
• IP reputation, proof of work, and client fingerprinting (e.g., J4 hash) are more effective in identifying and blocking automated clients 13:01.
• Rate limiting based on fingerprints—rather than IP addresses—provides better control over legitimate and malicious traffic 18:45.
These defenses should be layered, starting with robots.txt and user-agent checks, then progressing to IP analysis and fingerprinting for high-risk sites. 19:44
Sources:
Generate CPE Credits
Generate a professional CPE document from this video's transcript.
Estimated credit: 0.5 CPE hours
Estimate uses the video runtime (1 hour ≈ 1 CPE, rounded to the nearest 0.5, minimum 0.5, maximum 2.0). The final amount can be lower after review, never higher.
Topic: Cybersecurity. Commonly maps to: Security and Risk Management, Security Operations. Exact CISSP domains are assigned during generation.
Free account. One generation at a time, with a daily limit.
CPEBuddy is independent and not affiliated with or endorsed by ISC2, ISACA, or any certification body. Exports are formatted for common CPE submissions; acceptance is at your certification body's discretion.
Transcript Preview
First 800 characters of the transcript
[Music] Hi everyone. So my name is David. I'm the founder of ArtJet. We provide a security SDK for developers. So everything I'm going to be talking to you about today is what we've been building for the last few years, but how you can do it yourself. So, if you haven't had bots visiting your website and felt the pain, then you might be thinking, well, is this really a problem? Well, as as you just heard in the introduction, almost 50% of web traffic today is automated clients. And that varies depending on the industry. In gaming, that's almost 60% of all traffic is automated. And that's before the agent revolution has really kicked off. This isn't a new problem. It's been going on since the invention of the internet and there are bots that you want to visit your website like Google bot bu…