![\"A](\"https://vic.demuzere.be/assets/articles/images/TRYl3DRF0rsyvqqnV0drAUMoq12Qb1Mn.jpg\")
The last few years, web traffic from bots has increased rapidly.\nSome sources1 claim that automated activity now accounts for over half of all web traffic.\nThis is even more pronounced on small websites like mine, as I don't receive many human visitors.
\nThe table below shows traffic on this website for the last few days:
\n| \n | Path | \nHits | \n
|---|---|---|
| 1 | \n/wp-login.php | \n2751 | \n
| 2 | \n/wp-admin/index.php | \n2707 | \n
| 3 | \n/wp-admin/edit.php | \n2354 | \n
| 4 | \n/wp-admin/profile.php | \n2353 | \n
| 5 | \n/wp-admin/plugins.php | \n2351 | \n
| 6 | \n/robots.txt | \n1131 | \n
| 7 | \n/ | \n994 | \n
| 8 | \n/articles/feed.rss | \n989 | \n
| 9 | \n/articles/how-i-almost-lost-my-backups/ | \n314 | \n
| 10 | \n/wp-content/themes/vic/style.css | \n299 | \n
This clearly shows two things:
\nThat is to say, my own statistics show that I'm mainly serving our robot overlords.
\nThe situation is worse on other kinds of web apps, like my Forgejo instance.\nIt's a single-user instance, and most projects are private, but I do publish a bunch of open-source projects there.\nRecent bot traffic on applications like these is causing significant load on my server.
\nPreviously, the majority of crawlers were relatively decent.\nThey would crawl your website slowly, making sure not to cause too much load.\nTheir main purpose was serving search results to people looking for the content you publish.\nBringing down your website would only result in their own users landing on error pages.
\nThe recent Generative AI (GenAI) craze changed this completely.\nMultiple companies are now racing to collect as much data as possible.\nThey no longer care what happens with the websites they're crawling.\nOnce they have collected enough content, their GenAI agents provide users with answers directly.\nThe source website is treated as irrelevant.\nObviously, as a website owner, I do not want these thankless robots on my property3!
\nSo I went through the standard ways of blocking them:
\nrobots.txt.None of these methods work.
\nGenAI companies frequently ignore robots.txt files.\nThey also usually disguise themselves as normal browsers instead of identifying as crawlers.\nMore recently, I've also noticed them increasingly using consumer IP space.\nCombined with rapidly changing addresses, this makes it impossible to block them by IP address alone.
There are some promising new tools like Anubis that work4, for now.
\nMaking sure websites stay online is part of my job.\nI don't want to have to go through the same hassle for my own web apps when I get home.\nI'm tired, and the few things I'm hosting just aren't worth it.
\nSo, I decided to take two different approaches:
\nWebsites that are meant to be public, like the one you're reading now.\nI've always optimized these for high traffic, and will continue to do so.\nThese are built to withstand much more than the AI crawlers I'm currently seeing.\nI'm no longer blocking any bots on public websites.
\nWebsites (and web apps) that are primarily meant for myself.\nThese are now geoblocked and only accept traffic from Belgium.\nAs these are mainly open source applications that I don't maintain myself, they are harder to optimize.\nThis includes my Forgejo instance.\nIt's sad that part of my open source code disappears behind a firewall.\nI hope to open it up again once AI companies start behaving more responsibly.
\nMy geoblock page contains contact information.\nSo if people really want to go through the hassle of requesting access (which I very much doubt), they can be allowed in.
\nEvery article I've found claiming this reads like marketing material for a web security firm.\nFor that reason, I decided not to link to any of them.\nBased on my own traffic stats though, I don't doubt their numbers. ↩
\nI do read robots.txt on some sites I visit.\nIt's a treasure trove of information about the website you're visiting.\nBut that's probably not normal human behaviour. ↩
\nI don't really mind AI crawlers, as long as they behave.\nThey usually don't. ↩
\nAnubis is great!\nEspecially since it lets through the bots that clearly identify as a bot. ↩
\nSecurity on web services has improved a lot lately. Most of them have switched to being available over HTTPS only, use decent hashing and encryption for user credentials and support two-factor authentication by now. Authy provides an easy to use two-factor solution for organizations. However, this seems to be a bit of a poisoned gift.
\nOnline authentication used to rely on remembering a password. This is by far the easiest solution, but it has a few drawbacks. People are bad at remembering things, especially if they don't use them every day. This results in passwords being written down and attached to computer screens and keyboards. Even worse is that most people use the same password for every site they visit.
\nUsing the same password on multiple sites becomes a problem when one of them leaks user credentials. Trying out the leaked username/e-mail and password combinations may give access to a lot of other sites, creating an avalanche effect.
\nTwo-factor authentication attempts to solve this problem by combining the password with a physical item. A mobile device or hardware token generates an access code, usable for a limited time only. This means that someone who knows your password is still unable to access your account. Great!
\n
An authenticator app generates an access code based on a secret and the current time. This code changes every 30 seconds, but can usually be used for about 5 minutes. Every site gives you a unique secret, and thus you can't use a single code for multiple sites.
\nNotice that I talk about "an authenticator app" and not a specific application? These services support the TOTP or HOTP standard. There are a lot of applications that allow you to generate these access codes. The most well-known is the Google Authenticator, but there are many more! On every platform and for every device. Even if you're using a 10 year old smartphone, chances are you'll find a suitable two-factor application.
\nAuthy provides a solution for websites that makes it easy to implement two-factor authentication. This is a good thing as it allows more organisations to use 2FA. Their own authenticator app provides a few extra features too. Unfortunately, in order to do this, they decided to take an open standard and completely break it. They deliberately prevent you from using any other app than their own. As usual, this is a one way situation: you can import other accounts to authy, but not the other way around.
\nWhen asked about other apps, their customer support replied:
\n\n\nAre you asking about using Authy-powered tokens with third-party apps? If so, we do not support that feature, but many of our clients support authentication via SMS or phone calls.
\n
Sure, their own app has a few amazing features. It allows you to backup an encrypted copy of your secret keys to the cloud, for example. But is that worth it? What if you're using a platform that doesn't have an Authy app? As more organisations switch to Authy, they are effectively ending the freedom of choice when it comes to authenticator apps.
\nIs this a bad thing? Up to you to decide.
\n","url":"https://vic.demuzere.be/articles/how-authy-cripples-my-2fa-experience/","date_published":"2015-03-11T17:46:16Z","date_modified":"2026-05-13T17:23:33Z","language":"en"}]}