The last few years, web traffic from bots has increased rapidly.\nSome sources1 claim that automated activity now accounts for over half of all web traffic.\nThis is even more pronounced on small websites like mine, as I don't receive many human visitors.
\nThe table below shows traffic on this website for the last few days:
\n| \n | Path | \nHits | \n
|---|---|---|
| 1 | \n/wp-login.php | \n2751 | \n
| 2 | \n/wp-admin/index.php | \n2707 | \n
| 3 | \n/wp-admin/edit.php | \n2354 | \n
| 4 | \n/wp-admin/profile.php | \n2353 | \n
| 5 | \n/wp-admin/plugins.php | \n2351 | \n
| 6 | \n/robots.txt | \n1131 | \n
| 7 | \n/ | \n994 | \n
| 8 | \n/articles/feed.rss | \n989 | \n
| 9 | \n/articles/how-i-almost-lost-my-backups/ | \n314 | \n
| 10 | \n/wp-content/themes/vic/style.css | \n299 | \n
This clearly shows two things:
\nThat is to say, my own statistics show that I'm mainly serving our robot overlords.
\nThe situation is worse on other kinds of web apps, like my Forgejo instance.\nIt's a single-user instance, and most projects are private, but I do publish a bunch of open-source projects there.\nRecent bot traffic on applications like these is causing significant load on my server.
\nPreviously, the majority of crawlers were relatively decent.\nThey would crawl your website slowly, making sure not to cause too much load.\nTheir main purpose was serving search results to people looking for the content you publish.\nBringing down your website would only result in their own users landing on error pages.
\nThe recent Generative AI (GenAI) craze changed this completely.\nMultiple companies are now racing to collect as much data as possible.\nThey no longer care what happens with the websites they're crawling.\nOnce they have collected enough content, their GenAI agents provide users with answers directly.\nThe source website is treated as irrelevant.\nObviously, as a website owner, I do not want these thankless robots on my property3!
\nSo I went through the standard ways of blocking them:
\nrobots.txt.None of these methods work.
\nGenAI companies frequently ignore robots.txt files.\nThey also usually disguise themselves as normal browsers instead of identifying as crawlers.\nMore recently, I've also noticed them increasingly using consumer IP space.\nCombined with rapidly changing addresses, this makes it impossible to block them by IP address alone.
There are some promising new tools like Anubis that work4, for now.
\nMaking sure websites stay online is part of my job.\nI don't want to have to go through the same hassle for my own web apps when I get home.\nI'm tired, and the few things I'm hosting just aren't worth it.
\nSo, I decided to take two different approaches:
\nWebsites that are meant to be public, like the one you're reading now.\nI've always optimized these for high traffic, and will continue to do so.\nThese are built to withstand much more than the AI crawlers I'm currently seeing.\nI'm no longer blocking any bots on public websites.
\nWebsites (and web apps) that are primarily meant for myself.\nThese are now geoblocked and only accept traffic from Belgium.\nAs these are mainly open source applications that I don't maintain myself, they are harder to optimize.\nThis includes my Forgejo instance.\nIt's sad that part of my open source code disappears behind a firewall.\nI hope to open it up again once AI companies start behaving more responsibly.
\nMy geoblock page contains contact information.\nSo if people really want to go through the hassle of requesting access (which I very much doubt), they can be allowed in.
\nEvery article I've found claiming this reads like marketing material for a web security firm.\nFor that reason, I decided not to link to any of them.\nBased on my own traffic stats though, I don't doubt their numbers. ↩
\nI do read robots.txt on some sites I visit.\nIt's a treasure trove of information about the website you're visiting.\nBut that's probably not normal human behaviour. ↩
\nI don't really mind AI crawlers, as long as they behave.\nThey usually don't. ↩
\nAnubis is great!\nEspecially since it lets through the bots that clearly identify as a bot. ↩
\nCustomers regularly send me x509 key pairs at work1.\nMost of the time these have been forwarded a few times already, and people tend to mess up the files.\nThis means that the public and private keys don't match anymore, and I have to ask for a new pair.
\nLuckily, we can use OpenSSL to check whether a public and private key match.
\nThe classic way to compare RSA keys is to check whether the modulus is the same, as this is the only part that is shared between the public and private keys.\nIf the modulus isn't the same, the keys can't form a pair.
\nRead the modulus from the public key:
\nopenssl rsa -pubin -in public.pem -modulus -noout | openssl sha1\nThen do the same with the private RSA key:
\nopenssl rsa -in private.pem -modulus -noout | openssl sha1\nIf the two values match, the key files form a pair.
\nNote that we're piping the actual output of the -modulus command to openssl sha1 to get a shorter hash of the modulus.\nThis makes it easier to manually compare the two values, but it's not strictly necessary.\nYou could also compare the output directly if you're feeling brave.
Elliptic curve keys don't have an easy-to-compare modulus like the RSA keys above had.\nHowever, it's always possible to generate the public key2 part when we have the private key.
\nUsing the private key, generate (a hash of) the public key:
\nopenssl pkey -pubout -in private.pem | openssl sha1\nThen compare the output with the public key file you have.\nWe need to make openssl load it and print it out again to make sure we get the exact same format.
openssl x509 -pubkey -in public.pem -noout | openssl sha1\nIf the two values match, these keys form a pair.
\nPlease don't do this. Ask the company that needs a certificate to generate a private key and provide you with a Certificate Signing Request (CSR). This way, you don't have to handle the private key at all. ↩
\nWhen people say "public key" in the context of a x509 pair they usually refer to the certificate. The public key we're extracting here is the actual cryptographic public key, it doesn't contain the extra data like domain names that a certificate usually holds. You can't recover the full certificate when you only have the private key. ↩
\nKev Quirk recently wrote about removing content from public blogs:
\n\n\nGenerally speaking, I don't delete content from this site.\nHaving said that, if I posted something that I later feel is particularly egregious, I think I probably would.\nI personally don't think that a website should be a permanent record - how can it be?\nNothing last forever.
\n
This got me thinking about what to do with some of my older articles.\nI believe that URLs should be stable.\nThe internet is a large place, there’s no way to know who might be linking to any of my webpages.\nRemoving a page may also break other websites: someone may have written about it.\nDeleting the page would remove the context of their reference.
\nHowever, I’ve been working on this website in some form or another since 2007.\nTechnical articles tend to get outdated, and people change their minds about things they’ve written.\nSome of the articles on this website no longer reflect what I stand for.
\nArticles older than two years already include a banner noting their age, but that’s not always enough.\nNowadays, people often form an opinion based on the title alone — especially on social media.\nIt’s unlikely they’d notice a note about how old the article is.
\nSo I decided to hide certain articles.\nHidden articles are no longer linked on my website and don't appear in RSS feeds.\nYou can’t find them by clicking around.\nBut if you already have the link, the content is still accessible.\nBecause, as they say, cool URIs don't change.
\n","url":"https://vic.demuzere.be/articles/hiding-outdated-articles/","date_published":"2024-11-10T10:04:30Z","date_modified":"2024-11-11T18:18:07Z","language":"en"},{"id":"https://vic.demuzere.be/articles/not-so-redesign/","content_html":"Over the past few months I've been working on a new stylesheet for this website, and today I finally pushed the redesign branch to production!\nThe previous CSS code was largely written almost 10 years ago and only received small updates and enhancements.\nSince then, new features like flexbox and grids were added and made creating responsive websites much easier.
\nTurns out, the new design looks exactly the same as the old one!
\nA few things did change though:
\nfloat to position elements in favor of flexbox.I also tried to get some insight in how people used the website:
\nMost of the traffic to this website is routed through Cloudflare's CDN.\nI've configured it to cache everything, which means the access logs on my web server record only part of the actual traffic.\nThis creates an interesting dataset: I can see which pages are most visited, but I can't see spikes due to pages being shared somewhere.\nFor the same reason, my statistics about browsers, operating systems, referring websites and returning visitors are completely skewed.
\nArticles are the most popular pages on this website.\nThat makes sense, as those tend to show up in searches and are shared more frequently than others.\nMy most popular articles are technical ones:
\nThat means I need to write more articles!
\nThe most common 404s appear to be scanners looking for vulnerabilities.\nWordPress related pages like wp-login.php, xmlrpc.php and several files inside the wp-includes directory lead the race, followed by Joomla's administrator directory.
My work to fix broken links (using data from the access logs and Google Search Console) seems to have paid off.\nI've also been trying not to delete or move pages for a few years now.\nWhen a page is no longer relevant, I replace it with a small message pointing people to alternatives, like I did with my OTR fingerprint listing.
\nAnother interesting 404 page is /irc:irc.quakenet.org/sorcix,isnick which seems to be caused by a spider that doesn't understand irc: URLs.
I've been looking into IndieWeb after reading a few blog posts by Jan-Lukas Else.\nPeople can reply to your articles by linking back from their own website, or by using one of the provided IndieWeb-compatible services.\nAs I'm not active on mainstream social media, this may be a great way to connect with like-minded people.
\nFeatures I want to roll out next:
\nCertain visitors of your blog may want to print your articles.\nBy default, most browsers will present a simplified version of your website for printing.\nBackground images may be hidden, saving a lot of ink.\nBut we can do better: usually people just want to print your content, not the entire website.
\nPrinting things is not that common,\nso it's really not worth it to spend much time worrying about how your website looks on paper.\nThere are a few simple things that make a lot of difference, though!
\nModern browsers support media queries,\na way to limit part of your stylesheet to devices that match certain criteria.\nWhile this feature is most used to scale your website to the size of multiple devices,\nwe can also use it to define print styles.
\n@media print {\n\t/* insert CSS for printing here */\n}\nWe want these styles at the very bottom of our stylesheet:\nThey have to override everything that we have defined before.
\nIn case you want to support Internet Explorer 8\nand lower (or prefer your print styles in a separate file),\nthe alternative is using a link tag with media attribute:
<link rel="stylesheet" href="print-all-the-things.css" media="print" />\nWhen using a separate stylesheet, you no longer have to use the media queries in the examples below.
\nUsually you'll want to get rid of everything except the content of your website.\nNo header, footer, sidebar, sharing widgets, advertisements..\nMost of this should be possible by changing your stylesheet only.\nYou might want to introduce a few new classes to manage visibility, that'll come in handy!
\n.print {\n\tdisplay: none;\n}\n@media print {\n\t#header, #footer, #sidebar, .share, .ads, .no-print {\n\t\tdisplay: none;\n\t}\n\t.print {\n\t\tdisplay: block;\n\t}\n}\nAnother really simple tweak is the page margin.\nI prefer margins a bit wider than the defaults that Chrome gave me.\nNote that most browsers add some information in the page header and footer;\nMaking the top and bottom margins a bit bigger helps separating your content from the page info.
\n@media print {\n\t@page {\n\t\tmargin: 2cm 2cm 2cm 1.5cm;\n\t}\n}\nCloudflare is a cloud-based solution to protect your website against attacks.\nOne way to mitigate (DDoS) attacks is providing a geographically distributed network of POPs with a lot of bandwidth.\nAnd that's exactly what they did: Cloudflare maintains a content delivery network with 30 data centers around the world.\nEven if you have no reason to suspect being attacked any time soon, the speed improvement is well worth it.
\nSetting up their service is pretty straightforward. Move your domain to their nameservers and select the subdomains that have to be protected.\nImportant to note: Using the CDN routes all traffic to your website through the Cloudflare network. This means that you're basically allowing them\nto do a man-in-the-middle attack on your visitors. This article is aimed at static websites, so it doesn't really matter, but it's something to keep in mind.
\nBy default, Cloudflare caches resources that are supposed to be static on most websites. This includes images, stylesheets and javascripts.\nYour server is contacted for every pageview, but most assets are offloaded to their edge servers.\nThis is great for a dynamic website, as we want to make sure that visitors see their personalized version of the website.\nStatic websites, however, may benefit from being cached completely.
\nFirst things first. In the Cloudflare site settings page, we see three cache levels:
\nAggressive: The complete URL is used to identify a resource.
\nhttp://example.com/image.png?including=query-string
\nSimplified: The query string is ignored when identifying a resource.
\nhttp://example.com/image.png?ignore=this
\nBasic: Only caches resources without query string.
\nhttp://example.com/image.png
\nThe cache level setting only applies to those resources that Cloudflare decided to cache.\nI usually set this to Aggressive for static websites, as it allows me to change a hash in the query string to invalidate the cache when updating a page.
\nA second interesting setting is the minimum expire TTL, this allows you to overwrite the expire time in the cache-control header.\nWhen setting up a static website, I expect these headers to be set correctly on the origin server, but this may come in handy when you have little control.
The setting we're looking for is hidden deeper in the Cloudflare admin panel. In order to make their CDN cache everything, you need to create a page rule.\nPage rules define custom settings for a given subset of your website. Click the gear icon next to your website and select Page Rules. Between the rule settings an item named Custom caching appears. Set this to Cache everything to, well, cache everything. This only applies to the URL pattern specified for the page rule.
\nTwo other interesting settings:
\nEdge cache expire TTL: Specifies how long a resource has to be cached on the Cloudflare edge servers. The longer they are allowed to cache it, the more\nbandwidth you'll save. Decide how big this value may be depending on how much you update your static website. You may always clear their edge cache using the\nadministration panel. (This setting is only available when Cache everything is selected.)
\nBrowser cache expire TTL: This setting overwrites the minimum expire TTL from the global site settings, for this specific URL pattern.
\nTry it out! Depending on the number of visitors, caching everything may save quite a lot of bandwidth! If you have only a few visitors per month, you may not\neven see it, especially when they are geographically distributed. It looks like Cloudflare does not share resources between edge servers. If a resource\nis not available in the POP closest to your visitor, it contacts the your server.
\n","url":"https://vic.demuzere.be/articles/caching-static-website-with-cloudflare/","date_published":"2015-01-17T23:50:00Z","date_modified":"2026-05-13T18:17:26Z","language":"en"},{"id":"https://vic.demuzere.be/articles/trackerless-social-sharing/","content_html":"We seem to live in the golden age of social media: Sharing every useless article you've read with the world is important. Most visitors are of the lazy kind, though, so we need to make this as easy as possible. Fortunately, social media sites provide us with widgets to do exactly that! Phew, we're done here, right? ..right?
\nThe problem with these widgets is that we also allow social media companies to follow our visitors. The primary source of revenue for these companies is advertising. Knowing which sites their members visit is a valuable asset. But we don't want evil corporations to spy on our visitors, do we? The solution for this problem was invented in 1993, and is called a hyperlink. It was designed to send people from one page to the next.
\nNot convinced yet? Well, this probably means you don't value your visitors enough. But there is another reason: Social sharing widgets slow down your website. Every widget has to be loaded from an external website, causing a new HTTP request. Hyperlinks just sit there, waiting to be clicked!
\nLast but not least, social sharing hyperlinks can be styled as you wish. Widgets are a pain in the ass to integrate in your website. Most websites simply have them all together in a box. Choose your own social media icon for your links, or use my Social media vectors.
\nhttps://twitter.com/intent/tweet?url=URL
Facebook:
\nhttps://facebook.com/sharer/sharer.php?u=URL
Google+:
\nhttps://plus.google.com/share?url=URL
Linkedin:
\nhttps://www.linkedin.com/shareArticle?mini=true&url=URL&title=TITLE
Reddit:
\nhttps://www.reddit.com/submit?url=URL