The last few years, web traffic from bots has increased rapidly.\nSome sources1 claim that automated activity now accounts for over half of all web traffic.\nThis is even more pronounced on small websites like mine, as I don't receive many human visitors.
\nThe table below shows traffic on this website for the last few days:
\n| \n | Path | \nHits | \n
|---|---|---|
| 1 | \n/wp-login.php | \n2751 | \n
| 2 | \n/wp-admin/index.php | \n2707 | \n
| 3 | \n/wp-admin/edit.php | \n2354 | \n
| 4 | \n/wp-admin/profile.php | \n2353 | \n
| 5 | \n/wp-admin/plugins.php | \n2351 | \n
| 6 | \n/robots.txt | \n1131 | \n
| 7 | \n/ | \n994 | \n
| 8 | \n/articles/feed.rss | \n989 | \n
| 9 | \n/articles/how-i-almost-lost-my-backups/ | \n314 | \n
| 10 | \n/wp-content/themes/vic/style.css | \n299 | \n
This clearly shows two things:
\nThat is to say, my own statistics show that I'm mainly serving our robot overlords.
\nThe situation is worse on other kinds of web apps, like my Forgejo instance.\nIt's a single-user instance, and most projects are private, but I do publish a bunch of open-source projects there.\nRecent bot traffic on applications like these is causing significant load on my server.
\nPreviously, the majority of crawlers were relatively decent.\nThey would crawl your website slowly, making sure not to cause too much load.\nTheir main purpose was serving search results to people looking for the content you publish.\nBringing down your website would only result in their own users landing on error pages.
\nThe recent Generative AI (GenAI) craze changed this completely.\nMultiple companies are now racing to collect as much data as possible.\nThey no longer care what happens with the websites they're crawling.\nOnce they have collected enough content, their GenAI agents provide users with answers directly.\nThe source website is treated as irrelevant.\nObviously, as a website owner, I do not want these thankless robots on my property3!
\nSo I went through the standard ways of blocking them:
\nrobots.txt.None of these methods work.
\nGenAI companies frequently ignore robots.txt files.\nThey also usually disguise themselves as normal browsers instead of identifying as crawlers.\nMore recently, I've also noticed them increasingly using consumer IP space.\nCombined with rapidly changing addresses, this makes it impossible to block them by IP address alone.
There are some promising new tools like Anubis that work4, for now.
\nMaking sure websites stay online is part of my job.\nI don't want to have to go through the same hassle for my own web apps when I get home.\nI'm tired, and the few things I'm hosting just aren't worth it.
\nSo, I decided to take two different approaches:
\nWebsites that are meant to be public, like the one you're reading now.\nI've always optimized these for high traffic, and will continue to do so.\nThese are built to withstand much more than the AI crawlers I'm currently seeing.\nI'm no longer blocking any bots on public websites.
\nWebsites (and web apps) that are primarily meant for myself.\nThese are now geoblocked and only accept traffic from Belgium.\nAs these are mainly open source applications that I don't maintain myself, they are harder to optimize.\nThis includes my Forgejo instance.\nIt's sad that part of my open source code disappears behind a firewall.\nI hope to open it up again once AI companies start behaving more responsibly.
\nMy geoblock page contains contact information.\nSo if people really want to go through the hassle of requesting access (which I very much doubt), they can be allowed in.
\nEvery article I've found claiming this reads like marketing material for a web security firm.\nFor that reason, I decided not to link to any of them.\nBased on my own traffic stats though, I don't doubt their numbers. ↩
\nI do read robots.txt on some sites I visit.\nIt's a treasure trove of information about the website you're visiting.\nBut that's probably not normal human behaviour. ↩
\nI don't really mind AI crawlers, as long as they behave.\nThey usually don't. ↩
\nAnubis is great!\nEspecially since it lets through the bots that clearly identify as a bot. ↩
\nSQLite does not have built-in UUID functions, but by getting creative with the randomblob(N) function we can generate version 4 (random) UUIDs.
Before we continue, it's important to note that you should verify whether SQLite's random functions are adequate for your usecase.\nThey may not return truly random data.\nHowever, as the RFC notes, "they (UUIDs) MUST NOT be used as security capabilities".\nAnd since we all follow that to the letter, it doesn't really matter.. right?
\nThe excellent SQLite manual gives us a starting point:
\n\n\nHint: applications can generate globally unique identifiers using this function together with hex() and/or lower().
\n
Combining the randomblob(N) function with hex(N) does indeed create something that vaguely resembles a UUID.\nFor some purposes this may suffice, but the result isn't actually a valid UUID1:\nvalid UUIDs must have specific bits set to indicate their version and variant.
We can do better!
\nThe binary representation of a version 4 UUID consists of:
\nAs far as I know, we can't modify individual bytes in a blob using the available functions in SQLite.\nHowever, we do have access to a lot of string functions.\nTaking a detour using HEX encoding provides a solution.\nWe'll first concatenate a HEX string containing the data we want, then convert it back to a binary blob.\nNote that each byte is represented by two hexadecimal characters.
\nThe first 48 random bits are easy:
\nsqlite> select HEX(RANDOMBLOB(6));\r\n4004714237AF\r\nThe first 4 bits of the next byte need to be set to 0100 (or 4 in hex).\nAs every byte is represented by two characters in hexadecimal, we can just use a literal 4 for the next character.
sqlite> select '4';\r\n4\r\nNext, we need 6 bits of random data.\nWe can only generate random bytes using RANDOMBLOB(N), but we need 1.5 bytes.\nLuckily, because we are working with a hexadecimal string, we can generate 2 bytes (4 hex characters) and take the first three characters.
sqlite> select HEX(RANDOMBLOB(2));\r\n7B4A\r\n\r\nsqlite> select SUBSTR(HEX(RANDOMBLOB(2)),0,4);\r\n5BD\r\nThe next one is a little more complex.\nThe variant field consists of a variable number of bits.\nFor our version 4 UUID, we need to set the first two bits to 10, the others can be random.\nThat gives us a possible range between 1000 and 1011, or a hexadecimal character between 8 and B inclusive.
select FORMAT('%X', 8 + ABS(RANDOM() % 4));\r\nA\r\nAfter this, we need 60 more random bits.\nSince this again includes a half byte again (7.5 bytes), we're doing the same SUBSTR trick as before:
sqlite> select SUBSTR(HEX(RANDOMBLOB(8)),0,16);\r\nE83EF2136842130\r\nUsing CONCAT(X, ...) we create a single string:
sqlite> select CONCAT(\r\n(x1...> HEX(RANDOMBLOB(6)),\r\n(x1...> '4',\r\n(x1...> SUBSTR(HEX(RANDOMBLOB(2)),0,4),\r\n(x1...> FORMAT('%X', 8 + ABS(RANDOM() % 4)),\r\n(x1...> SUBSTR(HEX(RANDOMBLOB(8)),0,16)\r\n(x1...> );\r\n\r\n64DF281823BE47249BB2CA8D6450066B\r\nThen use UNHEX(X) to convert it back into a binary blob:
select UNHEX(CONCAT(\r\n HEX(RANDOMBLOB(6)),\r\n '4',\r\n SUBSTR(HEX(RANDOMBLOB(2)),0,4),\r\n FORMAT('%X', 8 + ABS(RANDOM() % 4)),\r\n SUBSTR(HEX(RANDOMBLOB(8)),0,16)\r\n));\r\nThe SQL above can be used as the default value for a blob column:
\nCREATE TABLE "animals" (\r\n "id" INTEGER PRIMARY KEY NOT NULL,\r\n "uuid" BLOB UNIQUE NOT NULL\r\n DEFAULT (UNHEX(CONCAT(\r\n HEX(RANDOMBLOB(6)), '4',\r\n SUBSTR(HEX(RANDOMBLOB(2)),0,4),\r\n FORMAT('%X', 8 + ABS(RANDOM() % 4)),\r\n SUBSTR(HEX(RANDOMBLOB(8)),0,16))))\r\n UNIQUE CHECK(LENGTH("uuid") = 16),\r\n "name" TEXT NOT NULL\r\n) STRICT;\r\nThis will generate a UUID if we don't specify one on insert:
\nsqlite> INSERT INTO "animals" ("name") VALUES ('Fluffy');\r\n\r\nsqlite> SELECT * FROM "animals";\r\n1|[blob]|Fluffy\r\n\r\nsqlite> SELECT "id", HEX("uuid"), "name" FROM "animals";\r\n1|85671F7120F14AA6802B8B4E9213B8BE|Fluffy\r\nDepending on your use case, you may want to store UUIDs in a TEXT column.
We were close to a "hex-and-dash" string format with the SQL above, until we call the UNHEX(X) function.\nRFC 9562 specifies that dashes are required, so we'll add them in the right places.
sqlite> select CONCAT(\r\n HEX(RANDOMBLOB(4)),\r\n '-',\r\n HEX(RANDOMBLOB(2)),\r\n '-',\r\n '4',\r\n SUBSTR(HEX(RANDOMBLOB(2)),0,4),\r\n '-',\r\n FORMAT('%X', 8 + ABS(RANDOM() % 4)),\r\n SUBSTR(HEX(RANDOMBLOB(2)),0,4),\r\n '-',\r\n SUBSTR(HEX(RANDOMBLOB(8)),0,13)\r\n);\r\n\r\n199ED82A-3102-42DB-B59B-8C356B6F3FF7\r\nVoila!
\nAs technically correct is the best kind of correct: if you get really lucky with the generated random data, they might actually be valid. ↩
\nCertain visitors of your blog may want to print your articles.\nBy default, most browsers will present a simplified version of your website for printing.\nBackground images may be hidden, saving a lot of ink.\nBut we can do better: usually people just want to print your content, not the entire website.
\nPrinting things is not that common,\nso it's really not worth it to spend much time worrying about how your website looks on paper.\nThere are a few simple things that make a lot of difference, though!
\nModern browsers support media queries,\na way to limit part of your stylesheet to devices that match certain criteria.\nWhile this feature is most used to scale your website to the size of multiple devices,\nwe can also use it to define print styles.
\n@media print {\n\t/* insert CSS for printing here */\n}\nWe want these styles at the very bottom of our stylesheet:\nThey have to override everything that we have defined before.
\nIn case you want to support Internet Explorer 8\nand lower (or prefer your print styles in a separate file),\nthe alternative is using a link tag with media attribute:
<link rel="stylesheet" href="print-all-the-things.css" media="print" />\nWhen using a separate stylesheet, you no longer have to use the media queries in the examples below.
\nUsually you'll want to get rid of everything except the content of your website.\nNo header, footer, sidebar, sharing widgets, advertisements..\nMost of this should be possible by changing your stylesheet only.\nYou might want to introduce a few new classes to manage visibility, that'll come in handy!
\n.print {\n\tdisplay: none;\n}\n@media print {\n\t#header, #footer, #sidebar, .share, .ads, .no-print {\n\t\tdisplay: none;\n\t}\n\t.print {\n\t\tdisplay: block;\n\t}\n}\nAnother really simple tweak is the page margin.\nI prefer margins a bit wider than the defaults that Chrome gave me.\nNote that most browsers add some information in the page header and footer;\nMaking the top and bottom margins a bit bigger helps separating your content from the page info.
\n@media print {\n\t@page {\n\t\tmargin: 2cm 2cm 2cm 1.5cm;\n\t}\n}\nWhen a new version of one of my Go apps is ready, I usually want to build and package the binary for multiple platforms.\nThis can easily be done in a shell script, but Make brings a bonus: it can run multiple builds at the same time.
\nIn case you've worked with make before, skip this section!\nIf not: let's see what a basic makefile looks like.
\nMake defines targets, these are things that need to be done.\nUsually a target represents a file that has to be created.\nEvery target contains a list of commands that have to be executed, we call that the recipe.\nMake will only execute the recipe if a file with the same name as the target does not exist.\nIf your target does not represent a file, it's a phony target.
\nmytarget:\n\techo Hello world.\n\nspace: mytarget foo\n\techo "I'm in space."\n\nfoo:\n\techo I am run only once\n\ttouch foo\n\n.PHONY: mytarget space\nThe above makefile contains 3 targets: mytarget, foo and space.\nThe second target tells us that it needs both other targets, so when you call make space, make will also execute mytarget and foo.
At the bottom of the file we define a special .PHONY target: Here we simply tell make which targets are phony targets.\nThose do not represent a file on disk.\nAs we usually don't compile separate Go files, most of our targets will be phony targets.
\nLet's see what this does: save the file as Makefile and run make space in the same directory:
$ make space \necho Hello world.\nHello world.\necho I am run only once\nI am run only once\necho "I'm in space."\nI'm in space.\nNotice that make always tells us what command it is executing.\nYou can hide a command by prefixing it with @.\nAs we'll be executing multiple targets concurrently, it may be useful to see which commands are running.
\nRun make space again.\nYou'll see that the foo target is not run anymore.\nThat's because we created a file named foo and foo is not a phony target.
GNU Make has two types of variables.\nWhen we assign to a variable with := the contents is evaluated at the time the variable is declared.\nIf we use just = the contents is evaluated every time we use the variable.
MYVAR := Hello there! I am currently in $(shell pwd)\nPWD = $(shell pwd)\nThe pwd shell command will be executed every time we use variable $(PWD) in the makefile, while $(MYVAR) will always contain the same value.
Make supports running multiple targets at the same time.\nThis means that we need a separate target for every GOOS and GOARCH in order to take advantage of the concurrency.
\nA simple way to do this would be:
\nlinux:\n\tGOOS=linux GOARCH=amd64 go build ...\n\nwindows:\n\tGOOS=windows GOARCH=amd64 go build ...\n\nrelease: windows linux\n\n.PHONY release windows linux\nThis builds both linux and windows binaries when we call make release.\nWe can improve this Makefile by looping over the platforms and architectures we want to build:
PLATFORMS := linux/amd64 windows/amd64\n\nrelease: $(PLATFORMS)\n\n$(PLATFORMS):\n\techo Hello, I am $@\n\n.PHONY release $(PLATFORMS)\nWhat we do here is create a target that contains all possible platforms.\nEvery platform is a separate target, just like it was in the previous Makefile.\nThat also means that we have to add every platform to the phony list.\nLuckily, we can do that using the same variable. Inside the recipe we can use the special $@ that contains the name of the current target.
The next part is a bit harder, we need to separate OS and ARCH.\nRemember that variables can be evaluated at runtime?\nWe'll use that feature to create an $(os) and $(arch) var.
temp = $(subst /, ,$@)\nos = $(word 1, $(temp))\narch = $(word 2, $(temp))\nString manipulation functions in make are limited, so we have to be a bit creative here.\nWe'll use these variables inside the $(PLATFORM) target.\nRemember that $@ contains the name of the current target?\nWhen we replace the slash in this string we can use the word function to get the first and last part.
Combined with our previous Makefile, this becomes:
\nPLATFORMS := linux/amd64 windows/amd64\n\ntemp = $(subst /, ,$@)\nos = $(word 1, $(temp))\narch = $(word 2, $(temp))\n\nrelease: $(PLATFORMS)\n\n$(PLATFORMS):\n\tGOOS=$(os) GOARCH=$(arch) go build -o '$(os)-$(arch)' mypackage\n\n.PHONY release $(PLATFORMS)\nNote that variables can only be set outside the recipe.\nAs the contents is only evaluated when we use them, it still contains the correct value.
\nWe now have a Makefile that creates binaries for the platforms we listed in the $(PLATFORMS) variable.\nWhen we call make release it builds everything we asked, one by one.\nThe main reason to use make is the easy concurrency: simply call make with the -j n argument, where n is the number of simultaneous jobs.