The table below shows traffic on this website for the last few days:

This clearly shows two things:

• The first useful page for a real visitor is in sixth place².
• My stylesheet has a low number of hits compared to the frontpage and my most recently published article. One would expect each unique visitor to download the stylesheet at least once.

That is to say, my own statistics show that I'm mainly serving our robot overlords.

The situation is worse on other kinds of web apps, like my Forgejo instance. It's a single-user instance, and most projects are private, but I do publish a bunch of open-source projects there. Recent bot traffic on applications like these is causing significant load on my server.

Keeping the bots out

Previously, the majority of crawlers were relatively decent. They would crawl your website slowly, making sure not to cause too much load. Their main purpose was serving search results to people looking for the content you publish. Bringing down your website would only result in their own users landing on error pages.

The recent Generative AI (GenAI) craze changed this completely. Multiple companies are now racing to collect as much data as possible. They no longer care what happens with the websites they're crawling. Once they have collected enough content, their GenAI agents provide users with answers directly. The source website is treated as irrelevant. Obviously, as a website owner, I do not want these thankless robots on my property³!

So I went through the standard ways of blocking them:

1. Asking nicely in robots.txt.
2. Denying requests based on the User-Agent header.
3. Blocking IP ranges from known data centers and cloud providers.

None of these methods work.

GenAI companies frequently ignore robots.txt files. They also usually disguise themselves as normal browsers instead of identifying as crawlers. More recently, I've also noticed them increasingly using consumer IP space. Combined with rapidly changing addresses, this makes it impossible to block them by IP address alone.

There are some promising new tools like Anubis that work⁴, for now.

Giving up

Making sure websites stay online is part of my job. I don't want to have to go through the same hassle for my own web apps when I get home. I'm tired, and the few things I'm hosting just aren't worth it.

So, I decided to take two different approaches:

1. Websites that are meant to be public, like the one you're reading now. I've always optimized these for high traffic, and will continue to do so. These are built to withstand much more than the AI crawlers I'm currently seeing. I'm no longer blocking any bots on public websites.

2. Websites (and web apps) that are primarily meant for myself. These are now geoblocked and only accept traffic from Belgium. As these are mainly open source applications that I don't maintain myself, they are harder to optimize. This includes my Forgejo instance. It's sad that part of my open source code disappears behind a firewall. I hope to open it up again once AI companies start behaving more responsibly.

My geoblock page contains contact information. So if people really want to go through the hassle of requesting access (which I very much doubt), they can be allowed in.

Before we continue, it's important to note that you should verify whether SQLite's random functions are adequate for your usecase. They may not return truly random data. However, as the RFC notes, "they (UUIDs) MUST NOT be used as security capabilities". And since we all follow that to the letter, it doesn't really matter.. right?

The excellent SQLite manual gives us a starting point:

Hint: applications can generate globally unique identifiers using this function together with hex() and/or lower().

Combining the randomblob(N) function with hex(N) does indeed create something that vaguely resembles a UUID. For some purposes this may suffice, but the result isn't actually a valid UUID¹: valid UUIDs must have specific bits set to indicate their version and variant.

We can do better!

As binary blob

The binary representation of a version 4 UUID consists of:

• 48 bits (6 bytes) random data
• 4 bits (0.5 bytes) version field
• 6 bits (1.5 bytes) random data
• 4 bits (0.5 bytes) variant field
• 60 bits (7.5 bytes) random data

As far as I know, we can't modify individual bytes in a blob using the available functions in SQLite. However, we do have access to a lot of string functions. Taking a detour using HEX encoding provides a solution. We'll first concatenate a HEX string containing the data we want, then convert it back to a binary blob. Note that each byte is represented by two hexadecimal characters.

The first 48 random bits are easy:

sqlite> select HEX(RANDOMBLOB(6));
4004714237AF

The first 4 bits of the next byte need to be set to 0100 (or 4 in hex). As every byte is represented by two characters in hexadecimal, we can just use a literal 4 for the next character.

sqlite> select '4';
4

Next, we need 6 bits of random data. We can only generate random bytes using RANDOMBLOB(N), but we need 1.5 bytes. Luckily, because we are working with a hexadecimal string, we can generate 2 bytes (4 hex characters) and take the first three characters.

sqlite> select HEX(RANDOMBLOB(2));
7B4A

sqlite> select SUBSTR(HEX(RANDOMBLOB(2)),0,4);
5BD

The next one is a little more complex. The variant field consists of a variable number of bits. For our version 4 UUID, we need to set the first two bits to 10, the others can be random. That gives us a possible range between 1000 and 1011, or a hexadecimal character between 8 and B inclusive.

select FORMAT('%X', 8 + ABS(RANDOM() % 4));
A

After this, we need 60 more random bits. Since this again includes a half byte again (7.5 bytes), we're doing the same SUBSTR trick as before:

sqlite> select SUBSTR(HEX(RANDOMBLOB(8)),0,16);
E83EF2136842130

Stitching it together

Using CONCAT(X, ...) we create a single string:

sqlite> select CONCAT(
(x1...>   HEX(RANDOMBLOB(6)),
(x1...>   '4',
(x1...>   SUBSTR(HEX(RANDOMBLOB(2)),0,4),
(x1...>   FORMAT('%X', 8 + ABS(RANDOM() % 4)),
(x1...>   SUBSTR(HEX(RANDOMBLOB(8)),0,16)
(x1...> );

64DF281823BE47249BB2CA8D6450066B

Then use UNHEX(X) to convert it back into a binary blob:

select UNHEX(CONCAT(
  HEX(RANDOMBLOB(6)),
  '4',
  SUBSTR(HEX(RANDOMBLOB(2)),0,4),
  FORMAT('%X', 8 + ABS(RANDOM() % 4)),
  SUBSTR(HEX(RANDOMBLOB(8)),0,16)
));

Generating UUIDs automatically

The SQL above can be used as the default value for a blob column:

CREATE TABLE "animals" (
  "id" INTEGER PRIMARY KEY NOT NULL,
  "uuid" BLOB UNIQUE NOT NULL
    DEFAULT (UNHEX(CONCAT(
      HEX(RANDOMBLOB(6)), '4',
      SUBSTR(HEX(RANDOMBLOB(2)),0,4),
      FORMAT('%X', 8 + ABS(RANDOM() % 4)),
      SUBSTR(HEX(RANDOMBLOB(8)),0,16))))
    UNIQUE CHECK(LENGTH("uuid") = 16),
   "name" TEXT NOT NULL
) STRICT;

This will generate a UUID if we don't specify one on insert:

sqlite> INSERT INTO "animals" ("name") VALUES ('Fluffy');

sqlite> SELECT * FROM "animals";
1|[blob]|Fluffy

sqlite> SELECT "id", HEX("uuid"), "name" FROM "animals";
1|85671F7120F14AA6802B8B4E9213B8BE|Fluffy

In human readable (hexadecimal) form

Depending on your use case, you may want to store UUIDs in a TEXT column.

We were close to a "hex-and-dash" string format with the SQL above, until we call the UNHEX(X) function. RFC 9562 specifies that dashes are required, so we'll add them in the right places.

sqlite> select CONCAT(
  HEX(RANDOMBLOB(4)),
  '-',
  HEX(RANDOMBLOB(2)),
  '-',
  '4',
  SUBSTR(HEX(RANDOMBLOB(2)),0,4),
  '-',
  FORMAT('%X', 8 + ABS(RANDOM() % 4)),
  SUBSTR(HEX(RANDOMBLOB(2)),0,4),
  '-',
  SUBSTR(HEX(RANDOMBLOB(8)),0,13)
);

199ED82A-3102-42DB-B59B-8C356B6F3FF7

Voila!

Printing things is not that common, so it's really not worth it to spend much time worrying about how your website looks on paper. There are a few simple things that make a lot of difference, though!

Style for printing

Modern browsers support media queries, a way to limit part of your stylesheet to devices that match certain criteria. While this feature is most used to scale your website to the size of multiple devices, we can also use it to define print styles.

@media print {
	/* insert CSS for printing here */
}

We want these styles at the very bottom of our stylesheet: They have to override everything that we have defined before.

Older browsers

In case you want to support Internet Explorer 8 and lower (or prefer your print styles in a separate file), the alternative is using a link tag with media attribute:

<link rel="stylesheet" href="print-all-the-things.css" media="print" />

When using a separate stylesheet, you no longer have to use the media queries in the examples below.

Hiding unnecessary parts of the page

Usually you'll want to get rid of everything except the content of your website. No header, footer, sidebar, sharing widgets, advertisements.. Most of this should be possible by changing your stylesheet only. You might want to introduce a few new classes to manage visibility, that'll come in handy!

.print {
	display: none;
}
@media print {
	#header, #footer, #sidebar, .share, .ads, .no-print {
		display: none;
	}
	.print {
		display: block;
	}
}

Page margins

Another really simple tweak is the page margin. I prefer margins a bit wider than the defaults that Chrome gave me. Note that most browsers add some information in the page header and footer; Making the top and bottom margins a bit bigger helps separating your content from the page info.

@media print {
	@page {
		margin: 2cm 2cm 2cm 1.5cm;
	}
}

Makefile 101

In case you've worked with make before, skip this section! If not: let's see what a basic makefile looks like.

Make defines targets, these are things that need to be done. Usually a target represents a file that has to be created. Every target contains a list of commands that have to be executed, we call that the recipe. Make will only execute the recipe if a file with the same name as the target does not exist. If your target does not represent a file, it's a phony target.

mytarget:
	echo Hello world.

space: mytarget foo
	echo "I'm in space."

foo:
	echo I am run only once
	touch foo

.PHONY: mytarget space

The above makefile contains 3 targets: mytarget, foo and space. The second target tells us that it needs both other targets, so when you call make space, make will also execute mytarget and foo.

At the bottom of the file we define a special .PHONY target: Here we simply tell make which targets are phony targets. Those do not represent a file on disk. As we usually don't compile separate Go files, most of our targets will be phony targets.

Let's see what this does: save the file as Makefile and run make space in the same directory:

$ make space  
echo Hello world.
Hello world.
echo I am run only once
I am run only once
echo "I'm in space."
I'm in space.

Notice that make always tells us what command it is executing. You can hide a command by prefixing it with @. As we'll be executing multiple targets concurrently, it may be useful to see which commands are running.

Run make space again. You'll see that the foo target is not run anymore. That's because we created a file named foo and foo is not a phony target.

Variables

GNU Make has two types of variables. When we assign to a variable with := the contents is evaluated at the time the variable is declared. If we use just = the contents is evaluated every time we use the variable.

MYVAR := Hello there! I am currently in $(shell pwd)
PWD = $(shell pwd)

The pwd shell command will be executed every time we use variable $(PWD) in the makefile, while $(MYVAR) will always contain the same value.

Cross compiling

Make supports running multiple targets at the same time. This means that we need a separate target for every GOOS and GOARCH in order to take advantage of the concurrency.

A simple way to do this would be:

linux:
	GOOS=linux GOARCH=amd64 go build ...

windows:
	GOOS=windows GOARCH=amd64 go build ...

release: windows linux

.PHONY release windows linux

This builds both linux and windows binaries when we call make release. We can improve this Makefile by looping over the platforms and architectures we want to build:

PLATFORMS := linux/amd64 windows/amd64

release: $(PLATFORMS)

$(PLATFORMS):
	echo Hello, I am $@

.PHONY release $(PLATFORMS)

What we do here is create a target that contains all possible platforms. Every platform is a separate target, just like it was in the previous Makefile. That also means that we have to add every platform to the phony list. Luckily, we can do that using the same variable. Inside the recipe we can use the special $@ that contains the name of the current target.

The next part is a bit harder, we need to separate OS and ARCH. Remember that variables can be evaluated at runtime? We'll use that feature to create an $(os) and $(arch) var.

temp = $(subst /, ,$@)
os = $(word 1, $(temp))
arch = $(word 2, $(temp))

String manipulation functions in make are limited, so we have to be a bit creative here. We'll use these variables inside the $(PLATFORM) target. Remember that $@ contains the name of the current target? When we replace the slash in this string we can use the word function to get the first and last part.

Combined with our previous Makefile, this becomes:

PLATFORMS := linux/amd64 windows/amd64

temp = $(subst /, ,$@)
os = $(word 1, $(temp))
arch = $(word 2, $(temp))

release: $(PLATFORMS)

$(PLATFORMS):
	GOOS=$(os) GOARCH=$(arch) go build -o '$(os)-$(arch)' mypackage

.PHONY release $(PLATFORMS)

Note that variables can only be set outside the recipe. As the contents is only evaluated when we use them, it still contains the correct value.

Concurrency

We now have a Makefile that creates binaries for the platforms we listed in the $(PLATFORMS) variable. When we call make release it builds everything we asked, one by one. The main reason to use make is the easy concurrency: simply call make with the -j n argument, where n is the number of simultaneous jobs.