![\"Alecto](\"https://vic.demuzere.be/wp-content/uploads/articles/images/DgqtPFnIaOyThLZaRMPkYUv7MXsowBfW.jpg\")
The last few years, web traffic from bots has increased rapidly.\nSome sources1 claim that automated activity now accounts for over half of all web traffic.\nThis is even more pronounced on small websites like mine, as I don't receive many human visitors.
\nThe table below shows traffic on this website for the last few days:
\n| \n | Path | \nHits | \n
|---|---|---|
| 1 | \n/wp-login.php | \n2751 | \n
| 2 | \n/wp-admin/index.php | \n2707 | \n
| 3 | \n/wp-admin/edit.php | \n2354 | \n
| 4 | \n/wp-admin/profile.php | \n2353 | \n
| 5 | \n/wp-admin/plugins.php | \n2351 | \n
| 6 | \n/robots.txt | \n1131 | \n
| 7 | \n/ | \n994 | \n
| 8 | \n/articles/feed.rss | \n989 | \n
| 9 | \n/articles/how-i-almost-lost-my-backups/ | \n314 | \n
| 10 | \n/wp-content/themes/vic/style.css | \n299 | \n
This clearly shows two things:
\nThat is to say, my own statistics show that I'm mainly serving our robot overlords.
\nThe situation is worse on other kinds of web apps, like my Forgejo instance.\nIt's a single-user instance, and most projects are private, but I do publish a bunch of open-source projects there.\nRecent bot traffic on applications like these is causing significant load on my server.
\nPreviously, the majority of crawlers were relatively decent.\nThey would crawl your website slowly, making sure not to cause too much load.\nTheir main purpose was serving search results to people looking for the content you publish.\nBringing down your website would only result in their own users landing on error pages.
\nThe recent Generative AI (GenAI) craze changed this completely.\nMultiple companies are now racing to collect as much data as possible.\nThey no longer care what happens with the websites they're crawling.\nOnce they have collected enough content, their GenAI agents provide users with answers directly.\nThe source website is treated as irrelevant.\nObviously, as a website owner, I do not want these thankless robots on my property3!
\nSo I went through the standard ways of blocking them:
\nrobots.txt.None of these methods work.
\nGenAI companies frequently ignore robots.txt files.\nThey also usually disguise themselves as normal browsers instead of identifying as crawlers.\nMore recently, I've also noticed them increasingly using consumer IP space.\nCombined with rapidly changing addresses, this makes it impossible to block them by IP address alone.
There are some promising new tools like Anubis that work4, for now.
\nMaking sure websites stay online is part of my job.\nI don't want to have to go through the same hassle for my own web apps when I get home.\nI'm tired, and the few things I'm hosting just aren't worth it.
\nSo, I decided to take two different approaches:
\nWebsites that are meant to be public, like the one you're reading now.\nI've always optimized these for high traffic, and will continue to do so.\nThese are built to withstand much more than the AI crawlers I'm currently seeing.\nI'm no longer blocking any bots on public websites.
\nWebsites (and web apps) that are primarily meant for myself.\nThese are now geoblocked and only accept traffic from Belgium.\nAs these are mainly open source applications that I don't maintain myself, they are harder to optimize.\nThis includes my Forgejo instance.\nIt's sad that part of my open source code disappears behind a firewall.\nI hope to open it up again once AI companies start behaving more responsibly.
\nMy geoblock page contains contact information.\nSo if people really want to go through the hassle of requesting access (which I very much doubt), they can be allowed in.
\nEvery article I've found claiming this reads like marketing material for a web security firm.\nFor that reason, I decided not to link to any of them.\nBased on my own traffic stats though, I don't doubt their numbers. ↩
\nI do read robots.txt on some sites I visit.\nIt's a treasure trove of information about the website you're visiting.\nBut that's probably not normal human behaviour. ↩
\nI don't really mind AI crawlers, as long as they behave.\nThey usually don't. ↩
\nAnubis is great!\nEspecially since it lets through the bots that clearly identify as a bot. ↩
\nMy backup strategy follows the 3-2-1 backup rule:
\nSome people argue that your production data (the copy stored on your laptop, for example) counts as one of those copies.\nI disagree.\nWhile it may be a copy of your current data, it's not a copy of your historical data.\nIf I need to recover a file that I modified or deleted 2 years ago, my laptop won't have it.
\nI use Restic to write encrypted backups to a 2.5" SSD inside an external USB enclosure.\nThis disk also contains a few scripts to automate the process.\nFriday is backup day: connect the disk, run the script, done.
\nUsing rsync, I make copies of this master disk:
To another USB 2.5" hard drive located at the office.\nThis is a physical copy I can easily access if my house burns down.\nRestic's encryption is vital here, as coworkers have access to the area where this disk is kept.
\nTo a dedicated server at Hetzner.\nA few friends have SSH access1 to the account that stores this copy.\nThis provides another "worst-case scenario" path to my data.
\nEvery few months I run a validation script on my Hetzner server.\nThis executes Restic with the check --read-data subcommand on the copy stored there.\nA few days later, I know whether the data is still correct.\nWhile not exactly the same as a full restore test, it gives me some piece of mind.
Important folders on my desktop and laptop are synced using Syncthing.\nIn addition to my personal devices, these also replicate to one of my servers.\nFrom there, an hourly Restic snapshot is made to Backblaze B2.\nIn case my 3-2-1 drives fail, I have one last chance to recovery my important documents here.
\nI was assured my backup scheme was solid, until it wasn't.
\nSuddenly, the data validation on the Hetzner server failed.2\nRestic saves backups as deduplicated chunks in what it calls "pack files."\nSeveral of these files no longer matched their checksums.\nSince pack files are named after their SHA-256 hash, I checked them again manually.\nSadly, sha256sum confirmed the worst: the data in these files was not what it was supposed to be.
This server has two 4TB disks in RAID 1.\nI assumed there was a hardware issue with the server, but everything looked fine.\nThe other services on the server ran fine, the disks reported no errors, and the RAID was healthy.
\nThen I realized: the Hetzner server was not the source of corruption.\nChecking the pack files on my USB disk quickly revealed the truth.\nThese files were also corrupt, and rsync had dutifully replicated the corrupted data to my server.
It quickly became obvious what was happening.\nDuring additional checks on my USB disk, my laptop started reporting I/O errors.\nThe SSD in this enclosure has been failing silently for a while, returning random data instead of my files.\nI hadn't noticed because I rarely use these backups.\nThe data is usually only read by rsync to push it to the server.
\nAn hour later, the SSD died completely.
\nTwo days later, when I was back at the office, I checked the second USB drive.\nI had little hope.\nI usually sync both the Hetzner server and this disk at the same time.\nAs expected, the same errors were present.
\nI had managed to corrupt all three copies of my backups simultaneously.
\nThere is no magic recovery procedure here.\nIf all copies of a piece of data are gone, they are gone forever.\nRegrettably, I did lose some data that day.
\nHowever, Restic was quite helpful during the recovery process:
\nA nice side-effect of the data deduplication is that you can "import" files again.\nBy creating a temporary snapshot with files gathered from other locations (like old laptop folders or emails), Restic recreates the deduplicated chunks.\nThis makes those files usable again in the older, previously "corrupt" snapshots!
\nDry-running restic repair snapshots --forget listed exactly which files would be removed due to missing data.\nUse this as a to-do list to gather missing files.
Part of the missing data turned out to be logfiles I didn't care about.\nPurging these from snapshots made it a lot easier to identify which important files were actually missing.3
\nIn the end, I lost only three possibly important files: pictures.\nSince the data is gone, I don't know exactly what was in them, but I know which albums they belonged to.\nMy backup disk now contains a text file listing those three missing filenames as a permanent reminder.
\nClearly, I needed safeguards.\nI made two specific changes to the script I use to sync backups from my local USB SSD to the other destinations:
\nBefore executing rsync, the script now runs restic check.\nWhile a full --read-data takes too long for a weekly routine, a basic check catches structural errors.\nBefore my previous SSD died, the basic check also found errors.\nI'm not sure it would have catched the SSD returning small bits of random data, though.
Except for the locks directory, restic never overwrites files.\nIt only creates new ones.\nOld files stay until you run restic forget to clean up your repository.\nI now run rsync with the --ignore-existing flag, as there is no reason to overwrite remote files.
I'm confident that this would prevent data loss if a drive fails silently again.
\nOnce hard drive prices stabilize, I'll likely buy another external drive for a yearly "cold" archive.\nThe hardest files to replace where those that have been removed from my devices years ago.\nHaving an additional offline drive with an old copy would allow me to recover those files.
\nNot all of them know about it though. ↩
\nI would have never discovered about this if it wasn't for this regular data check.\nBackups are not backups if you don't check them. ↩
\nFiguring out what data I had lost took a few days and I didn't really take notes.\nDocumenting which commands I used here would have been the most interesting part of the article.\nOh well.. ↩
\nSQLite does not have built-in UUID functions, but by getting creative with the randomblob(N) function we can generate version 4 (random) UUIDs.
Before we continue, it's important to note that you should verify whether SQLite's random functions are adequate for your usecase.\nThey may not return truly random data.\nHowever, as the RFC notes, "they (UUIDs) MUST NOT be used as security capabilities".\nAnd since we all follow that to the letter, it doesn't really matter.. right?
\nThe excellent SQLite manual gives us a starting point:
\n\n\nHint: applications can generate globally unique identifiers using this function together with hex() and/or lower().
\n
Combining the randomblob(N) function with hex(N) does indeed create something that vaguely resembles a UUID.\nFor some purposes this may suffice, but the result isn't actually a valid UUID1:\nvalid UUIDs must have specific bits set to indicate their version and variant.
We can do better!
\nThe binary representation of a version 4 UUID consists of:
\nAs far as I know, we can't modify individual bytes in a blob using the available functions in SQLite.\nHowever, we do have access to a lot of string functions.\nTaking a detour using HEX encoding provides a solution.\nWe'll first concatenate a HEX string containing the data we want, then convert it back to a binary blob.\nNote that each byte is represented by two hexadecimal characters.
\nThe first 48 random bits are easy:
\nsqlite> select HEX(RANDOMBLOB(6));\r\n4004714237AF\r\nThe first 4 bits of the next byte need to be set to 0100 (or 4 in hex).\nAs every byte is represented by two characters in hexadecimal, we can just use a literal 4 for the next character.
sqlite> select '4';\r\n4\r\nNext, we need 6 bits of random data.\nWe can only generate random bytes using RANDOMBLOB(N), but we need 1.5 bytes.\nLuckily, because we are working with a hexadecimal string, we can generate 2 bytes (4 hex characters) and take the first three characters.
sqlite> select HEX(RANDOMBLOB(2));\r\n7B4A\r\n\r\nsqlite> select SUBSTR(HEX(RANDOMBLOB(2)),0,4);\r\n5BD\r\nThe next one is a little more complex.\nThe variant field consists of a variable number of bits.\nFor our version 4 UUID, we need to set the first two bits to 10, the others can be random.\nThat gives us a possible range between 1000 and 1011, or a hexadecimal character between 8 and B inclusive.
select FORMAT('%X', 8 + ABS(RANDOM() % 4));\r\nA\r\nAfter this, we need 60 more random bits.\nSince this again includes a half byte again (7.5 bytes), we're doing the same SUBSTR trick as before:
sqlite> select SUBSTR(HEX(RANDOMBLOB(8)),0,16);\r\nE83EF2136842130\r\nUsing CONCAT(X, ...) we create a single string:
sqlite> select CONCAT(\r\n(x1...> HEX(RANDOMBLOB(6)),\r\n(x1...> '4',\r\n(x1...> SUBSTR(HEX(RANDOMBLOB(2)),0,4),\r\n(x1...> FORMAT('%X', 8 + ABS(RANDOM() % 4)),\r\n(x1...> SUBSTR(HEX(RANDOMBLOB(8)),0,16)\r\n(x1...> );\r\n\r\n64DF281823BE47249BB2CA8D6450066B\r\nThen use UNHEX(X) to convert it back into a binary blob:
select UNHEX(CONCAT(\r\n HEX(RANDOMBLOB(6)),\r\n '4',\r\n SUBSTR(HEX(RANDOMBLOB(2)),0,4),\r\n FORMAT('%X', 8 + ABS(RANDOM() % 4)),\r\n SUBSTR(HEX(RANDOMBLOB(8)),0,16)\r\n));\r\nThe SQL above can be used as the default value for a blob column:
\nCREATE TABLE "animals" (\r\n "id" INTEGER PRIMARY KEY NOT NULL,\r\n "uuid" BLOB UNIQUE NOT NULL\r\n DEFAULT (UNHEX(CONCAT(\r\n HEX(RANDOMBLOB(6)), '4',\r\n SUBSTR(HEX(RANDOMBLOB(2)),0,4),\r\n FORMAT('%X', 8 + ABS(RANDOM() % 4)),\r\n SUBSTR(HEX(RANDOMBLOB(8)),0,16))))\r\n UNIQUE CHECK(LENGTH("uuid") = 16),\r\n "name" TEXT NOT NULL\r\n) STRICT;\r\nThis will generate a UUID if we don't specify one on insert:
\nsqlite> INSERT INTO "animals" ("name") VALUES ('Fluffy');\r\n\r\nsqlite> SELECT * FROM "animals";\r\n1|[blob]|Fluffy\r\n\r\nsqlite> SELECT "id", HEX("uuid"), "name" FROM "animals";\r\n1|85671F7120F14AA6802B8B4E9213B8BE|Fluffy\r\nDepending on your use case, you may want to store UUIDs in a TEXT column.
We were close to a "hex-and-dash" string format with the SQL above, until we call the UNHEX(X) function.\nRFC 9562 specifies that dashes are required, so we'll add them in the right places.
sqlite> select CONCAT(\r\n HEX(RANDOMBLOB(4)),\r\n '-',\r\n HEX(RANDOMBLOB(2)),\r\n '-',\r\n '4',\r\n SUBSTR(HEX(RANDOMBLOB(2)),0,4),\r\n '-',\r\n FORMAT('%X', 8 + ABS(RANDOM() % 4)),\r\n SUBSTR(HEX(RANDOMBLOB(2)),0,4),\r\n '-',\r\n SUBSTR(HEX(RANDOMBLOB(8)),0,13)\r\n);\r\n\r\n199ED82A-3102-42DB-B59B-8C356B6F3FF7\r\nVoila!
\nAs technically correct is the best kind of correct: if you get really lucky with the generated random data, they might actually be valid. ↩
\nToday, I worked on a ticket for a customer with a Craft 4.x CMS that created a MySQL table with a staggering 270 columns.\nMost of them were of type TEXT, but it also contained about 50 VARCHAR columns of varying lengths.\nAttempting to import the table resulted in a Row Size Too Large error.
ERROR 1118 (42000): Row size too large (> 8126). Changing some columns to TEXT or BLOB may help. In current row format, BLOB prefix of 0 bytes is stored inline.\nThe first attempt to resolve the issue involved converting all columns to TEXT, as their contents are stored off-page. That didn't work: there were too much columns, and even the references consumed too much space.
Next, I considered changing the row format, which can sometimes resolve this issue. Unfortunately, the customer had already set it to DYNAMIC, the best available option for this scenario. No gains there.
The obvious fix for running out of space is creating more space.\nMySQL has an innodb_page_size parameter that defines the tablespace's page size.\nThe data file consists of virtual pages, which are completely overwritten each time a change is made.
MySQL stores two rows in a page, meaning that when you update one, you're effectively saving two.\nThis is an important setting for performance, but in this case, it's relevant for another reason: a row can use at most half of the InnoDB page size.
\nThe innodb_page_size setting is somewhat misleading because its value is stored inside the ibdata1 file, which contains your MySQL data. Once the first page has been written, you can't change the value anymore.\nAll pages inside the file must be the same size.
Thus, changing it means recreating everything, which involves a process like this:
\nmysqldump --all-databases > backup.sql\nsystemctl stop mysql\nrm -rf /var/lib/mysql/*\nmysqld --user mysql --initialize --innodb-page-size=32K\nsystemctl start mysql\ncat backup.sql | mysql\nHowever, on our Percona PXC cluster, I was met with an error:
\n2025-03-04T13:50:32.929013Z 0 [ERROR] [MY-010872] [InnoDB] InnoDB: Invalid page size=32768.\nThat was surprising, as the documentation lists 32768 (32K) as a valid option.
I spent way too much time debugging this because the error message didn't lead to any useful documentation.\nIt turns out that Percona PXC cluster does not allow changing innodb_page_size.\nThey only accept the default value of 16K, and any other setting will fail.
From (an older) Percona PDF manual:
\n\n\nOnly the default 16 KB page size (innodb_page_size=16384) is accepted until the relevant upstream bug is fixed by Codership (see https://github.com/codership/galera/issues/398).\nAll other sizes will report Invalid page size and shut down (the server will not start up).
\n
Newer documentation doesn't seem to mention this anymore, but it still applies.
\ninnodb_strict_modeWhile researching this issue, I found many Stack Overflow posts suggesting that disabling innodb_strict_mode might help. Be very careful with this!
Disabling strict mode only replaces the critical error with a warning. You'll be able to create the table, but the underlying issue remains: the rows are still too large.
\nIt may not fail immediately, but at some point, an INSERT or UPDATE operation will trigger an error if it exceeds the maximum size. This shifts the failure from a controlled event (such as during deployment) to a random action that a user might perform — not ideal.
Customers regularly send me x509 key pairs at work1.\nMost of the time these have been forwarded a few times already, and people tend to mess up the files.\nThis means that the public and private keys don't match anymore, and I have to ask for a new pair.
\nLuckily, we can use OpenSSL to check whether a public and private key match.
\nThe classic way to compare RSA keys is to check whether the modulus is the same, as this is the only part that is shared between the public and private keys.\nIf the modulus isn't the same, the keys can't form a pair.
\nRead the modulus from the public key:
\nopenssl rsa -pubin -in public.pem -modulus -noout | openssl sha1\nThen do the same with the private RSA key:
\nopenssl rsa -in private.pem -modulus -noout | openssl sha1\nIf the two values match, the key files form a pair.
\nNote that we're piping the actual output of the -modulus command to openssl sha1 to get a shorter hash of the modulus.\nThis makes it easier to manually compare the two values, but it's not strictly necessary.\nYou could also compare the output directly if you're feeling brave.
Elliptic curve keys don't have an easy-to-compare modulus like the RSA keys above had.\nHowever, it's always possible to generate the public key2 part when we have the private key.
\nUsing the private key, generate (a hash of) the public key:
\nopenssl pkey -pubout -in private.pem | openssl sha1\nThen compare the output with the public key file you have.\nWe need to make openssl load it and print it out again to make sure we get the exact same format.
openssl x509 -pubkey -in public.pem -noout | openssl sha1\nIf the two values match, these keys form a pair.
\nPlease don't do this. Ask the company that needs a certificate to generate a private key and provide you with a Certificate Signing Request (CSR). This way, you don't have to handle the private key at all. ↩
\nWhen people say "public key" in the context of a x509 pair they usually refer to the certificate. The public key we're extracting here is the actual cryptographic public key, it doesn't contain the extra data like domain names that a certificate usually holds. You can't recover the full certificate when you only have the private key. ↩
\nKev Quirk recently wrote about removing content from public blogs:
\n\n\nGenerally speaking, I don't delete content from this site.\nHaving said that, if I posted something that I later feel is particularly egregious, I think I probably would.\nI personally don't think that a website should be a permanent record - how can it be?\nNothing last forever.
\n
This got me thinking about what to do with some of my older articles.\nI believe that URLs should be stable.\nThe internet is a large place, there’s no way to know who might be linking to any of my webpages.\nRemoving a page may also break other websites: someone may have written about it.\nDeleting the page would remove the context of their reference.
\nHowever, I’ve been working on this website in some form or another since 2007.\nTechnical articles tend to get outdated, and people change their minds about things they’ve written.\nSome of the articles on this website no longer reflect what I stand for.
\nArticles older than two years already include a banner noting their age, but that’s not always enough.\nNowadays, people often form an opinion based on the title alone — especially on social media.\nIt’s unlikely they’d notice a note about how old the article is.
\nSo I decided to hide certain articles.\nHidden articles are no longer linked on my website and don't appear in RSS feeds.\nYou can’t find them by clicking around.\nBut if you already have the link, the content is still accessible.\nBecause, as they say, cool URIs don't change.
\n","url":"https://vic.demuzere.be/articles/hiding-outdated-articles/","date_published":"2024-11-10T10:04:30Z","date_modified":"2024-11-11T18:18:07Z","language":"en"},{"id":"https://vic.demuzere.be/articles/pwgen-not-secure-by-default/","content_html":"Command-line tool pwgen is a well known tool to generate a list of passwords.\nI use it almost daily: when creating new accounts on remote servers, or when I want to encrypt sensitive data with a temporary password to send to a customer.\nSometimes opening my terminal and typing pwgen 30 1 is faster than getting to the generator in my password manager.1
I had noticed before that the passwords generated by pwgen seemed to contain patterns: repeating vowels is something that stands out.\nI hadn't thought about it much, until I saw a colleague type pwgen -s at his terminal.
-s?As with most unix-y tools, a quick glance at the man page tells us everything we want to know.\nThe -s is shorthand for --secure:
\n\nGenerate completely random, hard-to-memorize passwords.\nThese should only be used for machine passwords, since otherwise it's almost guaranteed that users will simply write the password on a piece of paper taped to the monitor...
\n
As it turns out, pwgen actually doesn't generate secure passwords by default.\nThe passwords it generates are designed to be easily memorized.\nThat explains the patterns I noticed before!2
Recently, I was looking for an outside temperature sensor. Preferably in an instrument shelter, as my whole garden has direct sunlight during most of the day. I ended up buying the Alecto WS-5500, which is actually a rebranded Fine Offset WH2900. It has a decent temperature sensor and also supports uploading data to a configurable HTTP endpoint.
\n
The inside unit that comes with the WS-5500 can be configured using the WS View Android application. It's able to upload data to a custom HTTP endpoint, provided that you use the same protocol used by either Weather Underground or Ecowitt.net.
\nWeather Underground uses a HTTP GET request. This makes for rather long URLs since we have to put all the data into query parameters. In addition, the configured password is also visible in the URL and thus saved in my access logs. I didn't want to go down this path.
\nEcowitt uses POST requests which seems more sane.
\nData is sent through a POST request in standard application/x-www-form-urlencoded encoding. Looks like the contents depends on what sensors are connected to your weather station. I've found examples that contained other fields than I found in the data sent by the WS-5500.
For anyone else that wants to write a script to capture data from the all-in-one Alecto WS-5500 or Fine Offset WH2900, the fields below are sent in the POST request when Ecowitt protocol is selected:
\nstationtype=EasyWeatherV1.5.9\nmodel=WS2900_V2.01.12\nfreq=868M\n\nwh65batt=0\n\ndateutc=2021-06-21 20:16:45\n\ntempinf=70.9\nhumidityin=59\nbaromrelin=29.944\nbaromabsin=29.841\ntempf=59.2\nhumidity=86\nwinddir=355\nwindspeedmph=2.9\nwindgustmph=4.5\nmaxdailygust=12.5\nrainratein=0.000\neventrainin=0.000\nhourlyrainin=0.000\ndailyrainin=0.000\nweeklyrainin=0.000\nmonthlyrainin=0.000\ntotalrainin=0.000\nsolarradiation=0.00\nuv=0\nAs we use different units in Belgium, I used these conversions:
\n(fahrenheit - 32.0) * (5.0 / 9.0)inches * 33.6585mph * 1.60934inches * 25.4Due to the special characters used in json, the easiest way to use jq with inline scripts it by putting it between single quotes.\nThat makes it impossible to use bash variables inside your script.\nFortunately, jq has an --arg parameter to create a predefined variable from an external source.
You can use it to define a $foo variable with the contents of bash variable $FOO, for example:
$ export FOO="bar"\n$ jq -n --arg foo "$FOO" '{foo: $foo}'\n{\n "foo": "bar"\n}\nYou might notice the -n parameter: This tells jq to use null as input instead of reading json from stdin like it normally would.\nI'm using it in all examples on this page as it makes them a bit shorter.\nIn real world usage, you probably won't need it.
As usual in bash scripts, you can also execute a command inline:
\n$ jq -n --arg hostname "$(hostname)" '{foo: $hostname}'\n{\n "foo": "cypher"\n}\nEvery variable created with --arg is treated as a string.\nThat might cause some unexpected behavior when you work with numbers, like when you attempt to add a number to an argument.\nComparisons between strings and numbers seem to work fine, but I wouldn't count on it (pun intended).
$ echo '{"a": 3.0}' | jq --arg foo 5 '$foo + .a'\njq: error (at <stdin>:1): string ("5") and number (3) cannot be added\nThere are two ways to solve this: either convert the variable to a number with tonumber or use --argjson instead.\nI prefer to use tonumber because that way I'm sure jq forces it to be a number.\nAs my argument may be the result of an earlier command, I may be using an error message as input instead of the expected numeric result.
$ jq -n --arg foo 5 '($foo|tonumber) + 1'\n6\n\n$ jq -n --argjson foo 5 '$foo + 1'\n6\nIt's also possible to load variables from files.\nWith --slurpfile we can read a file containing json objects.\nAn array containing the separate objects is then made available as a predefined variable.\nNote that even if your file contains only a single json object, as the variable is still an array, the [0] is required to access it.
$ cat foo.json\n{"foo": "bar"}\n{"bar": "foo"}\n\n$ jq -n --slurpfile foo foo.json '$foo[0].foo'\n"bar"\nA second option is --rawfile, creating a string variable with the exact contents of the file.\nNote that as jq outputs json, reading and printing the file from the previous example results an escaped string.
$ jq -n --rawfile foo foo.json '$foo'\n"{\\"foo\\": \\"bar\\"}\\n{\\"bar\\": \\"foo\\"}\\n"\nThat's it!
\n","url":"https://vic.demuzere.be/articles/using-bash-variables-in-jq/","date_published":"2021-02-10T20:01:03Z","date_modified":"2024-11-11T18:21:44Z","language":"en"},{"id":"https://vic.demuzere.be/articles/not-so-redesign/","content_html":"Over the past few months I've been working on a new stylesheet for this website, and today I finally pushed the redesign branch to production!\nThe previous CSS code was largely written almost 10 years ago and only received small updates and enhancements.\nSince then, new features like flexbox and grids were added and made creating responsive websites much easier.
\nTurns out, the new design looks exactly the same as the old one!
\nA few things did change though:
\nfloat to position elements in favor of flexbox.I also tried to get some insight in how people used the website:
\nMost of the traffic to this website is routed through Cloudflare's CDN.\nI've configured it to cache everything, which means the access logs on my web server record only part of the actual traffic.\nThis creates an interesting dataset: I can see which pages are most visited, but I can't see spikes due to pages being shared somewhere.\nFor the same reason, my statistics about browsers, operating systems, referring websites and returning visitors are completely skewed.
\nArticles are the most popular pages on this website.\nThat makes sense, as those tend to show up in searches and are shared more frequently than others.\nMy most popular articles are technical ones:
\nThat means I need to write more articles!
\nThe most common 404s appear to be scanners looking for vulnerabilities.\nWordPress related pages like wp-login.php, xmlrpc.php and several files inside the wp-includes directory lead the race, followed by Joomla's administrator directory.
My work to fix broken links (using data from the access logs and Google Search Console) seems to have paid off.\nI've also been trying not to delete or move pages for a few years now.\nWhen a page is no longer relevant, I replace it with a small message pointing people to alternatives, like I did with my OTR fingerprint listing.
\nAnother interesting 404 page is /irc:irc.quakenet.org/sorcix,isnick which seems to be caused by a spider that doesn't understand irc: URLs.
I've been looking into IndieWeb after reading a few blog posts by Jan-Lukas Else.\nPeople can reply to your articles by linking back from their own website, or by using one of the provided IndieWeb-compatible services.\nAs I'm not active on mainstream social media, this may be a great way to connect with like-minded people.
\nFeatures I want to roll out next:
\n