Jujens’ blog

Jouer sous Linux

2025-12-20T00:00:00+01:00

Après plusieurs tests infructueux l’année dernière, j’ai décidé de retester le jeu sous Linux en avril 2025 avec une nouvelle config. Et je dois admettre que ça fonctionne vraiment très bien. Je n’ai presque pas de problèmes et un seul jeu aurait nécessité de repasser sous Windows le temps qu’un correctif le corrige. Il y a toutefois quelques points sur lesquels j’ai un peu coincé et qui méritent qu’on les détaille. Ce que je vais faire dans cet article.

J’ai essayé de faire des sections claires pour vous permettre de ne lire que ce qui vous intéresse.

TL;DR : ça fonctionne très bien et je ne joue plus du tout sous Windows. On peut quand même avoir des petits soucis qui finissent par être corrigés. Si vous voulez être sûr de pouvoir jouer à n’importe quel jeu dès sa sortie, mieux vaut garder un Windows dans un coin au cas où.

Contents

Contexte
Vers le jeu sous Linux
- Avec Steam
- Et GoG ?
Les jeux

Contexte

Tout d’abord, un peu de contexte. J’ai essayé l’année dernière vers décembre après que des collègues m’aient dit jouer sous Linux sans problème. Malheureusement, sur mon ancienne config avec une vieille Nvidia GTX 1070, je n’ai jamais réussi à lancer les jeux Steam à cause d’une erreur sur la carte graphique. Mes recherches n’ont rien donné et après plusieurs tests, j’ai lâché l’affaire et je suis retourné sous Windows.

Je suppose que le souci n’était pas matériel ni au niveau de l’OS car mes jeux GOG fonctionnaient sans souci apparent (je n’ai pas poussé mes tests très loin).

En début d’année 2025, comme plusieurs jeux qui m’intéressaient allaient sortir et que ma machine commençait à vieillir, j’ai décidé de refaire ma config. C’est surtout la carte graphique qui posait problème : j’avais acheté un écran 1440p fin d’année dernière et la GTX 1070 montrait ses limites sur les jeux récents. En 1080p, je trouve qu’elle fonctionne toujours très bien et les jeux, même récents, restaient beaux. Au delà des problèmes de performances, les mises à jour du noyau avaient tendance à rater sous openSUSE Tumbleweed (ma distro) car les pilotes ne sont pas mis à jour rapidement dans le dépôt Nvidia.

J’ai décidé de partir sur une carte AMD RX9070XT, en grande partie à cause de ces soucis de support des cartes Nvidia sous Linux. De ce que j’ai compris, sur les modèles plus récents, le pilote officiel est mieux intégré au noyau et on évite toutes ces galères. Je pense quand même qu’AMD est un meilleur choix, ne serait-ce que pour leur meilleur support de Linux depuis de nombreuses années. Tout test simple : rien à installer, tout vient de base avec le noyau.

Je signale quand même un souci que j’ai eu avec la carte : le son pouvait produire des genres de craquements via Display Port à la lecture de musique (je n’ai jamais eu de souci en jeu bizarrement). Une mise à jour de noyau vers une nouvelle version a résolu le souci. Comme la carte est un modèle récent, je pardonne. Cela reste un point à garder en tête : si vous prenez du matériel récent, mieux vaut une rolling release ou une distro comme Fedora bien à jour.

Une fois la config montée, j’ai commencé par vérifier que tout semblait bon sous Windows en jouant un peu. J’ai vite eu l’impression que le système n’était pas très stable avec plusieurs jeux qui ont planté et bien plus qu’avec ma vieille GTX 1070. Surement, car je n’ai pas réinstallé Windows en changeant de carte. J’ai simplement désinstallé les pilotes Nvidia et installé ceux d’AMD. Je suppose qu’une réinstallation propre de Windows règlerait le souci, mais, comme j’arrive à jouer sous Linux sans souci, eh bien, je ne vais pas me prendre la tête avec ça. Windows va rester installé au cas où, ne serait-ce que pour déboguer mon imprimante.

Vers le jeu sous Linux

Avec Steam

Comme dans mes tests de fin 2024, j’ai eu des soucis avec Steam, quand bien même c’est ce que tous les joueurs sous Linux que je connais me vendaient comme la panacée. J’ai décidé de passer par le Flatpak pour que les jeux n’aient pas accès aux documents système par sécurité. Cela vient peut-être du fait que je me souviens d’un gros bogue de Steam qui supprimait tous les fichiers de $HOME il y a quelques années.

Par défaut, seuls les jeux natifs Linux peuvent être installés et lancés. Pour lancer n’importe quels jeux, il faut activer Steam Play pour tous les jeux dans les paramètres section "Compatibilité". Une fois cela fait, je télécharge un jeu et j’essaie de le lancer.

Et c’est le drame. Rien ne se passe. Même pas un message d’erreur.

J’essaie avec un autre jeu, et même problème. J’essaie différentes versions de Proton (la couche de compatibilité de Valve pour jouer aux jeux Windows sous Linux qu’on peut configurer en cas de souci). Sans succès.

Je me suis dit qu’il me manquait peut-être certains paquets. J’ai essayé d’en installer plusieurs en lien avec Mesa ou AMDGPU. Sans succès.

J’ai essayé un jeu GOG via Heroic Game Launcher qui a fonctionné sans problème avec Wine.

Finalement, j’ai essayé d’installer le paquet Steam fourni par les dépôts. Le gestionnaire de paquets a ajouté plusieurs autres paquets et, miracles, je pouvais jouer. J’ai réessayé la version Flatpak qui fonctionne aussi correctement une fois ces paquets installés. J’ai décidé de rester sur la version Flatpak avec le paquet installé et pour séparer mes activités et être sûr qu’aucun jeu n’aura accès à mes fichiers, j’ai aussi créé un utilisateur dédié pour le jeu. Mais c’est peut-être juste moi qui suis un peu parano.

Après plusieurs mois de jeux sous Linux, je ne me vois plus lancer un Windows. Ça fonctionne trop bien ! Je suis même surpris d’à quel point ça fonctionne bien, y compris sur des jeux comme Avowed, The Outer Worlds 2, Assassin’s Creed Shadows qui sont sortis cette année.

J’ai même réussi à jouer à Mass Effect Legendary Edition et Dragon Age Inquisition qui nécessite cette 🤬 d’EA app. Notez que je passe par Steam pour lancer les jeux : Steam installe et lance l’EA app automatiquement et ça fonctionne aussi bien (ou mal) que sous Windows. Il suffit de se connecter à l’EA app. Malheureusement, après une mise à jour de l’EA app, DA:I ne se lançait plus (idem pour Mass Effect. Supprimer manuellement un fichier de cache a réglé le souci. Pour cela, il faut : trouver l’id Steam du jeu (visible dans les paramètres du jeu dans la section "Mise à jour") puis supprimer le dossier ~/.var/app/com.valvesoftware.Steam/.local/share/Steam/steamapps/compatdata/<APP_ID> (c’est le chemin avec Flatpak, ça devrait être quelque chose comme ~/.local/share/Steam/steamapps/compatdata/<APP_ID> pour la version standard). Ça me confirme dans l’idée que l’EA est naze.

Quelques petits soucis quand même :

Sur Dragon Age the Veilguard, les coiffures de certains personnages se sont mises à scintiller après une mise à jour système. Le jeu restait jouable, mais avait un aspect très bizarre. Une autre mise à jour quelque temps après a réglé le souci.
Avec Indiana Jones and the Great Circle, j’ai dû attendre plusieurs mois avant que le jeu ne se lance correctement. Les textures n’étaient pas rendues correctement et le jeu était injouable. C’est le plus gros souci que j’ai eu et le seul jeu non jouable.

Et GoG ?

En plus de Steam, j’ai aussi une belle bibliothèque sur GoG. Même si elle n’a pas d’app compatible Linux, elle reste ma boutique de choix : les jeux n’ont pas de DRM et elle est basée en Europe. Pour me simplifier la vie, j’ai choisi de passer par Heroic Game Launcher installé via Flatpak (qui gère aussi les jeux Epic Games Store). C’est surtout pour ça que j’aime Flatpak : être sûr d’avoir des logiciels pas très courants correctement packagé pour Linux.

En soi, le fonctionnement est très proche de Steam : on peut installer les jeux et les lancer. L’app se charge de les garder à jour et traque le temps de jeu. Les succès ne sont pas visibles dans l’app, mais sont correctement débloqués (j’ai vérifié sous l’app GoG sous Windows).

Petit point de configuration pour que les jeux natifs Linux se lancent : il faut configurer le lien vers les fichiers de compatibilité de Steam. Sans, certains jeux ne peuvent pas se lancer, car ils nécessitent une vieille version d’openSSL. Pour cela, il faut s’assurer que le paramètre "Chemin par défaut vers Steam" pointe bien vers ~/.var/app/com.valvesoftware.Steam/.local/share/Steam (je suppose que c’est ~/.local/share/Steam pour la version standard). Ça devrait être le cas par défaut.

Les jeux

Pour terminer, voici une liste des jeux auxquels j’ai joué cette année sous Linux avec la date de sortie du jeu, l’application utilisée (Steam ou Heroic) et quelques commentaires. Principalement mon temps de jeu ou si j’ai eu des soucis (et dans ce cas, lesquels).

Dragon Age: the Veilguard, 2024, Steam : presque complet (j’ai démarré sous Windows). Le jeu est plus stable avec les paramètres élevés que sous Windows. Quelques plantages à l’utilisation du pouvoir ultime ou lors de très gros combats. Réduire le ray tracing a résolu le souci. J’ai eu exactement le même problème au même endroit sous Windows avec mon ancienne config.

Le curseur n’est pas le même que sous Windows et semble troué. Étrange, mais pas impactant.

Plus grave, j’ai eu un souci avec le rendu des cheveux après une mise à jour : ils scintillaient. Le jeu restait jouable, mais était très moche. Ça a mis plusieurs semaines à revenir à la normale avec une mise à jour système.
Pillars of Eternity, 2015, Heroic : jeu de base complet, sur mon portable. J’ai utilisé la version Linux avec les compatibilités Steam pour éviter les soucis avec la vieille version d’openSSL dont le jeu a besoin. Ça fonctionne très bien.
Avowed, Steam, 2025 : fonctionne bien en paramètres max, et ce, dès sa sortie. Je n’ai pas eu un seul crash sur la totalité du jeu. Seul petit souci rencontré : des gels graphiques de temps en temps, en général après un changement de zone. Ces gels disparaissent après quelques secondes. C’est plus agaçant qu’autre chose.
Age of Empires IV, 2021, Steam : j’ai joué plusieurs dizaines d’heures en solo et en multi sans souci.
Age of Mythology Retold, 2024, Steam : j’ai joué plusieurs dizaines d’heures en solo et en multi sans souci.
Indiana Jones et le cercle ancien, 2024, Steam : j’ai joué plusieurs heures sans souci après des débuts compliqués. Quand j’ai essayé pour la première fois, les textures étaient complètement cassées. Ça a fini par fonctionner après une mise à jour système. J’ai des lenteurs quand la caméra tourne même en texture moyenne ou avec fsr et textures élevées ou moyennes. C’est le seul jeu avec ce problème et celui qui fonctionne le moins bien. Comme je n’y ai pas accroché, je n’ai pas cherché plus loin.
Mass Effect Legendary, 2021, Steam : jeu complet, aucun souci. Après avoir eu des soucis à cause de l’EA app avec Dragon Age Inquisition (voir le jeu suivant), j’ai essayé de le relancer et j’ai eu le même comportement. La même solution a corrigé le souci.
Dragon Age Inquisition, 2014, Steam : jeu complet, y compris les DLCs. J’ai eu quelques bogues de quêtes qui se valident en boucle. La quête finit par se valider correctement après plusieurs "essais". J’ai aussi eu un plantage avant le boss de fin.

Plus grave : suite à une mise à jour de l’EA app, le jeu ne se lançait plus du tout. J’ai dû vider le cache en supprimant un dossier à la main. Voir le détail dans la section avec Steam.
Baldur’s Gate 3, 2023, Heroic : quelques heures, RAS.
Dragon Age Origins, 2009, Heroic : j’ai seulement fait le prologue, RAS.
Witcher 3, 2015, Heroic : plusieurs heures de jeux, RAS.
The Outer Worlds Spacer Choice edition, 2019, Heroic : jeu complet, RAS.
Pathfinder Wrath of the Righteous, 2021, Heroic : quelques heures, RAS.
Assassin’s creed Odyssey, 2018, Steam : quelques heures, RAS.
Assassin’s creed Shadows, 2025, Steam : jeu de base, joué dès sa sortie, RAS.
Horizon Zero Dawn, 2017, Steam : jeu complet, RAS.
The Outer Worlds 2, 2025, Steam: jeu complet, joué dès sa sortie, RAS.
Horizon forbidden west, 2022, Steam : quelques heures, RAS.
Spellforce 3 versus, 2017, Steam : quelques heures, RAS.

Gaming on Linux

2025-12-20T00:00:00+01:00

After several unsuccessful tests last year, I decided to retest gaming on Linux in April 2025 with a new setup. And I have to admit that it works really well. I had almost no problems and only one game would have required switching back to Windows while waiting or a patch that eventually fixed it. There are, however, a few points on which I got a little stuck and which deserve to be detailed. That’s what I'm going to do in this article.

I tried to make clear sections to allow you to read only what interests you.

TL;DR: It works great and I don't play Windows at all anymore. You can still have small problems that end up corrected after a few months (at most). If you want to be sure you can play any game as soon as it comes out, it's probably best to keep a Windows just in case.

Table of contents

Context
Towards gaming on Linux
- With Steam
- And with GoG?
The games

Context

First, a little context. I tried it last year around December after colleagues told me they could play Linux without problems. Unfortunately, on my previous config with an old Nvidia GTX 1070, I never managed to launch Steam games due to an error relating with the graphics card. My searches turned up nothing and after several tests, I gave up and went back to gaming on Windows.

I suppose the problem was not hardware or OS because my GOG games worked without any apparent problems (I didn't push my tests very far).

At the start of 2025, as several games I wanted to play were about to be released and my machine was starting to age, I decided to redo my setup. It was especially the graphics card that posed a problem: I bought a 1440p screen at the end of last year and the GTX 1070 showed its limits on recent games. On 1080p, I find that it still works very well and the games, even recent ones, remained beautiful. Beyond performance issues, kernel updates had a tendency to fail under openSUSE Tumbleweed (my distro) because drivers are not updated quickly in the Nvidia repository.

I decided to go with an AMD RX9070XT card, largely because of these support issues for Nvidia cards under Linux. From what I understand, on more recent models, the official driver is better integrated into the kernel and thus avoids all these hassles. I still think AMD is a better choice, if only for their good Linux support for many years. Everything is more simple: nothing to install, everything comes directly with the kernel.

I must still point out a problem I had with the card: the sound could produce some kind of crackling sound when used via Display Port when playing music (I never had a problem in game strangely). A kernel update to a new version resolved the issue. As the card is a recent model, I’ll forgive it. This remains something to keep in mind: if you take recent hardware, it is better to have a rolling release or a distro like Fedora that is always up to date.

Once the config was set up, I started by checking that everything seemed good under Windows by playing a little. I quickly had the impression that the system was not very stable with several games crashing much more than with my old GTX 1070. Probably, because I didn't reinstall Windows after changing cards. I simply uninstalled the Nvidia drivers and installed the AMD ones. I suppose a clean installation of Windows would solve the problem, but, since I can play Linux without problem, I'm not going to bother with that. Windows will stay installed just in case, if only to debug my printer.

Towards gaming on Linux

With Steam

As in my tests at the end of 2024, I had problems with Steam, even though that's what all the Linux players I know sold me as a panacea. I decided to use Flatpak so that games would not have access to system documents for security. Maybe it's because I remember a big Steam bug that deleted all files from $HOME a few years ago.

By default, only native Linux games can be installed and launched. To launch any games, you must activate Steam Play for all games in the settings "Compatibility" section. Once I did that, I downloaded a game and tried to launch it.

And… Nothing happened. Not even an error message.

I tried with another game, and same problem. I tried with a different version of Proton (the Valve compatibility layer for playing Windows games on Linux that can be configured in case of problems). Again, without any success.

I figured I might be missing some packages. I tried to install several related to Mesa or AMDGPU. Still no success.

I tried a GOG game installed via Heroic Game Launcher which worked correctly with Wine.

Finally, I tried to install the Steam package provided by the repositories. The package manager added several more packages and, hooray, I could play. I tried the Flatpak version again which also works correctly once these system packages are installed. I decided to stay on the Flatpak version. To separate my activities and be sure that no game will have access to my personnal files, I also created a dedicated user for the game. But maybe that's just me being a little paranoid.

After several months of playing Linux, I no longer see myself launching Windows for gaming. It works too well! I'm even surprised by how well it works, including on games like Avowed, The Outer Worlds 2, Assassin's Creed Shadows that came out this year.

I even managed to play Mass Effect Legendary Edition and Dragon Age Inquisition which requires this 🤬 EA app. Note that I use Steam to launch the games: Steam installs and launches the EA app automatically and it works just as well (or poorly) as under Windows. You just have to log in to the EA app. Unfortunately, after an update of the EA app, DA:I no longer launched (same for Mass Effect. Manually deleting a cache file fixed the problem. To do this, you must: find the Steam id of the game (visible in the game settings in the "Update" section) then delete the folder ~/.var/app/com.valvesoftware.Steam/.local/share/Steam/steamapps/compatdata/<APP_ID> (this is the path with Flatpak, it should be something like ~/.local/share/Steam/steamapps/compatdata/<APP_ID> for the standard version). This confirms my idea that the EA app sucks.

A few small problems though:

On Dragon Age the Veilguard, some characters' hairstyles started to sparkle after a system update. The game remained playable, but looked weird. Another system update some time later fixed the problem.
With Indiana Jones and the Great Circle, I had to wait several months before the game launched properly. Textures were not rendered correctly and the game was unplayable. This is the biggest problem I've had and the only case of non-playable game.

And with GoG?

In addition to Steam, I also have a nice library on GoG. Even though it doesn't have a Linux compatible app, it's still my store of choice: the games don't have DRMs and it's based in Europe. To make my life easier, I chose to go through the Heroic Game Launcher installed via Flatpak (this app also manages Epic Games Store games). This is why I like Flatpak: being sure to have not very common software correctly packaged for Linux by its developpers.

In itself, it’s very close to Steam: you can install the games and launch them. The app takes care of keeping them up to date and tracks playing time. Achievements are not visible in the app, but are correctly unlocked (I checked under the GoG app on Windows).

Small configuration point for native Linux games to launch: you must configure the link to the Steam compatibility files. Without it, some games cannot launch because they require an old version of openSSL. To do this, you must ensure that the "Default path to Steam" parameter points to ~/.var/app/com.valvesoftware.Steam/.local/share/Steam (I assume that it is ~/.local/share/Steam for the standard version). This should be the case by default.

The games

Finally, here is a list of games I played this year on Linux with the release date of the game, the application used (Steam or Heroic) and some comments. That’s mostly my playing time and whether I had any problems (and in this case, which ones).

Dragon Age: the Veilguard, 2024, Steam: almost complete (I started on Windows). The game is more stable at high settings than under Windows. Some crashes when using ultimate power or during very big fights. Reducing ray tracing solved the problem. I had exactly the same problem in the same place on Windows with my old config.

The cursor is not the same as in Windows and appears to have holes. Strange, but not impactful.

More serious, I had a problem with the rendering of the hair after an update: it was sparkling. The game remained playable, but was very ugly. It took several weeks to return to normal with a system update.
Pillars of Eternity, 2015, Heroic: complete base game, on my laptop. I used the Linux version with Steam compatibility to avoid issues with the old version of openSSL that the game needs. It works very well.
Avowed, Steam, 2025: works well in max settings, as soon as it is released. I didn't have a single crash in the entire game. Only small problem encountered: graphic freezes from time to time, generally after a change of zone. These gels disappear after a few seconds. It's more annoying than anything else.
Age of Empires IV, 2021, Steam: I played several dozen hours of solo and multiplayer without problem.
Age of Mythology Retold, 2024, Steam: I played several dozen hours of solo and multiplayer without problem.
Indiana Jones and the Old Circle, 2024, Steam: I played several hours without problem after a complicated start. When I first tried it, the textures were completely broken. It ended up working after a system update. I have slowness when the camera runs even in medium texture or with high or medium fsr and textures. This is the only game with this problem and the one that works the worst. Since I didn't like it, I didn't look any further.
Mass Effect Legendary, 2021, Steam: full game, no worries. After having problems because of the EA app with Dragon Age Inquisition (see the next game), I tried to restart it and had the same behavior. The same solution corrected the problem.
Dragon Age Inquisition, 2014, Steam: full game, including DLCs. I've had a few quest bugs that validate over and over again. The quest ends up being validated correctly after several "tries". I also had a crash before the end boss.

More serious: following an update of the EA app, the game no longer launched at all. I had to clear the cache by deleting a folder by hand. See the details in the section with Steam.
Baldur's Gate 3, 2023, Heroic: a few hours, works fine.
Dragon Age Origins, 2009, Heroic: I only did the prologue, works fine.
Witcher 3, 2015, Heroic: several hours of games, works fine.
The Outer Worlds Spacer Choice edition, 2019, Heroic: full game, works fine.
Pathfinder Wrath of the Righteous, 2021, Heroic: a few hours, works fine.
Assassin's creed Odyssey, 2018, Steam: a few hours, works fine.
Assassin's Creed Shadows, 2025, Steam: base game, played upon release, works fine.
Horizon Zero Dawn, 2017, Steam: full game, works fine.
The Outer Worlds 2, 2025, Steam: full game, played upon release, works fine.
Horizon forbidden west, 2022, Steam: a few hours, works fine.
Spellforce 3 versus, 2017, Steam: a few hours, works fine.

My shell config

2025-09-14T00:00:00+02:00

I discovered a lot of tools this year, most of them I now rely on daily. So I thought it would be a nice time to share them with various custom configs I wrote.

TL;DR: install zoxide, direnv, starship and atuin and have this in your .zshrc (or .bashrc) and enjoy a better Shell experience:

eval "$(zoxide init zsh)"
eval "$(direnv hook zsh)"
eval "$(starship init zsh)"
# Beware, this doesn’t work with Bash by default.
# See https://docs.atuin.sh/guide/installation/#installing-the-shell-plugin
eval "$(atuin init zsh)"

All my configs are tracked with git to ease sharing (mostly) and track changes.

First, let’s talk about the shell. I switch to Fish in 2021 only to come back to ZSH (with oh-my-zsh) in 2024 because of work (long story short using Bash or ZSH like all my colleges makes things way easier for me). Having ZSH at home too is easier since Fish and ZSH differ in some important ways in how to write functions, conditions or loops. The gain of Fish aren’t big enough for me to use Fish only at home or to maintain work things in Fish. The fact that I don’t do advanced things in shell often and that most of my scripts remain basic and must be Bash compatible also has a play here: most of the time I don’t need the niceties of Fish. Furthermore, with the zsh-autosuggestions and zsh-syntax-highlighting plugins, I can have the two most useful features of Fish in my ZSH shell. So, definitely no pressure to use Fish.

This brings me to the most useful of the tools I want to present: starship. It allows you you configure your prompt (to display relevant data about git repo, Pythhon, Node, Rust…) directly in your shell. It’s compatible with Bash, ZSH, Fish, Elvish and more! So no matter what you are using, you can have a shell that looks the same with a very nice prompt. It has lots of configuration options, I personally only changed the colors to make it behave better on light themes.

Another very important tool to me I used for years is direnv. It can load/unload environment variables based on the directory you are in. It can also run command once the step into the directory of a project. For instance, it can automatically enable a Python virtual environment. This way, you are sure to always use the proper version of Python no matter where you are! It’s specially useful when you work on different projects.

A most recent find is atuin. It makes accessing and search your command history way easier. I find it hard to revert to "normal" history now. It even allows you to sync history between machines either by using their service or by self hosting the software. I personally don’t use this feature: commands between my work and personal projects don’t overlap enough besides simple commands to make it useful. And you have to trust the service with commands that potentially contains secrets (they say everything is encrypted, but you still need to trust them on that).

Other tools I like and use a lot (I note that most of the new ones are written in Rust and some are in Go):

git: you surely know and use it too, but I have lots of aliases you might find useful: nice shortcuts, to easily amend a commit, to easily do a fixup commit… And many more!
- I also use git-delta to have nicer output with syntax highlighting. A must have if you ask me. I have light themes as part of my configs too. difftastic seems like a nice alternative, but will change the output to make it more meaningful and I prefer to have the raw one.
- I heard of mergiraf to help solve merge conflicts, but haven’t tested it. Looks cool though!
ripgrep as a replacement of grep. It ignores files in .gitignore by default and has overall better default behaviors.
most as a replacement of less: it has better scrolling and supports binary files.
eza as a modern replacement of ls. I now even have an alias to use eza instead of ls if eza is installed!
just as a replacement of make without its weirdness, better Windows support and more reasonable defaults.
fd as a replacement of find.
zoxide to navigate more easily in directories. Its main feature is to enable easy nested navigation. Let’s say you have ~/Projects/org/my-project, do z ~/Projects/org/my-project once and then you can do z my-project where ever you are and move to Projects/org/my-project automatically. No need to remember the full path of start navigation from home! Note: I still use cd, maybe just because of old habits.
dust as a nice replacement of good old du.
micro as a nicer CLI editor than nano or vim.
httpie to have output coloring and formatting when I do HTTP requests in CLI.
tokei to have nice stats about code (number of files, lines…).

Other articles about shell configurations that you may find useful:

Please share your tools and configs in the comments!

À propos de mon blog

2025-09-05T00:00:00+02:00

Après plus de dix ans, voici mon 200e article de blog ! J’ai décidé que ce serait un bon moment pour faire un point sur ces années et envisager le futur.

J’ai démarré ce blogue en 2013 alors que j’étais encore étudiant à Centrale Marseille. Il était alors fait en Drupal et hébergé sur les serveurs de l’école.

J’ai assez rapidement décidé d’en faire un site statique pour avoir quelque chose de plus simple et de plus facilement hébergeable et de le migrer sur mon domaine pour être plus indépendant (ma remise de diplôme approchait). Comme je faisais déjà pas mal de Python, j’ai choisi Pelican un générateur écrit en Python. De mémoire, j’avais aussi regardé Jekyll une autre solution très populaire à l’époque écrite en Ruby. Aujourd’hui, j’étudierai aussi Hugo une solution écrite en Go qui me semble de plus en plus populaire.

Pour ce blogue, je compte rester sur Pelican : ça fonctionne très bien, c’est suffisant pour mes besoins et comme j’ai peu d’articles le site est généré en moins d’une seconde. Je n’ai donc pas besoin des performances d’Hugo. En somme, aucune bonne raison de changer. Changer serait même difficile : mes articles sont écrits en reStructuredText et pas en Markdown (plus d’info à ce sujet ci-dessous) ce qui fonctionne avec Python moins dans les autres langages.

Le blogue a assez peu changé : je continue à bloguer sur les sujets tech qui m’intéressent. Certaines catégories n’ont pas eu d’articles récemment n’en auront surement jamais : elles correspondent à mes sujets d’intérêts quand j’ai lancé le blog. Seule la programmation est vraiment restée. L’expérience aidant, j’ai aussi l’impression d’avoir moins de sujets qui méritent un article. De ce point de vue, 2025 s’annonce comme une année exceptionnelle avec beaucoup de sujets.

Mon blogue reste assez peu connu. C’est principalement ma faute, je fais peu de pub, car ça ne m’intéresse pas. Il faut croire qu’écrire pour moi-même me suffit et que je préfère clarifier mes idées au fait d’être lu. Au début, je postais les liens sur Twitter pour avoir un peu de visibilité. Désormais, j’ai quitté Twitter où je n’étais de toute façon pas actif. J’ai un compte Mastodon, plus pour suivre des choses que pour poster. Et peu de personnes me suivent.

J’ai toujours des statistiques anonymes et compatibles RGPD sans bannière de cookies. Au début, j’autohébergeais une instance Matomo. Pour me simplifier la vie et gagner du temps, je suis passé sur plausible cette année : c’est simple, suffisant pour mon usage, RGPD compatible par défaut et pas très cher (9 € par mois, ce que je peux me permettre maintenant). Mon article le plus vu d’année en année reste un article publié en 2013 sur la création d’une page de garde en LaTeX avec le double de vues des autres articles. Au coude à coude viennent ensuite un article sur l’utilisation de trap en Bash et sur un sur les espaces de nom dans Docker. Pour les articles publiés cette année, celui sur uv et ruff et celui sur Pydantic et les enums fonctionnent très bien.

La langue d’écriture a aussi changé. Initialement, j’écrivais tout en français, puis j’ai proposé des traductions de mes articles en anglais pour accroitre leur visibilité. Comme la traduction demandait beaucoup de temps, maintenant j’écris en anglais, sauf pour de rares articles spéciaux comme celui-ci.

L’hébergement a changé également. Je suis passé d’un site Drupal, à un site statique hébergé sur mon VPS à un site statique dans des buckets (chez Scaleway pour éviter les géants américains). C’est encore plus simple (plus de serveur), et fonctionne très bien. C’est aussi moins cher. J’ai configuré la CSP via le HTML, car je n’ai pas l’impression qu’on puisse configurer ces entêtes dans les buckets. Je trouve que c’est moins bon, mais ça fonctionne. Je garde un VPS pour d’autres projets qui ont besoin d’un serveur, dont mes commentaires qui restent autohébergés grace à isso.

Concernant le format des sources, je suis toujours sur reStructuredText. Pelican fonctionne avec reSTtructuredText (via docutils l’implémentation de référence pour ce format) et Markdown par défaut. À titre personnel, je n’ai jamais accroché au Markdown : je le trouve pas assez spécifié et il a trop de variantes pas forcément compatibles entre elles. Avec reStrucuturedText, je peux inclure des fichiers, avoir les numéros de lignes à côté du code… J’ai donc plus de fonctionnalités qui fonctionnent correctement et sont correctement standardisées. Cela me convient mieux, même si je dois reconnaitre que le format manque de flexibilité. C’est, je pense, la force de Mardown : les cas simples sont gérés correctement par tout le monde et pas de prise de tête avec l’indentation, on écrit du texte un peu n’importe comment et ça fonctionne. Je doute qu’un langage à balisage sans cette caractéristique puisse devenir populaire, même si parser les cas plus complexes ou avoir toutes les fonctionnalités dont vous avez besoin est plus difficile.

Et l’IA dans tout ça ? Je n’ai pas vraiment d’avis. Comme la visibilité n’a que peu d’impact, je vais continuer à bloguer même si moins de personnes visitent mon blog. Si ça devient vraiment un souci et que je ne veux pas nourrir les LLM, je pense que je continuerais à écrire, pour moi sans rien publier. Ce serait dommage, car je sais que certains de mes articles sont utiles. Affaire à suivre…

Pour finir : je suis très content de ce blogue. Il me force à mettre mes idées au clair, me permet de partager certaines trouvailles, et, je pense, qu’il a fait de moi un meilleur développeur. Je compte continuer à bloguer quand j’ai envie sur les sujets qui m’intéressent. Je ne pense pas changer quoique ce soit sur la partie technique : ça fonctionne et j’en suis satisfait. Le fait que le blogue soit statique est un gros avantage, je pense, et je ne peux que vous recommander de faire un site statique vous aussi, si vous voulez un blogue : tout est plus simple, on peut utiliser git pour gérer les articles et il y a pléthore de bons outils pour vous simplifier la tâche.

Note

Dans un bucket, on ne peut pas définir des entêtes HTTP personnalisés dont X-Frame-Options pour empêcher l’inclusion du site dans une iframe. Utiliser <meta http-equiv="X-Frame-Options" content="DENY"> n’a pas d’effet. De même, la directive frame-src de Content-Security-Policy n’a pas d’effet au sein d’un élément meta (contrairement aux autres directives). Voir X-Frame-Options header et Content-Security-Policy: frame-src directive.

Tant que ces entêtes ne pourront pas être utilisées, j’ai décidé d’utiliser ce code javascript pour afficher une page vide quand le site est inclus dans une iframe (pour les utilisateurs qui ont javascript activé, ce qui devrait être la plupart des utilisateurs de nos jours) :

if (window.parent !== window) {
    window.location.replace('about:blank');
}

About my blog

2025-09-05T00:00:00+02:00

After more than ten years, this is my 200th blog post! Well technically the French version is the 200th post and this one the 201st because of how Pelican counts them. I decided it would be a good time to take a look back and one ahead.

I started this blog in 2013 while I was still a student at Centrale Marseille, a French engineering school. At the time, it was built with Drupal and hosted on the school's servers.

I quickly decided to turn it into a static site to make it simpler and easier to host, and to migrate it to my own domain to be more independent (my graduation was approaching). Since I was already doing a lot of Python, I chose Pelican, a generator written in Python. From what I recall, I also looked at Jekyll, another very popular solution at the time written in Ruby. Today, I will also look at Hugo, a solution written in Go that seems to be growing a lot in popularity lately.

For this blog, I plan to stick with Pelican: it works very well, it's sufficient for my needs, and since I have few articles, the site is generated in less than a second. So I don't need Hugo's performance. In short, there's no good reason to change. Changing would even be difficult: my articles are written in reStructuredText and not in Markdown (more info on this below), which works with Python but less so with other markup languages.

The blog hasn't changed much: I continue to blog about tech topics that interest me. Some categories haven't had any recent posts and probably never will: they correspond to my interests when I started the blog. Only programming has really remained. With experience, I also feel like I have fewer topics that deserve an article. From this point of view, 2025 looks like it will be an exceptional year with lots of new articles.

My blog remains relatively unknown. This is mainly my fault, as I don't do much advertising because I'm not interested in it. I guess writing for myself is enough for me, and I prefer to clarify my ideas rather than be read. In the beginning, I posted links on Twitter to get a little visibility. Now I've left Twitter, where I wasn't active anyway. I have a Mastodon account, more to follow things than to post. And very few people follow me.

I still have anonymous, GDPR-compliant statistics without cookie banners. At first, I self-hosted a Matomo instance. To simplify my life and save time, I switched to Plausible this year: it's simple, sufficient for my needs, GDPR-compliant by default, and not very expensive (€9 per month, which I can afford now). My most viewed article year after year remains an article published in 2013 on creating a cover page in LaTeX with twice as many views as the next article. With almost the same number of views are an article on using trap in Bash and one on namespaces in Docker. Of the articles published this year, the one on uv and ruff and the one on Pydantic and enums are working very well.

The writing language also changed. Initially, I wrote everything in French, then I offered translations of my articles into English to increase their visibility. Since translating was very time-consuming, I now write in English, except for rare special articles like this one.

The hosting has also changed. I switched from a Drupal site to a static site hosted on my VPS to a static site stored in buckets (at Scaleway to avoid the American giants). It's even simpler (no more server) and works very well. It's also cheaper. I configured CSP via HTML, because I don't think you can configure these headers in buckets. I find it less good, but it works. I keep a VPS for other projects that need a server, including my comments, which remain self-hosted thanks to isso.

As for the source format, I'm still using reStructuredText. Pelican works with reSTtructuredText (via docutils, the reference implementation for this format) and Markdown by default. Personally, I've never been a fan of Markdown: I find it insufficiently specified and it has too many variants that aren't always compatible with each other. With reStructuredText, I can include files, have line numbers next to the code, etc. So I have more features that work properly and are properly standardized. It suits me better, even though I have to admit that the format lacks flexibility. I think that's the strength of Markdown: simple cases are handled correctly by everyone, and there's no hassle with indentation; you can write text any way you like and it works. I doubt that a markup language without this feature could ever become popular, even if parsing more complex cases or having all the features for your needs are harder.

And what about AI in all this? I don't really have an opinion. Since visibility has little impact, I'll continue blogging even if fewer people visit my blog. If it really becomes a concern and don’t want to feed the LLM, I think I'll continue writing, for myself, without publishing anything. That would be a shame, because I know some of my articles are useful. To be continued I guess…

To sum up: I'm very happy with this blog. It forces me to clarify my ideas, allows me to share some of my discoveries, and, I think, has made me a better developer. I plan to continue blogging when I feel like it on topics that interest me. I don't plan to change anything on the technical side: it works and I'm happy with it. The fact that the blog is static is a big advantage, I think, and I can only recommend that you also make a static site if you want a blog: everything is simpler, you can use git to manage articles, and there are plenty of good tools to make your job easier.

Note

On a bucket you cannot set the X-Frame-Options header (or any custom header) to prevent the inclusion of your website into an iframe. Using <meta http-equiv="X-Frame-Options" content="DENY"> will have no effect. Likewise the frame-src instruction of Content-Security-Policy will only have an effect if used if a meta tag unlike other directives. See X-Frame-Options header and Content-Security-Policy: frame-src directive.

Until headers can be set, I decided to use this JS code to hide the website if included in an iframe (at the condition JS is enabled for the visitor which nowadays should be most users):

if (window.parent !== window) {
    window.location.replace('about:blank');
}

My opinion on Django Ninja

2025-07-06T00:00:00+02:00

If you’re using the Django web framework (DRF) and are building APIs, you probably know about the Django Rest Framework a toolkit to build APIs in Django since more than a decade (the first 3.x version was release in 2014). It’s great and I’ve used it professionally to build several APIs. It’s also complex and showing its age when compared to what can be done in Flask or FastAPI.

Luckily, there’s a new possibility in Django’s space: django-ninja. It reached 1.0 in 2023 and was first release in 2020. It’s heavily inspired by FastAPI, relies on Pydantic for payload validation and supports async out of the box. It’s also easier to setup. That’s why I decided to use it on one of my personal projects which needed an API. After about a year, I’d like to give my opinion on it.

Please keep in mind that I only built a small API with it. I haven’t used it yet on bigger projects, so I can’t say much about this use case. I think it’s ready and mature enough and would definitely try it over good old DRF, but you may reach for someone else’s opinion if you plan to use it on a big project.

Overall, I’m pleased and very impressed by django-ninja. The biggest selling point for me is Pydantic. This library is awesome and used more and more. I already use it extensively at work. I also use it in my project to validate the input of JSON data stored in the database and to parse some JSON files. Being able to also use it for my API is a big plus for me: use cases are very similar and having only one library to learn is a breeze.

Regarding the creation of the API itself, it’s a bunch of decorated and type annotated functions. django-ninja takes care of the rest like validating POST payload or mapping GET parameters to function parameters. It even generate the documentation automatically. On that regard, it’s indeed very close to how FastAPI works. Simple and yet efficient. It seems to lack the ability to use class based views, which could be detrimental to grouping some functions.

I also like the fact that to change the HTTP response code, all you have to do is return a tuple of status_code, response like in Flask.

I’m a bit surprised it doesn’t handle more things regarding authentication. It supports Django’s auth out of the box, but it’s based on cookies. So it’s only useful to debug the API in your browser. For an API key, HTTP Bearer or HTTP Basic auth, it gives you a base class you can subclass to implement your custom method. I choose to use a JWT token passed in a header. I guess it’s not more integrated because there are so many options available and possible links with the user model that what you’d need would be there anyway.

While I’m very pleased with what django-ninja has to offer, I encountered a few issues:

I have Content Security Policy (CSP) with nonce enabled on my site thanks to django-csp. Because of this, the JS and CSS loaded by the docs couldn’t be loaded because no nonce was applied in the doc templates. I solved it by copying the swagger.html template and adding nonce="{{ request.csp_nonce }}" were the static files are loaded.
When not authenticated and targeting the API with curl, I got some CSRF error. Making the HttpBearer auth the default (ie before django_auth in the list of auth providers) solved the problem.
On some endpoints, I allow the PATCH HTTP method to be used to update only the supplied fields. Initially, I used PatchDict provided by django-ninja. Sadly, given how it’s implemented, it allowed some fields to be set to None in the payload without validation error when these fields are not nullable. This caused errors at the database level when the API tried to save these models. What I ended up doing to solve this is:
1. Create a custom type named NotSet integrated in Pydantic thanks to the __get_pydantic_core_schema__ class method. It takes the underlying type as an argument to display beautiful default values in the documentation. You can view its implementation here.
2. Create a custom update function that skips values set to NotSet when updating the Django model. You can view its implementation here.
3. Use the type like this read_at: datetime | SkipJsonSchema[NotSet] | None = NotSet(datetime.now) in my Ninja schemas. The SkipJsonSchema prevents the type to be mentioned in the documentation since it’s a technical type users can’t use directly.
It’s not as easy as I’d hoped, but it works and I think it’s an acceptable work around, at least for my needs. There’s probably a way to make this more generic and reusable on existing schemas, but I don’t need that right now.

To conclude, I’m glad this solution exists and I think I have a cleaner, more modern and more simple way to create APIs than DRF. I had some issues and spent time in the documentation at first, but nothing weird since it’s a new library for me. From what I’ve seen and after my usage, it seems like a powerful, feature complete solution to build API with Django. I hope I’ll be able to use on a bigger project!

As usual, I’d be delighted to hear your opinion in the comments below.

Some tips on observables

2025-06-29T00:00:00+02:00

Table of contents

Subscribing to the same observable twice
Various behaviors

You can download the Jupiter notebook or download the script.

Subscribing to the same observable twice

The source will be consumed twice. To prevent that, a Subject must be used instead.

const sourceMultipleSubs = from([1, 2]);

sourceMultipleSubs.subscribe(value => console.log("Subscription 1", value));
sourceMultipleSubs.subscribe(value => console.log("Subscription 2", value));

Various behaviors

const source$ = from([1, 2, 3, 1, 4]);
// Will emit the values of source$ each 3,9 seconds. The 1st value is emitted immediatly.
const delayedValues = zip(source$, timer(0, 3_900), (value, time) => value);

Behavior of `from(Promise)` and `combineLatest`

The promise merged with from will never be called again when used in combineLatest. It will only ever use its first value.

const someHttpRequest = () => {
    console.log('calling the function');
    return Promise.resolve(null);
}

const sub = combineLatest(interval(2_000).pipe(take(10)), from(someHttpRequest())).subscribe(console.log)

Behavior of `switchMap`

When multiple switchMap are chained, the observable created by the latest switchMap will continue to emit values until we pass again in this switch map.

const sub = delayedValues.pipe(
  tap((value) => console.log('Source value', value)),
  switchMap(value => interval(1000).pipe(
      filter(value => value <= 2),
      take(1),
      map(() => value * 10),
      tap(value => console.log('1st inner observable emmiting value:', value)),
      finalize(() => console.log('Ending 1st inner observable created from:', value)),
    )
  ),
  switchMap(value => interval(500).pipe(
      take(10),
      map(() => value * 10),
      tap(value => console.log('2nd inner observable emmiting value:', value)),
      finalize(() => console.log('Ending 2nd inner observable created from:', value)),
    )
  )
).subscribe(value => console.log('Value in subscription:', value), console.error, () => console.log('Ending subscription'));

Behavior of filter

Filter won’t trigger the rest of the pipeline until it accepts a value. This also means observable created by switchMap will continue to execute until a new value passes the filter.

const sub = delayedValues.pipe(
  tap((value) => console.log('Source value', value)),
  filter(value => value <= 2),
  switchMap(value => interval(1000).pipe(
      take(5),
      map(() => value * 10),
      finalize(() => console.log('Ending inner observable created from:', value))
    )
  ),
).subscribe(value => console.log('Value in subscription:', value), console.error, () => console.log('Ending subscription'));

Using `EMPTY` in a `switchMap`

The observable will complete as soon as the switch to empty is done. Must use of({}) instead to only cancel a previous switchMap.

const obs1 = from([1, 2])
    .pipe(switchMap(() => EMPTY))
    .subscribe(
        () => console.log('Next with EMPTY'),
        () => {},
        () => console.log('Complete EMPTY')
    );


const obs2 = from([1, 2])
    .pipe(switchMap(() => of({})))
    .subscribe(
        () => console.log('Next of({})'),
        () => {},
        () => console.log('Complete of({})')
    );

Behavior of `switchMap` with Promise

Promise can’t be canceled. When reaching a switchMap again, RxJS will just start the new promise. This means, many Promises can execute in parallel. The rest of the pipe will only be executed for promise that succeeds before switchMap is called again. So, if switchMap is called a second time while the initial is still running, only the second call will execute the full pipe.

With mergeMap, the rest of the pipe will be executed for each promise that resolves.

With contactMap, we will execute concatMap in order and wait for the previous one to complete fully (ie the full pipe that follows) before starting the next one. The rest of the pipe is executed as expected. This is the same behavior than with observables.

const sub = delayedValues.pipe(
  tap((value) => console.log('Source value', value)),
  switchMap(value => {
      console.log("Entering switchMap", value);
      return new Promise(resolve => setTimeout(() => {
          console.log("Resolving promise", value);
          resolve(value * 10);
      }, 10000));
  }),
  tap(value => console.log("Value in pipe", value)),
).subscribe(value => console.log('Value in subscription:', value), console.error, () => console.log('Ending subscription'));

const sub = delayedValues.pipe(
  tap((value) => console.log('Source value', value)),
  mergeMap(value => {
      console.log("Entering mergeMap", value);
      return new Promise(resolve => setTimeout(() => {
          console.log("Resolving promise", value);
          resolve(value * 10);
      }, 10000));
  }),
  tap(value => console.log("Value in pipe", value)),
).subscribe(value => console.log('Value in subscription:', value), console.error, () => console.log('Ending subscription'));

const sub = delayedValues.pipe(
  tap((value) => console.log('Source value', value)),
  concatMap(value => {
      console.log("Entering concatMap", value);
      return new Promise(resolve => setTimeout(() => {
          console.log("Resolving promise", value);
          resolve(value * 10);
      }, 10000));
  }),
  tap(value => console.log("Value in pipe", value)),
).subscribe(value => console.log('Value in subscription:', value), console.error, () => console.log('Ending subscription'));

If the Promise is resolved immediately, the observable will emit once with the last value from the promise and then end. So the chain can end prematurely if a source should continue to emit and Promise is used in the chain.

mergeMap and concatMap will execute the chain for all emitted values from the source.

const obs = from([1, 2])
    .pipe(switchMap(val => from(Promise.resolve(val * 10))))
    .subscribe(
        val => console.log('Next:', val),
        () => {},
        val => console.log('Completed with:', val)
    );

const obs = from([1, 2])
    .pipe(mergeMap(val => from(Promise.resolve(val * 10))))
    .subscribe(
        val => console.log('Next:', val),
        () => {},
        val => console.log('Completed with:', val)
    );

const obs = from([1, 2])
    .pipe(concatMap(val => from(Promise.resolve(val * 10))))
    .subscribe(
        val => console.log('Next:', val),
        () => {},
        val => console.log('Completed with:', val)
    );

Behavior of `catchError` and `finalize`

No matter where the finalize is, it will run! It will run after the error callback of the subscription. If there’s no error, it will run before the next callback of the subscription.
If we have multiple finalize, they will run in the order of definition, as part of the pipe.
Having a catchError prevents the error callback of the subscription to be run.
- It’s like a catch on a Promise chain: we revert to a normal flow. Using throwError cancels that flow.
- It will only catch errors that occurs before its definition. It has no impact on errors that occurs after. So it must be placed as far in the pipe as possible to catch as many errors as possible.

const obs = source$.pipe(
    map(value => {
        if (value <= 2) {
            return value
        }

        throw new Error("oops")
    }),
    tap(value => console.log('Tapping:', value)),
    finalize(() => console.log('running finalize')),
).subscribe()

const obs = source$.pipe(
    finalize(() => console.log('running finalize')),
    map(value => {
        if (value <= 2) {
            return value
        }

        throw new Error("oops")
    }),
    catchError(() => {
        console.log('An error occurred');
        return of("Kaboom");
    }),
    tap(value => console.log('Tapping:', value)),
).subscribe(console.log, v => console.warn('On error', v))

const obs = source$.pipe(
    finalize(() => console.log('running finalize')),
    map(value => {
        if (value <= 2) {
            return value
        }

        throw new Error("oops")
    }),
    catchError(() => {
        console.log('An error occurred');
        return throwError(() => new Error("Kaboom"));
    }),
    tap(value => console.log('Tapping:', value)),
).subscribe(console.log, v => console.warn('On error', v))

const obs = source$.pipe(
    finalize(() => console.log('running finalize 1')),
    catchError(() => {
        console.log('An error occurred');
        return of("Kaboom");
    }),
    map(value => {
        if (value <= 2) {
            return value
        }

        throw new Error("oops")
    }),
    tap(value => console.log('Tapping:', value)),
    finalize(() => console.log('running finalize 2')),
).subscribe(console.log, v => console.warn('On error', v))

const obs = source$.pipe(
    finalize(() => console.log('running finalize 1')),
    map(value => {
        if (value <= 2) {
            return value
        }

        throw new Error("oops")
    }),
    catchError(() => {
        console.log('An error occurred');
        return of("Kaboom");
    }),
    tap(value => console.log('Tapping:', value)),
    finalize(() => console.log('running finalize 2')),
).subscribe(console.log, v => console.warn('On error', v))

Switching to an observable that uses switchMap internally

The inner observable (and thus its inner switchMap) is correctly unsubscribed from and thus does nothing when it must do nothing.

const buildInnerObs = startingValue => delayedValues.pipe(
    map(value => startingValue * 10),
    tap(value => console.log('Inner obs:', value)),
    switchMap(value => interval(1000).pipe(
        take(10),
        map(() => value * 10),
        finalize(() => console.log('Ending inner interval')))
    ),
    tap(value => console.log('Inner obs switch map:', value)),
    finalize(() => console.log('Ending inner switchMap')),
);

const sub = delayedValues.pipe(
    tap(value => console.log('Starting with:', value)),
    switchMap(value => value <= 2 ? buildInnerObs(value) : of({})),
).subscribe(value => console.log('In subscription:', value));

`combineLatest`

It will wait for all observable to emit their first values before emitting.

const sub = combineLatest([delayedValues, source$.pipe(delay(1000))]).subscribe(console.log);

My opinions on HTMX

2025-06-24T00:00:00+02:00

HTMX is a JavaScript library designed to add interactivity to your web pages. With it, you don’t build a single page application, you use standard templates (written in Django, Jinja) rendered in your backend, add attributes to your HTML elements and let HTMX add the appropriate interactivity. From what I read, it’s getting lots of popularity in the Django world. That’s why I heard of it in the first place and one of the reason I choose it.

There are alternatives like Alpine.js or Turbo. After looking at their documentation and examples, I had the feeling HTMX is better suited for what I was building. HTMX also seems more popular and integration in Django is made easy thanks to django-htmx. It’s also quite small (at least compared to the cost of a SPA): only 17.5kb once minified and gzipped. It should be almost all the JS you need when you build something with it: if you need way more, switching to a SPA could make more sense. Likewise, if you need to build an app or handle complex state, a true SPA might be easier to maintain in the long run.

I think the library is powerful and relatively easy to use: all you have to do is include it and add the proper attributes. It also has lots of examples to help you get started. Pretty much everything I needed to use already had an example associated with it.

In my case, I use it in an article reading application: the backend fetches RSS feeds and displays the fetched articles on a page. The articles are grouped into various reading lists. To improve UX, when a user clicks on the "Mark as read" button, I don’t want a full page reload. That’s where HTMX comes in!

With HTMX, I can create a form and submit it to mark the article as read without this page reload. To do this, adding hx-boost and hx-swap to the form is enough to make HTMX handle the submission and handle the HTML from the response. A nice feature here is that you can update multiple part of the page in one go: the element associated with the update and something else thanks to a process named Out of Band Content swapping (oob for short): instead of just returning the element you need to swap, you can return multiple and HTMX will swap them all. In my case: the article card and the unread count are updated at the same time thanks to this process.

Two features required a bit more work to make them work correctly.

The first one is a feature, which when enabled, allows to read articles on scroll to easily scan big chunks of articles and only open those that interest you. This required some custom JS to correctly detect scroll end, wait a few seconds without interactions to trigger the action (to make sure the user won’t scroll up), find the articles that are not visible anymore and mark them all as read. It required some digging into the documentation to be able to hook on htmx and use its JavaScript API correctly. In the end, it worked really well!

The second one, is when users try to delete an article: I wanted to display a pretty modal and not a basic JS alert. Part of the problem comes from the fact that I have multiple possible action buttons and only for some I want the confirm modal to be triggered. This isn’t supported out of the box.

What I ended up doing was have multiple forms and use the form attribute of buttons to select which form the buttons must submit. This way, I can have a delete form with hx-confirm (use to display an alert or run a custom action on a click) and only open the modal in that case. As far as I can remember, putting hx-confirm on the button directly didn’t work (I don’t remember the exact behavior in that case). To have this work in all cases, I had to setup a custom event listeners on htmx:confirm to capture the event and run some custom processing code. I also had to make sure the name of the button was correctly transmitted in the request which wasn’t done automatically by HTMX in that case for some reason. The full code is available here if you need more details.

To write the proper solution, I had to debug HTMX. It’s a pain as in any library you don’t know but not a very big deal. In Django debug mode, I load the unminified version of the library and from there, I was able to add breakpoints from the browser tools and see what was happening. I had to do it another time when I tried an HTML minification library which broke HTMX completely. It turns out, it removed type="submit" attributes since it’s the default, but HTMX requires them to boost forms.

The library is great and I’m very pleased by it. There are however, as with any tech, some downsides:

HTMX behaviors can only be tested with end to end tests. Not much of an issue for me: they don’t change much and can be tested manually. It can be a problem on a bigger app, but I think it’s also hard to mess up something the smallest amount of manual testing would catch: you are only adding HTML attributes after all, the rest is handled by HTMX itself.
I think you must enable CSP (Content Security Policy) to make sure no HTML coming from an untrusted source is used by HTMX to swap elements and thus execute JavaScript. You should to it anyway, HTMX or not.
If you only enhance links and forms with hx-boost, your application will work without JS since browser have a fallback. If you do other things, JS becomes required since without it some behaviors will break. Not a very big deal in my opinion since all your users should have JS enabled. In my case, some menus managed by Bootstrap won’t open without JS, making the navigation in the app a pain anyway.
Since these attributes are specific to HTMX, I guess there is a form of vendor lock-in. But the attributes are easy to spot and since the alternatives also rely on attributes, I guess a search/replace would do most of the migration. It’s still way easier than changing SPA frameworks since it does way less!
The documentation promotes behaviors that are not accessible by default. To quote it:

Why should only <a> & <form> be able to make HTTP requests?

Why should only click & submit events trigger them?

Why should only GET & POST methods be available?

Why should you only be able to replace the entire screen?

I guess it’s up to you to make sure that what you do is accessible, just like with a SPA and use the proper attributes for accessibility. The examples could also promote accessibility more. Some already do, without explaining it enough in my taste, but most don’t. I think that accessibility is in such a bad place, that having all the official examples promote best practices would be a huge help.

According to this hacker news thread, the maintainers are aware of this and may improve the situation. There are also one issue about the accessibility implication of HTMX, one about improving the examples and there seems to have plans to use the new moveBefore API to improve the situation (this API is only available in Chrome and only since the beginning of the year).

To be fully honest, I don’t think my project is fully accessible. As a backend developer, I don’t know enough to see the issue (and I’m probably far from the only one if I trust my experience and the reports regarding the poor state of web accessibility). That’s also why, I rely as much as possible on forms with progressive enhancement and use Bootstrap which have an accessibility team. No matter the lib, most examples I see don’t even mention it. That’s not only an HTMX problem.

In conclusion, HTMX adds some complexity compared to basic pages, but it remains manageable. I’m happy with the lib and the fact it allows me to write most of my logic in Python and Django with standard HTML and almost no custom JavaScript. Now that the heavy lifting is done, it’s been months since I even had to touch HTMX features. Almost all my logic lives in the backend which feels easier and make me more productive: no language or context juggling. Plus, all the behaviors are defined in the HTML with the rest of the attributes. Depending on what you are building, it can be a good and more simple alternative to a full fledged SPA. It also gives me some memories of when I was a student and was building websites without a JS framework: use of plain JS files to extend your site’s features. It was a mess back then, even with the help of jQuery. Now, it feels clean and easy!

Simply how licenses are applied to your project with SPDX and reuse

2025-06-22T00:00:00+02:00

SPDX (Software Package Data Exchange) is a project managed by the Linux foundation created to standardize how licenses are identified in a human and machine readable way. In a nutshell, instead of adding a big header to your files to identify the applicable license, you apply a copyright text and a license identifier. For a Python file under the GPL, instead of this:

{{ project }}
Copyright (C) {{ year }}  {{ organization }}

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

It would give this:

# SPDX-FileCopyrightText: {{ year }} {{ organization }}
#
# SPDX-License-Identifier: GPL-3.0-or-later

It’s shorter and more readable (at least in my opinion). And a script can easily read this header and compile a list of all used licenses and to each file they apply.

You can go to the list of licenses to find the identifier and text of the licenses you want to use. Apparently, these identifiers have been around for several years (at least 2021) and are already used by big projects like KDE, Qt or Fedora. I personally only learned about them earlier this year and dug into the subject this month.

So far, I thought is was interesting, but wouldn’t impact my projects too much: I’d only have to switch one header for another. That’s until I learned about the REUSE tool by the FSFE in Fedora magazine. The goal of this tool is to help you choose licenses, download their full text and add the proper header to your files. For files that cannot have a license header, it can create a <FILE_NAME>.license file or define the license in a REUSE.toml file. Once this is done, it can enforce that all your files have a license.

It’s easy to setup (including in pre-commit hooks) and comes with a nice tutorial and FAQ to help you get started. The catch is that it can be a bit time consuming to enforce at first to find the proper licenses to apply to each file. Luckily, it can add headers to multiple files at the same time. This tool is used by KDE, Linux, Nextcloud and Curl!

In my case, I used it in one of my GNU AGLP licensed project. It contained many files of different types, some with a proper AGPL header, some without. And others were copy/pasted from the internet and thus the AGPL with my name in the copyright didn’t apply. So I had to review for each file what to do and apply the header with the proper copyright text in bulk of files. At least, now it’s done and files I don’t have the copyright on but I can use in my project (because they are published on a free software license like MIT) are clearly identified.

I wandered whether I should use only the SPDX-FileCopyrightText and SPDX-License-Identifier in the header or keep the full GNU AGPL copyright statement. According to the GNU AGLP full text, this header is only a strong suggestion to help readers identify the license of the file. So nothing mandatory on that side and I think the SPDX-License-Identifier fills that role too even if it’s less clear since it doesn’t give any explanation nor explicitly state the absence of warranty. After looking a bit in the Qt, KDE and REUSE repositories, which all use some variants of the GNU GLP licenses, I saw only the SPDX identifier and not the classical headers, I guess it’s fine. It’s also much better than most programs under non GLP licenses which tend to have no header at all! I also searched the internet for this but couldn’t find anything useful. If you have more information and this, please let me know in the comments below.

Some interesting articles I found on this subject:

SPDX Tutorial
Making sense of software licensing with FSFE REUSE: A beginner’s guide for open source developers
Understanding and Using SPDX License Identifiers and License Expressions
SPDX License Expressions in my opinion it has the best explanation of the AND operator.

Using super to reuse method with inheritance in Python

2025-06-09T00:00:00+02:00

If you use Python, you probably already know of super to call a method from the base class. You probably also know you can use a function as a method if it takes the instance as 1st parameter. What you cannot do, is call super().my_method() directly in such a function. To do that, you must use the parameters of super to call it just like it was done in Python 2, so with super(type(instance), instance).

To give more details (and testable code) to clarify what’s happening, please read below.

Let’s start by defining our base class and its methods:

class BaseClass:
    def __init__(self):
        self.base_attr = "Test"

    def base_method(self, a_parameter: int) -> int:
        print("Base class base method")
        return a_parameter

    def shared_method(self, a_parameter: int) -> int:
        raise NotImplementedError

base_method we will want to call, a base_attr to use and the shared_method that must be implemented in the sub-classes. That’s shared_method that will be defined with a function in our child classes.

Let’s try with a basic usage, where we only use a function as a method:

def _shared_no_super(instance: BaseClass, a_parameter: int) -> int:
    value = instance.base_method(a_parameter)
    print("Inside shared method no super", value, instance.base_attr)
    return value

class A(BaseClass):
    def base_method(self, a_parameter: int) -> int:
        print("Inside A base method")
        return super().base_method(a_parameter) + 1

    shared_method_no_super = _shared_no_super

a = A()
a.shared_method_no_super(0)

As expected, a.shared_method_no_super(0) will call the code in _shared_no_super and will call the method on our a instance and then on the base method. The output will be:

Inside A base method
Base class base method
Inside shared method no super 1 Test

Now let’s try to use super in the function to bypass the method on the class and go directly to the base class. The naive approach is to use super() directly like in the class. Let’s try that:

def _invalid_shared(instance: BaseClass, a_parameter: int) -> int:
    value = super().base_method(a_parameter)
    raise Exception("This should never be printed")

class Invalid:
    def base_method(self, a_parameter: int) -> int:
        raise Exception("This should never be reached")

    shared_method = _invalid_shared

instance = Invalid()
instance.shared_method(0)

This will result in a RuntimeError:

Traceback (most recent call last):
  File "main.py", line 47, in <module>
    instance.shared_method(10)
  File "main.py", line 34, in _invalid_shared
    value = super().base_method(a_parameter)
RuntimeError: super(): __class__ cell not found

It turns out that Python 3 does some magic behind the scene to make a simple super() work and do the right thing. Since here it’s not called with a class context, this magic cannot happen. Hence the error.

If like me you used Python 2, you may remember how super had to be used: super(MyClass, instance). The goal of these parameters was to help Python find the proper class on which to start looking for the super method. This is still possible in Python 3, both for compatibility reason and (I suppose) to allow more advanced uses of super.

Let’s see that in action:

def _shared_method(instance: BaseClass, a_parameter: int) -> int:
    value = super(type(instance), instance).base_method(a_parameter)
    print("Inside shared method", value)
    return value


class B(BaseClass):
    def base_method(self, a_parameter: int) -> int:
        print("Inside B base method")
        return a_parameter * 2

    shared_method = _shared_method


b = B()
b.shared_method(1)

Now this works as expected and prints, as expected:

Base class base method
Inside shared method 1

Of course, it also works if used as a fonction directly:

_shared_method(b, 2)

Will print:

Base class base method
Inside shared method 2

Fun trick isn’t it? I don’t think you’ll need it often since most of the time code between classes will be shared in a base class (or a mixin). But I still think it’s good to know for the weird and niche cases when you need to share code between classes, use super but just cannot rely on inheritance.

Promise returned by async functions

2025-05-24T00:00:00+02:00

Recently at work, I encountered a problem with a function that should have returned a custom promise subclass. The goal of this subclass is to be able to call a cancel method to cancel some pending action done within the promise and then reject it. It looks like this:

class CancellablePromise extends Promise {
    constructor(executor) {
        super(executor)
    }

    cancel() {
        console.log("Canceling");
        // Actual operation
    }
}

Unlike what I expected, I didn’t get a CancelablePromise but a standard Promise object. I initially thought it came from the chain: I didn’t stored the return promise directly but the output of a .catch method like this:

const myPromise = buildCancellable().catch(() => console.error("Oops"));

After running some tests, it turns out that chaining from a custom promise class will return an instance of the custom promise class. So the culprit wasn’t there. I continued debugging and acquired the certainty that buildCancellable didn’t return the custom class at all in the first place. I’m also certain it used to.

It turns out the buildCancellable function was made async a few months ago. Instead of being defined like this:

const buildCancellable = () => new CancellablePromise(() => { ... });

It’s now defined like this:

const buildCancellable = async () => {
    // Stuff using await
    return new CancellablePromise(() => { ... })
};

It turns out that returned values of async function are always wrapped in a proper Promise even if they are themselves already a Promise. It has a very little impact on most usage since nested promise are automatically unwrapped, so we have this behavior:

const innerPromise = new Promise(resolve => {
  setTimeout(() => {
    console.log("Resolving inner promise");
    resolve("Inner value");
  }, 2_000);
});

const outerPromise = new Promise(resolve => {
  setTimeout(() => {
    console.log("resolving outer promise");
    resolve(innerPromise);
  }, 1_000)
});

const value = await outerPromise;

// Will log "Inner value" as expected.
console.log("The value:", value);

But it made the cancel method non reachable since the actual CancellablePromise cannot be reached now. The only solution is to remove the async keywords and remove await from the body.

I hope you found this reminder useful. I did have an of course moment once I figured it out, but was a bit lost at the start. I really expected to get the proper object back (and I still think it’s reasonable to expect Promise to not be wrapped again) since it’s already a Promise. I guess it’s best for consistency to have JS behave this way. And it’s transparent in almost all cases anyway.

Using JavaScript and TypeScript in jupyter notebook

2025-05-23T00:00:00+02:00

While wandering how to run some tests with JavaScript, I discorvered that deno, a JavaScript runtime like NodeJS, has support for jupyter notebooks thanks to this article. It works perfectly. Install deno (probably available in your distribution repositories), run deno jupyter --install and you are good to go!

To get a package from npm, prefix its name with npm: like this: import _ from 'npm:lodash'. You can then use it normally:

import _ from 'npm:lodash'
_.defaults({ 'a': 1 }, { 'a': 3, 'b': 2 });

It even supports TypeScript code out of the box:

const test: string[] = [];
const myPush = <T>(array: T[], elt: T) => {
    array.push(elt);
};
myPush(test, 12);
console.log(test);

Neat isn’t it?

Using Pydantic models within enums

2025-04-26T00:00:00+02:00

In previous articles, I provided some tips on enums and explained how to use a custom class with Pydantic. Now, I’ll dig into all kinds of enum usages with Pydantic, including enums whose members are Pydantic models themselves! Let’s dive right in!

Just like before, you can access the code of this article as a Python script and as a notebook. I won’t include the imports in the code examples for brevity, refer to the notebook or the script if needed.

Table of contents

Using normal enums
With different types in the enum
Using enums with Pydantic models
Conclusion
Bonus

Using normal enums

Let’s start small with how Pydantic behaves with normal enums.

class IntValues(IntEnum):
    first = 1
    second = 2


class StrValues(StrEnum):
    valid = "valid"
    invalid = "invalid"


class NormalEnumModel(BaseModel):
    # False is the default. If True, will use 1 or "valid"
    # in the model instead of the enum members.
    model_config = ConfigDict(use_enum_values=False)

    int_value: IntValues
    str_value: StrValues


m = NormalEnumModel.model_validate({"int_value": 1, "str_value": "valid"})
print("The model:", m)
print("Dumped in Python mode:", m.model_dump(mode="python"))
print("Dumped in JSON mode:", m.model_dump(mode="json"))

The model: int_value=<IntValues.first: 1> str_value=<StrValues.valid: 'valid'>
Dumped in Python mode: {'int_value': <IntValues.first: 1>, 'str_value': <StrValues.valid: 'valid'>}
Dumped in JSON mode: {'int_value': 1, 'str_value': 'valid'}

Each field is correctly parsed as an enum member. When dumping in JSON mode, the value is used and the member remains in Python mode. I say this behaves as expected.

Once again, as expected, values outside the enum will raise a ValidationError:

try:
    # Values outside the enum, will fail!
    NormalEnumModel.model_validate({"int_value": 10, "str_value": "nope"})
    assert False, "Must fail"
except ValidationError as e:
    print("Failing as expected with values outside the enum with:")
    print(e)

Failing as expected with values outside the enum with:
2 validation errors for NormalEnumModel
int_value
  Input should be 1 or 2 [type=enum, input_value=10, input_type=int]
    For further information visit https://errors.pydantic.dev/2.11/v/enum
str_value
  Input should be 'valid' or 'invalid' [type=enum, input_value='nope', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/enum

Let’s illustrate the behavior when forcing Pydantic to use enum values instead of enum members with use_enum_values=True:

class MyModelWithValues(BaseModel):
    model_config = ConfigDict(use_enum_values=True)

    int_value: IntValues
    str_value: StrValues


m = MyModelWithValues.model_validate({"int_value": 1, "str_value": "valid"})
print("The model:", m)
print("Dumped in Python mode:", m.model_dump(mode="python"))
print("Dumped in JSON mode:", m.model_dump(mode="json"))

The model: int_value=1 str_value='valid'
Dumped in Python mode: {'int_value': 1, 'str_value': 'valid'}
Dumped in JSON mode: {'int_value': 1, 'str_value': 'valid'}

This time, the values is used in the model and no matter how it’s dumped.

With different types in the enum

Now that the basics are covered, let’s use more complex enums. Let’s start to see what happens if we try to mix types in IntEnum and StrEnum. If Python can parse a type automatically in IntEnum it will do so:

class WithIntAndIntAsString(IntEnum):
    member1 = 1
    # This will be parse automatically by Python.
    member2 = "10"


print(
    "member1",
    type(WithIntAndIntAsString.member1.value),
    WithIntAndIntAsString.member1,
)
print(
    "member2",
    type(WithIntAndIntAsString.member2.value),
    WithIntAndIntAsString.member2,
)

member1 <class 'int'> 1
member2 <class 'int'> 10

Otherwise, it will raise a TypeError:

try:
    class MoreStringValues(StrEnum):
        member1 = "member1"
        # Python won’t cast anything to int.
        value2 = 10
except TypeError as e:
    print("Failing with error:", e)

Failing with error: 10 is not a string

If we need to mix types, we can use a base Enum:

class MultipleBasicTypes(Enum):
    member1 = "member1"
    member2 = 10


print(
    type(MultipleBasicTypes.member1.value),
    type(MultipleBasicTypes.member2.value),
)

<class 'str'> <class 'int'>

If you need to enforce a consistent type within your enums, please refer to of my previous article and the enforce_member_type decorator.

Using enums with Pydantic models

Now that we know how enums behave with Pydantic, let’s dive the heart of this article: enums with Pydantic models as members and their usage in other Pydantic models! I’ll detail several possible strategies here and discuss the pros and cons of each one. You’ll probably have questions or remarks on these (and maybe even new methods), so don’t hesitate to use the comment section.

I’ll use an example to make things less abstract. I’ll leave your imagination work to find useful use-cases. You can also just see it as a way to have fun with Python and Pydantic.

Let’s imagine that you are running a blog platform. Each article on the platform is associated with a category. A category could be represented with this Pydantic model:

class Category(BaseModel):
    # The id of the category in the database
    id: int
    # A unique human-readable code identifying the category.
    code: str
    # The title to display to the users.
    title: str

Let’s say that for editorial reasons the list of categories is fixed. So this list can fit into an enum. The ids and the codes are set within the enum to ease referencing the categories in the codebase without needing to query the database. This will help to create new categories in the database when you add some. All you’ll have to do is loop over the members of the enum.

We’ll also assume it allows great simplification in you code to be able to hard code the id of the category directly. The enum makes this clean and readable. It will also make your JSON API more simple to use for your users. To create an article, they can use the human readable code of the category instead of the id. They can send this:

{"category": "programming", "title": "Fun with Pydantic"}

to get this back:

Article(category=ArticleCategory.programming, title="Fun with Pydantic")

No need to bother them with ids!

Let’s start this journey by creating the ArticleCategory enum:

class ArticleCategory(Enum):
    cooking = Category(id=1, code="cooking", title="Cooking")
    programming = Category(id=2, code="programming", title="Programming")

You’ll then be able to mention this in you article model:

class Article(BaseModel):
    # id of the article in the database.
    # Allow None in the model to enable user to create an article in the API.
    id: int | None = None
    category: ArticleCategory
    title: str

Note

Since ids don’t bring anything for this article, I’ll omit them from now on to make the examples a bit more simple.

How does this behave by default?

Let’s see:

article = Article.model_validate({
    "category": ArticleCategory.programming,
    "title": "Fun with Pydantic",
})
print(article)

id=None category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'

Our category field is a member of the enum as expected.

What about a call to model_dump?

print("Dump in Python mode:", article.model_dump(mode="python"))
print("Dump in JSON mode:", article.model_dump(mode="json"))

Dump in Python mode: {'id': None, 'category': <ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>, 'title': 'Fun with Pydantic'}
Dump in JSON mode: {'id': None, 'category': {'id': 2, 'code': 'programming', 'title': 'Programming'}, 'title': 'Fun with Pydantic'}

If we dump for Python, the member of the enum is preserved and as JSON our model is serialized. So far, it’s only standard Pydantic stuff.

What if we pass a new Category instance? Let’s start with one equal to a member of the enum:

article = Article.model_validate({
    "category": Category(id=2, code="programming", title="Programming"),
    "title": "Fun with Pydantic",
})
print(article)

id=None category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'

Once again it works and category is a member of the enum.

What if we use a new category not part of the enum?

try:
    article = Article.model_validate({
        "category": Category(id=3, code="travel", title="Travel"),
        "title": "Traveling",
    })
except ValidationError as e:
    print("It fails:")
    print(e)

It fails:
1 validation error for Article
category
  Input should be Category(id=1, code='cooking', title='Cooking') or Category(id=2, code='programming', title='Programming') [type=enum, input_value=Category(id=3, code='travel', title='Travel'), input_type=Category]
    For further information visit https://errors.pydantic.dev/2.11/v/enum

As expected, we get an error because this category is not part of the enum.

And with a dict representing a category?

try:
    article = Article.model_validate({
        "category": {"id": 2, "code": "programming", "title": "Programming"},
        "title": "Fun with Pydantic",
    })
except ValidationError as e:
    print("It fails:")
    print(e)

It fails:
1 validation error for Article
category
  Input should be Category(id=1, code='cooking', title='Cooking') or Category(id=2, code='programming', title='Programming') [type=enum, input_value={'id': 2, 'code': 'progra... 'title': 'Programming'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/enum

That’s a bit sad. Likewise, we cannot reference our category by id or code when creating an article by default. Luckily, it’s a problem that can be solved. So, let’s do it!

Creating a custom Enum base class

If you read my previous article on custom classes and Pydantic, this idea should come to mind. It sure was the first solution that came to mine. All we need to to is create a BasePydanticEnum class inheriting from Enum and hook this enum to Pydantic with the __get_pydantic_core_schema__ class method. It will allow enum members to be loaded from their code field and serialized to their code field. I won’t detail __get_pydantic_core_schema__ too much here.

Note

To simplify code samples, I’ll assume all models used this way have a field named code for the serialization. I’ll provide at the end, as a bonus, a set of classes that allow the field to be specified for more flexibility. This way, you’ll be able to have multiple enums with Pydantic model as member and use different identifier fields.

class BasePydanticEnum(Enum):
    @classmethod
    def __get_pydantic_core_schema__(
        cls, _source: Any, _handler: GetCoreSchemaHandler
    ):
        # Get the member from the enum no matter what we have as input.
        # If we fail to find a matching member, it will fail.
        # It accepts: code, enum member and enum member value as input.
        get_member_schema = core_schema.no_info_plain_validator_function(
            cls._get_member
        )

        return core_schema.json_or_python_schema(
            json_schema=get_member_schema,
            python_schema=get_member_schema,
            # Get the code from the value of the enum member.
            serialization=core_schema.plain_serializer_function_ser_schema(
                lambda member: member.value.code
            ),
        )

    @classmethod
    def _get_member(cls, input_value: Any):
        for member in cls:
            # If the input is already a member or is a member value, let’s use it.
            if input_value == member or input_value == member.value:
                return member

            # If not, search for the member with input_value as code.
            if member.value.code == input_value:
                return member

            # Try to validate the input as a model, in case the user supplied a dict
            # representing a member. Validating during each loop is suboptimal,
            # improve this if you care about this feature.
            # Not easy since you can’t know easily the type of you member values by
            # default. Forcing the child to implement a _get_value_type class method
            # would solve this.
            try:
                model = type(member.value).model_validate(input_value)
            except ValidationError:
                continue
            else:
                # Validated successfully and matches the current member.
                if member.value == model:
                    return member

        # Raise a ValueError if our search fails for Pydantic to create its proper
        # ValidationError.
        raise ValueError(f"Failed to convert {input_value} to a member of {cls}")


class BasePydanticEnumArticleCategory(BasePydanticEnum):
    cooking = Category(id=1, code="cooking", title="Cooking")
    programming = Category(id=2, code="programming", title="Programming")


class BasePydanticEnumArticle(BaseModel):
    category: BasePydanticEnumArticleCategory
    title: str

Let’s check it works as expected. I’ll just provide the full code examples with their output, since it should be straightforward at this point.

article = BasePydanticEnumArticle.model_validate(
    {"category": BasePydanticEnumArticleCategory.programming, "title": "Fun with Pydantic"}
)
print("Creation from member:", article)
print("Dump in Python mode:", article.model_dump(mode="python"))
print("Dump in JSON mode:", article.model_dump(mode="json"))

print(
    "From a category object that’s equal to a member:",
    BasePydanticEnumArticle.model_validate(
        {
            "category": Category(
                id=2, code="programming", title="Programming"
            ),
            "title": "Fun with Pydantic",
        }
    ),
)

print(
    "Creating from a valid code:",
    BasePydanticEnumArticle.model_validate(
        {"category": "programming", "title": "Fun with Pydantic"}
    ),
)

print(
    "Creating from a valid dict:",
    BasePydanticEnumArticle.model_validate(
        {
            "category": {"id": 2, "code": "programming", "title": "Programming"},
            "title": "Fun with Pydantic",
        }
    ),
)

try:
    article = BasePydanticEnumArticle.model_validate(
        {
            "category": Category(
                id=3, code="travel", title="Travel"
            ),
            "title": "Traveling",
        }
    )
except ValidationError:
    print("Creating with a non member category fails as expected.")

try:
    article = BasePydanticEnumArticle.model_validate(
        {"category": "travel", "title": "Traveling"}
    )
except ValidationError:
    print("Creating from an invalid code fails as expected.")

Creation from member: category=<BasePydanticEnumArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'
Dump in Python mode: {'category': 'programming', 'title': 'Fun with Pydantic'}
Dump in JSON mode: {'category': 'programming', 'title': 'Fun with Pydantic'}
From a category object that’s equal to a member: category=<BasePydanticEnumArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'
Creating from a valid code: category=<BasePydanticEnumArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'
Creating from a valid dict: category=<BasePydanticEnumArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'
Creating with a non member category fails as expected.
Creating from an invalid code fails as expected.

So far so good. And problem solved for all the use cases! But, this solution also has a set of drawbacks:

All your enums must inherit from a custom enum class.
All the logic is in this custom class while the logic for the serialization and deserialization is about the model.
It requires advanced concepts.
It’s complex. And to make it generic it’s even more complex since part of the logic must be spread: the serialization/deserialization logic stays in the base enum class and the name of the identifier field must go into the model or the child class. It’s easier to enforce in the child enum since we can create the method in the base class with a raise NotImplemented body.
- Advanced note: since enum already has a metaclass, the custom class cannot inherit from ABCMeta and use the @abstract_method decorator.
This serialization/deserialization is forced on all users.

Can we find a better way to do this? Please read-on!

Alternative with annotations

If you’ve used Pydantic, you may wander if we can use the Annotate pattern to do this. Using BeforeValidator and PlainSerializer we can. It’s verbose and error prone (we must specify both each time we define a field). It also relies on less advanced Pydantic concepts.

The serializer is easy and reusable:

EnumMemberToCodeSerializer = PlainSerializer(lambda member: member.value.code)

The validator a bit less:

def get_member(enum_cls: type[Enum], input_value: Any):
    # Same logic that in the class. I left the loading a dict into a model
    # out for simplicity.
    for member in enum_cls:
        if input_value == member or input_value == member.value:
            return member

        if member.value.code == input_value:
            return member

    raise ValueError(f"Failed to find a member in {enum_cls} from {input_value}")


# The actual validator must be built with the enum as input so it can access
# the proper members.
def create_enum_from_code_validator(enum_cls: type[Enum]):
    return BeforeValidator(lambda input_value: get_member(enum_cls, input_value))

Now let’s glue it together in a model and test how this behaves:

class AnnotatedArticle(BaseModel):
    category: Annotated[
        ArticleCategory,
        create_enum_from_code_validator(ArticleCategory),
        EnumMemberToCodeSerializer,
    ]


m = AnnotatedArticle.model_validate({"category": "programming"})
print("Checking behavior:")
print(m)
print(m.model_dump(mode="python"))
print(m.model_dump(mode="json"))
print(AnnotatedArticle.model_validate({"category": ArticleCategory.cooking}))
print(AnnotatedArticle.model_validate({
    "category": ArticleCategory.programming.value,
}))
try:
    AnnotatedArticle.model_validate({"category": "travel"})
    assert False, "Should not go there"
except ValidationError:
    print("Failed with invalid code")

Checking behavior:
category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>
{'category': 'programming'}
{'category': 'programming'}
category=<ArticleCategory.cooking: Category(id=1, code='cooking', title='Cooking')>
category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>
Failed with invalid code

Note

You may wander why I didn’t use something like category: build_pydantic_enum_type_annotation(ArticleCategoryForAnnotated) to build everything with one function. build_pydantic_enum_type_annotation could be defined like so:

def build_pydantic_enum_type_annotation(enum_cls: type[Enum]):
    return Annotated[
        enum_cls,
        BeforeValidator(lambda input_value: get_member(enum_cls, input_value)),
        EnumMemberToCodeSerializer,
    ]

The reason is: it won’t work. It requires to use a call expression with a type expression and that’s not allowed.

It works, but is also quite weird and complex. I think I prefer the first solution. The field must be annotated with both a custom serializer and a custom validator or it won’t work. At least, it only concerns the leaf model, the rest can remain standard. I’m still sure we can do better.

Logic into the model that’s the member of the enum

Let’s try another approach in which the enum stays standard and it’s its Pydantic model members which know how to serialize themselves. In this solution:

Serialization is made easy with a model_serializer placed on the enum that is a member of the class.
Derialization is harder:
- We cannot hook into the model because it won’t be used: we will receive a string instead of an enum value and this string won’t be equal to any member of the enum. So Pydantic will stop there and won’t use any model_validator(mode="before") we may have to load the data. We don’t have an easy access to the enum there anyway to identify the member to use.
- We cannot use a model_validator on the enum since it’s not a Pydantic model. It would also mean we have part of the logic in the model and part on the enum which breaks locality. It would go against what we are trying to achieve here.
- We could put a model_validator on the leaf model but we break locality even further.
- What works is to hook into the __eq__ method of the model: if the value we receive is the identifier of the model, we can let Pydantic know it found the right member of the enum. When it loads the data, Pydantic will loop over the enum member and check to see whether its input is equal to one of their value. If so, it will use it.

class BaseComplexEnumModel(BaseModel):
    @model_serializer()
    def _serialize_model(self):
        return self.code

    def __eq__(self, other) -> bool:
        # Both are Pydantic models, let's see whether Pydantic thinks they are
        # equal by using the BaseModel.__eq__
        if isinstance(other, BaseModel):
            return super().__eq__(other)

        # We have an identifier, let's see whether it matches this model's one.
        return self.code == other


class ComplexCategory(BaseComplexEnumModel):
    code: str
    title: str


class ComplexArticleCategory(Enum):
    programming = ComplexCategory(code="programming", title="Programming")
    cooking = ComplexCategory(code="cooking", title="Cooking")

Let’s run some basic tests:

class ComplexArticle(BaseModel):
    category: ComplexArticleCategory


m = ComplexArticle.model_validate({"category": "programming"})
print("Checking behavior:", m)
print("From a member value:", ComplexArticle.model_validate({
    "category": ComplexArticleCategory.programming.value,
}))
print("From a member:", ComplexArticle.model_validate({
    "category": ComplexArticleCategory.cooking,
}))
try:
    ComplexArticle.model_validate({"category": "travel"})
    assert False, "Should not go there"
except ValidationError:
    print("Failed with invalid code")

# weird but it works since we overloaded __eq__ to load that.
assert ComplexCategory(code="some_code", title="Some title") == "some_code"

Checking behavior: category=<ComplexArticleCategory.programming: ComplexCategory(code='programming', title='Programming')>
From a member value: category=<ComplexArticleCategory.programming: ComplexCategory(code='programming', title='Programming')>
From a member: category=<ComplexArticleCategory.cooking: ComplexCategory(code='cooking', title='Cooking')>
Failed with invalid code

I find this solution very weird and magical. And it leaks outside the Pydantic context by changing deeply how equality behaves. It works, it illustrates a “creative” use of overloading __eq__ for advanced needs, but I can’t remand it. Once again, I’m sure we can find something better.

Putting the logic in the leaf model

After trying to put all the logic in the model that’s used in the enum, let’s try to put it at the other end: in the model that uses the enum. Just like with the annotated example, this solution uses a custom serializer and a custom validator.

class BaseLeafModelWithCodeSupport(BaseModel):
    @model_validator(mode="before")
    @classmethod
    def _load_pydantic_enum(cls, values):
        for field_name, field_def in cls.model_fields.items():
            # Do nothing for fields whose are not defined as enums
            if field_name not in values or not issubclass(field_def.annotation, Enum):
                continue

            # Loop over the enum members now that we know that field_def.annotation
            # is an enum.
            for member in field_def.annotation:
                # Find the allowed values as input: the member itself, the member
                # value and the code.
                # If one of them matches, member is the member we are looking for.
                if values[field_name] in cls._get_allowed_input_values_for_member(
                    member
                ):
                    values[field_name] = member

        return values

    @classmethod
    def _get_allowed_input_values_for_member(cls, member):
        # It’s an enum member without a code. Can’t do anything with it.
        # Let Pydantic load it its usual way.
        if not hasattr(member.value, "code"):
            return (member, member.value)

        return (member, member.value, member.value.code)

    @model_serializer(when_used="json", mode="wrap")
    def _serialize_pydantic_enums(
        self, handler: SerializerFunctionWrapHandler, info: SerializationInfo,
    ):
        # Let Pydantic serialize the values automatically.
        values = handler(self)
        for field_name, field_def in self.__pydantic_fields__.items():
            # Find the fields defined as enums.
            if not issubclass(field_def.annotation, Enum):
                continue

            # A field may not have a serialization alias, use its name in
            # this case instead.
            serialization_alias = field_def.serialization_alias or field_name
            # Use the serialization alias, only when dumping by alias.
            serialization_field_name = (
                serialization_alias if info.by_alias else field_name
            )

            field_value = getattr(self, field_name)
            # Not an enum member with a code. We don’t care about it.
            if not hasattr(field_value, "code"):
                continue

            # Some fields may be excluded from serialization (because they are
            # unset for instance). Let's ignore those.
            if serialization_field_name in values:
                values[serialization_field_name] = field_value.code

        return values

Let’s test:

class ArticleWithCodeSupport(BaseLeafModelWithCodeSupport):
    category: ArticleCategory
    category_with_alias: ArticleCategory = Field(serialization_alias="cat", default=ArticleCategory.programming)


m = ArticleWithCodeSupport.model_validate({"category": "programming", "category_with_alias": "programming"})
print("Checking behaviors")
print(m)
print(m.model_dump(mode="python"))
print(m.model_dump(mode="json", by_alias=True))
print(m.model_dump(mode="json", by_alias=False))
print(ArticleWithCodeSupport.model_validate({"category": ArticleCategory.cooking}))
print(ArticleWithCodeSupport.model_validate({"category": ArticleCategory.programming.value}))
try:
    ArticleWithCodeSupport.model_validate({"category": "travel"})
    assert False, "Should not go there"
except ValidationError:
    print("Failed with invalid code")

Checking behaviors
category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')> category_with_alias=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>
{'category': <ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>, 'category_with_alias': <ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>}
{'category': {'id': 2, 'code': 'programming', 'title': 'Programming'}, 'cat': {'id': 2, 'code': 'programming', 'title': 'Programming'}}
{'category': {'id': 2, 'code': 'programming', 'title': 'Programming'}, 'category_with_alias': {'id': 2, 'code': 'programming', 'title': 'Programming'}}
category=<ArticleCategory.cooking: Category(id=1, code='cooking', title='Cooking')> category_with_alias=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>
category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')> category_with_alias=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>
Failed with invalid code

Works very well! It’s based on model_validator and model_serializer which are more common than __get_pydantic_core_schema__. It requires care in the implementation to work correctly and to prevent issues with “normal” enum fields. Most notably, it must loop over the field definitions and access advanced attributes of the model.

On the bright side, all the logic is in the model that actually cares about it (well, in its base class to be exact). I think it’s the best solution since everything is done locally in one place and it only impacts the model that requires the behavior regarding the code attribute. The implementation is not too complex if properly documented (at least in my opinion).

Conclusion

Which solution would I choose? The “logic in the leaf model with model serialize and model validator” since I think it’s the cleanest and one of the more simple. It’s also the one with the least impact on other part of the code: only the model that need the weird “code for member behavior” is impacted. I may also consider the first solution that relies on __get_pydantic_core_schema__ since I think it’s quite clean even if logic is in a weird place. I’d probably do this if I have many custom classes and thus, __get_pydantic_core_schema__ has become very familiar to me.

I’d just discard the other possibilities as too weird or complex.

I hope you enjoyed this article about a weird use case of Pydantic with enums as much as I did and learned things on the way. I myself really liked the exploration and the different ideas that required each case to work. Don’t hesitate to leave a comment below if needed!

Bonus

In all the examples above, I relied on a field named code to do the serialization/deserialization. I didn’t try to support any other field. Here, I’ll give ways for the 1st and 4th method to select which field must be used instead. I’ll assume you’ve read the article and are at ease with it so I won’t comment the code as much. If you have a problem, let me know in the comments!

Note

I haven’t tested these solutions as much so they may be a bit buggy.

class EnumModel(BaseModel):
    code: str


class BasePydanticEnumWithIdentifier(Enum):
    """Allow enum member to be loaded from their identifier (a field of the model) and then serialized to their identifier.
    """

    @classmethod
    def __get_pydantic_core_schema__(cls, _source: Any, _handler: GetCoreSchemaHandler):
        # Get the member from the enum no matter what we have as input. If we fail to find a matching member, it will fail.
        # It accepts: code, enum member and enum member value as input.
        get_member_schema = core_schema.no_info_plain_validator_function(cls._get_member)

        return core_schema.json_or_python_schema(
            json_schema=get_member_schema,
            python_schema=get_member_schema,
            serialization=core_schema.plain_serializer_function_ser_schema(cls._serialize_to_identifier),
        )

    @classmethod
    def _get_member(cls, input_value: Any):
        for member in cls:
            if input_value == member or input_value == member.value:
                return member

            if getattr(member.value, cls._get_identifier()) == input_value:
                return member

        raise ValueError(f"Failed to convert {input_value} to a member of {cls}")

    @classmethod
    def _serialize_to_identifier(cls, member) -> Any:
        return getattr(member.value, cls._get_identifier())

    @classmethod
    def _get_identifier(cls) -> str:
        # Used to override in each enum what field of the member to use to load/serialize the data.
        # Note: using @abstractmethod is NOT straightforward: both Enum and ABC uses metaclass and they are not compatible.
        # We should be able to create our own to adapt it. Probably not worth it.
        raise NotImplementedError


class ModelValues(BasePydanticEnum):
    first_model = EnumModel(code="first_model")
    second_model = EnumModel(code="second_model")

    @classmethod
    def _get_identifier(cls):
        return "code"


class MyModel(BaseModel):
    model_value: ModelValues


m = MyModel.model_validate({"model_value": "first_model"})
print(m)
print(m.model_dump())

model_value=<ModelValues.first_model: EnumModel(code='first_model')>
{'model_value': 'first_model'}

class EnumModel(BaseModel):
    code: str

    @property
    def identifier(self):
        # Sad that it must be on this model and on no the leaf.
        return "code"


class ModelDecoratorModelValues(Enum):
    first_choice = EnumModel(code="first_choice")
    second_choice = EnumModel(code="second_choice")


class BaseModelDecoratorWithComplexEnum(BaseModel):
    @model_validator(mode="before")
    @classmethod
    def _load_pydantic_enum(cls, values):
        for field_name, field_def in cls.model_fields.items():
            if field_name not in values or not issubclass(field_def.annotation, Enum):
                continue

            # Loop over the enum members now that we know that field_def.annotation is an enum.
            for member in field_def.annotation:
                if values[field_name] in cls._get_allowed_input_values_for_member(member):
                    values[field_name] = member

        return values

    @classmethod
    def _get_allowed_input_values_for_member(cls, member):
        if not hasattr(member.value, "identifier"):
            return (member, member.value)

        member_identifier_field_name = member.value.identifier
        member_identifier = getattr(member.value, member_identifier_field_name)
        return (member, member.value, member_identifier)

    @model_serializer(when_used="json", mode="wrap")
    def _serialize_pydantic_enums(self, handler: SerializerFunctionWrapHandler, info: SerializationInfo):
        values = handler(self)
        for field_name, field_def in self.model_fields.items():
            if not issubclass(field_def.annotation, Enum):
                continue

            # A field may not have a serialization alias, use its name in this case instead.
            serialization_alias = field_def.serialization_alias or field_name
            serialization_field_name = serialization_alias if info.by_alias else field_name

            # Some fields may be excluded from serialization (because they are unset for instance). Let's ignore those.
            if serialization_field_name in values:
                values[serialization_field_name] = self._get_allowed_input_values_for_member(getattr(self, field_name))

        return values


class ModelDecoratorWithComplexEnum(BaseModelDecoratorWithComplexEnum):
    model_value: ModelValues
    enum_value: ModelDecoratorModelValues


m = MyModel.model_validate({"model_value": "first_model"})
print(m)
print(m.model_dump())

model_value=<ModelValues.first_model: EnumModel(code='first_model')>
{'model_value': 'first_model'}

Reusing Pydantic validators and serializers

2025-04-21T00:00:00+02:00

I’ll give some tips to reuse field and model validators and serializers. All my examples will use validators, but it works exactly the same for serializers.

Field validators

Assign the validator to a variable like so:

def _is_above_ten(value: int):
    if value > 10:
        return value

    raise ValueError(f"{value} is not above 10")


field_validator = AfterValidator(_is_above_ten)


class MyModel(BaseModel):
    my_field: Annotated[int, field_validator]

Model validators

With inheritance

This is specially useful if all the models requiring the validator can inherit from the same base model.

class BaseModelWithValidators(BaseModel):
    my_field: int
    my_other_field: int

    @model_validator(mode="after")
    def my_validator(self) -> Self:
        if self.my_field > self.my_other_field:
            raise ValueError("my_field must be lower or equal to my_other_field")

        return self


class MyChildModel(BaseModelWithValidators):
    third_field: int

With factories

You can achieve the same result with factories that will build the validator. It’s useful to avoid a type hierarchy since this gives you smaller pieces easier to combine. It also allows for the validator to have some configurations passed to it when the model is defined.

def validator_factory(min_value: int):
    @model_validator(mode="after")
    def my_validator(self) -> Self:
        if self.my_field > self.my_other_field:
            raise ValueError("my_field must be lower or equal to my_other_field")

        if self.my_field <= min_value:
            raise ValueError(f"my_field must be higher than {min_value}")

        return self

    return my_validator

class ModelWithValidatorFactory(BaseModel):
    my_validator = validator_factory(80)

    my_field: int
    my_other_field: int

Using Pydantic with custom classes

2025-04-19T00:00:00+02:00

Pydantic is an awesome data validation library for Python. It’s getting very popular these days because it’s fast, has lots of features and is pleasant to use. I’m using it at work and it personal projects. It’s one of the strong points of FastAPI, the new popular Python framework. It can even integrate with Django thanks to django-ninja.

While it integrates with standard Python types out of the box, using it with custom classes requires a bit more work. It’s documented here for generalities about custom classes and here for specificities about container types, but I don’t think this part of the documentation is very clear. That’s why, in this article, I’ll share my experience with how I integrated a custom class with Pydantic. I hope you can use it to understand better how it works and thus facilitate using your custom classes with Pydantic. In a second article, I’ll give another example of custom classes with Pydantic by using enums whose members are Pydantic models.

You can access the code of this article as a notebook and as a Python script. I’ll strip imports from inner examples for brevity. Feel free to download these files and experiment! The code is tested on Python 3.11 but should work with any recent enough version of Python.

For the purpose of this article, I’ll suppose you know Python and have a basic understanding of Pydantic, generic types in Python and know the common dunder methods.

Table of contents

Why I needed to integrate a custom class in the first place
The class without Pydantic
Adding Pydantic
- Testing
Conclusion

Why I needed to integrate a custom class in the first place

My goal was to create a read only mapping compatible with Pydantic. Here’s how it must work:

Usage with Pydantic must be transparent:
- When the model is serialized, the mapping must be serialized as a dict.
- It must be created from a dict containing values. Both the keys and the values from this input dict must be parsed and validated. To increase reusability, the model must be made generic to allow validation of different types: for instance a ReadonlyMapping[str, int] and ReadonlyMapping[int, float].
- Just like any other Pydantic models, if I pass an instance to the model, the model must use the class directly.
Once set in the model, the field must not be editable any more. The goal being that the model must be fully immutable so that neither its fields nor the content of the mapping can be edited by mistake.

So my goal is to arrive at something like this:

class ImmutableModel(BaseModel):
    # Make the model immutable: cannot add nor change attributes.
    model_config = ConfigDict(frozen=True)

    my_field: int
    my_dict: dict[str, str]


m = ImmutableModel.model_validate({"my_field": 1, "my_dict": {}})

try:
    m.my_field = 2
    assert False, "Must fail"
except ValidationError as e:
    print("Fails as expected with:", e)

# The underlying structure is still mutable and can be changed.
# despite the model being frozen. That’s what we want to prevent.
m.my_dict["key"] = "value"

The class without Pydantic

Let’s start by building a class that will be our generic readonly mapping. At its core, this class will just be a wrapper around a good old dict. It will rely on typing.Generic to be generic and will implement __getitem__ to provide item access. It’ll also implement __str__ and __repr__ to ease debugging. __setitem__ won’t be implemented to prevent setting an item and get a TypeError if we try to.

Note

I didn’t use MappingProxyType because I wanted a class that’s easier to extend to fit my needs here. In another context, that’s the solution I’d have chosen to have a readonly mapping.

# Generic type for the key.
Key = TypeVar("Key")
# Generic type for the value.
Value = TypeVar("Value")


class ReadonlyMapping(Generic[Key, Value]):
    def __init__(self, mapping: dict[Key, Value] | None = None):
        self._mapping: dict[Key, Value] = mapping or {}

    def __getitem__(self, key: Key):
        return self._mapping[key]

    def __str__(self):
        return str(self._mapping)

    def __repr__(self):
        return repr(self._mapping)

This class can then be used this way:

readonly_mapping = ReadonlyMapping[int, str]({1: "Test"})
print(readonly_mapping[1])

Test

And it fails as expected on key assignment:

try:
    readonly_mapping[2] = "Testing"
    assert False, "Must fail"
except TypeError as e:
    print("Fails as expected with:", e)

Fails as expected with: 'ReadonlyMapping' object does not support item assignment

Adding Pydantic

Now that the base is built, let’s iterate on that. To hook Pydantic with this custom class, a class method named __get_pydantic_core_schema__ must be added. It’ll use low level Pydantic functions to generate a proper Pydantic schema based on your generic arguments and let Pydantic handle the parsing and the validation. These methods come from pydantic_core and are used internally by Pydantic. They are documented here. Here’s how it looks like (with inline comments to explain its tricky part):

 1 class ReadonlyMapping(Generic[Key, Value]):
 2     ...
 3 
 4     @classmethod
 5     def __get_pydantic_core_schema__(
 6         cls, source: Any, handler: GetCoreSchemaHandler
 7     ) -> core_schema.CoreSchema:
 8         # Retrieve the generic arguments. args is a tuple of the type used for
 9         # Key and the type used for Value.
10         args = get_typing_args(source)
11         # If no generics are passed, args will be an empty tuple.
12         # Let’s fallback to a raw dict in this case.
13         if args:
14             # Make Pydantic generate a schema for dict[Key, Value] that it can
15             # directly use for validation.
16             # This allows for Pydantic to parse the keys and the values and to
17             # validate them without further work.
18             validate_dict_data_schema = handler.generate_schema(
19                 dict[args[0], args[1]]
20             )
21         else:
22             validate_dict_data_schema = handler.generate_schema(dict)
23 
24         # Create a schema that will build our instance from cls
25         # (ie ReadonlyMapping) from the validated dict.
26         from_dict_schema = core_schema.no_info_after_validator_function(
27             cls, validate_dict_data_schema
28         )
29 
30         # Validator to hook an already built instance of ReadonlyMapping
31         # into the model.
32         from_instance_schema = core_schema.is_instance_schema(cls)
33 
34         # Combine from_dict_schema with from_instance_schema to be able
35         # to use both.
36         # They will be tried in order: 1st from_instance_schema and if
37         # it fails from_dict_schema.
38         # If the last one fails, validation will fail.
39         schema = core_schema.union_schema(
40             [from_instance_schema, from_dict_schema]
41         )
42 
43         return core_schema.json_or_python_schema(
44             # Use our schema when raw JSON data is used directly.
45             json_schema=schema,
46             # With Python data.
47             python_schema=schema,
48             # When the model is serialized to Python or JSON, return the
49             # underlying dict that Pydantic can handle directly.
50             serialization=core_schema.plain_serializer_function_ser_schema(
51                 lambda instance: instance._mapping
52             ),
53         )

Testing

Let’s add a test model for testing:

class MyModel(BaseModel):
    my_mapping: ReadonlyMapping[str, int]

And on to the tests we go!

Loading from a `dict`

m = MyModel.model_validate({"my_int": 1, "my_mapping": {"key": "77"}})
print("The model:", type(m), m)
m_python = m.model_dump(mode="python")
print("As Python:", type(m_python), m_python)
m_json = m.model_dump_json()
print("As JSON:", type(m_json), m_json)

The model: <class '__main__.MyModel'> my_mapping={'key': 77}
As Python: <class 'dict'> {'my_mapping': {'key': 77}}
As JSON: <class 'str'> {"my_mapping":{"key":77}}

Loading from json

m = MyModel.model_validate_json("""{"my_mapping": {"key": "77"}}""")
print("The model from JSON:", m)

The model from JSON: my_mapping={'key': 77}

Key access

print(m.my_mapping, m.my_mapping["key"])

{'key': 77} 77

Access invalid key

try:
    print(m.my_mapping["nope"])
    assert False, "Must fail"
except KeyError as e:
    print("Accessing an invalid key fails as expected with error:", e)

Accessing an invalid key fails as expected with error: 'nope'

Direct creation

m = MyModel(my_mapping=ReadonlyMapping({"key": 77}))
print("Direct creation:", m)

Direct creation: my_mapping={'key': 77}

Validating with a ReadonlyMapping instance

m = MyModel.model_validate({"my_mapping": ReadonlyMapping({"key": 77})})
print("Validating with a ReadonlyMapping instance:", m)

Validating with a ReadonlyMapping instance: my_mapping={'key': 77}

Setting a key

try:
    m.my_mapping["key"] = 0
    assert False, "Must fail"
except TypeError as e:
    print("Setting a key fails as expected with error:", e)

Setting a key fails as expected with error: 'ReadonlyMapping' object does not support item assignment

Key parsing

class MyOtherModel(BaseModel):
    mapping: ReadonlyMapping[int, int | bool]


m = MyOtherModel.model_validate({"mapping": {"1": "1", "2": False}})
print("Keys and values are correctly parsed and validated:", m)

Keys and values are correctly parsed and validated: mapping={1: 1, 2: False}

Validation failure

try:
    MyOtherModel.model_validate({"mapping": {"key": "1", "2": "Some value"}})
    assert False, "Must fail"
except ValidationError as e:
    print("This fails as expected with error e:", e)

This fails as expected with error e: 4 validation errors for MyOtherModel
mapping.is-instance[ReadonlyMapping]
  Input should be an instance of ReadonlyMapping [type=is_instance_of, input_value={'key': '1', '2': 'Some value'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/is_instance_of
mapping.function-after[ReadonlyMapping(), dict[int,union[int,bool]]].key.[key]
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='key', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/int_parsing
mapping.function-after[ReadonlyMapping(), dict[int,union[int,bool]]].2.int
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='Some value', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/int_parsing
mapping.function-after[ReadonlyMapping(), dict[int,union[int,bool]]].2.bool
  Input should be a valid boolean, unable to interpret input [type=bool_parsing, input_value='Some value', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/bool_parsing

Without any generic annotation:

class NoTypeAnnotationModel(BaseModel):
    mapping: ReadonlyMapping

m = NoTypeAnnotationModel.model_validate({"mapping": {"key": "value", 1: 1}})
print("Read and parsed, no validation:", m)

Read and parsed, no validation: mapping={'key': 'value', 1: 1}

Conclusion

I hope I demystified how to use a custom class with Pydantic. Once you know a bit how __get_pydantic_core_schema__ can be used and how to use the core validation schemas, it’s not that hard. Just like me, you’ll probably have to experiment a bit to fully grasp what is going on. And if something is unclear, don’t hesitate to ask a question below!

Converting jupyter notebooks

2025-04-17T00:00:00+02:00

You can export a notebook directly from the interface under File > Save and Export Notebook As. You can also do it in CLI with:

jupyter nbconvert --execute --to <FORMAT> my_notebook.ipynb

The --execute will execute the notebook to make sure all output cells are up to date. Many export formats are supported including: python, markdown, HTML and restructured text:

jupyter nbconvert --execute --to python my_notebook.ipynb
jupyter nbconvert --execute --to markdown my_notebook.ipynb
jupyter nbconvert --execute --to html my_notebook.ipynb
jupyter nbconvert --execute --to rst my_notebook.ipynb

Sending all nginx logs to journald

2025-04-13T00:00:00+02:00

As part of my work to uniformize my server setup around systemd, I decided to make nginx log everything with journald. By default, it logs accesses and errors in log files under /var/log/nginx. I already had one log file per vhost.

To do the change, I changed my access_log and error_log directives in each vhost to:

access_log syslog:server=unix:/dev/log main;
error_log syslog:server=unix:/dev/log;

Here:

/dev/log is a socket you can use to send logs to journald.
main is my logging configuration. I can’t use the default combined because it may not log the domain for which the request was made under some conditions I didn’t bother to dig into. It would cause problems to filter logs by domain.

Note

error_log cannot have a logging configuration.

Here’s how the main logging configuration is (in /etc/nginx.conf in the main http block):

log_format  main  '$http_host - $remote_addr - $remote_user [$time_local] "$request" '
            '$status $body_bytes_sent "$http_referer" '
            '"$http_user_agent" "$http_x_forwarded_for"';

I only added $http_host at the start from the default main logging configuration suggested in the default configuration. Here’s how I filter logs by domains:

# View new entries with --follow
# Select service nginx with --unit nginx
# grep the results with --grep jujens.eu (regular expressions possible)
journalctl --follow --unit nginx --grep "jujens.eu"

Other articles regarding my switch to more systemd:

Some tips on Python’s enums

2025-04-12T00:00:00+02:00

I’d like to share a few tips I recently (re)discovered about Python’s enums. While interesting on its own, this article is also paving the way on a more advanced article I hope to write soon. Let’s dive right in!

`auto`

By using the function enum.auto, you can set integer values automatically in you enum. So this:

class MyEnum(enum.IntEnum):
    FIRST = enum.auto()
    SECOND = enum.auto()

is equivalent to this:

class MyEnum(enum.IntEnum):
    FIRST = 1
    SECOND = 2

See https://docs.python.org/3/library/enum.html#enum.auto

`unique`

Utility decorators to make sure all values are unique in the enum. So this will raise ValueError: duplicate values found in <enum 'MyEnum'>: THIRD -> SECOND:

@enum.unique
class MyEnum(enum.IntEnum):
    FIRST = 1
    SECOND = 2
    THIRD = 2

See https://docs.python.org/3/library/enum.html#enum.unique

`nonmember`

This function will prevent a field to become an enum member. It will be a standard attribute instead. So this code:

class MyEnum(enum.IntEnum):
    FIRST = 1
    SECOND = 2
    my_field = enum.nonmember(5)

print(repr(MyEnum.FIRST))
print(repr(MyEnum.my_field))

will print:

<MyEnum.FIRST: 1>
5

See https://docs.python.org/3/library/enum.html#enum.nonmember

`enum.Flag`

A utility class to create an enum that supports bitwise operations.

class Permission(enum.Flag):
    READ = enum.auto()
    WRITE = enum.auto()

read_or_write = Permission.READ | Permission.WRITE

See https://docs.python.org/3/library/enum.html#enum.Flag

`enum.verify`

A decorator to apply various predefined verifications on your enums. You can check that your values are unique (like with enum.unique), continuous or valid flags. For instance, this code will raise ValueError: invalid enum 'MyEnum': missing values 2.

@enum.verify(enum.CONTINUOUS)
class MyEnum(enum.IntEnum):
    FIRST = 1
    THIRD = 3

See https://docs.python.org/3/library/enum.html#enum.verify and https://docs.python.org/3/library/enum.html#enum.EnumCheck

Extra

Sadly, the provided verify decorator can only run pre-defined checks. Luckily, you can get inspiration from it to create your own. For instance, to check that all members are of the same type and raise a ValueError if not, you can use:

def enforce_member_type(type_to_enforce: type):
    def wrapper(cls):
        for member in cls:
            if isinstance(member.value, type_to_enforce):
                continue

            raise ValueError(f"{member} is not of type {type_to_enforce.__name__}")

        return cls

    return wrapper


@enforce_member_type(int)
class WrongTypeInEnforcedEnum(enum.Enum):
    FIRST = 1
    SECOND = "second"

PostgreSQL using ANY instead of IN

2025-03-31T00:00:00+02:00

I recently learned while reading psycopg’s documentation (a Python driver for PostgreSQL), that you should use WHERE id = ANY(:values) instead of WHERE id IN :values when filtering over lists. That’s because the ANY operator works with empty list while IN doesn’t. Psycopg will also correctly adapt lists (even empty) in this case and won’t convert them to arrays like it does with IN. Tuples and set cannot be used with neither ANY nor IN.

This could help to simplify some queries.

As far as I know, SQLite and Mysql don’t have a similar operator.

Here is a small script to test by yourselves. You need to have a PG running accepting all connection. This can be done with: docker run -it --rm -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust postgres:latest The script can be run with uv like this to install the dependencies before running the script: uv run postgresql-any.py

# To make this executable directly with uv run:
# /// script
# dependencies = [
#   "psycopg[binary]",
#   "sqlalchemy",
# ]
# ///

from sqlalchemy import create_engine, text


connection_url = "postgresql+psycopg://postgres@localhost:5432/postgres"

for query in (
    "SELECT * FROM test WHERE id IN :ids",
    "SELECT * FROM test WHERE id = ANY(:ids)",
):
    for ids in ([1, 2], [], (1, 2), {1, 2}):
        print(query, ids)
        engine = create_engine(connection_url)
        with engine.connect() as connection:
            connection.execute(text("CREATE TEMPORARY TABLE test (id serial PRIMARY KEY)"))
            connection.execute(text("INSERT INTO test(id) VALUES(1)"))
            connection.execute(text("INSERT INTO test(id) VALUES(2)"))
            connection.execute(text("INSERT INTO test(id) VALUES(3)"))

            try:
                results = connection.execute(
                    text(query),
                    {"ids": ids},
                )
                print(f"with {type(ids)}", results.fetchall())
            except Exception as e:
                print(f"Failed with {type(ids)}:", e)

        print("\n\n")
        engine.dispose()

Monitor certificates statuses

2025-03-30T00:00:00+01:00

Let’s encrypt recently announced they will stop sending emails when certificates must be renewed. Following this announcement, I created a small script to monitor the status of my certificates and receive an email if they are not renewed in time. Since it can be handy, here it is!

The check-certs.sh script. It will sent an email if a certificate in the supplied list is due to expired in 20 days.

#!/usr/bin/env bash

set -eu

WARN_IF_LESS_THAN_DAYS_LEFT=20
EMAIL=jujens@jujens.eu
# Replace with our list of sites to moniftor.
sites=(jujens.eu www.jujens.eu comments.jujens.eu)
certificates_will_expired_soon=""

now_ts=$(date +%s)

for site in ${sites[@]}; do
    expiration_date=$(echo -n | openssl s_client -servername $site -connect $site:443 |& openssl x509 -noout -dates |& grep notAfter | cut -d = -f 2)
    expiration_ts=$(date -d "$expiration_date" +%s)

    ((diff=expiration_ts-now_ts, seconds_to_days=60*60*24, nb_days_left=diff/seconds_to_days))
    message="$site will expire in $nb_days_left days"
    echo $message
    if [[ $nb_days_left -le $WARN_IF_LESS_THAN_DAYS_LEFT ]]; then
        certificates_will_expired_soon+="$message\n"
    fi
done

if [[ -n "$certificates_will_expired_soon" ]]; then
    echo -e $certificates_will_expired_soon | mail -s "Certs renewal issue!" $EMAIL
fi

I launch it weekly with a systemd timer.

The script to renew the certificates (also launched weekly with a systemd timer) is much more simple:

#!/usr/bin/env bash

set -eu

certbot renew
systemctl restart nginx