Julien Enselme personal blog - Programmation

My shell config

2025-09-14T00:00:00+02:00

I discovered a lot of tools this year, most of them I now rely on daily. So I thought it would be a nice time to share them with various custom configs I wrote.

TL;DR: install zoxide, direnv, starship and atuin and have this in your .zshrc (or .bashrc) and enjoy a better Shell experience:

eval "$(zoxide init zsh)"
eval "$(direnv hook zsh)"
eval "$(starship init zsh)"
# Beware, this doesn’t work with Bash by default.
# See https://docs.atuin.sh/guide/installation/#installing-the-shell-plugin
eval "$(atuin init zsh)"

All my configs are tracked with git to ease sharing (mostly) and track changes.

First, let’s talk about the shell. I switch to Fish in 2021 only to come back to ZSH (with oh-my-zsh) in 2024 because of work (long story short using Bash or ZSH like all my colleges makes things way easier for me). Having ZSH at home too is easier since Fish and ZSH differ in some important ways in how to write functions, conditions or loops. The gain of Fish aren’t big enough for me to use Fish only at home or to maintain work things in Fish. The fact that I don’t do advanced things in shell often and that most of my scripts remain basic and must be Bash compatible also has a play here: most of the time I don’t need the niceties of Fish. Furthermore, with the zsh-autosuggestions and zsh-syntax-highlighting plugins, I can have the two most useful features of Fish in my ZSH shell. So, definitely no pressure to use Fish.

This brings me to the most useful of the tools I want to present: starship. It allows you you configure your prompt (to display relevant data about git repo, Pythhon, Node, Rust…) directly in your shell. It’s compatible with Bash, ZSH, Fish, Elvish and more! So no matter what you are using, you can have a shell that looks the same with a very nice prompt. It has lots of configuration options, I personally only changed the colors to make it behave better on light themes.

Another very important tool to me I used for years is direnv. It can load/unload environment variables based on the directory you are in. It can also run command once the step into the directory of a project. For instance, it can automatically enable a Python virtual environment. This way, you are sure to always use the proper version of Python no matter where you are! It’s specially useful when you work on different projects.

A most recent find is atuin. It makes accessing and search your command history way easier. I find it hard to revert to "normal" history now. It even allows you to sync history between machines either by using their service or by self hosting the software. I personally don’t use this feature: commands between my work and personal projects don’t overlap enough besides simple commands to make it useful. And you have to trust the service with commands that potentially contains secrets (they say everything is encrypted, but you still need to trust them on that).

Other tools I like and use a lot (I note that most of the new ones are written in Rust and some are in Go):

git: you surely know and use it too, but I have lots of aliases you might find useful: nice shortcuts, to easily amend a commit, to easily do a fixup commit… And many more!
- I also use git-delta to have nicer output with syntax highlighting. A must have if you ask me. I have light themes as part of my configs too. difftastic seems like a nice alternative, but will change the output to make it more meaningful and I prefer to have the raw one.
- I heard of mergiraf to help solve merge conflicts, but haven’t tested it. Looks cool though!
ripgrep as a replacement of grep. It ignores files in .gitignore by default and has overall better default behaviors.
most as a replacement of less: it has better scrolling and supports binary files.
eza as a modern replacement of ls. I now even have an alias to use eza instead of ls if eza is installed!
just as a replacement of make without its weirdness, better Windows support and more reasonable defaults.
fd as a replacement of find.
zoxide to navigate more easily in directories. Its main feature is to enable easy nested navigation. Let’s say you have ~/Projects/org/my-project, do z ~/Projects/org/my-project once and then you can do z my-project where ever you are and move to Projects/org/my-project automatically. No need to remember the full path of start navigation from home! Note: I still use cd, maybe just because of old habits.
dust as a nice replacement of good old du.
micro as a nicer CLI editor than nano or vim.
httpie to have output coloring and formatting when I do HTTP requests in CLI.
tokei to have nice stats about code (number of files, lines…).

Other articles about shell configurations that you may find useful:

Please share your tools and configs in the comments!

My opinion on Django Ninja

2025-07-06T00:00:00+02:00

If you’re using the Django web framework (DRF) and are building APIs, you probably know about the Django Rest Framework a toolkit to build APIs in Django since more than a decade (the first 3.x version was release in 2014). It’s great and I’ve used it professionally to build several APIs. It’s also complex and showing its age when compared to what can be done in Flask or FastAPI.

Luckily, there’s a new possibility in Django’s space: django-ninja. It reached 1.0 in 2023 and was first release in 2020. It’s heavily inspired by FastAPI, relies on Pydantic for payload validation and supports async out of the box. It’s also easier to setup. That’s why I decided to use it on one of my personal projects which needed an API. After about a year, I’d like to give my opinion on it.

Please keep in mind that I only built a small API with it. I haven’t used it yet on bigger projects, so I can’t say much about this use case. I think it’s ready and mature enough and would definitely try it over good old DRF, but you may reach for someone else’s opinion if you plan to use it on a big project.

Overall, I’m pleased and very impressed by django-ninja. The biggest selling point for me is Pydantic. This library is awesome and used more and more. I already use it extensively at work. I also use it in my project to validate the input of JSON data stored in the database and to parse some JSON files. Being able to also use it for my API is a big plus for me: use cases are very similar and having only one library to learn is a breeze.

Regarding the creation of the API itself, it’s a bunch of decorated and type annotated functions. django-ninja takes care of the rest like validating POST payload or mapping GET parameters to function parameters. It even generate the documentation automatically. On that regard, it’s indeed very close to how FastAPI works. Simple and yet efficient. It seems to lack the ability to use class based views, which could be detrimental to grouping some functions.

I also like the fact that to change the HTTP response code, all you have to do is return a tuple of status_code, response like in Flask.

I’m a bit surprised it doesn’t handle more things regarding authentication. It supports Django’s auth out of the box, but it’s based on cookies. So it’s only useful to debug the API in your browser. For an API key, HTTP Bearer or HTTP Basic auth, it gives you a base class you can subclass to implement your custom method. I choose to use a JWT token passed in a header. I guess it’s not more integrated because there are so many options available and possible links with the user model that what you’d need would be there anyway.

While I’m very pleased with what django-ninja has to offer, I encountered a few issues:

I have Content Security Policy (CSP) with nonce enabled on my site thanks to django-csp. Because of this, the JS and CSS loaded by the docs couldn’t be loaded because no nonce was applied in the doc templates. I solved it by copying the swagger.html template and adding nonce="{{ request.csp_nonce }}" were the static files are loaded.
When not authenticated and targeting the API with curl, I got some CSRF error. Making the HttpBearer auth the default (ie before django_auth in the list of auth providers) solved the problem.
On some endpoints, I allow the PATCH HTTP method to be used to update only the supplied fields. Initially, I used PatchDict provided by django-ninja. Sadly, given how it’s implemented, it allowed some fields to be set to None in the payload without validation error when these fields are not nullable. This caused errors at the database level when the API tried to save these models. What I ended up doing to solve this is:
1. Create a custom type named NotSet integrated in Pydantic thanks to the __get_pydantic_core_schema__ class method. It takes the underlying type as an argument to display beautiful default values in the documentation. You can view its implementation here.
2. Create a custom update function that skips values set to NotSet when updating the Django model. You can view its implementation here.
3. Use the type like this read_at: datetime | SkipJsonSchema[NotSet] | None = NotSet(datetime.now) in my Ninja schemas. The SkipJsonSchema prevents the type to be mentioned in the documentation since it’s a technical type users can’t use directly.
It’s not as easy as I’d hoped, but it works and I think it’s an acceptable work around, at least for my needs. There’s probably a way to make this more generic and reusable on existing schemas, but I don’t need that right now.

To conclude, I’m glad this solution exists and I think I have a cleaner, more modern and more simple way to create APIs than DRF. I had some issues and spent time in the documentation at first, but nothing weird since it’s a new library for me. From what I’ve seen and after my usage, it seems like a powerful, feature complete solution to build API with Django. I hope I’ll be able to use on a bigger project!

As usual, I’d be delighted to hear your opinion in the comments below.

My opinions on HTMX

2025-06-24T00:00:00+02:00

HTMX is a JavaScript library designed to add interactivity to your web pages. With it, you don’t build a single page application, you use standard templates (written in Django, Jinja) rendered in your backend, add attributes to your HTML elements and let HTMX add the appropriate interactivity. From what I read, it’s getting lots of popularity in the Django world. That’s why I heard of it in the first place and one of the reason I choose it.

There are alternatives like Alpine.js or Turbo. After looking at their documentation and examples, I had the feeling HTMX is better suited for what I was building. HTMX also seems more popular and integration in Django is made easy thanks to django-htmx. It’s also quite small (at least compared to the cost of a SPA): only 17.5kb once minified and gzipped. It should be almost all the JS you need when you build something with it: if you need way more, switching to a SPA could make more sense. Likewise, if you need to build an app or handle complex state, a true SPA might be easier to maintain in the long run.

I think the library is powerful and relatively easy to use: all you have to do is include it and add the proper attributes. It also has lots of examples to help you get started. Pretty much everything I needed to use already had an example associated with it.

In my case, I use it in an article reading application: the backend fetches RSS feeds and displays the fetched articles on a page. The articles are grouped into various reading lists. To improve UX, when a user clicks on the "Mark as read" button, I don’t want a full page reload. That’s where HTMX comes in!

With HTMX, I can create a form and submit it to mark the article as read without this page reload. To do this, adding hx-boost and hx-swap to the form is enough to make HTMX handle the submission and handle the HTML from the response. A nice feature here is that you can update multiple part of the page in one go: the element associated with the update and something else thanks to a process named Out of Band Content swapping (oob for short): instead of just returning the element you need to swap, you can return multiple and HTMX will swap them all. In my case: the article card and the unread count are updated at the same time thanks to this process.

Two features required a bit more work to make them work correctly.

The first one is a feature, which when enabled, allows to read articles on scroll to easily scan big chunks of articles and only open those that interest you. This required some custom JS to correctly detect scroll end, wait a few seconds without interactions to trigger the action (to make sure the user won’t scroll up), find the articles that are not visible anymore and mark them all as read. It required some digging into the documentation to be able to hook on htmx and use its JavaScript API correctly. In the end, it worked really well!

The second one, is when users try to delete an article: I wanted to display a pretty modal and not a basic JS alert. Part of the problem comes from the fact that I have multiple possible action buttons and only for some I want the confirm modal to be triggered. This isn’t supported out of the box.

What I ended up doing was have multiple forms and use the form attribute of buttons to select which form the buttons must submit. This way, I can have a delete form with hx-confirm (use to display an alert or run a custom action on a click) and only open the modal in that case. As far as I can remember, putting hx-confirm on the button directly didn’t work (I don’t remember the exact behavior in that case). To have this work in all cases, I had to setup a custom event listeners on htmx:confirm to capture the event and run some custom processing code. I also had to make sure the name of the button was correctly transmitted in the request which wasn’t done automatically by HTMX in that case for some reason. The full code is available here if you need more details.

To write the proper solution, I had to debug HTMX. It’s a pain as in any library you don’t know but not a very big deal. In Django debug mode, I load the unminified version of the library and from there, I was able to add breakpoints from the browser tools and see what was happening. I had to do it another time when I tried an HTML minification library which broke HTMX completely. It turns out, it removed type="submit" attributes since it’s the default, but HTMX requires them to boost forms.

The library is great and I’m very pleased by it. There are however, as with any tech, some downsides:

HTMX behaviors can only be tested with end to end tests. Not much of an issue for me: they don’t change much and can be tested manually. It can be a problem on a bigger app, but I think it’s also hard to mess up something the smallest amount of manual testing would catch: you are only adding HTML attributes after all, the rest is handled by HTMX itself.
I think you must enable CSP (Content Security Policy) to make sure no HTML coming from an untrusted source is used by HTMX to swap elements and thus execute JavaScript. You should to it anyway, HTMX or not.
If you only enhance links and forms with hx-boost, your application will work without JS since browser have a fallback. If you do other things, JS becomes required since without it some behaviors will break. Not a very big deal in my opinion since all your users should have JS enabled. In my case, some menus managed by Bootstrap won’t open without JS, making the navigation in the app a pain anyway.
Since these attributes are specific to HTMX, I guess there is a form of vendor lock-in. But the attributes are easy to spot and since the alternatives also rely on attributes, I guess a search/replace would do most of the migration. It’s still way easier than changing SPA frameworks since it does way less!
The documentation promotes behaviors that are not accessible by default. To quote it:

Why should only <a> & <form> be able to make HTTP requests?

Why should only click & submit events trigger them?

Why should only GET & POST methods be available?

Why should you only be able to replace the entire screen?

I guess it’s up to you to make sure that what you do is accessible, just like with a SPA and use the proper attributes for accessibility. The examples could also promote accessibility more. Some already do, without explaining it enough in my taste, but most don’t. I think that accessibility is in such a bad place, that having all the official examples promote best practices would be a huge help.

According to this hacker news thread, the maintainers are aware of this and may improve the situation. There are also one issue about the accessibility implication of HTMX, one about improving the examples and there seems to have plans to use the new moveBefore API to improve the situation (this API is only available in Chrome and only since the beginning of the year).

To be fully honest, I don’t think my project is fully accessible. As a backend developer, I don’t know enough to see the issue (and I’m probably far from the only one if I trust my experience and the reports regarding the poor state of web accessibility). That’s also why, I rely as much as possible on forms with progressive enhancement and use Bootstrap which have an accessibility team. No matter the lib, most examples I see don’t even mention it. That’s not only an HTMX problem.

In conclusion, HTMX adds some complexity compared to basic pages, but it remains manageable. I’m happy with the lib and the fact it allows me to write most of my logic in Python and Django with standard HTML and almost no custom JavaScript. Now that the heavy lifting is done, it’s been months since I even had to touch HTMX features. Almost all my logic lives in the backend which feels easier and make me more productive: no language or context juggling. Plus, all the behaviors are defined in the HTML with the rest of the attributes. Depending on what you are building, it can be a good and more simple alternative to a full fledged SPA. It also gives me some memories of when I was a student and was building websites without a JS framework: use of plain JS files to extend your site’s features. It was a mess back then, even with the help of jQuery. Now, it feels clean and easy!

Simply how licenses are applied to your project with SPDX and reuse

2025-06-22T00:00:00+02:00

SPDX (Software Package Data Exchange) is a project managed by the Linux foundation created to standardize how licenses are identified in a human and machine readable way. In a nutshell, instead of adding a big header to your files to identify the applicable license, you apply a copyright text and a license identifier. For a Python file under the GPL, instead of this:

{{ project }}
Copyright (C) {{ year }}  {{ organization }}

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

It would give this:

# SPDX-FileCopyrightText: {{ year }} {{ organization }}
#
# SPDX-License-Identifier: GPL-3.0-or-later

It’s shorter and more readable (at least in my opinion). And a script can easily read this header and compile a list of all used licenses and to each file they apply.

You can go to the list of licenses to find the identifier and text of the licenses you want to use. Apparently, these identifiers have been around for several years (at least 2021) and are already used by big projects like KDE, Qt or Fedora. I personally only learned about them earlier this year and dug into the subject this month.

So far, I thought is was interesting, but wouldn’t impact my projects too much: I’d only have to switch one header for another. That’s until I learned about the REUSE tool by the FSFE in Fedora magazine. The goal of this tool is to help you choose licenses, download their full text and add the proper header to your files. For files that cannot have a license header, it can create a <FILE_NAME>.license file or define the license in a REUSE.toml file. Once this is done, it can enforce that all your files have a license.

It’s easy to setup (including in pre-commit hooks) and comes with a nice tutorial and FAQ to help you get started. The catch is that it can be a bit time consuming to enforce at first to find the proper licenses to apply to each file. Luckily, it can add headers to multiple files at the same time. This tool is used by KDE, Linux, Nextcloud and Curl!

In my case, I used it in one of my GNU AGLP licensed project. It contained many files of different types, some with a proper AGPL header, some without. And others were copy/pasted from the internet and thus the AGPL with my name in the copyright didn’t apply. So I had to review for each file what to do and apply the header with the proper copyright text in bulk of files. At least, now it’s done and files I don’t have the copyright on but I can use in my project (because they are published on a free software license like MIT) are clearly identified.

I wandered whether I should use only the SPDX-FileCopyrightText and SPDX-License-Identifier in the header or keep the full GNU AGPL copyright statement. According to the GNU AGLP full text, this header is only a strong suggestion to help readers identify the license of the file. So nothing mandatory on that side and I think the SPDX-License-Identifier fills that role too even if it’s less clear since it doesn’t give any explanation nor explicitly state the absence of warranty. After looking a bit in the Qt, KDE and REUSE repositories, which all use some variants of the GNU GLP licenses, I saw only the SPDX identifier and not the classical headers, I guess it’s fine. It’s also much better than most programs under non GLP licenses which tend to have no header at all! I also searched the internet for this but couldn’t find anything useful. If you have more information and this, please let me know in the comments below.

Some interesting articles I found on this subject:

SPDX Tutorial
Making sense of software licensing with FSFE REUSE: A beginner’s guide for open source developers
Understanding and Using SPDX License Identifiers and License Expressions
SPDX License Expressions in my opinion it has the best explanation of the AND operator.

Promise returned by async functions

2025-05-24T00:00:00+02:00

Recently at work, I encountered a problem with a function that should have returned a custom promise subclass. The goal of this subclass is to be able to call a cancel method to cancel some pending action done within the promise and then reject it. It looks like this:

class CancellablePromise extends Promise {
    constructor(executor) {
        super(executor)
    }

    cancel() {
        console.log("Canceling");
        // Actual operation
    }
}

Unlike what I expected, I didn’t get a CancelablePromise but a standard Promise object. I initially thought it came from the chain: I didn’t stored the return promise directly but the output of a .catch method like this:

const myPromise = buildCancellable().catch(() => console.error("Oops"));

After running some tests, it turns out that chaining from a custom promise class will return an instance of the custom promise class. So the culprit wasn’t there. I continued debugging and acquired the certainty that buildCancellable didn’t return the custom class at all in the first place. I’m also certain it used to.

It turns out the buildCancellable function was made async a few months ago. Instead of being defined like this:

const buildCancellable = () => new CancellablePromise(() => { ... });

It’s now defined like this:

const buildCancellable = async () => {
    // Stuff using await
    return new CancellablePromise(() => { ... })
};

It turns out that returned values of async function are always wrapped in a proper Promise even if they are themselves already a Promise. It has a very little impact on most usage since nested promise are automatically unwrapped, so we have this behavior:

const innerPromise = new Promise(resolve => {
  setTimeout(() => {
    console.log("Resolving inner promise");
    resolve("Inner value");
  }, 2_000);
});

const outerPromise = new Promise(resolve => {
  setTimeout(() => {
    console.log("resolving outer promise");
    resolve(innerPromise);
  }, 1_000)
});

const value = await outerPromise;

// Will log "Inner value" as expected.
console.log("The value:", value);

But it made the cancel method non reachable since the actual CancellablePromise cannot be reached now. The only solution is to remove the async keywords and remove await from the body.

I hope you found this reminder useful. I did have an of course moment once I figured it out, but was a bit lost at the start. I really expected to get the proper object back (and I still think it’s reasonable to expect Promise to not be wrapped again) since it’s already a Promise. I guess it’s best for consistency to have JS behave this way. And it’s transparent in almost all cases anyway.

Using Pydantic models within enums

2025-04-26T00:00:00+02:00

In previous articles, I provided some tips on enums and explained how to use a custom class with Pydantic. Now, I’ll dig into all kinds of enum usages with Pydantic, including enums whose members are Pydantic models themselves! Let’s dive right in!

Just like before, you can access the code of this article as a Python script and as a notebook. I won’t include the imports in the code examples for brevity, refer to the notebook or the script if needed.

Table of contents

Using normal enums
With different types in the enum
Using enums with Pydantic models
Conclusion
Bonus

Using normal enums

Let’s start small with how Pydantic behaves with normal enums.

class IntValues(IntEnum):
    first = 1
    second = 2


class StrValues(StrEnum):
    valid = "valid"
    invalid = "invalid"


class NormalEnumModel(BaseModel):
    # False is the default. If True, will use 1 or "valid"
    # in the model instead of the enum members.
    model_config = ConfigDict(use_enum_values=False)

    int_value: IntValues
    str_value: StrValues


m = NormalEnumModel.model_validate({"int_value": 1, "str_value": "valid"})
print("The model:", m)
print("Dumped in Python mode:", m.model_dump(mode="python"))
print("Dumped in JSON mode:", m.model_dump(mode="json"))

The model: int_value=<IntValues.first: 1> str_value=<StrValues.valid: 'valid'>
Dumped in Python mode: {'int_value': <IntValues.first: 1>, 'str_value': <StrValues.valid: 'valid'>}
Dumped in JSON mode: {'int_value': 1, 'str_value': 'valid'}

Each field is correctly parsed as an enum member. When dumping in JSON mode, the value is used and the member remains in Python mode. I say this behaves as expected.

Once again, as expected, values outside the enum will raise a ValidationError:

try:
    # Values outside the enum, will fail!
    NormalEnumModel.model_validate({"int_value": 10, "str_value": "nope"})
    assert False, "Must fail"
except ValidationError as e:
    print("Failing as expected with values outside the enum with:")
    print(e)

Failing as expected with values outside the enum with:
2 validation errors for NormalEnumModel
int_value
  Input should be 1 or 2 [type=enum, input_value=10, input_type=int]
    For further information visit https://errors.pydantic.dev/2.11/v/enum
str_value
  Input should be 'valid' or 'invalid' [type=enum, input_value='nope', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/enum

Let’s illustrate the behavior when forcing Pydantic to use enum values instead of enum members with use_enum_values=True:

class MyModelWithValues(BaseModel):
    model_config = ConfigDict(use_enum_values=True)

    int_value: IntValues
    str_value: StrValues


m = MyModelWithValues.model_validate({"int_value": 1, "str_value": "valid"})
print("The model:", m)
print("Dumped in Python mode:", m.model_dump(mode="python"))
print("Dumped in JSON mode:", m.model_dump(mode="json"))

The model: int_value=1 str_value='valid'
Dumped in Python mode: {'int_value': 1, 'str_value': 'valid'}
Dumped in JSON mode: {'int_value': 1, 'str_value': 'valid'}

This time, the values is used in the model and no matter how it’s dumped.

With different types in the enum

Now that the basics are covered, let’s use more complex enums. Let’s start to see what happens if we try to mix types in IntEnum and StrEnum. If Python can parse a type automatically in IntEnum it will do so:

class WithIntAndIntAsString(IntEnum):
    member1 = 1
    # This will be parse automatically by Python.
    member2 = "10"


print(
    "member1",
    type(WithIntAndIntAsString.member1.value),
    WithIntAndIntAsString.member1,
)
print(
    "member2",
    type(WithIntAndIntAsString.member2.value),
    WithIntAndIntAsString.member2,
)

member1 <class 'int'> 1
member2 <class 'int'> 10

Otherwise, it will raise a TypeError:

try:
    class MoreStringValues(StrEnum):
        member1 = "member1"
        # Python won’t cast anything to int.
        value2 = 10
except TypeError as e:
    print("Failing with error:", e)

Failing with error: 10 is not a string

If we need to mix types, we can use a base Enum:

class MultipleBasicTypes(Enum):
    member1 = "member1"
    member2 = 10


print(
    type(MultipleBasicTypes.member1.value),
    type(MultipleBasicTypes.member2.value),
)

<class 'str'> <class 'int'>

If you need to enforce a consistent type within your enums, please refer to of my previous article and the enforce_member_type decorator.

Using enums with Pydantic models

Now that we know how enums behave with Pydantic, let’s dive the heart of this article: enums with Pydantic models as members and their usage in other Pydantic models! I’ll detail several possible strategies here and discuss the pros and cons of each one. You’ll probably have questions or remarks on these (and maybe even new methods), so don’t hesitate to use the comment section.

I’ll use an example to make things less abstract. I’ll leave your imagination work to find useful use-cases. You can also just see it as a way to have fun with Python and Pydantic.

Let’s imagine that you are running a blog platform. Each article on the platform is associated with a category. A category could be represented with this Pydantic model:

class Category(BaseModel):
    # The id of the category in the database
    id: int
    # A unique human-readable code identifying the category.
    code: str
    # The title to display to the users.
    title: str

Let’s say that for editorial reasons the list of categories is fixed. So this list can fit into an enum. The ids and the codes are set within the enum to ease referencing the categories in the codebase without needing to query the database. This will help to create new categories in the database when you add some. All you’ll have to do is loop over the members of the enum.

We’ll also assume it allows great simplification in you code to be able to hard code the id of the category directly. The enum makes this clean and readable. It will also make your JSON API more simple to use for your users. To create an article, they can use the human readable code of the category instead of the id. They can send this:

{"category": "programming", "title": "Fun with Pydantic"}

to get this back:

Article(category=ArticleCategory.programming, title="Fun with Pydantic")

No need to bother them with ids!

Let’s start this journey by creating the ArticleCategory enum:

class ArticleCategory(Enum):
    cooking = Category(id=1, code="cooking", title="Cooking")
    programming = Category(id=2, code="programming", title="Programming")

You’ll then be able to mention this in you article model:

class Article(BaseModel):
    # id of the article in the database.
    # Allow None in the model to enable user to create an article in the API.
    id: int | None = None
    category: ArticleCategory
    title: str

Note

Since ids don’t bring anything for this article, I’ll omit them from now on to make the examples a bit more simple.

How does this behave by default?

Let’s see:

article = Article.model_validate({
    "category": ArticleCategory.programming,
    "title": "Fun with Pydantic",
})
print(article)

id=None category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'

Our category field is a member of the enum as expected.

What about a call to model_dump?

print("Dump in Python mode:", article.model_dump(mode="python"))
print("Dump in JSON mode:", article.model_dump(mode="json"))

Dump in Python mode: {'id': None, 'category': <ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>, 'title': 'Fun with Pydantic'}
Dump in JSON mode: {'id': None, 'category': {'id': 2, 'code': 'programming', 'title': 'Programming'}, 'title': 'Fun with Pydantic'}

If we dump for Python, the member of the enum is preserved and as JSON our model is serialized. So far, it’s only standard Pydantic stuff.

What if we pass a new Category instance? Let’s start with one equal to a member of the enum:

article = Article.model_validate({
    "category": Category(id=2, code="programming", title="Programming"),
    "title": "Fun with Pydantic",
})
print(article)

id=None category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'

Once again it works and category is a member of the enum.

What if we use a new category not part of the enum?

try:
    article = Article.model_validate({
        "category": Category(id=3, code="travel", title="Travel"),
        "title": "Traveling",
    })
except ValidationError as e:
    print("It fails:")
    print(e)

It fails:
1 validation error for Article
category
  Input should be Category(id=1, code='cooking', title='Cooking') or Category(id=2, code='programming', title='Programming') [type=enum, input_value=Category(id=3, code='travel', title='Travel'), input_type=Category]
    For further information visit https://errors.pydantic.dev/2.11/v/enum

As expected, we get an error because this category is not part of the enum.

And with a dict representing a category?

try:
    article = Article.model_validate({
        "category": {"id": 2, "code": "programming", "title": "Programming"},
        "title": "Fun with Pydantic",
    })
except ValidationError as e:
    print("It fails:")
    print(e)

It fails:
1 validation error for Article
category
  Input should be Category(id=1, code='cooking', title='Cooking') or Category(id=2, code='programming', title='Programming') [type=enum, input_value={'id': 2, 'code': 'progra... 'title': 'Programming'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/enum

That’s a bit sad. Likewise, we cannot reference our category by id or code when creating an article by default. Luckily, it’s a problem that can be solved. So, let’s do it!

Creating a custom Enum base class

If you read my previous article on custom classes and Pydantic, this idea should come to mind. It sure was the first solution that came to mine. All we need to to is create a BasePydanticEnum class inheriting from Enum and hook this enum to Pydantic with the __get_pydantic_core_schema__ class method. It will allow enum members to be loaded from their code field and serialized to their code field. I won’t detail __get_pydantic_core_schema__ too much here.

Note

To simplify code samples, I’ll assume all models used this way have a field named code for the serialization. I’ll provide at the end, as a bonus, a set of classes that allow the field to be specified for more flexibility. This way, you’ll be able to have multiple enums with Pydantic model as member and use different identifier fields.

class BasePydanticEnum(Enum):
    @classmethod
    def __get_pydantic_core_schema__(
        cls, _source: Any, _handler: GetCoreSchemaHandler
    ):
        # Get the member from the enum no matter what we have as input.
        # If we fail to find a matching member, it will fail.
        # It accepts: code, enum member and enum member value as input.
        get_member_schema = core_schema.no_info_plain_validator_function(
            cls._get_member
        )

        return core_schema.json_or_python_schema(
            json_schema=get_member_schema,
            python_schema=get_member_schema,
            # Get the code from the value of the enum member.
            serialization=core_schema.plain_serializer_function_ser_schema(
                lambda member: member.value.code
            ),
        )

    @classmethod
    def _get_member(cls, input_value: Any):
        for member in cls:
            # If the input is already a member or is a member value, let’s use it.
            if input_value == member or input_value == member.value:
                return member

            # If not, search for the member with input_value as code.
            if member.value.code == input_value:
                return member

            # Try to validate the input as a model, in case the user supplied a dict
            # representing a member. Validating during each loop is suboptimal,
            # improve this if you care about this feature.
            # Not easy since you can’t know easily the type of you member values by
            # default. Forcing the child to implement a _get_value_type class method
            # would solve this.
            try:
                model = type(member.value).model_validate(input_value)
            except ValidationError:
                continue
            else:
                # Validated successfully and matches the current member.
                if member.value == model:
                    return member

        # Raise a ValueError if our search fails for Pydantic to create its proper
        # ValidationError.
        raise ValueError(f"Failed to convert {input_value} to a member of {cls}")


class BasePydanticEnumArticleCategory(BasePydanticEnum):
    cooking = Category(id=1, code="cooking", title="Cooking")
    programming = Category(id=2, code="programming", title="Programming")


class BasePydanticEnumArticle(BaseModel):
    category: BasePydanticEnumArticleCategory
    title: str

Let’s check it works as expected. I’ll just provide the full code examples with their output, since it should be straightforward at this point.

article = BasePydanticEnumArticle.model_validate(
    {"category": BasePydanticEnumArticleCategory.programming, "title": "Fun with Pydantic"}
)
print("Creation from member:", article)
print("Dump in Python mode:", article.model_dump(mode="python"))
print("Dump in JSON mode:", article.model_dump(mode="json"))

print(
    "From a category object that’s equal to a member:",
    BasePydanticEnumArticle.model_validate(
        {
            "category": Category(
                id=2, code="programming", title="Programming"
            ),
            "title": "Fun with Pydantic",
        }
    ),
)

print(
    "Creating from a valid code:",
    BasePydanticEnumArticle.model_validate(
        {"category": "programming", "title": "Fun with Pydantic"}
    ),
)

print(
    "Creating from a valid dict:",
    BasePydanticEnumArticle.model_validate(
        {
            "category": {"id": 2, "code": "programming", "title": "Programming"},
            "title": "Fun with Pydantic",
        }
    ),
)

try:
    article = BasePydanticEnumArticle.model_validate(
        {
            "category": Category(
                id=3, code="travel", title="Travel"
            ),
            "title": "Traveling",
        }
    )
except ValidationError:
    print("Creating with a non member category fails as expected.")

try:
    article = BasePydanticEnumArticle.model_validate(
        {"category": "travel", "title": "Traveling"}
    )
except ValidationError:
    print("Creating from an invalid code fails as expected.")

Creation from member: category=<BasePydanticEnumArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'
Dump in Python mode: {'category': 'programming', 'title': 'Fun with Pydantic'}
Dump in JSON mode: {'category': 'programming', 'title': 'Fun with Pydantic'}
From a category object that’s equal to a member: category=<BasePydanticEnumArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'
Creating from a valid code: category=<BasePydanticEnumArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'
Creating from a valid dict: category=<BasePydanticEnumArticleCategory.programming: Category(id=2, code='programming', title='Programming')> title='Fun with Pydantic'
Creating with a non member category fails as expected.
Creating from an invalid code fails as expected.

So far so good. And problem solved for all the use cases! But, this solution also has a set of drawbacks:

All your enums must inherit from a custom enum class.
All the logic is in this custom class while the logic for the serialization and deserialization is about the model.
It requires advanced concepts.
It’s complex. And to make it generic it’s even more complex since part of the logic must be spread: the serialization/deserialization logic stays in the base enum class and the name of the identifier field must go into the model or the child class. It’s easier to enforce in the child enum since we can create the method in the base class with a raise NotImplemented body.
- Advanced note: since enum already has a metaclass, the custom class cannot inherit from ABCMeta and use the @abstract_method decorator.
This serialization/deserialization is forced on all users.

Can we find a better way to do this? Please read-on!

Alternative with annotations

If you’ve used Pydantic, you may wander if we can use the Annotate pattern to do this. Using BeforeValidator and PlainSerializer we can. It’s verbose and error prone (we must specify both each time we define a field). It also relies on less advanced Pydantic concepts.

The serializer is easy and reusable:

EnumMemberToCodeSerializer = PlainSerializer(lambda member: member.value.code)

The validator a bit less:

def get_member(enum_cls: type[Enum], input_value: Any):
    # Same logic that in the class. I left the loading a dict into a model
    # out for simplicity.
    for member in enum_cls:
        if input_value == member or input_value == member.value:
            return member

        if member.value.code == input_value:
            return member

    raise ValueError(f"Failed to find a member in {enum_cls} from {input_value}")


# The actual validator must be built with the enum as input so it can access
# the proper members.
def create_enum_from_code_validator(enum_cls: type[Enum]):
    return BeforeValidator(lambda input_value: get_member(enum_cls, input_value))

Now let’s glue it together in a model and test how this behaves:

class AnnotatedArticle(BaseModel):
    category: Annotated[
        ArticleCategory,
        create_enum_from_code_validator(ArticleCategory),
        EnumMemberToCodeSerializer,
    ]


m = AnnotatedArticle.model_validate({"category": "programming"})
print("Checking behavior:")
print(m)
print(m.model_dump(mode="python"))
print(m.model_dump(mode="json"))
print(AnnotatedArticle.model_validate({"category": ArticleCategory.cooking}))
print(AnnotatedArticle.model_validate({
    "category": ArticleCategory.programming.value,
}))
try:
    AnnotatedArticle.model_validate({"category": "travel"})
    assert False, "Should not go there"
except ValidationError:
    print("Failed with invalid code")

Checking behavior:
category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>
{'category': 'programming'}
{'category': 'programming'}
category=<ArticleCategory.cooking: Category(id=1, code='cooking', title='Cooking')>
category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>
Failed with invalid code

Note

You may wander why I didn’t use something like category: build_pydantic_enum_type_annotation(ArticleCategoryForAnnotated) to build everything with one function. build_pydantic_enum_type_annotation could be defined like so:

def build_pydantic_enum_type_annotation(enum_cls: type[Enum]):
    return Annotated[
        enum_cls,
        BeforeValidator(lambda input_value: get_member(enum_cls, input_value)),
        EnumMemberToCodeSerializer,
    ]

The reason is: it won’t work. It requires to use a call expression with a type expression and that’s not allowed.

It works, but is also quite weird and complex. I think I prefer the first solution. The field must be annotated with both a custom serializer and a custom validator or it won’t work. At least, it only concerns the leaf model, the rest can remain standard. I’m still sure we can do better.

Logic into the model that’s the member of the enum

Let’s try another approach in which the enum stays standard and it’s its Pydantic model members which know how to serialize themselves. In this solution:

Serialization is made easy with a model_serializer placed on the enum that is a member of the class.
Derialization is harder:
- We cannot hook into the model because it won’t be used: we will receive a string instead of an enum value and this string won’t be equal to any member of the enum. So Pydantic will stop there and won’t use any model_validator(mode="before") we may have to load the data. We don’t have an easy access to the enum there anyway to identify the member to use.
- We cannot use a model_validator on the enum since it’s not a Pydantic model. It would also mean we have part of the logic in the model and part on the enum which breaks locality. It would go against what we are trying to achieve here.
- We could put a model_validator on the leaf model but we break locality even further.
- What works is to hook into the __eq__ method of the model: if the value we receive is the identifier of the model, we can let Pydantic know it found the right member of the enum. When it loads the data, Pydantic will loop over the enum member and check to see whether its input is equal to one of their value. If so, it will use it.

class BaseComplexEnumModel(BaseModel):
    @model_serializer()
    def _serialize_model(self):
        return self.code

    def __eq__(self, other) -> bool:
        # Both are Pydantic models, let's see whether Pydantic thinks they are
        # equal by using the BaseModel.__eq__
        if isinstance(other, BaseModel):
            return super().__eq__(other)

        # We have an identifier, let's see whether it matches this model's one.
        return self.code == other


class ComplexCategory(BaseComplexEnumModel):
    code: str
    title: str


class ComplexArticleCategory(Enum):
    programming = ComplexCategory(code="programming", title="Programming")
    cooking = ComplexCategory(code="cooking", title="Cooking")

Let’s run some basic tests:

class ComplexArticle(BaseModel):
    category: ComplexArticleCategory


m = ComplexArticle.model_validate({"category": "programming"})
print("Checking behavior:", m)
print("From a member value:", ComplexArticle.model_validate({
    "category": ComplexArticleCategory.programming.value,
}))
print("From a member:", ComplexArticle.model_validate({
    "category": ComplexArticleCategory.cooking,
}))
try:
    ComplexArticle.model_validate({"category": "travel"})
    assert False, "Should not go there"
except ValidationError:
    print("Failed with invalid code")

# weird but it works since we overloaded __eq__ to load that.
assert ComplexCategory(code="some_code", title="Some title") == "some_code"

Checking behavior: category=<ComplexArticleCategory.programming: ComplexCategory(code='programming', title='Programming')>
From a member value: category=<ComplexArticleCategory.programming: ComplexCategory(code='programming', title='Programming')>
From a member: category=<ComplexArticleCategory.cooking: ComplexCategory(code='cooking', title='Cooking')>
Failed with invalid code

I find this solution very weird and magical. And it leaks outside the Pydantic context by changing deeply how equality behaves. It works, it illustrates a “creative” use of overloading __eq__ for advanced needs, but I can’t remand it. Once again, I’m sure we can find something better.

Putting the logic in the leaf model

After trying to put all the logic in the model that’s used in the enum, let’s try to put it at the other end: in the model that uses the enum. Just like with the annotated example, this solution uses a custom serializer and a custom validator.

class BaseLeafModelWithCodeSupport(BaseModel):
    @model_validator(mode="before")
    @classmethod
    def _load_pydantic_enum(cls, values):
        for field_name, field_def in cls.model_fields.items():
            # Do nothing for fields whose are not defined as enums
            if field_name not in values or not issubclass(field_def.annotation, Enum):
                continue

            # Loop over the enum members now that we know that field_def.annotation
            # is an enum.
            for member in field_def.annotation:
                # Find the allowed values as input: the member itself, the member
                # value and the code.
                # If one of them matches, member is the member we are looking for.
                if values[field_name] in cls._get_allowed_input_values_for_member(
                    member
                ):
                    values[field_name] = member

        return values

    @classmethod
    def _get_allowed_input_values_for_member(cls, member):
        # It’s an enum member without a code. Can’t do anything with it.
        # Let Pydantic load it its usual way.
        if not hasattr(member.value, "code"):
            return (member, member.value)

        return (member, member.value, member.value.code)

    @model_serializer(when_used="json", mode="wrap")
    def _serialize_pydantic_enums(
        self, handler: SerializerFunctionWrapHandler, info: SerializationInfo,
    ):
        # Let Pydantic serialize the values automatically.
        values = handler(self)
        for field_name, field_def in self.__pydantic_fields__.items():
            # Find the fields defined as enums.
            if not issubclass(field_def.annotation, Enum):
                continue

            # A field may not have a serialization alias, use its name in
            # this case instead.
            serialization_alias = field_def.serialization_alias or field_name
            # Use the serialization alias, only when dumping by alias.
            serialization_field_name = (
                serialization_alias if info.by_alias else field_name
            )

            field_value = getattr(self, field_name)
            # Not an enum member with a code. We don’t care about it.
            if not hasattr(field_value, "code"):
                continue

            # Some fields may be excluded from serialization (because they are
            # unset for instance). Let's ignore those.
            if serialization_field_name in values:
                values[serialization_field_name] = field_value.code

        return values

Let’s test:

class ArticleWithCodeSupport(BaseLeafModelWithCodeSupport):
    category: ArticleCategory
    category_with_alias: ArticleCategory = Field(serialization_alias="cat", default=ArticleCategory.programming)


m = ArticleWithCodeSupport.model_validate({"category": "programming", "category_with_alias": "programming"})
print("Checking behaviors")
print(m)
print(m.model_dump(mode="python"))
print(m.model_dump(mode="json", by_alias=True))
print(m.model_dump(mode="json", by_alias=False))
print(ArticleWithCodeSupport.model_validate({"category": ArticleCategory.cooking}))
print(ArticleWithCodeSupport.model_validate({"category": ArticleCategory.programming.value}))
try:
    ArticleWithCodeSupport.model_validate({"category": "travel"})
    assert False, "Should not go there"
except ValidationError:
    print("Failed with invalid code")

Checking behaviors
category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')> category_with_alias=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>
{'category': <ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>, 'category_with_alias': <ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>}
{'category': {'id': 2, 'code': 'programming', 'title': 'Programming'}, 'cat': {'id': 2, 'code': 'programming', 'title': 'Programming'}}
{'category': {'id': 2, 'code': 'programming', 'title': 'Programming'}, 'category_with_alias': {'id': 2, 'code': 'programming', 'title': 'Programming'}}
category=<ArticleCategory.cooking: Category(id=1, code='cooking', title='Cooking')> category_with_alias=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>
category=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')> category_with_alias=<ArticleCategory.programming: Category(id=2, code='programming', title='Programming')>
Failed with invalid code

Works very well! It’s based on model_validator and model_serializer which are more common than __get_pydantic_core_schema__. It requires care in the implementation to work correctly and to prevent issues with “normal” enum fields. Most notably, it must loop over the field definitions and access advanced attributes of the model.

On the bright side, all the logic is in the model that actually cares about it (well, in its base class to be exact). I think it’s the best solution since everything is done locally in one place and it only impacts the model that requires the behavior regarding the code attribute. The implementation is not too complex if properly documented (at least in my opinion).

Conclusion

Which solution would I choose? The “logic in the leaf model with model serialize and model validator” since I think it’s the cleanest and one of the more simple. It’s also the one with the least impact on other part of the code: only the model that need the weird “code for member behavior” is impacted. I may also consider the first solution that relies on __get_pydantic_core_schema__ since I think it’s quite clean even if logic is in a weird place. I’d probably do this if I have many custom classes and thus, __get_pydantic_core_schema__ has become very familiar to me.

I’d just discard the other possibilities as too weird or complex.

I hope you enjoyed this article about a weird use case of Pydantic with enums as much as I did and learned things on the way. I myself really liked the exploration and the different ideas that required each case to work. Don’t hesitate to leave a comment below if needed!

Bonus

In all the examples above, I relied on a field named code to do the serialization/deserialization. I didn’t try to support any other field. Here, I’ll give ways for the 1st and 4th method to select which field must be used instead. I’ll assume you’ve read the article and are at ease with it so I won’t comment the code as much. If you have a problem, let me know in the comments!

Note

I haven’t tested these solutions as much so they may be a bit buggy.

class EnumModel(BaseModel):
    code: str


class BasePydanticEnumWithIdentifier(Enum):
    """Allow enum member to be loaded from their identifier (a field of the model) and then serialized to their identifier.
    """

    @classmethod
    def __get_pydantic_core_schema__(cls, _source: Any, _handler: GetCoreSchemaHandler):
        # Get the member from the enum no matter what we have as input. If we fail to find a matching member, it will fail.
        # It accepts: code, enum member and enum member value as input.
        get_member_schema = core_schema.no_info_plain_validator_function(cls._get_member)

        return core_schema.json_or_python_schema(
            json_schema=get_member_schema,
            python_schema=get_member_schema,
            serialization=core_schema.plain_serializer_function_ser_schema(cls._serialize_to_identifier),
        )

    @classmethod
    def _get_member(cls, input_value: Any):
        for member in cls:
            if input_value == member or input_value == member.value:
                return member

            if getattr(member.value, cls._get_identifier()) == input_value:
                return member

        raise ValueError(f"Failed to convert {input_value} to a member of {cls}")

    @classmethod
    def _serialize_to_identifier(cls, member) -> Any:
        return getattr(member.value, cls._get_identifier())

    @classmethod
    def _get_identifier(cls) -> str:
        # Used to override in each enum what field of the member to use to load/serialize the data.
        # Note: using @abstractmethod is NOT straightforward: both Enum and ABC uses metaclass and they are not compatible.
        # We should be able to create our own to adapt it. Probably not worth it.
        raise NotImplementedError


class ModelValues(BasePydanticEnum):
    first_model = EnumModel(code="first_model")
    second_model = EnumModel(code="second_model")

    @classmethod
    def _get_identifier(cls):
        return "code"


class MyModel(BaseModel):
    model_value: ModelValues


m = MyModel.model_validate({"model_value": "first_model"})
print(m)
print(m.model_dump())

model_value=<ModelValues.first_model: EnumModel(code='first_model')>
{'model_value': 'first_model'}

class EnumModel(BaseModel):
    code: str

    @property
    def identifier(self):
        # Sad that it must be on this model and on no the leaf.
        return "code"


class ModelDecoratorModelValues(Enum):
    first_choice = EnumModel(code="first_choice")
    second_choice = EnumModel(code="second_choice")


class BaseModelDecoratorWithComplexEnum(BaseModel):
    @model_validator(mode="before")
    @classmethod
    def _load_pydantic_enum(cls, values):
        for field_name, field_def in cls.model_fields.items():
            if field_name not in values or not issubclass(field_def.annotation, Enum):
                continue

            # Loop over the enum members now that we know that field_def.annotation is an enum.
            for member in field_def.annotation:
                if values[field_name] in cls._get_allowed_input_values_for_member(member):
                    values[field_name] = member

        return values

    @classmethod
    def _get_allowed_input_values_for_member(cls, member):
        if not hasattr(member.value, "identifier"):
            return (member, member.value)

        member_identifier_field_name = member.value.identifier
        member_identifier = getattr(member.value, member_identifier_field_name)
        return (member, member.value, member_identifier)

    @model_serializer(when_used="json", mode="wrap")
    def _serialize_pydantic_enums(self, handler: SerializerFunctionWrapHandler, info: SerializationInfo):
        values = handler(self)
        for field_name, field_def in self.model_fields.items():
            if not issubclass(field_def.annotation, Enum):
                continue

            # A field may not have a serialization alias, use its name in this case instead.
            serialization_alias = field_def.serialization_alias or field_name
            serialization_field_name = serialization_alias if info.by_alias else field_name

            # Some fields may be excluded from serialization (because they are unset for instance). Let's ignore those.
            if serialization_field_name in values:
                values[serialization_field_name] = self._get_allowed_input_values_for_member(getattr(self, field_name))

        return values


class ModelDecoratorWithComplexEnum(BaseModelDecoratorWithComplexEnum):
    model_value: ModelValues
    enum_value: ModelDecoratorModelValues


m = MyModel.model_validate({"model_value": "first_model"})
print(m)
print(m.model_dump())

model_value=<ModelValues.first_model: EnumModel(code='first_model')>
{'model_value': 'first_model'}

Using Pydantic with custom classes

2025-04-19T00:00:00+02:00

Pydantic is an awesome data validation library for Python. It’s getting very popular these days because it’s fast, has lots of features and is pleasant to use. I’m using it at work and it personal projects. It’s one of the strong points of FastAPI, the new popular Python framework. It can even integrate with Django thanks to django-ninja.

While it integrates with standard Python types out of the box, using it with custom classes requires a bit more work. It’s documented here for generalities about custom classes and here for specificities about container types, but I don’t think this part of the documentation is very clear. That’s why, in this article, I’ll share my experience with how I integrated a custom class with Pydantic. I hope you can use it to understand better how it works and thus facilitate using your custom classes with Pydantic. In a second article, I’ll give another example of custom classes with Pydantic by using enums whose members are Pydantic models.

You can access the code of this article as a notebook and as a Python script. I’ll strip imports from inner examples for brevity. Feel free to download these files and experiment! The code is tested on Python 3.11 but should work with any recent enough version of Python.

For the purpose of this article, I’ll suppose you know Python and have a basic understanding of Pydantic, generic types in Python and know the common dunder methods.

Table of contents

Why I needed to integrate a custom class in the first place
The class without Pydantic
Adding Pydantic
- Testing
Conclusion

Why I needed to integrate a custom class in the first place

My goal was to create a read only mapping compatible with Pydantic. Here’s how it must work:

Usage with Pydantic must be transparent:
- When the model is serialized, the mapping must be serialized as a dict.
- It must be created from a dict containing values. Both the keys and the values from this input dict must be parsed and validated. To increase reusability, the model must be made generic to allow validation of different types: for instance a ReadonlyMapping[str, int] and ReadonlyMapping[int, float].
- Just like any other Pydantic models, if I pass an instance to the model, the model must use the class directly.
Once set in the model, the field must not be editable any more. The goal being that the model must be fully immutable so that neither its fields nor the content of the mapping can be edited by mistake.

So my goal is to arrive at something like this:

class ImmutableModel(BaseModel):
    # Make the model immutable: cannot add nor change attributes.
    model_config = ConfigDict(frozen=True)

    my_field: int
    my_dict: dict[str, str]


m = ImmutableModel.model_validate({"my_field": 1, "my_dict": {}})

try:
    m.my_field = 2
    assert False, "Must fail"
except ValidationError as e:
    print("Fails as expected with:", e)

# The underlying structure is still mutable and can be changed.
# despite the model being frozen. That’s what we want to prevent.
m.my_dict["key"] = "value"

The class without Pydantic

Let’s start by building a class that will be our generic readonly mapping. At its core, this class will just be a wrapper around a good old dict. It will rely on typing.Generic to be generic and will implement __getitem__ to provide item access. It’ll also implement __str__ and __repr__ to ease debugging. __setitem__ won’t be implemented to prevent setting an item and get a TypeError if we try to.

Note

I didn’t use MappingProxyType because I wanted a class that’s easier to extend to fit my needs here. In another context, that’s the solution I’d have chosen to have a readonly mapping.

# Generic type for the key.
Key = TypeVar("Key")
# Generic type for the value.
Value = TypeVar("Value")


class ReadonlyMapping(Generic[Key, Value]):
    def __init__(self, mapping: dict[Key, Value] | None = None):
        self._mapping: dict[Key, Value] = mapping or {}

    def __getitem__(self, key: Key):
        return self._mapping[key]

    def __str__(self):
        return str(self._mapping)

    def __repr__(self):
        return repr(self._mapping)

This class can then be used this way:

readonly_mapping = ReadonlyMapping[int, str]({1: "Test"})
print(readonly_mapping[1])

Test

And it fails as expected on key assignment:

try:
    readonly_mapping[2] = "Testing"
    assert False, "Must fail"
except TypeError as e:
    print("Fails as expected with:", e)

Fails as expected with: 'ReadonlyMapping' object does not support item assignment

Adding Pydantic

Now that the base is built, let’s iterate on that. To hook Pydantic with this custom class, a class method named __get_pydantic_core_schema__ must be added. It’ll use low level Pydantic functions to generate a proper Pydantic schema based on your generic arguments and let Pydantic handle the parsing and the validation. These methods come from pydantic_core and are used internally by Pydantic. They are documented here. Here’s how it looks like (with inline comments to explain its tricky part):

 1 class ReadonlyMapping(Generic[Key, Value]):
 2     ...
 3 
 4     @classmethod
 5     def __get_pydantic_core_schema__(
 6         cls, source: Any, handler: GetCoreSchemaHandler
 7     ) -> core_schema.CoreSchema:
 8         # Retrieve the generic arguments. args is a tuple of the type used for
 9         # Key and the type used for Value.
10         args = get_typing_args(source)
11         # If no generics are passed, args will be an empty tuple.
12         # Let’s fallback to a raw dict in this case.
13         if args:
14             # Make Pydantic generate a schema for dict[Key, Value] that it can
15             # directly use for validation.
16             # This allows for Pydantic to parse the keys and the values and to
17             # validate them without further work.
18             validate_dict_data_schema = handler.generate_schema(
19                 dict[args[0], args[1]]
20             )
21         else:
22             validate_dict_data_schema = handler.generate_schema(dict)
23 
24         # Create a schema that will build our instance from cls
25         # (ie ReadonlyMapping) from the validated dict.
26         from_dict_schema = core_schema.no_info_after_validator_function(
27             cls, validate_dict_data_schema
28         )
29 
30         # Validator to hook an already built instance of ReadonlyMapping
31         # into the model.
32         from_instance_schema = core_schema.is_instance_schema(cls)
33 
34         # Combine from_dict_schema with from_instance_schema to be able
35         # to use both.
36         # They will be tried in order: 1st from_instance_schema and if
37         # it fails from_dict_schema.
38         # If the last one fails, validation will fail.
39         schema = core_schema.union_schema(
40             [from_instance_schema, from_dict_schema]
41         )
42 
43         return core_schema.json_or_python_schema(
44             # Use our schema when raw JSON data is used directly.
45             json_schema=schema,
46             # With Python data.
47             python_schema=schema,
48             # When the model is serialized to Python or JSON, return the
49             # underlying dict that Pydantic can handle directly.
50             serialization=core_schema.plain_serializer_function_ser_schema(
51                 lambda instance: instance._mapping
52             ),
53         )

Testing

Let’s add a test model for testing:

class MyModel(BaseModel):
    my_mapping: ReadonlyMapping[str, int]

And on to the tests we go!

Loading from a `dict`

m = MyModel.model_validate({"my_int": 1, "my_mapping": {"key": "77"}})
print("The model:", type(m), m)
m_python = m.model_dump(mode="python")
print("As Python:", type(m_python), m_python)
m_json = m.model_dump_json()
print("As JSON:", type(m_json), m_json)

The model: <class '__main__.MyModel'> my_mapping={'key': 77}
As Python: <class 'dict'> {'my_mapping': {'key': 77}}
As JSON: <class 'str'> {"my_mapping":{"key":77}}

Loading from json

m = MyModel.model_validate_json("""{"my_mapping": {"key": "77"}}""")
print("The model from JSON:", m)

The model from JSON: my_mapping={'key': 77}

Key access

print(m.my_mapping, m.my_mapping["key"])

{'key': 77} 77

Access invalid key

try:
    print(m.my_mapping["nope"])
    assert False, "Must fail"
except KeyError as e:
    print("Accessing an invalid key fails as expected with error:", e)

Accessing an invalid key fails as expected with error: 'nope'

Direct creation

m = MyModel(my_mapping=ReadonlyMapping({"key": 77}))
print("Direct creation:", m)

Direct creation: my_mapping={'key': 77}

Validating with a ReadonlyMapping instance

m = MyModel.model_validate({"my_mapping": ReadonlyMapping({"key": 77})})
print("Validating with a ReadonlyMapping instance:", m)

Validating with a ReadonlyMapping instance: my_mapping={'key': 77}

Setting a key

try:
    m.my_mapping["key"] = 0
    assert False, "Must fail"
except TypeError as e:
    print("Setting a key fails as expected with error:", e)

Setting a key fails as expected with error: 'ReadonlyMapping' object does not support item assignment

Key parsing

class MyOtherModel(BaseModel):
    mapping: ReadonlyMapping[int, int | bool]


m = MyOtherModel.model_validate({"mapping": {"1": "1", "2": False}})
print("Keys and values are correctly parsed and validated:", m)

Keys and values are correctly parsed and validated: mapping={1: 1, 2: False}

Validation failure

try:
    MyOtherModel.model_validate({"mapping": {"key": "1", "2": "Some value"}})
    assert False, "Must fail"
except ValidationError as e:
    print("This fails as expected with error e:", e)

This fails as expected with error e: 4 validation errors for MyOtherModel
mapping.is-instance[ReadonlyMapping]
  Input should be an instance of ReadonlyMapping [type=is_instance_of, input_value={'key': '1', '2': 'Some value'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/is_instance_of
mapping.function-after[ReadonlyMapping(), dict[int,union[int,bool]]].key.[key]
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='key', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/int_parsing
mapping.function-after[ReadonlyMapping(), dict[int,union[int,bool]]].2.int
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='Some value', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/int_parsing
mapping.function-after[ReadonlyMapping(), dict[int,union[int,bool]]].2.bool
  Input should be a valid boolean, unable to interpret input [type=bool_parsing, input_value='Some value', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/bool_parsing

Without any generic annotation:

class NoTypeAnnotationModel(BaseModel):
    mapping: ReadonlyMapping

m = NoTypeAnnotationModel.model_validate({"mapping": {"key": "value", 1: 1}})
print("Read and parsed, no validation:", m)

Read and parsed, no validation: mapping={'key': 'value', 1: 1}

Conclusion

I hope I demystified how to use a custom class with Pydantic. Once you know a bit how __get_pydantic_core_schema__ can be used and how to use the core validation schemas, it’s not that hard. Just like me, you’ll probably have to experiment a bit to fully grasp what is going on. And if something is unclear, don’t hesitate to ask a question below!

Some tips on Python’s enums

2025-04-12T00:00:00+02:00

I’d like to share a few tips I recently (re)discovered about Python’s enums. While interesting on its own, this article is also paving the way on a more advanced article I hope to write soon. Let’s dive right in!

`auto`

By using the function enum.auto, you can set integer values automatically in you enum. So this:

class MyEnum(enum.IntEnum):
    FIRST = enum.auto()
    SECOND = enum.auto()

is equivalent to this:

class MyEnum(enum.IntEnum):
    FIRST = 1
    SECOND = 2

See https://docs.python.org/3/library/enum.html#enum.auto

`unique`

Utility decorators to make sure all values are unique in the enum. So this will raise ValueError: duplicate values found in <enum 'MyEnum'>: THIRD -> SECOND:

@enum.unique
class MyEnum(enum.IntEnum):
    FIRST = 1
    SECOND = 2
    THIRD = 2

See https://docs.python.org/3/library/enum.html#enum.unique

`nonmember`

This function will prevent a field to become an enum member. It will be a standard attribute instead. So this code:

class MyEnum(enum.IntEnum):
    FIRST = 1
    SECOND = 2
    my_field = enum.nonmember(5)

print(repr(MyEnum.FIRST))
print(repr(MyEnum.my_field))

will print:

<MyEnum.FIRST: 1>
5

See https://docs.python.org/3/library/enum.html#enum.nonmember

`enum.Flag`

A utility class to create an enum that supports bitwise operations.

class Permission(enum.Flag):
    READ = enum.auto()
    WRITE = enum.auto()

read_or_write = Permission.READ | Permission.WRITE

See https://docs.python.org/3/library/enum.html#enum.Flag

`enum.verify`

A decorator to apply various predefined verifications on your enums. You can check that your values are unique (like with enum.unique), continuous or valid flags. For instance, this code will raise ValueError: invalid enum 'MyEnum': missing values 2.

@enum.verify(enum.CONTINUOUS)
class MyEnum(enum.IntEnum):
    FIRST = 1
    THIRD = 3

See https://docs.python.org/3/library/enum.html#enum.verify and https://docs.python.org/3/library/enum.html#enum.EnumCheck

Extra

Sadly, the provided verify decorator can only run pre-defined checks. Luckily, you can get inspiration from it to create your own. For instance, to check that all members are of the same type and raise a ValueError if not, you can use:

def enforce_member_type(type_to_enforce: type):
    def wrapper(cls):
        for member in cls:
            if isinstance(member.value, type_to_enforce):
                continue

            raise ValueError(f"{member} is not of type {type_to_enforce.__name__}")

        return cls

    return wrapper


@enforce_member_type(int)
class WrongTypeInEnforcedEnum(enum.Enum):
    FIRST = 1
    SECOND = "second"

Writing a browser extension

2025-02-16T00:00:00+01:00

I recently wrote a browser extension for Firefox and Chromium based browsers for my Legadilo (RSS feeds aggregator and articles saver) project. The goal is to make it easier and faster to save an article or to subscribe to a feed directly on a web page without forcing you to open the app. I never wrote a browser extension before and it didn’t go as I thought it would. So I’d like to share a bit of my experience.

At high level, an extension is just a bunch of HTML, JS and CSS files zipped together with a manifest.json file and some icons. I rely on Mozilla’s web-ext tool to do the packaging for me. The manifest.json is there to:

Define where the icons are.
Define what permissions the extension will need
The version (v2 or v3) of the extension.
Link your files to what the extension needs:
- Point to the HTML file for options.
- Point to the HTML file for each action you will define.
- Point to the background script or service worker to use to handle network requests.

The other JS files are loaded directly in their respective HTML file. I load them with <script src="action.js" type="module"></script> to be able to use new style imports in them without transpilling.

To keep things simple, I decided not to use a framework and to code interactions myself. So I have all "screens" directly in the same HTML hidden by default with display: none. My JS file then hides/displays what is needed and fills the required content by updating and inserting relevant DOM nodes. Like in the old days. It works fine for me since my extension remains very simple, but I think that for more complex use cases, a lightweight framework like React, Vue or Svelte would come in handy. I also wander if you could use web components (eventually with a library like LitElement) to keeps things small and simple and rely on the new features of the browser instead.

I took me some time to figure out how to wire everything together. I didn’t find the doc that great and ended up mostly relying on existing extension to understand how to do what I wanted to do. With hours of tries and errors.

I developed my extension of Firefox because I prefer their dev tools (just like for normal web development). The most frustrating part was understanding how the dev tools work to have proper debugging and logging. On Firefox, it happens on a dedicated page debug: and then opening the proper dev tools. Sometimes, I had to manually refresh the extension in there, and sometimes it was automatic. With more experience, it may become more obvious when a manual action is needed.

Once I got everything working in Firefox, I tried the extension on Chromium. And it didn’t work. I decided to use manifest v3 right away: for this extension the new restrictions don’t apply and I thought both browsers support it correctly in the same way. It turns out, for the background script part (both how it runs and how it communicates with other JS scripts), Firefox still uses the manifest v2 way while Chromium is on the v3 way. I think it’s to still allow network based ad blockers to work with Firefox. This introduced lots of small differences even for my tiny extension.

What I ended up doing: instead of trying to solve them at once, I copied the FF extension and ported it to "full" v3 so it would work correctly on Chrome. I then diffed the two implementations and worked on merging them. It took some time, but I managed to do that by introducing glue functions:

In the background.js script, both browsers don’t subscribe to events the same way. Symmetrically, they don’t send responses the same way.

// Check whether we are on FF.
const isFirefox = typeof browser === "object";

if (isFirefox) {
    browser.runtime.onConnect.addListener(function (port) {
        port.onMessage.addListener(
            (request) => onMessage(request, port.postMessage)
        );
    });
} else {
    chrome.runtime.onMessage.addListener((request, sender, sendResponse) => {
        onMessage(request, sendResponse);

        // This is required to correctly get the response on the other side!
        // I lost lots of time because I forgot it because it seems so useless!
        return true;
    });
}

const onMessage = async (request, sendResponse) => {
    // request is on the format you need. Same for response.
    // I choose a very simple { name: "", payload {} } structure.
    try {
        switch (request.name) {
            case "save-article":
                await processSaveArticleRequest(sendResponse, request.payload);
                break;
            default:
                console.warn(`Unknown action ${request.name}`);
                break;
        }
    } catch (err) {
        sendResponse({ error: err.message });
    }
};

On the action.js end, I must connect to the port on FF but not on Chromium. This gives a little wrapper like this:

// Only used on FF.
let port;
const isFirefox = typeof browser === "object";

document.addEventListener("DOMContentLoaded", async () => {
    connectToPort();
    // Do extension related stuff
});

const connectToPort = () => {
    // Chromium based browsers will receive the response directly
    // from the function call sending the message.
    if (!isFirefox) {
        return;
    }

    port = chrome.runtime.connect({ name: "legadilo-popup" });
    port.onMessage.addListener(onMessage);
};

And then to send messages to the background script (still in action.js):

const doStuff = () => {
sendMessage({
    name: "update-feed",
        payload: {
            feedId: 1,
        },
    });
}

const sendMessage = async (message) => {
    if (isFirefox) {
        // On response will be called as part of the port behavior.
        port.postMessage(message);
        return;
    }

    // Manually call onMessage to process the response in the same way
    // on all browsers.
    const response = await chrome.runtime.sendMessage(message);
    onMessage(response);
};

const onMessage = (request) => {
    if (request.error) {
        displayErrorMessage(request.error);
        return;
    }

    hideErrorMessage();
    hideLoader();
    switch (request.name) {
        case "saved-article":
            savedArticleSuccess(request.article, request.tags);
            break;
        default:
            console.warn(`Unknown action ${request.name}`);
    }
};

Once I had everything working on both browsers, the time to publish this extension on the web stores came:

On Firefox it was really easy: a small form to fill with the extension attached to it. An automated review later, the extension was online.
For Chrome, it took much longer:
- First, you must pay a small fee (to protect against spam I suppose).
- Then I had to fill a form, which took much longer than for Firefox: I needed extra icons, screenshots and I had to give much more details mostly about data collection and a privacy. I couldn’t help but ask myself: "like Google cared about privacy and data collection". That’s the only reason I have an almost empty privacy page on the app.
- Only then and after one rejection because I was missing some infos, I managed to publish the extension.

Now my extension is online and I use it. I still have works to do to improve it, but the base is here and useful. I’m glad of the UX it gives. I didn’t expect cross browser compatibility to be that hard. I guess it comes from the fact we are still in the transition to manifest v3. I also wasn’t ready for how much infos the Chrome webstore asks to publish even this basic extension. On the bright side, it forced me to write a simple privacy policy that states I don’t collect nor sell anything (so you know that as well).

Using podman for containers

2025-02-09T00:00:00+01:00

Podman is an alternative to Docker and Docker compose. It uses the same CLI interface than Docker and uses the same standardized image format. So you can use an image built with Docker with it or build an image and then use it with Docker. Its podman-compose command is compatible with Docker compose files, so again nothing to adapt there.

If both tools are so close, you are probably wandering why not use Docker since it’s more famous? While close in its usage, Podman is very different in its implementation. It doesn’t rely on a daemon nor requires root privileges to run containers. So you don’t have to enable anything to get started and you have better isolation by default: within the container root will be mapped to the current user and other users to dedicated UID and GIU on the host. For more on this subject and Docker, see my "use linux user namespaces in docker" article.

Note

This relies on /etc/subuid and /etc/subguid files on linux. Make sure they are filled with something like this (this should be the case on modern distributions):

# username:1st sub id:number of sub id
# Typically this will be:
jujens:100000:65536

See my article linked above for more on this.

Since there is no daemon started by default on boot, it means containers won’t be restarted after a reboot in the default configuration. I think it can be quite good for development containers you may not need after each reboot. But if you want to use it to run your production containers, that’s a problem. Luckily, Podman let you manage your containers with systemctl thanks to dedicated unit files. This means you have all the power of systemd to run your containers and manage their dependencies.

To do this (full example below):

Create a file ending with .container in ~/.config/containers/systemd/. For instance, ~/.config/containers/systemd/legadilo.container
Refresh the daemon with systemctl --user daemon-reload (--user is paramount since we are using a non root user to run the unit).
Start the unit with systemctl --user start <UNIT> In this case: systemctl --user start legadilo
Check that the container is running with podman ps
Reboot and check that the container is still running. By default, your containers will be named systemd-<UNIT> In this case, systemd-legadilo. Any volumes or networks will be created automatically from their associated definition files.

What’s very nice is that you can also rely on this system to auto-update containers. Let’s say you are using the legadilo:latest image. If it changes, running podman auto-update will pull and restart the service with the new image automatically. This can also be run automatically thanks to the podman-auto-update.timer provided systemd timer! For more on systemd timers, see this article.

Here is a sample legadilo.container file to help you get started:

[Unit]
Description=Legadilo container
# Dependencies can be any service as usual.
# Or any other containers referenced by their implicit service file.
Requires=postgres.service
After=postgres.service

[Container]
Image=rg.fr-par.scw.cloud/legadilo/legadilo-django:latest
# Force a name to prevent the systemd- prefix
ContainerName=legadilo
# To rely on journald to view the logs (optional).
LogDriver=journald
# Configure directly the environment variables here.
Environment=ENV_VAR=value
# Or use an env file.
EnvironmentFile=~/.private/legadilo.env
PublishPort=8080:8000
Network=legadilo.network

[Install]
# Required to make the container start on boot.
WantedBy=multi-user.target
WantedBy=default.target

And the related legadilo.network (put it next to the .container file):

[Unit]
Description=Legadilo Network

[Network]
Subnet=192.168.30.0/24
Gateway=192.168.30.1

And the legadilo-postgres.volume for the PostgreSQL database (still next to the .container file):

[Unit]
Description=PG Volume

[Volume]

And the legadilo-postgres.container file for completeness:

[Unit]
Description=Podman PostgreSQL

[Container]
Image=docker.io/postgres:17
ContainerName=legadilo-postgres
LogDriver=journald
Volume=postgres.volume:/var/lib/postgresql/data
Environment=POSTGRES_HOST=postgres
Environment=POSTGRES_PORT=5432
Environment=POSTGRES_DB=legadilo
Environment=POSTGRES_USER=legadilo
Environment=POSTGRES_PASSWORD=<RETRACTED>
Network=legadilo.network

[Install]
WantedBy=multi-user.target
WantedBy=default.target

You can then inspect the logs with journalctl CONTAINER_NAME=legadilo and journalctl CONTAINER_NAME=legadilo-postgres

Note

I gave a name the legadilo container since it will never clash with something system related. I think it’s easier that way. But, do it the way you want.

Note

If needed, you can access the host with the host.containers.internal name. I think it requires Podman 5.3 or more if you use 5.x versions of Podman to due a limitation in earlier releases 5.x releases.

If you do, please make sure the service listens to the proper interface. Otherwise, it won’t accept the connection coming from the container. It can be 0.0.0.0. to listen to all interfaces or the interface associated with the docker container to limit incoming traffic. In the default configuration, it appears to be 192.168.100.133. By running ip a on the host, you should be able to find the right address.

To conclude, I think Podman is a great replacement for Docker. It solves nicely and by default the long standing issue I had with isolation (ie not running the containers as root while still having files created by the root user in the container owned by my user outside). Being able to manage containers with systemd is awesome and allows me to run all services the same way: with systemctl! I’m now using podman locally on personal work and intent to use it on my servers.

That’s it for today. If you have any questions or remarks, please leave them below!

Resources:

Systemd Timers

2025-02-01T00:00:00+01:00

After using anacron for years to run a backup script regularly, I decided to have a look at systemd timers. Overall, anacron worked fine: I could run tasks as my user and it would start tasks if they missed a run. But, I was still frustrated with how it worked: I had to configure it to be able to run cron tasks with a non root user and never remembered where the config file was. I also wanted better notifications in case of failure: CRON programs send logs by email by default which is of no use to me on my local machine. And I am in the process of using systemd more on my servers. So it only makes sense to use it more locally too and to rely on its features to replace CRON jobs.

After digging a bit, the feature that convinced me to use systemd timers is the systemctl --user list-timers command (omit the --user option to list system timers): it lists all the timers with their next and last execution dates as well as how much time is remaining before the next run. I can see everything in a human readable way with one command! Of course, everything ends logged in journald! As with any systemd service, you can also specify dependencies of the service to run. I can also use Weekly and Monthly keywords to more easily define when to run a task without bothering with CRON syntax! I couldn’t ask for more.

Note

With Weekly and Monthly, scripts will run of the 1st day of the week or of the month at midnight which is fine for the small scripts I have, but may run lots of big scripts at the same time on your setup.

The only pain point I see is when writing a new timer. Timers are divided in two parts: the timer itself which define when something must run and an associated service file to run something. It’s a bit tiresome to write, but since I don’t do it often, it’s not a very big deal. On the bright side, you can run the service without the timer and it will behave the same.

Now that the introduction is over, let’s look at an example.

Note

I’ll focus on user run services since that’s what I need, but if you save the files under /etc/systemd/system and omit --user from the commands, you can create custom and system wide timers and services.

All the config files must be under ~/.config/systemd/user/. Here is a sample .timer file to help you get started (for instance backup-stuff.timer):

[Unit]
Description="Weekly backup"

[Timer]
# Supports keywords like weekly and monthly and CRON syntax.
OnCalendar=weekly
# Run now if the scheduled run was missed.
Persistent=true
# Which service to run with this timer.
Unit=backup-stuff.service

[Install]
WantedBy=timers.target

And the accompanying .service file (backup-stuff.service):

[Unit]
Description="Sync server backup to the local machine"
# Specify dependencies as usual in systemd units.
After=network.target
# %n is the name of the current unit.
# This service will be called in case of failure with the name of the current unit
# so it can mention it in its error message.
OnFailure=notify-errors@%n.service

[Service]
# Should be a oneshot service: we start with the timer, the script runs and then stops.
Type=oneshot
# %h will be replaced by the home directory of the user.
ExecStart=%h/bin/backup-servers.sh

[Install]
WantedBy=default.target

If you want to run it outside the timer, you can do systemctl --user start backup-stuff.service. Even if you do that, the service won’t skip its next timer run. That’s very useful for debugging. And just like a normal run, the logs will end up in journald.

And now, the service to notify failures, saved in .config/systemd/user/notify-errors@.service:

[Unit]
Description="Notify errors to the user"

[Service]
Type=oneshot
# %i will be replaced by the parameter (ie what comes after @ when the service was called).
ExecStart=%h/bin/systemd-notify-errors.sh %i

And the accompanying script:

service=$1
# Get the last execution timestamp to filter the logs for this execution.
last_execution_timestamp=$(systemctl --user show --property ExecMainStartTimestamp --value $service)
# Extract the logs of this unit.
logs=$(journalctl --user --unit $service --since "$last_execution_timestamp")

# Display the error to the user.
# Notification is displayed during 5 minutes (in ms).
# This requires the libnotify-tools package.
notify-send --urgency normal --expire-time 300000 "$service failed" "$logs"

Don’t forget to refresh the systemd daemon with systemctl --user daemon-reload or you won’t be able to see your new services and timers.

All we have left to do is to enable the timer with systemctl --user enable backup-stuff.timer And we are done!

Note

User timers will only be executed for users that logs in by default. To always execute them when the machine is up even if the user don’t log in, enable lingering with sudo loginctl enable-linger USER

Resources:

systemd/Timers on the Archlinux wiki.
Working with ``systemd` Timers <https://documentation.suse.com/smart/systems-management/html/systemd-working-with-timers/index.html>`__ on the openSUSE documentation.

My take on UV and Ruff

2025-01-25T00:00:00+01:00

I recently tried the new and shiny tools made by astral. I only used them on my personal projects yet, but I’m still very impressed! You may already have heard of them. I’ll try to keep the article concise and won’t dig too deep into the tools to prevent this article to be outdated fast: these tools are under very active development and thus change quickly. The drawbacks listed here may not be a think any more when you’ll read it.

Ruff

ruff the linting tool to unite them all. It replaces flake8 and most of its plugins, pylint and your formatter (most likely black nowadays). It’s very feature complete with over 800 rules. And growing! It’s main selling point is: it’s blazing fast.

I was personally impressed by its speed both to validate a file and to reformat it. The difference of speed with flake8 is noticeable and huge from pylint (which was always slow).

Integration with pre-commit comes out of the box, so it should be easy to run it both locally and in CI. The documentation also notably mentions GitHub actions, Gitlab CI and Docker.

Formatting works great in Pycharm on save by configuring a save action. Linting doesn’t work out of the box (it might change, see this GitHub issue). But since PyCharm has good linting built in, I only encountered a few linting errors on pre-commit. So not a big deal for me.

It also has an official VSCode extension. I never tested it, so no opinions on that. Given its rating and usage, I guess it’s very good.

With any luck, it will also replace type checkers one day with a much faster alternative.

The only issue I see is if you have custom flake8 plugins: ruff doesn’t yet support plugins. But you could only rely on flake8 for that and use ruff for everything else.

uv

uv is a new package manager. It can replace pip and tools like poetry to manage your dependencies. It supports dependencies groups and lock files. Everything will be into your pyproject.toml file following the latest PEP. Sadly, dependabot is still not compatible with it, so no update notifications yet. I hope this comes soon.

It can also replace pipx to install and run Python commands. All you need to do is to install the tool with uv tool install flake8. You can then use the flake8 command as usual (provided ~/.local/bin is in your PATH). You can also download and run a tool once with uvx flake8.

Even better, you can add dependencies to your script and let uv create a venv and run the script in it with all its dependencies automatically! To do so:

Create your script.
Add the dependencies with uv add --script example.py DEPS
Run with uv run example.py

It can also download a Python interpretor if you don’t have it already installed on your machine. So if a script is only compatible with a version of Python you don’t have, uv will still be able to run it for you automatically.

Once again, with uv you don’t need anything else to run Python scripts and projects!

Conclusion

I’m very impressed by what they built and how fast they made it. I strongly suggest you test these tools. Given the vide of the community, you’ll probably use them too! And you’ll probably be able to replace all your flake8 and pylint workflows with them and make them run faster! And to run your script without a sweat and built in dependency management!

I also think the future of these tools will be great: they are open source after all, so the community should be able to take them back if needed. Given how used and liked they are, I don’t see them disappear. The only thing that could be an issue: astral is a VC backed company, so they will have to make money somehow at some point. No idea what form it will take though.

And you, do you have any thoughts on these tools? Did you use and like them?

Weird test behavior in my Django project test suite after an IntegrityError

2024-12-01T00:00:00+01:00

Recently I encountered a very weird behavior in my Django project test suite. I created a view that caught an IntegrityError from the database (caused by duplicates in a unique index). When this error occurs, I want to respond with a 409 - CONFLICT status code and an error message.

TL;DR: it’s caused by how transactions works in the test suite. Using a transaction.atomic() on the view solved it. Read on to learn more.

Everything worked fine in the browser. But in the test suite, I always got a 400 - BAD REQUEST error with the default HTML error page as content. Not my custom status nor custom JSON. Worst, if I tried to log the queries, django_assert_num_queries listed none. Yet, I was sure they were executed. And if I launched the debugger, my view correctly caught the IntegrityError and responded with the proper response and the proper status.

Note

I’m using pytest, but if I’m right, it doesn’t have an impact on the problem nor on how to solve it.

I had no clue what was going on. Luckily, I had to do the same modification (catch some IntegrityError) in two other views as well. I wandered if I’d always get the same behavior. So I updated the second view and got the same issue. I moved on the the third view and it worked as expected!

At least I now had a comparison point to start investigating. The main difference was that my third view was decorated with a transaction.atomic(). The other weren’t.

So it hit me: each test functions is run in a transaction. This way, we can easily and quickly rollback all the changes made by test test when it ends (successfully or not). So, when my IntegrityError hit, it caused a rollback to happen at test level before the test completed. That’s why django_assert_num_queries couldn’t list the queries and what caused me not to have access to the proper response. I don’t know why in this case the response is always a generic 400 and not a more specific error message though.

After testing with my existing views by adding/removing transaction.atomic(), I was able to confirm that I only got the correct behavior when transaction.atomic() was present. So I updated the view, made my test suite pass and went on to write this blog post!

In the end, I think I was lucky to have the need to update a view wrapped in transaction.atomic(). Without this basis to compare, I think I’d have search way longer what was going on and why the problem was even occurring!

Offline support almost without Javascript

2024-02-27T00:00:00+01:00

Recently I wandered wether I could build a website with offline support without building a full SPA. The answer is yes it’s doable: you only need Javascript for the service worker. Just for the fun, I also tried it with navigation done with HTMX and without much surprise it also works.

Warning

To do these tests, you must use a small webserver to serve your content. I used python3 -m http.server.

If you use Firefox, you must shutdown the server to be offline. If you go offline in the dev tools (for some reason I don’t know), your navigation won’t be handled by the service worker and you will appear truly offline.

Let’s start by building a simple index.html page:

 1 <!DOCTYPE html>
 2 <html>
 3     <head>
 4         <base href="/">
 5         <script>
 6             if ('serviceWorker' in navigator) {
 7               window.addEventListener('load', () => {
 8                 navigator.serviceWorker.register('/service-worker.js')
 9                   .then(registration => {
10                     console.log('Service Worker registered with scope:', registration.scope);
11                   })
12                   .catch(error => {
13                     console.error('Service Worker registration failed:', error);
14                   });
15               });
16 
17               navigator.serviceWorker.ready.then((registration) => {
18                 fetch('/page2.html');
19               })
20             }
21           </script>
22           <!-- <script src="/htmx.min.js"></script> -->
23     </head>
24 <body>
25 
26 <h1>Home page</h1>
27 
28 <p>My first paragraph.</p>
29 
30 <ul>
31     <li><a href="/">Index</a></li>
32     <li><a href="/page1.html">Page 1</a></li>
33     <li><a href="/page2.html">Page 2</a></li>
34 </ul>
35 
36 </body>
37 </html>

The HTML part should be straightforward. In the script tag, I register my service worker (lines 7 to 15) and prefetch a page once it is ready so it’s in the cache (line 17 to 19). The rest is just code to navigate between the different pages and a title to identify the page.

I build two other page by copy/pasting this into page1.html and page2.html and changing the h1. I then created my service-worker.js:

 1 const CACHE_NAME = 'my-cache-v1';
 2 const urlsToCache = [
 3   '/',
 4   '/page1.html',
 5 ];
 6 
 7 self.addEventListener('install', event => {
 8   event.waitUntil(
 9     caches.open(CACHE_NAME)
10       .then(cache => {
11         return cache.addAll(urlsToCache);
12       })
13   );
14 });
15 
16 self.addEventListener('fetch', event => {
17   console.log('Captured fetching', event.request)
18   event.respondWith(
19     caches.match(event.request)
20       .then(response => {
21         if (response) {
22           console.log('Serving from cache')
23           return response;
24         }
25         console.log('Fetching from server')
26         return fetch(event.request)
27           .then(fetchResponse => {
28             return caches.open(CACHE_NAME)
29               .then(cache => {
30                 cache.put(event.request, fetchResponse.clone()); // Add the fetched response to the cache
31                 return fetchResponse;
32               });
33           });
34       })
35   );
36 });

In it, I add some content to the cache by default (lines 7 to 14).

I then register a listener to the fetch event so I can choose how to respond to HTTP requests (lines 16 to 36). I choose to serve the response from the cache if I have it. If not, I fetch it then put it in the cache for later usage.

It’s very basic, but it’s enough to make it work. You can also test it by navigating here.

Note

page2.html (fetched from the main page) won’t be put into the cache until you refresh the page. From what I understood, it’s for consistency: never serve from the cache until you are sure the service worker is active and working.

You can now test the app and see how it behaves.

To test with HTMX, I replaced the ul (lines 30 to 34) with either:

<!-- Test with HTMX boost to use standard links -->
<ul hx-boost="true">
    <li><a href="/">Index</a></li>
    <li><a href="/page1.html">Page 1</a></li>
    <li><a href="/page2.html">Page 2</a></li>
</ul>

or:

<!-- Test with HTMX links to use standard links -->
<ul>
    <li><a hx-get="/" hx-push-url="true">Index</a></li>
    <li><a hx-get="/page1.html" hx-push-url="true">Page 1</a></li>
    <li><a hx-get="/page2.html" hx-push-url="true">Page 2</a></li>
</ul>

That’s it. The prefetching is very basic, but I think you get the idea: generate a list of URL and then call them.

If you want to learn more about a related topic, I wrote an article about PWA and Django a while back.

Development containers

2024-01-02T00:00:00+01:00

I recently discovered the dev containers standard (or development containers for long) recently after trying to contribute to a project which had them enabled by default. It seems to be a new standard way to work with containers in development.

The goal is to provide a container you can use as a full featured environment for development. Your editor and its terminal are connected to the container and you run everything in it. So instead of running some things locally or wrapping everything in docker compose exec to run them on the container, you run everything directly in the container thanks to your editor. With VSCode, you can even personalize the container to install your favorite shell and theme (in my case, I use ZSH or Fish with starship), use your customized git configuration as well as any extra tools you might need.

It is compatible with plain docker as well as docker compose and supports VSCode and Pycharm (currently in beta in Pycharm). According to my tests, it works very well with VSCode and will be picked up automatically if you have the proper extensions installed (see the getting started section of the doc which is really good) and is still rocky in Pycharm (it even broke for a while after a Pycharm update). Once it supported correctly, it means no matter what editor you use (provided it supports the standard) you will have the same experience and behaviors. And for Pycharm, it means you can use it inside a container without the usual slowness regarding container creation for every command you launch.

I think it’s a very good idea and that we should use it more. It does come with a few issues if you use user namespaces with Docker like me. Let’s dig deeper for that use case.

User namespaces

The goal of user namespaces is to map a user inside the container to another user outside. I’ve explored this extensively in this article. For instance, I map the root user inside the container to my user outside. This way, all files created by root in the container will belong to me outside. It makes sharing files on volumes much easier on Linux where there is no abstraction layer to do UID conversions automatically. It also brings extra security: root inside the container is a mere user on the host. If you don’t run the container with the root user, files will belong to a weird UID outside the container.

Let’s illustrate this with an example. Let’s take this Dockerfile:

FROM debian:latest

RUN apt-get update && \
    apt-get install -y passwd

RUN groupadd --gid 1000 dev-user \
    && useradd --uid 1000 --gid dev-user --shell /bin/bash --create-home dev-user

With user namespacing enabled in the Docker daemon, build the image with docker build -f Dockerfile --tag test:latest . then run it with docker run -it --rm test:latest

If you look at permissions in the home folder or /usr/bin/bash you should see something like this:

root@435ec3552812:/# ls -alhn /home/
total 0
drwxr-xr-x 1    0    0  16 Dec 24 13:24 .
drwxr-xr-x 1    0    0 174 Dec 24 13:24 ..
drwxr-xr-x 1 1000 1000  54 Dec 24 13:24 dev-user
root@435ec3552812:/# ls -alh /home/
total 0
drwxr-xr-x 1 root     root      16 Dec 24 13:24 .
drwxr-xr-x 1 root     root     174 Dec 24 13:24 ..
drwxr-xr-x 1 dev-user dev-user  54 Dec 24 13:24 dev-user

Now, rerun with: docker run -it --rm --userns host test:latest to disable user namespacing for this run. You should see something like this:

root@565d6ae60c50:/# ls -alh /home/
total 0
drwxr-xr-x 1 dev-user  dev-user   16 Dec 24 13:24 .
drwxr-xr-x 1 dev-user  dev-user  174 Dec 24 13:26 ..
drwxr-xr-x 1 100000999 100000999  54 Dec 24 13:24 dev-user
root@565d6ae60c50:/# ls -alhn /home/
total 0
drwxr-xr-x 1      1000      1000  16 Dec 24 13:24 .
drwxr-xr-x 1      1000      1000 174 Dec 24 13:26 ..
drwxr-xr-x 1 100000999 100000999  54 Dec 24 13:24 dev-user

Root became user with UID 1000 (including for already present files like /usr/bin/bash) while user became something non existent (the remapped UID used during the build). Before, inside the container everything was correct including for files like /usr/bin/bash which was correctly owned by root with UID 0 inside the container.

Disabling user namespacing

Let’s now disable user namespacing in the Docker daemon and retry: docker run -it --rm test:latest. You should get something like this:

root@ed2f0d20f197:/# ls -aln /usr/bin/bash
-rwxr-xr-x 1 0 0 1265648 Apr 23  2023 /usr/bin/bash
root@ed2f0d20f197:/# ls -alh /usr/bin/bash
-rwxr-xr-x 1 root root 1.3M Apr 23  2023 /usr/bin/bash
root@ed2f0d20f197:/# ls -alh /home/
total 0
drwxr-xr-x 1 root     root     16 Dec 24 13:30 .
drwxr-xr-x 1 root     root      0 Dec 24 13:31 ..
drwxr-xr-x 1 dev-user dev-user 54 Dec 24 13:30 dev-user
root@ed2f0d20f197:/# ls -alhn /home/
total 0
drwxr-xr-x 1    0    0 16 Dec 24 13:30 .
drwxr-xr-x 1    0    0  0 Dec 24 13:31 ..
drwxr-xr-x 1 1000 1000 54 Dec 24 13:30 dev-user

So everything is fine. But, files created with root in the container will be owned by root outside. Files created in the container will be owned by whatever user has UID 1000 on your system (most likely you).

Problems with VSCode and devcontainers

By default, VSCode uses the root user inside the container: you will perform all your operations as root whether user namespacing is enabled or not. With user namespacing on, it’s not a big deal since everything will be mapped to the correct user. Without it, everything belongs to root which is a pain. You are also doing all your actions as root which is still weird and probably a bad idea since it may not be realistic to enforce user namespacing to all members of your team (without even thinking about open source project).

You can setup VSCode to use a different user. That will work fine without user namespacing. But with it, you UID outside the container will be whatever UID user namespacing attributed to you. So you won’t have read access to the files (unless you open read access for everybody but that’s just another bad idea). So you will be stuck and won’t be able to develop.

You could ask VScode to run the containers with the --userns host option (it’s also doable with docker compose by adding userns_mode: host to your service definition). But, since the user will be added during the build step and you can’t build your image with user ns disabled (or if it’s possible, I haven’t found how), the files of the newly added users won’t belong to it as we’ve seen earlier and VSCode won’t be able to install its server.

It turns out, VSCode has a nice feature: updateRemoteUserUID. It’s on by default and will change the UID of the user of the container (unless it’s root which must stay with UID 0) to the UID of your user. So, you can transparently create file inside or outside the container they will have the correct UID! Sadly (and despite what the doc is saying) it doesn’t seem to work for GID which will still be whatever it is in the container. But it’s still a huge step forward.

Here is the docker file I used (inside a .devcontainer folder):

FROM debian:latest

RUN apt-get update && \
    apt-get install -y passwd git

# Let’s use something that proably doesn’t exit on the system.
RUN groupadd --gid 5000 dev-user \
    && useradd --uid 5000 --gid dev-user --shell /bin/bash --create-home dev-user

RUN mkdir -p /app

USER dev-user

And the devcontainer.json (also inside the .devcontainer folder):

{
  "build": { "dockerfile": "Dockerfile" },
  "workspaceMount": "source=${localWorkspaceFolder},target=/app,type=bind",
  "workspaceFolder": "/app",
  "updateRemoteUserUID": true
}

Let’s recap:

Enabling user namespacing: besides using root as a user for all your actions, I don’t think it’s usable. And it’s probably a bad idea (you wouldn’t do it on your host system after all) and for your team mates that don’t have user namespacing enabled it will create a security issue since they are also running everything as root on the host.
Disabling user namespacing and using a dedicated user: thanks to VSCode and its updateRemoteUserUID feature everything will work correctly out of the box. I still think it’s better to explicitly set the UID and GID to 1000 when building the container: it’s probably the right value. Since the GID isn’t updated, it’s a very good default and will hide the internals away for most users.

Wrapping up

I’ve been a huge proponent of user namespacing for years since it makes everything go more smoothly on Linux while increasing security. It doesn’t seem to work well with dev containers. Since the spec allows you to remap the UID of a standard user inside the container to your UID outside, it’s not a very big deal.

I think from now on, I’ll only enable user namespacing if I really need it and launch all my dev works in containers as a standard user letting my IDE doing the UID remapping if needed.

I also think I’d leave user namespacing on in my production environment so that if an attacker gains root access in a container, they don’t have root access on the host.

And you, what do you think? Do you know more about devcontainers? If so, please leave a comment below.

Django async

2023-12-10T00:00:00+01:00

Now that Django is fully async (views, middleware and ORM), I though it was a good time to test how it behaves when run asynchronously. I’ll try to keep this article concise with only relevant data and resources. Code can be seen in a sample project so you can check the code and go further if you want. I won’t explain it, but I think it’s simple enough to be easy to understand if you already know Python and Django. I also provide a synthesis and conclusion at the end of the article. I checked the most common solutions to serve a project: the classic (and included in Django) runserver (dev only), gunicorn, uvicorn, daphne and hypercorn.

Table of contents

Running a basic view
Sync vs async behavior
HTTP2 support
- How about HTTP2 PUSH feature?
Load testing
Wrapping up!
- My recommendations

Running a basic view

I started by checking the behavior of a basic view: it renders a template and loads a CSS file. I wanted to check how each solution behaves when the template and the static file are updated and whether everything is served correctly. This is mostly to check the behavior of each solution during development: in production, you won’t update your templates on the fly and will rely on something else to serve your static assets. I think it is still interesting and useful if you want to be as close as possible from your production environment in development.

runserver: as expected, everything went smoothly, the static file is served and when the template is updated a simple page reload allowed me to see the newest version.
daphne: when launched directly, the template modification is not available until I do a restart and the static file is not found. It’s not surprising since allowing this is a feature of runserver for ease of development.
gunicorn: same as daphne.
uvicorn: same as daphne.
hypercorn: same as daphne.

What’s interesting is that you can change this default behavior with command line options for development:

daphne: you can install it as an app in your Django project. It will then be picked up by runserver. So runserver will behave like an ASGI server while still being able to serve static files and to see template modifications immediately. You can make sure daphne is used if you see Starting ASGI/Daphne in the startup logs.
For all other servers, you will need to rely on Whitenoise to serve static assets and use their reloading option to update when source code change and their watch extra files one to restart when HTML files are updated. Please note that during my test, reloading in hypercorn seemed broken.

Regarding template reloading, you could also choose to disable template caching to always use the latest one. See this article for more.

Note

If you are in an async view, you must use only async operation and wrap sync operations (like a call to the ORM) in sync_to_async. You will get errors if you don’t. Same goes in reverse with async_to_sync. You can make sync and async views cohabit without any issues whether you serve your project on WSGI or ASGI. You won’t really benefit from async on a WSGI serve app though.

Note

You can hook into the signal autoreload_started to make runserver restart on any file you want. But it’s not documented and thus may break at any time. See here for more.

Relevant commits:

Do basic tests with API & template views
Setup daphne
Test with model
Test app servers (includes script to launch each servers in a dev like and prod like fashion).
Use whitenoise for static file serving (I basically followed the tutorial).

Sync vs async behavior

To spot any differences between sync and async behaviors, I created a very simple view that returns JSON. I then sleep 10s with time.sleep or asyncio.sleep.

runserver: I passed the --nothreading option to avoid having multiple threads that could handle the requests simultaneously. By default, if I launch two requests in parallel, the first completes in 10s and the second one in 20s. So they are handled one after the other. The fact that no, one or both requests are made to an async view doesn’t change a thing. That was what I was expecting. I’ll call this the fully sync behavior.
daphne: both requests always ends after 10s. Even if I target two times the sync view. I suppose it is using sync_to_async to run the non async views, which, as far as I know, will make it run in a thread. Using it directly or through runserver doesn’t change the behavior. I’ll call this the fully async behavior.
gunicorn: as expected, I get the fully sync behavior.
uvicorn: as expected, I get the fully async behavior, whether I launch it directly or as a gunicorn worker as suggested in the documentation for production environment.
hypercorn: as expected, I get the fully async behavior.

Relevant commits:

Getting more serious with async: using the StreamingHttpResponse

The StreamingHttpResponse is not new and allows us to stream a response, ie instead of sending it in one go you send it chunk by chunk. The use case for this in the documentation is to send a big CSV file. As the documentation points out, in WSGI, you will need a worker for the whole duration of the response. This worker won’t be able to serve any other clients. That’s where ASGI really comes handy: your worker can still server clients while it is waiting on IO. Let’s test this.

I created two new views to test this: one sync and one async. I used HTTPIE like this to view the streaming: http 'http://localhost:8000/stream' --stream (sync stream) and http 'http://localhost:8000/astream' --stream (async stream).

When using gunicorn I can serve both views. The sync one is correctly streamed. The async one responds but all in one go (ie without any streaming). And I got this warning: StreamingHttpResponse must consume asynchronous iterators in order to serve them synchronously. Use a synchronous iterator instead. It’s consistent with what the doc says:

When serving under WSGI, this should be a sync iterator. When serving under ASGI, then it should be an async iterator. […] Under WSGI the response will be iterated synchronously. Under ASGI the response will be iterated asynchronously. (This is why the iterator type must match the protocol you’re using.)

When using an ASGI server, I got the reversed behavior: the async view streamed its content while the sync one didn’t. And I got this warning: StreamingHttpResponse must consume synchronous iterators in order to serve them asynchronously. Use an asynchronous iterator instead (except for daphne which for some reason didn’t print anything).

Relevant commits:

Test StreamingHttpResponse

Going further with async: Server Sent Events (SSE)

Instead of just stream data, how about streaming it to a browser and allowing the browser to react? This could be handy to notify the browser of some changes (like new data being inserted). Like websockets, the client could see the update immediately. Unlike Websockets, communication is unidirectional: from the server to the client. But it should be enough for many use cases and doesn’t require any extra lib.

This is done with StreamingHttpResponse and an async iterator.

How to get notified of events? You could use PostgreSQL directly thanks to its listen/notify feature or rely on the pub/sub feature of Redis.

There are several things you must pay attention to:

Each message must be ended with two line breaks.
The data must start with data: or you won’t be able to access the data in JS. So your payload must be like f"data: {data}".
The content type of your response must be "text/event-stream".
You can then create an EventSource in JS and parse each event data property.

For more details on this, please read this article.

Relevant commits:

Test server sent events

How about Websockets?

To use Websockets, you still need to use Django Channels. It works on all servers except gunicorn (you need ASGI for this, no work around or compatibility with async_to_sync this time!).

It’s very easy to set up. To test it you don’t even need Redis (the only supported channel used to dispatch messages to all consumers) and can rely on a channel that works in memory. I only followed the official tutorial.

Relevant commits:

Test Websocket with channel

HTTP2 support

HTTP2 is only supported by hypercorn out of the box. For daphne you need to install two new packages: one for HTTP2 and one TLS. That’s because daphne only does HTTP2 through TLS. Since your browser won’t open a HTTP2 connexion unless it’s under TLS it’s not a big deal. gunicorn doesn’t support HTTP2 and uvicorn decided not to add it because they are alternatives (hypercorn and daphne as well as using a good old reverse proxy like nginx or Apache in from of the ASGI server).

It worked fine with both hypercorn and daphne.

Note

To test an HTTP2 connection, you can use curl with its --http2 option or Firefox (as far as I know Chrome doesn’t display the HTTP version of the connection).

Note

To test this in Firefox, I had to generate self signed certificates. I used the method described here. See the script for how to launch hypercorn and daphne with HTTP2 and certificates.

Relevant commits:

Setup HTTP2 for daphne

How about HTTP2 PUSH feature?

When HTTP2 was launched I clearly remember that push was the feature everybody was exited about. I never bothered to dig and enable it, but it sounded compelling: you could push assets to the browser before it even asked for it to load them faster! This test sounded like the perfect time to give it a try.

And… it turns out Chrome removed it a couple of years ago and nginx deprecated it in June 2023 in version 1.25.1 (the directives have no effect now, but don’t yet trigger an error). Ouch!

It turns out that it’s not easy to use, makes caching a lot harder and can lead to needless resources being pushed or the same resources being pushed more than once. For more, please read this article.

I still decided to test it for the sake of it. I made it run easily with Apache. I’ll extend a bit since it not obvious:

I used the default httpd.conf file from the container as a basis.

Then I added this at the top of the file:

Listen 80
LoadModule http2_module modules/mod_http2.so
Protocols h2 http/1.1

LoadModule ssl_module modules/mod_ssl.so
SSLEngine on
Listen 443
SSLCertificateFile "/var/certs/server.crt"
SSLCertificateKeyFile "/var/certs/server.key"

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
ProxyPass /static !
ProxyPass "/" "http://172.17.0.1:80/"
ProxyPassReverse "/" "http://172.17.0.1:80/"

<VirtualHost *>
    DocumentRoot /var/www/html
    <Directory "/var/www/html">
        Require all granted
    </Directory>
    ServerName host
    ServerAlias *
</VirtualHost>

I then updated my view so to would add a Link header listing the resources to push:

response[
    "Link"
] = "</static/home.css>; as=style; rel=preload, </static/list.css>; as=style; rel=preload"

You can see in Firefox in the dev tools: the CSS files are loaded but not associated request is displayed in the explorer.
You can test it more easily with nghttp a CLI tool dedicated to testing HTTP2 connections. To do so, once you’ve installed it, you can launch it like this: nghttp -ans https://localhost:8080/async. It will list the resources loaded by the page with an asterix if they were pushed.

I never managed to make it work with hypercorn though with the same settings. Since it is being deprecated, I didn’t dig any further.

Relevant commits:

Test with reverse proxy for benchmark & HTTP push <https://gitlab.com/Jenselme/dj-test-async/-/commit/42fa4713047291716637f3eb99709faa10b392d9>

Load testing

After testing all that, I decided to do some load testing to see how everything behaved. I did all my testing with h2load which comes with nghttp since it supports load testing with HTTP1 and 2. I won’t post the detail results and, as any benchmark made by an amateur in non real conditions, you should take it with a big grain of salt.

I tested on HTTP2 (always with TLS) and with HTTP1 (with and without TLS). When I used a reserve proxy, daphne in HTTP1 was always the application server. I tested with 1 concurrent connection, 200 and 500.

Here is a summary of my results:

Adding HTTPS slows the app. This is mostly seen when comparing HTTP1 with HTTP1 + TLS. No surprises there: TLS has a cost. Letting a reverse proxy handle TLS removes this slowness.
In this test, the app was slower when server in HTTP2, whether directly on in front of a reverse proxy. I guess HTTP2 has an extra cost (in addition to TLS). Relying on a reverse proxy, just like with HTTP1, greatly improved performance (note that the proxy and daphne communicated in HTTP1).
I got some errors (between 25% and 44%) when using HTTP2 with 500 concurrent clients. I got more error with Apache than with raw daphne. I think this case is very theoretic anyway since you wouldn’t have this in on a real production environment: your app would be much more complex and you would probably have more workers to process the load thus preventing this issue.
I found no performance difference between all servers. Nor between sync and async views.
I got errors with hypercorn in HTTP2. I don’t know why and it seemed to work fine in a browser. I didn’t try to dig any further.
I also got lots of errors with nginx in HTTP2. A quick search yielded this result so it may be a protection against attack. I tried some configuration to prevent this without success.

Relevant commits:

Wrapping up!

Let’s summarize what I’ve learned so far:

All solutions can serve both sync and async views quite efficiently. But you can only really benefit from async on ASGI and most notably from websockets and SSE.
All solutions recommended in the Django documentation seem mature and performant.
All solutions can be used in development.
You probably still want a reverse proxy to handle HTTPS and HTTP2 connections. Letting something else will degrade performance quite a bit. But if you can’t or don’t want to, it’s not an obligation.

Synthesis
	HTTP2	SSE	Websockets	Usage in dev
Daphne	✅ (with extra deps, TLS only)	✅	✅	⚠️ (the easiest to integrate with Django but harder to watch for extra files)
Gunicorn	❌ (not a very big deal if you have a reverse proxy anyway)	❌ (requires ASGI)	❌ (requires ASGI)	✅
Uvicorn (with Gunicorn in prod)	❌ (not a very big deal if you have a reverse proxy anyway)	✅	✅	✅
Hypercorn	✅ (couln’t make server push work)	✅	✅	✅ (had an issue with reloading, but should work)

My recommendations

After all that, what would I recommend?

If you don’t need async and thus ASGI, you can probably stick with your current stack. It’s solid and won’t go away.
I’d still put a reverse proxy in front of my app, even for ASGI.
For a pure ASGI project, I think I’d use daphne and install it as an app. I’d to it because it’s the easiest to integrate with Django, including the runserver command I’m used to having during development. It also makes static files serving and template changes easier out of the box.
uvicorn under gunicorn looks like a very good alternative. And you benefit from all the options of gunicorn in production.
I’d reach for hypercorn only if I needed all of its features (like the ability to use uvloop or its experimental HTTP3 feature).
If you are in the process of migration to async, I think you should start by running "old" views under WSGi and let a reverse proxy route traffic to an ASGI server for async views. Once most of your app is migrated, I think you can switch to ASGI and let is serve your sync views until you change them (or for the end of time because you’ll never have time to migrate something that works).

Having said all that, I’ll gladly hear what you think in the comments!

Writing RSS reading app with various frontend frameworks

2023-09-09T00:00:00+02:00

During the summer, I decided to test a few frontend framework to see what’s going on in this space and form a better opinions over alternatives to React. I tested Svelte because after hearing from it I felt attracted to it, Vue because it is popular, React to have a good comparison point and Angular because I was about to take a new job that required Angular. I also included Aurelia, the framework I first used to create the app I used to test all the others.

To do these tests, I decided to code a RSS reading app: you connect on the first page and you are then redirected to the list of articles and categories. You see the articles of the current category (the unread ones by default), you can switch categories, mark articles as (un)read, read articles on scroll and on swipe.

Table of contents

My tests
- Aurelia
- Svelte
- VueJS
- ReactJS
- Angular
- Summary
Conclusion

My tests

Aurelia

Aurelia is a complete frontend framework I used on some of my personal projects. The framework is really good although a lot less known than the alternatives. From my experience, the community is small but very active. The main drawback I experimented was the lack of good tooling (things may have changed, I haven’t used Aurelia in a while).

The framework has pretty much everything you may need: a way to define components, dependency injection, a HTTP service, a logger… You can also use libraries that manipulate the DOM directly with it.

When I created the first app years ago, I had a good experience using the framework. I can’t really say more today. Please note that, just like with angular, the framework is big and designed to create apps, not enhance a page and that the resulting bundles can be big.

Last but not least, the app is rather big: 1,764 lines of code (1387 of TypeScript, 231 HTML). Part of this is explained by the fact it has features no other apps have (like offline support with a Service Worker).

Svelte

Svelte is a relatively new framework. You can use it as a library to enhance your pages or as a framework (with SSR, routing, stores) thanks to svelte kit. It has lots of features and a good community.

It’s the first framework I used during my tests. I found it easy to handle. What I specially liked is the way you define components: everything goes into the same file, you use a good templating language for your HTML that reminds me of the Django one and you define your CSS for just your component. Overall, it feels like you are using standard tech. Same goes for the stores: it feels like I was using JS and you can easily use them outside Svelte if needed (although reading values isn’t easy)!

I decided to use Skeleton to help me style the app. I think it was either a wrong choice or that I needed to configure it better to have something more pleasing to the eye.

What I found interesting is that the framework also help you define animations on elements. It help me a great deal to do the read on swipe feature correctly. Beware though, by default, it also tried to animate all the articles when I switched categories which created slowness and bugs (we didn’t really change the page). Understanding and fixing this took me a long time (but it occurred again with other frameworks, so it’s not a Svelte thing).

I didn’t like how reactive statements work: dependencies are implicit and I encountered a problem where a statement was run each time an object changed while I was only concerned with one of its properties.

I also found strange that I had to run two commands to compile the app and have all the errors.

Last but not least, the resulting app size is small: 1,317 lines of code (450 lines of Svelte, 978 lines of TypeScript).

VueJS

Vue is the main competitor to React. It’s a rendering lib you can extend to have all you need (router, store…). Most of these "extensions" are maintained by vue.

In terms of structure of a component, it’s very similar to Svelte. The way it handles state is very similar to React while being less verbose.

When I tried to add translations, I choose vue-i18n which seemed like a popular choice. But I couldn’t extract strings automatically. Using FormatJS which is compatible with both React and Vue might have been a better choice (although I never succeed to make string extraction work with it).

I think that watchEffect works a bit better than in Svelte, even if they also rely on automatic dependency discovery.

I also found strange that I had to run two commands to compile the app and have all the errors.

I didn’t do read on scroll nor read on swipe: I found no easy helpers to help me and wanted to spend time on other tests.

Last but not least, the resulting app size is small: 1,165 lines of code (444 lines of Vue, 717 lines of TypeScript). I’d say on par with Svelte given it’s a bit smaller but I have less features here.

A tiny thing I found strange: with the default rules, semi-colons are not used. I don’t think I encountered any other default with this rules.

ReactJS

React is probably the most popular one. It’s a rendering lib you can extend with lib maintained by third party to meet all your needs.

Here you are forced to use JSX to define your components. I have always disliked it since I think it forces you to do React things to define your view and your behaviors. On the bright side, you can define multiple components in the same file and create many tiny reusable components.

The hooks system to perform side effects or have cached values is very verbose but also very explicit about dependencies. It has pros (it’s obvious and you can manage them by hand if needed) as well as cons (it’s verbose, easy to forget without a linter, if some deps are missing you must document why).

Material UI is an astonishing component library: complete, well documented and easy to use. It’s the best component library I used.

One the good side, you can view all errors in one command.

I found a helper lib to help me implement the read on swipe.

I also tested for the first time the new features of Redux: async thunks. I think that having this included out of the gate is a good thing as we won’t need to rely on external libs to perform something that we almost always need to do. Redux is still very decoupled in different concepts: it makes it very clean and also very verbose.

At last but not least, the app is rather big: 1,684 lines of code (630 TSX, 1219 TypeScript).

Angular

Angular is a big batteries included framework. It’s the one that has the big enterprise touch and the one where you don’t really need to install anything but Angular libs to achieve your goals.

It contains everything you need from the HTTP router, Angular Material to help you design your UI, Service worker service, form validation and building… to dependency injection. It also comes with more concepts: components, directives, services. Like Aurelia, it is centered around classes whereas other framework are more centered around functions. By default, a component is split into 3 files: the TypeScript code, the HTML and the CSS.

Its binding system is also more complex: while in all other frameworks you can bind anything, here you must bind data and event (to provide output to your component) in a different manner. If you just bind a function, call it won’t refresh the state of your app.

You will also need to learn and use RxJS since it is heavily used by the framework, including its HTTP service. It’s a world of its own that will take you time to get used to (let alone master), will increase the size of your bundles but is very powerful and even pleasing to use once you get the gist of it.

During development, I found that it required more full page reloads than any other solution. It’s a pain since it can make you loose context and may force you to reenter data/reopen modals.

While it includes a translation system, it was way too complicated for my needs. I felt it was tailor for big enterprise usage. Luckily, I could use another library that was more aligned with my needs.

At last but not least, the app is quite big. I had to do some improvements to limit the size. It’s better that the raw one, but still big. I have 1,859 lines of code (1,590 TypeScript, 286 HTML, 132 CSS).

Summary

Note

To run the size tests, I always used a full reload on each page.

Framework	Link to project	My opinion	Lines of code	Connection page size	Articles page size
Aurelia	aurss	Good framework, not really known/used. Tooling could be improved.	1,764	37 requests to load 1.57MB	39 requests to load 1.66MB
Svelte	svelte-rss	Very good framework, pleasant to use, modern and lightweight.	1,317	18 requests to load 235KB	18 requests to load 240KB
Vue	vue-rss	More or less the same as Svelte. After these small tests, I’d favour Svelte over Vue.	1,165 (but some features are lacking)	7 requests to load 220KB	9 requests to load 231BK
React	react-rss	Solid framework, good community, possibilities to make native app, very good components libraries. Shows its age.	1,684	7 requests to load 528KB	8 requests to load 530KB
Angular	ng-rss	Complete framework, heavy to use and load. Overall good dev experience even if it’s verbose.	1,859	8 requests to load 1.04MB	7 requests to load 1.04MB

Conclusion

I enjoyed testing these frameworks and dive into their differences. I loved Svelte the most: code is pleasing to write, I like its architecture and conventions, the community is big enough so you shouldn’t have any problems (or if you do, someone should be able to help). I’d probably pick it to start my next project if I were starting from scratch.

Vue seemed nice too. But a bit stuck between Svelte and React, like I could have part of the joy of Svelte while still having to endure a bit of pain from React.

React is my least favorite by far. I’ve had a long and complex relationship with React, always failing to understand the hype around it (and more recently the hype around the hooks). I only used it because my job required me to. I was glad to see I could get exited by a frontend tech. For more on this, I invite you to read two articles by Josh Collinsworth who expressed quite eloquently this feeling I have: The self-fulfilling prophecy of React and Things you forgot (or never knew) because of React.

Angular is its own beast. If you work in a big enterprise, it’s probably the best choice for you. I think it can be pleasing to use, but it’s also complex and verbose.

In a nutshell

There is a world outside React.
You should probably checkout Svelte or Vue if you haven’t already.
Keep a very close eye on web components: they are very likely to be the future. In fact, whatever you are using and whatever your opinion is (or just because you have little time and cannot checkout many things), just keep an eye on that. You can read this article to get a good start.

My opinion after testing some AI code assistant

2023-08-21T00:00:00+02:00

With all the hype around AI and since I had time to spare, I decided to test some AI coding assistants to make my own opinion about them. I'll start by giving my opinion on each assistant I tried. I will be a bit fuzzy since I didn't intend to write this blog post and didn't take as many notes as I should have. Then I'll wrap up with my general opinion and some advices.

I ran most of my tests writing a small task manager in React, VueJS, Svelte and Elm. I tried to switch between all of them between each to view the differences. I tried CodeWhisperer last on a dedicated test in TypeScript and in Python, not on the task manager project. I ran the same tests with Codeium to have a comparison point. I mostly used VSCode for my tests and use Codeium a bit with PyCharm.

Tested code assistants

AWS CodeWhisperer

AWS CodeWhisperer is AWS take on the subject. It's tied to your AWS account and AWS extension for your editor. It's free for individuals.

According to my tests, it's the worst overall. On the bright side, it did generate a few test cases that were more relevant to my use case than other tools and can handle imports automatically. But it suggested any way too often in its TS completions, proposed less variations between suggestions, proposed more irrelevant suggestions, proposed more complex suggestions and relied too often on line to line suggestion (versus block/function suggestion for other assistants). It's also way slower and felt unresponsive at times.

It's also the only tool that proposes a security scanner. I tried in on a piece of Python code and its suggestions were good: I was lacking a context manager and doing a SQL injection which the tool spotted. It was very slow, even on my small file (~10 lines of code). I don't know how it behaves nor what it reports on larger and more complex pieces of code.

To conclude: It's definitely a pass for me. In this current state, it's also a pass if you are tied to AWS product. Just code without, you should have a better time. The security scanner can be a good thing if it works properly on large code bases though.

GitHub Copilot

GitHub Copilot is you guessed it made by GitHub. It's linked to your GitHub account and requires a dedicated extension in your editor. It's not free even for individuals, but comes with a 1 month trial period (that's what I used).

According to my tests, it proposed the best suggestions of all tested tools. It's even good at not messing up too much parentheses and curly braces completions in existing code (where all other tools had some/more issues). It's also fast at responding. It does the job and does it well (for an IA assistant).

There is also a chat in beta. I couldn't try it: it's only opened for business accounts right now. But, from my experience with codeium it should be a valuable additions.

Codeium

Codeium is a product made by a startup. It requires an account (you can use your Google account) and a dedicated extension in your editor. The VSCode extension is the most complete since it supports all features (completion and chat), but I tested it in PyCharm and code completion work fine (I lacked the chat though). You can test it in their playground before creating an account nor installing anything. That's a good thing in my opinion. It's free for individual use. It's also the tool I used beyond my small IA assistants tests (because it's good, the chat is quite handy and it's free).

It's suggestions aren't always the most relevant and it tends to fail for parentheses/curly braces completions. When generating test cases, I had to guide it to have something good. It invented a method named toBeUpperCase and in a forEach loop instead of using the item it used the array and the proper index. Weird (but working). Inventions struck me more with this tool than with the others (but it's also the one I used most, so I may be biased). Its different proposals are good and different. By default, in TypeScript, it create fat arrow functions on multiple lines even when the result could be a one liner. In Python, it tends to propose pass as the body of the function by default (you can then make it write it).

It's also the only assistant I tested that propose more features than mere code completions:

It comes with a chat that is a must use. Instead of searching Stack Overflow for something, most of the time I could just ask it how to do something (like a form in Angular) and it would respond. Depending on the answers, I could get a code snippet I could use, a detailed explanation or I had to ask it again (and sometimes go to Stack Overflow).
It proposes tools to refactor code: you select a piece of code, run a default refactoring action or explain what refactoring you want. Then it proposes a code snippet you can apply to take the refactoring into account. I only used it on simple tasks, but it worked well. This sounds really promising since refactoring can be really boring.

My opinion

With the notable exception of CodeWhisperer I enjoyed working with these tools. I mostly wrote the name of the function I wanted and let the tool complete it. Test cases aside, I don't think I had to explain in natural language what I wanted to get a good result (please note that I used very explicit function names). I tried to disable it to see if I would miss it. And I did. Mostly for writing boring code in my place!

There are a few things to keep in mind while using these tools:

They are not always up to date.
They can propose really stupid things or "hallucinate". What's more surprising (even if you know a bit how they work), is that they propose both really relevant suggestions and really stupid ones in the same session. It also help put into perspective our potential replacement as a developer (and you still have to explain to it what to do!). So for now, developers are safe.
It's a bit frustrating that they get parentheses/curly braces wrong so often.
Depending on the context you use it in (existing file, opened files), results can vary. Try to open relevant files to help it do more relevant completions.
It can be tempting to not refactor the supplied code because it arrived fast and was "good enough". Since the tool allowed me to be more productive, sometimes I just wanted to be a bit too "productive" and move on to other part of the project and leave the code as is.
Beware not to get stuck on the proposed solution for too long: if it doesn't work and you fail to get it working, starting from scratch by yourself may save you time in the end (or chatting with the bot, or search Stack Overflow).
Solutions (both autocomplete suggestions and answers from the chat) are without context. So you can't know easily (besides your own experience) whether they are good. At least on Stack Overflow you have ratings, comments and other answers to help you (I always read comments and other answers when I'm not sure about the top one, I hope you do too!).

How about learning?

You can use these assistants to help you write code in a language you don't really know. I tried that with Elm. It was more pleasing that doing it only with the documentation and web searches: you can rely on completions to get to your goal more rapidly and with less frustration. Especially with a language as different from the ones I know as Elm.

Did I really learn or practice Elm? No! That's not really a problem in this context, because I didn't really want to learn it. I think it can be if I really tried: how could I get into the right state of mind, learn by practising if a tool gives me the answers? I could understand the code and edit it a bit thanks to my experience, but overall, I don't really know if the code is actually good.

More generally, I don't think you can truly learn anything with a tool like this: you need to think, fail and write code by your own to learn. I don't think there is an alternative to this. But these tools will make it harder to do it correctly. For instance, I tried to do some type challenges in TS. With the autocomplete on, I just got the answers. Answers I could not have found on my own without trying a bunch of things (and some I just couldn't get without the solutions since they relied on things I didn't know).

And you also need to know whether the code is good, secure and maintainable.

The chat can be useful to get relevant information right from your editor though. But I think you'll still need to write code by yourself, experience developers to review your code and blog/courses/books to move forward in your programming journey. So I don't advise using completions when trying to learn: you could feel more productive at the start, but I think it will come and bite you in the long run.

How about using it on the job?

With all that said, these assistants sound like the prefect tool to help to you in your job. There are some points to consider before that though:

Technically, you must pay for them to use them in a professional context. They are not expensive, but depending on your company size, it can become quite expensive.
They are many legal considerations to take into account:
- You could violate copyrights: the assistants learned from OSS projects and may generate license protected code that will end up in your project. It's mostly dangerous for GPL protected code. While technically you could do it before, you had to conscientiously do it, search for it and use it. Now, it's at the finger tip of millions of developers that will have no idea they are infringing any copyright.
- Who owns the output of an IA tool is not clear.
- This requires to send code to a distant server you don't control. This could result in code/data/secret leakage. You need to make sure your provider won't learn from your code. You need to trust it and read its data policy: to be more accurate, it's too important to just accept without understanding it in a professional context and you need to get legal involved.

So don't do this on your own, involve your co-workers and management. Even to use the tool from time to time at your job with a personal account. To avoid having problems over this, you must get management support before using the tool. Otherwise, this too can come and bite you!

Wrapping up

AI assistant can be very powerful tools to help you refactor code, get answers faster and help you write code (either tricky or boring).
Probably not the best tool to learn and practice programming. I think it applies to all other activities.
You still need to rely on your experience to maintain code quality and security (and to make the whole thing work).
Make sure to have management support if you want to use them at work: they bring a lot of non technical issues you cannot solve on your own.

Other takes on this:

Exploring Generative AI.

My opinion on enums in TypeScript

2023-08-18T00:00:00+02:00

Today I'd like to dig a bit on enums in TypeScript and their potential alternatives. For this, I'll be using the playground and TypeScript 5.1.6 (latest released version when I am writing this). I expect you to know what enums are and to be comfortable in TypeScript.

Let's start with a very basic definition and usage of an enum:

enum ProgrammingLanguagesNumberEnum {
    Python,
    TypeScript,
}

const simpleLog = (language: ProgrammingLanguagesNumberEnum) => console.log(language)

// We can use the members.
simpleLog(ProgrammingLanguagesNumberEnum.Python)
// We can use member values directly (by default enums maps to numbers, the first member is 0 and then each member increment by 1).
simpleLog(0)
// We cannot use a value that is not part of the enum.
simpleLog(10)

Note

On previous versions of TypeScript (that is until version 5), simpleLog(10) was allowed and would not be reported as an error.

We can also use specify numbers explicitly:

enum ProgrammingLanguagesExplicitNumberEnum {
    Python = 10,
    TypeScript = 20,
}

const simpleLogBis = (language: ProgrammingLanguagesExplicitNumberEnum) => console.log(language)

simpleLogBis(ProgrammingLanguagesExplicitNumberEnum.Python)
// This yields an error since 0 isn't valid any more.
simpleLogBis(0)
// This works.
simpleLogBis(10)

We can also use strings:

enum ProgrammingLanguages {
    Python = 'Python',
    TypeScript = 'TypeScript',
}

const simpleLogTer = (value: ProgrammingLanguages) => console.log(value)

// Using members directly still works.
simpleLogTer(ProgrammingLanguages.Python)
// This fails since Ruby is not an allowed value of the enum.
simpleLogTer('Ruby')
// This also fails, even if Python is an allowed value.
simpleLogTer('Python')

Now that we've reviewed the basics, let's see why we would use enums:

It's obvious when reading the code that the value belongs to a collection of values. We are not seeing an random 0 or 'Python' but ProgrammingLanguages.Python.

Note

Even though we can use member values directly with "number" enums, I think it's a bad idea. I'd even say, the compiler should behave like with string enums and report an error.

We can check that all cases are handled. In the code samples below, if a branch is not covered (or if we add values to the enum), we will get an error:

function assertNever(value: never): never {
    throw new Error(
        `Unhandled discriminated union member: ${JSON.stringify(value)}`,
    );
}

const logProgrammingLanguageStringEnum = (language: ProgrammingLanguages) => {
    switch (language) {
        case ProgrammingLanguages.Python:
            console.log('Python');
            break;
        case ProgrammingLanguages.TypeScript:
            console.log('TypeScript')
            break;
        default:
            assertNever(language);
    }
}

const toEmoji = (language: ProgrammingLanguages): string => {
    switch (language) {
        case ProgrammingLanguages.Python:
        return '🐍'
        case ProgrammingLanguages.TypeScript:
        return '💎'
    }
}

To map enum values to other values. Just like with the previous example, if a value is missing or when we add values to the enum, we will get an error:

const languagesToEmoji: Record<ProgrammingLanguages, string> = {
    [ProgrammingLanguages.Python]: '🐍',
    [ProgrammingLanguages.TypeScript]: '💎',
}

We can get all the members of the enum like this (it works because once transpiled to JavaScript, enums are just objects, more on that later):
```
// We can of course also use Object.values and Object.entries.
const languages = Object.keys(ProgrammingLanguages);
```

If enums have nice qualities, then why would we want to use something else?

We can have those same properties in other ways. So it makes sense to compare the different possibilities before making a choice.

TypeScript will generate extra code that must be shipped and executed. So this enum:

enum ProgrammingLanguages {
    Python = 'Python',
    TypeScript = 'TypeScript',
}

Will become:

var ProgrammingLanguages;
(function (ProgrammingLanguages) {
    ProgrammingLanguages["Python"] = "Python";
    ProgrammingLanguages["TypeScript"] = "TypeScript";
})(ProgrammingLanguages || (ProgrammingLanguages = {}));

So when we have many enums, their impact won't be negligible both in term of size and execution time.

What are the alternatives?

Plain constants. We define const PYTHON = 'PYTHON' and const TYPESCRIPT = 'TYPESCRIPT'. It works but we loose the fact that these variables are grouped and the compiler cannot notify us about invalid cases in usage. So, not acceptable.

Union types. We can define type Languages = 'Python' | 'TypeScript'. This will still work as expected:

const languagesToMessage: Record<Languages, string> = {
    Python: '🐍',
    TypeScript: '💎',
};

const languagesUnionToEmoji = (language: Languages): string => {
    switch (language) {
        case 'Python':
            return '🐍'
        case 'TypeScript':
            return '💎'
    }
}

However, when reading, it's not obvious that the value comes from a union type and we cannot get all the members programmatically in code (types are compiled away).

Using a const objects. It's probably the alternative that is closer to using enums. But we need an intermediate type to ease its usage:

const allowedLanguages = {
    Python: 'Python',
    TypeScript: 'TypeScript',
} as const;

// This is not allowed on a const object. So no risk of adding members by mistake.
allowedLanguages['Ruby'] = 'Ruby'

// We must extract the exact keys of our object.
type AllowedLanguages = keyof typeof allowedLanguages;

const allowedLanguagesToMessage: Record<AllowedLanguages, string> = {
    Python: '🐍',
    TypeScript: '💎',
};

const allowedLanguagesToEmoji = (language: AllowedLanguages): string => {
    switch (language) {
        case 'Python':
            return '🐍'
        case 'TypeScript':
            return ''
    }
}

Using const enum. Instead of declaring our enum like we did, we declare it like this:
```
const enum ProgrammingLanguages {
    Python = 'Python',
    TypeScript = 'TypeScript',
}
```
This way, it will be compiled away. It's an enum behaving like an enum with a notable exception const languages = Object.keys(ProgrammingLanguages) is triggering an error since it doesn't transpile into an object.

However, do note that they come with their own set of pitfalls.

If we wrap all this up, what should we use?

I think we should use string union types as much as possible to replace enums: they fit nicely into the language and are easy to define and use. On reading, it's a bit harder to see they come from a collection of values, but I think that with habit you can guess when a union type is used (at least that was the case for me). And you editor should be able to help you out. On writing, our editor will complete the strings, so it's not a problem.

I don't think we should use other union types (like number union types). Reading a plain number won't give enough information to understand what is going on.
If you need to access all the members in you code, const objects are a good alternative. They are even recommended in TypeScript documentation!
Given the pitfall coming with const enum, I think they should be avoided (and if you don't use them often, you'll probably forget about those pitfalls).

TL;DR: Enums can still be used, but we have better alternatives now. So I think we should use them only sparingly and always start with either a union type or a const object.

Last question: should you migrate away from enums? I think you may but don't need to. They add code to your bundles and may slow code executions, but it this the first thing you must do to improve this? I don't think so. Checking duplicated or big libs will probably contribute more to code bloats than enums. But I guess it depends on your code base and on how much you use enums. It's also easy to do (even if it can be time consuming).

For reference and to help you test what I explained here, here's the full code:

// Basic enums

enum ProgrammingLanguagesNumberEnum {
    Python,
    TypeScript,
}

const simpleLog = (language: ProgrammingLanguagesNumberEnum) => console.log(language)

// We can use the members.
simpleLog(ProgrammingLanguagesNumberEnum.Python)
// We can use member values directly (by default enums maps to numbers, the first member is 0 and then each member increment by 1).
simpleLog(0)
// We cannot use a value that is not part of the enum.
simpleLog(10)

enum ProgrammingLanguagesExplicitNumberEnum {
    Python = 10,
    TypeScript = 20,
}

const simpleLogBis = (language: ProgrammingLanguagesExplicitNumberEnum) => console.log(language)

simpleLogBis(ProgrammingLanguagesExplicitNumberEnum.Python)
// This yields an error since 0 isn't valid any more.
simpleLogBis(0)
// This works.
simpleLogBis(10)



// String enums

enum ProgrammingLanguages {
    Python = 'Python',
    TypeScript = 'TypeScript',
}

const simpleLogTer = (value: ProgrammingLanguages) => console.log(value)

// Using members directly still works.
simpleLogTer(ProgrammingLanguages.Python)
// This fails since Ruby is not an allowed value of the enum.
simpleLogTer('Ruby')
// This also fails, even if Python is an allowed value.
simpleLogTer('Python')

function assertNever(value: never): never {
    throw new Error(
        `Unhandled discriminated union member: ${JSON.stringify(value)}`,
    );
}

const logProgrammingLanguageStringEnum = (language: ProgrammingLanguages) => {
    switch (language) {
        case ProgrammingLanguages.Python:
            console.log('Python');
            break;
        case ProgrammingLanguages.TypeScript:
            console.log('TypeScript')
            break;
        default:
            assertNever(language);
    }
}

const toEmoji = (language: ProgrammingLanguages): string => {
    switch (language) {
        case ProgrammingLanguages.Python:
        return '🐍'
        case ProgrammingLanguages.TypeScript:
        return '💎'
    }
}

logProgrammingLanguageStringEnum(ProgrammingLanguages.Python)
toEmoji(ProgrammingLanguages.TypeScript)

const languagesToEmoji: Record<ProgrammingLanguages, string> = {
    [ProgrammingLanguages.Python]: '🐍',
    [ProgrammingLanguages.TypeScript]: '💎',
}

// We can of course also use Object.values and Object.entries.
const languages = Object.keys(ProgrammingLanguages);



// Exploring alternavites.

type Languages = 'Python' | 'TypeScript'

const languagesToMessage: Record<Languages, string> = {
    Python: '🐍',
    TypeScript: '💎',
};

const languagesUnionToEmoji = (language: Languages): string => {
    switch (language) {
        case 'Python':
            return '🐍'
        case 'TypeScript':
            return '💎'
    }
}

const allowedLanguages = {
    Python: 'Python',
    TypeScript: 'TypeScript',
} as const;

type AllowedLanguages = keyof typeof allowedLanguages;

// This is not allowed on a const object.
allowedLanguages['Ruby'] = 'Ruby'

const allowedLanguagesToMessage: Record<AllowedLanguages, string> = {
    Python: '🐍',
    TypeScript: '💎',
};

const allowedLanguagesToEmoji = (language: AllowedLanguages): string => {
    switch (language) {
        case 'Python':
            return '🐍'
        case 'TypeScript':
            return ''
    }
}

const enum ProgrammingLanguagesConst {
    Python = 'Python',
    TypeScript = 'TypeScript',
}

Exploring a weird HTTP issues

2022-05-04T00:00:00+02:00

Today I'd like to explain how I tried to solve a weird HTTP issues that I found at work. I hope you will find the method and the trials I used the useful/interesting if/when you will encounter something similar.

The issue was this: I needed to dynamically generate a ZIP file with a list of files that couldn't be known before generation. For small ZIP files, everything went fine. For bigger ones, the download would stop before the end and I would only get a corrupted ZIP file. This was of course only happening in production: locally, the download could be long but it would always complete. The stop would always happen when a request was taking at least 30s to complete. Since the project is configured have timeouts at 30s at multiple level to avoid too long requests, that wasn't specific enough to avoid some digging.

The website is written in Python using the Django web framework. In production it's deployed in kubernetes and it's run with gunicorn a well known and used Python application server. In front of gunicorn, there is a nginx reverse proxy, mostly to buffer file upload requests (see this article for more information on that).

Now that you have this context, let's dive in.

gunicorn?

I didn't start my exploration methodically. I had an intuition it could be linked to gunicorn: the Django application would take too much time to generate the ZIP and gunicorn would time out. By default [1], a request times out after 30s. This couldn't happen locally since I use Django's built in server which doesn't have a timeout. And in production, I could see in the logs that gunicorn was restarting its worker process.

So, I timed the ZIP creation process. In some cases, it could get close to 30s and even be longer than this. This was mostly due to the fact we were trying to compress the files. Since they were PDF and images, we didn't really need to compress them (and it was causing issues anyway). So, I changed how we create the ZIP file to skip the compression entirely and create an uncompressed ZIP file. In worst cases, the ZIP file creation now only took around 5s.

At this stage, I thought I solved the problem. However, after some more testing in pre-production, I found out that sometimes, the request would still abort and result in a corrupted ZIP file. I had to dig some more!

More gunicorn issues?

Since gunicorn is serving the file, I was thinking that maybe it was blocked until the file was completely downloaded. It's an issue I already encountered with file uploads and that I solved with a nginx reverse proxy. Maybe something similar was happening, although I was expecting nginx to buffer the response in both directions.

To test this, I configured a nginx instance locally and added the limit_rate 1M; [2] to the server block to slow down downloads so I could witness the problem. I then launched the site with gunicorn with only one worker and triggered a download. I tried to make requests to gunicorn both directly and through nginx. gunicorn responded as expected while the download continued. I then switched gunicorn off, just to be sure. The download continued as expected. So, I could rule out gunicorn. nginx was correctly buffering the response.

Let's go beyond 30s

So far, so good. However, locally, the download took less than 30s. So, I tweaked the limit_rate parameter again to be sure to hit this 30s mark. The download failed and stopped at 30s.

For reasons that are beyond the scope of this article, when the user triggers the download through the interface as I was, we actually download the file in JS and then on something like this URL.createObjectURL(new Blob([response.body])) to actually save the file. It's not something I'd recommend generally since it adds complexity and the browser can handle it just fine on its own. But, it's how it is.

So, I tried to query the API directly without relying on any JS layer. It worked. It looks like a timeout issue in the frontend code.

Once again, I had similar issues with file uploads a while back: the frontend also had timeouts configured. As it turned out, these timeouts were also an issue here. The project uses SuperAgent for its HTTP request, so I just had to configure it so this request could have a longer timeout.

I tested again, and it worked! I thought I solved it. It turned out there was another issue I couldn't spot locally.

To the server!

At this stage, I once again though the problem was solved. So I deployed my solution in pre-production and tested. And it failed. Again.

Our pre-production environnement is a bit peculiar, and we have two nginx reverse proxies in front of the app (we have only one in production). I though it could be an interaction between these two.

So I tweaked the configuration of both nginx. I increased all relevant timeout values: proxy_connect_timeout 600;, proxy_send_timeout 600;, proxy_read_timeout 600; and send_timeout 600;. I even disabled proxy buffering just in case with proxy_buffering off;.

It had no effect. At this stage, I resorted to try each step in the test environnement separately. To do this, I disabled all HTTPS redirections and disabled permissions on my backend view so I could access the URL directly with curl [3] .

Our test environnement is in kubernetes, but the same method applies to any environnement. I connected to the pod where gunicorn and its nginx side car were running and tried this:

Talking directly to gunicorn: curl -v --output gunicorn.zip localhost:8000/api/download-url [4]. It worked perfectly and the ZIP file was valid (I unzipped it).
Talking to the nginx sidecar: curl -v --output proxy.zip localhost:80/api/download-url/. Again, it worked.

I then connected to the pod of the second reverse proxy and ran:

curl -v --output proxy.zip https://test.example.com/api/download-url/

And it failed.

What I am missing?

To dig deeper, still thinking the issue was in the second proxy, I increased its log level with error_log stderr debug;. I got this error:

epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while reading upstream, client: <IP>, server: <SERVER_NAME>, request: "GET /api/download-url/ HTTP/1.1", upstream: "http://<IP>:80/api/download-all/", host: "<HOST>"

That's a weird error. curl (what I thought as the client) didn't closed the connection. What could be happening?

I googled a bit but didn't find anything useful. I tried again thinking at each step what the request actually did and traversed. And then, I got it: by using https://test.example.com I was not talking directly to the second proxy. I was going through the load balancer first. It's obvious once you know it, but I had to think about it.

So I tried again with curl -v --output proxy.zip -X 'Host: test.example.com' localhost/api/download-all/ to skip the load balancer and really talk to the proxy directly. And it worked!

So the problem came from the load balancer after all! And the message reported by nginx is coherent with this: the load balancer is its client and it's closing the connection. If it were not, I'd be out of ideas this time and would need to find an alternative solution to my problem.

I searched the documentation and found out that there is a timeout for these requests (it's not hit for file uploads for some reasons or I'd have know about it sooner) and that we can configure it. The documentation for GKE is here. I applied the suggested configuration and… TADA! Problem solved for good. At least until the download isn't longer than the supplied timeout, which is unlikely for our use cases.

Conclusions

It took more time that I originally anticipated. The fact that many problems were tied together made the resolution harder:

The ZIP creation was too slow. So gunicorn could time out on some occasions [5].
The frontend had a timeout, so the request couldn't also complete because of that.
The request was so long that we also hit a timeout at infrastructure level. This one could only be found in a test environnement close enough of production. To make things worse, I forgot about the load balancer and thought I was talking to a component directly when I actually wasn't.

Luckily for me, I encountered a similar problem with file upload. This time, a rapid Google search indicated me to use a reverse proxy to solve the issue. So, as always, the more experience who have, the more likely you are to have an idea or a lead to where the problem is.

In the end, I had to use to the good old method of testing each component after another to find out what is actually causing the issue. It's long a a bit tedious, but it works.

While I was doing this, I couldn't help but think of other solutions. It's a good thing I didn't stop, because most of them wouldn't have worked and all of them would have needlessly increased the complexity of the project. The solutions I thought of were:

Create the ZIP file with gunicorn and use a nginx feature named X-Accel-Redirect to transfer quickly the request to nginx. It would have been useless since nginx was already buffering the request correctly.
Uploading the file to bucket storage and redirect the user to it. It would have worked but I would also have to clean the ZIP files after a while.
Switch to an extra service altogether if the ZIP creation had to be long. It could have helped, but I would still have the long download issue.

I hope you will find this useful. If you have a question or a remark, please leave a comment below.

[1]	And we use this default value.

[2] You may wander why I used this instead of the browser tools. I started with the throttle tool of the browser, but by default it was way too slow. I managed to configure it, but on some occasions the limit wasn't enforced correctly. And later, I needed to test with curl anyway. So in the end, it saved me quite a lot of time.

[3]	It goes without saying than this environnement doesn't contain any sensitive information. I used random public test files to test my ZIP download.

[4]	It's a placeholder URL as you probably guessed.

[5]	It could also seems unresponsive to kubernetes probes so kubernetes would restart it. I didn't mention this sooner to make reading easier. Since our requests are now fast enough, tweaking the probes isn't necessary, so I didn't do it.