Jujens' blog

Server tips

2020-09-17T00:00:00+02:00

Sommaire

Start systemd services automatically for a user

Start systemd services automatically for a user

By default sysemtd services linked to a user – created with systemctl --user enable --now service for instance – are only started after the first login of the user. To start them when the server starts, you must enable lingering for the user with (as root):

loginctl enable-linger USER

You can also soo the lingering status for a user with (as root):

loginctl show-user USER --property Linger

Source: https://wiki.archlinux.org/index.php/Systemd/User#Automatic_start-up_of_systemd_user_instances

Make JS code communicate with a ServiceWorker

2020-08-18T00:00:00+02:00

If you need to send data from your JS code to your ServiceWorker, you can do it like this:

navigator.serviceWorker.register("./sw.js").then(registration => {
    registration.active.postMessage("Some message");
});

You can then rely on this code in the ServiceWorker to read the message:

self.addEventListener("message", event => {
    console.log(event.data); // This will log: Some message
});

Aurelia Store and Immer

2020-08-09T00:00:00+02:00

I spoke last year of Aurelia store in this article At the time, it wasn't possible to combine Aurelia Store and Immer. The root cause of the issue was fixed in the latest version of Immer (7.0), so we can now combine both.

Let's study how to do this:

Install immer with npm install --save immer@^7.0.7 or yarn add immer@^7.0.7
Because we need Aurelia to update the state when it's used in component to add some observers, we must disable immer's auto-freezing. You can do this in main.js or app.js with:
```
import {setAutoFreeze} from 'immer';
// This is required to allow Aurelia to add its observer on objects in the state.
setAutoFreeze(false);
```
Update you actions to use produce. This will allow you to modify the state directly without mutating it. This way, the history features of the store will be preserved. For a simple example:
```
// Before.
export function selectBackend(state: State, selectedBackend: SelectableBackend) {
    const newState = {...state};
    newState.selectedBackend = selectedBackend;
    return newState;
}

// After
export const selectBackend = produce((state: State, selectedBackend: SelectableBackend) => {
    state.selectedBackend = selectedBackend;
});
```
Thanks to immer, you don't have to copy the state explicitly, immer will create a new state based on your modification automatically. The produce function takes a function that will receive the draft of the state you can modify as first argument and all other arguments passed to you action as extra arguments. So the function you need to pass to produce takes the same arguments as a standard action. You still need to be aware and pay attention to some details:
- All the objects in the store are changed so immer can track the changes without modifying the original objects. This means that this won't work:
```
export deleteArticle = produce((state, article) => {
    // Will return -1 because article is not in the articles array, it's proxied version is.
    const indexOfArticleToDelete = state.articles.indexOf(article);
});
```
- Only the state will be a "draft" (to use immer terminology) and can be modified directly. If you were to modify other argument, you would mutate them. And if they are in the store, you would update them the store as well. So this won't work as expected and will mutate the object:
```
export markAsRead = produce((state, article) => {
   article.isRead = true;
});
```
For both these issues, you need to first find the proper object in the draft state and apply the modification to it. You can view an example here.

If you want more detail, you can check this commit of one of my project when I updated my actions to use immer.

As noted by Vildan Softic (the creator of Aurelia Store), you can organize how you update the state differently to update the state correctly while keeping things clear without requiring immer. You can see an example here. Many thanks to him for pointing that out and giving me some pointer to correctly test Aurelia Store with Immer.

So while I think it's really nice to be able to use Immer, whether to use it or not highly depends on your programming style and on whether you can afford adding yet another library to your project (immer is about 15kB minified which is tiny, but it adds up with the rest).

Testing a clojure web framework

2020-08-02T00:00:00+02:00

After learning Clojure thanks to the awesome book Clojure for the Brave and True I decided to test some web programing with the language to practice a bit.

Before dwelling on this experience, I'd like to say a few words about the language itself. I heard about the language when I was still completing my masters degree in 2014 or 2015. I didn't gave it much attention back then. Later, I noticed that Robert C. Martin (author of the book Clean Code among other books) talked about it and seemed to enjoy the language. Since I respect his technical opinion, it raised my interest again and I look a bit into it and bought the book Clojure for the Brave and True which then sat on my bookshelf for a couple of years. Until the middle of last year when I decided to seriously look at functional programming and finally went through the book.

This was mind blowing! In addition to the great qualities of the book who made the learning journey really fun, I found the language itself to be really well designed and interesting to learn. I like the notion than almost everything is a function (the example with the simple + function at the beginning of the book really hooked me up), the way it handles "polymorphism", atoms and references… I also had to change perspective and get used to everything being immutable which requires lot of thinking at first but forced me to reconsider many things about my daily programing practice. The fact that the book comes with many exercices after each chapter is a great plus to sharpen the reader's skills. It's a great learning experience and I do encourage you to learn the language. I must confess I still have issues with the syntax (it's a LISP after all with lots of parentheses everywhere) and how to organize my code. But I don't practice much, so I guess it's normal.

Moving on to web development and Luminus. In order to practice and to do something that ressembles what I do daily (web development), I decided to write a small website using the language. I looked around for a library/framework to help me in this endeavour and found Luminus which looked very good: the documentation is very detailed and it has a great getting started. It also has a lot of template to create projects with different setup easily (like choose which database you want or to bootstrap the frontend for you). I also dwelled a bit on frontend development with ClojureScript (the version of Clojure designed to run in the browser) and Reagent, a library written in ClojureScript that leverages React to create dynamic interfaces. After some quick tests, I knew I would be able to do what I wanted.

My goal was to create a site to help manage various competitions. In the end, I mostly played with Reagent and coded the home page of the site as well as the detail pages for each competitions. I initially wanted to add a form to add competitions and a search feature but gave up due to lack of time. The frontend was entirely done in Reagent and communicated with the backend thanks to REST. The code is available here if you want to take a look at it.

I liked how the frontend and backend can work together and never require a full server restart. The code, from both the frontend and the backend, is updated via hot updates which makes the developer experience very fluid. That's also very important since the server is very slow to start, without this auto-reload feature, I think the development experience would be very poor. It also means I almost never had to fully reload the page. That was really pleasing.

On the unpleasing side of things: I decided to use a SQL database (as I saw while making the getting started). From what I tested, that's where the framework doesn't shine: I had to write SQL for each query I had to make which doesn't seem very flexible to me. Maybe because I'm too used to the Django ORM by now to revert to raw SQL for every requests. I also had to restart the server each time I edited a query for my changes to be taken into account. That was adding to the pain I felt when writing SQL queries and one of the reason I decided not to take more time on the project: it would require even more queries. I looked for ways to avoid this, but didn't find anything (I may not have looked hard enough) and still thinks the framework should handle this for me. What's strange is other than that, the experience is very fluid and hot reload work perfectly. This made me think that it's also partially why MongoDB is so popular: all you need is to send JSON to the server and all languages have good tools to do this, mostly Clojure with its emphasis on data and using maps [1] everywhere. No need for someone to build and maintain the complex piece of technology that is an ORM.

All in all, I'm satisfied with what I did and intend to continue trying stuff with Clojure. If I do something else, I'll probably use MongoDB to avoid SQL altogether.

[1]	Also known as dict in Python.

Retrospective on my biggest personal project

2020-07-19T00:00:00+02:00

Today I'd like to look back at my longest (and biggest) personal project and what I learned along the way. The project is now called Last Run (but was named Arena of Titans when we started). It's a board game adapted for web usage. It went live a few month ago and is available here. The project is also entirely Open Source under the AGPLv3 license with the code source hosted on gitlab. If you play and have feedback, let me know! We are on social media or you can leave a comment here.

I started the project in 2014 when I was completing my studies in Centrale Marseille, a French engineering school. At the time, we were 4 on the project: the creator of the game, myself and two other developers. This project was accepted as our final project assignment which was a great way to combine passion and usefulness. Now we are three: the creator of the game who does a bit of development but is mostly working on the gameplay, myself as the main developer and a new developer who participates from time to time.

The original version of the game was written in Java and AngularJS. I remember I looked at Apache Wicket and Vaadin to do everything in Java, including the frontend. This idea was influenced by a fellow student (but not teammate) who really liked Vaadin (and maybe still does). Eventually, after some tests, it became clear the project wasn't really fitting into this model (don't ask me exactly why, it was years ago) and I settled on AngularJS (for the best I think). At this time, the frontend did HTTP requests to the API because it was easier this way given what we knew and all players had to play on the same computer because it was easier for us and we didn't really know how to update other browsers even if they were connected to the same API. Since HTTP requests should be stateless, I stored the game state in Redis which is the only original technical decision that remained over the years. It was also the early days of Docker, which looked cool and could allow me to avoid installing Java on my server, so I initially deployed the app with it. It was a bit painful (mostly because I did it at the very end of the project so the examiners could see it on the web) but it was interesting and fun to do.

After graduation, the creator of the game and myself still wanted to work on the game to improve it while the other members of the team didn't and left. I tried to improve the Java version, mostly to add proper multiplayer and to rely on websocket to provide real time, two-way communication with the API. However, I've always been a Python guy and choose Java for the first version of the game only because it was more convenient for other team members. I was also a bit tired of the ES5/AngularJS frontend and wanted to switch to ES6 (we were in 2015 at this time, JS already made its "revolution" with ES6). The fact that ES5/AngularJS was what I was using at work also influenced my decision: I wanted to play with something new and different.

So, I decided to rewrite the thing. I tested few options in the backend:

Go: it felt complicated given what I knew and what I was trying to achieve. Furthermore, I dislike the type system, maybe because I came from a Java and Python background. My main paint point was that I was already using sets in the algorithm and since Go doesn't have sets in the standard library, I needed an extra package. And since Go doesn't support generics, I had to allow any type in my sets which defeats the purpose of a type system in my opinion. That's the equivalent of doing Set<Object> in Java which is strongly discouraged.
NodeJS: it felt very young and was lacking some modern APIs (like Set) at the time. That would have required to either not use them or to somehow transpile the code which I wasn't willing to do. And the future of the project wasn't very clear at the time with the io.js fork.
Python: I was able to do all I wanted and it's a language I always used and loved. I was also using it at work, so it made sense to use it here too. Did I really need to test anything else you may ask? For the sake of testing and experimenting I may answer. I choose (and still use today) the Autobahn library for websocket communication. What I found interesting was that I could use it with asyncio (which landed in the standard library in Python 3.4, the Python version of the time) which seemed like the way to go for async programing. That was a very goo choice, mostly in Python 3.5 when the async and await keywords were introduced and completely made asyncio the way to go for new async code.

In the frontend, I tried several options (VueJS wasn't really a thing at this time, so I didn't tested it):

Angular 2 (now just named Angular): it seemed like the logical choice since the existing app was written in AngularJS. The framework was in Beta at the time. I really didn't like the experience it provided: it felt very complicated and cluttered. I also encounter several problems with the router which didn't invite me to trust the framework. Even after reading a book about it a few months later, after it was officially released, it still felt over-engineered and not that clear to me. So I was glad I discarded it.
Ember: it felt nice but limiting because of how it handles data. I didn't really adhere to its ways of doing things. Still felt like a strong framework.
React: it didn't left much of an impression at the time. I don't think it was that popular back then and it felt like I would lack some structure and need to add many things by myself (which is consistent with the fact it's more a rendering library than a framework).
Aurelia: that's the framework I immediately loved. It felt like an updated and simplified (in a good sense) version of AngularJS. I saw how I could transfer and improve my existing practices of working with AngularJS. I still think it is an under-appreciated framework that deserves to be more known.

So after these tests, I decided to rewrite the project in Python with Autobahn using asyncio and Aurelia in the frontend. I started in July 2015 and I think I completed a first version during the rest of the summer. Of course the project evolved over time:

I added CI to run my tests automatically.
I add many linters (both in the backend and in the frontend). I also added prettier and black to automatically format my code. I check formatting with a pre-commit hook in git and even let PyCharm reformat the code on save.
I switched to pre-commit to manage my pre-commit hooks.
I switched to Aurelia store to manage the application state. I initially relied on a custom service but relying on a more complete solution made sense in the end. See this article to learn more about this process.
I added validation in the API. Initially I always assumed I was given proper data (I guess I was young and unafraid back then). By the way, in Python, Cerberus is a very good library to do that!
I'm stricter in my usage of functions: initially I embraced the flexibility of Python a bit to much and allowed both Card object and tuple (card_name, card_color) (which allowed me to find the proper Card object) to be passed around which made the code more complex and more fragile. I'm stricter now in how I define the interfaces of my functions.
I reduce various subjects of tech debts and simplified many portions of the code thanks to all the things I learned in Python, JS and programming in general over the last 6 years.
I changed a few times of package managers and bundles for the frontend. See this article to learn more.

One very important point I haven't talked about yet is the deploy process. That's another area where I learned much over these years! I had a server even as a student, so I used it even for the earliest versions of the game. I don't remember exactly how I deployed the first version of the app using docker though. I think I just exported the image as a tar.gz and imported it on the server with some kind of custom Bash script. Since it was the early days of docker which wasn't very stable at the time, I switched to standard uwsgi directly on the server and packaging all Python dependencies as RPM packages installed on the system (luckily I didn't have much of those because it takes a lot of time!). Later, when I wanted to switch to a more powerful server, I think I switched back to docker because it made dependency management and deploys easier. At this stage, I was still using custom Bash scripts to build and deploy images. The front was deployed with Rsync. It was a bit weird, not always stable but globally worked. After that, I switched jobs and discovered Ansible as a tool to deploy stuff. Since it way more stable and complete than what I could create by hand, I switched to it a few years back. That's where I am now: a bunch of Ansible playbooks to build and deploy both the frontend and the backend. The backend runs in docker and the frontend is served directly by nginx because it doesn't need anything else. I still have a Redis instance that was deployed once, so it doesn't bother me much.

The last thing I'd like to share is how I handle multiple versions of the game. It's not very useful right now since we don't have many players but it felt like something I had to do correctly from the start because it would be harder to add later on. Since the game can change from one version to another (because we had features), incompatibilities can arise: the frontend can send a request in an older format or the API can require new information the frontend cannot send yet. The result would be a broken game for the player. I could use feature flags to detect which code path to use and have a single instance of the API but I think it would clutter the code more than serving the actual purpose. So I decided that each version of the frontend would talk to a dedicated version of the backend. To do that, I developed a small gateway which reads the version of the API the frontend wants and transfer the traffic to the proper container. It's written in Go because I really wanted to give another try to this language and though this routing gateway would be a very good fit for the language (which I still think it is). When you go on the site, nginx will always serve the latest version of the frontend by default. Once you start a game, the version of the frontend is used in the URL so if you refresh the page, you will get the proper version.

To conclude, I really like the project. It took time to go to where we are now for various reasons: I made mistakes because I was a beginner, I can only work on the project during my spare time, I'm mostly the only developer working on the project and we made some changes in the gameplay that had a huge impact on the structure of the project. Overall, I am pleased and proud of the work accomplished. I learned a lot, was able to try thing and spent countless hours over Python, JS, Bash or Docker problems. This may sound like a loss of time but it was a great way to learn. I still plan to work on this project and don't know where it will be in 6 years from now. If it's still around, I'll try to publish a follow up article!

Switching from Aurelia CLI to Webpack

2020-07-04T00:00:00+02:00

Recently I decided to change the build system I use on my main Aurelia project. I changed in April 2016 from JSPM to Webpack (JSPM was really not awesome at the time, I can't say for now), again in August 2016 from Webpack to the CLI and now I am switching back to webpack. At the time, I moved away from webpack because of a big bundle of 1.3 MB that was always loaded. I didn't manage to split it correctly using Webpack while Aurelia CLI gave me the flexibility I needed to load less JS by default. So why did I decided to switch again?

To get smaller bundles: since Webpack is not used just by Aurelia it supports many optimisation to reduce the size of your bundles (like tree shaking to remove unused code).
To simplify assets management: Webpack can do it directly. I used to rely on gulp-rev to correctly version them. But it required manual work to get it working. Now I can just rely on Webpack.
Currently, Server Side Rendering with Aurelia is only supported by Webpack. This could be a big plus for my site where I have public facing pages that would gain from being correctly indexed by Google.
Hot Module Reload to speed development (alone this wouldn't have been enough for a switch, but it's nice).

Since my last try, Webpack and Aurelia's support for Webpack both significantly improved so I guessed this was worth a try to help me reduce my bundles (that the main reason I tried to be honest, the other ones are more bonuses).

Here's how I made the switch. I:

Copied required dependencies from a new project created with the CLI (it was of course configured to use webpack, which is now the default option for a bundler). That included of course webpack and webpack-dev-server.
Changed .css into .scss in HTML files so styles are correctly loaded by webpack on my require lines.
Used PLATFORM.moduleName where necessary. This is required to bridge Aurelia with Webpack. See the documentation. This mainly implied to:
1. Add PLATFORM.moduleName to link moduleId to their module my route configurations.
2. Add PLATFORM.moduleName in plugin definition.
Updated my storage service so it uses a prefix for my keys. This was required because webpack use localStorage to store things and I looped on all the keys which resulted in errors in my code.
Updated aurelia.json to match one provided by the webpack skeleton.
Added a webpack configuration. It's based on the one from the skeleton. I had to change (to match what I do in my project):
1. Where I get sources from src to app.
2. The name of the static folder from public to assets.
3. Where I send the static files when the app is built.

Import all my images and assets directly in JS files so Webpack would correctly copy and version them. This way, in JS I can use the imported name to point to the assets and let Webpack handle the versioning. I had a bunch of gulp tasks to do this before and it was a bit cumbersome. See this page and this commit for more details on how to manage assets with webpack. To help me generate the import lines and the mapping objects I often needed, I relied on this small Python script:

files_list = """COPY HERE THE RESULT OF ``ls -d PATH``""".strip().split()

import_name_to_file_name = {}
for elt in files_list:
    file = elt.split("/")[-1]
    file = file[:-4]
    components = file.split('-')
    components = [c.title() if index > 0 else c for index, c in enumerate(components)]
    import_name = ''.join(components)
    import_name_to_file_name[file] = import_name
    print(f'import {import_name} from "../../{elt}"')
print('{')
for key, val in import_name_to_file_name.items():
    print(f'"{key}": {val},')
print('}')

Added index.ejs as a base index file for webpack and adapt it to suit my needs.
Updated my jest configuration so I could run my tests again. I even went a bit further and stopped using Aurelia CLI to use Jest CLI directly. This allowed me to have access to all the options provided by Jest more easily. To do this, aside from updating the configuration to support Webpack, I had to make sure the environment variable BABEL_TARGET was correctly set to node. Without it, Babel doesn't target the proper runtime and it doesn't work. Since I already used nps to launch some scripts, this wasn't an issue. I defined my tasks to run tests like this:
```
test: {
    default: crossEnv("BABEL_TARGET=node jest"),
    coverage: crossEnv("BABEL_TARGET=node jest --coverage"),
    watch: crossEnv("BABEL_TARGET=node jest --watch"),
},
```
See this commit for more details.
Used a webpack plugin to help me generate my ServiceWorker to preload all bundles and assets: serviceworker-webpack-plugin. The details are here.

I had one big issue during the switch: compose. I use this to dynamically select a view model to display in an element like this: <compose view-model="./${type}" model.one-way="popupModel"></compose> (where type is passed to my component and corresponds to JS and HTML files to load on the view (for instance if type is transition Aurelia will load and inject the content of transition.js and transition.html). Sadly, Webpack doesn't know what to do with this out of the box. Luckily, I found a thread on the forum that gave the solution: declare all you need to use with PLATFORM.moduleName in the component definition. See this commit for more details (or the thread on the forum).

I'd also like to point out that the configuration generated by the CLI can optimize for two major use cases:

Usage over HTTP 1.1: you get fewer but bigger bundles.
Usage over HTTP 2: you get a lot of small bundles (around 50 in my case).

Since my site is only compatible with modern browser and supports HTTP 2, I decided to optimize for that. One important point to know is that by default, all bundles will be loaded by the index.html even those you won't need yet. To prevent this (and put some pages in dedicated bundles), you can specify a chunk name to PLATFORM.moduleName like this: PLATFORM.moduleName('myRouteModule', 'chunkName'). This way, all that's required for myRouteModule to run will only be loaded when the route becomes first active. That's a good thing to know to avoid loading to much JS from the start if your app is big. See this page to learn more about this.

Results:

Bundle sizes before webpack:
- Total first party JS loaded on the whole site: 1.54 MB
- Total first party JS on the home page: 1.02 MB
Bundle sizes with webpack:
- Total first party JS loaded on the whole site: 1.26 MB
- Total first party JS on the home page: 618 kB

So that's a gain of about a 400 kB. Not bad.

To conclude, I am satisfied with the switch: I reached my goal and can take advantages of webpack which is nice! I'd also like to point out that the transition went way smother that I expected: I got it working in a couple of hours and then had to spend some time with various issues (like assets management or the issue with compose) and to tailor the build to my needs which are not standard. All in all, I think I made the switch in less than 30 hours. Not bad. If your needs are less complex than mine, it should be way less than this. I guess both Webpack and Aurelia matured a lot since the last time I made a build system switch. That's a very good thing!

Should you build a SPA?

2020-06-15T00:00:00+02:00

This article is an update to a previous article why you want to build a SPA which I think was laking some nuance and precision. Reading this article is not required to read this one. I'm writing this update because SPAs are very popular and I recently though more about them and realized my previous article was missing some important points. If you see some points that are still missing after reading this article or just want to share your opinion, don't hesitate to react in the comments!

Some note on the context: I am currently working professionally on a website that is not a SPA but I built big SPAs as side projects. I'd like to rely on my experience to explain the pain points I encounter with this project in the hope to convince you to consider building a SPA from scratch on you next project. I hope that with this article, whether you finally build a SPA or not will be based on relevant arguments for your team and project.

Since I work on a "hybrid" project with pages being generated server side and part of the page managed by jQuery or React, I experience many pain points due to this architecture. I guess it was the main motivation why I wrote the previous article: these points wouldn't exist if we had built a SPA. I'd also like to note that most of these decisions were made before I arrived in the company. Let's review these pain points:

Interactions between jQuery or vanilla JS and React (I'll talk mostly of React here because that's what I use at work, but my examples should apply to any other "framework" [1]). That is a very big pain point. Some actions are triggered by jQuery by interactions made on the page. The event handlers are attached by jQuery to the DOM. This is mainly done to hide and show elements on click. The problem is: React can destroy any part of the DOM at any time when re-rendering a component, thus destroying the elements you attached an event handler on, therefore breaking interactivity. Some strategies were built to prevent this: mainly to hook on component lifecycle to readd the event handlers once React added the new elements in the DOM. It is impractical and a big source of bugs. Since React can handle events for us and can show/hide elements, there is no point to use jQuery for this. Just go with strait React, the result will be more simple and you will avoid lots of bugs that are hard to find and fix. Luckily, this practice died in our project, but I still insist on it because it was such a pain and I guess we aren't (weren't?) the only ones doing this.
Big DOM manipulations in plain JS or jQuery. I don't really know why it's this way since we have React for some pages but sometimes we just create lots of HTML elements with jQuery to avoid a full reload of the page or to hide and display form elements based on very complex business rules. These pages don't even load React, all is done with jQuery. The code is bad and hiding/displaying elements based on events is a source of bugs. I guess it was made this way because it was perceive as easier and quicker. What's sad is that from what I understand the price for React was already paid when these pages were built:
- We already had pages (part of pages to be exact) in React so it's there and available (and a better fit for these problems).
- The build system was configured and could load only small bundles of JavaScript.
- Team members had to learn React at this point and from my perspective it makes sense to pay the investment where/when it makes sense. In the long run, it will allow us to go faster with less bugs.
- The code is hard to understand and buggy.
On some pages we do the initial render with Django and then React kicks in to re-render the page and add interaction. This feels like duplicated work that can be optimized.
Interactions between different part of the page can be hard: we can have several small React apps on a page and sometime they must communicate or share common data (for instance: whether the user is logged in or not). Since they are independent and not part of a bigger app, communications is harder than it could be.

The question now is: is this enough to build a SPA? The ecosystem is now mature enough so we can have a way to create and reuse components, very good support for templating, updating components if some data changes and routing with any framework. By routing I mean that the URL in the URL will be updated as if it were rendered on the server (and not just a hash change) with history and back/forward buttons working as expected. Now we can also have Server Side Rendering (or SSR for short) to pre-render our app on the server and thus speed up initial page load as well as making our app easy to index by crawler. This will avoid us any SEO impact. So, it looks like everything would be better in the wonderful world of SPAs, but is it?

Why wouldn't we build a SPA?

Maybe instead of thinking at why we want to make a SPA, we can start by looking at why we wouldn't want to build one (the points below are not really in importance order):

Habit or never built a SPA before. That's the weakest argument since SPA can definitely be a good thing. My advise would be: learn about SPAs and their advantages and downsides, learn how to build one and then try to build one in a test project to see how it goes. You will then have better insights on whether a SPA would be a good fit for your project.
Lack of skills: a much better argument, in my opinion. Building a SPA requires much more JS and frontend skills than building what I'll call a "standard" or "normal" site. But if you avoid building a mess with jQuery, it will be easier for you to hire developers with the right skill set (I don't think they'll join you to maintain the jQuery mess) and to maintain and add feature to it in the long term. And these skills can be learned: from my experience, modern JS and its frameworks can be learned (more or less rapidly depending on which framework you choose and your experience with JS and they'll of course come with their own set of quirks). But it's definitely doable. If you team is lacking UX or design skill, try to rely a contractor to help you, since (from my perspective) those skills will be harder to learn for developers.
SPA are complex to build and also require a mindset changes from how we do things in the backend. That's true, and if you are not used to building them, depending on the application your are building it may not be worth the trouble. So keep reading to learn more!
SPAs hurt SEO: since the server will respond with a HTML file without content, crawlers can't parse anything interesting from the page. It's a valid concern that can be mitigated with Server Side Rendering (SSR) or pre-rendering. Read the sections about this below if you want to know more.
Slow initial page load: same as above, the browser needs to load and parse JS before displaying anything. It may even require some extra HTTP requests to get required data before displaying anything. This can also be mitigated by SSR or pre-rendering, see below.
Extra work: you must re-implement things that comes for free in the browser like a loading icon or error handling. Frameworks or libraries can help, but it sure is extra work.
Bundle size: the browser will need to load more JS which can be an issue on low end devices (like some phones) or on poor connections (mostly on phones too). Depending on who you are targeting this may be a big issue. This can be mitigated by the fact that you can load only part of the JS on each page. Don't forget that marketing typically also includes lots of JS because of various trackers and advertisers. So "your" JS may not be that big (or big enough to be an actual issue) compared to all the other things you have to load (but it is more important for the page to render).
Memory leaks: since your user will stay a very long time in you app, if you don't pay attention, some objects may never be garbage collected resulting in poor performance and high memory usage. But normally, if you follow the best practices of JS and your framework this shouldn't be an issues (like clearing timeouts and interval, unsubscribing to store events…).
Security: how do you authenticate? How do you protect against XSS and CSRF? Those are hard questions and I think we find many misleading information on the internet. I'm no expert and encourage you to do your own research on this (it's also way outside the scope of this article). I'll just write some notes based on mine:
- You can still use cookies with SPA: if your app and your API are on the same domain, even an AJAX request can set a cookie easily. If you do use cookies, make sure they are HTTP only (so JS can't access them to prevent attackers to steal authentication information with XSS), secure (so they are only transmitted over HTTPS). You should use CSRF tokens and probably also use the Same Site header (to prevent sending to cookie to your site when your user are on another site) to prevent CSRF attacks. I think that even if they are not the most trendy choice any more, cookies are a very good mean of authentication which is secure and robust and whose problems are well known with ways to prevent those that exist. This is not always the case for other means of identification.
- Enable CSP (Content Security Policy). This will allow you to choose what your browser can do. For instance, it can be used to disallow execution of inline JS to prevent XSS attacks.
- If you use tokens (with JWT for instance), beware of how and where you store them. See this article for more details on this.
User experience: it can be harder to achieve a good UX because we need to correctly re-implement things that our browser handles natively. And we can do it poorly. I'm thinking of GitHub here: once you loaded a page, you will navigate without a full page reload. Sometimes, it's slow and I seem stuck on a page and just by pressing F5, I can reach the page I'm trying to access immediately (you probably experienced this too). So I guess even for very good tech teams this can be hard.
The technology is still evolving fast, how can we know it will stick around? A fair and complex question. If the list big names seems to have stabilized, the frameworks themselves are still evolving. I think the "big ones" (React, Vue.js, Angular…) will stick for a while because of the sheer number of projects using them. Let's also be honest: what says your backend tech will stay around or won't add breaking changes? It's used widely now but it can be replaced by something else (by a new shiny thing on Deno?).
Overload the backend with too many requests: I guess it can happen if your app need to do a bazillion requests on each page. But I think it's more a design issue than a problem with SPA: why is it doing so many requests? Can we group some? Can tools like GraphQL help?
Reinventing the wheel: For example you will need to find a way to restore the scroll position by any page change, add extra loaders to show that the contents are not yet ready, keep the page <title> in sync with your content and much more… It's free in the browser and it can take time to do all of these little features correctly.
Improper status code: I don't mean error handling of an AJAX requests that failed, I mean that your server must always answer something and, a priori, cannot know whether a route exist. So it will always respond with 200 and the index file. This can be mitigated with SSR, see below.
Accessibility: you can have issues but it's definitely getting better and your classic sites may lack important accessibility features too, let's be honest here. I think it's more whether you will take the time to add these features to your site than a problem with the tools themselves. See this article for more.
Your application requirements don't need it. That's a traitorous argument I think because the requirements can evolve and you may end up needing much more interactivity than you initially though. In the end, it will highly depend on the project you're building and your trust in the requirements to continue not to need a lot of JS. Here are some:
- Your app is read only.
- Your are just dealing with basic forms, where HTML5 validation (or just a bit of JS validation) will be enough.
- Your app must work without JS.

SSR

Since SSR is trendy and can help mitigate some of the most blocking issues with SPA (namely initial load and SEO), here are a few notes SSR.

What is it and how does it work? As I said in the introduction, the point of SSR is to pre-render your app on the server.

Without it, when you request the first page of your SPA, the server will respond with a HTML file that only contains links to JavaScript files. It cannot display anything directly, it has to load the JS and execute it so your JS code can render the content of the page. During this rendering phase, your browser may need to do extra requests to get extra data. The following pages will be handled by the routing module of your SPA and won't require a page reload, but they still can require additional network requests to get the content of the page. The communication between your app and the server is typically done using the REST conventions with JSON as the exchange format (although, now, GraphQL is gaining popularity).

If you enable SSR, your app will first run on the server thanks to NodeJS. Since the server ran the app, it can supply the browser with a complete HTML file that is ready for display. This means the page will display faster in the browser of the user, it will work correctly with search engines and environments that don't support JS. You should also be able to make the server correctly respond 404 for pages that don't exist (which is not easy to do normally since the app written in JS knows whether the route exists or not, the server that serves the base HTML file doesn't).

It comes at a cost though:

Your app must be able to run in NodeJS. Depending on your host provider, architecture…, it may not be possible. One thing is sure, your architecture will be more complex if you go this way.
Your application must be designed to run in the browser and in NodeJS (code that can do that is said to be isomorphic). But you cannot code as if you were only supporting the browser. For instance, you cannot access the DOM directly since it doesn't exist on Node, you must go through an abstraction which should be provided by your framework. I think making your app run correctly in the two environments will be one of the most challenging part when doing SSR.
If your app takes time to start after the HTML is sent to the user, you will have to capture events (like button clicks) to replay them once your app has started. And if it takes really too long, I think you defeated the point of SSR and displaying fast an usable app to your users is worse than starting with a spinner that at least indicates the app is loading.
Since the code will be run many times on the server, you need to pay attention to:
- Not use setTimeout since it can block rendering until the timeout has expired or use setInterval because intervals may pile on causing memory and performance issues (this can be mitigated if you don't forget to clean them correctly).
- Don't update prototype of global objects.
- Call required cleanup functions (like unbind, detach, unsubscribe in Aurelia) to clean the memory of the process and make sure page renders don't leak memory.
The response time can increase if the server is busy and it will be harder to support high loads: without SSR, most of the work is done on the client side, the server that serves the app only has to supply static files, so the server will handle high loads better. If you have many or big API calls to populate the content of the page, it may be a problem in both cases though. But keep in mind it's another potential bottle neck.

So depending on what you do and what your stack is, it may not be possible, very hard or not worth the cost. But if you can do it, it can be a huge plus to avoid major pain points. Small disclaimer to conclude with this section: although SSR looks mature for all the major frameworks I have never used it in production yet, so I have no idea of what other issues you can encounter if you go down this path (I did test it though).

Pre-render

One alternative to SSR, is pre-rendering. It's almost the same except instead of rendering the page when we ask it, we pre-renders all the public pages of the website, save the result and serve the result directly when requested. So it should be faster and reduce server load since we mostly server static files on the first request.

I haven't tested this, but I can guess that if it is done properly, when you app boots, you can override part of it (let's say to display a username) just like you would have with a normal site. It may flicker a bit while you update the pre-rendered version though (but again, just like on a normal site). I also guess you can make sure only anonymous users get the pre-rendered pages to avoid this altogether.

I'll also note that this technique can work with SSR: you rely on SSR to pre-render your website and you then send this pages to your users.

There are also services that allow you to do just that like prerender (which also has an OpenSource version), bromdone, getseojs or seo4ajax (and probably many more). Even if they are paid services, it would reduce the complexity of your infrastructure since you won't have to maintain servers for that. From what I understand, all these services work more or less the same way: they launch a headless browser to render your pages and save the results so they can serve them later (which mean you don't have to support SSR to use them). When a request is made to your server, you choose whether the response should come from the pre-render service or not. Meaning that for your users, the content will always come from your servers and domains. If the content served by the service is close enough of the content without it, google won't consider it like cloacking and penalize your SEO (both content should be the same anyway). Google even recommends this technique if your site uses technologies that search engines have difficulty accessing, like JavaScript, images, or Flash (to quote their page about cloaking linked before).

So when would we?

Now that we've discussed in detail when not to build a SPA while giving nuance to each counter arguments to correctly understand whether or why it is valid, let's review some arguments in favor of building one (yes that's way shorter, but it doesn't mean the advantages aren't worth it):

Your app needs a complex and interactive user interface or state that must be persisted across many pages. The result will be way more simple in the end if you build a SPA than some kind of hybrid stuff with a static page that will need to interact with JS. That's the main reason why you would build a SPA and still accept the drawbacks of this method.
Your team is already so familiar with JS and SPA (and eventually SSR), that you know in the end it will just be quicker this way.
You must expose a public API. You can then reuse it directly in your SPA avoiding duplicating some work to get the data on the site.

I'd also like to point out:

Your life will be easier if you can update framework and build tools frequently and if you use supported tooling (like create-react-app). They evolve fast and bring enhancement frequently.
Building a SPA is not required for a PWA. See for instance my article about PWA and Django where I transform a normal Django site into a PWA.
You should uniformize conventions. This can be a function that convert snake case into camel case (and reciprocally) so you don't have a mixed conventions in your frontend (or in both your frontend and your backend).
You can still handle style with CSS (or SCSS) files, you don't have to go down the CSS in JS road.
SPA can handle translations and internationalization. The framework you choose should have libraries to help you do that.

How about hybrid?

You may already have a website and need to add interactivity to it. Should you rewrite everything? Probably not, it would take a lot of time and your site already works. If your pages are private and don't require SEO, maybe you can convert some pages to "SPA" (you won't get an app, just a page but it will be close of a SPA for the principles used).

If not, you can include interactivity to part of a page to build what I'll call an hybrid app. This is typically done with React or Vue.js because they have a much smaller footprint than the other frameworks. This is because they do less things and are only concerned with rendering. It also means you may need more extra libraries to do what you want than with frameworks like Angular or Aurelia.

Will the result be a monstrosity like most hybrids in fiction? Not necessarily. Here are some tips that can help you:

Use CSS and vanilla JS if you can since this will be less complex. Just keep in mind this will only work for the most basic cases.
For more advanced cases, you will need a framework like React or Vue.js. One important thing to remember, is to always use the framework for rendering and handling events: it can react to all events you'll need (click, form submit, hover…) and conditional hide/display something. Don't interfere with it, if you do, you will run into trouble and hard to correct bugs (as I pointed out earlier). Once you are in the framework, you use the framework.
You can pre-render the content of the app in your backend with your backend templating language if you don't wish to display a spinner or if you wish to display content faster or to support SEO. Just make sure the app loads fast so everything works when your users try to interact with the page. Also note you will have to duplicate the display and probably part of the logic in the backend templates. Make sure it's worth it.
Use JSON for communication with your backend if you can. It will be easier to read and debug when a problem arises.
Display a loader and handle errors (eg display a message to the user) when things go wrong as you would in a SPA.
Rely on your backend for security: don't try to use tokens, just go with cookies and session authentication just as you would if your were building a "classic" site. Don't disable CSRF protection!
If many small independent JS applications need to communicate on the same page, you can create a global store to allow them to share the same source of data and interact with each other.
If you are in this situation, your team is probably mostly backend or full stack developers. Don't hesitate to train them so they are efficient with the frontend tech you choose and to rely on a designer for design and UX.
By all means, avoid jQuery in the JS apps. As I said earlier this can lead to issues when interacting with the framework.
Please don't build a hybrid with jQuery, it's awful to work with and thus can hurt hiring and developers retention. jQuery can still be used if your needs for interactivity are low, in my opinion, but be ready to move away from it if your needs grow bigger and to rewrite part of your app.

If you are starting a new project and intend to build a hybrid instead of a SPA (or a classical site), please:

Think again, you may need more interactivity than you initially though and a SPA may make things easier and more simple in the long run despite a slower start.
Pay attention to the skills in your team and train/prototype if needed.
Don't underestimate the work of maintaining two templates if you need SEO (and don't go for CSS in JS since you will need to duplicate the style in CSS for your main templates to render correctly). This of course only apply if you don't use pre-rendering.
SSR can be harder to achieve: depending on your stack, you may need to launch another process from your main web process to execute the JavaScript. This may not be possible nor desirable. Pre-rendering should work exactly like with a SPA though, making it a very good solution.

Wrapping up

To conclude:

Whether to build a SPA highly depends on your team and requirements. There is no general answer.
Please don't make a SPA only because it's popular or that's what trendy startup or big companies like Facebook or Google do. This can turn out to be a disaster for you. Make sure to understand the tradeoffs involved.
On many occasions, a static website (ie a site only made of static pages that can be served directly by a web server) like this blog can be the best option. There are many generators out there to help you build one with plain HTML file or markup format like markdown. Jekyll, Pelican and Hugo are popular solutions for this.
Don't use jQuery for complex use cases, we have better solutions now (mostly if you want to do big DOM manipulations).
Don't try to make jQuery (or vanilla JS) interact with the framework. Rely on the framework all the way!
Keep in mind that simple is different than easy: something hard now can turn out to be simple in the long term. So learning how to do SPAs and building one, although harder at first may be the right option in the long term.
Whatever you do now, learn frontend dev (or backend dev if your are a frontend person), you will need these skills to at least help make the best decisions (like choosing when to do a SPA) and to interact with the other side of the spectrum.
You probably will need more JS and interactions than you initially think.

To give a small conclusion about the project I work on: all in all, the fact that it's not a SPA is not a big deal. I have issues with some pages true, and I think we should use React more in some cases and rewrite some pages in React. But I still value Django and its ecosystem for the type of site I'm currently building. I think it gave us ease and speed to develop the site. So I think for us, a hybrid is a good choice: it gives us the possibility to add JS interactions where we need while preserving SEO. I just wished we went for React on some pages that requires lots of interactivity instead of jQuery. Since we don't need much interaction on most pages, all in all this works well. And we do have a SPA for some back office stuff where it makes sense.

If you have a remark, a question or just disagree, please leave a comment below.

Small frameworks list

Here's a small list of frameworks (at time of writing in June 2020) that you may want to look into to build a SPA:

ReactJS: Most likely the most popular choice right now. While it was initially design to be included in a "static" page to add some dynamism to it, you can build full SPAs with it and even mobile apps. React itself is quite small so you will probably need to add some other libraries to extend its capabilities a bit. React can be added incrementally to you site to add dynamic parts step by step.
Vue.js: Another very popular option. Very similar to React in principles (small, incremental) with a different approach that you may prefer coming from a "traditional web development" background.
Angular: While the first two are relatively small and can be added incrementally, Angular is a big framework designed to build full SPAs. I don't think you will be able to easily enhance an existing static website with it. While its size may be daunting, it is complete and you probably won't need much additional libraries with it. Furthermore, its tooling is mature and able to reduce the size of your code to only include the code you actually use to keep the JS you send small. Given its approach and use of TypeScript, it might be the right choice for you if you come from the Java or .NET world.
Aurelia: Not as popular as the other options, but I think it is a sweet spot between the completeness and complexity of Angular and minimalism of React or Vue. Like Angular it is a complete framework designed to build SPAs, but it has a good focus on web standards and tries to get out of your way so you can write standard JavaScript or include standard JavaScript libraries as easily as possible. Which makes it easier to learn and develop with (in my opinion at least). It's my go to choice for my personal projects.

Resources

Some interesting resources I read to help me improve my opinion on SPAs:

[1]	You may say React and Vue are more libraries than frameworks. I agree, but I'll keep calling them frameworks to ease writing since I also want to include SPA frameworks like Angular or Aurelia.

JavaScript fatigue

2020-05-31T00:00:00+02:00

Today, I'd like to dig into something that has been bothering me lately: why I often feel tired of JavaScript (and even sometimes programming in general). I'm not alone in this, there's even a name for this JavaScript fatigue. Here I'll give my personal thoughts on this (and probably rant a bit).

As developers, we do live in a fast moving world. It's even more true in the world of frontend development and JavaScript: it seems a new framework or practice appears every day and everything seems to force us to know about it and to learn it because it is so new and so cool and so much better than everything else that came before it.

This doesn't come from nowhere: with the rise of NodeJS and interactive apps, JavaScript is more and more used and so its community is expanding. Fast, very fast. This means the number of developers using the language, contributing to it and its ecosystem is increasing too. Quickly, very quickly because of the need for these skills and the "hype effect".

Add to that the fact that not so long ago (before Chrome and V8 and the evolution ES2015/ES6 of the language), JavaScript was still lacking what I'd call "important features to be productive" and was seen more as a side language to add some interactions in the browser. So there were some frameworks but nothing that got a real big momentum and pretty much everything still had to be built. This means, in this space, we are all using recent tech. This is quite unlike what we had (and still have) in the Python world for instance with frameworks like Django (first released in 2005) or Flask (first released in 2010) or in the Ruby world with Ruby on Rails (first released in 2005). Even if nodeJS itself was release in 2009, it takes time for an ecosystem to raise and mature with all of its libraries and frameworks and then for developers to adopt it (beyond the early adopters of course). In the frontend side of things, I think it really started with AngularJS in 2009 which was the first attempt (that I know of at least and that really became popular) for a framework to allow you to build a true SPA that became really popular. It was then followed by React in 2013, Vue.js in 2014, Angular in 2016 and Aurelia in 2016 to name only the most known. They are of course many others, most of them will probably remain experiments.

You can object that some of the projects I listed earlier are more or less as recent as the JS ones, after all both AngularJS and NodeJS were first released in 2009. However, they have share an important difference: they were built on concepts and rely on technologies that are much older, mainly:

The MVC model which dates from 1978.
SQL which dates from 1974. Of course both PostgreSQL and MySQL are more recent: 1996 with sources in the INGRES project way back in the 70s and 1995 respectively. But since they are at least 25 years old, they are kind of venerable now.
Python or Ruby which date from 1991 and 1995 respectively.

And even if the projects can be new, they are based on "old" tech and concepts that matured before these projects came to be. And they are more or less 10 years older than most framework we use today in the frontend. These concepts evolved and were refined over time which took time. We were able to test many possibilities, to fail, to learn before getting where we are now: mature stuff that works in many cases and are stable.

Contrast this with Express.js which was 1st released the year after Node was or the NoSQL movement which really starts near 2009 (MongoDB was release this year and the first NoSQL convention happened), like NodeJS and AngularJS. Or JavaScript itself which added "basic" feature only recently (like the import system in 2015). You can of course argue the root of NoSQL is older but my point is it gained momentum and was formalized very recently (at least in the form we know and use).

So, I feel we are still missing maturity for frontend tech. The technologies are all recent, evolving fast and don't seem close to a stabilization (I hope this will change in the future). The fact that there is so many developers using it and wanting to bring their ideas to it doesn't help either.

Which leads to the situation I was describing in introduction: we see a lot of experimentation, technologies move fast, become deprecated quickly and we have a lot to learn just to stay up to date. This can lead to a feeling of exhaustion because no matter what we do, even if we try to only stay relevant, we still feel out of date without any clue to what we should learn, what would be most valuable, what will last. That applies to tech that have been around for some time like React: they recently introduced hooks which are a brand new and very different from what we did until then to manage state. And we are expected to by peer pressure (on social media or internally where we work) and to pass interviews (mostly at the latest trendy startup but that where the cool kids go, isn't?). And if like me you like learning things and experimenting, this can be daunting and exhausting.

Of course, they are things we can do to mitigate this:

Only take a close look at tech that is popular and that passed the trial of time. This way, we only look at tech that had time to mature. But it can already be perceived as obsolete and it can still be evolving (I'm thinking of React Hooks that were introduced recently).
If we want to look at some latest stuff to have fun, we can, just do it only once or twice per year and don't dive all in since we probably won't use it anyway (or it will just disappear). And remember that we are learning it for fun not to use it in production.
Reduce our usage social media where we can be submerged by the hype wave and where it looks like everybody gives their opinion even if they haven't though on the matter and where people can be quite enthusiastic about anything, making ou feel bad your are not and giving us the false impression the thing we are looking at is very popular and that we just must use it (and that everybody already is). The problem is also similar at conferences but I can't really advise not to go at one or watch their recorded videos because they are a magnificent way to learn. And they have a big difference: when you go to a conference, you can talk to people to get a more nuance picture.
Remember about cognitive bias: speakers are most likely normal folk too and many of them are just average. That's just statistics. Speaking at a conference doesn't make them heroes we must follow no matter what.
Remember even those cool folks have to maintain legacy code and don't always do the cool stuff they talk about. But that's easier said than done. And, like most of us, they'll talk about the cool stuff they are doing.

That's good and that can help. It can still be hard to do, if like me, you like to experiment and learn new things. Since as developers we all work with our brain and are attracted to technology, I think it's the case for many of us.

But it's only part of the solution and making it only rely on people, on us personally, can make us feel bad and worsen the problem. Even if that's what our culture incite us to do. I want to emphasize that it's also a global context. And the big pression to follow the hype will still be there and each time we'll go to a meetup or a conference. It's also present in how we make job interviews. And if we don't work in a big company or a startup where we can test, experiment and work the latest stuff during work hours, it's even harder because we must do it all on our valuable spare time. So I feel like we need to collectively do something and stop chasing the brand new thing, for us and for others. Here is a small list of things we can do:

Stop reinventing the wheel because we didn't to it or because we think we could do it a bit better.
Stop blaming and criticizing the old tech and the users of old tech:
- The tech may be old but it doesn't meant it's irrelevant. We can even learn things from it and become better developers (and dare I say like it?).
- It's maybe a much better match for what they are doing than the shiny stuff (whose main quality is often to be new and shiny).
- The user of old tech may just not be following the hype and be convinced for very good reasons that the tech they use is good for their problem. Even if it has some defects and it's not prefect, the issues and imperfections are known and very often can be dealt with. They know how to be productive with it and keep the software going with few bugs.
- They may also not have a choice and have to maintain a big legacy project.
- There is a place for proven tech even outside legacy systems.
- Please note that I'm not advising never to learn, experiment or move forward. Don't get stuck into your habits: think about them and make them evolve when needed. Don't choose an old tech because you're used to it but because you know it is good. New tech can spawn great values. This implies to read books and experiment from time to time. It just doesn't need to happen at a fast pace and for everything that happens out there. Focus on the basis and high level architecture, it can be reused across languages and tech so it always pays off.
Accept we are no all working at Google, Facebook or any other very big player. We don't need to work exactly like they do, with their tech and their procedures.
- We will probably be more productive if we don't. Most of what they do is design for the enormous load they have and for the huge number of developers that work for them.
- Even in a startup with big ambitions, we are likely to miss something that will prevent your architecture to scale correctly and force you to rewrite it. Even if the software was written with all the latest shiny tech and separated in µ-services. And let's face it, the startup is very likely to disappear anyway. So maybe we should try to build a product that works and get customers before thinking about scaling at the level of Google.
- We will probably apply the visible stuff without all the technical rigour that comes into it. This will result in us building a big mess of little balls of mud that will shine it meetups but prevents us to deliver.
- So maybe try to build a good product following proven code advise (like those described in the book Clean Code) and it will be easier to you to pivot or scale should it be needed. And if we have to rewrite it, do it. If our software was done properly, it will take time, but it will be doable (normally at least, we all hope and we all fear the tech debt ^^).
- Old stuff is always a pain to maintain anyway, whether written in a shiny new µ-service architecture or in a big monolith.
Accept that the hype is the hype and probably isn't mature enough to be used in the real world just yet for most companies.
- Most of us can't afford to rewrite everything each time something new – a framework or a version of a framework – comes along. Don't blame anyone for not being into it. They may have other priorities and they don't have the resources (the army of developers and the billions of dollars) to make it happen anyway.
Find a job that allows you to learn and stay relevant during work hours to better enjoy your spare time.
Accept that old tech with known tradeoffs can be a good thing.
Don't forget that most of the common problems with "old" can be mitigated because we understand the tech so well with decades of experience with it.
Don't forget old tech evolves. For instance recent versions PostgreSQL has astonishing support for JSON and can help you apply schema migrations without downtime. So bringing this as a hammer argument for MongoDB may not be relevant.
Don't forget some issues with old tech comes for a particular framework, lib, or software. For instance, PostgreSQL supports array and has for a very long time unlike MySQL (which as far as I know still doesn't). Arrays are even in the SQL standard, so criticizing SQL for this is in fact criticizing MySQL. Take care of this. And they too can evolve and provide those features, so you may not need change them.
We need time to understand the tradeoffs of new tech. Embracing it too early can lead to bigger problems in the long run that using old and proven tech. It is cool that some can do it and find these issues. Not everybody can do it though, and for many solutions that brings money to a company, it's not a good fit. And we may encounter problems nobody has before and nobody knows how to solve which may put us in a delicate solution.
In the real word, behind the buzzword, many are probably not doing like the Big Ones and building a big ball of µ-services or fail at understanding a problem correctly even if they proudly display their state of the art practices. Even in the big ones, I sure the reality is a bit less interesting than what they present at conferences.
Stop blaming people when we ask genuine questions about new tech because we don't understand their tradeoffs yet and don't fully grasp their advantages. It takes time to correctly understand a subject and build an intuition about when, how to use it and whether it's a good fit for our project and my team. Just replying with You're just resistant to change is neither helpful nor mature. They can be valid reasons for a change (even proposed by Facebook) to be bad so don't use the argument of authority It's made at Facebook or Google, try to convince us with good arguments instead. And don't forget the goodness of a change may not be obvious at first.
Don't forget that something good for one team may not be for another. For instance, take µ-services: they are very interesting to structure big teams. If you're just 2 or 3, I think it's overkill. Or write a application in a language nobody knows in your team because you think it may be a good fit for it. But if we think about it, the advantages it brings are not that obvious. The languages you know may not be the best theoretical fit, but you know them and the ecosystem around it, which gives it a very clear advantages. So unlike it is obviously not a good choice, you should use it. Making your team to a new language and learn about its ecosystem takes time, don't forget it!
Must stop idolizing Facebook, Google, Netflix and co. They can do shitty things and mistakes or at least things that don't match our needs and our teams. Stay critical. Don't use in a discussion That's what Google does or It's made by the bright folks at Facebook. Even bright teams can make mistakes and this tech or this way of doing things may not be a good fit for your team. See my point above.
Can use other languages or framework other than the most popular ones and still be awesome persons and developers. Not everybody has to use JS everywhere every time. For instance, I like Python, some may like Ruby, others Java. Find something you like, use it in your day job if you can. Diversity in languages and frameworks is a good thing. Learn something else than what you are currently using so proudly. It can help you get a new perspective on things and make us better developers.

Thanks for reading! Keep coding and learning and be nice and empathic ;-) Don't hesitate to react in the comment.

Feedback after switching to openSUSE Tumbleweed

2020-05-15T00:00:00+02:00

This article is a follow up to Installing openSUSE next to Fedora with BTRFS where I detailed how I switched to openSUSE Tumbleweed. In this article, now that I have been using Tumbleweed for about 2 months and a half, I'll give some feedback on my experience.

The issues I had:

For some reason, I lost an EFI file used by Fedora to boot which prevented me to run my Fedora installation. I was able to restore it by copying another EFI file (mmx.efi) into the proper location.
Unlocking the LUCKS drive in Grub is slow. Although I find it's a bit faster lately (and it's not an issue with Tumbleweed per say).
Docker sometimes fails to pull images. It only happens in script, running docker pull SOME_IMAGE always runs correctly.

Unlike in Fedora, /tmp is not cleaned automatically after a reboot. You have to enable this manually by putting the content below in /etc/tmpfiles.d/tmp.conf (see this thread for the solution and the reasoning behind it):

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See tmpfiles.d(5) for details

# Clear tmp directories separately, to make them easier to override
# SUSE policy: we don't clean those directories
D! /tmp 1777 root root -
q /var/tmp 1777 root root -

Updating the kernel to a new minor version (eg from 4.5 to 4.6) can break the Nvidia drivers and thus your display. The patched driver comes relatively fast but it was a bad surprise when it happened to me! I hope the process will be smother in the future. It was also the occasion to experiment booting from a read-only BTRFS snapshot. And it worked well!

Despite these issues (for which I have workarounds anyway), I am very pleased with my experience. The update process works well (with the exception of Nvidia drivers as pointed above) and the system is stable. So I decided to switch all my computers to it (I switched to a rolling release to avoid Fedora updates after all, didn't I?).

I must also say that YAST (which acts as a configuration panel for you computer from firewall to installing packages) is well done and a pleasure to use. No need to search/install different tools, just use YAST!

I also find the use of openSUSE of BTRFS snapshots to be really cool. I haven't used them much since the system is stable and I haven't encountered issues with updates (except once with Nvidia). But it's still a nice thing. I'll just note that while installing openSUSE to another computer, I made a big configuration mistake that locked me out of the system. Thanks to the snapshots and the rollback mechanism, it wasn't such a big deal (because yes, /etc will be in the snapshot too). Without it, I might have needed to reinstall everything. So I don't need them often, but I'm really glad they are here when I do.

This post wouldn't be complete without some observation on KDE. As a long time Gnome user, I also decided to switch to KDE to see how it goes (and while your changing, why not change more?). I am pleased with the change and was able to adapt to KDE very quickly (despite many years of using Gnome). I do appreciate the fact that many things are integrated by default (like the clipboard manager or media player integration) whereas you need extension in Gnome to do this. I also really like the fact that the command that ask for the root password is mentioned on the popup that ask for it. I think it is a big plus for security.

I had some issues though, here is the list and how I solved them:

I cannot configure my gmail account to use it easily with KMail. From what I read, I have to enable IMAP support in gmail. In Gnome this is supported by default (calendar works fine though). Since I only have my professional mailbox in it, I can just keep gmail opened in a pined tab. So not a big deal.
Since I use the same user on both Fedora and openSUSE, when KDE launched it changed some configuration and now the close, maximise and minimise icons looks a bit funny on Gnome.
I use the bépo keyboard disposition (which was made specifically for French). It was really easy to configure this keyboard disposition in the session but harder to enable it on the login screen (although it may come from the fact that openSUSE doesn't support this disposition during install unlike Fedora). Luckily, this post on the ArchLinux forum had the solution: update /etc/X11/xorg.conf.d/00-keyboard.conf with the content below.
```
Section "InputClass"
    Identifier         "Keyboard Layout"
    MatchIsKeyboard    "yes"
    MatchDevicePath    "/dev/input/event*"
    Option             "XkbLayout"  "fr"
    Option             "XkbVariant" "bepo_latin9"
    Option             "XkbOptions" "terminate:ctrl_alt_bksp"
EndSection
```
I had to create a script in .config/autostart-scripts/ssh-add.sh with the content below to correctly pre-load my SSH keys, something that worked by default on Gnome. The script is based on this article from the ArchLinux wiki.
```
#!/usr/bin/env bash

for key in $(ls -d ~/.ssh/* | grep -v pub | grep -v authorized_keys | grep -v config | grep -v known_hosts | grep -v seahorse); do
    ssh-add -q "${key}"
done
```

I also find the combo KDE + openSUSE to need less RAM that my Fedora + Gnome.

So, to conclude, I am very pleased with the change and don't regret doing it.

PWAs and Django

2020-02-29T00:00:00+01:00

In this article, I'll guide you to create a PWA with the Django framework. All the code for this project is available under my gitlab account. You will find in each sections links to the relevant commits to help you follow. If something is not clear or if you have a question, please leave a comment below.

Updates:

On 2020-10-12: as pointed out by Ernest by email, the workbox version was missing the version number in the ServiceWorker. It is required to update it and recache pages. It is now fixed.

Sommaire

Prerequisites
PWA basics
Building a PWA with Django
Conclusion

Prerequisites

I expect you to be at ease with Django and to have some knowledge of modern JavaScript (ES6+). You don't need much JavaScript: basic knowledge, fat arrow functions and Promise should be enough. No pre-requisites on PWA are required, this article includes a section to demystify them if needed.

PWA basics

Simply put, PWAs are "installed" web pages: when you browse on a website that is a PWA, you will be prompted to install it. If you accept, you will see the icon of the APP on your home screen (like any other app) and if you press this icon, the PWA will launch in full screen mode just like a native app. But you didn't go to the app store to install it. It's still just a website.

As a developer, building a PWA means:

Your users don't have to install it: they can use the app directly on the website without installing anything and you'll still be able to provide them advanced features like caching and offline support.
You will have more control on when you can update it: no need to submit it to the app store and wait for validation. Just push files to you server. The update will occur the next time your users reopen the app.
You are not bound by the rules of any app stores.
The app will work on any device with a recent browser. Chrome has the best support for PWAs right now, Firefox is quite good but Safari support can be an issue (even if it's getting better, see this article for instance).
The app can be progressively enhanced (eg by adding better offline support or better caching).
You show notifications the user with the Notification API and the Push API.
You can access some native features (like the camera or the microphone). If you want to access all native features or require very high performance (for a video game for instance), you still need to do a native app.

Here are the basic components needed to make a PWA:

manifest.json: it's a JSON file describing how the app must be displayed and installed. You'll use it to specify the name, icon, orientation and some display properties of your app. See here for more details. It is required to make your app a PWA.
ServiceWorker (SW for short): it behaves like a proxy server between your app and the network. It will keep working even when the tab is closed. So it will allow you to:
- Have caching and offline support.
- Sync things in the background.
- Subscribe to notifications of the Push API and display them to users.
Please note that:
- It only works over HTTPS (except for localhost so you can develop without certificates).
- It cannot access the DOM.
- It may apply to all or only a subset of your site.
- It must be registered to have an effect.
- It will only get updated if its content changed and after the tab is closed and reopen (unless you use the dev tools and force it to reload on change but that's not an option for your users, more on that below).
See here for more. It is required to make your app a PWA.
indexDB: where you put data to exchange with the SW (SW cannot access localStorage). See here for more information.
Many other APIs I'll just list here so you know what you can do:
- Notification API to display notification to the user.
- Push API to receive messages from a server.
- Many more: geolocation, camera access, …

Note

This section is heavily inspired from the similar section from my article Writing a PWA with Aurelia.

Building a PWA with Django

Create a Django website

Obvious isn't it? For the purpose of this tutorial, I created a basic site with Django and a few pages. You can of course follow the following sections to add PWA support to your own site.

The repository with all the code is located here. To use the project, you will need to create a virtual environment and install Django in it. The code was tested with Django 2.2 but it should work correctly with any more recent versions of Django. You can install the project with Pipenv:

Installation of the dependencies: pipenv install
Run the default migrations (this will create a SQlite database): pipenv run python manage.py migrate
Start the server: pipenv run python manage.py runserver

The commits associated with this section (nothing relating with PWA so far):

Adding a manifest.json

As said in the previous section, the manifest is a JSON file that will allow your app to be installed. Since its content shouldn't change often, I think we can include in in the static folder with hard-coded values.

You can find Django apps that will allow you to configure this with settings variable but I don't recommend them. The file is simple and rarely changes, so I don't think it's worth adding a dependency to you project just for that.

In my case, it is in djangopwa/static/manifest.json with this content:

{
    "name": "Django PWA",
    "short_name": "djangopwa",
    "start_url": "/",
    "scope": ".",
    "display": "standalone",
    "background_color": "#FFF",
    "theme_color": "#493174",
    "description": "Test app for Django and PWA",
    "dir": "ltr",
    "lang": "en-US",
    "orientation": "portrait-primary",
    "icons": [{
        "src": "/static/icons/aurss.96x96.png",
        "type": "image/png",
        "sizes": "96x96"
    }, {
        "src": "/static/icons/aurss.512x512.png",
        "type": "image/png",
        "sizes": "512x512"
    }]
}

We can now update our base template to load it on every page:

<!-- This is to defined the viewport and how the app must behave when on full screen. This is required too. -->
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="manifest" href="{% static 'manifest.json' %}">

While we are at it, we can also update our base template to add better compatibility with IE11 and Safari by adding:

<meta name="theme-color" content="#493174">
<link rel="apple-touch-icon" href="{% static 'icons/aurss.192x192.png' %}">

Now your app can be installed on the home screen but it's not a PWA yet. It's missing a ServiceWorker. That's the subject of the next section.

The commits associated with this section:

Adding a basic ServiceWorker

The service worker is defined in a JavaScript file that must be served at the root of your application. If it is under /static its URL will be something like /static/sw.js. It means that the scope of your ServiceWorker (ie the pages it is activated on), must be under /static. This means that it won't be enabled for our whole site (which is what we want, limiting it to /static makes it unusable on standard pages so it would be quite useless). So it must be served from something like http://wwww.example.com/sw.js.

To achieve this, I created a simple TemplateView that will serve a file named sw.js. The file is created from a template. It will also be interesting to have a template for this later.

We do this by adding the following code into our main URL file djangopwa/urls.py:

Then, in djangopwa/templates/sw.js, we can add the basic structure of our ServiceWorker:

// Bump this version number each time a cached or asset changes.
// If you don't, the SW won't be reinstalled and the pages you cache initially won't be updated
// (by default at least, see next sections for more on caching).
const VERSION = '{{ version }}';

self.addEventListener('install', (event) => {
    console.log('[SW] Installing SW version:', VERSION);
});

The commits associated with this section:

Is this a PWA?

We should check shouldn't we? The easiest way to do this is to open Chrome and open the dev tools (although other browsers support PWA, Chrome right now provides the best developer experience with its Audits and Application tabs). Open the Audits tab, make sure Progressive Web App is selected under the Audits section and run one. The only problematic item should be Does not redirect HTTP traffic to HTTPS which is normal since we are running the application locally without HTTPS.

So yes, we successfully made a PWA. A very basic one we will improve in the upcoming sections. And you know the last item you'll have to respect before putting it into production: serve it under HTTPS and make sure all HTTP traffic is redirected to HTTPS.

Before going further, I'd like to present to you the Application tab of the Chrome dev tools. When working with a PWA, it will be the most useful tab. It allows you to:

Inspect the manifest.
Manage the service worker. I suggest you tick Update on reload to update the Service Worker each time it is modified and you reload the page. If you don't, you will have to close the tab and open it again for the Service Worker to update which is cumbersome during development. If you have issue with this option, disable it, close the tab and reopen it to trigger a standard update of the ServiceWorker.
See the content of the various storage.
Inspect and flush the cache storage where the Service Worker will save the content of your HTTP requests. This is very handy when testing.

Add basic caching

Now we can cache some pages and serve the pages from the cache instead of from the network. The pages will be added in the cache when the SW is installed (they will be pre-cached) and won't be updated afterwards. Pre-caching is interesting for pages you know will be accessed or for static assets. We will handle updates later.

To do so, we will update the function that reacts to the install event (which is triggered when the SW installs) to pre-cache some pages:

const VERSION = '{{ version }}';
const staticCachePrefix = 'static';
const staticCacheName = `${staticCachePrefix}-${VERSION}`;

event.waitUntil(
    caches.open(staticCacheName)
        .then(cache => {
            console.log('[SW] Caching app shell');
            // All the pages supplied here will be set in the cache when the service worker installs.
            // If the service worker is updated, these pages will be updated in the cache.
            // So it can be a good idea to inject the version of the project in the template
            // to be sure the SW will be updated (and thus the pre-cached assets) when the project is.
            cache.addAll([
                '/static/manifest.json',
                '/',
                '/offline/',
            ]);
        }),
);

Then we must add an event listener for the fetch event so we can respond with the pages from the cache. This event is triggered each time the browser tries to do an HTTP request when you click on a link or use the fetch API (don't use the xhr object or jQuery.ajax, it doesn't fit well with SW because the fetch event is not triggered). We can do this by adding the code below in the sw.js file:

self.addEventListener('fetch', (event) => {
    // Let the browser do its default thing
    // for non-GET requests. It's not safe to cache them anyway.
    if (event.request.method !== 'GET') {
        return;
    }

    event.respondWith(
        caches.match(event.request)
            .then((response) => {
                // If we have the response in the cache, we return it.
                // If not, we try to fetch it.
                return response || fetch(event.request);
            }),
    );
});

Pre-caching is nice but you will most likely have many dynamic pages on your site and you can't pre-cache them all (most of them won't be accessed by your user anyway). But you may still want to cache them. If you don't want to cache all of them, you can rely on event.request.url to test if the page must be cached or not. By slightly updating the handler above, we can cache dynamic pages:

// Move this line with the constants at the top of the file.
const dynamicCacheName = 'dynamic';

event.respondWith(
    caches.match(event.request)
        .then((response) => {
            console.log(`[SW] Requesting ${event.request.url}.`)
            // If we have the response in the cache, we return it.
            // If not, we try to fetch it.
            if (response) {
                console.log(`[SW] Served response to ${event.request.url} from the cache.`);
                return response;
            }

            return fetch(event.request)
                .then(res => {
                    return caches.open(dynamicCacheName)
                        .then(cache => {
                            // We can read a response only once. So if we don't clone it here,
                            // we won't be able to see anything in the browser.
                            cache.put(event.request.url, res.clone());
                            return res;
                        })
                })
                .catch(err => console.warn('Warning: app is offline', err));
        }),
);

Tada, you can now cache any page. Be sure to use a different cache for the dynamic pages. It will be useful later when we will clean the cache. To test this, you can go to any URLs like /say-something/SOMETHING. If you use hi or bye you will see the page. Otherwise, you will see a 404.

This version has a problem though. It will cache all pages even if the request fails (with a 500 or a 404 for instance). So we will cache pages we don't want to cache. To avoid this, before return caches.open(dynamicCacheName), you can add the code below to answer with the raw request with the request failed:

const requestSucceeded = res.status >= 200 && res.status <= 300;
if (!requestSucceeded) {
    return res;
}

The commits associated with this section:

Add offline support

Now that we have setup caching, we can provide offline support to our user. If the network is down, we will display them a page explaining they are currently offline instead of displaying the error page from the browser. To do so, we will once again update the handler associated with the fetch event. We will replace the line .catch(err => console.warn('Warning: app is offline', err)); by the code below:

.catch(err => {
    console.warn('[SW] Network request failed, app is probably offline', err);
    return caches.open(staticCacheName)
        .then((cache) => {
            // If the request expects an HTML response, we display the offline page.
            if (event.request.headers.get('accept').includes('text/html')) {
                return cache.match('/offline/');
            }

            return Promise.reject();
        })
        .catch(err => console.warn('[SW] failed to get response from network and cache.', err));
});

Now, in the dev tools, go offline (you may have to un-tick Update on reload to witness the proper behavior and to re-open the tab). You can do this in the Network tab under or in the Application tab under Service Workers. Navigate to a new page (that's not in the cache, for instance /toto). You should see your offline page. Go back online, refresh the page and you should see the standard 404 page from Django.

The commit associated with this section:

Caching strategies

That's all very interesting, but right now, if a page of your site updates and it is in the cache, you must remove it from the cache with the dev tools and reload the page to see the update. Not very practical and none of your users will do this. So we need to tune our caching strategy. Let's detail them now so we know what to choose.

They are various strategies you can implement:

Network only: always go to the network, cache nothing. That's the default behaviour.
Cache only: always go to the cache, never reach the network.
Cache with network fallback: if the response is in the cache, use it. Otherwise, go to the network. That's what we are using now.
Network with cache fallback: try to use the network, if it fails, use the cache. If the network is slow, the requests will take time to fail so this will provide a poor experience.
Cache then network: give the response that's in the cache (if present) to answer fast then make an HTTP request to update the cache.

The strategy to use depends on what kind of app you are building, the type of content associated with the URL, the domain you are requesting (yours or someone else's)… I can't give you a general answer.

I can strongly suggest you to never cache any request that's not a GET request: if a request have sides effects (PUT, POST, PATCH), it's not safe to cache the results. It's common HTTP wisdom and practice. However, you can rely on the Service Worker to save the payload of a POST in indexDB when the network is down so you can do the request once the network is back online. I won't talk about it in this article. If you are interested, I have an example in a small app I built.

Here, I'll propose a slight variation of the Network witch cache fallback: we will try to reach the network to provide the most up to date version of the page but if it fails within a timeout, then we will provide the version from the cache. If the request were to succeed anyway, we will update the cache. It looks like a good compromise.

Before implementing this, I added a view that responds in a variable amount of time between 0s and 10s to easily simulate a page that takes time to respond. You can access it under /random-response. I also refactored the code of the Service Worker a bit to clarify the code and ease the implementation. Now each "step" is in their own functions: getFromCache, tryToFetchAndSaveInCache, fetchAndSaveInCache and provideOfflineFallback. You can find links to the relevant commits below, I won't detail in the article.

Now that it's done, I can create a new function networkThenCache that does what we wish to implement:

// In milliseconds.
const networkWaitTime = 2000;

function networkThenCache(event) {
    return Promise.race([
        tryToFetchAndSaveInCache(event, dynamicCacheName),
        new Promise((resolve, reject) => setTimeout(reject, networkWaitTime))
    ])
        .then(
            (response) => response,
            () => getFromCache(event).catch(() => provideOfflineFallback(event))
        );
}

With Promise.race returns a promise that will resolve when the first promise in the array we give it resolves. So here, it will resolve when we get a response from the network or after 2s with the response from the cache. If we don't have a response in the cache, we will display the offline fallback. It's not perfect: if you don't have a response in the cache you may want to wait more for the network to respond. But at least it's a start and you can adapt it to fit your needs.

We also have to update our response to the fetch event:

event.respondWith(
    networkThenCache(event),
);

Now if you go to /random-response, you will see (the time spent sleeping in the view is displayed in the console and on the page):

The offline page if the first response takes too much time.
The response supplied by the view if it answered in less than 2s.
The response from the cache if the view took more than 2s.

One interesting thing is that since promise chains can't be cancelled, if the network request do succeed, it will update the cache. You can test it and see in if you inspect the dynamic cache of the application.

The commits associated with this section:

Cleaning the cache

We have to kind of caches to clean: the static cache and the dynamic cache. We'll start with the static cache.

The easier way to clean it, is to remove all static caches that are not it. Since we made a clear separation between static and dynamic caches with a prefix, all we need to do is:

self.addEventListener('activate', (event) => {
    console.log('[SW] Cleaning old cache shell');
    event.waitUntil(
        caches.keys()
            .then((keys) => Promise.all(
                keys
                    .filter((key) => key !== staticCacheName && key.startsWith(staticCachePrefix))
                    .map((key) => caches.delete(key))
            )),
    );
});

Cleaning the dynamic cache is a bit trickier. We would like to remove the oldest entries first. You can access all entries in the cache with cache.keys() but you don't have access to the time the item was added to the cache. And you can't rely on the order because it appears to be somewhat random. Furthermore, it is impossible to read headers of a request once it has been cached. Even before, only a small subsets of headers (those viewed as safe) are available. See: this and this.

What can we do? They are several ways to walk around this:

Just remove entries until we have an acceptable amount of them in the cache. That's quick and dirty but it's really simple to implement.
Until a service worker is updated it keeps all objects in memory (you can't see this behavior if Update on reload is ticked). So we can create a map mapping a URL to its response timestamp and use that to find the response to delete. We will also have to delete entries in the dynamic cache we don't know about (or delete the dynamic cache entirely upon installation). From what I read here, this is how the workbox plugin (a plugin to ease working with ServiceWorkers by google, I talk about it in the Using Workox section) actually works.
Store the mapping information in indexDB so it persists between updates of the ServiceWorker and delete the entries based on that.

As much as I would like to introduce indexDB in this tutorial, I'll go for solution 2 which is more straightforward (and I think this tutorial is already long enough).

To do so, we will add a constant describing how much items we want to keep in the cache and create a map to keep the association between the requested URLs and the date at which they were seen:

const maxNumberItemsDynamicCache = 5;
const urlsToCacheTimes = new Map();

We now must update it each time we put a response in the dynamic cache and launch a clean:

cache.put(event.request.url, res.clone())
    .then(() => {
        urlsToCacheTimes.set(event.request.url, Date.now());
        return trimCache(cache, maxNumberItemsDynamicCache, urlsToCacheTimes);
    });

And we must add the function to clean the cache:

function trimCache(cache, maxItems, cacheTimeInfos) {
    if (cacheTimeInfos.size <= maxItems) {
        console.log('[SW] Nothing to trim from the cache.');
        return Promise.resolve();
    }

    // We sort all entries by descending dates.
    // We keep a slice of the maxItems more recent items.
    const urlsToKeep = Array.from(cacheTimeInfos.entries())
        .sort((a, b) => a[1] - b[1])
        .reverse()
        .slice(0, maxItems)
        .map(([url, _]) => url);

    console.log('[SW] Keeping in cache', urlsToKeep);
    return cache.keys()
        .then((keys) => {
            const deletions = keys.map(key => {
                if (urlsToKeep.includes(key.url)) {
                    return Promise.resolve();
                }

                console.log(`[SW] Removing ${key.url} from the cache.`);
                cacheTimeInfos.delete(key.url);
                return cache.delete(key);
            });
            return Promise.all(deletions);
        })
        .then(() => console.log('[SW] Done trimming cache.'))
        .catch(() => console.log('[SW] Error while trimming cache.'));
}

Note

You should be able to read the Date header from the response with res.headers.get('Date') if you prefer to rely on the date given by the server.

The commits associated with this section:

Suggestions to further improve dynamic caching

You can read the HTTP header from the responses to fine tune how you cache a page. For instance, we will not cache a page if it contains the no-cache value in the Control-Cache header. To do so, we will update the fetchAndSaveInCache function:

const requestSucceeded = res.status >= 200 && res.status <= 300;
const mustNotCache = res.headers.get('cache-control').includes('no-cache');
if (!requestSucceeded) {
    console.log('[SW] Request failed.');
    return res;
} else if (mustNotCache) {
    console.log('[SW] The page must not be cached.');
    return res;
}

You can of course use headers in more advance ways: to invalidate the cache based on what the response tells you to do or to fetch a resource directly from the cache if the header allows you.

Astuce

You can list all the headers of a response with Array.from(res.headers.entries())). This must be run before the response is cached (as stated above).

The commits associated with this section:

Caching pages with JS and CSS

So far so good. But our application is lacking styling and JavaScript! To illustrate how it goes, we will only add some CSS to the site. The process for JavaScript is the same.

Let's add some basic style in djangopwa/static/style.css:

h1 {
    color: gray;
    font-weight: bold;
}

.thing-said {
    color: red;
}

And load it in base.html:

<link rel="stylesheet" href="{% static 'style.css' %}">

If you reload the page, you should see all the titles in grey and the file style.css will be added to the dynamic cache. It's looking quite good by default.

Should we pre-cache it? It's a file that will be requested for every page and that's required on the offline page too. So it makes sense to pre-cache it. But this lead to the question: what should we pre-cache? Maybe it's time to answer.

The general advise I've seen is to pre-cache:

The app shell. That's the resources (CSS and JS) that are required to display the base of the application. They most likely don't change very often and they are required to display all the pages (including our offline page). This means these files will be requested them anyway. So we don't waste anything by pre-loading them.
The offline page because once offline we won't be able to load it! It also mean it's probably a good idea to keep it small and simple.

We could cache other things too. But I think for most cases that's all we need to pre-cache. We don't want to request too many pages or files: they will all be downloaded when the app starts. For our mobile users, we don't want to download too many things to save bandwidth, storage space and power. Plus, we don't know what the users will access so we may cache stuff that's not required. And for the normal pages of the site, we have the dynamic cache to speed things up.

Because these resources are pre-cached, we can switch to a cache only strategy: each time they are requested we load them from the cache. To avoid issues when you update them, you should make sure their names contain a hash of the file so we can keep them indefinitely. It's a good practice anyway so your CDN can cache these assets indefinitely too. This can be done easily in Django by using the proper static file storage: the ManifestStaticFilesStorage. You can enable it by adding in your setting file: STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'. You may want to disable this during development not to hit the cache and see the updated files.

Here is how the ServiceWorkerView and sw.js now looks:

class ServiceWorkerView(TemplateView):
    template_name = 'sw.js'
    content_type = 'application/javascript'
    name = 'sw.js'

    def get_context_data(self, **kwargs):
        return {
            'version': version,
            'icon_url': static('icons/aurss.512x512.png'),
            'manifest_url': static('manifest.json'),
            'style_url': static('style.css'),
            'home_url': reverse('home'),
            'offline_url': reverse('offline'),
        }

const appShell = [
    '{{ icon_url }}',
    '{{ manifest_url }}',
    '{{ style_url }}',
    '{{ home_url }}',
    '{{ offline_url }}',
].map((partialUrl) => `${location.protocol}//${location.host}${partialUrl}`);

Note

I could have used the static and url template tags directly but that would have required to add {% load static %} in sw.js. Since this is not valid JS, my editor report it as an error which I find annoying.

The commits associated with this section:

Using Workox

So far, we've all done everything by hand. It's interesting and it's a very good way to learn. However, it can take a lot of time. Here comes Workbox a tool made by google to automate much of the process. Google describe it as a JavaScript Libraries for adding offline support to web apps. If you search about PWAs you will probably hear about it as it's quite popular and feature complete.

Since the documentation is very complete, I'll just provide an example of ServiceWorker using Workbox with a CDN to help you getting started. It should provide more or less the same features than what we wrote by hand. It's much smaller though (and probably less buggy). The code is on the workbox branch of the repository.

importScripts('https://storage.googleapis.com/workbox-cdn/releases/5.0.0/workbox-sw.js');

const VERSION = '{{ version }}';

if (workbox) {
  console.log(`Yay! Workbox is loaded 🎉`);
} else {
  console.log(`Boo! Workbox didn't load 😬`);
}

const OFFLINE_URL = '{{ offline_url }}';
const appShell = [
    '{{ icon_url }}',
    '{{ manifest_url }}',
    '{{ style_url }}',
    '{{ home_url }}',
    '{{ offline_url }}',
].map((partialUrl) => `${location.protocol}//${location.host}${partialUrl}`);

// Precache the shell.
workbox.precaching.precacheAndRoute(appShell.map(url => ({
    url,
    revision: null,
})));

// Serve the app shell from the cache.
workbox.routing.registerRoute(({url}) => appShell.includes(url), new workbox.strategies.CacheOnly());

// Serve the other pages from the cache and make a request to update the value in the cache.
// Limit the cache to 5 entries.
workbox.routing.registerRoute(
    ({url}) => !appShell.includes(url),
    new workbox.strategies.StaleWhileRevalidate({
        cacheName: 'dynamic-cache',
        plugins: [new workbox.expiration.ExpirationPlugin({
            maxEntries: 5,
        })],
    })
);

// Handle offline.
// From https://developers.google.com/web/tools/workbox/guides/advanced-recipes#provide_a_fallback_response_to_a_route
workbox.routing.setCatchHandler(({ event }) => {
    console.log(event)
    switch (event.request.method) {
        case 'GET':
            return caches.match(OFFLINE_URL);
        default:
            return Response.error();
    }
});

The commits associated with this section:

Switch to workbox
` Put version number in workbox version <https://gitlab.com/Jenselme/django-pwa/-/commit/b3d13d3302813294db8eb85252b1a5d4eae46d0b>`__

Conclusion

That was quite a journey! Normally you should now be able to create a PWA with Django from scratch. You should also be able to understand the main concepts of a PWA. And you'll probably want to use Workbox to ease the writing of the ServiceWorker in your own app. But learning how it works behind the scene will ease debugging and understanding of the documentation of Workbox.

If you have any remarks or questions, please leave a comment below.

Installing openSUSE next to Fedora with BTRFS

2020-02-23T00:00:00+01:00

Update: My feedback article is now available.

I wanted to install openSUSE Tumbleweed (the rolling release version of openSUSE) on one of my computers to see how it looked outside a VM (and thus to try to use it daily). I am thinking about switching to this distribution to avoid major updates of Fedora. They happen every 6 months and I have multiple machines to update, so I find it cumbersome to update them all (and I like to test new stuff too ^^).

Why openSUSE? Because it looks stable, well tested, it provides a graphical installer (I already installed Arch and Gentoo in the past and wasn't enthusiastic about doing this again) and has some nice defaut with BTRFS (snapshots and booting from snapshots without having to configure it manually). I also know a contributor and a long time user of openSUSE and Tumbleweed who's very happy about the system.

In this article, I'll explain how I installed it next to my Fedora (which I don't want to loose it until I commit to making the switch). Some things are specific to my setup, others are more generic and should be reusable (more or less easily). If you have any question or remark, please leave a comment. I'll assume you have some knowledge of BTRFS.

Starting point

I had fedora installed with a BTRFS top level volume and 3 child volumes root for /, var for /var and home for /home. Since I use docker with the BTRFS driver I also had many subvolumes of /var for docker.

I also installed Tumbleweed in a VM to see how it behaves. I created another VM with Fedora installed just like on my computer so I could try installing Tumbleweed next to it without breaking my current setup.

Preparing Fedora

I removed the docker containers and images as well as all docker related subvolumes that remained. I don't know whether this could have an impact and didn't want to risk it (nor did I wish to migrate them).
The two most useful commands you can use at anytime to check something are:
- btrfs subvolume list PATH to list all subvolumes in the supplied filesystem.
- btrfs subvolume show PATH to see which subvolume is mounted under the supplied path.
Disable SELinux: it's not used in openSUSE so I think some files will be mislabeled which may prevent Fedora to work properly (or maybe just prevent me to access some files, I don't know and I don't want to test). Let's simply as much as possible and remove it from the equation.
Since my subvolumes are named var, root and home, the Tumbleweed installer will mount them under /var, /root and /home. I didn't find a way to prevent this. For /home it's the behavior I want since I want to keep my data. For the other two, it's not. You cannot rename subvolumes directly but you can rely on snapshots to achieve it. To do so, I (as root):
1. Mounted the top level BTRFS volume (which is not accessible by default) with: mount -t btrfs -o subvolid=5 /dev/vda3 /mnt/toplevel/ (most of my commands are copied from when I tested in a VM, hence the vda, adapt this part to your setup).
2. Created a snapshot for the root subvolume: btrfs subvolume snapshot /mnt/toplevel/root/ /mnt/toplevel/fedoraroot/
3. Emptied /mnt/toplevel/fedoraroot/var to remove the directory with rmdir /mnt/toplevel/fedoraroot/var
4. Created another snapshot for var (I decided to make it a subvolume of fedoraroot because I think it makes sense to have one top volume for Fedora): btrfs subvolume snapshot /mnt/toplevel/var/ /mnt/toplevel/fedoraroot/
5. Migrated the nested subvolumes for /var:
```
[root@baley toplevel]# rmdir /mnt/toplevel/fedoraroot/var/lib/machines/
[root@baley toplevel]# btrfs subvolume snapshot /mnt/toplevel/var/lib/machines/ /mnt/toplevel/fedoraroot/var/lib/
[root@baley toplevel]# rmdir /mnt/toplevel/fedoraroot/var/lib/portables/
[root@baley toplevel]# btrfs subvolume snapshot /mnt/toplevel/var/lib/portables/ /mnt/toplevel/fedoraroot/var/lib/
```
6. Identified each subvolume so I could know which ones where mounted as my main filesystem:
```
[root@localhost ~]# touch /mnt/toplevel/root/stdroot
[root@localhost ~]# touch /mnt/toplevel/fedoraroot/fedoraroot
[root@baley toplevel]# touch fedoraroot/var/fedoravar
[root@baley toplevel]# touch ./var/stdvar
```
7. Updated fstab in the snapshot (that's in fedoraroot) since that what we want to use from now on. To do so, I replaced subvol=root by subvol=fedoraroot```and ``subvol=var by subvol=fedoraroot/var.
8. Rebooted.
9. Edited the grub line from the menu before booting from it. I updated the subvol parameter of the kernel to be subvol=fedoraroot. This way grub mounted the proper subvolume during boot.
10. Checked that everything was correct:
```
[root@localhost ~]# btrfs subvolume show /
fedoraroot
        Name:                   fedoraroot
        UUID:                   1fc8ffa2-76f5-a343-aae9-4b499073d805
        Parent UUID:            5effda61-179e-9b48-900f-b7ad85d5f9d9
        Received UUID:          -
        Creation time:          2020-02-22 16:14:19 +0100
        Subvolume ID:           268
        Generation:             4554
        Gen at creation:        4447
        Parent ID:              5
        Top level ID:           5
        Flags:                  -
        Snapshot(s):
[root@localhost ~]# ls /fedoraroot
/fedoraroot
```
11. Remounted the top level volume.
12. Deleted the nested subvolumes: btrfs subvolume delete /mnt/toplevel/root/var/lib/machines/ (same for portables).
13. Deleted root and var:
```
btrfs subvolume delete /mnt/toplevel/root/
btrfs subvolume delete /mnt/toplevel/var/
```
14. Updated grub: grub2-mkconfig -o /boot/grub2/grub.cfg # or /boot/efi/EFI/fedora/grub.cfg if you use EFI
15. Checked the generated grub configuration:
```
[root@localhost ~]# grep -R subvol /boot/grub2/grub.cfg  # The subvolume parameter is correct.
set default_kernelopts="root=UUID=ce254a08-22e0-481e-b027-498aa059cc1a ro rootflags=subvol=fedoraroot resume=UUID=163891ab-ac20-4e58-8731-306810f99c9c rhgb quiet "
```
16. Rebooted. All should still be working.

Installing Tumbleweed

Just do the normal installation process until you reach the disk setup. Here you'll have to choose Expert partition and Start with existing partitions. Then, on the disk that contains you BTRFS volume:

Mount the swap
Mount the EFI partition under /boot/efi (if you use EFI).
Mount the main BTRFS volume / and be sure to skip the formatting of the volume. /boot must be under / in the BTRFS volume even if you use luks. See below for why.
Check installation summary: should not remove anything but should create a lot of new subvolume. None should conflict with fedora's.

That should be it. Inspect in the summary the operations done. You should see that the installer:

Will create all the subvolumes it needs.
Won't delete anything.
Will mount the fedora subvolumes. Not a big deal, we will remove that from the fstab once the installation is done.

You can launch the installation.

Finishing the setup of Tumbleweed

If you use luks, you will have to enter you luks passphrase in Grub with a US keyboard. You will then have to enter you passphrase normally as part of the boot process.

By default Tumbleweed will create a subvolume named @ and create its other subvolumes as subvolumes of it. Here however, it didn't (I don't know why). Since I still want a main subvolume for each system (having home as a direct subvolume of the top level volume sounds good to me since it is independent of any other system). So, it's time to move subvolumes again. I won't detail much this time, I think you know the drill (do this from fedora, the directories are in use in Tumbleweed):

mount -t btrfs -o subvolid=5 /dev/vda3 /mnt/toplevel/
cd /mnt/toplevel/
btrfs subvolume list .

# Repeat for each subvolume.
rmdir /mnt/toplevel/@/var/
btrfs subvolume snapshot /mnt/toplevel/var /mnt/toplevel/@

# Check. Don't remove subvolume while in use, the deletion will come later.
btrfs subvolume list /mnt/toplevel
btrfs subvolume list .

Let's edit the fstab. We are just appending @/ to each subvol arguments like this: subvol=@/var.

We also need to update /boot/grub2/x86_64-efi/load.cfg so grub mounts the correct subvolume during the boot process:

set btrfs_relative_path='y'
cryptomount -u bf63369674824969a7726fda0cb46537
btrfs-mount-subvol ($root) /boot/grub2/x86_64-efi @/boot/grub2/x86_64-efi

Note

I didn't need this step in my VM and given the content of the file, I think it is specific to luks.

We can now reboot. Everything should go well. Boot on the other system if not.

We can now delete the old subvolumes from the installation:

mount -t btrfs -o subvolid=5 /dev/vda3 /mnt/toplevel/
cd /mnt/toplevel/
# Repeat for each subvolume.
btrfs subvolume delete ./var

We can also enable snapper now to have snapshot. It's done automatically during a normal installation but not here (because we did weird things I guess). We can do this with snapper -c root create-config /. And we are done!

Avis

Fedora uses subvol in grub.cfg but not Tumbleweed. Tumbleweed changes the default subvolume of the top level volume. It's not a problem with this setup but if you want to install two systems that do this, you will have to make at least one explicitly use subvol everywhere.

Wrapping up

It required a lot of preparation and tests but I'm glad I made it. I find Tumbleweed fast (faster than fedora at boot and when installing package) and reliable (from what I used until now). I'll need to do more testing to confirm that (the system is very fresh after all). I also learned a lot about BTRFS, subvolumes and snapshots along the way. I still have some issues though:

The nextcloud client ask me to reconnect each time I launch it. It may come from a configuration shared with Fedora. I'll fix this later once I know which system I want to keep.
The grub menu to choose the kernel (or the snapshot) is sometimes not displayed.
Grub takes some time to start once the key is entered.
If I fail to enter my passphrase for grub of the first try, it fails and I have to do a hard reboot.
I decided to switch from Gnome to KDE and I think I hit a bug with the online accounts that prevent me to setup my gmail account. I'll have to dig into this later (I'll try to update this post if I find the solution). I also spent a lot of time to configure KMail (which has a lot of options by the way).

Bonus

Enter luks passphrase only once

If you use luks, you have to enter your luks passphrase twice: once to unlock /boot so grub can start the system and then so the system can access the disk. Sadly, they can't communicate (yet?).

You could put /boot on an unencrypted filesystem (that's what fedora does), but according to this thread you won't be able to boot from a BTRFS snapshot (which is one cool thing Tumbleweed enables by default).

You can also fellow these steps so you'll only have to enter the passphrase for grub. Check also this link to update the initramfs in Tumbleweed (don't forget to launch dracut --force). For reference, my /etc/dracut.conf contains:

# PUT YOUR CONFIG IN separate files
# in /etc/dracut.conf.d named "<name>.conf"
# SEE man dracut.conf(5) for options
install_items+="/etc/luks-keys/root.key"

Note

If you need to add many files, separate them with spaces like this: install_items+="/etc/luks-keys/root.key /etc/luks-keys/swap.key"

Astuce

Don't add key-slot=1 immediately, if you did a mistake you will be locked out of the system because luks will try to match the key against this slot. Don't forget to adapt the slot if needed. You may also choose not to add it since I found it doesn't really change the boot time anyway.

Tumbleweed default install

For reference:

localhost:~ # btrfs subvolume list /
ID 256 gen 32 top level 5 path @
ID 258 gen 4192 top level 256 path @/var
ID 259 gen 4125 top level 256 path @/usr/local
ID 260 gen 4192 top level 256 path @/tmp
ID 261 gen 3045 top level 256 path @/srv
ID 262 gen 4166 top level 256 path @/root
ID 263 gen 3620 top level 256 path @/opt
ID 264 gen 4192 top level 256 path @/home
ID 265 gen 26 top level 256 path @/boot/grub2/x86_64-efi
ID 266 gen 3550 top level 256 path @/boot/grub2/i386-pc
ID 267 gen 4118 top level 256 path @/.snapshots
ID 268 gen 4191 top level 267 path @/.snapshots/1/snapshot
ID 274 gen 197 top level 267 path @/.snapshots/2/snapshot
ID 275 gen 262 top level 267 path @/.snapshots/3/snapshot
ID 278 gen 485 top level 267 path @/.snapshots/4/snapshot
ID 282 gen 1614 top level 267 path @/.snapshots/5/snapshot
ID 283 gen 2100 top level 267 path @/.snapshots/6/snapshot
ID 290 gen 3024 top level 267 path @/.snapshots/11/snapshot
ID 291 gen 3066 top level 267 path @/.snapshots/12/snapshot
ID 292 gen 3233 top level 267 path @/.snapshots/13/snapshot
ID 295 gen 3551 top level 267 path @/.snapshots/14/snapshot
ID 296 gen 3593 top level 267 path @/.snapshots/15/snapshot
ID 297 gen 3613 top level 267 path @/.snapshots/16/snapshot
ID 301 gen 3631 top level 267 path @/.snapshots/17/snapshot
ID 305 gen 4005 top level 267 path @/.snapshots/18/snapshot
ID 306 gen 4051 top level 267 path @/.snapshots/19/snapshot
ID 307 gen 4113 top level 267 path @/.snapshots/20/snapshot

localhost:/mnt/toplevel # cat /etc/fstab
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /                       btrfs  defaults                      0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /var                    btrfs  subvol=/@/var                 0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /usr/local              btrfs  subvol=/@/usr/local           0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /tmp                    btrfs  subvol=/@/tmp                 0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /srv                    btrfs  subvol=/@/srv                 0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /root                   btrfs  subvol=/@/root                0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /opt                    btrfs  subvol=/@/opt                 0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /home                   btrfs  subvol=/@/home                0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /boot/grub2/x86_64-efi  btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /boot/grub2/i386-pc     btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /.snapshots             btrfs  subvol=/@/.snapshots          0  0
UUID=6f2fce23-6460-4d69-9e69-a5b6b5e4456c  swap                    swap   defaults                      0  0

Why you probably want to make a SPA

2020-01-12T00:00:00+01:00

Update: I created an updated version of this article in should you build a SPA which provides a more complete and balanced view on this subject. Please read it instead of this current article.

I am currently working professionally on a website that is not a SPA. I'd like to rely on this experience to explain the pain points I encounter in the hope to convince you to build a SPA from scratch on you next project.

First, I'll discuss some arguments you may have against SPAs hoping to change you mind:

Habit: you are maybe used to build website with frameworks like Django which, because of the context in which they were created, favors building websites with their template engines and to add JavaScript (most likely with jQuery) only to parts of the site, where you need the page to be dynamic. I think it's the weaker argument against building a SPA: in web development, technologies are changing and evolving and if you keep doing things out of habit, your skills will become irrelevant sooner or later. Note that I am not suggesting you should learn every new shiny thing that appears: it will take you a lot of time and it will never really pay off since the shiny thing will be replaced by something shinier very quickly. What I am saying is that we now have some mature and popular frameworks and libraries to build SPAs. They will stay around and thus are worth learning (you'll find a list near the end of this article). So, my advise is: learn one of them to expand your skill set and make SPAs your new habit!
You've never done it: I think it is very similar to the habit argument seen above. So I'll just say: it can be learned, others have done it and have shared how they did it. So you should be able to get help to start by reading some articles or books, browsing StackOverflow or by hiring a consultant.
Too complex for what you are doing: I have to admit these frameworks can be daunting, mostly at first. They come with new concepts, ideas and ways of doing things. And you will be running a full application in the browser of your users which is objectively more complex than just displaying a good old HTML page. I think it can be a very good argument against building a SPA. Depending on your context and current skills, a SPA may not make sense. For instance, you are prototyping or the site you are building is just for internal use or you are building it on your spare time to have fun. But for a website that you build professionally and that will release to the public, I still think you should go for a SPA even if you think you may not need it at first because you probably will need one (keep reading 😉).
You'll ever have the need: I think you probably will have the need and sooner than you think. For better or worse, users expect more and more dynamic and interactive web pages that respond quickly without page reloads. The more your site will grow, the more you will need dynamic parts: the header to show the user is logged in, some search results to display dynamically, a small dynamic component to improve user experience on a page, a tiny embed app… And these parts probably won't stay isolated: they will have to talk/interact with each other. It will be much easier to do in a SPA: you have a top component or a store to store global information so each sub-components can be updated correctly when one changes. Without, you may need to rely on custom events, global state that you manage yourself, maybe even a page reload or you'll have different technologies you'll have to make interact (let's say jQuery because that's what you started with and ReactJS because you needed it for complex interactions). Note that part of the pain can be mitigated with a global store injected in multiple components. But it will probably end up in a strange mix of technologies.
Find the skills: If you go for a SPA, you will need to be (or need your colleges to be) proficient in both backend and frontend development. If you already have frontend development skills or frontend developers in your team, then you're fine. If not, there are two different kind of skills you'll need:
- UX/design/HTML/CSS skills: This is to design the page and have something both pretty and usable. Let's face it, you'll need that anyway and you probably already have it (maybe with a contractor). You can make it easier for you by relying on a CSS framework so you don't have to do all the heavy lifting (backend developers tend not to like that anyway and it's harder than it seems). Just get some basic skills to be able to do some adaptations to what the designer gives you. In the end, UX/design is a job on its own and I don't think it is fair to expect this from you (nor to expect you designers to be developers).
- JavaScript skills with a JavaScript framework: Probably the easiest thing to get. The code will be similar to what you are already writing and most backend developers don't really mind writing JS from time to time (mostly modern JS though, not JS for IE though). And since you rely on a framework, it will handle most of the compatibility between browsers for you. Since JavaScript skills are in high demand (even in the backend side with NodeJS), learning these skills is in your interest too. You may stumble on browser specific issues from time to time or have trouble to adapt some CSS code, but you can leverage this with either getting some time to learn, hiring a contractor or a "true" frontend developer (if you can afford it).
You can use frameworks like ReactJS or Vue.js [1] to enhance only part of the page, this way you'll have the best of both worlds: a globally static website with dynamic parts only where needed. The problem is that as stated above, as your site grow, you will have more and more of those dynamic parts and they will have to interact which will be way easier to do in a SPA.
They come with their own set of challenges: True. You will have to organize your code differently and your backend will be more like an API than a "good old website". But this can be learned. You will also most likely need server side rendering (or SSR for short). It may require some adaptation in your infrastructure but I think it is mature for the big frameworks so it shouldn't really be an issue. You should be able to find many resources about this online for your framework of choice.

Now, I'll give some arguments about why SPAs are cool and so why you really want to do them:

If you don't start with a SPA, it will be hard to change later. You will have to redo your whole site: the frontend in a JS framework and part of your backend so it can be used in a API fashion. That's a lot of work you can avoid by doing a SPA first.
For better or worse, users expect JS and dynamic things. So you will need dynamic components on your site whether you want it or not. I think you should go for the easiest world and make your site dynamic first.
All the dynamic parts are done in the same technology. You may be tempted to use jQuery to add basic dynamic parts. But as your site will grow, this will likely not suffice or become too complex to manage. So you'll add something like ReactJS or Vue.js. But then you have a weird hybrid and will have to make your jQuery parts work with your framework parts which is more complicated than it sounds and results in a poor developer experience (and once your developers will be used to modern frameworks, I think they won't be pleased to work with ever jQuery again).
Maintaining two templates: for SEO or user experience to speed up initial page load, you might want to render the dynamic components on the server and cache them before they are rendered in the browser of the user. Since this is traditionally done in your backend templates, you'll have templates for server side rendering in your backend and JavaScript components that will take over in the browser to add dynamism. So you end up duplicating the work. Or to avoid this, you may want to render your application directly on the server. But part of the page will be rendered by your backend templating engine while some elements will be rendered by something else. It is way harder to do correctly than do all the rendering work in one technology.

Before concluding, a few word if you are currently in a hybrid site with part of it done in the "good old way" and part of it in a JavaScript framework. No matter why you got there, it's where you are. Converting everything to a SPA will be long, painful and costly. It will look a lot like a rewrite and you should avoid it if possible. Although I favour SPAs and if the developer experience won't be that great in an hybrid website, I think you should avoid the cost and risk associated with a rewrite (even of just the frontend) if possible. Most issues impact you as a developer but can be worked around to still provide a good user experience. My opinion is: use the JavaScript framework everywhere you can, always use it for new work and convert steps by steps the older part of your site as need rises. Look into SSR because you will need it sooner or later. In the end, let your new dynamic parts eat up the older static parts.

Here's a small list of frameworks (at time of writing in the beginning of 2020) that you may want to look into to build a SPA:

ReactJS: Maybe the most popular choice right now. While it was initially design to be included in a "static" page to add some dynamism to it, you can build full SPAs with it and even mobile apps. React itself is quite small so you will probably need to add some other libraries to extend its capabilities a bit. React can be added incrementally to you site to add dynamic parts step by step.
Vue.js: Another popular option. Very similar to React in principles (small, incremental) with a different approach that you may prefer coming from a "traditional web development" background.
Angular: While the first two are relatively small and can be added incrementally, Angular is a big framework designed to build SPA. I don't think you will be able to enhance an existing static website with it (not easily at least). While its size may be daunting, it is complete and you probably won't need additional libraries with it. Furthermore, its tooling is mature and able to reduce the size of your code to only include the code you actually use to keep the JS you send small. Given its approach, it might be the right choice for you if you come from a Java/.NET background.
Aurelia: Not as popular as the other options, but I think it is a sweet spot between the completeness and complexity of Angular and minimalism of React or Vue. Like Angular it is a complete framework designed to build SPA, but it focuses on web standards and tries to get out of your way so you can write standard JavaScript or include standard JavaScript libraries as easily as possible. Which makes it easier to learn and develop with (in my opinion at least).

To conclude, I am now convinced that SPAs are the way to go by default. There may still be use cases for "good old website" but I think the list is small and is getting smaller. Even if SPAs may require you to learn new skills and if they come with their own set of challenges, I think (and I hope I have convinced you) that SPAs will make you life easier in the long run.

If you have a remark or disagree, please leave a comment below.

[1]	You may say these are more libraries than frameworks. I agree, but I'll keep calling them frameworks to ease writing since I also want to include SPA frameworks like Angular or Aurelia.

Writing a PWA with Aurelia

2019-09-01T00:00:00+02:00

Sommaire

PWA basics
The App
- manifest.json
- ServiceWorker
Conclusion
Resources

I feel like Progressive Web Application (PWA for short) are a more and more popular way to build mobile apps. That's understandable since you can use the same technology you use to build a SPA to build one! So you stay in known grounds, can use your usual web browser to develop and test the app. And, unlike hybrid apps, you also enhance your standard SPA! In this article, I'll explain how I build a PWA named aurss with the Aurelia framework. I'll start by explaining what PWAs are and some specific tech you must use to make them work. Then I'll explain how to build one with Aurelia.

PWA basics

As a developer, building a PWA means:

Your users don't have to install it: they can use the app directly on the website without installing anything and you'll still be able to provide them advanced features like caching.
You will have more control on when you can update it: no need to submit it to the app store and wait for validation. Just push files to you server. The update will occur the next time your users reopen the app.
You are not bound by the rules of any app stores.
The app will work on any device with a recent browser. Chrome has the best support for PWAs right now, Firefox is quite good but Safari support can be an issue (even if it's getting better, see this article for instance).
The app can be progressively enhanced (eg by adding better offline support or better caching).
You show notifications the user with the Notification API and the Push API.
You can access some native features (like the camera or the microphone).

Here are the basic components needed to make a PWA:

manifest.json: it's a JSON file describing how the app must be displayed and installed. You'll use it to specify the name, icon, orientation and some display properties of your app. See here for more details. It is required to make your app a PWA.
ServiceWorker (SW for short): it behaves like a proxy server between your app and the network. It will keep working even when the tab is closed. So it will allow you to:
- Have caching and offline support.
- Sync things in the background.
- Subscribe to notifications of the Push API and display them to users.
Please note that:
- It only works over HTTPS (except for localhost).
- It cannot access the DOM.
- It may apply to all or only part of the pages.
- It must be registered to have an effect.
- It will only get updated if its content changed and after the tab is closed and reopen (unless you use the dev tools and force it to reload on change).
See here for more. It is required to make your app a PWA.
indexDB: where you put data to exchange with the SW (SW cannot access localStorage). See here for more information.
Many other APIs I'll just list here so you know what you can do:
- Notification API to display notification to the user.
- Push API to receive messages from a server.
- Many more: geolocation, camera access, …

The App

Now that we've seen what the basic components are, let's see how to do this with Aurelia.

In a nutshell, you want to build a normal app while paying attention to some details, relating with the SW: it will rely on indexDB for storage, on the fetch API for network (luckily that's what the Aurelia HTTP client is using), all its APIs are promise based and it will be reinstalled only when it changes.

I assume you are familiar with Aurelia so I won't explain Aurelia specific things (since they don't exist, I just build a standard app using the framework) and focus on PWA stuff. Anything I explain here is applicable with a basic app created by the CLI (if possible use webpack since this article will be easier to follow if you do).

So to make your app a PWA, you will have to:

Add the manifest.json.
Add the service worker.
Check with lighthouse that the app is installable.

You can then add other capabilities if needed.

manifest.json

Once you've build the app, you can add the manifest.json file. To do so add in your index.html or index.ejs:

<link rel="manifest" href="/manifest.json">

Then in the static/ folder (if you are not using webpack as configured by aurelia-cli, you may need to adapt this, the goal is to have this file in the root of the build files next to the index.html file), you can create the manifest.json file. Below is an example, you can adapt it to your needs. It also has more properties, see the documentation to view all its possibilities.

{
    "name": "aurss – RSS reader",
    "short_name": "aurss",
    "start_url": "/",
    "scope": ".",
    "display": "standalone",
    "background_color": "#FFF",
    "theme_color": "#493174",
    "description": "RSS reader for TTRSS",
    "dir": "ltr",
    "lang": "en-US",
    "orientation": "portrait-primary",
    "icons": [{
        "src": "/icons/aurss.96x96.png",
        "type": "image/png",
        "sizes": "96x96"
    }, {
        "src": "/icons/aurss.512x512.png",
        "type": "image/png",
        "sizes": "512x512"
    }]
}

ServiceWorker

Then, include the service worker:

Create a file named sw.js in your src folder. Leave it empty for now.

if ('serviceWorker' in navigator) {
    runtime.register()
        .then((registration) => this.logger.info('Service worker is registered', registration))
        .catch((registrationError) => this.logger.error(
            'Service worker failed to register', registrationError,
        ));
} else {
    this.logger.info('Service worker is not available in this browser.');
}

Make sure it will be copied when you build the application next to the index.html. For instance, with webpack, you can use this plugin (which will also be handy later for caching so I recommend that you use it or something similar).

Note

Even you are using TypeScript, I suggest you rely on plain JavaScript for the SW file. It will be much easier to write and you won't have to worry about transpilation.

Astuce

If you are using Chrome, the Application tab of the dev tools is really useful. It will show you information about the manifest file, the service worker and the used storage.

Astuce

ES6 imports won't work with SW, you must use something like importScripts('/src/js/idb.js'); to import things.

Now the SW is bootstrapped, we can start to use it to:

Cache the shell of the app (ie the files it needs to work) and provide offline support. This is done once the SW is installed, so when the installed event has been fired (self represent the SW instance):

// Name of our static cache.
const staticCachePrefix = 'static';
// Update this each time you app shell changes.
// This way, the SW will have changed and get reinstalled
// allowing for the new files to be cached.
const VERSION = '1.0.0';
const staticCacheName = `${staticCachePrefix}-${VERSION}`;

self.addEventListener('install', (event) => {
    log('Installing SW version:', VERSION);
    // waitUntil is there to make the sure SW is neither paused nor stopped
    // while we expect it to do some work until the promise we passed to it completes.
    // See: https://stackoverflow.com/a/37906330
    event.waitUntil(
        caches.open(staticCacheName)
            .then(cache => {
                console.log('Caching app shell');
                // serviceWorkerOption comes from serviceworker-webpack-plugin.
                // Adapt if needed.
                cache.addAll(serviceWorkerOption.assets);
            }),
    );
});

You can then use in your app.js file, in the constructor, something like the code below to display notification to the user about online/offline status:

this.onOffline = () => this.store.dispatch(isOffline.name);
this.onOnline = () => this.store.dispatch(isOnline.name);

if (!navigator.onLine) {
    this.store.dispatch(isOffline.name);
} else {
    this.store.dispatch(isOnline.name);
}

window.addEventListener('offline', this.onOffline);
window.addEventListener('online', this.onOnline);

Catch network requests, for instance to implement dynamic caching.

self.addEventListener('fetch', (event) => {
    // Let the browser do its default thing
    // for non-GET requests.
    if (event.request.method !== 'GET') {
        return;
    }

    event.respondWith(
        caches.match(event.request)
            .then((response) => {
                return response || fetch(event.request);
            }),
    );
});

When offline, you can store data in indexDB to send it as soon as the browser gets online again. To do this, you must listen to the sync event in the SW. You can take inspiration from this code for the SW part and from these utility functions (see a usage here) to achieve this.

Note

There are many caching strategies: with network fallback, cache only, network only, network with cache fallback, cache then network. I won't detail them here. Search on the internet if needed.

Note

Don't forget to clean the cache from time to time. This can be done with:

self.addEventListener('activate', (event) => {
    console.log('Cleaning old cache shell');
    event.waitUntil(
        caches.keys()
            .then((keylist) => Promise.all(
                keylist
                    .filter((key) => key !== staticCacheName && key.startsWith(staticCachePrefix))
                    .map((key) => caches.delete(key))
            )),
    );
});

Finally, you should test you app with lighthouse to check it actually is a PWA. The tool will also check for performance which is a good thing. In the Chrome dev tools, go to the Audit tab, verify that Progressive Web App is checked under the audit section and run the audit. The report should tell you that you have indeed created a PWA (and if not, what you are missing).

Conclusion

To sum up:

PWAs are not tied to any framework. You can make a PWA without one or add PWA capabilities to a SPA written with any framework.
They are quite fast to develop and allow for good code re-use because you are not making an app distinct from you main web site.
You don't have to learn many new technologies, most of the things you will do will be standard we development.
You can progressively enhance an app with PWA capabilities.
Pay attention to what they support and what they don't before starting and to iOS support.

Resources

My test application: aurss It's MIT licensed so you can re-use it. Don't hesitate to ask me questions in the comment or in Gitlab.
A course I took (I highly recommand it)

Some thoughts on Aurelia Store

2019-08-24T00:00:00+02:00

On my spare time, I am developing a board game that uses the Aurelia framework. I currently manage the state in a service without any dedicated state management solutions (like RxJS, Aurelia store, Redux, …). And I feel I am reaching to the limit of what I can do with it and feel the need for a dedicated solution. So I decided to give aurelia-store the official plugin for state management for the framework. I did so in at test project: aurss a small app to list rss feeds.

First, let me explain a bit why we would need a store: in Aurelia, you can use services (ie instances of a object) that are injected into your components. You can use them for actions (eg interaction with an external API) and to store data. If you app is big, you will end up with many services. You'll have to keep them all in sync (maybe with observers or the event aggregator) and make sure your component are correctly updated.

It turns out this approach can be quite tricky and that's why solutions like Redux (from the React world) or VueX for Vue.js and of course Aurelia store have emerged: to help you manage the state in a single place so you don't have to do the hard work of keeping everything in sync. Instead of having multiple services, you have one store, you rely on functions to change its state and the store "notifies" your components that something changed and that they must be updated. It also comes with some nice features like easy persistence and recovery from local storage and time travel.

Aurelia store more specifically is a plugin for Aurelia to help you use this pattern with the Aurelia framework. It's based on RxJS a popular library for event programming. The idea of RxJS is register actions to do when a given event occurs as specified by the observer pattern, ie do something like this:

const observable = fromEvent(document, 'click');
observable.subscribe(() => console.log('clicked!'));
observable.subscribe((event) => sendEventToAnalytics(event));

I'll let you see the doc of RxJS to learn more about that.

What's very interesting about Aurelia store is that while based on RxJS (and its full power), it remains easy to use at first: you don't need to know RxJS to start and don't need much knowledge to do even quite advanced things. Please read the introduction to Aurelia store to have a more detailed explanation about why the plugin exists and its relationship with RxJS.

Let's see briefly how to use it (I assume you are already familiar with Aurelia, if not, the official documentation will give you more details on how to integrate it in your app):

In your main.js file, you must enable the plugin and define an initial state:
```
aurelia.use.plugin('aurelia-store', { players: [] });
```

Inject the Store in your components:

import { inject } from 'aurelia-framework';
import { Store } from 'aurelia-store';

@inject(Store)
class MyComponent { …

Listen to events with:

// `this.state` will receive a new value each time the state is updated.
// This way, the component will re-render.
this.subscription = this.store.state.subscribe((state) => this.state);

Dispatch actions to update the state:

function addPlayer(state, player) {
    // Don't mutate the state directly, create a new one. See the doc for why.
    const newState = {..state};
    newState.players = [...newState.players];
    newState.players.push(player);
    return newState;
}
this.store.registerAction(addPlayer.name, addPlayer);
setInterval(() => this.store.dispatch(addPlayer, "Player " + parseInt(Math.random() * 100, 10), 10000);

Astuce

Don't forget to dispose of subscription with this.subscription.unsubscribe. See the doc.

I must say the experience of using Aurelia store is very pleasant: the plugin is easy to integrate, it works well, it's not hard to understand, it seems to play nice with RxJS if you need some advance features (for instance to register to updates to part of the state or to get only the first change).

If I compare this to Redux which is the other state management solution I used, I think that using directly functions when we dispatch actions is a very good idea: instead of having actions and reducers like in Redux, you only have one concept to achieve the same result. It's also way easier to get started and is easier to understand (each time I have to touch a Redux code base, I have to re-understand how it works and what the link between actions and reducers is).

The fact that the plugin hides the complexity of RxJS is also a really good thing. By comparison with Angular (from the tutorial I made at least) where RxJS is fully exposed, it really simplifies the development and startup.

So I am really pleased with the plugin, what I can do with it and the overall developer experience it provides. There are some small gotchas though:

Don't forget to register your actions. You may forget at first. But the store will provide an helpful error message if you forget so it's not the end of the world.
Pay attention to async actions: unlike other solutions, the Aurelia store allows you to do something like this: await this.store.dispatch(costlyApiCall). If you do this, your app will freeze until the API call is done which is not what you want (and may not be noticeable locally if your API is on your machine). So for expensive async operations, split the actions in two: one to do the call and one to update the store when the call is done. To view an example of this, look at fetchArticles and receivedArticles from my test app. But if you know your async actions will resolve fast (querying IndexDB for instance), this shouldn't be an issue.
Immer.js (useful to ease the writing of state mutation) doesn't work with the plugin out of the box yet (see this discussion which also gives a possible solution and this issue). Maybe alternative solutions work (like immutable.js) but I haven't tried them. Update of 2020-08-09: it is now possible, see this update.

Issues while writing tests for a component using i18n

2019-08-04T00:00:00+02:00

Some time ago, I wanted to add some tests (behavior and render) on an Aurelia component that uses the i18n plugin and more precisely the df attribute in the view to display a localized date.

Since it was a while back (I wanted to write it earlier but couldn't), my memory may not completely accurate. I hope it can help you nonetheless. If you have questions, please post a comment.

My component is pretty basic: it takes some bindable as inputs, define a time format to format the date correctly, has 3 methods to add custom behavior and that's it. You can view the model here and the view here.

Also note I am using TypeScript, Webpack and Jest. If you use different tooling, you may need to adapt the example a bit.

In my test, I setup my component like this:

component = StageComponent.withResources(PLATFORM.moduleName('../../src/resources/elements/aurss-article'))
    .inView(`
    <aurss-article
        value.bind="article"
        mark-article-as-read.bind="markArticleAsRead"
        mark-article-as-unread.bind="markArticleAsUnread"
        open-article.bind="openArticle"
    ></aurss-article>
    `).boundTo(viewModel);

await component.create(bootstrap);  // bootstrap comes from aurelia-bootstrapper

I got this error when running the tests: Error: No ValueConverter named "df" was found!. I then tried to register the component with:

component = StageComponent.withResources([
    PLATFORM.moduleName('../../src/resources/elements/aurss-article'),
    PLATFORM.moduleName('aurelia-i18n'),
])

But all I got was: Error: Error invoking RelativeTime. Check the inner error for details..

After reading tests in the plugin repo and this article (which solves a similar problem but the proposed solution didn't work for me), here is how I nailed it:

I created an test/helpers.ts file with the code below. Its purpose is to bootstrap the i18n plugin (it resembles what you do in main.ts):

import {Aurelia, PLATFORM} from 'aurelia-framework';
import {Backend} from 'aurelia-i18n';

export const prepareI18nComponent = (component) => {
    component.bootstrap((aurelia: Aurelia) => {
        return aurelia.use.standardConfiguration()
            .plugin(PLATFORM.moduleName('aurelia-i18n'), (instance) => {
                const aliases = ['t', 'i18n'];

                // register backend plugin
                instance.i18next.use(Backend.with(aurelia.loader));
                const config = {
                    resources: {
                        en: {
                            translation: {
                                hello: undefined,
                            },
                        },
                    },
                    skipTranslationOnMissingKey: true,
                };

                return instance.setup(Object.assign({
                    attributes: aliases,
                    backend: {
                        loadPath: './locales/{{lng}}/{{ns}}.json',
                    },
                    debug: false,
                    defaultNS: 'translation',
                    fallbackLng: 'en',
                    interpolation: {
                        prefix: '{{',
                        suffix: '}}',
                    },
                    lng: 'en',
                }, config));
            });
    });
};

In the beforeEach function, I now setup the component like this:

beforeEach(async (done) => {
    viewModel = {
        article: createArticle(),
        markArticleAsRead: jest.fn(),
        markArticleAsUnread: jest.fn(),
        openArticle: jest.fn(),
    };

    component = StageComponent.withResources(PLATFORM.moduleName('../../src/resources/elements/aurss-article'))
        .inView(`
        <aurss-article
            value.bind="article"
            mark-article-as-read.bind="markArticleAsRead"
            mark-article-as-unread.bind="markArticleAsUnread"
            open-article.bind="openArticle"
        ></aurss-article>
        `).boundTo(viewModel);

    prepareI18nComponent(component);  // Bootstrap i18n

    await component.create(bootstrap);

    done();
});

I also configured jest to load a file name test/jest-pretest.ts before it executes the tests suite by adding, under the jest section of my package.json file:

"setupFiles": [
    "<rootDir>/test/jest-pretest.ts",
    "jest-localstorage-mock"
]

This file contains:

import {Options} from 'aurelia-loader-nodejs';
import {globalize} from 'aurelia-pal-nodejs';
import 'aurelia-polyfills';
import * as IntlPolyfill from 'intl';
import * as path from 'path';

Options.relativeToDir = path.join(__dirname, 'unit');
globalize();

(global as any).navigator = {};

global.Intl.NumberFormat   = IntlPolyfill.NumberFormat;
global.Intl.DateTimeFormat = IntlPolyfill.DateTimeFormat;
(window as any).Intl = global.Intl;

(window as any).Intl.NumberFormat   = IntlPolyfill.NumberFormat;
(window as any).Intl.DateTimeFormat = IntlPolyfill.DateTimeFormat;

It is meant to be sure the Intl API used by the plugin is correctly defined during the tests.

That's it! You can also take a look at the full project code in gitlab if you need more details: https://gitlab.com/Jenselme/aurss

Extract translations from your Aurelia app

2019-08-03T00:00:00+02:00

If you use Aurelia with the i18n plugin, you will have to maintain JSON files that will map a translation key to an actual translation. Manually editing the file to add/remove keys can be tedious.

Luckily, keys can be extracted automatically thanks to i18next-scanner. It can extract translation keys from JavaScript, TypeScript and HTML files.

To do this:

Install i18next-scanner with npm install --save-dev i18next-scanner or yarn add -D i18next-scanner.
Add the code below in i18next-scanner.config.js file at the root of your project.
Run ./node_modules/.bin/i18next-scanner src/**/*.{js,ts,html} to extract the translations. They will be under src/locales/{{lng}}/translation.json. Keys to be translated will have the value __STRING_NOT_TRANSLATED__.

The extraction supports the following patterns:

In code files: it will extract keys directly from i18next (usage like i18next.t('my-key')) and from the injected I18N service (ie if you use it like this: this.i18n.tr('my-key')).
In HMTL files:
- With the following attributes: data-i18n, data-t, t, i18n (usage like this <p t='my-key'></p>).
- With the following behaviors: t (usage ${ 'myKey' | t } and ${ 'myKey' & t } – the whitespace is not important).

If this doesn't exactly work for your use cases, please adapt the code below and ask a question in the comments if needed.

 1 const fs = require('fs');
 2 const path = require('path');
 3 const typescript = require("typescript");
 4 
 5 module.exports = {
 6     input: [
 7         'src/**/*.{ts}',
 8     ],
 9     output: './',
10     options: {
11         debug: true,
12         func: {
13             list: ['i18next.t', 'i18n.t', 'this.i18n.tr'],
14             extensions: ['.js'],
15         },
16         lngs: ['en', 'fr'],
17         ns: [
18             'translation',
19         ],
20         defaultLng: 'en',
21         defaultNs: 'translation',
22         defaultValue: '__STRING_NOT_TRANSLATED__',
23         resource: {
24             loadPath: 'src/locales/{{lng}}/{{ns}}.json',
25             savePath: 'src/locales/{{lng}}/{{ns}}.json',
26             jsonIndent: 4,
27             lineEnding: '\n'
28         },
29         nsSeparator: false, // namespace separator
30         keySeparator: false, // key separator
31         interpolation: {
32             prefix: '{{',
33             suffix: '}}'
34         },
35         removeUnusedKeys: true,
36     },
37 
38     transform: function customTransform(file, enc, done) {
39         const {ext} = path.parse(file.path);
40         const content = fs.readFileSync(file.path, enc);
41 
42         if (ext === '.ts') {
43             const {outputText} = typescript.transpileModule(content.toString(), {
44                 compilerOptions: {
45                     target: 'es2018',
46                 },
47                 fileName: path.basename(file.path),
48             });
49 
50             this.parser.parseFuncFromString(outputText, {list: ['i18next.t', 'i18next.tr', 'this.i18n.tr']});
51         } else if (ext === '.html') {
52             this.parser
53                 .parseAttrFromString(content, {list: ['data-i18n', 'data-t', 't', 'i18n'] });
54 
55             // We extra behaviours `${ 'myKey' | t }` and `${ 'myKey' & t }` from the file.
56             const extractBehaviours = /\${ *'([a-zA-Z0-9]+)' *[&|] *t *}/g;
57             const strContent = content.toString();
58             let group;
59             while (true) {
60                 group = extractBehaviours.exec(strContent);
61                 if (group === null) {
62                     break;
63                 }
64                 this.parser.set(group[1]);
65             }
66         }
67 
68         done();
69     },
70 };

You can also view the code in gitlab: https://gitlab.com/snippets/1881948

Because of the removeUnusedKeys: true options, keys that are not found will be removed from the JSON files. If you don't want this behavior, switch this option to false.

It is compatible with Javascript and Typescript. If you don't use Typescript, remove the line const typescript = require("typescript"); to avoid issues at import.

CSS tips

2019-03-10T00:00:00+01:00

Sommaire

Apply style to IE11 only

Apply style to IE11 only

Put the style in this media query:

@media screen and (-ms-high-contrast: active), (-ms-high-contrast: none) {
    // Put CSS here
}

PyCharm tips

2019-03-10T00:00:00+01:00

Sommaire

View print output immediately
Auto-load .env file
Use shell_plus in the integrated console
Make sure commit hooks run in a venv

View print output immediately

Add PYTHONUNBUFFERED=1 in run config or .env (only use .env if you use the plugin). This is meant to immediately view the output of the print function without the need for flush=True.

Auto-load .env file

Install the EnvFile and .env file support plugins. Then configure the Django server to run with the custom command runserver_plus (optional) and in the EnvFile tab, tick Enable EnvFile and in the box below, add you .env file. Do the same for pytest.
Use [direnv](http://direnv.net/) to autoload the .env. To do this, create a .envrc file with the content below and run direnv allow . to allow the .envrc file to be loaded:
```
# Auto load venv.
layout_pipenv
# Auto load .env file
dotenv
```
Add this to the activate script:

if [[ "${TERMINAL_EMULATOR}" == "JetBrains-JediTerm" && -f .env ]]; then
    load-dotenv .env
fi

Use shell_plus in the integrated console

Use this starting script in Build, Execution, Deployment > Console > Django console:

# Load env var
import environ
environ.Env.read_env('.env')

import sys; print('Python %s on %s' % (sys.version, sys.platform))
import django; print('Django %s' % django.get_version())
sys.path.extend([WORKING_DIR_AND_PYTHON_PATHS])
if 'setup' in dir(django): django.setup()
import django_manage_shell; django_manage_shell.run(PROJECT_ROOT)
from django_extensions.management import shells
from django.core.management.color import color_style

# Default settings for shell_plus
shell_plus_default_settings = {
    'bpython':         False,
    'connection_file': None,
    'dont_load':       [],
    'ipython':         True,
    'kernel':          False,
    'no_browser':      False,
    'no_color':        False,
    'notebook':        False,
    'plain':           False,
    'print_sql':       False,
    'ptipython':       False,
    'ptpython':        False,
    'pythonpath':      None,
    'quiet_load':      False,
    'settings':        None,
    'traceback':       False,
    'use_pythonrc':    False,
    'verbosity':       1,
    'vi_mode':         False,
    'dont_load':       [],
}

g = globals()
objects_to_import = shells.import_objects(shell_plus_default_settings, color_style())
g.update(objects_to_import)

Make sure commit hooks run in a venv

You can use this script:

#!/usr/bin/env bash

set -eu

readonly INSIDE_DOCKER=$(grep -q docker /proc/self/cgroup && echo true)
readonly command="$1"
# To be sure $@ will only contain the parameters of the command, not the command itself.
shift

# If we are already in a venv or if the are in docker, run directly.
if [[ -v VIRTUAL_ENV || "${INSIDE_DOCKER}" == "true" ]]; then
    echo "Already in venv"
    ${command} "$@"

    exit $?
else
    # Load bashrc to be sure PATH is correctly set. We can't do this before this it could mess up with the enabled venv.
    # We don't want to fail if the bashrc file contains unbound variables.
    set +u
    source ~/.bashrc || echo "Bash RC file not found"
    set -u
    # If not and pipenv is installed and we have a Pipfile, run with pipenv.
    if command -v pipenv > /dev/null && [[ -f Pipfile ]]; then
        echo "Running in venv with pipenv"
        pipenv run "${command}" "$@"

        exit $?
    # If poetry is installed and we have a pyproject.toml, run with poetry.
    elif command -v poetry > /dev/null && [[ -f pyproject.toml ]]; then
    echo "Running in venv with poetry"
    poetry run "${command}" "$@"
    else
        echo "Not in an virtual env and don't know how to run in one." >&2
        exit 1
    fi
fi

Then, if you use pre-commit, you can do something like this:

- id: flake8
  name: flake8
  entry: ./bin/ensure-runs-in-venv.sh
  exclude: '/snapshots/.*\.py$'
  args: [flake8]
  language: system
  types: [python]

ZSH tips

2018-10-09T00:00:00+02:00

Sommaire

ZBell

ZBell

Useful to have a notification when a long command completes.

To enable it, add zbell to your plugins array if you are using oh-my-zsh or source the definition file.

To configure:

The minimum time commands must take for the notification to happen, use: ZBELL_DURATION. For instance ZBELL_DURATION=60.
The command zbell must ignore, use: ZBELL_CMD_IGNORE. For instance: ZBELL_CMD_IGNORE=($EDITOR $PAGER git nano less emacs bash ssh su tmux zsh "pipenv shell" "python manage.py shell_plus")

Here is the plugin. It was inspired by https://gist.github.com/jpouellet/5278239/ (I made changes to ignore commands made of multiple words, to report duration only once and to use notify-send to send the notification).

#!/usr/bin/env zsh

# From https://gist.github.com/jpouellet/5278239

# This script prints a bell character when a command finishes
# if it has been running for longer than $zbell_duration seconds.
# If there are programs that you know run long that you don't
# want to bell after, then add them to $zbell_ignore.
#
# This script uses only zsh builtins so its fast, there's no needless
# forking, and its only dependency is zsh and its standard modules
#
# Written by Jean-Philippe Ouellet <jpo@vt.edu>
# Made available under the ISC license.

# only do this if we're in an interactive shell
[[ -o interactive ]] || return


# get $EPOCHSECONDS. builtins are faster than date(1)
zmodload zsh/datetime || return


# make sure we can register hooks
autoload -Uz add-zsh-hook || return

# initialize zbell_duration if not set
(( ${+zbell_duration} )) || zbell_duration=${ZBELL_DURATION:-15}

# initialize zbell_ignore if not set
if [[ ${#ZBELL_CMD_IGNORE} -gt 0 ]] && (( ! ${+zbell_ignore} )); then
    zbell_ignore=()
    for ignore_cmd in ${ZBELL_CMD_IGNORE[*]}; do
        zbell_ignore+=("${ignore_cmd}")
    done
fi
(( ${+zbell_ignore} )) || zbell_ignore=($EDITOR $PAGER)

# initialize it because otherwise we compare a date and an empty string
# the first time we see the prompt. it's fine to have lastcmd empty on the
# initial run because it evaluates to an empty string, and splitting an
# empty string just results in an empty array.
zbell_timestamp=$EPOCHSECONDS

# right before we begin to execute something, store the time it started at
zbell_begin() {
    zbell_timestamp=$EPOCHSECONDS
    zbell_lastcmd=$1
}

# when it finishes, if it's been running longer than $zbell_duration,
# and we dont have an ignored command in the line, then print a bell.
zbell_end() {
    if [[ -z "${zbell_lastcmd}" ]]; then
        return
    fi

    time_run=$(( $EPOCHSECONDS - $zbell_timestamp ))
    ran_long=$(( $EPOCHSECONDS - $zbell_timestamp >= $zbell_duration ))

    has_ignored_cmd=0
    for ignore_cmd in ${zbell_ignore[*]}; do
        if [[ "${zbell_lastcmd}" == *"${ignore_cmd}"* ]]; then
            has_ignored_cmd=1
            break
        fi
    done

    if (( ! $has_ignored_cmd )) && (( ran_long )); then
        notify-send "'${zbell_lastcmd}' completed in ${time_run}"
    fi

    zbell_lastcmd=""
}

# register the functions as hooks
add-zsh-hook preexec zbell_begin
add-zsh-hook precmd zbell_end

Bash tricks

2018-10-03T00:00:00+02:00

Sommaire

Scripts
Signals

Scripts

Use this at the top of all your Bash scripts to avoid problems:

# Exit on error.
set -e
# Don't allow undefined variable.
set -u
# Make pipeline fail if any command in it fail.
set -o pipefail

Signals

See this article.

Heroku tips

2018-10-03T00:00:00+02:00

Use an editor

By default there is no editor on Heroku. To get one, you can use the heroku-nano plugin like this (or by installing as a plugin):

mkdir bin
cd bin
curl https://github.com/Ehryk/heroku-nano/raw/master/heroku-nano-2.5.1/nano.tar.gz --location --silent > nano.tar.gz
tar -xvf nano.tar.gz
chmod +x nano
cd -
export PATH="$PATH:$(realpath bin)"

Then use nano README.md.

Papertail

Download logs

This requires httpie. But you can adapt this to use curl instead.

#!/usr/bin/env bash

set -eu

# Change this to start at another date.
start_date=2019-08-24
nb_days_to_download=18
# Set your paper trail token here.
papertrail_token=''

for i in $(seq 0 ${nb_days_to_download}); do
  log_date=$(date -I -d "${start_date} +${i} days")
  mkdir -p "${log_date}"
  pushd "${log_date}"
    for log_hour in $(seq -f "%02g" 0 23); do
      if [[ ! -f "${log_date}-${log_hour}.tsv.gz" ]]; then
        http --follow "https://papertrailapp.com/api/v1/archives/${log_date}-${log_hour}/download" "X-Papertrail-Token:${papertrail_token}" > "${log_date}-${log_hour}.tsv.gz"
      fi
    done
  popd
done

Download files

From google drive

Define this function:

gdrive-download() {
    file_id=$1
    file=$2
    base_url="https://drive.google.com/uc?export=download&id=${file_id}"
    echo "Downloading ${base_url}"
    CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "${base_url}" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1/p')
    wget --load-cookies /tmp/cookies.txt "https://drive.google.com/uc?export=download&confirm=${CONFIRM}&id=${file_id}" -O "${file}"
    rm -rf /tmp/cookies.txt
}

And use it like this: gdrive-download FILE_ID_FROM_THE_SHARE_LINK DEST_FILENAME.

Python tricks

2018-10-03T00:00:00+02:00

Sommaire

Inspect TLS certificates with requests

Inspect TLS certificates with requests

Use the snippet below to capture and inspect the certificates of a distant site and its chain with requests.

import requests

HTTPResponse = requests.packages.urllib3.response.HTTPResponse
orig_HTTPResponse__init__ = HTTPResponse.__init__
def new_HTTPResponse__init__(self, *args, **kwargs):
    orig_HTTPResponse__init__(self, *args, **kwargs)
    try:
        self.peer_certificate = self._connection.peer_certificate
        self.peer_cert_chain = self._connection.peer_cert_chain
    except AttributeError:
        pass
HTTPResponse.__init__ = new_HTTPResponse__init__

HTTPAdapter = requests.adapters.HTTPAdapter
orig_HTTPAdapter_build_response = HTTPAdapter.build_response
def new_HTTPAdapter_build_response(self, request, resp):
    response = orig_HTTPAdapter_build_response(self, request, resp)
    try:
        response.peer_certificate = resp.peer_certificate
        response.peer_cert_chain = resp.peer_cert_chain
    except AttributeError:
        pass
    return response
HTTPAdapter.build_response = new_HTTPAdapter_build_response

HTTPSConnection = requests.packages.urllib3.connection.HTTPSConnection
orig_HTTPSConnection_connect = HTTPSConnection.connect
def new_HTTPSConnection_connect(self):
    orig_HTTPSConnection_connect(self)
    try:
        self.peer_certificate = self.sock.connection.get_peer_certificate()
        self.peer_cert_chain = self.sock.connection.get_peer_cert_chain()
    except AttributeError:
        pass
HTTPSConnection.connect = new_HTTPSConnection_connect


response = requests.get('https://www.jujens.eu')
response.peer_certificate.get_subject()
response.peer_cert_chain

response.peer_cert_chain[0].get_subject()

Sources: https://stackoverflow.com/a/52072170 and https://stackoverflow.com/a/47931103

Deploy to your test server with git hooks

2018-09-09T00:00:00+02:00

You probably already wanted to have your own test environment to allow others in the company to tests what you did. Perhaps you have a common one, but as your team is growing, it is probable that the common environment is a bottleneck and you wish each developer could have its own to test things in isolation. That's the subject of this article where we will create such an environment and deploy to it with a simple git push (a lot like Heroku but a lot cheaper). You will also have full control over it and you will be able to modify the files directly on the server if needed (just don't do that last part in production ;-)).

In a nutshell, we need to support:

Code updates:
1. Push to a current branch.
2. Force push to a current branch
3. Erasing unclean work tree on the server
4. Pushing a new branch
Restart of the server (meaning the test environments must be up and running if the server restarts).
Environment variables to configure the application.

To achieve all this, all you need is a linux server. Your machine or a VM will do if you want to test along. If you want to use a VM, you may want to check my tips for libvirt to use a static ip for the VM and Connect to a VirtualBox VM to ease SSH connection in you VM.

To do it for real, a server in the office or in the cloud will do. If you need help, leave a comment, I'll do my best to provide feedback.

For the purpose of this tutorial, we will deploy a small hello work app. You can fork it and use the fork as a remote repository if you want to follow along. The app is made in Python and there are a few Python specific things here and there but globally, you should be able to adapt it to any other languages. Leave a comment if you need help.

Before we start, some precision:

If your server is running SELinux, you'll have to add proper context (the web one) to some file and allow the web server to connect to the UNIX or TCP socket of the application server. I won't go into this here, the tutorial is already long enough.
I'll use jujens.bl.test as my test domain. Replace it with your own.
Commands that must be run as root with start with # and as a normal user with $. I'll precise the host too.
You need new commits to push the code with git. If you change nothing, no hooks (more on that later) will run on the server because git is smart enough to see nothing changed.
I assume your OS runs systemd. If not, adapt the commands.
I'll present things in a tutorial manner so you can understand and follow what I do. Go to the Summary section to view a recap and what you need to do each time you want to add a developer.

Sommaire

Prepare the server
Prepare the repo
- Configure git
- Prepare the server repo
How to send the code to the server
Configuring the web server
- Basic setup of the application server
- Basic nginx configuration
Configuration for multiple users
Environment variables
Secure access
Going further
Summary
- Full configurations

Prepare the server

Get a linux server (cloud, VM, your machine, Raspberry Pi, …)
Install the prerequisites
- SSH
- git
- nginx (apache or any other web server can work too, you'll have to adapt the conf though)
- Python (or anything else you need to run your application)
Configure SSH: you probably want to edit /etc/ssh/sshd_config to disallow direct root login (with PermitRootLogin no), force SSH key usage (with PasswordAuthentication no) and perhaps change the default port (eg with Port 422). Search the internet to learn more.
Enable SSH: on the server, run systemctl start sshd and systemctl enable sshd
Configure the firewall to let SSH and HTTP traffic through (adapt these commands if you changed any ports):
- With firewalld: # firewall-cmd --add-service=http --permanent && firewall-cmd --add-service=http and # firewall-cmd --add-service=ssh --permanent && firewall-cmd --add-service=ssh
- With iptable: # iptables -t filter -A INPUT -p TCP --dport 22 -j ACCEPT and # iptables -t filter -A INPUT -p TCP --dport 80 -j ACCEPT
Create at least one user (each dev should have their own user, more on that later). I'll use jujens.
Create the deploy group: # groupadd -r deploy and add your user to it: # usermod -aG deploy jujens
Start and enable nginx: # systemctl start nginx and # systemctl enable nginx

To use the bl.test and jujens.bl.test domains, you must either modify your /etc/hosts by adding something like this (change the ip of the server):

192.168.122.45 bl.test jujens.bl.test

Or use dnsmasq to redirect all requests made to bl.test and its subdomains to an ip. See my article on that

If your server is publicly accessible, you may add a wildcard entry to your DNS server so all subdomains of test.mypublicdomain.com are served by the test server. This way you will avoid the need to change your DNS configuration when you add a new developer.

Prepare the repo

Configure git

First, we will configure git. You can use a global configuration by editing /etc/gitconfig (users can still change it for all their repos in ~/.gitconfig or per each repo in .git/config).

Note that if you intend to do other things than git deploys with your server, the global configuration may not be the best choice.

Here is the git configuration we'll use

[core]
    hooksPath = /etc/git/hooks

[receive]
    denyCurrentBranch = updateInstead

You can check it by running: git config receive.denyCurrentBranch and git config core.hooksPath.

You must adapt the path to where the git hooks will be. If you choose a per repo configuration, you don't need to specify it and it will default to .git/hooks

By default git won't allow you to push to a non bare repo (ie a repo in which you can access files easily) because it is dangerous. But in our case, it is required, otherwise, we won't be able to push anything while making the site work. That's the point of the ``denyCurrentBranch = updateInstead`` option. See the documentation for more details.

Now we can create the directory where the global hooks will lie: mkdir -p /etc/git/hooks. We will add scripts here later.

Prepare the server repo

Clone the remote repo on your dev machine (where you want) so you can push from it.
Clone the remote repo on the server. You could also push directly from your dev machine to it but I find it easier this way (and you still have to create a repo on the server). Do it:
1. Create the folder /var/www/bl.test/jujens.bl.test (the location may seem odd right now, but it will have its importance later): # mkdir -p /var/www/bl.test/jujens.bl.test
2. Set user and group: # chown jujens:nginx /var/www/bl.test/jujens.bl.test
3. Set the proper permissions: # chmod u=rwX,g=rXs,o=- /var/www/bl.test/jujens.bl.test (this will give all permission the current user, read and "execute" permissions to the group and nothing to others. The s is there to set the "sticky bit". Thanks to it, all file created inside the directory will belong to the group you specify.
4. Clone: # git clone https://gitlab.com/Jenselme/hello-flask.git /var/www/bl.test/jujens.bl.test
Add the test remote on your dev repo (so you can push to the test server): $ git remote add test jujens@bl.test:/var/www/bl.test/jujens.bl.test

Check everything is configure properly with a $ git push test It should output:

The authenticity of host 'bl.test (192.168.122.45)' can't be established.
ECDSA key fingerprint is SHA256:nb/a1ypBTTgahruVf92GzLNGTAdrrDi2afIWfXaFAa0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'bl.test' (ECDSA) to the list of known hosts.
Everything up-to-date

How to send the code to the server

First, let me explain briefly what git hooks are since we will need them. Simply put, git hooks are scripts that are run before or after certain actions (commit, push, …). They are named pre-ACTION or post-ACTION and are handy to automatically run linting before committing or tests before pushing. By default they lie in PROJECT/.git/hooks (by the way, you can find samples in this folder). Here, we used the hooksPath to have global hooks.

Don't forget to make these script files executable or they won't work (I forget almost all the time!).

See the doc for more details and the full list of available hooks.

Now that's you know what git hooks are, we will do each step at a time (git commands are run on you machine):

Step 1 Push to a current branch: Nothing to do since we configured git properly. Just modify hello.py, commit and git push test. You should see the commit and the modified file on the server.
Step 2 Force push to a current branch: Let's say we made a mistake in our previous commit, amended it and want to force push that. Modify the hello.py again and do git commit --amend -a to amend the last commit. git push test will fail but git push test --force will do just what we need. This will also work if you have new commits on the test server, just be aware they will be deleted.
Step 3 Erasing unclean work tree on the server: Modify hello.py on the server and don't commit. This is meant to simulate a debug modification you made after deploying the app. Now modify hello.py on your machine, commit and push. Neither git push test nor git push test --force will work because the work tree is unclean: you have a modified file whose content would be lost if the push succeeded. You could connect to the machine and run git checkout . yourself. But we can automate this with git hooks. Here we want to use the pre-receive hook that will be run before git gets the pushed content. In /etc/git/hooks/pre-receive (or .git/hooks/pre-receive if you want local hooks, I won't precise again), add:
```
#!/usr/bin/env bash

# We launch the command in a subshell to avoid impacting the true environment with our unset.
# We unset GIT_DIR because it points to the wrong directory.
# We are in the .git directory when this hook is run, so we need to move one step up to run git commands.
# We then force a checkout of the repo to discard any changes.
( unset GIT_DIR && cd .. && git checkout -f )
```
And voilà! Our git push works!

Step 4 Pushing a new branch: this must be done once git received all the commits with the post-receive hook to update to the new branch. Git hooks are generally passed some parameters so we can do "smart" actions with them. Here, we will read the ref (name of the pushed branch). Create /etc/git/hooks/post-receive and add:

#!/usr/bin/env bash

# Parameters are passed to stdin, we need to use read to get them. Only ref matters for what we do.
read oldrev newrev ref
# By default, ref contains refs/heads/branch-name. We use bash substitution mechanism to remove "refs/heads/".
branch_name=${ref/refs\/heads\//}

# Once again, force a checkout to the branch.
( unset GIT_DIR && cd .. && git checkout -f $branch_name )

Now we can correctly push our code the repo on the test server. On big leap forward. Now let's see how we can configure the server.

Configuring the web server

Basic setup of the application server

First, we will prepare our venv (or whatever else you need in your language) with the proper dependencies. This section must be adapted if you are not using Python. Run on the server:

Create the venv: $ python3 -m venv .venv
Activate it: $ source .venv/bin/activate
Install the dependencies: $ pip install -r requirements.txt
Start the application server: $ gunicorn hello:app
Check it is running with $ curl localhost:8000 You should see the following output
```
Hello World!
```

In a true server, you may want to use --forwarded-allow-ips to limit who can connect to guincorn. See: gunicorn's documentation

Now that it's done, we can add to the post-receive hook (/etc/git/hooks/post-receive) the following line to update our venv each time code is pushed:

# We are again in .git by default.
( cd .. && source .venv/bin/activate && pip install -r requirements.txt )

Basic nginx configuration

Create nginx configuration file in /etc/nginx/conf.d/bl.test.conf (or /etc/nginx/site-enabled/bl.test.conf depending on your system, I'll use /etc/nginx/conf.d for the rest of this tutorial) with this content:

server {
    listen 80;
    server_name jujens.bl.test;
    # Put it to the folder that will contains all the repos.
    root /var/www/bl.test/;

    # Prevent access to source files.
    location ~ .*\.pyc? {
        return 404;
    }

    # Prevent access to git
    location ~ .git {
        return 404;
    }

    # Relay everything else to the application server.
    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://127.0.0.1:8000;
    }
}

Test the configuration with nginx -t and if everything is fine, reload nginx: systemctl reload nginx. On your development machine, run curl jujens.bl.test to check everything is fine. You should see something like:

Hello World!

Configuration for multiple users

In the basic configuration section, we used a TCP socket (with a port) to access our application server. It works but if you want to add an environment for a new developer, you will need to:

Find an used port and configure gunicorn to use it.
Add a new nginx configuration so nginx accept request from the proper domain and routes them to the proper application server.

Thankfully, we can also use another kind of sockets: UNIX sockets. They exist as files on the server and if named with a proper convention, we can route our traffic easily. That's why I suggested to used the domain as the name of the folder that contains the code. It should become obvious once we completed it.

Configuring the application server

We will rely on systemd to start and restart the application servers. First, create a file named /etc/systemd/system/gunicorn@.service. It is a service template: it is meant to run multiple instances of a service based on a name (jujens.bl.test for instance). The name will be accessible as a variable (%i) in the template so we can alter the behavior of the service based on it.

Put this content in the file (adapted from http://docs.gunicorn.org/en/stable/deploy.html#systemd):

[Unit]
Description=gunicorn daemon for %i
After=network.target

[Service]
PIDFile=/var/run/gunicorn/%i.pid
User=nginx
Group=nginx
Environment="PATH=/var/www/bl.test/%i/.venv/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
RuntimeDirectory=gunicorn
WorkingDirectory=/var/www/bl.test/%i
ExecStart=/var/www/bl.test/%i/.venv/bin/gunicorn \
        --pid /var/run/gunicorn/%i.pid   \
        --bind unix:/var/run/gunicorn/%i.sock \
        hello:app
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

I won't go into the details here (archlinux has a great doc for that). It is just how we create new unit in systemd. A unit being used to run a service. We use a description and dependencies in the Unit section. We then define the service:

PIDFile is used by systemd to know whether the service is working.
We define as who the service will run.
We update the path so it can find our venv.
We defined the working directory to where the app of any given user will live thanks to %i
We defined how we want to start the service and how it can be reloaded and restarted.

I didn't use the --reload option of guincorn. That is because we may need to update the dependencies of our project before restarting it.

You may want to use the --worker X option to run multiple worker and thus allow your app to handle more traffic. In a test server it may not be needed though.

Now we need to "tell systemd" the file has changed with systemctl daemon-reload. We must also prepare the directories required by gunicorn to run:

Where the PID and socket files will be: mkdir -p /var/run/gunicorn
Correct permissions: chmod u=rwX,g=rX,o=- /var/run/gunicorn/

We can now start and enable the server by using the template and specifying the name of the instance like this: guincorn@DOMAIN. Or in my case: systemctl start gunicorn@jujens.bl.test and systemctl enable gunicorn@jujens.bl.test

The trailing .service in the service name is optional.

Use journalctl -u gunicorn@INTANCE to view the logs. For instance: journalctl -u gunicorn@jujens.bl.test

You don't actually need to create the /var/run/gunicorn directory. systemd will create it based on the value of RuntimeDirectory and set the permissions based on the values of User and Group.

On Ubuntu, /var/run/ is mounted on a tmpfs so the directory will disappear. It also seems that when you will launch multiple instances, the folder will be deleted and recreated making its usage as described in this tutorial impossible (guincorn cannot start by putting its PID and Socket directly in /var/run because of a permission issue). I suggest you use a runtime directory per user with RuntimeDirectory=gunicorn-%i or create a /var/myrun/ folder and adapt the configuration or dig into how Ubuntu does things.

Configuring nginx

We need to update the location / section of our nginx configuration to use the proper UNIX socket:

location / {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_redirect off;
    proxy_pass http://unix:/var/run/gunicorn/$host.sock;
}

We also need to update the server name into server_name *.bl.test;.

nginx will respond with a 502 Bad Gateway if there is no socket for that domain (either because it doesn't exist or because gunicorn is down).

Static files

We could use something like whitenoise to serve our static files. But since we have a nginx server, we can also rely on it to do the job:

server {
    listen 80;
    server_name *.bl.test;
    root /var/www/bl.test;

    # Prevent access to source files.
    location ~ .*\.pyc? {
        return 404;
    }

    # Prevent access to git.
    location ~ .git {
        return 404;
    }

    # If you request /page, we will check if /$host/page exists. If so we serve it.
    # Otherwise we pass the request to gunicorn for processing.
    # For instance, if I request jujens.bl.test/requirements.txt, nginx will check whether
    # /var/www/bl.test/jujens.bl.test/requirements.txt exists and serve it to me.
    # /var/www/bl.test comes from the server root and jujens.bl.test from the $host variable.
    location / {
        try_files /$host/$uri @gunicorn;
    }

    # Relay everything else to the django web server.
    location @gunicorn {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://unix:/var/run/gunicorn/$host.sock;
    }
}

Serving directly static files this way is insecure. You may want to choose an empty folder as root and relay everything to your app to serve only relevant files. Another good idea would be to put all static assets in a static directory and serve only its content. For Django users, you can adapt the configuration I mention here.

Testing deployment

We are almost ready to test the deployment of our app. We still need two things. First, we must configure sudo to allow the user to restart gunicorn (because git hooks are run as the currently logged in user, jujens in my case, they may not be allowed to run systemctl restart by default). To do that:

Run visudo to edit the sudoers file (visudo will check the syntax of the file before exiting. You can use another editor by setting the EDITOR environment variable).
Add at the bottom of the file: %deploy ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart gunicorn@* This will allow all users in the deploy group to restart any instance of gunicorn. I think it is the easiest way to achieve our goal. Be more restrictive if you need to.

We can now add to our post-receive hook (/etc/git/hooks/post-receive):

# We move to the folder that contains our code.
# We extract the domain (and guincorn instance name) from the folder
# by keeping only the name of the current directory of the full path.
# Finally, we restart the proper instance of gunicorn.
( cd ..; unit_name=$(pwd); unit_name=${unit_name##*/}; sudo systemctl restart "gunicorn@${unit_name}" )

Now if you deploy a new commit, gunicorn will restart and if you do curl http://jujens.bl.test you will be able to see you modification.

Environment variables

So far so good. But you may want to configure your application on a per user basis (to use a specific database for instance). To do this, you can rely on environment variables as recommended by the twelve factor app. To do this, we will configure systemd to load an environment file which will contain all our variables. gunicorn will them be able to access them and so will our app.

To do that:

Add to the gunicorn@.service file (/etc/systemd/system/gunicorn@.service), after Environment:
```
EnvironmentFile=/var/www/bl.test/%i/.env
```
Notify systemd about the change with: systemctl daemon-reload.

You may also want to prevent access to this file with nginx:

location ~ ^/.env$ {
    return 404;
}

Now create /var/www/bl.test/jujens.test.bl/.env and put in it:

MY_ENV_VAR="Loaded from env"

Redeploy your app, run curl http://jujens.bl.test and you should see Loaded from env in the response.

Secure access

Since this is a test server, you may want to restrict access to the application. You can do this by putting the server in a local network not accessible to the outside world, basic auth or X509 auth. I'll only detail the basic auth method. For X509, see: https://arcweb.co/securing-websites-nginx-and-client-side-certificate-authentication-linux/

To protect your server with a username and a password at the nginx level:

Install httpd-tools (or apache2-utils or whatever provides /usr/bin/htpasswd)
Create the password file and add username to it: htpasswd -c /etc/nginx/bl.test.passwd username
Type password
Configure nginx to use this file and prevent access without authentication by adding in the server block:

auth_basic "Test server";
auth_basic_user_file /etc/nginx/bl.test.passwd;

Test the configuration with nginx -t and reload if the configuration is good with: systemctl reload nginx.

Going further

At this point, we are done and everything should work as intended. I just want to point out some other ways to do things:

You could use the post-receive hooks to move static files into a dedicated folder (eg /var/www/bl.test/static/jujens.bl.test) and use /var/www/bl.test/static as the root of nginx to be sure not to serve source files. Since it will communicate with gunicorn with a socket, its root need not be where the source lives. You don't need to bother about a dedicated path for each user if your static files contains their hash in their name. You'll have to setup a cleanup mechanism though. I didn't do it in this tutorial to keep things more simple.
If you repo is big, you can use something like: git --work-tree=/var/www/bl.test/jujens.bl.test --git-dir=/home/jujens/bl.test/.git checkout -f to make everyone push to one git repository and update their source tree. You can use the $USER environment variable to know who pushed and copy to the proper directory. You'll have to adapt the file structure.
A very good article form digital ocean has almost the same subject as this tutorial (they give more precision about git hooks and handle only static applications).
You can also extract the subdomain name (jujens in my case) to use just it in your paths and instance names. Something like this should work (see https://stackoverflow.com/a/33681072/3900519):
```
map $host $env_name {
    default ...;
    ~* ^(?<x_env_name>[a-z0-9]+)\..* $x_env_name;
}
```
You can also add HTTPS with something like let's encrypt or you usual provider.
Depending on the level of customization you want, you could also use a project like dokku.

Summary

Thank you for going this far. The article is quite dense and I think it deserves a quick summary. You can also find all the configurations below with a download link.

So in a nutshell, to setup the server, you need to (refer to Prepare the server for the details):

Install the prerequisites.
Configure SSH.
Add and configure the base user and group.
Setup the global domain if possible.

Each time you want to add a user, you need to:

Add a new DNS entry if you can't use "glob domains".
Add a new user in the deploy group.
Prepare the code repository (clone and setup hooks if you didn't choose to use global ones).
Prepare the environment (.env file, venv, …)
Start the application server.

You could use an adaptation of this script to make your life easier (run as root):

 1 #!/usr/bin/env bash
 2 
 3 # Usage: ./add-user.sh USERNAME REPO_TO_CLONE
 4 # USERNAME: username used for ssh and in the domain
 5 # REPO_TO_CLONE: the repository to clone.
 6 # example: ./add-user.sh jujens https://gitlab.com/Jenselme/hello-flask.git
 7 # will create a jujens user to use the jujens.bl.test subdomain with the specified app.
 8 
 9 # Configure bash to exit on errors, consider undefined variables as errors
10 # and fail if any step in a pipe does.
11 set -e
12 set -u
13 set -o pipefail
14 
15 GIT_REPO_BASE_DIR="/var/www/bl.test"
16 BASE_DOMAIN="bl.test"
17 
18 USERNAME="$1"
19 REPO_URL="$2"
20 REPO_DIR="${GIT_REPO_BASE_DIR}/${USERNAME}.${BASE_DOMAIN}"
21 SUB_DOMAIN="${USERNAME}.${BASE_DOMAIN}"
22 
23 echo "Creating user and add to group."
24 useradd "${USERNAME}"
25 usermod -aG deploy "${USERNAME}"
26 
27 echo "Preparing repo."
28 mkdir -p "${REPO_DIR}"
29 chown jujens:nginx "${REPO_DIR}"
30 chmod u=rwX,g=rXs,o=- "${REPO_DIR}"
31 git clone "${REPO_URL}" "${REPO_DIR}"
32 
33 # Adapt to your needs and language.
34 echo "Preparing venv"
35 pushd "${REPO_DIR}"
36     echo "Opening .env file"
37     ${EDITOR:-vim} .env
38 
39     python3 -m venv .venv
40     source ".venv/bin/activate"
41     pip install -r requirements.txt
42 pushd
43 echo "Launching service"
44 systemctl start "gunicorn@${SUB_DOMAIN}"
45 systemctl enable "gunicorn@${SUB_DOMAIN}"
46 
47 
48 # That's assuming you use the default SSH port, if not, adapt.
49 echo "To add the git remote: git remote add test ${USERNAME}@${BASE_DOMAIN}:${REPO_DIR}"
50 echo "Done"

Full configurations

nginx configuration:

 1 server {
 2     listen 80;
 3     server_name *.bl.test;
 4     root /var/www/bl.test;
 5 
 6     auth_basic "Test server";
 7     auth_basic_user_file /etc/nginx/bl.test.passwd;
 8 
 9     # Prevent access to pyc and py files.
10     location ~ .*\.pyc? {
11         return 404;
12     }
13 
14     location ~ .git {
15        return 404;
16     }
17 
18     location ~ ^/.env$ {
19         return 404;
20     }
21 
22     location / {
23         try_files /$host/$uri @gunicorn;
24     }
25 
26     # Relay everything else to the django web server.
27     location @gunicorn {
28         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
29         proxy_set_header Host $http_host;
30         proxy_set_header X-Forwarded-Proto $scheme;
31         proxy_redirect off;
32         proxy_pass http://unix:/var/run/gunicorn/$host.sock;
33     }
34 }

gunicorn configuration

 1 [Unit]
 2 Description=gunicorn daemon
 3 After=network.target
 4 
 5 [Service]
 6 PIDFile=/var/run/gunicorn/%i.pid
 7 User=nginx
 8 Group=nginx
 9 EnvironmentFile=/var/www/bl.test/%i/.env
10 Environment="PATH=/var/www/bl.test/%i/.venv/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
11 RuntimeDirectory=gunicorn
12 WorkingDirectory=/var/www/bl.test/%i
13 ExecStart=/var/www/bl.test/%i/.venv/bin/gunicorn \
14           --pid /var/run/gunicorn/%i.pid   \
15           --bind unix:/var/run/gunicorn/%i.sock \
16           hello:app
17 ExecReload=/bin/kill -s HUP $MAINPID
18 ExecStop=/bin/kill -s TERM $MAINPID
19 PrivateTmp=true
20 Restart=on-failure
21 RestartSec=5s
22 
23 [Install]
24 WantedBy=multi-user.target

git hooks:

pre-receive

1 #!/usr/bin/env bash
2 
3 # We launch the command in a subshell to avoid impacting the true environment with our unset.
4 # We unset GIT_DIR because it points to the wrong directory.
5 # We are in the .git directory when this hook is run, so we need to move one step up to run git commands.
6 # We then force a checkout of the repo to discard any changes.
7 ( unset GIT_DIR && cd .. && git checkout -f )

post-receive

 1 #!/usr/bin/env bash
 2 
 3 # Parameters are passed to stdin, we need to use read to get them. Only ref matters for what we do.
 4 read oldrev newrev ref
 5 # By default, ref contains refs/heads/branch-name. We use bash substitution mechanism to remove "refs/heads/".
 6 branch_name=${ref/refs\/heads\//}
 7 
 8 # Once again, force a checkout to the branch.
 9 ( unset GIT_DIR && cd .. && git checkout -f $branch_name )
10 
11 # You can add additionnal steps to complete checkout on the current branch (install deps, build JS/CSS files, …)
12 # For instance:
13 ( cd .. && source .venv/bin/activate && pip install -r requirements.txt )
14 
15 # We move to the folder that contains our code.
16 # We extract the domain (and guincorn instance name) from the folder
17 # by keeping only the name of the current directory of the full path.
18 # Finally, we restart the proper instance of gunicorn.
19 ( cd ..; unit_name=$(pwd); unit_name=${unit_name##*/}; sudo systemctl restart "gunicorn@${unit_name}" )
20 # You could simply this line by using the $USER variable and creating BASE_DOMAIN="bl.test" into ``sudo systemcl restart "gunicorn@${USER}.${BASE_DOMAIN}"``

Use dnsmasq with NetworkManager

2018-09-08T00:00:00+02:00

This will, for instance, allow you to redirect all matching domains to a certain server: dnsmasq will intercept the DNS query and return the ip you specified.

To do this, edit NetworkManager configuration in /etc/NetworkManager/NetworkManager.conf and add in the main section dns=dnsmasq:

[main]
dns=dnsmasq

You can then create a file in /etc/NetworkManager/dnsmasq.d/redirect.conf to redirect the domain you want. For instance to redirect jujens.test to localhost host, add this line:

address=/jujens.test/127.0.0.1

VSCode tips

2018-09-02T00:00:00+02:00

Sommaire

Python
- venv

Python

venv

To use a venv with VSCode, add in the folder settings:

{
    "python.pythonPath": "~/.virtualenvs/bureauxlocaux--Uxk5jFn/bin/python",
    "python.unitTest.pyTestEnabled": true
}

Deploy your source maps to Rollbar with Ansible

2018-07-16T00:00:00+02:00

I recently had to integrate Rollbar (a service to track errors) with a JS application. One nice thing about Rollbar is that if you provide it with source maps, it will point to you where the error is in the original unminified code.

Since the application is deployed with Ansible, I wanted to upload the source maps with Ansible. There is a built in module to register deploys with Ansible: rollbar_deployment but that only solves half the problem since it only registers deployments. Example of usage:

- name: Register deploy
  rollbar_deployment:
    token: "{{ lookup('env','ROLLBAR_API_KEY') }}"
    environment: "{{ env }}"
    revision: "{{ version }}"

with:

{{ env }} the environment you are deploying to.
{{ version }} the revision you want to register.
You also need the ROLLBAR_API_KEY to be set and to contain your Rollbar server API key so you can authenticate to Rollbar.

Since I didn't found a module already done, I decide to build my own.

Below is the code of the Ansible module. Place it under library/rollbar_source_maps.py in your Ansible project. Note: it only works with Python 3+ and requires the requests library.

You will then be able to use it with:

- hosts: fronts
  tasks:
    - name: Upload source maps to Rollbar
      connection: local
      rollbar_source_maps:
        src: "{{ src }}/dist/scripts"
        base_url: "{{ base_url }}"
        version: "{{ version }}"
        token: "{{ lookup('env','ROLLBAR_API_KEY') }}"

Where:

{{ src }} points to the root of your JS project. The src key in the YAML must point to the folder that contains your source maps. Feel free to adapt to your path.
{{ base_url }} points to the base URL of your scripts. So, if they are all located under http://aot.testing/dist/scripts/ that what you must use here.
{{ version }} is the current version of the code you are deploying so Rollbar can track your deploys and help you find the one that caused a problem.
You also need the ROLLBAR_API_KEY to be set and to contain your Rollbar server API key so you can authenticate to Rollbar.

#!/usr/bin/python3

# Copyright 2018, Julien Enselme, <jujens@jujens.eu>
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

ANSIBLE_METADATA = {
    'metadata_version': '1.0',
    'status': ['preview'],
    'supported_by': 'community',
}

DOCUMENTATION = '''
---
module: rollbar_source_maps
version_added: 1.0
author: "Julien Enselme (@Jenselme)"
short_description: Upload source maps to Rollbar
description:
- Upload source maps from a folder to Rollbar. We associate scripts and source maps thanks to the base URL and the name of the files. A source map must have the same name as the files and ends with .map
    (see https://docs.rollbar.com/docs/source-maps)
options:
token:
    description:
    - Your project access token.
    required: true
src:
    description:
    - The directory in which to look for source maps.
    required: true
base_url:
    description:
    - The base URL of your site so Rollbar can have the mapping of a file on the site to a file on the local system.
    required: true
version:
    description:
    - The version of the source maps.
    required: true
url:
    description:
    - Optional URL to submit the notification to.
    required: false
    default: 'https://api.rollbar.com/api/1/sourcemap'
validate_certs:
    description:
    - If C(no), SSL certificates for the target url will not be validated.
        This should only be used on personally controlled sites using
        self-signed certificates.
    required: false
    default: 'yes'
    choices: ['yes', 'no']
'''

EXAMPLES = '''
- name: Upload source maps to Rollbar
rollbar_source_maps:
    src: dist
    base_url: http://www.example.com/scripts/
    version: v1.0.0
    token: AAAA
'''

import os
import traceback

from pathlib import Path
from urllib.parse import urljoin

import requests

from ansible.module_utils.basic import AnsibleModule
from ansible.module_utils.six.moves.urllib.parse import urlencode
from ansible.module_utils._text import to_native
from ansible.module_utils.urls import fetch_url


def main():

    module = AnsibleModule(
        argument_spec=dict(
            token=dict(required=True),
            src=dict(required=True),
            base_url=dict(required=True),
            version=dict(required=True),
            url=dict(
                required=False,
                default='https://api.rollbar.com/api/1/sourcemap'
            ),
            validate_certs=dict(default='yes', type='bool'),
        ),
        supports_check_mode=True,
    )

    if module.check_mode:
        module.exit_json(changed=True)

    params = dict(
        access_token=module.params['token'],
    )

    base_url = module.params.get('base_url')
    version = module.params.get('version')
    url = module.params.get('url')
    src = module.params.get('src')

    for file in Path(src).glob('*.js.map'):
        with file.open('rb') as source_map:
            source_map_content = source_map.read()

        minified_url = urljoin(base_url, file.name).replace('.map', '')

        try:
            resp = requests.post(url, files={
                'source_map': source_map_content,
            }, data={
                'access_token': module.params.get('token'),
                'minified_url': minified_url,
                'version': version,
            })
        except Exception as e:
            module.fail_json(msg='Unable to upload source maps to Rollbar: %s' % to_native(e), exception=traceback.format_exc())
        else:
            if resp.status_code != 200:
                module.fail_json(msg='HTTP result code: %d connecting to %s with error %s' % (resp.status_code, url, resp.text))

    module.exit_json(changed=True)


if __name__ == '__main__':
    main()

Timeout a function in python

2018-06-02T00:00:00+02:00

You can use signals and a context manager to acheive this. The idea is to register a function that will raise an exeption when the alarm signal is received and schedule the alarm signal for the timeout.

import signal
from contextlib import contextmanager


@contextmanager
def timeout(time):
    # Register a function to raise a TimeoutError on the signal.
    signal.signal(signal.SIGALRM, raise_timeout)
    # Schedule the signal to be sent after ``time``.
    signal.alarm(time)

    try:
        yield
    except TimeoutError:
        pass
    finally:
        # Unregister the signal so it won't be triggered
        # if the timeout is not reached.
        signal.signal(signal.SIGALRM, signal.SIG_IGN)


def raise_timeout(signum, frame):
    raise TimeoutError


def my_func():
    # Add a timeout block.
    with timeout(1):
        print('entering block')
        import time
        time.sleep(10)
        print('This should never get printed because the line before timed out')

Even if this solution works well, you can't nest timeout which can be a problem. Perhaps a solution relying on concurrent.futures may work but during my tests, I never managed to correctly stop a function once a timeout has been reached.

Source: https://stackoverflow.com/a/494273/3900519

Some tips for django

2018-05-21T00:00:00+02:00

Sommaire

View last executed query
Logging
- Queries
Migrations
- Checks
- Fake all pending migrations
Model proxies
Easily increment migrations numbers
Create a widget with custom display
Use a nginx reverse proxy in dev
Checking PO files
Testing

View last executed query

Run (the query will be as executed by the database with values correctly replaced and escaped):

from django.db import connection
print(connection.queries[-1])

Logging

Queries

You can configure your logger to view all requests made to the database with:

LOGGING = {
    # ...
    'loggers': {
        # ...
        'django.db': {
            'handlers': ['console'],
            'level': 'DEBUG',
        },
    },
}

Source: Surviving Django (if you care about databases) under Another random bit of advice.

Migrations

Checks

You can use this check to verify your migrations during CI (no dependency issue and no creation):

python manage.py makemigrations
if [[ $(git status --porcelain | grep migrations | wc -l) -gt 0 ]]; then
    echo 'New migrations were created in the project. Please fix that.' >&2
    exit 1
fi

If you want to use it in a git hook, you should only consider untracked migrations:

python manage.py migrate
migs_exit_code=$?

if [[ "${migs_exit_code}" -ne 0 ]]; then
    exit 1
fi

if [[ $(git status --porcelain | grep migrations | grep .py | grep '??' | wc -l) -gt 0 ]]; then
    echo 'New migrations were created in the project. Please fix that.' >&2
    exit 1
fi

Fake all pending migrations

#!/usr/bin/env bash

set -eu
set -o pipefail

# This will contain a list like this:
#wagtailusers
# [ ] 0001_initial
# [ ] 0002_add_verbose_name_on_userprofile
# [ ] 0003_add_verbose_names
# [ ] 0004_capitalizeverbose
# [ ] 0005_make_related_name_wagtail_specific
# [ ] 0006_userprofile_prefered_language
# [ ] 0007_userprofile_current_time_zone
# [ ] 0008_userprofile_avatar
# [ ] 0009_userprofile_verbose_name_plural

pending_migrations="$(python manage.py showmigrations | grep --color '\[ \]\|^[a-z]' | grep --color '[  ]' -B 1)"

declare -A app_to_last_migration

# Make sure we loop over lines, not words.
IFS=$'\n'
# Array must not be quoted to correctly loop over each line.
# shellcheck disable=SC2068
for line in ${pending_migrations[@]}; do
    if [[ "$line" =~ ^[a-z]+ ]]; then
        app_name="${line}"
    elif [[ "$line" =~ \[ ]]; then
        # Capture the last migration of the app to pass it and all migrations before it.
        migration_name=$(echo "${line}" | awk '{print $3}')
        app_to_last_migration["${app_name}"]="${migration_name}"
    fi
done

for app_name in "${!app_to_last_migration[@]}"; do
    python manage.py migrate ${app_name} ${app_to_last_migration[${app_name}]} --fake
done

Model proxies

They are useful to change the behavior of a model (based on a type column for instance) or to work on a subset of a table (based on a type column for instance). See the documentation for more details.

from django.db import models

class Person(models.Model):
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)

class MyPerson(Person):
    class Meta:
        proxy = True

    def do_something(self):
        # ...
        pass

class OrderedPerson(Person):
    class Meta:
        ordering = ["last_name"]
        proxy = True

Easily increment migrations numbers

Here is a small bash script to increment the numbers of your migrations. Put it into your ~/.profile or ~/.bashrc. Use it like: inc-migrations PROJECT/apps/APP_NAME/migrations 0012

It will rename all migrations from 0012 (ie transform 0012_mig.py into 0013_mig.py and so on) and replace all occurrences of the previous name (eg 0012_mig.py) with the new one (eg 0013_mig.py).

function inc-migrations() {
    local folder="$1"
    local number="$2"
    local start_file
    local new_number
    local next_file

    if ! [[ "$(pwd)" =~ "${folder}$" ]]; then
        cd "${folder}"
    fi

    if [[ -f "${number}" ]]; then
        start_file="${number}"
        number=$(echo "${number}" | cut -d _ -f 1)
    else
        start_file=$(ls ${number}* 2> /dev/null)
    fi

    if [[ ! -f "${start_file}" ]]; then
        echo "${start_file} doesn't exits" >&2
        return 1
    fi
    let "new_number = ${number} + 1"
    new_number=$(printf "%04d\n" "${new_number}")
    next_file=$(ls ${new_number}* 2> /dev/null)

    new_file_name="${start_file/${number}/${new_number}}"
    git mv "${start_file}" "${new_file_name}"
    sed -i "s/${start_file/.py/}/${new_file_name/.py/}/g" *.py

    if [[ -f "${next_file:-}" ]]; then
        inc-migrations "${folder}" "${next_file}"
    fi
}

Create a widget with custom display

class BonusTimeWidget(AdminIntegerFieldWidget):
    UNITS_TO_TRANSLATIONS = {
        'month': partial(ungettext_lazy, '%(count)d month', '%(count)d free months'),
    }

    def __init__(self, *args, unit='month', **kwargs):
        super().__init__(*args, **kwargs)
        if unit not in self.UNITS_TO_TRANSLATIONS:
            supported_units = ','.join(self.UNITS_TO_TRANSLATIONS.keys())
            raise ValueError(
                f'{unit} is not a supported unit. Supported units are: {supported_units}',
            )

        self.bonus_translation = self.UNITS_TO_TRANSLATIONS[unit]

    def render(self, name, value, attrs=None):
        """Render the value in a custom span."""
        if value == 0:
            return ''

        text = self.bonus_translation(value) % {'count': value}
        return mark_safe(f'<span>+ {text}</span>')

Use a nginx reverse proxy in dev

The django web server works well but can be slow when it needs to handle many requests (to load many images for instance). One way to solve this is to use a production web server (nginx in this case) to handle most of the work (ie everything but dynamically generated pages).

Prerequisites:

Install nginx
Add the domain you want to use in your /etc/hosts file. For instance 127.0.0.1 myproject.localhost.
Make sure nginx has access to the files of your project. Most of the time a chmod 755 /PATH/TO/PROJECT will do (repeat on each subdirectory nginx need to pass to access to your files). If you are using a shared computer, you may need to think about a more secure way to allow nginx to access the file (ACL may help you).
If you are using SELinux, don't forget to add the proper context to the files. For instance, do something like:
1. Add these files to the proper SELinux context by copying the one from the default web folder: semanage fcontext --add --equal /var/www/html /PATH/TO/PROJECT
2. Restore the context of the files: restorecon -R /home/jenselme/Work/bureauxlocaux
3. Check that the context is correct: ls -Z The output should contain something like system_u:object_r:httpd_sys_content_t:s0.

Here is the nginx configuration to put in /etc/nginx/conf.d (or /etc/nginx/sites-enabled):

server {
    listen 80;
    # Make sure this host is in the ALLOWED_HOSTS variable in the settings.
    server_name PROJECT.localhost;
    root /PATH/TO/PROJECT;

    # Prevent access to pyc and py files.
    location ~ .*\.pyc? {
        return 404;
    }

    # Search for files in the media folder. Change this if you configured Django to store your uploaded files elsewhere.
    location ~ ^/files/(.*) {
        try_files /media/$1 =404;
    }

    # Look for static files in the static folder or at the root of the project.
    location ~ ^/static {
        # Look both in the production static folder at the root of you project and in PROJECT
        # (where you have the apps directory and the static directory you use in dev).
        try_files /PROJECT/$uri /$uri $uri;
    }

    # Relay everything else to the django web server.
    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://127.0.0.1:8000;
    }
}

To add a connect timeout (eg to mimic Heroku's timeout) to the Django dev server, add the lines below in the @django block:

proxy_connect_timeout   30;
proxy_send_timeout      30;
proxy_read_timeout      30;
send_timeout            30;

Checking PO files

#!/usr/bin/env bash

set -eu

for file in "$@"; do
  msgfmt -v --check "${file}"
done

Testing

Inserting pytest fixtures in a test case

class SpamTest(unittest.TestCase):

    @pytest.fixture(autouse=True)
    def inject_fixtures(self, caplog):
        self._caplog = caplog

    def test_eggs(self):
        with self._caplog.at_level(logging.INFO):
            spam.eggs()
            assert self._caplog.records[0].message == 'bacon'

Source: https://stackoverflow.com/a/50375022

Reset PK sequence

def reset_database_sequences(*models_or_factories):
    models = [
        model_or_factory._meta.model for model_or_factory in models_or_factories
    ]
    sequence_sql = connection.ops.sequence_reset_sql(no_style(), models)
    with connection.cursor() as cursor:
        for sql in sequence_sql:
            cursor.execute(sql)

Source: https://stackoverflow.com/a/50275965/3900519

Reset factoryboy sequence

def reset_sequences(*factories):
    reset_database_sequences(*factories)
    for factory_cls in factories:
        factory_cls.reset_sequence()

JavaScript tips

2018-03-07T00:00:00+01:00

Sommaire

Normalize an UTF-8 string

Normalize an UTF-8 string

Rely on String.prototype.normalize().

const str = "Crème Brulée"
str.normalize('NFD').replace(/[\u0300-\u036f]/g, "")
> 'Creme Brulee'

Source: https://stackoverflow.com/questions/990904/remove-accents-diacritics-in-a-string-in-javascript/37511463#37511463

PostgreSQL tips

2018-03-07T00:00:00+01:00

Sommaire

Update fields in JSONB format
CSV export/import
Update rows in bulks
Understand lock issues
Use variables in query
Database sizes

Update fields in JSONB format

Use the jsonb_set(COLUMN_NAME, PATH_TO_CHANGE, VALUE) function. The value must be a valid value in JSON (ie use '500' for numbers, 'true' for boolean and '"str"' for strings). Examples:

-- Set the postcode field to the 92600 integer.
UPDATE sales SET address = jsonb_set(address, '{postcode}', '92600') WHERE id = 47880;
-- Set a nested field to the "FRA" value. Note: the nested object must exists.
UPDATE sales SET address = jsonb_set(address, '{country,code}', '"FRA"');
-- Set a value at a given position in an array (an object must exist at this position).
UPDATE sales SET phones = jsonb_set(phones, '{0,number}', '"06"');

CSV export/import

To export content of a table to a CSV file (eg to re-import it in a different database on a different server). Update the query and connection options to fit your needs:

psql -U user -c 'COPY (SELECT * FROM sales WHERE id = 110683) To STDOUT WITH CSV HEADER;' database > /tmp/sales.csv

To import the data from a CSV file, you need to specify the columns you want to import. To find them, run head -n 1 /tmp/sales.csv to get the first lines of the CSV which contains the headers. You can then run the command below.

psql -U user database -c "COPY sales(id,organization_id,date,name,address) FROM STDIN WITH CSV HEADER;" < /tmp/sales.csv

Note 1: it should be possible to import the data with psql -U user database -c "COPY sales(id,organization_id,date,name,address) FROM '/tmp/sales.csv' WITH CSV HEADER;". However, when I tried that, I got: ERROR: could not open file "/tmp/sale.csv" for reading: No such file or directory even though the file existed.

Note 2: you can also achieve this with pgAdmin with you want a nice GUI.

Update rows in bulks

This will update rows by batch of 100 without being blocked by locks. It is useful to prepare for the addition of an NOT NULL constraint without downtime.

UPDATE server_info
SET    status = 'active'
WHERE  server_ip = (
    SELECT server_ip
    FROM   server_info
    WHERE  status = 'standby'
    LIMIT  100
    FOR    UPDATE SKIP LOCKED
)

Understand lock issues

This will display all queries blocked by locks and which query is blocking them (pg 9.6+):

SELECT pid,
    usename,
    pg_blocking_pids(pid) AS blocked_by,
    state,
    age(now(), query_start) AS age,
    query AS blocked_query
FROM pg_stat_activity
ORDER BY age DESC NULLS LAST;

Source: https://stackoverflow.com/a/43363536

Use variables in query

Use a CTE:

WITH myconstants (var1, var2) as (
    values (5, 'foo')
)
SELECT *
FROM somewhere, myconstants
WHERE something = myconstants.var1
OR something_else = myconstants.var2;

Source: https://stackoverflow.com/a/54220318/3900519

Database sizes

To view the size of tables in a database:

SELECT nspname || '.' || relname AS "relation",
    pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
FROM pg_class C
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE nspname NOT IN ('pg_catalog', 'information_schema')
    AND C.relkind <> 'i'
    AND nspname !~ '^pg_toast'
ORDER BY pg_total_relation_size(C.oid) DESC
LIMIT 20;

To view the sizes of databases:

SELECT
  pg_database.datname,
  pg_size_pretty(pg_database_size(pg_database.datname)) AS size
  FROM pg_database
  ORDER BY size DESC;

Source: http://www.postgresqltutorial.com/postgresql-database-indexes-table-size/

Run multiple docker daemons on the same host

2018-02-25T00:00:00+01:00

Today I am going to explain how you can run multiple docker daemons on the same host. This can be useful if you want several users to use docker and want each of them to be isolated from one another (ie don't see images, containers, … of other users). The solution relies on user namespaces [1] and sudo.

Sommaire

Configure sudo
Configure username spaces
Docker unit file
Usage
docker-compose
Conclusion

Configure sudo

First, we will update the sudoers file by launching visudo to allow the users to use the docker command and force them to use the proper daemon. To do that, append the following lines at the end of the file:

USER ALL=(root) NOPASSWD: /usr/bin/docker -H unix\:///var/run/docker-USER.sock *, ! /usr/bin/docker *--priviledged*, ! /usr/bin/docker *host*

You must replace USER (present two times) by the username of the user you want to allow (you must duplicate the line for each user you want to allow). This will allow USER to launch the docker command as root, without a password, while restricting the user to use its docker instance by talking to the proper socket. We also prevent the user to run a command that contains the priviledged option and host keyword. This is meant to avoid them disabling namespaces for security reasons. If we don't, they could run containers as true root on the machine. This would make our isolation less useful. If you don't care about that, you can simply use:

USER ALL=(root) NOPASSWD: /usr/bin/docker -H unix\:///var/run/docker-USER.sock *

Configure username spaces

Edit /etc/subuid and /etc/subgid so you have two lines per users which will look like (depending on your system, you may already have the second line):

USER:USERID:1
USER:1XXXXX:65536

The first line is used to map the user of id 1 in the container (ie root) to the id of a normal user outside. The second line, defines which uid will be used to map other users: the first number gives the first id and the second the number of ids. The ranges must not collide between users. Since this is still probably obscure, let me give an example:

I have a user named jenselme. It has uid 1000 on the system.
I have a user named julien. It has uid 1001 on the system.
To enable mapping of root in the container to jenselme on the host for jenselme's containers, I add jenselme:1000:1 to /etc/subuid and /etc/subgid
To enable mapping of other users in jenselme's containers, I add: jenselme:100000:65536. So users will be dispatched to host uids between 100000 and 165535. For instance, user with id 33 in a container will have id 100032 on the host (id mapping starts at 100000, hence 100032 and not 100033).
To enable mapping of root in the container to julien on the host for julien's containers, I add julien:1001:1 to /etc/subuid and /etc/subgid
To enable mapping of other users in julien's containers, I add: julien:165536:65536. So users will be dispatched to host uids between 165536 and 231071. Which are the 65536 ids avaiable after jenselme's ids range.

So my files will contain:

jenselme:1000:1
jenselme:100000:65536
julien:1001:1
julien:165536:65536

To learn more about this, you can read my article dedicated to this subject

Docker unit file

To do that, you first need to create a docker@.service file so you can select for which user you want to run docker. This file must be located under /etc/systemd/system/ and have this content:

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker-containerd.service
Wants=docker-storage-setup.service
Requires=docker-containerd.service rhel-push-plugin.socket registries.service

[Service]
Type=notify
EnvironmentFile=/run/containers/registries.conf
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
ExecStart=/usr/bin/dockerd-current \
          --add-runtime oci=/usr/libexec/docker/docker-runc-current \
          --default-runtime=oci \
          --authorization-plugin=rhel-push-plugin \
          --containerd /run/containerd.sock \
          --exec-opt native.cgroupdriver=systemd \
          --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
          --init-path=/usr/libexec/docker/docker-init-current \
          --seccomp-profile=/etc/docker/seccomp.json \
          --userns-remap %i \
          --host unix:///var/run/docker-%i.sock \
          --pidfile /var/run/docker-%i.pid \
          $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $ADD_REGISTRY \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY \
          $REGISTRIES
ExecReload=/bin/kill -s HUP $MAINPID
TasksMax=8192
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

I added three command line options to the standard service. The %i will be replace by what comes after the @ when we will run docker (eg USER with systemctl start docker@USER.service). Let me detail the new options:

--userns-remap %i: enables the username space for the user. This way, each users won't be able to see/use the images and containers of each other: when usermapping is enabled, docker keeps everything (images, containers, networks, volume …) separate for each mapping.
--host unix:///var/run/docker-%i.sock: change the name of the socket to listen to. If we don't do this, all our instances would try to listen to unix:///var/run/docker.sock which wouldn't work.
--pidfile /var/run/docker-%i.pid: change the name of the PID file of the process. If we don't do this, all our instances would try to use /var/run/docker.pid which wouldn't work.

If you want to use a different docker configuration file for each instance, you can add --config-file /etc/docker/daemon-%i.json to the list of option. This will make docker use /etc/docker/daemon-USER.json as a config file when running docker@USER. Note: the file must exist and be valid JSON, otherwise, docker won't start.

Note: I have commented the part that differs from the standard docker.service on fedora. If you use a different distribution, the content of this file may differ but you should be able to adapt it from mine. If you have a question, please leave a comment.

Don't forget to make systemd take the new file into account with systemctl daemon-reload.

Usage

At this point, everything is functional, you can start your daemons with systemctl start docker@USER.service. To use a daemon, you need to specify to docker which socket to use with the -H option like this: sudo docker -H unix:///var/run/docker-USER.sock. To ease the process, you can create an alias like this in your .bashrc file (adapt to your shell):

alias docker="sudo docker -H unix:///var/run/docker-$(whoami).sock"

After opening a new shell or doing source ~/.bashrc, you can use the docker command directly, eg:

docker run --rm busybox ls

I suggest you play around with it on multiple accounts to see how all of this works before read the end of the article.

docker-compose

You can use docker-compose like this sudo docker-compose -H unix:///var/run/docker-$(whoami).sock if you add the line below to you sudoers:

USER ALL=(root) NOPASSWD: /usr/bin/docker-compose -H unix:///var/run/docker-USER.sock *

However, since docker compose does HTTP request to your docker socket, you cannot rely on sudo to prevent the user from launching priviledged or unamespaced containers. You will need an authorization plugin for that like HBM or Twistlock AuthZ Broker. Note: I have not tested these plugins.

Conclusion

Everything should be working and you should be able to run as many docker daemons as you need. I hope I was clear on this quite technical subject. If I wasn't or if you just want to leave a remark, please leave a comment below!

[1]	User namespaces allow to differentiate user ids inside the container from user ids outside. For instance root (ie uid 1) in the container will be a standard user outside of it (eg uid 1000). To learn more about user namespaces and their usage with docker, read this article.

Can we clean password from PHP memory?

2017-11-26T00:00:00+01:00

At work in a PHP application, we rely on libsodium to erase a password from $_POST. It may sound like a good idea: once the password is not in memory any more, it can't leak. But the question is: is it really erased from memory? That's the question will answer here.

To do this, we will create a docker container with PHP script to check this. We will dump the memory of the process into a file and we will inspect this file to see if the password is in it.

You can the scripts and the Dockerfile here if you want to run all tests by yourself.

First, let's look at the Dockerfile:

FROM php:7.1-apache

COPY *.php /var/www/html/

# Install libsodium
RUN apt-get update \
    && apt-get install build-essential -y \
    && curl -fsSL 'https://github.com/jedisct1/libsodium/releases/download/1.0.11/libsodium-1.0.11.tar.gz' -o libsodium.tar.gz \
    && mkdir -p libsodium \
    && tar -xf libsodium.tar.gz -C libsodium --strip-components=1 \
    && rm libsodium.tar.gz \
    && cd libsodium && ./configure && make check && make install \
    && cd .. && rm -r libsodium
RUN pecl install libsodium-1.0.6 \
    && docker-php-ext-enable libsodium

# Install packages to dump memory and inspect the dump
RUN apt-get install -y gdb

RUN rm -rf /var/lib/apt/lists/*

Nothing fancy: we start from an image that include Apache and PHP 7.1. We then install libsodium to erase the memory and gdb to have tools to dump the memory and analyse the dump.

Now the scripts. I made 2. The first check-post.php is meant to check that the value in $_POST is correctly erased:

<?php

// Before clean
var_dump($_POST);

Sodium\memzero($_POST['password']);

// After clean.
var_dump($_POST);

The second is meant to dump the memory of the process: we erase the memory then run an infinite loop so we have time to dump the memory of the process:

<?php

Sodium\memzero($_POST['password']);

while(true);

I didn't do var_dump in this one to be sure the password won't be in memory somewhere.

Let's build the image: docker build -t php-clean-password-tests .

Let's run it with docker run -d --name php-clean-password-tests -p 8080:80 php-clean-password-tests

Note: if you use user namespaces, you must start the container with docker run -d --name php-clean-password-tests -p 8080:80 --userns=host --privileged php-clean-password-tests instead, because you need to be the true root to dump the memory of a process.

Let's tests the first script with httpie or good old curl:

http -f POST localhost:8080/check-post.php password=superpasswd
curl -X POST -d 'password=superpasswd' localhost:8080/check-post.php

We have this output (with httpie):

HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 81
Content-Type: text/html; charset=UTF-8
Date: Sun, 26 Nov 2017 10:00:07 GMT
Keep-Alive: timeout=5, max=100
Server: Apache/2.4.10 (Debian)
Vary: Accept-Encoding
X-Powered-By: PHP/7.1.11

array(1) {
["password"]=>
string(11) "superpasswd"
}
array(1) {
["password"]=>
NULL
}

So far so good! Before running the second script, let's enter in the container with: docker exec -it php-clean-password-tests /bin/bash

Run http -f POST localhost:8080/wait-memory-dump.php password=superpasswd in your host shell to launch the second script.

In the container, run ps aux to list all running processes. You should see an apache2 command with non zero time, like:

www-data    17  9.5  0.0 173156 11916 ?        R    10:10   0:02 apache2 -DFOREGROUND

That's the process running our script. Get the PID (first number from the left, 17 in this example) and dump the memory with:

gcore 17  # Replace 17 by the PID you got before

You should now have a file named like core.17 in the current directory. This is a binary file we could inspect directly with gdb but to keep things simple, we will convert it to text with objdump:

objdump -s core.17 > core.17.txt

Now, let's grep for our password:

grep pass core.17.txt

Note: given how the file is structured, search direcly for a long string won't work. Try breaking your password into sequence of 3/4 characters to find it.

Here is the result (excerpt):

563aaa079c90 67696e5f 73686132 35365f70 61737377  gin_sha256_passw
7fd64d457030 70617373 776f7264 3d737570 65727061  password=superpa
7fd64d45d090 70617373 776f7264 00000000 00000000  password........
(…)
7ffd7df771b0 70617373 776f7264 3d737570 65727061  password=superpa
7ffd7df78e50 70617373 776f7264 0826f252 d67f0000  password.&.R....
7ffd7df79050 70617373 776f7264 3d737570 65727061  password=superpa

Conclusion: the password is still in memory. So it can still leak and the memzero wasn't that useful (you wouldn't output the password from $_POST anywhere anyway, right?). Where can it come from? Since the script are deliberately small, I suspect it comes from php://input a virtual, read only file where PHP stores the raw input of the request. You can read it with file_get_contents('php://input') and as far as I know, you can't erase it with memzero. Now imagine a true application: you validate the password? Are you sure it is not copied somewhere by your validation code? You use an ORM? Did it copy it somewhere for its internal usage? I guess that to be sure the password is just present in one place and erase it correctly from that one place is a herculean thing to do.

Last question: does it matter? From my point of view no. In modern web development, we don't manage the memory ourselves which makes the risk of leaking it by accident very low (unlike in C if you recall the heartbleed vulnerability). Which means, if you make sure your application core dumps (memory dump generated in cases of same application crash) are disable or not accessible from the outside, you are good. And between you and me, if an attacker can dump memory of a process, (s)he is root on your server, so your security sucks. Period.

So I don't think your application will gain much security if you do try to clean password from memory yourself (and you probably can't do it anyway or at least not without breaking promises from your programming language). The only thing it may do is give you a false sense of security. This doesn't meant you should stick to good password related practices:

Hash them with algorithms designed to hash passwords like bcryt or argon2 (and not sha related algorithm or worst md5).
Slat them.
Don't load the hash from the database unless you need it.
Don't put silly restrictions on the password content or length (but check that it is strong).
Protect yourself against XSS and CRSF
Secure you servers and databases

How to cache Python module in Gitlab CI

2017-08-20T00:00:00+02:00

By default, pip cache will be in ~/.pip. However, this folder cannot be cached by Gitlab. The trick is to force pip to use a folder located in the build directory with the --cache-dir option. You can then cache this folder.

For instance, you can use .pip as in the .gitlab-ci.yml excerpt below:

image: python:3.6

cache:
  paths:
    - .pip

before_script:
  - mkdir -p .pip
  - pip install -U pip
  - pip --cache-dir=.pip install -r requires.txt
  - pip --cache-dir=.pip install -r tests_requires.txt

nginx tips

2017-08-19T00:00:00+02:00

CORS for multiple domains

location ~* \.(?:ttf|ttc|otf|eot|woff|woff2)$ {
    if ( $http_origin ~* (https?://(.+\.)?(domain1|domain2|domain3)\.(?:me|co|com)$) ) {
        add_header "Access-Control-Allow-Origin" "$http_origin";
    }
}

Source: https://stackoverflow.com/a/27879729/3900519

Integrate Rollbar with Aurelia

2017-08-19T00:00:00+02:00

I recently integrated Rollbar (a service to Detect, diagnose and debug errors with ease) into an Aurelia application. I explain here how I did it. I start by explaining how logging works with Aurelia and then how I integrated Rollbar into this.

Sommaire

Logging with Aurelia
Integration with Rollbar
- Using Rollbar within Aurelia
- Loading Rollbar

Logging with Aurelia

The main module to log things with Aurelia is simply named aurelia-logging. It is responsible for:

Setting the log level you want with setLevel.
Create/get loggers with getLogger. When you get a logger with this function, logs messages will be dispatched to all registered log appenders (more on that in a minute) while respecting the level of log you set.
Defining the Appender interface.

So the aurelia-logging is a generic module to deal with logging but doesn't log anything by itself. It is defined this way so it can work in various environment (eg the browser and Node) and send logs where you want (let it be the browser console or Rollbar).

To actually do some logging, we need to add at least one Appender with addAppender. For instance, you can use the aurelia-logging-console module to append log to the browser console. Or you can use au-rollbar to log the messages into Rollbar.

Let's illustrate this with an example. Let's say I configure the log level to log only warnings and above with Logger.setLevel(Logger.logLevel.warn); and that you added the ConsoleAppender and RollbarAppender to the list of appenders with Logger.addAppender(new ConsoleAppender()) and Logger.addAppender(new RollbarAppender());.

You then create your logger with let myLogger = Logger.getLogger('my-module'). You then use the logger, eg:

myLogger.warn('A warning')
myLogger.debug('A debug message')

The A warning message will be dispatched to the console and rollbar while A debug message won't be logged anywhere.

Integration with Rollbar

Using Rollbar within Aurelia

In order to integrate Rollbar into Aurelia, I created a RollbarAppender that implements the Appender interface (see the interface definition here). What it does is pretty straightforward: it dispatches incoming log messages to the proper methods of the Rollbar object. Just like in ConsoleAppender I prepend the level and id of the logger to the message.

Since according to the doc I can only give one message, I put the eventual extra parameters of the function in a JSON object to log them as extra arguments.

Here is the full code. You can download it here. You can also install it with npm install au-rollbar.

 1 export default class RollbarAppender {
 2     debug(logger, ...rest) {
 3         let mainArgs = rest[0];
 4         this.getRollbar().debug(`DEBUG [${logger.id}]: ${mainArgs}`, this._formatRest(rest));
 5     }
 6 
 7     info(logger, ...rest) {
 8         let mainArgs = rest[0];
 9         this.getRollbar().info(`INFO [${logger.id}]  ${mainArgs}`, this._formatRest(rest));
10     }
11 
12     warn(logger, ...rest) {
13         let mainArgs = rest[0];
14         this.getRollbar().warning(`WARN [${logger.id}] ${mainArgs}`, this._formatRest(rest));
15     }
16 
17     error(logger, ...rest) {
18         let mainArgs = rest[0];
19         this.getRollbar().error(`ERROR [${logger.id}]  ${mainArgs}`, this._formatRest(rest));
20     }
21 
22     _formatRest(rest) {
23         try {
24             return JSON.stingify({ extra: rest.slice(1) });
25         } catch (e) {
26             return undefined;
27         }
28     }
29 
30     getRollbar() {
31         if (!window.Rollbar) {
32             if (console.warn) {  // eslint-disable-line no-console
33                 console.warn('Rollbar is not defined');  // eslint-disable-line no-console
34             }
35 
36             return {
37                 debug: () => {},
38                 info: () => {},
39                 warning: () => {},
40                 error: () => {},
41             };
42         }
43 
44         return window.Rollbar;
45     }
46 }

You then need to import it with something like import RollbarAppender from './logging'; (or import RollbarAppender from 'au-rollbar' if you installed it with npm). You then need to configure your logging. Since I doubt you will want to enable it in development, I suggest you configure Aurelia like this (in your main.js file):

 1 import RollbarAppender from 'au-rollbar'
 2 import * as Logger from 'aurelia-logging';
 3 
 4 export function configure(aurelia) {
 5     aurelia.use.standardConfiguration();
 6 
 7     if (environment.debug) {
 8         // Only log to the console when developing (default configuration of developmentLogging)
 9         aurelia.use.developmentLogging();
10     } else {
11         // Production logging is inspired by developmentLogging
12         // This loads the ``aurelia-logging-console`` module and then configures logging.
13         aurelia.use.preTask(() => {
14             return aurelia.loader.normalize(
15                     'aurelia-logging-console',
16                     aurelia.bootstrapperName
17             ).then(name => {
18                 return aurelia.loader.loadModule(name).then(m => {
19                     // Once the module is loaded, we configure logging.
20                     // We add the ConsoleAppender so all messages are logged in the console of the browser.
21                     Logger.addAppender(new m.ConsoleAppender());
22                     // If the Rollbar exists in the window object (ie if Rollbar is loaded),
23                     // we add the RollbarAppender so messages are sent to rollbar.
24                     // If it is not, we print a warning in the browser console.
25                     if (window.Rollbar) {
26                         Logger.addAppender(new RollbarAppender());
27                     } else {
28                         console.warn('Rollbar is not defined');  // eslint-disable-line no-console
29                     }
30                     // Configure logger to log only messages of level warning or above.
31                     Logger.setLevel(Logger.logLevel.warn);
32                 });
33             });
34         });
35     }
36 
37     if (environment.testing) {
38         aurelia.use.plugin('aurelia-testing');
39     }
40 
41     aurelia.start().then(() => aurelia.setRoot('app', document.body));
42 }

Loading Rollbar

Now we need to load the Rollbar script for Rollbar to actually work. Since in addition to logging messages with the Rollbar object, Rollbar can log uncaught exception, it is better to load it as early as possible (to detect errors when Aurelia is bootstraping for instance). To achieve this, I decided to put the configuration and quickstart script (from here) into a rollbar.js script located in the scripts folder. Then, since I use the CLI to build my project, I edited aurelia_project/aurelia.json to add "scripts/rollbar.js", in the prepend section of the vendor bundle. So the definition of the vendor bundle looks like:

 1 {
 2     "name": "vendor-bundle.js",
 3     "prepend": [
 4         "node_modules/bluebird/js/browser/bluebird.js",
 5         "scripts/polyfills.js",
 6         "scripts/rollbar.js",
 7         "node_modules/requirejs/require.js"
 8     ],
 9     "dependencies": [
10         "aurelia-binding",
11         "aurelia-bootstrapper",
12         "aurelia-dependency-injection",
13         "aurelia-event-aggregator",
14         "aurelia-framework",
15         "aurelia-history",
16         "aurelia-history-browser",
17         "aurelia-loader",
18         "aurelia-loader-default",
19         "aurelia-logging",
20         "aurelia-logging-console",
21         "aurelia-metadata",
22         "aurelia-pal",
23         "aurelia-pal-browser",
24         "aurelia-path",
25         "aurelia-polyfills",
26         "aurelia-route-recognizer",
27         "aurelia-router",
28         "aurelia-task-queue",
29         "aurelia-templating",
30         "aurelia-templating-binding",
31         "text",
32         {
33             "name": "aurelia-templating-resources",
34             "path": "../node_modules/aurelia-templating-resources/dist/amd",
35             "main": "aurelia-templating-resources"
36         },
37         {
38             "name": "aurelia-templating-router",
39             "path": "../node_modules/aurelia-templating-router/dist/amd",
40             "main": "aurelia-templating-router"
41         },
42         {
43             "name": "aurelia-testing",
44             "path": "../node_modules/aurelia-testing/dist/amd",
45             "main": "aurelia-testing",
46             "env": "dev"
47         }
48     ]
49 }

If you want to load Rollbar from another location, you can use the rollbarJsUrl property in the configuration object and give it the URL from which you want to load the script.

That's it. If you have any question or remark, please leave a comment.

A sample application with Aurelia UX

2017-08-14T00:00:00+02:00

This is a follow up to my Small comparison of ionic2 and Aurelia + Framework7 for hybrid mobile applications article where I compared ionic2 and Aurelia with Framework 7 for building an hybrid applications. At the time, Aurelia UX was not yet ready (and we were expecting a different, closed source project called Aurelia Interface). Now, Aurelia UX made huge process and I was able to write the sample application in it. I describe here how it went. The code is available on github with the other applications.

As a reminder, here are the features of the TODO application:

List all the saved todos (home page)
Edit an existing todo
Add a todo
Mark a todo as done on the home page. Done todos must have their title strike-through on the home page.

I wrote the application in TypeScript and CSS, since they are the languages Aurelia UX and its showcase application are written in. Here are screenshots of the result:

I based my application on the Aurelia UX showcase app: I cloned this repo, remove the code I didn't need and started to add my code. It is structured like this:

An assets folder with the images and fonts
An aurelia_project folder that contains the configuration for the cli
A custom_typings folder for TypeScript type definitions
A scripts folder created by the cli for some useful scripts (eg require.js)
A src folder that contains the application in itself. It contains:
- The main.ts file which is the entry point to the application
- routes.ts to configure the routes of the application
- CSS files common to all the application
- app.ts, app.css and app.html which configure the layout of the page and configures the menu and the router
- A models folder that contains type definition of the application for TypeScript
- A pages folder which contains the HTML, JS and CSS code for each routes
- A services folder that contains services for the application

Now that's done, let's take a look at the code. I won't detail the main.ts too much since it is very classic. I'll just point out to line 18:

.plugin('aurelia-ux')

This is where we enable Aurelia UX in our application. If we don't, we won't be able to use its tags and attributes in our application.

In the app.ts, we import AureliaUX from the aurelia-ux module and inject it into our application. If you look at the constructor, you'll see the first interesting piece:

constructor(private ux: AureliaUX) {
    ux.design.primary = MAIN_THEME.primary;
    ux.design.accent = MAIN_THEME.accent;

    this.showNavigationMenu = false;
}

Here is the definition of MAIN_THEME:

const MAIN_THEME = {
    primary: '#009688',
    accent: '#4CAF50',
};

We are using this to set the color of the style of our application. If you look at the corresponding CSS, you can see something like:

styles.header {
    background: ${$design.primary};
    color: ${$design.primaryForeground};
}

As you may guess, thanks to the design property of the AureliaUX object, our CSS code and our TypeScript code are linked. The content of ux.design.primary will replace all the occurrences of ${$design.primary} in the CSS. And the best part is: it is dynamic! If you change ux.design.primary in your application, the style will update to reflect it. You can test this by opening the hamburger menu and clicking on the Alternate theme check box. This will call the theme changed function to update the style of the application.

To use this feature in our HTML, we must:

Require the CSS file with the ux hash:

<require from="./app.css#ux"></require>

Use styles.header as a custom attribute, like this:

<header styles.header>
    <h1 styles.header-h1>${router.currentInstruction.config.navModel.title}</h1>
    <a styles.clickable target="_self" click.delegate="toggleNavigationMenu()">
        <i styles.header-burger class="fa fa-bars" aria-hidden="true"></i>
    </a>
</header>

Simple and efficient, isn't?

Navigation in the application is handled by Aurelia's router. Since you are probably already familiar with it, I won't detail that here.

Lets take a look at the edit page. There isn't much in the TS code: we basically setup the validation (to know more about validation, see this article), rely on the storage service to load the TODO to edit (if we are editing an existing TODO) and save it in the cache. We also use this.router.navigateToRoute('list'); to navigate in the application.

The more interesting part is the HTML where I use tags defined by Aurelia UX. The form to add a TODO is located in a ux-form tag. I then use the custom tags to create:

The input for the title:

<ux-input
             placeholder="Title"
             value.bind="todo.title & validate"
></ux-input>

The textarea:

<ux-textarea
              auto-resize
              placeholder="Description"
              value.bind="todo.description & validate"
></ux-textarea>

The done checkbox:

<ux-field>
   <ux-checkbox checked.bind="todo.done">Done</ux-checkbox>
</ux-field>

The save button:

<ux-button type="raised" click.delegate="save()" disabled.bind="!canSave">Save</ux-button>

As you can see, they are simply custom elements and behave like any other custom elements would.

The list page is quite basic so I won't detail it. If you have questions, please leave a comment.

What I didn't test:

Building the application and sending it to my phone. I suspect this will be similar to what you can do with Framework 7 or ionic.
Using SCSS instead of CSS. I frankly have no idea how this would behave in conjuction with how you can interact with CSS inside Aurelia UX (things like ux.design.primary = MAIN_THEME.primary; in your JS and background: ${$design.primary}; in the CSS.

There is currently no documentation about Aurelia UX but the showcase app contains example for all the available components. You can clone the repo and launch it with npm install to install the dependencies and then au run to run it.

To conclude: I find the project promising. It's easy to work with, has some really nice features (such as the ability to have easy links between the code and the CSS). The look and feel of the application is great. The integration with Aurelia is perfect. It also looks a lot like ionic so developers used to this framework shouldn't be lost. They are however some downsides: this is still a work in progress and some components don't exist yet (eg select). I also found that the checkboxes are not always reliable. Some clicks where not taken into accounts. I don't know if this came from my browser (FireFox for desktop, I haven't tested on mobile) or from the way the check boxes are rendered (they are not real check boxes but divs so they can be styled correctly).

That's it for today. If you have a question or remark, please leave a comment below.

Chiffrer des fichiers avec GPG

2017-08-08T00:00:00+02:00

Voici deux fonctions bash qui permettent de chiffrer et déchiffrer un fichier ou un dossier avec GPG.

Les dossiers sont compressés dans une archive ZIP avant le chiffrement.

Le chiffrement laisse les fichiers d'origine intacts. Le déchiffrement laisse le fichier chiffré intact.

Ces fonctions auront leur place dans votre .bashrc ou équivalent.

Pour chiffrer : gpgcrypt FICHIER

Pour déchiffrer : gpgdecrypt FICHIER

gpgcrypt() {
    if [[ -z "$1" ]]; then
        echo "You must supply an argument"
        return 1
    elif [[ -d "$1" ]]; then
        zip -r "$1.zip" "$1"
        gpg --output "$1.zip.gpg" --symmetric "$1.zip"
    else
        gpg --output "$1.gpg" --symmetric "$1"
    fi
}

gpgdecrypt() {
    if [[ -z "$1" ]]; then
        echo "You must supply an argument"
        return 1
    else
        gpg --output "${1%.*}" --decrypt "$1"
        if [[ "$(file --mime-type --brief "${1%.*}")" == 'application/zip' ]]; then
            unzip "${1%.*}"
        fi
    fi
}

Chiffrer des fichiers avec GPG

2017-08-08T00:00:00+02:00

Here are two bash functions to crypt/decrypt files with GPG.

Folder are compressed in a ZIP file before being encrypted.

Crypting leaves original files untouched. So does decrypting.

These functions have their place in your .bashrc or equivalent.

To crypt: gpgcrypt FICHIER

To decrypt: gpgdecrypt FICHIER

gpgcrypt() {
    if [[ -z "$1" ]]; then
        echo "You must supply an argument"
        return 1
    elif [[ -d "$1" ]]; then
        zip -r "$1.zip" "$1"
        gpg --output "$1.zip.gpg" --symmetric "$1.zip"
    else
        gpg --output "$1.gpg" --symmetric "$1"
    fi
}

gpgdecrypt() {
    if [[ -z "$1" ]]; then
        echo "You must supply an argument"
        return 1
    else
        gpg --output "${1%.*}" --decrypt "$1"
        if [[ "$(file --mime-type --brief "${1%.*}")" == 'application/zip' ]]; then
            unzip "${1%.*}"
        fi
    fi
}

Extraire le HTML d'un email au format mbox

2017-08-07T00:00:00+02:00

Voici un petit script Python pour convertir un mail au format mbox en HTML. Pour que le script fonctionne, il faut soit que le corps du message soit du HTML (recommandé) soit que la première pièce jointe du message contienne le message en HTML.

Par défaut, le script traitera tous les fichiers dont l'extension est mbox dans le dossier courant. Vous pouvez lui passer le chemin vers un dossier pour qu'il traite les fichiers de ce dossier. Le HTML est sauvegardé dans un fichier du même nom que l'original, dans le même dossier avec l'extension .html.

#!/usr/bin/env python3

import sys
from glob import glob
from mailbox import mboxMessage
from os.path import join, splitext


path = '.'
if len(sys.argv) > 1:
    path = sys.argv[1]

for path in glob(join(path, '*.mbox')):
    with open(path, 'r') as mail_file:
        message = mboxMessage(mail_file.read())

    content = message.get_payload()
    if isinstance(content, str):
        html = message.get_payload(decode=True).decode('utf-8')
    else:
        html = content[0].get_payload(decode=True).decode('utf-8')
        html = html.replace('charset=iso-8859-1', 'charset=utf-8')

    file_name, _ = splitext(path)
    with open(f'{file_name}.html', 'w') as html_file:
        html_file.write(html)

Vous pouvez aussi le trouver sur github : https://github.com/Jenselme/dot-files-shell/blob/master/bin/extract-html-email.py

Extraire le HTML d'un email au format mbox

2017-08-07T00:00:00+02:00

Here is a small script to convert mail in the mbox format to HTML. For the script to work, the body of the message (recommended) or its first attachment must contain the HTML code.

By default, the script will convert all mbox files of the current folder. You can also give it a path to another folder as first argument. The HTML is saved in the same folder than the original file in a file with the same name and the .html extension.

#!/usr/bin/env python3

import sys
from glob import glob
from mailbox import mboxMessage
from os.path import join, splitext


path = '.'
if len(sys.argv) > 1:
    path = sys.argv[1]

for path in glob(join(path, '*.mbox')):
    with open(path, 'r') as mail_file:
        message = mboxMessage(mail_file.read())

    content = message.get_payload()
    if isinstance(content, str):
        html = message.get_payload(decode=True).decode('utf-8')
    else:
        html = content[0].get_payload(decode=True).decode('utf-8')
        html = html.replace('charset=iso-8859-1', 'charset=utf-8')

    file_name, _ = splitext(path)
    with open(f'{file_name}.html', 'w') as html_file:
        html_file.write(html)

You can also find it on github: https://github.com/Jenselme/dot-files-shell/blob/master/bin/extract-html-email.py

Use Linux user namespaces to fix permissions in docker volumes

2017-07-02T00:00:00+02:00

Sommaire

Configure docker
Configure the subordinate uid/gid
Tests
Bonus
- Create files
- Unix sockets
External links

Not long ago, I publish an article about using Unix sockets with docker. These sockets where in docker volumes so they could be shared between various containers. The key idea was to change the UID and GID of the user that owns the socket in the container so they match those of the user that built the image. The main issue with this approach is that it requires you to build to container with the user that will run it. This makes the solution not portable.

Hopefully, the Linux kernel allows us to use an alternative to map user id inside the container to a predictable user id outside: user id namespaces. According to wikipedia: Namespaces are a feature of the Linux kernel that isolates and virtualizes system resources of a collection of processes. Examples of resources that can be virtualized include process IDs, hostnames, user IDs, network access, interprocess communication, and filesystems. Namespaces are a fundamental aspect of containers on Linux.

For instance, thanks to the PID namespace, a process run inside a container can "think" it has the PID 1 inside a container while in fact it has another one. The same is true with user namespace: a user can "think" it has the 0 uid (root) while it fact it has the 1000 user id (some standard user). This will allow us to be sure for the files in a docker volumes that:

All files belonging to the root user in the container will belong to a user of the system that is not root in the host.
All files belonging to other users in the container will be mapped to predictable uid (more on that latter).

Configure docker

Lets configure docker to do all that.

First we either need to start the docker daemon with the --userns-remap USER flag or make sure the configuration file of the docker daemon (/etc/docker/daemon.json) contains something like:

{
  "userns-remap": "USER"
}

Notes:

In both cases, USER must be a valid user of the system (ie present in /etc/passwd).
Don't forget to restart the daemon if you have to edit the file.

Configure the subordinate uid/gid

subuid and subgid are used to specify the user/group ids an ordinary user can use to configure id mapping in a user namespace. They are written like: username:id:count. For instance, with jenselme:100000:65536 it means that user jenselme can use 65536 user ids starting at 100000.

This will be used by docker to properly remap uid in the container to the host. For instance, with jenselme:100000:65536, a file with a uid of 33 in the container, will be a file with a uid of 100032 in the host. And you will have access to that file. Neat, isn't it?

Now that we've seen the theory, let's configure them properly. First, edit /etc/subuid and add (replace jenselme by your own user name):

jenselme:1000:1
jenselme:100000:65536

You should be able to understand the second line. The first one is there for a slightly different purpose: make sure that all files created by root belong to the user with uid 1000. That's me on my machine, you should of course use your uid (you can get it with id -u USER). Otherwise, they will belong to uid 100000.

Now, edit /etc/subgid and add (replace jenselme by your own user name):

jenselme:982:1
jenselme:100000:65536

The second line is the name in both cases. I didn't use jenselme:1000:1 but jenselme:982:1. On my machine, 982 is the gui of the docker group (you can get it with getent group docker). This means that all files created by root, will belong to me and to the docker group. This "trick" can be handy if for some reason you need to share files with the docker daemon. For instance, software like traefic may need to read/write to the docker socket. By default, for this socket we have:

[root@fastolfe ~]# ll /var/run/docker.sock
srw-rw----. 1 root docker 0 Jun 11 18:18 /var/run/docker.sock

This means that if in the outside the container the uid of root and its guid are mapped to those of jenselme, traefic won't be able to communicate with the socket because of the permissions of the file. Map the gid of root in the container to the gid of docker in the host allows us to prevent that issue.

Note on security: Giving access to the docker socket is a problem from a security standpoint since it allows a container to create new containers thus giving it access to the whole host system with root permissions, eg by running docker run -it -v --privileged -v /:/host --userns=host fedora chroot /host. That is why SELinux will prevent the docker socket to be mounted in a volume by default. You should be aware of that when you do this. See this for more on that topic.

Tests

Now that we are all set, let's start the docker daemon (or restart it).

Note to SELinux users: You need to append Z (capital z) when mounting the volumes, like this: -v $(pwd)/test:/test/:Z. Otherwise, the SELinux context will not be correct and you won't be able to access the volumes from the container. See this docker tip.

The first thing you should notice is that if you had downloaded images or created containers, you will not see them with docker images or docker ps -a. That's because, when user re-mapping is enabled, all images and containers are located in a dedicated subfolder. On my machine, that is /var/lib/docker/1000.982.

Now that we know this is expected, let's try things. Run somewhere:

docker run -it -v "$(pwd)/test:/test/" nginx /bin/bash

This will open a bash prompt as root in the container. Go to the volume with cd /test and create a file: touch rootfile. If you run a ls -l inside the container, you should see something like:

root@02a5bcc1757c:/test# ls -l
total 0
-rw-r--r--. 1 root   root 0 Jun 11 16:25 rootfile

Let's check the uid and gid to be sure:

root@02a5bcc1757c:/test# ls -ln
total 0
-rw-r--r--. 1 0 0 0 Jun 11 16:25 rootfile

So the file belongs to root and its uid is 0 as well as its gid.

Now run ls -l in the host:

▶ ls -l
total 0
-rw-r--r--. 1 jenselme docker 0 Jun 11 18:25 rootfile

Let's check the uid and guid:

▶ ls -ln
total 0
-rw-r--r--. 1   1000 982 0 Jun 11 18:25 rootfile

That's correct. Now let's do the same thing wit the www-data user. First, let's give some permissions on the /test folder to the www-data user. Since this is just a test, let's run chmod 777 /test. Now, switch to this user with su -s /bin/bash www-data. You should now be in the /test directory connected as www-data. Create a file with touch www-data-file. You should see something like:

www-data@02a5bcc1757c:/test$ ls -l
total 0
-rw-r--r--. 1 root     root     0 Jun 11 16:36 rootfile
-rw-r--r--. 1 www-data www-data 0 Jun 11 16:38 www-data-file

And:

www-data@02a5bcc1757c:/test$ ls -ln
total 0
-rw-r--r--. 1  0  0 0 Jun 11 16:36 rootfile
-rw-r--r--. 1 33 33 0 Jun 11 16:38 www-data-file

As far as the host is concerned, we have:

▶ ls -l
total 0
-rw-r--r--. 1 jenselme docker 0 Jun 11 18:36 rootfile
-rw-r--r--. 1   100032 100032 0 Jun 11 18:38 www-data-file

And

▶ ls -ln
total 0
-rw-r--r--. 1   1000    982 0 Jun 11 18:36 rootfile
-rw-r--r--. 1 100032 100032 0 Jun 11 18:38 www-data-file

Now let's create some files from the host. For instance, let's do touch www-data-file-from-host. In the host it currently belongs to the current user. Let's see in the container:

www-data@02a5bcc1757c:/test$ ls -l
total 0
-rw-r--r--. 1 root     root     0 Jun 11 16:36 rootfile
-rw-r--r--. 1 www-data www-data 0 Jun 11 16:38 www-data-file
-rw-r--r--. 1 root     nogroup  0 Jun 11 16:41 www-data-file-from-host

It belongs to root and nogroup as expected (in the host, the file belongs to jenselme:jenselme not jenselme:docker, hence the nogroup, I could run chown jenselme:docker www-data-file-from-host to fix the gid). If you check the uid and gid, you will see it is also as expected.

Now let's change the owner of www-data-file-from-host to 100032:100032 with chown 100032:100032 www-data-file-from-host (this must be run as root to prevent an Operation not permitted). I let you check the owner, uid, gid in the container. You can also check that the www-data user can write in the file with echo 'test' > www-data-file-from-host.

This looks good isn't it? I found however one dark spot in this. If you try to edit www-data-file-from-host or www-data-file in the host, it will fail with permission denied. As far as I understand the subuid and subgid, this is not normal. If someone has an explanation for this, please leave a comment. I see two workarounds for that:

The basic:
1. Create a group with id 100032 (as root): groupadd -g 100032 docker-www-data
2. Add yourself to this group (as root): usermod -aG docker-www-data jenselme
3. Disconnect/reconnect or use the newgrp docker-www-data command to take this change into account.
4. Give write permission to the group in the container: chmod g=rw www-data-file
5. Write in the file.
Note: You cannot do anything about the user since you can only have one user id.
The elegant: use ACL (Access Control List). See the external links section to learn more about ACL. TL;DR, ACLs are a way to extend the standard permissions of the filesystem. With them, you can set permissions for a file or directory with very thin granularity for each users and groups of the system. To enable ACLs, run as root:
1. setfacl -Rdm u:USER:rwX DIR (replace USER by a username and DIR by a path to a directory or file). This will:
  - -R recurse on subfolders.
  - -d default to this rule. This means that the ACL will apply to all files and directories created in DIR after the setfacl was run.
  - -m modify the rule to u:USER:rwX that is give to the user (u:) USER the permissions rwX. The capital X means give execution permission to all folders and to files that have the execute permissions. This prevent us to make all files executable.
  - apply to DIR
2. setfacl -Rm u:USER:rwX DIR (replace USER by a username and DIR by a path to a directory or file). This will apply the ACL rule on the existing files in DIR.

Bonus

Create files

If you can't or don't want to create the files (eg logs) when the images is created or when you start the container and be sure the container will be able to create them without running chmod -R 777 DIR, you can use the commands below. We assume in this example that in a log folder, we will have many volumes. Each volume will contain the logs of a container.

Create all the containers with the proper volumes.
Run chmod u=rwX,g=rwX,o=rwX -R log/ so all log files can be created.
Run find log -type d > dir-to-create to save a list of the directories you will need.
Run find log -type f > files-to-create to save a list of the files you will need.
Stop the container and destroy them.
Delete the folder: rm -rf log

Create the directories:

for dir in $(cat dir-to-create); do
    mkdir -p "${dir}"
done

Create the files:

for file in $(cat files-to-create); do
    touch "${file}"
done

Fix the permissions (run the commands as root):
1. To change the owner: find log -type f -exec chown 100032:100032 {} \;
2. To give write access to the group: find log -type f -exec chmod g=rw {} \;

Unix sockets

I guess you now see how to improve the solution described in my previous article. Since we can forsee for a given uid and gid in a container to which uid and gid it will map to in the filesystem, what we need to do is:

Change the uid and gid in the containers so they are the same. For instance usermod -u 60 uwsgi and groupmod -g 60 uwsgi. This will map to 100059 in the host.
Create a group with gid 100059 on the host.
Put the proper user in the newly created group.
Give permission on the socket to the group.

This is a bit easier than before since we only have to do the operation once per machine and we can share without any issues the image of the containers since it is the kernel that will do the mapping dynamically.

Note: you can't just rely on ACLs here, since the uid and gid of all containers that will use the socket need to be the same. You can however use ACLs to give permissions to the socket to a user of the host.

External links

That's it. If you have a question or a remark, please leave a comment below.

Docker compose tips

2017-06-11T00:00:00+02:00

For my tips about docker, go here.

Sommaire

Use docker-compose.override.yml

Use docker-compose.override.yml

As describe here, if you create a docker-compose.override.yml next to your docker-compose.yml, you can override or add values to the docker file. This file is loaded by default. To ignore it, you must use the --ignore-override option.

For instance, with this docker-compose.yml:

version: '2'

web:
  image: example/my_web_app:latest
  links:
    - db
    - cache

db:
  image: postgres:latest

cache:
  image: redis:latest

and this docker-compose.override.yml:

version: '2'

web:
  volumes:
    - '.:/code'
  ports:
    - 8883:80

db:
  ports:
    - 5432:5432

when you run docker-compose up, it will be like you used this docker-compose.yml:

version: '2'

web:
  image: example/my_web_app:latest
  volumes:
    - '.:/code'
  ports:
    - 883:80
  links:
    - db
    - cache

db:
  image: postgres:latest
  ports:
    - 5432:5432

cache:
  image: redis:latest

Create python virtual enviroments on Windows

2017-06-11T00:00:00+02:00

Before creating the venv you will need to open a PowerShell terminal as root and run the commands below to allow the script that activates the virtual env to run:
```
cd ..
Set-ExecutionPolicy Unrestricted
```
Create the venv. Run in a terminal as a normal user: python3 -m venv .venv If the creation fails due to python3 not found, check that:
- python3 was added to your your PATH during install.
- Make a copy of your python.exe executable into python3.exe
Enable the venv: .venv\Scripts\activate
Install the dependencies: pip install -r requires.txt
Install the tests dependencies: pip install -r tests_requires.txt
To install libraries that requires compiled files (like lxml) you can use the wheel files. Most libraries have such a file on pypi so you can install them directly with pip. If not, check this site to find the proper wheel file for your platform and your version of Python. Then to install it, download it and run pip install \path\to\the\weeh.whl

Use squid and squidGuard to redirect all URLs from a domain to another one

2017-06-09T00:00:00+02:00

Sommaire

Configure squid
Configure squidGuard
Activate squid

You may find yourself in a situation where you will need to redirect all URLs from a domain (lets say www.example.com) to another one (lets say www.example.org). This can be done with squid, a proxy server, and squidGuard, an extension to squid designed to do redirections. In this article I will deal only with redirections of HTTP requests made to that domain.

First install squid and squidGuard on your machine (you should be able to find packages for them in the repository of your distribution).

Configure squid

We need to tweak a bit the default configuration for it to fit our needs. The config file should be located at /etc/squid/squid.conf.

First we need to disable the forwarded_for feature by adding forwarded_for off in the config file. We also need to the name of the proxy with visible_hostname fastolfe.localdomain.

Then we need to configure the program used to rewrite the URL (you may need to change the path to the squidGuard executable and config file): url_rewrite_program /usr/bin/squidGuard -c /etc/squid/squidGuard.conf.

At last but not least, we must allow all connections that are not HTTP or that are on different ports to be passed untouched by the proxy. To achieve that, we need to add the following ACLs (add them below the other ones and before acl CONNECT method CONNECT):

acl Safe_ports port 993         # secured imap
acl Safe_ports port 587         # secured smpt
acl Safe_ports port 25          # default smtp
acl Safe_ports port 465         # gmail smtp
acl Safe_ports port 53          # DNS

We then need to change the http_access rule to allow not only HTTPS on port 443 but all the other ones. To do that, replace http_access deny CONNECT !SSL_ports by http_access deny CONNECT !Safe_ports.

Since all your HTTP request will go through the proxy, you may want to disable the log to prevent them to use lot of disk space. To do that, add to the config file:

access_log none
cache_log none
cache_log none
logfile_rotate 0

You can download my full configuration file here.

Configure squidGuard

We will first need to write the main configuration file that should be at /etc/squid/squidGuard.conf. If the file already exists, I advise you to replace its content by the code below. You will have to adapt it to mach your needs.

#
# CONFIG FILE FOR SQUIDGUARD
#

# Where to store the database of the rules.
dbhome /etc/squid
# Log dir. You may need to create the directory manually, create the /var/log/squidGuard/squidGuard.log file and give it to the squid user.
logdir /var/log/squidGuard

#
# REWRITE RULES:
#

rew example {
    # Rewrite (replace in the URL) www.example.com by www.example.org
    s@://www.example.com/@://www.example.org/@i
}

#
# SOURCE ADDRESSES:
#

src local {
    # Define what source addresses the proxy must proxy. Here we limit ourselves to local host.
    ip 127.0.0.1
}

#
# DESTINATION CLASSES:
#

dest local {
}


acl {
    default {
        # Rewrite the URL based on the example rewrite rule.
        rewrite example
        # Don't block any addresses, configure the proxy to pass everything.
        pass        local all
    }
}

To test the configuration run:

echo 'http://www.example.com/some/url - - GET' | squidGuard -c squidGuard.conf -d

You should get something like:

2017-06-10 17:07:41 [22756] New setting: dbhome: /etc/squid
2017-06-10 17:07:41 [22756] New setting: logdir: /var/log/squidGuard
2017-06-10 17:07:41 [22756] destblock local missing active content, set inactive
2017-06-10 17:07:41 [22756] squidGuard 1.4 started (1497107261.140)
2017-06-10 17:07:41 [22756] squidGuard ready for requests (1497107261.141)
2017-06-10 17:07:41 [22756] source not found
2017-06-10 17:07:41 [22756] no ACL matching source, using default
OK rewrite-url="http://www.example.org/some/url"
2017-06-10 17:07:41 [22756] squidGuard stopped (1497107261.141)

If you use another URL, you should get this response:

2017-06-10 17:07:49 [22759] New setting: dbhome: /etc/squid
2017-06-10 17:07:49 [22759] New setting: logdir: /var/log/squidGuard
2017-06-10 17:07:49 [22759] destblock local missing active content, set inactive
2017-06-10 17:07:49 [22759] squidGuard 1.4 started (1497107269.252)
2017-06-10 17:07:49 [22759] squidGuard ready for requests (1497107269.252)
2017-06-10 17:07:49 [22759] source not found
2017-06-10 17:07:49 [22759] no ACL matching source, using default
ERR
2017-06-10 17:07:49 [22759] squidGuard stopped (1497107269.252)

You can download my full configuration file here.

Activate squid

You can now enable and start the proxy: systemctl enable squid and systemctl start squid.

You now need to configure your software to use it. For instance, if you use NetworkManager, to make all requests go through it:

Open the network settings.
Go under Network proxy.
Select Manual in Method.
Fill the first field of HTTP Proxy with localhost and put 3128 in the second one.

That's it. If you have a problem or a question, please leave a comment.

Require valid-user on Apache only for some request types

2017-03-29T00:00:00+02:00

You can use the Limit directive like this:

AuthType basic
AuthName "Restricted area"
AuthUserFile /var/www/passwd
Require valid-user
# Only GET and OPTIONS request are allowed without authentication.
<Limit GET OPTIONS>
    Require all granted
</Limit>

Mon passage à LineageOS

2017-03-29T00:00:00+02:00

Tout d'abord, je présente un projet intéressant : oandbackup que j'ai découvert via cet article. oandbanck vous permet de sauvegarder les APK installés sur votre téléphone pour pouvoir les réinstaller une fois la mise à jour effectuée. Malheureusement, suite à un problème lors de la mise à jour, j'ai dû formater ma carte SD, du coup, je n'ai pu tester que la sauvegarde et pas la restauration. Mais d'après ce que j'ai lu, ça fonctionne très bien.

Concernant l'installation à proprement parler : j'avais déjà tenté une installation début février avec une nightly pour mon galaxy s2. Ça c'était mal passé : l'installation n'arrivait pas à se terminer. Après plusieurs tentatives infructueuses, les builds de cyanogenmod n'étant plus disponibles, j'ai fini par installer une vieille build de replicant pour pouvoir passer des appels et recevoir des SMS. Comme le projet lineageos a pas mal bougé (et que la build de replicant était vraiment vieille), j'ai retenté le week-end dernier une installation. J'ai eu quelques soucis mais j'ai réussi à les contourner. Voici ce que j'ai fait :

Télécharger une image récente : https://download.lineageos.org/
Télécharger l'utilitaire SU : https://download.lineageos.org/extras
Copier les fichiers sur la carte SD
Installer TWRP (le mode recovery) : c'est à partir de là que les choses ont commencés à ce gâter. J'ai tenté la commande issue de la documentation:
```
heimdall flash --KERNEL twrp-3.1.0-0-i9100.img --no-reboot
```
Malheureusement, impossible de démarrer sous TWRP : le téléphone démarrait sous l''ancien recovery. J'ai bien tenté heimdall flash --KERNEL twrp-3.1.0-0-i9100.img --no-reboot (ce que je faisais avec mes anciennes recovery) mais là, impossible de démarrer. D'après ce que j'ai compris, cela vient du fait que TWRP est plus qu'un simple mode recovery, il a besoin d'un noyau pour démarrer. La solution, c'est d'utiliser le noyau contenu dans l'image LineageOS pour pouvoir démarrer en mode recovery. Pour cela :
1. Dézipper boot.img de l'image.
2. Lancer avec le téléphone en mode téléchargement :
```
heimdall flash --KERNEL boot.img --RECOVERY twrp-3.1.0-0-i9100.img
```
Normalement, maintenant, le téléphone peut démarrer en mode recovery.
Effacer les "caches" dalvik, data, system, et cache.
Lancer l'installation du zip. Dans mon cas, l'installation se bloquait aux trois quart environ. En regardant les messages, j'ai vu : Failed to mount '/system'. J'ai testé pas mal de choses (revider les caches, changer le système de fichier avec parted, …) mais rien n'y a fait. Heureusement, j'ai fini par trouver ce thread reddit qui donne la solution : réparer le fichier PIT du téléphone. Pour cela, télécharger le PIT "qui va bien" (ou via le lien original). Ensuite, lancer en mode téléchargement (où boot.img est le noyau de LineageOS dézipper précédemment):
```
heimdall flash --repartition --pit I9100_1.5GB-System_6GB-Data_512MB-Preload_by-the.gangster.pit --KERNEL boot.img --RECOVERY twrp-3.1.0-0-i9100.img
```
À ce stade, l'installation du zip devrait fonctionner (j'ai revidé les caches avant on ne sait jamais).
Installer su
Redémarrer
Normalement tout va bien

Malheureusement, je ne sais plus à quel moment j'ai dû effacer la carte SD (je n'ai pas pris de notes pensant pouvoir écrire ce billet plus tôt). Mais de mémoire, c'était évident donc j'espère que pour vous aussi. Et si vous rencontrez un problème, vous pouvez laisser un commentaire !

Angular2 and SVG

2017-02-21T00:00:00+01:00

Disclaimer: I use the Aurelia framework, a competitor of Angular2. I wrote several articles about Aurelia. I am not an expert with Angular2. If you spot a mistake, please leave a comment.

Update 2017-03-06: Give the solution of how to use binding with SVG attributes. Thanks NexusVI.

In an interview about a year old, Rob Eisenberg, the creator of the Aurelia framework, said:

Over a year ago the Angular 2 team introduced their symbolic binding syntax. While that was technically standards compliant HTML, it was pointed out by the community that it was not compliant SVG (I have not confirmed that myself). Although members of the community pointed this out, the Angular 2 team made no changes to their design.

Since I recently completed a book about Angular2 and that I heavily use SVG in my applications, I wanted to test it myself. So I created an app with the cli (you can install it with npm install -g angular-cli@1.0.0-beta.22-1). Nothing fancy, just ng new test-svg. I then created a small SVG with Inkscape (it just contains a rectangle) and copied it into the template of the application.

I ran ng serve to build the application, opened it in a browser and BOOM, first error:

zone.js:388Unhandled Promise rejection: Template parse errors:
':sodipodi:namedview' is not a known element:

From what I know, Angular2 has its own HTML parser (mostly to parse camel cased syntax like ngIf which is not HTML compliant) and it fails if it encounters an HTML tag or attribute it doesn't know. While it's nice to help you spot typos in the name of your component, if a piece of HTML or SVG has a non standard tag or attribute, it will crash. Here it is not a big deal, my SVG is small, I can get rid of <sodipodi:namedview /> and <metadata />. But in one of my application, I need to display user uploaded SVGs. Once uploaded, I add AngularJS tags so I can display only certain elements of it depending on actions of the user. These SVGs will contain non standard tags like the Inkscape ones. How can I get rid of all of them so that Angular2 is happy? Whitelist? But what should I put in it? Blacklist? Same question. There are many SVG software out there, each may have its quirks.

This made me wonder: what if I use a brand new HTML 5.1 tag? Will it crash? I tried the code below, and it worked. I guess the parser is already 5.1 compliant. But to support 5.2 you will probably need to update Angular.

<details> // Start Of A New Accordion Element
    <summary> TITLE OF ACCORDION ELEMENT </summary>
    // Anything Here Will Be Displayed After Accordion Is Toggled To Open
    <p>
        Cras dictum. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Aenean lacinia mauris vel est.
    </p>
</details>

Now that we have a working SVG, it is time to add Angular2 markup. Let's start simple and add an ngIf attribute to show/hide the SVG based on the value of a variable:

<svg *ngIf="showSvg"

Let's add a button to toggle the value of the SVG:

<button (click)="toggleSvg()">Toggle SVG</button>

And the toggleSvg function:

toggleSvg() {
    this.showSvg = !this.showSvg;
}

And this works. The SVG is correctly displayed/hidden when I press the button.

Next step: hide the SVG when I click on the rectangle. Let's add the proper attribute:

<rect (click)="toggleSvg()" … />

If I click on the rectangle, the SVG is hidden as expected.

Next step: display multiple rectangles. Let's add a rectTops property to the component. It contains the y value for SVG rectangles. We will loop over it to display a rectangle for each value (and in a moment, we will try to bind the value to the y attribute of our rectangles).

rectTops = [30, 50, 70, 90];

<rect *ngFor="let top of rectTops"
   (click)="hideSvg()" … />

If we inspect the DOM, we see all the rectangles in it. Now, let's use the value:

<rect *ngFor="let top of rectTops"
   (click)="hideSvg()"
   [y]="top"
   … />

And it fails:

Unhandled Promise rejection: Template parse errors:
Can't bind to 'y' since it isn't a known property of ':svg:rect'. ("

You may say, no problem, let's use the mustache notation instead:

<rect *ngFor="let top of rectTops"
   (click)="hideSvg()"
   y="{{ top }}"
   … />

No luck, it crashes too with the same error. And bind-y="top"? No luck either, still the same error.

It appears that the correct way to do this is:

<rect *ngFor="let top of rectTops"
   (click)="hideSvg()"
   [attr.y]="top"
   … />

Thanks NexusVI.

Use Unix sockets with Docker

2017-02-15T00:00:00+01:00

By default, you are supposed to use TCP sockets to communicate with your applications running in Docker. But what if you want to use Unix sockets instead? The answer is you can: you make the application create the socket file in a volume and set the proper permissions to it. You can them use it for your webserver to talk to your application or for cross container communications. The tricky part is to set the proper permissions on the socket.

Let's say you are creating a Python web application running in a container with UWSGI. It will run in the container as the uwsig user. You want it to use a Unix socket to communicate with your nginx webserver running on the host under the nginx user. You need to the socket:

To belong to the uwsgi user inside the container so your application can use it.
To belong to the nginx user outside it, so nginx can use it.

The question is, how do we fullfil these two requirements? First, let's explain how file ownership works.

In Unix each file has a owner and a group. You can see them with ls -l:

$ ls -l
-rw-rw-r--. 1 jujens jujens 366 Sep 12 14:13 README.md

Here we have a file named README.md that belongs to a user named jujens and a group named jujens. What you must know is: the names are just the human readable version of these attributes. They are encoded in the filesystem as numbers respectively named uid and gid.

So what we need to do, is make sure that both the nginx user (in the host system) and the uwsgi user (in the container) have the same uid (and gid). Depending on where we do ls -l we'll get different human readable names but the permissions will be correct.

To do that, we need 3 commands:

id to get the uid and gid of a user: id -u will give you the uid of the current user and id -g the gid of the current user. You can pass an optional username to get the uid and gid for a specific user: id -u nginx.
usermod to change the uid of a user: usermod -u <NEW_ID> <USER>
groupmod to change the gid of a user: groupmod -g <NEW_ID> <GROUP>

Now we just have to use them correctly. First, let's create a setup.sh script next to your Dockerfile. It will change the uid and gid for the uwsgi user inside the container. It takes two parameters, the uid and the gui:

 1 #!/usr/bin/bash
 2 
 3 # Quit on error.
 4 set -e
 5 # Treat undefined variables as errors.
 6 set -u
 7 
 8 
 9 function main {
10     local uwsgi_uid="${1:-}"
11     local uwsgi_gid="${2:-}"
12 
13     # Change the uid
14     if [[ -n "${uwsgi_uid:-}" ]]; then
15         usermod -u "${uwsgi_uid}" uwsgi
16     fi
17     # Change the gid
18     if [[ -n "${uwsgi_gid:-}" ]]; then
19         groupmod -g "${uwsgi_gid}" uwsgi
20     fi
21 
22     # Setup permissions on the run directory where the sockets will be
23     # created, so we are sure the app will have the rights to create them.
24 
25     # Make sure the folder exists.
26     mkdir /var/run/uwsgi
27     # Set owner.
28     chown root:uwsgi /var/run/uwsgi
29     # Set permissions.
30     chmod u=rwX,g=rwX,o=--- /var/run/uwsgi
31 }
32 
33 
34 main "$@"

In the Dockerfile, we need to:

Define the variables we what to pass to the script.
Copy the script in the container.
Pass the correct variables to the script.

To do so, we need the lines below (note: this Docker file is not complete, I just present here the relevant lines for what we need to do):

# Define the variables available.
ARG uwsgi_uid
ARG uwsgi_gid

# Copy the script.
COPY setup.sh /

# Launch setup.sh
RUN bash /setup.sh "${uwsgi_uid}" "${uwsgi_gid}"

At this point, all we have to do is use this Dockerfile. I'll describe two ways to use it:

With the docker command.
With the docker-compose command. To learn more about docker-compose, see the documentation.

Directly with the docker command

First we need to build our container with the proper values in our uwsgi_uid and uwsgi_gid variables. To achieve that, we need to use the --build-arg option (note: the $(…) construct allows us to execute a command. $(…) will be replaced by the output of the command):

docker build --build-arg uwsgi_uid=$(id -u nginx) --build-arg uwsgi_gid=$(id -g nginx)

Now we can run the application by mounting /var/run/uwsgi (the folder in which the socket of our application will be) into a volume so it is accessible by the host:

docker run --volume "/var/run/myapp/:/var/run/uwsgi/:rw"

And that's it. If something is not clear, please leave a comment.

With docker-compose

First we need to create a docker-compose.yml file. We need to specify an args section inside build to define the variables and a volumes section inside myapp to define the volume:

 1 version: '2'
 2 services:
 3     myapp:
 4         build:
 5             context: .
 6             args:
 7                 uwsgi_uid: ${UWSGI_UID}
 8                 uwsgi_gid: ${UWSGI_GID}
 9         restart: always
10         volumes:
11         - ${SOCKET_DIR}/:/var/run/uwsgi/:rw

Then we need to export in our shell the UWSGI_UID, UWSGI_GID and SOCKET_DIR variables so docker-compose can find them.

export UWSGI_UID=$(id -u nginx)
export UWSGI_GID=$(id -g nginx)
export SOCKET_DIR="/var/run/myapp"

Finally, in the same shell, we can launch docker-compose:

docker-compose up -d

And that's it. If something is not clear, please leave a comment.

Extraire toutes les images encodées en base64 d'un SVG

2017-02-13T00:00:00+01:00

Récemment, j'ai eu besoin d'intégrer un SVG dans un template Aurelia. Malheureusement, il contenait beaucoup d'images et elles étaient toutes incluses au format base64. Cela rendait le fichier quasiment inutilisable avec de gros pâtés qui empêchent de voir le code utile et d'ajouter les attributs « Aurelia » (comme if.bind). Heureusement, Python (3) est là !

Le script ci-dessous prend comme paramètre le chemin vers un fichier SVG et va extraire toutes les images encodées en base64 dans le dossier courrant. Le SVG utilisant ces images extraites est sauvegardé dans le dossier courrant avec le nom svg-without-images.svg.

 1 import sys
 2 
 3 from base64 import b64decode
 4 from lxml import etree
 5 
 6 
 7 # On définit les espaces de nom XML (namespace) qui vont nous permettre de faire des requêtes dans le SVG.
 8 NS = {
 9     'svg': 'http://www.w3.org/2000/svg',
10     'xlink': 'http://www.w3.org/1999/xlink',
11 }
12 
13 
14 def print_help():
15     print('./extract-images-svg.py SVG_FILE')
16     print('This will extract the images included in b64 in the SVG.')
17 
18 
19 def extract_images_svg(file_name):
20     # On ouvre le fichier source et on le parse.
21     with open(file_name) as svg_file:
22         svg = etree.parse(svg_file)
23 
24     # On utilise xpath pour trouver toutes les images contenues dans le fichier.
25     images = svg.xpath('.//svg:image', namespaces=NS)
26     for index, img in enumerate(images):
27         # On récupère le contenu de l'image.
28         content = img.get('{http://www.w3.org/1999/xlink}href')
29         # On regarde si l'image est au format base64. Si oui, on extrait le contenu.
30         if content.startswith('data:image/'):
31             # On extrait l'image et ses méta-données (seul le format nous intéresse ici).
32             meta, img_b64 = content.split(';base64,')
33             _, img_format = meta.split('/')
34             # On extrait l'image dans le format utilisé avant l'encodage en base64.
35             img_file_name = 'img-{index}.{format}'.format(index=index, format=img_format)
36             # On remplace le lien vers l'image extraite.
37             img.set('{http://www.w3.org/1999/xlink}href', img_file_name)
38             # On sauvegarde le fichier.
39             with open(img_file_name, 'wb') as img_file:
40                 img_file.write(b64decode(img_b64))
41 
42     # On sauvegarde notre nouveau SVG.
43     with open('svg-without-images.svg', 'w') as svg_file:
44         svg_content = etree.tostring(svg)\
45                 .decode('utf-8')\
46                 .replace('&gt;', '>')
47         svg_file.write(svg_content)
48 
49 
50 if __name__ == "__main__":
51     if len(sys.argv) != 2:
52         print_help()
53         sys.exit(0)
54 
55     extract_images_svg(sys.argv[1])

Logger les données d'une requête POST avec apache

2017-02-13T00:00:00+01:00

D'après le manuel, une façon simple de logger le contenu d'une requête POST (a priori uniquement en développement pour débugger l'application, sinon la taille des logs de production va exploser et on risque de faire fuiter des informations) est d'utiliser le module dumpio, soit pour Apache 2.4 :

# On met les logs dans des fichiers à part pour faciliter la lecture.
CustomLog /var/log/httpd/website.log combined
ErrorLog /var/log/httpd/website.error.log

# On active les logs debug
LogLevel debug

# On active le module
DumpIOInput On
DumpIOOutput On
LogLevel dumpio:trace7

Malheureusement, je n'ai jamais réussi à le faire fonctionner. Heureusement, il y a un autre module qui permet de faire ça : mod_security. Il se configure comme suit :

# On active le module.
SecRuleEngine On
SecAuditEngine on
# On lui donne un fichier de log.
SecAuditLog /var/log/httpd/website-audit.log
# On l'autorise à accéder au corps des requêtes.
SecRequestBodyAccess on
SecAuditLogParts ABIFHZ

# On configure une action par défaut.
SecDefaultAction "nolog,noauditlog,allow,phase:2"

# On définit une règle qui nous permet de logger le contenu des requêtes POST
SecRule REQUEST_METHOD "^POST$" "chain,allow,phase:2,id:123"
SecRule REQUEST_URI ".*" "auditlog"

Source : https://www.technovelty.org/web/logging-post-requests-with-apache.html

Extract all base64 encoded images from a SVG

2017-02-13T00:00:00+01:00

Recently, I had to integrate a SVG in an Aurelia template. Sadly, it contained a lot of images and they were all included in base64. It made the file almost unreadable with big blobs that prevented me to see the code and add the "proper Aurelia attributes" (eg if.bind). Hopefully, Python 3 is there to help!

The script below takes as parameter the path to a SVG file and will extract all images encoded in base64 in separate files in the current folder. The SVG using these extracted images will be saved in the current folder with the name svg-without-images.svg.

 1 import sys
 2 
 3 from base64 import b64decode
 4 from lxml import etree
 5 
 6 
 7 # We define the proper XML namespaces to query the SVG.
 8 NS = {
 9     'svg': 'http://www.w3.org/2000/svg',
10     'xlink': 'http://www.w3.org/1999/xlink',
11 }
12 
13 
14 def print_help():
15     print('./extract-images-svg.py SVG_FILE')
16     print('This will extract the images included in b64 in the SVG.')
17 
18 
19 def extract_images_svg(file_name):
20     # We open the file.
21     with open(file_name) as svg_file:
22         svg = etree.parse(svg_file)
23 
24     # We find all images with xpath.
25     images = svg.xpath('.//svg:image', namespaces=NS)
26     for index, img in enumerate(images):
27         # We get the value of the image.
28         content = img.get('{http://www.w3.org/1999/xlink}href')
29         # We check it is a base64 image. If so, we extract it.
30         if content.startswith('data:image/'):
31             # We take the content of the image and its metadata
32             # (only the format of the image is relevant to us).
33             meta, img_b64 = content.split(';base64,')
34             _, img_format = meta.split('/')
35             # Replace the base64 data by a link to an external image in the proper format.
36             img_file_name = 'img-{index}.{format}'.format(index=index, format=img_format)
37             img.set('{http://www.w3.org/1999/xlink}href', img_file_name)
38             # Save the extracted image.
39             with open(img_file_name, 'wb') as img_file:
40                 img_file.write(b64decode(img_b64))
41 
42     # Save the "corrected" SVG file.
43     with open('svg-without-images.svg', 'w') as svg_file:
44         svg_content = etree.tostring(svg)\
45                 .decode('utf-8')\
46                 .replace('&gt;', '>')
47         svg_file.write(svg_content)
48 
49 
50 if __name__ == "__main__":
51     if len(sys.argv) != 2:
52         print_help()
53         sys.exit(0)
54 
55     extract_images_svg(sys.argv[1])

Log POST data with apache

2017-02-13T00:00:00+01:00

According to the manual, the easiest way to log the content of a POST request (for debugging purpose, in a production environment, it would increase the size of the logs and increase the probability to leak information) is to use the dumpio module. For Apache 2.4, all you should do is:

# Put the logs in specific files to ease reading.
CustomLog /var/log/httpd/website.log combined
ErrorLog /var/log/httpd/website.error.log

# Enable debug logging.
LogLevel debug

# Enable the module.
DumpIOInput On
DumpIOOutput On
LogLevel dumpio:trace7

Sadly, I never managed to make it work. Hopefully, there is another module which allows us to do that: mod_security. You should configure it as follows:

# Enable the module.
SecRuleEngine On
SecAuditEngine on

# Setup logging in a dedicated file.
SecAuditLog /var/log/httpd/website-audit.log
# Allow it to access requests body.
SecRequestBodyAccess on
SecAuditLogParts ABIFHZ

# Setup default action.
SecDefaultAction "nolog,noauditlog,allow,phase:2"

# Define the rule that will log the content of POST requests.
SecRule REQUEST_METHOD "^POST$" "chain,allow,phase:2,id:123"
SecRule REQUEST_URI ".*" "auditlog"

Source: https://www.technovelty.org/web/logging-post-requests-with-apache.html

Form validation with Aurelia

2017-01-24T00:00:00+01:00

Update (2017-07-20): aurelia-validation now supports events, so it is much easier to disable the submission of the form until it is valid. See the Disable the submission until the form is valid section for the code.

Update (2017-01-25): after new exchanges with Doug Kent I was able to improve my workaround to disable the submission of the form until it is valid. See the Disable the submission until the form is valid section for the code.

I promised some months ago that I will talk about form validation with Aurelia. At that time, the validation plugin was still under heavy development. Furthermore, aurelia-validation was dropped in favour of aurelia-validatejs and then aurelia-validatejs was dropped for aurelia-validation. Now things look to have settled down for aurelia-validation. That's what I'll talk about today.

From what I see in the documentation, the plugin is both powerful and expressive (many rules available, possibility to chain them, error reporting, possibility to create your own rules, possibility to hook on the render phase, …).

Sommaire

Quick overview
More advanced example
- Disable the submission until the form is valid
- Conclusion

Quick overview

To get started, you need to:

Install aurelia-validation: npm install --save aurelia-validation
Enable it in your application (in your main.js):

aurelia.use
  .standardConfiguration()
  .plugin('aurelia-validation')
  .developmentLogging();

The more direct way to use validation is to:

Import ValidationControllerFactory and ValidationRules from aurelia-validation
Inject ValidationControllerFactory in you component @inject(ValidationControllerFactory)
Create the validation controller: this.controller = controllerFactory.createForCurrentScope();
Use ValidationRules to create some validation rules. For instance to make a field named title of the current class mandatory: ValidationRules.ensure('title').required().on(this);

In a nutshell, this JavaScript code:

import { inject } from 'aurelia-framework';
import { ValidationControllerFactory, ValidationRules } from 'aurelia-validation';


@inject(ValidationControllerFactory)
export class TodoPage {
    constructor(controllerFactory) {
        this.controller = controllerFactory.createForCurrentScope();
        ValidationRules
            .ensure('title').required()
            .on(this);
    }
}

with this template:

<template>
    <input type="text" placeholder="Title" value.bind="title & validate" required>
</template>

You probably want to view the list of validation errors. To do that, you can use the validation-errors attribute like that:

<template>
    <div validation-errors.bind="errors">
        <input type="text" placeholder="Title" value.bind="title & validate" required>

        <ul if.bind="controller.errors">
            <li repeat.for="error of controller.errors">
                ${error.message}
            </li>
        </ul>
    </div>
</template>

This will display the default error message for each rule that failed. You can change this message with withMessage: ValidationRules.ensure('title').required().withMessage('Title must be provided in this form.').on(this). You can also use aurelia-i18n to translate these messages (see the documentation).

More advanced example

Now that we've covered the basics, let's get back to my mobile todo application built with Aurelia and Framework7. I have a form to create a TODO that requires a title:

My goal is to use aurelia-validation to:

Prevent the user to create a TODO until a title with at least 3 characters is supplied (just to see how we can chain rules for a property). In order to to that, I just need to use the proper validation rules (since the title is stored in a todo property of my object, I don't use .on(this) but .on(this.todo):
```
ValidationRules
      .ensure('title').required().minLength(3).withMessage('Title must at least be 3 chars long.')
      .on(this.todo);
```
Display an error message next to the title with a proper error message. This message will of course be updated as the user types. To achieve that, I need to:
1. Change how validation is trigger: by default it is on blur (that is when the user leaves the field). Let's import validateTrigger and update our controller: this.controller.validateTrigger = validateTrigger.changeOrBlur;
2. Use the validation-errors attribute on an element that only contains the input for my title. This way, the value bound to it will only contain errors for this field (by construction, validation-errors contains errors only for the fields it contains):
```
<div class="item-inner" validation-errors.bind="titleErrors">
    <div class="item-title label">
        Title
        <strong repeat.for="errorInfo of titleErrors">${errorInfo.error.message}</strong>
    </div>
    <div class="item-input">
        
        <input type="text" placeholder="Title" value.bind="todo.title & validate" required>
    </div>
</div>
```

Disable the submission until the form is valid

At this point I ran into a problem: how can I disable the submission of a TODO until it is valid? If I use validation-errors.bind="errors" on an element that includes the full form, I can use errors.length === 0 to check that it is valid. The problem is, that when the page loads, the form is invalid but since validation hasn't run yet, the submit button will be active. If I validate the form in the attached method of the component, errors is not empty (so the button is correctly disabled) but the error message for the title is displayed too soon (before the user event interacted with the form which is not what I want).

In Angular2, you can rely on the dirty property of your form controller to know if the user has started to interact with the form. So I could use something like controller.dirty && errors.length === 0 to enable/disable my button (see this article for form validation with Angular2).

~~Sadly there is no such property with aurelia-validation. There is an issue about this with a link to this blog post suggesting to hook on the rendering phase to solve this.~~

There a way to easily do this with aurelia. See Workaround 3 (how you should do it).

Workaround 1

I asked on gitter for help and Doug Kent suggested various solutions the most promising being to use two controllers: one to validate the button, one to display the error. I didn't managed to get it to work but, based on this idea, I found a workaround (and I hope we will have a clean method to do this in the future):

I keep the controller as is for validation and error display.
I use a Validator to manually validate the form without impacting error reporting. The idea is to:
1. Validate the form in attached: this form can also be used to edit an existing todo (which has a valid title), so the save button can be enabled or disabled on page load. This validation is done with the Validator which means no errors will be reported to the user.
2. Watch on the title property of my todo and validate the form each time it changes.

Here is the relevant part of the code:

import { inject, ObserverLocator } from 'aurelia-framework';
import { ValidationControllerFactory, ValidationRules, Validator, validateTrigger } from 'aurelia-validation';


@inject(ValidationControllerFactory, Validator, ObserverLocator)
export class TodoPage {
    constructor(controllerFactory, validator, ol) {
        // We rely on the controller to display the errors.
        this.controller = controllerFactory.createForCurrentScope();
        // We rely on the validator to know if the canSave the todo or not
        // (update the canSave property in the validation method)
        this.validator = validator;
        this.canSave = false;

        this.todo = {};
        ValidationRules
            .ensure('title').required().minLength(3).withMessage('Title must at least be 3 chars long.')
            .on(this.todo);
        this.controller.validateTrigger = validateTrigger.changeOrBlur;

        ol.getObserver(this.todo, 'title').subscribe(() => {
            this.validate();
        });
    }

    attached() {
        this.validate();
    }

    validate() {
        this.validator.validateObject(this.todo).then(results => {
            let valid = true;

            // results is an array of validation results. Each result has a
            // valid property set to true if the rule is valid.
            for (let result of results) {
                valid = valid && result.valid;
            }

            this.canSave = valid;
        });
    }
}

I then use canSave in my template:

<a href="#"
   class="button button-big button-fill color-green ${canSave ? '' : 'disabled'}"
   click.delegate="saveTodo()">Save</a>

You can view the full code here and here. It feels a bit hackish and I don't think this will scale to big forms.

Workaround 2 (looks more solid)

After new exchanges on gitter, Doug Kent made another suggestion: use a custom validator instead of StandardValidator. This way I should be able to hook on the validation phase and call a callback. Guess what: it worked. Let's see how.

We start by creating an new class respecting the Validator interface and taking as argument an instance of the StandardValidator which will do the actual validation work and a callback:

constructor(validator, cb) {
    this.validator = validator;
    this.cb = cb;
}

The idea is then to use the promise returned by this.validator.validateObject to call our callback each time the promise is resolved:

validateObject(object, rules) {
    return this.validator.validateObject(object, rules).then(results => {
        this.cb(results);
        return results;
    });
}

We are almost done. We also need to trigger the validation when a property of the form is validated. For instance, here, when the user modifies the title of the todo, validateObject won't be called. The controller will only call validateProperty. That's not a big problem: once the property is validated, we can validate the whole form to update TodoPage.canSave:

validateObject(object, rules) {
    return this.validator.validateObject(object, rules).then(results => {
        this.cb(results);
        return results;
    });
}

Now let's see how we can use this. We initialize the validator and the controller in the constructor:

// validator is an instance of StandardValidator that was injected by DI.
this.validator = new TodoValidator(validator, results => this.updateCanSave(results));
this.controller = controllerFactory.createForCurrentScope(this.validator);
this.controller.validateTrigger = validateTrigger.changeOrBlur;

Since this page can be used in two cases (creating a new todo and editing an existing one), we differ the creation of the validation rules a little bit: if we set them up in the constructor, they wouldn't apply on the proper object when editing a TODO. We wait for the activation and the page to distinguish between these to cases:

Creating a new TODO

this.todo = {};
this.setupValidation();

Editing a TODO

this.storage.getTodo(params.id)  // Fetch the TODO.
    .then(todo => this.todo = todo)  // Save the TODO in the current object.
    .then(() => {
        // Setup the validation rules.
        this.setupValidation();
        // Launch validation to enable the save button if the edited TODO is valid.
        this.validate();
    });

The setupValidation method is just here to create our validation rules:

ValidationRules
        .ensure('title').required().minLength(3).withMessage('Title must at least be 3 chars long.')
        .on(this.todo);

The validate method just calls this.validator.validateObject(this.todo);. As for updateCanSave it updates the canSave property based on the results of the validation:

updateCanSave(validationResults) {
    let valid = true;

    for (let result of validationResults) {
        valid = valid && result.valid;
    }

    this.canSave = valid;
}

And that's it. The main advantage of this method, is that no matter how many fields you have, you don't need extra work. It can also easily be reused for different forms. You can see the full code here (with additional comments) and here (template).

Workaround 3 (how you should do it)

As a follow up to this issue work has been done by Aurelia's contributor Jeremy Danyow so we can execute a callback each time the validator is called. This greatly simplifies the code. You can view the documentation here.

In the constructor, we create the controller with a new validator and register the callback:

constructor(validator, controllerFactory) {
    this.canSave = false;
    this.controller = controllerFactory.createForCurrentScope(validator);
    this.controller.validateTrigger = validateTrigger.changeOrBlur;
    this.controller.subscribe(event => this.validateWhole());
}

The callback passed to this.controller.subscribe will be called each time the validator is run, for reset and validate events. It has one parameter: the event. You can use it to filter by validation type or to get the errors associated with this run. Go here to view the full interface.

Note: We don't need to manually run this.validateWhole manually to initialize canSave correctly. When the form is created and validation is bootstraped, the validator will run with a reset event.

In our case, we want to validate the whole form each time a field is validated so we can update our canSave property:

private validateWhole() {
    this.validator.validateObject(this.todo)
        .then(results => this.canSave = results.every(result => result.valid));
}

And we are done. This is much simplier and shorter that all the other workaround.

Note: Running validation will not display errors messages associated to any field. So even if you validate multiple fields and display error messages for each one, validateWhole will not have an impact.

You can view the full file here and the associated template here.

Conclusion

You should now be able to validate your forms with Aurelia. If you have a question or a remark, please leave a comment.

Trouver tous les boutons sans attributs type

2017-01-16T00:00:00+01:00

Certains navigateurs (comme Chrome) afficheront une erreur dans la console si vous avez des formulaires créés avec la balise form et des boutons sans l'attribut type. Il peut donc être intéressant d'avoir un petit script qui trouve tous ces boutons pour ajouter l'attribut type. La bonne nouvelle c'est qu'avec Python et BeautifulSoup 4 c'est assez facile :

import glob

from bs4 import BeautifulSoup


def fix_buttons_type():
    for html_file_path in glob.glob('**/*.html', recursive=True):
        with open(html_file_path, 'r') as html_file:
            soup = BeautifulSoup(html_file.read(), 'html.parser')
            number_buttons_without_type = 0
            buttons = soup.find_all('button')
            for button in buttons:
                if button.get('type') is None:
                    if number_buttons_without_type == 0:
                        print(html_file_path, end=' ')
                    number_buttons_without_type += 1

            if number_buttons_without_type > 0:
                print(number_buttons_without_type)


if __name__ == '__main__':
    fix_buttons_type()

Note : Si vous utilisez Python 3.4 ou inférieur, remplacez import glob par import glob2 as glob et installez glob2.

On peut aussi légèrement modifier le script pour ajouter l'attribut avec la valeur par défaut type="button" qui signale au navigateur que c'est un simple bouton qui ne doit rien faire vis à vis du formulaire (soumission ou remise à zéro). Le problème étant que cela peut casser le formatage du votre fichier.

import glob

from bs4 import BeautifulSoup


def fix_buttons_type():
    for html_file_path in glob.glob('**/*.html', recursive=True):
        with open(html_file_path, 'r') as html_file:
            soup = BeautifulSoup(html_file.read(), 'html.parser')
            number_buttons_without_type = 0
            buttons = soup.find_all('button')
            for button in buttons:
                if button.get('type') is None:
                    button['type'] = 'button'

        with open(html_file_path, 'w') as html_file:
            html_file.write(soup.prettify())


if __name__ == '__main__':
    fix_buttons_type()

Find all buttons without a type attribute

2017-01-16T00:00:00+01:00

Some browsers (like Chrome) will display an error in the console if you have forms created with the form tag and buttons without the type attribute. So it can be useful to have a script that find all these buttons so you can add them the type attribute. The good news is that with Python and BeautifulSoup 4 it is really easy:

import glob

from bs4 import BeautifulSoup


def fix_buttons_type():
    for html_file_path in glob.glob('**/*.html', recursive=True):
        with open(html_file_path, 'r') as html_file:
            soup = BeautifulSoup(html_file.read(), 'html.parser')
            number_buttons_without_type = 0
            buttons = soup.find_all('button')
            for button in buttons:
                if button.get('type') is None:
                    if number_buttons_without_type == 0:
                        print(html_file_path, end=' ')
                    number_buttons_without_type += 1

            if number_buttons_without_type > 0:
                print(number_buttons_without_type)


if __name__ == '__main__':
    fix_buttons_type()

Note: If you are using Python 3.4 or below, replace import glob by import glob2 as glob and install glob2.

It is possible to slightly modify the script to add the type attribute with a default value like this type="button". This signals the browser that the buttons is not related to the form (submission or reset). The problem is: this may break the formating of you file.

import glob

from bs4 import BeautifulSoup


def fix_buttons_type():
    for html_file_path in glob.glob('**/*.html', recursive=True):
        with open(html_file_path, 'r') as html_file:
            soup = BeautifulSoup(html_file.read(), 'html.parser')
            number_buttons_without_type = 0
            buttons = soup.find_all('button')
            for button in buttons:
                if button.get('type') is None:
                    button['type'] = 'button'

        with open(html_file_path, 'w') as html_file:
            html_file.write(soup.prettify())


if __name__ == '__main__':
    fix_buttons_type()

Make navigation user friendly on your Aurelia SPA

2016-08-28T00:00:00+02:00

Sommaire

Add a spinner
Use the wait cursor while navigating
Display a message to no-script user
More

Add a spinner

Add the font awesome files to your project (CSS and fonts files).

Import the font awesome style sheet in your index.html

<link rel="stylesheet" href="/assets/fonts/fontawesome/font-awesome.min.css">

Add the CSS code below to create the spinner either directly in the index.html with style tag or in a CSS file that you load in it:

.splash {
    text-align: center;
    margin: 10% 0 0 0;
    box-sizing: border-box;
}

.splash .message {
    font-size: 72px;
    line-height: 72px;
    text-shadow: rgba(0, 0, 0, 0.5) 0 0 15px;
    text-transform: uppercase;
    font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
}

.splash .fa-spinner {
    text-align: center;
    display: inline-block;
    font-size: 72px;
    margin-top: 50px;
}

Update the div in which your Aurelia application will be instantiated. Eg, replace:

<body aurelia-app="main">
  <div>
  </div>
</body>

By:

<body aurelia-app="main">
  <div class="splash">
      <div class="message">Arena of Titans</div>
      <i class="fa fa-spinner fa-spin"></i>
   </div>
</body>

Use the wait cursor while navigating

Change your router-view from:
```
<router-view></router-view>
```
to:
```
<router-view class="${router.isNavigating ? 'navigating' : ''}"></router-view>
```
this will add a class navigating to the router-view tag when the router is navigating and nothing otherwise.
Add this global CSS code to your application:
```
.navigating {
    cursor: wait !important;
    opacity: 0.7;
}

.navigating button {
    cursor: wait !important;
    opacity: 0.5;
}
```
This will display the wait cursor on the page and reduce the opacity on the page to signal the user a navigation is going on. You can of course change this code to match your application.
Don't forget to add the router to your object. In the configureRouter(config, router) function, do:
```
this.router = router;
```

Display a message to no-script user

Some of your users may use an extension like noscript which disable JavaScript on page you visit. By default, you will greet them with either a blank page or a never-ending spinner. I think it is more user friendly to show them a message asking to enable JavaScript. To do, add in your body tag:

 <noscript>
     <p>Arena of Titans cannot work without JavaScript. Please enable it.</p>
     <p>Arena of Titans ne peut pas fonctionner sans JavaScript. Merci de l'activer.</p>
</noscript>

To view the full details, look at these two commits:

Switching an application to aurelia-cli

2016-08-17T00:00:00+02:00

About a month ago, I started to make experiments with the webpack plugin for Aurelia in order to split my applications into multiple bundles. The application in question is a strategy game called Arena of Titans. You can see it there (click play to create a game, or use this link).

It was working well but the initial load of any page was slow. It was easy to find out why: the bundle loaded by webpack was over 1.3 megabytes big. Why is it that big? Well, it contained the game board which is about 1.1 megabytes minified. Hence my will to split it into several bundles in order to speed up the loading of the site. We still have to load the big board on the game page but at least the site would work at an acceptable speed.

I didn't manage to do it with webpack and after some time trying I thought what about the new aurelia-cli? I was able to do it just fine. Here's how I did it. You can skip to the part about bundles if you're not interested in the rest.

Sommaire

Presentation of aurelia-cli
Preparation
- Updates in main.js
Style
Bundles
- Problems encountered with bundles and their solutions
Tests
Conclusion

Presentation of aurelia-cli

aurelia-cli is a project aimed to provide a command line interface to create and run projects written with the Aurelia JavaScript framework. You can:

Create a new project with au new <project-name>. You will then be guided by an assistant to configure the project: standard ES6 or typescript, pure CSS or SCSS/Less, …
Run the project with au run or au run --watch to rebuild the app when you save a modification.
Test the project with au test or au test --watch to retest the app when you save a modification.

It relies on gulp to launch its tasks and on Browsersync to provide a small HTTP server that will reload you page when you do a modification.

Preparation

To avoid messing with the current code, I created a new aurelia-cli project in a separate folder and I copied the proper files in my project:

The folder aurelia_project which contains the gulp tasks required by aurelia-cli to run and the configuration of the project.
The scripts folder which contains requirejs the module loader used by project running with aurelia-cli and its text plugin to load the HTML and CSS.
test/aurelia-karma.js a small file that allows you to use karma with aurelia-cli projects. See the section about tests to learn more on that.
karma.conf.js so my karma configuration file doesn't rely on webpack any more and meets the expectations of aurelia-cli.

I then updated my package.json file. To do that, I just compared my current package.json with the one from the aurelia-cli project: I added the proper dependencies (like aurelia-cli) and removed the old ones (like webpack).

To avoid style warning with on the environement.js file, I added it to the .eslintignore. This files contains utilities values to configure aurelia (are we testing? are we in debug mode?):

export default {
    debug: true,
    testing: false
};

I then removed all of webpack configuration files and run rm -rf node_modules && npm install to update the dependencies.

Then, I adapted aurelia.json to my needs:

The sources of my project are in a folder named app and not in src as it is by default. So I changed all paths (a replace src by app did it). I also had to correct aurelia-karma.js. See the section about tests to learn more about that.
I configured the router to use push states. Which means my URLs look like this: /game/create and not like this #/game/create. The problem is that, by default, requirejs will load its bundle under /game/scripts/bundle.js instead of /scripts/bundle.js. So I did my first patch in aurelia-cli to add an option to use absolute path in requirejs. This way if you use "useAbsolutePath": true in the build.targets section of your aurelia.json file, your bundles will always be loaded in /scripts/bundle.js.

I also updated my global.scss file to load fonts with an absolute URL and aurelia_project/tasks/run.js to change the port of Browsersync and disable the ghost mode which mimics on all browsers the action you do in one: I want to be able to test the game in different browsers running different players. I don't want it to reproduce what I do in every browsers I have opened.

Updates in main.js

I reverted Aurelia's configuration function to the classical export function configure(aurelia) and I added the two lines below to enable development logging only on debug mode and to enable the testing plugin when testing:

if (environment.debug) {
    aurelia.use.developmentLogging();
}

if (environment.testing) {
    aurelia.use.plugin('aurelia-testing');
}

I also updated my configuration of bluebird.js a promise library for javascript in main.js from:

  let Promise = require('bluebird').config({
    longStackTraces: false,
    warnings: false,
});

to:

Promise.config({
    longStackTraces: false,
    warnings: false,
});

Style

With webpack, you include the SCSS files like javascript but with the extension, like this: import ./style/global.scss. Webpack will then compile the file and include it in the bundle thanks the loader you've configured. With aurelia-cli, things are a little different. You include the file in your HTML files like you require custom elements:

<require from="./style/global.css"></require>

Note that the extension of the file must be .css whether you're using true CSS or another language compiled to CSS like SCSS. The reason is that it will be loaded as CSS. You just have to configure the cssPreprosessor in your aurelia.json for the code to be correctly processed:

"cssProcessor": {
   "id": "sass",
   "displayName": "Sass",
   "fileExtension": ".scss",
   "source": "app/**/*.scss"
}

Bundles

The ability to split my app into multiple bundles is the main reason I switched to aurelia-cli. To create a bundle, all you have to do is to add an object with a name and a list of dependencies to the build.bundles array of your aurelia.json file.

It works great but the first time I tried, my bundles were not filled correctly. I tried to include files like this: "[site/**/*.js]" but it didn't work. I did some search and I found this comment by TylerJPresley on a issue about creating multiple bundles with aurelia-cli which gave me the solution: you have to put stars in front of the file name. Like this: "[**/site/**/*.js]":

{
    "name": "site-bundle.js",
    "source": [
        "[**/site/**/*.js]",
        "**/site/**/*.{css,html}"
    ]
}

With that, my bundles were correctly created and filled but they were not loaded. However, it turns out that I was missing some files in them: I did put the site and game files but not app.js, environment.js and main.js. As you might expect, without these files Aurelia could not work correctly. So I create a common-bundle.js for them and other files needed in the whole application:

{
    "name": "common-bundle.js",
    "source": [
        "[**/locale/**/*.js]",
        "[**/app.js]",
        "[**/environment.js]",
        "[**/main.js]",
        "[**/services/options.js]",
        "[**/services/storage.js]",
        "[**/services/browser-sniffer.js]",
        "[**/widgets/aot-options/*.js]",
        "**/app.html",
        "**/widgets/aot-options/*.html",
        "**/widgets/aot-options/*.css",
        "**/style/*.css"
    ]
}

In addition to bundling your app files, you can also include non Aurelia dependencies from outside your project if they rely on AMD. For instance, here's how I define my game-create-bundle.js with clipboardjs:

{
   "name": "game-create-bundle.js",
   "dependencies": [
       {
           "name": "clipboard",
           "path": "../node_modules/clipboard/dist",
           "main": "clipboard"
       }
   ],
   "source": [
       "[**/game/create/**/*.js]",
       "**/game/create/**/*.{css,html}"
   ]
}

You can also prepend javascript files to a bundle. That's how requirejs and bluebird are loaded. Here's the relevant excerpt from the vendor-bundle.js (defined here):

"prepend": [
     "node_modules/bluebird/js/browser/bluebird.js",
     "scripts/require.js"
]

At last but not least, some Aurelia modules contains multiple files that must be resolved. To bundle those correctly, you need to use an object instead of the name of the module in you dependencies array. Like this:

"dependencies": [
    "aurelia-binding",
    {
        "name": "i18next",
        "path": "../node_modules/i18next/dist/commonjs",
        "main": "index"
    }
]

You can view the full definition of my vendor-bundle.js where this example is taken here and the definition of all my bundles here.

All in all, I think that bundling with aurelia-cli is very powerful, works well and, once you know for the double stars, is quite easy to setup with the help of a new project to give you the base configuration.

In the end, I have 6 bundles for the application:

vendor-bundle.js with requirejs, bluebird.js a promise library and all Aurelia related files.
common-bundle.js that contains the translations, app.js, main.js, environement.js and various services and widgets common to the site and the game.
site-bundle.js that contains all pages and widgets for the sites.
game-common-bundle.js that contains files required to create and play the game.
game-create-bundle.js that contains all the widgets to create the game.
game-play-bundle.js that contains all you need to actually play the game.

Problems encountered with bundles and their solutions

aurelia-cli loads its bundles with requirejs which I find great. Until during a test session a friend reported that he was not redirected to the game page when he first tried to create a game. The second time he tried everything worked fine.

I dug into the problem and found out why it was failing. The bundle containing the game is still big (about 1.3 megabytes). On some slow connection, it took more than 7 seconds to load it. Which means that we encountered the default timeout for requirejs and the app would consider the bundle could not be loaded and thus didn't redirect the player to the page even after the script was loaded. requirejs nicely logged an error with this link in the console.

The solution is to use the waitSeconds option of requirejs to increase the value of the timeout (you can also disable it as the documentation says). I did some tests directly in the generated bundle and it work.

The problem is that at the time of this writing, aurelia-cli doesn't allow you to give custom options to the loader. So, I made a pull request to allow the user to do just that in order to correctly solve my problem while still using the cli. I hope it will be merged soon.

In order to further improve load time on these slow connection, I preload the big game-play-bundle.js with the board as soon as a user reaches the create game page. To do that, I added the line below in the constructor of the create game page:

require(['game/play/widgets/board/board'], () => {});

You must use this syntax to load the script asynchronously. If you use:

require('game/play/widgets/board/board')

the script will be loaded synchronously which means the user can't prepare the game until it is completely loaded. That is of course not my goal.

Tests

First, since the code is not in the src folder but in app, I had to correct aurelia-karma.js (see Preparation for where it comes from) the file that boostrap karma for usage with aurelia-cli. I had to correct this line (52 at the time of this writing):

originalDefine('/base/src/' + name, [name], function (result) { return result; });

into:

originalDefine('/base/app/' + name, [name], function (result) { return result; });

I think it would be better for aurelia-karma to use paths.root for aurelia.json (more general). I gave it a try but didn't succeed. But there may be other ways to do this. See my issue on the subject for more details paths.root from aurelia.json.

In my test files, I had to change some import paths, remove my import ../setup since it is automatically loaded by aurelia-karma. The test/unit/setup.js has the same content as before to import ployfills and initialize Aurelia:

import 'aurelia-polyfills';
import { initialize } from 'aurelia-pal-browser';

initialize();

I also had to move my test-utils.js which contains various stubs I use in my unit tests from the test/unit folder into the app folder so it is correctly transpiled by babel. I also created a dedicated bundle named test-utils-bundle.js to avoid loading it in the true application.

Conclusion

This is it! Converting the project wasn't very hard but some problems (absolute loading of bundles, some problems with testing and the inability to configure requirejs) required some time to make it work correctly. Now I think that the project is ready for the future of Aurelia tooling and I don't think I'll need to switch tools again.

If you want all the gory details, take a look at this merge commit and the commits before it in the aurelia-cli branch.

Otherwise, you also take a look at my aurelia.json file here, browse the complete code of the app there and see it in action on the website (click play to create a game, or use this link).

If you have a remark or question about this article or the game, leave a comment below or contact me on twitter.

Next step on my agenda, setup code coverage of the original sources of the app (and not the bundle, that would be too easy). I hope I'll be able to make it work and write about it soon. Stay tuned!

Utiliser des métaclasses pour créer simplement des enums en Python 3

2016-08-01T00:00:00+02:00

Depuis la version 3.4, Python dispose d'une classe Enum qui permet de créer des enums avec quelques propriétés intéressantes (itération, nombre d'éléments, accès aux éléments de l'enum comme ceux d'un objet ou d'un dictionnaire). Je vous laisse lire la documentation pour les détails.

Cependant, dans mon cas, je les utilise pour traquer certaines valeurs d'une requête faites sur une application (par exemple : le nom de la requête envoyée au serveur ou l'état d'un créneau – ouvert, fermé, pris). Les enum de base me posent plusieurs problèmes :

Il faut entrer le nom et la valeur pour chaque membre, comme ceci :

class SlotState(Enum):
    OPEN = 'OPEN'
    CLOSED = 'CLOSED'
    RESERVED = 'RESERVED'
    TAKEN = 'TAKEN'

J'oublie le .value à chaque fois lorsque je fais un test entre la requête et l'enum: je fais SlotState.OPEN == request['slot']['state'] au lieu de SlotState.OPEN.value == request['slot']['state'].

Du coup, je me suis demandé si on ne pouvais pas faire plus simple pour mon utilisation. Ce que j'aimerais c'est :

Utiliser le nom de l'attribut pour remplir automatiquement sa valeur.
Tester directement avec l'attribut MonEnum.membre == 'VALEUR'.
Ne pas avoir à mettre manuellement en majuscule les propriétés, ie pouvoir accéder aux éléments avec MonEnum['Valeur'] ou avec MonEnum['VALEUR'].
Garder les propriétés sympathiques des « vrais » enums Python :
- Longueur.
- Boucler sur tous les éléments de l'enum.
- Accès aux membres avec la notation crochet et comme aux attributs d'une classe.
- Tester si un élément existe dans l'enum avec le mot clé in.

C'est possible en utilisant les métaclasses et en s'inspirant un peu de EnumMeta la métaclasse de la classe Enum.

Qu'est-ce qu'une métaclasse ? Pour faire simple, vous savez certainement qu'en Python tout est objet et les classes ne font pas exception : ce sont des instances d'une métaclasse. Cela nous permet donc de faire des opérations sur la classe et non sur l'objet. Par exemple, si vous déclarer une méthode __getitem__ dans une classe, vous pouvez utiliser sur l'instance comme ceci instance['toto'] mais pas directement sur la classe comme cela MaClasse['toto'].

C'est ce que permet de faire une métaclasse : vous déclarer la méthode __getitem__ dans la métaclasse puis vous pouvez utiliser MaClasse['toto'].

Voici ma métaclasse SimpleEnumMeta (voir les explications ci-dessous) :

 1 class SimpleEnumMeta(type):
 2     def __new__(metacls, cls, bases, classdict):
 3          object_attrs = set(dir(type(cls, (object,), {})))
 4          simple_enum_cls = super().__new__(metacls, cls, bases, classdict)
 5          simple_enum_cls._member_names_ = set(classdict.keys()) - object_attrs
 6          non_members = set()
 7          for attr in simple_enum_cls._member_names_:
 8              if attr.startswith('_') and attr.endswith('_'):
 9                  non_members.add(attr)
10              else:
11                  setattr(simple_enum_cls, attr, attr)
12 
13          simple_enum_cls._member_names_.difference_update(non_members)
14 
15          return simple_enum_cls
16 
17     def __getitem__(cls, key):
18        return getattr(cls, key.upper())
19 
20     def __iter__(cls):
21        return (name for name in cls._member_names_)
22 
23     def __len__(cls):
24        return len(cls._member_names_)

Concernant le fonctionnement de cette métaclasse :

Le constructeur __new__ (à ne pas confondre avec la méthode d'initialisation __init__) nous permet d'interagir avec la classe avant sa "création" ce qui nous permet d'affecter des valeurs à nos attributs de classe. Pour cela :
1. On récupère les attributs communs à tous les objets pour pouvoir les exclure de la liste des attributs dont on va affecter la valeur automatiquement (ligne 3).
2. On crée la classe grâce au constructeur de type, la classe mère de toutes les métaclasses (ligne 4).
3. On stocke dans une propriété privée l'ensemble des attributs spécifiques à notre enum (ligne 5). Cela nous permet de donner le nombre d'élément contenu dans l'enum ainsi que de fournir un itérateur sur ces éléments (méthodes __len__ et __iter__).
4. Pour chaque attribut de l'enum, on affecte sa valeur comme le nom de l'attribut (lignes 7 à 11).
5. On supprime de la liste les attributs les éventuels attributs privés.
6. Enfin, on renvoie la classe nouvellement créée (ligne 15).
Je pense que les autres méthodes parles d'elles mêmes

Pour l'utiliser, il suffit de créer une classe avec un argument metaclass=SimpleEnumMeta :

class SlotState(metaclass=SimpleEnumMeta):
    OPEN = ()
    CLOSED = ()
    RESERVED = ()
    TAKEN = ()
    AI = ()

Vous devez assigner une valeur aux attributs puisque :

class MyClass:
   test

n'est pas du Python valide et provoque une exception de type NameError. L'utilisation d'un tuple vide me semble un bon compromis pour avoir du code valide et signaler que la valeur de l'attribut n'a pas d'importance.

On peut ensuite vérifier que l'on a bien les propriétés attendues :

SlotState.OPEN == 'OPEN'
SlotState['open'] == 'OPEN'
len(SlotState) == 5
'OPEN' in SlotState
for state in SlotState: print(state)

Et voilà ! Si vous avez une question ou une remarque, n'hésiter pas à utiliser les commentaires.

Use metaclasses to create simple enums in Python 3

2016-08-01T00:00:00+02:00

Since version 3.4, Python has an Enum class. It allows you to create enums with some neat properties:

Iteration.
Length.
Access to the members with the "attribute" or "dict" notation.

I leave you to the documentation for the details.

In my case, I am using them to track some values of requests send to my application (eg: the name of the request or the state of a slot – opened, closed, taken). In this use case, I have some problem with the "standard" enums as provided by the enum package:

I have to enter the name and value for each member which is kind of sad since they are the same:

class SlotState(Enum):
    OPEN = 'OPEN'
    CLOSED = 'CLOSED'
    RESERVED = 'RESERVED'
    TAKEN = 'TAKEN'

I always forget the .value when I am testing: I do SlotState.OPEN == request['slot']['state'] instead of SlotState.OPEN.value == request['slot']['state'].

So I asked myself: can I create my own enums that would solve these problems? I want them to:

Use the name of the attribute to fill its value.
Test directly with the attribute: MonEnum.membre == 'VALEUR'.
Not to require to use caps in all access, ie access with either MonEnum['Valeur'] or MonEnum['VALEUR']
Keep the neat properties of "true" Python enums:
- Length
- Loop over the elements
- Access to the elements with either the "attribute" or the dict notation.
- Test if an element is part of an enum with the in keyword.

The answer is, it is possible with metaclasses and by getting some inspiration from EnumMeta the metaclass of the Enum class.

What is a metaclass? You probably know that in Python everything is an object. Well, classes are no exceptions: they are instances of a metaclass. This allows you to make operations on a class instead of an object. For instance, if you declare a __getitem__ method on a class, you must use it on an instance of the class like this instance['toto'] but you cannot use it directly on the class like this: MyClass['toto'].

However, you can do this with a metaclass: if you declare __getitem__ in the metaclass of MyClass you can use MyClass['toto'].

Here's my SimpleEnumMeta metaclass (explanations below):

 1 class SimpleEnumMeta(type):
 2     def __new__(metacls, cls, bases, classdict):
 3          object_attrs = set(dir(type(cls, (object,), {})))
 4          simple_enum_cls = super().__new__(metacls, cls, bases, classdict)
 5          simple_enum_cls._member_names_ = set(classdict.keys()) - object_attrs
 6          non_members = set()
 7          for attr in simple_enum_cls._member_names_:
 8              if attr.startswith('_') and attr.endswith('_'):
 9                  non_members.add(attr)
10              else:
11                  setattr(simple_enum_cls, attr, attr)
12 
13          simple_enum_cls._member_names_.difference_update(non_members)
14 
15          return simple_enum_cls
16 
17     def __getitem__(cls, key):
18        return getattr(cls, key.upper())
19 
20     def __iter__(cls):
21        return (name for name in cls._member_names_)
22 
23     def __len__(cls):
24        return len(cls._member_names_)

How does it work?

The constructor __new__ (not to be mistaken with the initialization function __init__) allows us to interact with the class before its "creation". That allows us to affect values to the attribute of our class. To do that:
1. We get the set of attributes common to all objects in order to exclude them from the set of attributes whose value we want to set automatically (line 3).
2. We create the class thanks to the constructor of type the super class of all metaclasses (line 4).
3. We store in a private property the set of attributes specific to our enum (line 5). This allows us to give the correct number of elements of you enum and to create an iterator to loop over them (methods __len__ and __iter__).
4. For each attribute in the enum, we affect its value as the name of the attribute (line 7 to 11).
5. We then remove the private attributes from the list of the list of attributes.
6. In the end, we return the newly created class (ligne 15).
I don't think I need to explain the other methods.

To use it, you just need to create a class with the metaclass=SimpleEnumMeta parameter:

class SlotState(metaclass=SimpleEnumMeta):
    OPEN = ()
    CLOSED = ()
    RESERVED = ()
    TAKEN = ()
    AI = ()

You must assign a value to the attributes since:

class MyClass:
   test

is not valid Python code and results in a NameError exception. Using an empty tuple looks like a good way to write valid code while signaling you don't need to set the properties.

You can then check that your class behave as intended:

SlotState.OPEN == 'OPEN'
SlotState['open'] == 'OPEN'
len(SlotState) == 5
'OPEN' in SlotState
for state in SlotState: print(state)

Et voilà ! If you have a question, please leave a comment.

Plugin for statistics with Piwik on an Aurelia SPA

2016-06-05T00:00:00+02:00

On Arena of Titans a game based on the Aurelia web framework I am developing with a friend, we use Piwik to have statistics about our visitors. When we were about to launch the game, I realized that Piwik was not logging all the pages but only the pages on which you entered. It makes perfect sense: navigation on the site is handled by Aurelia which means there is no page load and if there is no page load, Piwik doesn't load the navigation.

To go around this problem, I created a small plugin for Aurelia aurelia-piwik. What it does is straightforward: it register a callback for the router:navigation:success event which is fired by Aurelia's router on successful navigation. In this callback, I use Piwik's API to log the navigation.

You can easily use this plugin once installed by registering it in your main.js:

aurelia.use
 .standardConfiguration()
 .developmentLogging()
 .plugin('aurelia-piwik')

I must say I am impressed by how easy it was to implement and use. After nearly seven months of Aurelia usage, I am really glad to use this framework. If you haven't checked it out yet, please do!

The source are available under the MIT license on GitHub. The package is also available on npm.

Resolving blank page problem of Aurelia app with webpack on Safari

2016-06-03T00:00:00+02:00

While asking users to test an Aurelia application, it appeared that it didn't work on Safari due to this error:

It comes from the fact that Safari doesn't support the internationalization (window.Intl) API.

In order to solve it, I had to manually include the Intl.js polyfill. Since it is already pulled as a dependency, I just had to adapt the bootstrap function of Aurelia with the setup code for Webpack of Intl.js:

bootstrap(aurelia => {
     if (!global.Intl) {
         console.log('Intl not present')
         require.ensure([
             'intl',
             'intl/locale-data/jsonp/en.js'
         ], function (require) {
             require('intl');
             require('intl/locale-data/jsonp/en.js');
             boot(aurelia);
         });
     } else {
         boot(aurelia);
     }
 });


 function boot(aurelia) {
     aurelia.use
         .standardConfiguration()
         .developmentLogging()
         .plugin('aurelia-i18n', (instance) => {
             let language = navigator.language.split('-')[0];

             // register backend plugin
             instance.i18next.use(XHR);

             function loadLocales(url, options, callback, data) {
                 switch (url) {
                     case 'en':
                         callback(enTranslations, { status: '200' });
                         break;
                     case 'fr':
                         callback(frTranslations, { status: '200' });
                         break;
                     default:
                         callback(null, { status: '404' });
                         break;
                 }
             }

             instance.setup({
                 backend: {
                     loadPath: '{{lng}}',
                     parse: (data) => data,
                     ajax: loadLocales,
                 },
                 lng: language,
                 attributes: ['t', 'i18n'],
                 fallbackLng: 'en',
                 debug: false,
             });
         });

     aurelia.start().then(() => aurelia.setRoot('app', document.body));
 }

How I switched an Aurelia application to webpack

2016-04-24T00:00:00+02:00

Note: I wrote this post when aurelia-webpack wast still in pre 1.0.0 release. As niieani pointed out in the comments part of it may be outdated and you way want to use aurelia-webpack with easy-webpack instead. The general ideas are still valid though.

First thing first: why would I want to use webpack with Aurelia instead of the default JSPM?

I was tired of JSPM which I find buggy
No need for gulp anymore
All deps are in the package.json file and node_modules folder
webpack can build all kind of files (JS, TS, CSS, JSON, …) and perform optimization on them
Awesome dev server with hot reload.

If you are interested in the subject, you can read this article from Dwayne Charrington which gives more details on why you may want to use webpack.

Here are the steps I followed. I did them nearly a month back but haven't found the time to write this article before today, so it is based on my small notes and memories. I apologize if this article is not as precise as it could be. If you have a specific question or run into trouble, please leave a comment. Anyway, the thing to remember is that the migration went well and webpack works very well with Aurelia.

Update index.html. Copy/Pasting the index.html from the skeleton with little adaptation should do the trick. You will also need a index.prod.html file for use in production. Again, the skeleton has what you need.
Update dependencies in package.json. Adapt the one from the skeleton. You may need to add dependencies beside webpack specific ones (specified in the skeleton) to get your app working. For instance, aurelia-logging-console was not explicitly in my package.json (not even under the jspm key). You can also get rid of the jspm key. In my case, Aurelia will give you a nice debug message if you are lacking a required dependency.
Update configuration of Karma. See karma.conf in the skeleton. For your tests to work, you are very likely to need to bootstrap your tests by adding to each file (you can also put that in a setup.js file and import it in your test files):

import { initialize } from 'aurelia-pal-browser';

initialize();

Update main.js (create it if you don't have one). See the skeleton for inspiration. Important note: if you want you style files to be loaded with webpack, they must be imported in a JS file. main.js can be a good place to place these import.
Copy webpack.config.js and webpack.prod.config.js and adapt them if need be.
Remove JSPM files: rm -rf jspm_packages config.js
Remove unneeded node modules (eg jspm) rm -rf node_modules
Remove gulp related files: rm -rf gulpfile.js build
Install new dependencies: npm install
Take care of submodules Not needed with aurelia-webpack 1.0.0: if you see an error like this:

.*$:106 Uncaught (in promise) Error: Cannot find module './aurelia-i18n/t'.
.*$:106 Uncaught (in promise) Error: Cannot find module './aurelia-i18n/nf'.
.*$:106 Uncaught (in promise) Error: Cannot find module './aurelia-i18n/df'.
Uncaught (in promise) Error: Cannot find module './aurelia-i18n/rt'.

You need to enable sub-modules for this modules, with something like this:

new AureliaWebpackPlugin({
    includeSubModules: [
        { moduleId: "aurelia-i18n" }
    ]
})

See the documentation of the webpack plugin for more information on that.

Normally you are ready to launch ./node_modules/.bin/webpack-dev-server --config webpack.config.js --hot --inline --progress --devtool eval and start developing with Aurelia and webpack.

Small comparison of ionic2 and Aurelia + Framework7 for hybrid mobile applications

2016-03-15T00:00:00+01:00

Update (2017-08-14): I finally published the part on Aurelia UX. See it here.

Updates (2017-01-24):

aurelia-interface has been deprecated in favour of aurelia-ux.
Add a link to my article about aurelia-validation.
I don't think I'll start a project to use Aurelia with Framework7. I want to test aurelia-ux first. However, someone is starting one: https://github.com/alflennik/au7

Sommaire

Application features
Ionic2
Aurelia and Framework7
- Bonus
Conclusion

I spent some time recently to develop a mobile application with the ionic framework. This framework allows you to create mobile applications for Android and iOS with HTML5 and CSS3 and AngularJS. Which means you can develop native like applications with the ease of development of a web application: debug in a browser, use Javascript and a framework you probably already know and that's it! No java or Swift, just the power of the web. The application is then build for your phone with Apache Cordova.

Since they have a version 2 coming soon based on Angular2, I decided to use directly this version (currently in beta).

However, on a another project, I use Aurelia a JavaScript framework built with ES6 and around web standards aimed to create Single Page Applications easily. And I must say I love it. It embraces ES6 and the new possibilities offered by the language and its use of convention over configuration makes it easy to work with.

So I wanted to try to build a mobile application with it. There are plans for a commercial addon called Aurelia Interface. Sadly there is not much information about it available apart from that video. After a small discussion on Aurelia's gitter, Scalpal suggested to look at Framework7 an HTML/CSS (with small bits of JavaScript) framework to do pretty much the same thing as ionic without the dependency on a particular framework. That's what I did and it turned out to work well with Aurelia.

Let's test!

Application features

To compare the two possibilities, I wanted to do a basic todo list applications that can:

List all the saved todos (home page)
Edit an existing todo
Add a todo
Mark a todo as done on the home page. Done todos must have their title strike-through on the home page.

All todos must be stored in the permanent SqlStorage of the phone.

In additions to these features, both applications must work on a browser (for debug and development purposes), on an android emulator and on an android phone.

Here is how the home page looks like (Aurelia + Framework7 on the left, Ionic2 + Angular2 on the right):

And the todo page (Aurelia + Framework7 on the left, Ionic2 + Angular2 on the right):

The complete code of the applications is available on github under the MIT license. Normally, the READMEs are complete enough to let you try the applications. If you encounter any problem, leave a comment or open an issue!

Ionic2

I wrote this version in TypeScript a language made by Microsoft to ease the development of huge frontend applications and not ES6 since it is the recommended way for Angular2 applications (Angular2 is itself written in TypeScript).

To organize the application, I followed the standard structure proposed by ionic:

An app folder with the code. In this app folder:
- each pages (JS, HTML and eventually SASS) are grouped in a folder within pages.
- Global theme SASS files are in the theme folder.
- Boostrap (app.ts) and utils (models.ts) are directly in app.
A build folder used for debugging and by Cordova to generate the final application. It contains the index.html generated by ionic and will contain all the built files.

First, let's talk of models.ts. It only contains one class and is used in the application as a type for the TypeScript compiler (and completion).

export class Todo {
    public id: number;
    public title: string;
    public description: string;
    public done: boolean;
}

In app.ts the application is bootstrapped. It is generated by ionic. I only changed the value of MyApp#rootPage so its value is ListTodosPage the JavaScript class that defines my root page.

That leads us to the first important concept of Ionic2: navigation doesn't work with URLs. As you'll see if you launch the app, the URL is never updated. To navigate between pages, you use a NavController object and pass the page on which you want to go, like that:

this.nav.push(TodoPage)

A page being a class decorated with @Page a decorator that comes from the ionic framework.

On the main page, we display the navbar and the list of all todos. To do that, you use in your template the ion-navbar element. This allows you to easily customize the navbar for each page. Ionic will also automatically add a Back button in this navbar when you navigate to another page.

The main content of the page must be inside the ion-content element. Almost all the elements you will put in it are defined by ionic to help you and starts with ion.

You can also add Angular2 markup:

if the tag starts by a star (*), it's a template, like *ngFor="#todo of todos, #index = index". Note that the syntax is particular: to create a variable in a template, you must put a pound sing (#) before the name of the variable.
if the tag is between parenthesis it is bound to an event like (click)="viewTodo(todo)". The value of the tag will be executed when the event occurs.
if the tag is between bracket like [class]="getClass(todo)" it means its value is one way bound to the expression you pass.

<ion-navbar *navbar>
       <ion-title>TODO list</ion-title>
</ion-navbar>

<ion-content padding>
       <ion-list>
           <ion-item *ngFor="#todo of todos, #index = index">
               <ion-label [class]="getClass(todo)" (click)="viewTodo(index)">{{ todo.title }}</ion-label>
               <button item-right (click)="markTodoDone(todo)">Toggle done</button>
           </ion-item>
       </ion-list>

       <button full round (click)="addTodo()">Add</button>
</ion-content>

On the TypeScript side, we have a class ListTodosPage decorated by @Page. This decorator is only used here to defined the template used for the page but it is also used to define which pipes (filters in Angular2 terminology) and directives are available in the template.

In the constructor, we create the SQL database if it doesn't exist and retrieve the todos:

this.storage.query('CREATE TABLE IF NOT EXISTS todos(id integer primary key unique, title VARCHAR(10), description TEXT, done BOOLEAN)')
  .then(() => this.storage.query('SELECT * FROM todos'))
  .then(results => {
      for (let result of results.res.rows) {
          console.log(result)
          this.todos.push(result);
      }
  })
  .catch(err => console.log(err));

To view a particular todo, we use the navigation controller with navigation parameters:

this.nav.push(TodoPage, {
    todo: this.todos[index],
    index: index,
});

To add a todo, we just omit the navigation parameter.

We can then retrieve the todo on the TodoPage like this:

constructor(private nav: NavController, navParams: NavParams) {
     if (navParams.data.todo) {
         this.todo = navParams.data.todo;
     } else {
         this.todo = new Todo();
     }
}

Once we have our todo (the one we want to edit or a new one), we can create the form and bind its default value to the existing todo. This way, if we have an existing todo, the from will be prefilled with the correct values.

I must say that forms in Angular2 are a huge beast and I am not sure to fully see the interest of such a complex system (at least in most cases). The programming API and the template tags are powerful and allow you to create fine grained validators to suit your needs but they are hard to understand at first. You have a simple syntax which suffers from lack of functionality and a powerful and verbose one (which I use here). Let's look at the code:

let group: {[key: string]: Control} = {
    title: new Control(this.todo.title, Validators.compose([Validators.required])),
    description: new Control(this.todo.description),
    done: new Control(this.todo.done),
};

this.todoForm = new ControlGroup(group);
this.title = this.todoForm.controls['title'];
this.description = this.todoForm.controls['description'];
this.done = this.todoForm.controls['done'];

First we defined the group of input we want. We give them a default value and eventually the validator use to determine if the value is correct or not. Then we create the form and extract from it the controls that will be used in the template for validation.

Now we can look at the template. I won't dig into it. If you want to learn more about forms in Angular2, visit this page.

Just see how we use the controls to see if a field is correct or not and note that we use the [ngFormModel] tag to bind the form to the proper form object.

<form [ngFormModel]="todoForm" (ngSubmit)="save(todoForm.value)">
     <ion-item [class.error]="!title.valid && title.touched">
         <ion-label floating>Title</ion-label>
         <ion-input type="text" value="" [ngFormControl]="title"></ion-input>
     </ion-item>
     <div *ngIf="title.hasError('required') && title.touched"
         class="error-box">
             * Title is required!
     </div>
     <ion-item [class.error]="!description.valid && description.touched">
         <ion-label floating>Description</ion-label>
         <ion-textarea type="text" value="" [ngFormControl]="description"></ion-textarea>
     </ion-item>
     <ion-item [class.error]="!done.valid && done.touched">
         <ion-label>Done</ion-label>
         <ion-checkbox [ngFormControl]="done"></ion-checkbox>
     </ion-item>

     <button full round danger type="reset" (click)="cancel()">Cancel</button>
     <button full round type="submit" [disabled]="!todoForm.valid">Save</button>
 </form>

We can then save the todo with an SQL query and parameters replacement:

this.storage.query(
    'UPDATE todos SET title = ?, description = ?, done = ? WHERE id = ?',
    [todoValues.title, todoValues.description, todoValues.done, this.todo.id]
)

Aurelia and Framework7

I decided to use webpack (an official skeleton is available) instead of the default SystemJS. Two reasons for that:

It's a good way to test webpack support
During my initial research I found that people had problems with SystemJS and cordova. Since webpack is used by ionic2, I expected it to work well.

I followed the same structure than for ionic: each pages in app/pages/<pagename>, services in app/services and bootstrap (main.js and app.js directly in the app folder.

The HTML code in index.html and app.html is almost a copy/paste from the getting started of Framework7. There is just one point in app.html that is different: in the getting started, the code is design for iOS. To use the material design stylesheet (like I did), replace <div class="pages navbar-through toolbar-through"> by <div class="pages navbar-fixed toolbar-through">. Otherwise, the content of the page will partially be in the navbar (due to a missing padding-top: 46px).

main.js contains nothing particular, it's just the standard main provided by the skeleton.

Since navigation in the app will rely on the Aurelia router, we have to defines our route in app.js:

configureRouter(config, router) {
     config.title = 'Aurelia';
     config.map([
         {
             route: ['', 'todos'],
             name: 'todos',
             moduleId: 'pages/list-todos/list-todos',
             nav: true,
             title: 'todos'
         }, {
             route: ['todo', 'todo/:id'],
             name: 'todo',
             moduleId: 'pages/todo/todo',
             nav: true,
             title: 'View Todo',
         }
     ]);

     this.router = router;
 }

And to navigate, I use either:

this.router.navigateToRoute('todo', {id: todo.id});

To view/edit a precise todo. This allows me to get the todo with the given id from storage. To add a new todo, I use:

this.router.navigate('/todo');

To navigate to the route without the parameter.

Concerning the templates, nothing fancy. Just think to put your code in a div with either the list-block or the content-block class depending on what you want to do:

<ul>
    <li class="item-content" repeat.for="todo of todos" click.delegate="viewTodo(todo)">
       <div class="item-inner ${todo.done === true || todo.done === 'true' ? 'done' : ''}">
           ${todo.title}
       </div>
       <p><a href="#" class="button button-fill" click.delegate="toggleDone(todo)">Toggle Done</a></p>
    </li>
</ul>

We use Aurelia's binding mechanism to make the link with the JavaScript:

We repeat the element with repeat.for="todo of todos"
Bind a click event to a function with click.delegate
Use string interpolation to display the values with ${todo.title} or to update the value of a tag with class="item-inner ${todo.done === true || todo.done === 'true' ? 'done' : ''}". The part from ${ to } will be replaced by the value of the expression computed by Aurelia.

The main difference with the ionic app is that I used localForage in a custom class to store the todos. localForage is a small library by Mozilla that uses localStorage if neither IndexedDB nor WebSQL is supported with an API close the localStorage. With ionic, you have a simple way to use SQLite on the phone and WebSQL on your browser with nice capabilities for SQL queries but that's a ionic thing. Since behind the scenes, it relies on Cordova, it should be possible to use it with Aurelia and Framework7 but it felt overkill for my small test.

On the page to edit todos, nothing fancy. I create a nice looking form with the good classes. The structure is a little bit complicated but is necessary to have the proper theme. The rest is just classic html (here the code for one field):

<li>
    <div class="item-content">
        <div class="item-inner">
            <div class="item-title label">Title</div>
            <div class="item-input">
                <input type="text" placeholder="Title" value.bind="todo.title" required>
            </div>
        </div>
    </div>
</li>

Just note that in forms, Aurelia will do two way data bindings with value.bind whereas everywhere else it will do one way data bindings by default for performance reasons.

In the JavaScript, we just have to load the proper todo:

activate(params) {
     this.todo = {};
     if (params.id) {
         this.storage.getTodo(params.id)
             .then(todo => this.todo = todo);
     }
 }

To allow the todo to be saved, I do a basic check on the title (it must not be empty). No message saying that the field is required if the user touches it is displayed. That's the other import difference with ionic. It is due to a bug in aurelia-validation about how translations are loaded with webpack. This is clearly behind what I did in ionic but I'll update the code and this article once it is fixed.

Update: I created a new post about validation and Aurelia. The code samples used in it are based on this application You can see it here.

get canSave() {
     return this.todo.title !== undefined && this.todo.title.length > 0;
 }

Bonus

It's not useful for my application, but I wanted to show how to use Framework7's JavaScript should you need it. In a javascript file (you can name it f7.js) put the code below:

// Load JS for webpack
import 'framework7';


// Create F7 object
export const F7 = new Framework7({
    material: true,
});

You can then use it as you would expect:

import { F7 } from 'f7'
import {inject} from 'aurelia-framework';

@inject(F7)
export class ListTodosPage {

  constructor(f7) {
     this.f7 = f7;
  }
}

And in your HTML:

<a click.delegate="f7.openPanel('left')" href="#" class="button button-big button-fill">Open Left Panel</a>

To see a usage, go here for my code and there for the associated template.

Conclusion

Both application have roughly the same amount of code: 164 lines of JS and 40 lines of HTML for ionic and 143 lines of JS and 103 lines of HTML for Aurelia/Framework7. Since my storage service is about 40 lines that are handled by directly by ionic, I have about 100 lines of JS code for Aurelia. There is more HTML but it's mostly the bootstrap required for Framework7.
I find Aurelia lighter and easier to use. Even if typescript has some nice features like type error detection and a better auto-completion (since it can be based on the type of the object), it looks Javaish with type declarations and interfaces. In addition to that, the syntax for the template is weird (and it breaks some HTML validators like the one integrated in NetBeans IDE). I've used it for some days now, so I begin to get use to it but with my fist experience with Aurelia I didn't have this incomfortable moments at first: templating is plain and simple tag.bind (default) or tag.two-way or tag.one-way depending on what you want to do, string interpolation works everywhere and is just plain ES6 template strings.
ionic uses a lot of custom elements and tags. I guess the goal is to make the code less verbose, but sometime it is confusing. For instance, to make a text input field, you use <ion-input type="text" />. So logically to make a checkbox, I tried <ion-input type="checkbox" /> (which doesn't work) instead of <ion-checkbox [ngFormControl]="done" />. On the contrary, Framework7 is just plain HTML. You just have to respect a certain structure (well detailed on the documentation) and put the proper classes on your element. So it can integrate with pretty much everything and it's quite easy to learn.
ionic has some good bindings with Corodova. For instance its ability to use WebSQL on your browser but SQLite on your phone. There's no easy way (yet?) to do that with Aurelia/Framework7. Maybe Aurelia-Interface will have that too.
the startup time of the application was quite slow (several seconds) with both solutions. Maybe there is a way to improve that, I admit I haven't dug into it yet.
webpack is awesome ;-)

In the end, ionic has a good cli, documentation and some bindings for Cordova which make it complete and interesting. But in my opinion, the Aurelia/Framework7 feels more natural mostly because of the weird design choices of Angular2 and ionic has lots of custom element over HTML.

Maybe I should start something (skeleton and command line tool) to work with Aurelia, Framework7 and Cordova. If there's no news of Aurelia-Interface soon, I think it's highly probable that I'll try. I'll update this post if I do (or decide not to).

Fichiers service pour seafile

2015-11-21T00:00:00+01:00

J'ai écrit des fichiers services pour utiliser seafile plus facilement avec systemd (à placer dans /etc/systemd/system/). Vous pouvez les télécharger :

[Unit]
Description=The seafile cloud service
Requires=mariadb.service
After=mariadb.service


[Service]
Type=forking
PIDFile=/var/www/haiwen/pids/seaf-server.pid
User=seafile
ExecStart=/var/www/haiwen/seafile-server-latest/seafile.sh start
ExecStop=/var/www/haiwen/seafile-server-latest/seafile.sh stop


[Install]
WantedBy=basic.target

[Unit]
Description=The seafile cloud service (web interface)
Requires=seafile.service
After=seafile.service


[Service]
Type=forking
User=seafile
ExecStart=/var/www/haiwen/seafile-server-latest/seahub.sh start-fastcgi
ExecStop=/var/www/haiwen/seafile-server-latest/seahub.sh stop


[Install]
WantedBy=basic.target

Fichiers service pour seafile

2015-11-21T00:00:00+01:00

I wrote service files to use seafile easily with systemd (put them in /etc/systemd/system/). You can download them:

[Unit]
Description=The seafile cloud service
Requires=mariadb.service
After=mariadb.service


[Service]
Type=forking
PIDFile=/var/www/haiwen/pids/seaf-server.pid
User=seafile
ExecStart=/var/www/haiwen/seafile-server-latest/seafile.sh start
ExecStop=/var/www/haiwen/seafile-server-latest/seafile.sh stop


[Install]
WantedBy=basic.target

[Unit]
Description=The seafile cloud service (web interface)
Requires=seafile.service
After=seafile.service


[Service]
Type=forking
User=seafile
ExecStart=/var/www/haiwen/seafile-server-latest/seahub.sh start-fastcgi
ExecStop=/var/www/haiwen/seafile-server-latest/seahub.sh stop


[Install]
WantedBy=basic.target

Utiliser un tableau dans un Makefile

2015-11-12T00:00:00+01:00

J'ai récemment eu besoin de lancer une tâche make pour plusieurs arguments. Je me suis demandé si on pouvait dans le Makefile, en passant une variable en argument sous la forme portals=geojb,n16, répéter une tâche plusieurs fois. Pour cela, il faut réussir à boucler sur l'entrée. Cela se fait comme suit :

.PHONY: toto
toto:
     @echo ${portals}
     IFS=',' && \
     for portal in $${portals}; do \
         echo "$${portal}"; \
         $(call process_portal, $${portal}); \
     done

define process_portal
     echo "processing $1"; \
     echo "done"
endef

Note : on peut tout à fait passer les arguments multiples sous la forme portals="geojb n16". Il faut alors supprimer la ligne IFS=','.

On notera que c'est beaucoup plus facile avec manuel (un petit lanceur de tâche écrit en moins de 150 lignes de bash que j'ai décidé d'utiliser à la place de make) :

#! /usr/bin/env bash

function hello {
    for portal in "$@"; do
        process "${portal}"
    done
}


function _process {
    echo "processing $1"
    echo "done"
}

Tips for libvirt/virtmanager

2015-11-12T00:00:00+01:00

Sommaire

Static IP
Launch Virtual Machine manager without a root password
Compress QCOW2 images
- Linux guest
- Windows guest

Static IP

You can affect a static IP to a guest of a libvirt instance with the following command:

virsh net-update default add ip-dhcp-host "<host mac='52:54:00:7d:2a:5e' name='leap' ip='192.168.122.42' />" --live --config

Launch Virtual Machine manager without a root password

Add the users that need to do this to the libvirt group.
Create a new polkit rule in /etc/polkit-1/rules.d/10-libvirt.rules containing:

polkit.addRule(function(action, subject) {
 if (action.id == "org.libvirt.unix.manage"
         && subject.local
         && subject.active
         && subject.isInGroup("libvirt")) {
     return polkit.Result.YES;
  }
 });

Compress QCOW2 images

Linux guest

Run in the guest:

fstrim -av
dd if=/dev/zero of=/mytempfile
rm -f /mytempfile

Run on the host:

mv image.qcow2 image.qcow2_backup
qemu-img convert -O qcow2 image.qcow2_backup image.qcow2 or if you want compression to further reduce the disk size but reduce performance: qemu-img convert -O qcow2 -c image.qcow2_backup image.qcow2

Then boot your machine and check everything is OK.

Windows guest

Run in the guest:

The defragementer tool
Download sdelete
Run sedelete -z c:

Run on the host:

mv image.qcow2 image.qcow2_backup
qemu-img convert -O qcow2 image.qcow2_backup image.qcow2 or if you want compression to further reduce the disk size but reduce performance: qemu-img convert -O qcow2 -c image.qcow2_backup image.qcow2

Then boot your machine and check everything is OK.

Afficher un SVG progressivement avec reveal.js

2015-11-12T00:00:00+01:00

Pour cela, il faut commencer par installer un plugin : svg fragments. Dans votre installation reveal, dans le sous dossier plugin, cloner le dépôt :

git clone https://gist.github.com/bollwyvl/fe1d2806449487cdf88a

Ensuite, chargez le plugin :

Ajoutez la bibliothèque D3 <script src="reveal/js/d3.min.js"></script> (à télécharger séparément)

Activez le plugin en ajoutant dans l'objet de configuration de reveal (celui passé à Reveal.initialize) :

{src: 'reveal/plugin/svg-fragments/reveal-svg-fragment.js', condition: function () {
               return !!document.querySelector('[data-svg-fragment]');
           }}

Puis dans votre SVG ajoutez un tag comme celui-ci inkscape:label="fragment1" sur les éléments à faire apparaître progressivement. Vous pouvez voir un exemple complet sur github.

Enfin, ajoutez le code correspondant dans votre html. Par exemple, pour le test.svg précédent :

<div data-svg-fragment="test.svg#[*|label=base]">
  <a class="fragment" title="[*|label=fragment1]"></a>
  <a class="fragment" title="[*|label=fragment2]"></a>
</div>

Disponible complètement sur github.

Utiliser un tableau dans un Makefile

2015-11-12T00:00:00+01:00

I recently had to launch a make target for multiple arguments. I tried to do it in a Makefile :

Pass the arguments to loop over portals=geojb,n16
Loop in the Makefile

It turned out to be harder than I thought:

.PHONY: toto
toto:
     @echo ${portals}
     IFS=',' && \
     for portal in $${portals}; do \
         echo "$${portal}"; \
         $(call process_portal, $${portal}); \
     done

define process_portal
     echo "processing $1"; \
     echo "done"
endef

Note: it is possible to pass the arguments like: portals="geojb n16". In this case, you must remove the line IFS=','.

I noted that it is much easier with manuel (a small task launcher written in fewer than 150 lines of Bash which I decided to use instead of make):

#! /usr/bin/env bash

function hello {
    for portal in "$@"; do
        process "${portal}"
    done
}


function _process {
    echo "processing $1"
    echo "done"
}

Se connecter à une machine virtuelle sous VirtualBox

2015-11-09T00:00:00+01:00

J'ai récemment eu besoin de créer un serveur pour faire des tests de déploiement en production pour le travail. Ce serveur est une machine virtuelle créée avec VirtualBox sous Fedora 22.

Pour que les tests soient concluant, je dois pouvoir me connecter à cette machine via SSH. Voici la méthode que j'ai suivie :

Dans File > Preferences > Network, sous l'onglet Host-only Networks, cliquez sur le bouton ajouter. Cette adaptateur sera utilisé par l'hôte et l'invité pour être sur un même réseau privé.
Dans les préférences de la machines, allez dans Network. Activez un adaptateur non encore utiliser et paramétrez le comme suit :
1. Attached to: choisissez Host-only Adapter
2. Name devrait être mis automatiquement à l'adaptateur créé précédemment. Si ce n'est pas le cas, corriger le paramètre.

On peut maintenant démarrer la machine virtuelle. Dans une console, tapez ip a pour récupérer son adresse ip. Vous pouvez l'utiliser pour vous connectez à la machine virtuelle. Vous pouvez également la renseigner dans /etc/hosts pour avoir un nom plus facile à retenir et faciliter la configuration de votre serveur HTTP.

Et c'est tout !

Connect to a VirtualBox VM

2015-11-09T00:00:00+01:00

I recently had to create a server to perform deployment tests for work. This server is a virtual machine created with VirtualBox and runs Fedora 22.

For the tests to be realistic, I had to connect with SSH to the VM. Here is the method I followed:

In File > Preferences > Network, under the Host-only Networks tab, click on the add button. This adapter will be used by the host and the guest to communicate on the same private network.
In the machine settings, go in Network. Activate a new adapter and configure it like this:
1. Attached to: choose Host-only Adapter
2. Name should be set automatically on the adapter you previously created. If it is not, set its value manually.

You can now boot the machine. In a terminal, type ip a to get the ip address fo the VM. You can now connect to the VW with SSH and do HTTP requests to you VM server. You should put the ip address in your /tc/hosts to have a easy to remember name and to ease the configuration of your HTTP server.

Se connecter en wifi à une raspberry pi

2015-11-08T00:00:00+01:00

Sommaire

Connexion
- Rendre la configuration permanente
- Configurer son router
En complément
Problèmes possibles

Par défaut la raspi peut communiquer en Ethernet mais on peut brancher sur un de ses ports USB un petit dongle wifi pour communiquer sans fil (cela se présente comme une petite clé USB). J'ai pour ma part acheté un Edimax EW-7811Un.

Normalement rien de très compliqué. Je détaille ici la procédure pour Archlinux. Pour une autre distribution, les manipulations devraient être similaires.

NB : il est tout à fait possible de tout configurer sans écran via SSH en étant relié par éthernet à la raspi.

Connexion

Normalement, il suffit de brancher le dongle et de démarrer la raspi. Pour tester que tout est correct, vous pouvez utiliser la commande suivante :

wifi-menu -o

Vous devriez voir la liste de tous les réseaux wifi disponibles et pouvoir vous connecter au votre.

Voir Problèmes possibles si ça ne marche pas.

Rendre la configuration permanente

Nous allons utiliser netctl pour nous connecter automatiquement en wifi à notre réseau. Pour cela, nous allons utiliser un profil pré-rempli fournit avec netctl.

cd /etc/netctl
cp examples/wireless-wpa .
vim wireless-wap # Éditer le fichier pour renseigner le SSID et la clé
netctl start wireless-wpa # Démarre netctl, vous devriez pouvoir utiliser le wifi
netctl enable wireless-wpa # Active netctl au démarrage.

Pour plus d'informations sur netctl voir la page de wiki d'Archlinux.

Configurer son router

Pour que la solution soit vraiment pratique, il reste à configurer le routeur wifi pour qu'il donne une ip fixe à la raspi. Cela permettra de renseigner l'adresse de la raspi dans le /etc/hosts et de pouvoir l'utiliser facilement.

Dans le cas de ma BBox, il faut aller sur http://192.168.1.254/ puis configuration du routeur puis DHCP. L'interface permet facilement d'associer une ip à une adresse MAC. Si vous êtes fleimard comme moi, vous connectez la raspi au réseau puis choisissez sa MAC dans la liste déroulante avant de la redémarrer.

Et voilà ! Si vous avez des questions, vous pouvez laisser un commentaire.

En complément

Relier sa Raspberry Pi à son PC directement en Ethernet

Problèmes possibles

Si la raspi ne détecte pas votre carte wifi (possible si votre noyau est un peu ancien), il faut installer le driver à la main. Sous Archlinux, vous pouvez utiliser

pacman -Sy dkms-8192cu

pour l'installer. La compilation du module noyau prend un peu de temps (~12 minutes chez moi). Si la compilation rate car la fonction strnicmp n'est pas trouvé, utiliser la version patchée du module. Pour l'installer :

git clone https://github.com/Jenselme/8192cu-dkms.git
cd 8192cu-dkms
makepkg -si

Utiliser istanbul pour voir le code coverage de tests lancés avec protractor

2015-09-25T00:00:00+02:00

Récemment j'ai eu besoin d'avoir du code coverage (assuré par istanbul) pour des tests d'intégration d'une application AngularJS. Ces tests sont lancés avec protractor et ce n'est pas aussi simple qu'il n'y parait. L'idée de base est :

D'associer une fonction à la clé onPrepare dans la configuration de protractor. Dans cette fonction on définit un rapporteur qui va pour chaque test collecter le code coverage.
D'utiliser un plugin (l'idée initiale est de Jeffrey Barrus) afin d'attendre que le rapport soit écrit (via une promise). En effet, protractor dispose d'une option onComplete qui est une fonction appelée lorsque les tests sont terminés. Cependant, cette fonction ne supporte pas l'asynchronisme : elle ne supporte pas les callbacks et ne va pas attendre qu'une promise soit résolue. Par conséquent, il est possible que le processus de protractor se termine avant que le rapport ne soit écrit ou que istanbul n'ait reçu les données de coverage. Heureusement, la méthode tearDown d'un plugin peut retourner une promise et protractor attendra sa résolution avant de quitter.

Je suppose que vous savez comment préparer vos fichiers javascript pour istanbul (étape appelée instrumentation). Si ce n'est pas le cas, vous pouvez lire la doc.

Les configurations sont les suivantes (disponibles en téléchargement en fin d'article) :

 1 var istanbul = require('istanbul');
 2 var collector = new istanbul.Collector();
 3 var reporter;
 4 var waitPlugin = require('./waitPlugin');
 5 
 6 function report() {
 7   if (reporter) {
 8     reporter.add('json');
 9     reporter.write(collector, true, function () {
10       console.log('Coverage report successfully written');
11     });
12   }
13 }
14 
15 exports.config = {
16   specs: [
17     '../test/integration/*.spec.js',
18     '../test/selenium/*_test.js'
19   ],
20   seleniumAddress: '${seleniumAddress}',
21   maxSessions: 1,
22   multiCapabilities: [
23     {browserName: 'firefox', shardTestFiles: true, maxInstances: 1}
24   ],
25   framework: 'jasmine2',
26   plugins: [{path: './waitPlugin.js'}],
27   onPrepare: function () {
28     var jasmineEnv = jasmine.getEnv();
29     waitPlugin.setOnComplete(report);
30     browser.driver.manage().window().maximize();
31     browser.get('${testPortalAddress}');
32 
33     jasmineEnv.addReporter(new function () {
34       this.specDone = function (spec) {
35         if (spec.status !== 'failed') {
36           var name = spec.fullName.replace(/ /g, '_');
37           var reportfile = 'coverage/integration/json/' + name;
38           reporter = new istanbul.Reporter(undefined, reportfile);
39           var promise = browser.driver.executeScript('return __coverage__;')
40                   .then(function (coverageResults) {
41                     collector.add(coverageResults);
42                   });
43           waitPlugin.waitList.push(promise);
44         }
45       };
46     });
47   }
48 };

On commence par charger les modules (istanbul, le collecteur, le plugin). On crée également la variable reporter. Elle contiendra par la suite un objet utilisé dans différentes fonctions.
Ensuite on crée la fonction report qui sera utilisé par la suite pour écrire le rapport. Pour cela, on donne le format du rapport (ici json). Plusieurs formats sont supportés et istanbul peut générer un rapport dans plusieurs formats à la fois.
Dans la configuration de protractor, rien de particulier jusqu'à la ligne 27 pour la fonction onPrepare.
1. À la ligne 29, on définit la fonction à appeler une fois que toutes les promises du plugin « attendre » sont résolues, en l'occurrence notre fonction report.
2. À la ligne 33, on ajoute un rapporteur à jasmine, le framework de tests. Ce rapporteur comprend une méthode specDone qui est appelée (comme son nom l'indique) à chaque fois qu'un spec est terminé. Si les tests ont réussi, on récupère le code coverage depuis la variable globale __coverage__ de istanbul et on ajoute le résultat au collecteur. Ces opérations étant asynchrone et renvoyant une promise, on ajoute cette dernière à la liste de promises à attendre dans notre plugin.

Pour le plugin wait, rien de très compliqué, je pense que le code parle de lui même. Pensez simplement à installer la bibliothèque q dont dépend le script avec npm install q ou npm install --save q pour ajouter q à votre package.json. En cas de problèmes, vous pouvez laisser un commentaire.

 1 var q = require('q');
 2 
 3 var waitList = [];
 4 var onComplete;
 5 
 6 exports.waitList = waitList;
 7 
 8 exports.setOnComplete = function(cb) {
 9   onComplete = cb;
10 };
11 
12 exports.teardown = function () {
13   return q.all(waitList)
14       .then(function() {
15         if (onComplete) {
16           return onComplete();
17         }
18       });
19 };

Si comme moi, vous devez lancer chaque fichier spec dans un nouveau navigateur, pour avoir la couverture globale, vous devez écrire le résultat de chaque jeu de tests dans un fichier json puis rassembler les résultats (par exemple en un rapport html plus lisible) avec la commande suivante :

istanbul report --include 'coverage/integration/json/**/*.json' --dir 'coverage/integration'  html

Si en revanche vous pouvez lancer tous les tests dans un seul navigateur, vous pouvez récupérer le rapport au format HTML directement en changeant ligne 8 de la configuration de protractor json en html.

Téléchargements

À réutiliser sans modération !

Use istanbul to get code coverage of integration tests launched with protractor

2015-09-25T00:00:00+02:00

Recently, I needed code coverage (generated by istanbul) for integration tests of an AngularJS application. These tests are launched with protractor. Sadly, this is not as easy as it may sound. The idea is to:

associate a function to the onPrepare key in protractor's configuration. In this function, we define a reporter which will for each test collect the code coverage.
use a plugin (the idea is from Jeffrey Barrus) in order to wait for the report to be written (with a promise): protractor has a onComplete option (a function called once all the tests are completed). However, this function doesn't support asynchronism: you cannot use callback or promise. So protractor may quit before the report is written or before the reporter has received any data. Hopefully, the tearDown method of a plugin can return a promise and protractor will wait its fulfillment before quitting.

I suppose that you know how to prepare your js files for istanbul (this step is called instrumentation). If you don't, you should read the doc.

Here are the configurations (you can download them at the end of the article):

 1 var istanbul = require('istanbul');
 2 var collector = new istanbul.Collector();
 3 var reporter;
 4 var waitPlugin = require('./waitPlugin');
 5 
 6 function report() {
 7   if (reporter) {
 8     reporter.add('json');
 9     reporter.write(collector, true, function () {
10       console.log('Coverage report successfully written');
11     });
12   }
13 }
14 
15 exports.config = {
16   specs: [
17     '../test/integration/*.spec.js',
18     '../test/selenium/*_test.js'
19   ],
20   seleniumAddress: '${seleniumAddress}',
21   maxSessions: 1,
22   multiCapabilities: [
23     {browserName: 'firefox', shardTestFiles: true, maxInstances: 1}
24   ],
25   framework: 'jasmine2',
26   plugins: [{path: './waitPlugin.js'}],
27   onPrepare: function () {
28     var jasmineEnv = jasmine.getEnv();
29     waitPlugin.setOnComplete(report);
30     browser.driver.manage().window().maximize();
31     browser.get('${testPortalAddress}');
32 
33     jasmineEnv.addReporter(new function () {
34       this.specDone = function (spec) {
35         if (spec.status !== 'failed') {
36           var name = spec.fullName.replace(/ /g, '_');
37           var reportfile = 'coverage/integration/json/' + name;
38           reporter = new istanbul.Reporter(undefined, reportfile);
39           var promise = browser.driver.executeScript('return __coverage__;')
40                   .then(function (coverageResults) {
41                     collector.add(coverageResults);
42                   });
43           waitPlugin.waitList.push(promise);
44         }
45       };
46     });
47   }
48 };

We start by charging the modules (istanbul, the collector, the plugin) and we create the reporter variable. It will contain the Reporter object used in various functions.
Then we create the report function which will be used to write the report. To do that, we specify the format of the report (json). More formats are supported by istanbul.
In protractor's configuration, nothing special until line 27 in the onPrepare function:
1. Line 29, we define the function that must be called once all the promises of the waitPlugin are fulfilled (here our report function).
2. Line 33, we add a reporter to jasmine, our test framework. This reporter only has a specDone method which is called each time a spec file is completed. If the tests were successful, we get the code coverage from the __coverage__ global variable and we add the result to the collector. These operation are asynchronous and return a promise. We add this promise to the list of promise to wait for in our plugin.

For the waitPlugin, nothing special, I think the code is obvious enough. Just think to install the q library which is required by the plugin to have promises. You can install it with npm install q or npm install --save q if you want to save this dependency in your package.json. If you have questions, you can leave a comment.

 1 var q = require('q');
 2 
 3 var waitList = [];
 4 var onComplete;
 5 
 6 exports.waitList = waitList;
 7 
 8 exports.setOnComplete = function(cb) {
 9   onComplete = cb;
10 };
11 
12 exports.teardown = function () {
13   return q.all(waitList)
14       .then(function() {
15         if (onComplete) {
16           return onComplete();
17         }
18       });
19 };

If like me, you must launch each spec file in a new browser, in order to have the global coverage, you need to write the result of each spec in a json file and then merge all the results with the following command:

istanbul report --include 'coverage/integration/json/**/*.json' --dir 'coverage/integration'  html

If you can however launch all the tests in a single browser, you can get the report directly in HTML by changing line 8 of protractor's configuration (json should become html).

Downloads

Utiliser postfix pour relayer ses mails vers un autre serveur

2015-08-18T00:00:00+02:00

J'ai récemment eu besoin de pouvoir envoyer des mails depuis ma machine avec un serveur de mails tournant en local. La façon la plus simple de faire ça est à mon avis d'avoir un serveur de mail qui relaie les mails vers un autre serveur mail. Pour des cas simples (juste l'utilisation de la commande mail, ssmtp est suffisant, voir configuration à la fin). Pour avoir un vrai serveur smtp, j'ai décidé d'utiliser postfix qui fait ça très bien et que j'utilise sur mon serveur.

En prérequis :

installer postfix
disposer d'un autre serveur smtp sur lequel on peut se connecter (le vôtre, Gmail, Outlook, …)

Éditer /etc/postfix/main.cf et ajouter :

relayhost = smtp.exemple.com:587
smtpd_sasl_path = smtpd
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_type = cyrus
smtp_sasl_auth_enable = yes
smtp_use_tls = yes

# Pour que le seveur ne relaie que les mails venant de localhost
# Attention, c'est le réseau 127.0.0.0/8 qui doit être utilisé et
# pas l'ip (127.0.0.1 ou 127.0.0.1/8)
mynetworks = 127.0.0.0/8, [::1]/128
smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination

# Nécessaire uniquement si le serveur smtp a besoin que le mot de passe
# soit envoyé en texte clair.
smtp_sasl_security_options = noanonymous

# Nécessaire uniquement votre fournisseur utilise du load balancing et
# transfère les mails vers un autre serveur SMTP.
smtp_cname_overrides_servername = no

Il reste ensuite à créer le fichier de mots de passes. Dans /etc/postfix/sasl_passwd avec :

smtp.exemple.com:587 nomutilisateur:mdp

Ensuite, utiliser la commande postmap hash:/etc/postfix/sasl_passwd pour générer fichier empreintes des mots passes utilisé par postfix. Vous pouvez tester la configuration avec postmap -q smtp.provider.com:25 /etc/postfix/sasl_passwd qui doit renvoyer nomutilisateur:mdp. Vous pouvez désormais supprimer /etc/postfix/sasl_passwd et redémarrer postfix.

Note : il est possible que vous ayez besoin de lancer newaliases pour terminer la configuration initiale du serveur.

Comme j'ai écrit la conf pour ssmtp, la voilà (dans /etc/ssmtp/ssmtp.conf) :

mailhub=smtp.example.com:587
AuthUser=nomutilisateur
AuthPass=mdp
# Si le serveur utilise StartSSL, ne pas utiliser avec UseTLS=yes
UseSTARTTLS=YES

A light branching strategy with mercurial and bookmarks

2015-07-22T00:00:00+02:00

Sommaire

Using bookmarks
Other mercurial tips
Conclusion

I am a contributor to nbpython the netbeans plugin for python. Since the DCVS used is mercurial and I have almost exclusively used git before I add to adapt my workflow to work properly.

What I want to do:

create a branch from the current default branch (the equivalent of master in git) to work on my contributions. This branch is temporary, used for development and code review (so it must be pushed and have an easy way to update to it) and should be deleted once the code is merge in the main trunck.
the code must be "rebased" in git terminology above the latest commit in default to preserve a linear history.

If you are coming from a git background, you must know that mercurial has four branching mechanism available:

branching by cloning: you just do a hg clone of the repo in another folder. This is far from efficient in term of disk space and network usage but it is the older branching mechanism and I only present it for the sake of completeness. I don't see the point to use it now.
anonymous branch: to create such a branch, you go to a previous revision and make a new commit. The only way to switch to this branch is with the commit hash (hence the name anonymous). In git, you just cannot do that. If you go to a commit that is associated with a branch, git warns you that you are in detached HEAD and that the commit you do will be discarded at some point (probably in the next garbage collection). In mercurial, you can do this as it allows multiple heads.
named branch: again, there is no equivalent in git. Theses branches are global and permanent: you can close them so they are not listed by default with hg branches (which list all the named branches) but you cannot delete them. You create them with hg branch <branch-name>. Furthermore, the name of the branch will be stored with the commit metadata. This look good for long lasting branch for some important features but this is too heavy for what I am looking for.
bookmarks: you can bookmark any change in mercurial. If a bookmark is active and you do a commit, the bookmark will be updated to point to this commit. Nothing is stored as metadata. This what looks more like the branch in git and this is what I want.

Using bookmarks

You can easily:

navigate to a bookmark with hg up <bookmark> (this is the command you use to go to any revision let it be a commit, a branch or a bookmark)
list them with hg bookmarks
delete them with hg bookmark -d <bookmark-name>. This will only delete the bookmark, not the commits associated with it which are still accessible with their hashes (even if they are not merged in a named branch). See below to learn how to do that.
push them to the remote repository with hg push -B <bookmark-name. This will create the bookmark if it doesn't exist or update it if it is already present.

One disturbing side effect when you use bookmark is that a branch will have multiple heads. So I use the rebase extension and do a hg pull --rebase to get the changes from http://hg.netbeans.org/main/contrib. If you used mercurial before you may have done instead hg pull and then hg update (and eventually hg merge to solve conflict) or used the fetch extension and did hg fetch which just does all this in one command.

So, how do I work?

Before doing anything, I created a main bookmark at the current state of default so that I could go back there easily. You should do the same. Once this is done, here's what I do:

I do a hg pull --rebase netbeans to get the latest code from hg.netbeans.org (my default remote point to my bitbucket clone so I don't push by error a bookmark on netbeans).
I create a bookmark with hg bookmark <feature>.
I work and commit normally.
I do a hg push -B <feature> to push my patches to my bitbucket repo so the other contributors can review my work.

Once my changes are validated, I must rebase my changes to the top of default. The fist way I found out was to export my changes with hg export -r <list of revisions to export> > file.patch and then do a hg up main and finally hg import < file.patch. This worked but the graft extension makes it easier. graft is the equivalent of cherry-pick. So here's what I do:

hg up main
hg graft -r <list of revision to export>
hg push -r main netbeans to push the new revision under the bookmark main to hg.netbeans.org to the default branch without creating the bookmark.

Now I want to delete the branch. To do that, I use the strip extension. It deletes a commit and all its children. So a hg strip -r <fist commit of bookmarked branch will to the trick. But it has an option to delete a bookrmark and all its parents (until it reaches the original parent of course). So I do: hg strip -B <feature>.

Other mercurial tips

Here are some tips I learned working with mercurial. There are too few of them for a dedicated article yet. I will update this list when I learn more of them.

To do the equivalent of git rebase -i, use hg histedit -r <revision>.
To discard the last commit: hg strip --keep. This will remove the last commit from history but let the files like they were after this commit.
To get hg st to display the relative path of files, add this to your .hgrc:

[alias]
__mystatus = status
st = !hg __mystatus $($HG root) $HG_ARGS

Some extensions that I think are a must have:

pager: by default, mercurial don't page its output. So for instance if you have lots of commits and do a hg log, they will all be printed at once. Not very practical. pager allow you to give the result to less.
progress: to view the progress of network operation (like push, pull, clone). Sadly not enabled by default. Once the extension is enabled, you must configure it:
```
[pager]
pager = LESS='FRX' less
```

To activate an extension, add it to the [extensions] section of your .hgrc like this:

[extensions]
progress =

You can view my complete .hgrc here.

Conclusion

Coming from git, mercurial is a bit disturbing at first. The main critic I have is that even for things that I consider to be basic like a less pager or to view progress of some operations you must enable some extensions. However, despite being different, it is just a matter of habit to be at ease with mercurial. If you haven't tried it yet, you really should:

it works great
I think that the command names are better and less confusing
it has nice local number to identify commits. So instead of doing hg up -r <hash> you can just do hg up -r 78.
it has the tip which identify the last commit you made
and many more.

Déplacer des lignes correspondant à une regexp en bash

2015-07-22T00:00:00+02:00

Récemment sur un projet j'ai dû mettre à jour closure compiler. Afin de faire cette mise à jour, j'ai dû déplacer certaines lignes de code utilisées par closure : le compilateur utilise des goog.require et des goog.provide pour trouver les dépendances entre les fichiers. Dans la version utilisée par le projet, ces lines étaient dans une IEF :

(function() {
       goog.require('toto');
       goog.provide('titi');

Mais cette façon d'écrire le code n'est plus possible dans les versions récentes du closure compiler : les dépendances doivent être gérées directement au niveau du fichier. Les goog.require et goog.provide doivent donc être déplacées dans tous les fichiers en dehors de l'IEF. Vu le nombre de fichiers concernés, j'ai écrit le script suivant pour le faire automatiquement.

# Boucle sur tous les fichiers qui ont une IEF dans les sources
for file in $(grep -rl '^(function() {' src | grep -v lib); do
    # On supprime la ligne de l'IEF
    sed -i '/^(function() {/d' "${file}"
    # On supprime l'indentation des goog.*
    sed -i -r 's/^  (goog.(provide|require))/\1/g' "${file}"
    # Affiche le fichier de bas en haut.
    # Pour la première ligne qui correspond à la gestion de dépendance, on
    # "imprime" la première ligne de l'IEF et on affiche la ligne de dépendance.
    # Remet le ficher à l'endroit.
    tac "${file}" | awk '/goog.(provide|require)/ && ! seen {print "(function() {"; seen=1} {print}' | tac > "${file}.new"
    # On doit utiliser un nouveau fichier car du fait des pipes, le fichier
    # va s'écrire avant d'avoir été complètement lu, ce qui va tout faire planter.
    mv -f "${file}.new" "${file}"
done

Supprimer la ligne encoding de fichiers python

2015-07-22T00:00:00+02:00

Lors du passage à python 3 sur un projet, j'ai décidé de supprimer les lignes qui donnent l'encodage du fichier. En effet, ils étaient tous en UTF-8 et c'est l'encodage par défaut sous python 3.

J'ai écrit ce petit script pour automatiser tout ça :

for file in $(find chsdi/ -name \*.py); do
  # Remove coding line
  sed -i '/^# *-\*- .*coding: utf-8 -\*-/d' $file
  # Remove whitespace line after it if it exists
  if head -n 1 $file | grep '^$'; then
    sed -i '1d' $file
  fi
done

Move line matching a regexp in bash

2015-07-22T00:00:00+02:00

Recently I had to update the closure compiler on a project. To do so, I had to move some code lines used by closure: the compiler uses goog.require and goog.provide to manage dependencies. In the version used by the project, these lines were in an IEF:

(function() {
       goog.require('toto');
       goog.provide('titi');

But this way to write code is not possible any more with newer versions of the closure compiler: dependencies must be managed at file level. goog.require and goog.provide must be moved out of the IEF. Since lost of files were to be updated, I wrote the following script to automate the process.

# Loop on all files that have an IEF in source folder
for file in $(grep -rl '^(function() {' src | grep -v lib); do
    # Remove the first line of the IEF
    sed -i '/^(function() {/d' "${file}"
    # Remove goog.* indent
    sed -i -r 's/^  (goog.(provide|require))/\1/g' "${file}"
    # Print the file from bottom to top.
    # For the fist line concerning dependencies management, we print the
    # first line of the IEF and we print the dep line.
    # Reverse the file again.
    tac "${file}" | awk '/goog.(provide|require)/ && ! seen {print "(function() {"; seen=1} {print}' | tac > "${file}.new"
    # We must use a new file because due to the pipes, the file may get
    # written before it is completely read. That will prevent the script to
    # work as expected.
    mv -f "${file}.new" "${file}"
done

Remove the encoding line in python files

2015-07-22T00:00:00+02:00

When making a python project python 3 only, I decided to delete the lines giving the encoding of the files. All files were in UTF-8 and it is the default encoding in python 3.

I wrote this little script to get the job done.

for file in $(find chsdi/ -name \*.py); do
  # Remove coding line
  sed -i '/^# *-\*- .*coding: utf-8 -\*-/d' $file
  # Remove whitespace line after it if it exists
  if head -n 1 $file | grep '^$'; then
    sed -i '1d' $file
  fi
done

CORS and HTTP authentication

2015-06-27T00:00:00+02:00

Sommaire

Nginx
Apache
Javascript
See also

Before doing a request (POST, GET, PUT, …) on another domain with Javascript, web browsers will perform an OPTIONS request to verify that the request is likely to be accepted by the server. They mostly check for CORS headers.

This doesn't cause any troubles if your service doesn't require authentication: the OPTIONS request is performed without the authentication headers. So, for instance, if you are trying to connect to an owncloud instance with the webdav protocol, the OPTIONS request will failed due to a 401 unauthorized error.

To avoid that, you must make sure that when requested with the OPTIONS method, the endpoint always respond with a 200 status code and with the proper header even without authentication. You can either do it programmatically in the code of your application or with your web server configuration.

I provide the correct configuration sample for nginx and apache. You should be able to adapt them for other web server or for your favorite programming language. Both include the headers and how to return the 200 status code on options. I also provide a sample JS code that can make the request.

Nginx

add_header 'Access-Control-Allow-Methods' 'GET,OPTIONS,PUT,DELETE' always;
add_header 'Access-Control-Allow-Credentials' 'true' always;
add_header 'Access-Control-Allow-Origin' '$http_origin' always;
add_header 'Access-Control-Allow-Headers' 'Authorization,DNT,User-Agent,Keep-Alive,Content-Type,accept,origin,X-Requested-With' always;

if ($request_method = OPTIONS ) {
    return 200;
}

You may also need to add the Access-Control-Expose-Headers to allow the browser to export the headers. This will be required to use them from AngularJS for instance:

add_header 'Access-Control-Expose-Headers' 'Authorization,DNT,User-Agent,Keep-Alive,Content-Type,accept,origin,X-Requested-With,Content-Disposition' always;

Apache

Header always set Access-Control-Allow-Origin "http://waffle"
Header always set Access-Control-Allow-Methods "POST, GET, OPTIONS"
Header always set Access-Control-Allow-Credentials "true"
Header always set Access-Control-Allow-Headers "Authorization,DNT,User-Agent,Keep-Alive,Content-Type,accept,origin,X-Requested-With"

RewriteCond %{REQUEST_METHOD} OPTIONS
RewriteRule ^(.*)$ $1 [R=200,L]

If you need Access-Control-Allow-Origin to be exactly the domain from the origin header, you can use:

SetEnvIf Origin "http(s)?://(.+)$" CORS=$0
Header always set Access-Control-Allow-Origin %{CORS}e env=CORS

You may also need to add the Access-Control-Expose-Headers to allow the browser to export the headers. This will be required to use them from AngularJS for instance:

Header always set Access-Control-Expose-Headers 'Authorization,DNT,User-Agent,Keep-Alive,Content-Type,accept,origin,X-Requested-With,Content-Disposition';

Javascript

Your javascript code must supply the Authorization and Content-Type headers. For instance, with the angular framework:

$http({
  method: 'POST',
  url: scope.webdav.url,
  withCredentials: true,
  headers: {
    Authorization: 'Basic ' + btoa(user + ':' + password),
    'Content-Type': 'application/vnd.google-earth.kml+xml; charset=utf-8'
  },
  data: getKml()
})

Use udev to setfacl when mounting a usb drive

2015-05-31T00:00:00+02:00

Write the line below (and adapt it) in /etc/udev/rules.d/:

SUBSYSTEMS=="block",ACTION=="add",KERNEL=="sd?1",RUN+="/usr/bin/setfacl -m    u:apache:r-x /run/media/jenselme"

Attention: if the path of the command is not absolute, udev will search of it in /usr/lib/udev.

Use docker to deploy a Glassfish application

2015-05-24T00:00:00+02:00

Recently I had to deploy an application on Glassfish. Since Glassfish is not packaged on the distribution I have on my server (Fedora Server) and I wanted to avoid to pollute my system I decided to use Docker.

In order to deploy my application correctly, I based my Dockerfile on the latest official image of glassfish from the docker hub. This image is based on debian and it supports java 8 (there is another image which support java 7).

Nothing fancy in the dockerfile. I copy the the war file in the container and set the entry points to a custom script. This script will start Glassfish, deploy the application and then restart the server in verbose mode.

## -*- docker-image-name: "aot-java" -*-
# AoT
#
# Version 1

FROM glassfish:latest

MAINTAINER Jujens "jujens@jujens.eu"

COPY aot.war /
COPY start.sh /

EXPOSE 8080

ENTRYPOINT ["/start.sh"]

Why didn't I just deploy the application in the dockerfile? In order to deploy the app, Glassfish must be running. If we used the following code to deploy the application, what would have happened?

RUN /usr/local/glassfish4/bin/asadmin start-domain
RUN /usr/local/glassfish4/bin/asadmin -u admin deploy /aot.war

Docker create a temporary container and launch the command in it. Once the server has started, the command will exit and the container will stop. Then docker will create a new temporary container and launch /usr/local/glassfish4/bin/asadmin -u admin deploy /aot.war in it. But since the first container is stopped, Glassfish is not running any more. So the deploy command fails.

If we started the server with /usr/local/glassfish4/bin/asadmin start-domain --verbose, the command would never have exited and docker would have been stuck.

So the solution is to use a custom script as entry point which will:

Start the server
Deploy the application
Stop the server
Restart it in verbose mode so this command doesn't exit and the container doesn't stop.

You can then run the container with: docker run --name <container-name> -d -p 8080:8080 <image-name>.

Here is the script:

#!/bin/sh

/usr/local/glassfish4/bin/asadmin start-domain
/usr/local/glassfish4/bin/asadmin -u admin deploy /aot.war
/usr/local/glassfish4/bin/asadmin stop-domain
/usr/local/glassfish4/bin/asadmin start-domain --verbose

Another solution would have been to simply use the default image, launch a container from it, connect to it with nsenter and deploy the war from there. But I find this method quite tricky and you have to use docker in a way that was never intended.

Docker tips

2015-05-24T00:00:00+02:00

For my tips about docker compose, go here.

Sommaire

Create an image
Save and import
Run
Volumes
Networking
Attach to a running container
Use customs DNS
Usage with sudo

Create an image

docker build -t <name> .

Save and import

docker save <image> > file
docker load < <image>

Run

docker run --name aot -d -p 8080:8080 --link redis:redis aot

Volumes

docker run -it --rm -v $(pwd)/test:/root busybox /bin/sh

If SELinux is enabled, use: Z (capital z, recommended, better matches the context of the container, eg system_u:object_r:container_file_t:s0:c667,c788 with the c667,c788 part changing for each container) or z (lower caps z to have a context appliable to all containers, eg system_u:object_r:container_file_t:s0):

docker run -it --rm -v $(pwd)/test:/root:Z busybox /bin/sh

If you cannot run with Z, run chcon -R system_u:object_r:container_file_t:s0 FOLDER. If you must use a different context on your system, start a container with a volume mounted with z (lower caps z indented) and run ls -dZ FOLDER to get the proper context.

See https://blog.voina.org/docker-selinux/, http://www.projectatomic.io/blog/2015/06/using-volumes-with-docker-can-cause-problems-with-selinux/ and https://opensource.com/business/14/9/security-for-docker

You can also use multiple mount options by separating them with a comma. For instance to mount a read only file with SELinux support, you can do this:

docker run -it --rm -v $(pwd)/test/README.rst:/root/README.rst:ro,Z busybox /bin/sh

Networking

If you have SELinux enabled and use user namespaces, you will hit two SELinux errors related to ip tables:

avc:  denied  { net_admin } for  pid=7464 comm="iptables" capability=12  scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:system_r:iptables_t:s0 tclass=cap_userns permissive=1
avc:  denied  { net_raw } for  pid=8158 comm="iptables" capability=13  scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:system_r:iptables_t:s0 tclass=cap_userns permissive=0

To allow these actions in SELinux, run as root (the plain text description of the rule is avaiable in dk-userns-iptables.te):

ausearch -c 'iptables' --raw | audit2allow -M dk-userns-iptables
semodule -X 300 -i dk-userns-iptables.pp

Attach to a running container

If you are running docker 1.3 or above, you should use: docker exec -it CONTAINER COMMAND to run COMMAND within the container. You can easily create a function to ease the thing and run bash by default:

dk-enter() {
    docker exec -it "$1" "${2:-/bin/bash}"
}

You can easily open a shell in a running docker container with the following commands:

Get the container pid: docker inspect -f {{.State.Pid}} <container-id>
Connect: nsenter --target <pid> --mount --uts --ipc --net --pid

Use customs DNS

During development you may need to use custom domains to reach your site. Since these domains aren't real and must point to your machine, you need to use a custom DNS resolver. I'll detail here, how to acheive this with dnsmaq and Network Manager.

First, you need to install dnsmaq. It should be in the repositories of your distribution.

Then you need to map your domains to the proper ip addresses. To do that, create as root /etc/NetworkManager/dnsmasq.d/mapping.conf and add something like this to map the domain to the ip:

address=/dev.here/127.0.0.1

Now, as described here you need to configure Network Manager to listen to the docker bridge, so the DNS requests from your containers are correctly passed to dnsmasq. To do so, create /etc/NetworkManager/dnsmasq.d/docker-bridge.conf with this content (if you use another ip than the default one, you will have to adapt this):

listen-address=172.17.0.1

Restart Network Manager for the changes to be taken into account: systemctl restart NetworkManager.

Then, as described here, we need to configure the docker daemon to use our DNS. To to that, create /etc/docker/daemon.json with this content (or add the dns entry if the file already exists):

{
  "dns": [
    "172.17.0.1"
  ]
}

Restart docker for the changes to be taken into account: systemctl restart docker.

Finally, you need to allow DNS requests in your firewall (the commands below are for firewalld, default firewall on Fedora):

Allow permanently the DNS service in the firewall (remove --permanent and don't run the second command to make this temporary): firewall-cmd --add-service=dns --permanent
Reload the configuration: firewall-cmd --reload

Usage with sudo

If you want your users to use the docker command but want to prevent some options, you should rely on sudo to allow your users to run sudo docker SOMETING. You can then configure sudo to prevent some action. Here is an example that allows user jenselme to run sudo docker without a password without allowing the --privileged option nor to disable namespaces:

jenselme ALL=(root) NOPASSWD: /usr/bin/docker,
    ! /usr/bin/docker *--privileged*,
    ! /usr/bin/docker *=host*

AngularJS tips

2015-04-26T00:00:00+02:00

This page lists all the angularjs tips I have collected.

Sommaire

Disable HTTP request caching
Launch an action after a $digest
Download binary data
Debugging
- Get the scope
- Get a service

Disable HTTP request caching

This is mostly useful on Internet Explorer: every HTTP requests made with the $http service will be cached by the browser. This means that if a resource changed on the server, refetching it, by default, will not show the modification to the client. The application will be presented the request from the cache of the browser. To prevent that, we disable the cache for HTTP requests.

module.config($httpProvider => {
        // Disable caching for Internet Explorer.
        // See: http://stackoverflow.com/a/19771501
        $httpProvider.defaults.cache = false;
        if (!$httpProvider.defaults.headers.get) {
                $httpProvider.defaults.headers.get = {};
        }
        // Disable IE ajax request caching
        $httpProvider.defaults.headers.get['If-Modified-Since'] = '0';
});

Launch an action after a $digest

Use $timeout with only one parameter:

$timeout(function do_after_digest() {});

Download binary data

You must set responseType to 'arraybuffer' in the options of the requests, like this:

$http({
        url: 'your/webservice',
        method: "POST",
        data: json, //this is your json data string
        headers: {
        'Content-type': 'application/json'
        },
        responseType: 'arraybuffer'
}).success(function (data, status, headers, config) {
        var blob = new Blob([data], {type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"});
        var objectUrl = URL.createObjectURL(blob);
        window.open(objectUrl);
}).error(function (data, status, headers, config) {
        //upload failed
});

Cf : http://stackoverflow.com/a/22448640

Debugging

Note: $0 refers to the last selected element by the debug tool. Works at least on Firefox and Chrome. In all these examples, this element must be under the controller.

Get the scope

var scope = angular.element($0).scope()

Get a service

var myService = angular.element($0).injector().get('myService')

Git tips

2015-04-26T00:00:00+02:00

This pages provides all gitips I have collected.

Sommaire

Rebase
- Split commit
- Autosquash
Cancel file deletion
Patch
Branches
Ignoring versioned files
Permanently remove files or folder from repo
Run multiple hooks in parallel
Revert a merge commit

Rebase

Split commit

After git rebase -i, run git reset HEAD~ to split current commit.

To split 3rd most recent commit, git rebase -i HEAD~3.

Autosquash

You can auto-squash all commits you made to correct a previous commit with:

Create a fixup commit: git commit --fixup=COMMIT_HASH
Run rebase --interactive --autosquash REF

Cancel file deletion

touch <filename>
git checkout <filename>

Patch

Create a patch: git format-patch -1 <commit>
Apply as patch git apply <patch>
Apply as commit: git am <patch>

Branches

Delete a local branch: git branch -d <branch-name>
Delete a local branch with unmerge work: git branch -D <branch-name>
Delete a branch on the server: git push origin --delete <branch-name> or git push origin :branchname on old version of git

Ignoring versioned files

Run git update-index --assume-unchanged path/to/file.txt To cancel the operation, run: git update-index --no-assume-unchanged path/to/file.txt.

Note: git stash and git checkout . will have an effect on the file.

Source: https://stackoverflow.com/questions/18276951/how-do-i-stop-git-from-tracking-any-changes-to-a-file-from-this-commit-forward/18277622#18277622

Permanently remove files or folder from repo

This is an potentially dangerous command as it rewrites the history. It can discard commits if you have shared your code with other. Use carefully. You will need to use git push -f to update any distant repo.

It also implies that if you are working on a forked repository (on github for instance) you won't be able to make pull requests any more as all the rewritten commits will be included in it (that can amount to several hundreds) and it is most likely that upstream won't want them.

git filter-branch -f --prune-empty --index-filter 'git rm -r --cached --ignore-unmatch src/main/webapp/inc/img'  -- HEAD

With:

--ignore-unmatch allows you to ignore nonexistent files
-r to delete recursively
--prune-empty to discard empty commit

If you use a commit range (hash1..ref):

the first commit is not filtered
the end of the range must be a reference (a branch name for instance). You can try to use an temporary branch as describe here if needed.

If you messed up, you can use the reflog to reset your repo until you run the following commands. You must then remove the backup and force garbage collection to regain your disk size:

rm -rf .git/refs/original
git reflog expire --all --expire=now
git gc --prune=now --aggressive

Normally, after a git push -f you should see the reclaimed space. You can also manually update the remote tracked branch by using something like origin/master as reference.

You can add the --all option to rewrite all branches and tags:

git filter-branch --index-filter 'git rm -rf --cached --ignore-unmatch src/main/webapp/inc/img' -- --all

If you use --tree-filter with the command rm -rf instead of --index-filter you will get the result slower as it checkouts the tree.

You can use this command to find big files. It lists all files with their size for each commit in master. Then, it selects only the size and the file. Lastly, it sorts files by size, big size first and each file is counted only once.

git ls-tree -lr master | awk '{print $4 " " $5}' | sort -unr

The documentation page.

Sources:

For the exact command to run garbage collection: http://stackoverflow.com/questions/2116778/reduce-git-repository-size
For the other explanations: http://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository

Run multiple hooks in parallel

git hooks are very useful because they allow you to run a script before some actions (for instance: run a linter before committing or run the tests before pushing). If the script exists with 0, the action proceeds, if not, the action is cancelled.

The problem is that if you have many actions to run, it can be slow. One way to mitigate this is to run the commands of the script in parallel by terminating the line with & and rely on the wait builtin to wait for all commands to complete before continuing.

You cloud launch all the commands and at the end of the script simply put wait to wait for them but the script will always exit with the 0 status code making the hook useless.

One solution, to exit with a non zero status if a command had an error to do (adapt to your needs):

#!/usr/bin/bash
# See https://stackoverflow.com/a/26504775 for some background

make lint-js &
JS_LINT_PID=$!

make lint-py &
PY_LINT_PID=$!

pipenv run python manage.py compilemessages &
MO_LINT_PID=$!

wait ${JS_LINT_PID}
JS_LINT_EXIT_CODE=$?

wait ${PY_LINT_PID}
PY_LINT_EXIT_CODE=$?

wait ${MO_LINT_PID}
MO_LINT_EXIT_CODE=$?

EXIT_CODE=$((${JS_LINT_EXIT_CODE} + ${PY_LINT_EXIT_CODE} + ${MO_LINT_EXIT_CODE}))

exit ${EXIT_CODE}

In a nutshell:

we capture the PID of the command we just run in a variable with $!.
wait for the process associated to this PID to complete with wait $PID
capture the exit code of wait (which will match the exit code of the command) with $?
sum the code. If any of them is upper than 0, the sum will be too and we will exit with a non zero status code.

The problem with this basic script is that you don't have access to the output of the commands. Thus, you know you have a problem but don't know which one. This can be solved by sending the output of each command into a temporary file and displaying the content of this file if the command failed:

#!/usr/bin/bash
# https://stackoverflow.com/a/26504775

JS_LINT_LOG=$(mktemp)
make lint-js > ${JS_LINT_LOG} 2>&1 &
JS_LINT_PID=$!

PY_LINT_LOG=$(mktemp)
make lint-py > ${PY_LINT_LOG} 2>&1 &
PY_LINT_PID=$!

MO_LINT_LOG=$(mktemp)
pipenv run python manage.py compilemessages > ${MO_LINT_LOG} 2>&1 &
MO_LINT_PID=$!

wait ${JS_LINT_PID}
JS_LINT_EXIT_CODE=$?

wait ${PY_LINT_PID}
PY_LINT_EXIT_CODE=$?

wait ${MO_LINT_PID}
MO_LINT_EXIT_CODE=$?

EXIT_CODE=$((${JS_LINT_EXIT_CODE} + ${PY_LINT_EXIT_CODE} + ${MO_LINT_EXIT_CODE}))

if [[ "${JS_LINT_EXIT_CODE}" -ne 0 ]]; then
    echo "JS LINT OUTOUPUT" >&2
    cat ${JS_LINT_LOG} >&2
    echo -e "\n\n"
fi

if [[ "${PY_LINT_EXIT_CODE}" -ne 0 ]]; then
    echo "PY LINT OUTOUPUT" >&2
    cat ${PY_LINT_LOG} >&2
    echo -e "\n\n"
fi

if [[ "${MO_LINT_EXIT_CODE}" -ne 0 ]]; then
    echo "MO LINT OUTOUPUT" >&2
    cat ${MO_LINT_LOG} >&2
    echo -e "\n\n"
fi

rm ${JS_LINT_LOG} ${PY_LINT_LOG} ${MO_LINT_LOG}

exit ${EXIT_CODE}

Revert a merge commit

Because they have two ancestors, you can't revert merge commits with a simple git revert HASH. You need to specify the ancestor to revert to. For instance, imagine the graph below:

* c7545ec - Tue, 26 Mar 2019 16:52:39 +0100 (2 minutes ago)
|           Add titi - Julien Enselme
*   df849a2 - Tue, 26 Mar 2019 16:51:53 +0100 (3 minutes ago)
|\            Merge branch 'tmp' - Julien Enselme
| * a23acae - Tue, 26 Mar 2019 16:51:47 +0100 (3 minutes ago) (tmp)
|/            Add tata - Julien Enselme
* 8be0cb2 - Tue, 26 Mar 2019 16:51:34 +0100 (3 minutes ago)
|           Add toto - Julien Enselme
* 75182c4 - Tue, 26 Mar 2019 16:51:24 +0100 (3 minutes ago)
            “root” - Julien Enselme

The merge commit (df849a2) has two ancestors:

a23acae which comes from a feature branch.
8be0cb2 which comes from the main branch.

To cancel it and cancel the commits from the feature branch, run git revert df849a2 -m 1.

Connect to a running docker container

2015-03-25T00:00:00+01:00

dk-enter() {
    docker exec -it "$1" "${2:-/bin/bash}"
}

Otherwies, you can easily open a shell in a running docker container with the following commands:

Get the container pid: docker inspect -f {{.State.Pid}} <container-id>
Connect: nsenter --target <pid> --mount --uts --ipc --net --pid

VirtualBox

2015-03-01T00:00:00+01:00

Sommaire

Transform img disk (qemu) to vdi (Virtual Box)
Transform vdi to img
Compress an image
Resize a disk
Mount a VDI to the local FS

Transform img disk (qemu) to vdi (Virtual Box)

You need to convert img in raw with qemu-img and then raw to vdi. It will take time.

$ qemu-img convert WindowsXP.img -O raw WindowsXP.raw
$ VBoxManage convertdd WindowsXP.raw $HOME/.VirtualBox/VDI/WindowsXP.vdi

Transform vdi to img

$ VBoxManage clonehd sdb.vdi sdb.raw --format RAW
$ qemu-img convert sdb.raw -O img sdb.img

Compress an image

Prepare Windows guest

run defrag in the guest (Windows)
nullify free space: sdelete –z

Prepare Linux guest

sudo dd if=/dev/zero of=/bigemptyfile bs=4096k
sudo rm -rf /bigemptyfile

Compess

shutdown the guest VM
Run VBoxManage's compact command: VBoxManage.exe modifyhd thedisk.vdi --compact

Resize a disk

vboxmanage modifyhd /home/CHEMIN/VIRTUALBOX\VOTRE.vdi –resize 80000 # 80 Go HDD

Mount a VDI to the local FS

Run these commands as root:

Enable the proper kernel module: modprobe nbd
Create a virtual device with at /dev/nbd0 with qemu-nbd -c /dev/nbd0 /PATH/TO/YOUR/VDI/disk.vdi
List the partitions on the disk with parted /dev/nbd0 print
Mount the partition in /mnt/vdi (eg if you want partition 2):
1. mkdir /mnt/vdi
2. mount /dev/nbd0p2 /mnt/vdi/
Extract the data you need by navigating under /mnt/vdi.
Unmount the volume: umount /dev/nbd0p2
Disable the virtual drive with qemu-nbd --disconnect /dev/nbd0

Note: If you want to mount a LVM volume, you will get an mount: /mnt/vdi: unknown filesystem type 'LVM2_member'. error. You need to do before instead of the mount step:

Identify the LVM volume with lvscan. You should see a path to the LVM volume to mount. /dev/fedora/root in may case with this output:
```
ACTIVE            '/dev/fedora/swap' [3.00 GiB] inherit
ACTIVE            '/dev/fedora/root' [<26.00 GiB] inherit
```
You can then mount it with mount /dev/fedora/root /mnt/vdi/
To unmount, use umount /dev/fedora/root
Disable the virtual drive normally.

Sources:

Fusionner deux dépôts git

2015-01-17T00:00:00+01:00

Récemment, j'ai eu besoin de fusionner deux dépôts en un seul. Ça se fait plutôt bien. Voilà la procédure :

# On clone les dépots
git clone projet1
git clone projet2
cd projet1

# On ajoute le projet2 comme source dans le projet1
git remote add projet2 ../projet2
git fetch projet2

# On liste les branches du dépôt 2
git branch -a

# On crée dans le dépôt 1 la branche à fusionner (par exemple master)
git branch projet2-master projet2/master

# On fusionne
git merge projet2-master

Surtout ne pas tentez de faire directement un git pull projet2

Si vous avez des sous-modules, réinitialisez les avec :

git submodule update --init

L'article (en anglais) qui m'a bien aidé : http://blog.caplin.com/2013/09/18/merging-two-git-repositories/

Importer périodiquement un flux ical dans owncloud

2015-01-09T00:00:00+01:00

J'ai toujours trouvé dommage que owncloud ne soit pas capable d'importer périodiquement des calendriers que l'on trouve sur le web. Dans mon cas, mon emploi du temps est un flux ical régulièrement mis à jour et j'aimerais bien que owncloud soit capable de l'importer. Je me suis finalement résigné à faire mon propre script pour régler ce problème.

Je pensais initialement le faire en Bash et utiliser curl pour faire les requêtes vers owncloud. Cependant, un flux ical peut contenir des sauts de lignes et les tenter de gérer correctement m'a fait souffrir. Du coup, je me suis rabattu sur python et son excellente bibliothèque requests.

Concrètement, ce script :

lit dans ~/.owncloud l'adresse et les identifiants du serveur owncloud,
récupère les événements définis dans les flux icals que je lui demande de synchroniser. Ces événements sont stockés dans une instance de la classe Vevent crée à cet effet. J'ai ajouté la possibilité de filtrer certains événements (pour exclure des cours que je n'ai pas),
supprime les événements présents dans owncloud mais absent des flux icals,
poste (crée ou met à jour) les événements.

Il ne reste plus qu'à mettre le script dans un cron pour faire l'import périodiquement.

Normalement le script est assez simple et clair pour être facilement compréhensible. En cas de problème, postez un commentaire.

Vous pouvez aussi télécharger ce script et le voir sur le dépôt mercurial.

  1 #!/usr/bin/python3
  2 
  3 import requests
  4 from requests.auth import HTTPBasicAuth
  5 import re
  6 import xml.etree.ElementTree as ET
  7 
  8 
  9 class Vevent:
 10     UID_REGEXP = re.compile('^UID:.*')
 11     SUMMARY_REGEXP = re.compile('^SUMMARY:.*')
 12 
 13     def __init__(self, lines):
 14         self._lines = lines
 15         self._uid = self._get_line_from_regexp(self.UID_REGEXP)
 16         self._summary = self._get_line_from_regexp(self.SUMMARY_REGEXP)
 17 
 18     def _get_line_from_regexp(self, regexp):
 19         for line in self._lines:
 20             if regexp.match(line):
 21                 index = self._lines.index(line)
 22                 return line + self._lines[index + 1]
 23 
 24     def get_vevent_for_put(self):
 25         return '\r\n'.join(self._lines)
 26 
 27     def get_as_vcal_for_put(self):
 28         str_vevent = 'BEGIN:VCALENDAR\r\n' + self.get_vevent_for_put() \
 29                      + '\r\nEND:VCALENDAR'
 30         # If you experience problem with the line below, try
 31         # return strvevent
 32         # instead
 33         return str_vevent.encode('utf-8')
 34 
 35     @property
 36     def uid(self):
 37         return self._uid
 38 
 39     @property
 40     def summary(self):
 41         return self._summary
 42 
 43     def __repr__(self):
 44         return '{}\n{}'.format(self._uid, self._summary)
 45 
 46 
 47 def get_login():
 48     owncloud_omis_url = ''
 49     login = ''
 50     password = ''
 51     with open('/home/jenselme/.owncloud') as owncloud:
 52         owncloud_omis_url, login, password = [line for line in
 53                                               owncloud.read().split('\n') if line]
 54     return owncloud_omis_url, login, password
 55 
 56 def fetch_all_vevents(urls, filter_dict):
 57     vevents = []
 58     for name, url in urls.items():
 59         current_vevents = get_vevents(url)
 60         if name in filter_dict:
 61             current_vevents = [vevent for vevent in current_vevents
 62                                if filter_dict[name].match(vevent.summary)]
 63         vevents.extend(current_vevents)
 64     return vevents
 65 
 66 def get_vevents(get_url):
 67     calendar = requests.get(get_url).content.decode('utf-8')
 68     calendar_lines = [line for line in calendar.split('\r\n') if line]
 69     # Remove VCALENDAR lines
 70     del calendar_lines[0]
 71     del calendar_lines[-1]
 72 
 73     vevent_lines = []
 74     vevents = []
 75     for line in calendar_lines:
 76         vevent_lines.append(line)
 77         if VEVENT_END.match(line) and vevent_lines:
 78             vevents.append(Vevent(vevent_lines))
 79             vevent_lines = []
 80     return vevents
 81 
 82 def delete_all_removed_vevents(fetch_vevents, destination_url, request_params):
 83     destination_uids = get_destination_calendar_uids(destination_url, request_params)
 84     source_uids = [vevent.uid for vevent in fetch_vevents]
 85     removed_from_source_uids = [uid for uid in destination_uids if uid not in source_uids]
 86     delete_all(removed_from_source_uids, destination_url, request_params)
 87 
 88 def get_destination_calendar_uids(destination_url, request_params):
 89     resp = requests.request('PROPFIND', destination_url, **request_params)
 90     xml_str = resp.content.decode('utf-8')
 91     root = ET.fromstring(xml_str)
 92     hrefs_uid = []
 93     for response in root:
 94         for child in response:
 95             if child.tag == '{DAV:}href':
 96                 hrefs_uid.append(child.text)
 97     uids = []
 98     for href in hrefs_uid:
 99         uid = href.split('/')[-1]
100         if uid:
101             uids.append(uid)
102     return uids
103 
104 def delete_all(uids, url, request_params):
105     for uid in uids:
106         delete_url = '{}/{}'.format(url, uid)
107         requests.delete(delete_url, **request_params)
108 
109 def put_all_vevents_as_vcalendars(vevents, calendar_put_url, request_params):
110     for vevent in vevents:
111         put_url = '{}/{}.ics'.format(calendar_put_url, vevent.uid)
112         data = vevent.get_as_vcal_for_put()
113         requests.put(put_url, data=data, **request_params)
114 
115 
116 if __name__ == '__main__':
117     # Global variables
118     VEVENT_END = re.compile('^END:VEVENT$')
119     SOCIOLOGIE_ORGANISATION = re.compile('.*Sociologie des Organisations.*')
120     EDT_FETCH_URLS = {'omis-org': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=804&projectId=15&calType=ical&firstDate=2015-01-06&lastDate=2015-06-30',
121     'tc-electif': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=1043&projectId=15&calType=ical&firstDate=2014-11-24&lastDate=2015-04-30',
122     'ENT': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=208&projectId=15&calType=ical&firstDate=2014-10-20&lastDate=2015-05-31',
123     'omis-gB': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=507&projectId=15&calType=ical&firstDate=2014-09-15&lastDate=2015-05-31',
124     'tc2-td3': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=194&projectId=15&calType=ical&firstDate=2014-09-01&lastDate=2015-06-30',
125     'omis-i': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=814&projectId=15&calType=ical&firstDate=2014-09-01&lastDate=2015-05-31'}
126 
127     owncloud_omis_url, login, password = get_login()
128 
129     request_params = {'verify': False, 'auth': HTTPBasicAuth(login, password)}
130     fetched_vevents = fetch_all_vevents(EDT_FETCH_URLS, {'omis-org': SOCIOLOGIE_ORGANISATION})
131     delete_all_removed_vevents(fetched_vevents, owncloud_omis_url, request_params)
132     put_all_vevents_as_vcalendars(fetched_vevents, owncloud_omis_url, request_params)

Un shell pour pelican

2015-01-09T00:00:00+01:00

Comme je l'ai déjà dit ici, mon blog est géré par Pelican. Il est également placé sous gestion de version. Pour régénérer automatiquement après chaque modification la version html des pages, il faut utiliser la commande make regenerate. Pour lancer le serveur de test pour voir les pages, on utilise make serve. Enfin, pour synchroniser le contenu sur le serveur, j'utilise make rsync_upload. J'en profite pour pousser les modifications de mon dépôt local vers mon serveur et vers Bitbucket pour en avoir une copie de sauvegarde.

Jusqu'à présent, je gérais tout ça à la main, dans un terminal. J'avais déjà écrit une petite fonction bash pour me déplacer dans le bon dossier et activer le venv python. Mais ce n'était pas très pratique : les commandes make serve et make regenerate doivent tourner en permanence en tâche de fond, il faut pusher sur deux dépôts différents et synchroniser les pages sur le serveur. J'ai donc décider de me créer un script pour améliorer tout ça.

Comme j'ai besoin de lancer plusieurs commandes bien définies dans un même dossier, sans sortir de l'environnement virtuel et que je veux pouvoir interagir facilement avec les processus en tâche de fond, j'ai décidé de créer un mini-shell. Je dois admettre que j'en suis très satisfait.

Quelques explications sur le script :

Tout est dans une boucle infinie pour imiter un vrai shell. echo -ne "> " permet d'afficher un prompt et de laisser le curseur sur cette ligne.
exec 3< <(make serve)
- La syntaxe <(make serve) est remplacée par chemin vers un fichier virtuel qui est connecté à la sortie standard de la commande située entre les parenthèses. Ici make serve qui démarre le serveur.
- Comme on veut que le processus soit en background, on lance la commande avec exec.
- Enfin, comme on veut pouvoir récupérer facilement la sortie de la commande, on envoie le contenu du fichier vers une nouvelle sortie 3. On lit son contenu avec cat <&3. Tant que le fichier n'est pas fermé, cette commande ne se retourne pas.
- Voir cette réponse sur stackoverflow pour plus de détails.
Sorties additionnelles : j'ai tenté de stocker le numéro des sorties dans des variables. Le code est plus clair et cela permet de le factoriser. Malheureusement, exec "${var}"< <(make serve) provoque une erreur. Idem pour cat <&"${var}".
[ -n "${serve_pid}" ] && has_died "${serve_pid}" : la fonction has_died ne doit pas être entre crochets (donc passée en argument à la fonction test) pour que le script fonctionne correctement.
echo ${pid} : on ne peut pas renvoyer autre chose que des codes d'erreur avec return. Pour tout le reste, il faut utiliser echo.

Ci-dessous le script complet. Vous pouvez aussi le télécharger ou le voir sur github pour avoir la version la plus à jour.

  1 #!/usr/bin/bash
  2 
  3 stop_command() {
  4     if [ "$1" ]; then
  5         kill "$1"
  6     fi
  7 }
  8 
  9 has_died() {
 10     if kill -0 "$1" > /dev/null 2>&1; then
 11         return 1
 12     else
 13         return 0
 14     fi
 15 }
 16 
 17 find_pid() {
 18     echo $(ps -elf | grep "$1" | grep -v '&&' | grep -v 'grep' | awk '{print $4}')
 19 }
 20 
 21 get_pid() {
 22     pid=$(find_pid "$1")
 23     until [ -n "${pid}" ]; do
 24         pid=$(find_pid "$1")
 25     done
 26     echo "${pid}"
 27 }
 28 
 29 # Activate venv
 30 cd ~/server/blog/
 31 source bin/activate
 32 cd pelican/jujens.eu
 33 
 34 regenerate_pid=''
 35 serve_pid=''
 36 # These variables cannot be used. If you try to use them, you will get an error like 3< not found.
 37 #regenerate_output=3
 38 #serve_output=4
 39 
 40 while true; do
 41     # Print errors for regenerate
 42     if [ -n "${regenerate_pid}" ] && has_died "${regenerate_pid}"; then
 43         echo -e "Regenerate has died with ouput:\n"
 44         cat <&3
 45     fi
 46     # Print errors for serve
 47     if [ -n "${serve_pid}" ] && has_died "${serve_pid}"; then
 48         echo -e "Serve has died with ouput:\n"
 49         cat <&4
 50     fi
 51 
 52     echo -en "(blog) > "
 53     read command
 54 
 55     case "${command}" in
 56         deploy)
 57             hg push > /dev/null
 58             hg push bitbucket >/dev/null
 59             stop_command "${serve_pid}"
 60             stop_command "${regenerate_pid}"
 61             serve_pid=''
 62             regenerate_pid=''
 63             cat <&3 > /dev/null 2>&1
 64             cat <&4 > /dev/null 2>&1
 65             make rsync_upload > /dev/null
 66             ;;
 67         push)
 68             hg push > /dev/null
 69             hg push bitbucket > /dev/null
 70             ;;
 71         st|status)
 72             hg st
 73             ;;
 74         add)
 75             echo "Enter the filename to add (. for all files)"
 76             read file_name
 77             hg add "${file_name}"
 78             ;;
 79         ci|commit)
 80             echo "Please enter the commit message:"
 81             read commit_msg
 82             hg ci -m "${commit_msg}"
 83             ;;
 84         serve)
 85             if [ -n "${serve_pid}" ] && ! has_died "${serve_pid}"; then
 86                 echo "Serve is already running."
 87             else
 88                 exec 4< <(make serve 2>&1)
 89                 serve_pid=$(get_pid 'python3 -m pelican.server')
 90             fi
 91             ;;
 92         regenerate)
 93             if [ -n "${regenerate_pid}" ] && ! has_died "${regenerate_pid}"; then
 94                 echo "Regenerate is already running."
 95             else
 96                 exec 3< <(make regenerate 2>&1)
 97                 regenerate_pid=$(get_pid 'make regenerate')
 98             fi
 99             ;;
100         "stop serve")
101             stop_command "${serve_pid}" > /dev/null
102             serve_pid=''
103             cat <&4 > /dev/null
104             ;;
105         "stop regenerate")
106             stop_command "${regenerate_pid}" > /dev/null
107             regenerate_pid=''
108             cat <&3 > /dev/null
109             ;;
110         stop)
111             echo "Stop requires an argument: serve or regenerate"
112             ;;
113         quit)
114             break
115             ;;
116         help)
117             echo "Available commands:"
118             echo -e "\tdeploy"
119             echo -e "\tpush"
120             echo -e "\tserve"
121             echo -e "\tregenerate"
122             echo -e "\tstop serve"
123             echo -e "\tstop regenerate"
124             echo -e "\thelp"
125             ;;
126         *)
127             if [ -n "${command}" ]; then
128                 echo -e "${command} is invalid."
129             fi
130     esac
131 done
132 
133 echo "Done"

Utiliser trap en bash pour pièger des signaux

2015-01-09T00:00:00+01:00

Mise à jour 2017-09-96: ajout d'un exemple d'utilisation des noms des signaux. Merci à Thomas et Lokta.

Bash est capable d'intercepter les signaux envoyés par certains raccourcis claviers (comme Ctrl-C) et de changer le comportement par défaut de ces raccourcis. Il suffit pour cela d'utiliser la commande trap. Elle prend en premier argument la commande à exécuter puis les signaux sur lesquels elle doit réagir.

Par exemple :

trap "echo yollo" 2 3

affichera yollo quand on fait Ctrl-C.

On obtient la liste des signaux avec leurs numéros et leurs noms avec la commande kill.

kill -l
1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL
5) SIGTRAP      6) SIGABRT      7) SIGEMT       8) SIGFPE
9)SIGKILL     10) SIGBUS      11) SIGSEGV     12) SIGSYS

Pour utiliser les noms des signaux, utiliser:

trap "echo 'sig INT caught' " INT

Periodically import an ical calendar into owncloud

2015-01-09T00:00:00+01:00

I always thought that owncloud should be able to periodically import calendars found on the web. In my case, I have a schedule as an ical feed I would like synchronize with my owncloud. I finally wrote a script to do that.

Initially, I intended to do it in Bash and to use curl to do requests to owncloud. However, an ical feed contains line breaks. I had a lot of problems to manage them correctly. So I wrote my script in python the excellent requests library.

This script :

reads in ~/.owncloud the address and the ids of the owncloud server,
fetches the events described in the ical feed I ask it to synchronize. Theses events are stored in an instance of Vevent. I had the possibility to filter out some events,
deletes the events present in my owncloud but not in the ical feeds,
put (create or update) the events in owncloud.

I think this script is simple and clear enough to be easily read. In case of troubles, you can post a comment.

You can also download this script and see it on my mercurial on bitbucket.

  1 #!/usr/bin/python3
  2 
  3 import requests
  4 from requests.auth import HTTPBasicAuth
  5 import re
  6 import xml.etree.ElementTree as ET
  7 
  8 
  9 class Vevent:
 10     UID_REGEXP = re.compile('^UID:.*')
 11     SUMMARY_REGEXP = re.compile('^SUMMARY:.*')
 12 
 13     def __init__(self, lines):
 14         self._lines = lines
 15         self._uid = self._get_line_from_regexp(self.UID_REGEXP)
 16         self._summary = self._get_line_from_regexp(self.SUMMARY_REGEXP)
 17 
 18     def _get_line_from_regexp(self, regexp):
 19         for line in self._lines:
 20             if regexp.match(line):
 21                 index = self._lines.index(line)
 22                 return line + self._lines[index + 1]
 23 
 24     def get_vevent_for_put(self):
 25         return '\r\n'.join(self._lines)
 26 
 27     def get_as_vcal_for_put(self):
 28         str_vevent = 'BEGIN:VCALENDAR\r\n' + self.get_vevent_for_put() \
 29                      + '\r\nEND:VCALENDAR'
 30         # If you experience problem with the line below, try
 31         # return strvevent
 32         # instead
 33         return str_vevent.encode('utf-8')
 34 
 35     @property
 36     def uid(self):
 37         return self._uid
 38 
 39     @property
 40     def summary(self):
 41         return self._summary
 42 
 43     def __repr__(self):
 44         return '{}\n{}'.format(self._uid, self._summary)
 45 
 46 
 47 def get_login():
 48     owncloud_omis_url = ''
 49     login = ''
 50     password = ''
 51     with open('/home/jenselme/.owncloud') as owncloud:
 52         owncloud_omis_url, login, password = [line for line in
 53                                               owncloud.read().split('\n') if line]
 54     return owncloud_omis_url, login, password
 55 
 56 def fetch_all_vevents(urls, filter_dict):
 57     vevents = []
 58     for name, url in urls.items():
 59         current_vevents = get_vevents(url)
 60         if name in filter_dict:
 61             current_vevents = [vevent for vevent in current_vevents
 62                                if filter_dict[name].match(vevent.summary)]
 63         vevents.extend(current_vevents)
 64     return vevents
 65 
 66 def get_vevents(get_url):
 67     calendar = requests.get(get_url).content.decode('utf-8')
 68     calendar_lines = [line for line in calendar.split('\r\n') if line]
 69     # Remove VCALENDAR lines
 70     del calendar_lines[0]
 71     del calendar_lines[-1]
 72 
 73     vevent_lines = []
 74     vevents = []
 75     for line in calendar_lines:
 76         vevent_lines.append(line)
 77         if VEVENT_END.match(line) and vevent_lines:
 78             vevents.append(Vevent(vevent_lines))
 79             vevent_lines = []
 80     return vevents
 81 
 82 def delete_all_removed_vevents(fetch_vevents, destination_url, request_params):
 83     destination_uids = get_destination_calendar_uids(destination_url, request_params)
 84     source_uids = [vevent.uid for vevent in fetch_vevents]
 85     removed_from_source_uids = [uid for uid in destination_uids if uid not in source_uids]
 86     delete_all(removed_from_source_uids, destination_url, request_params)
 87 
 88 def get_destination_calendar_uids(destination_url, request_params):
 89     resp = requests.request('PROPFIND', destination_url, **request_params)
 90     xml_str = resp.content.decode('utf-8')
 91     root = ET.fromstring(xml_str)
 92     hrefs_uid = []
 93     for response in root:
 94         for child in response:
 95             if child.tag == '{DAV:}href':
 96                 hrefs_uid.append(child.text)
 97     uids = []
 98     for href in hrefs_uid:
 99         uid = href.split('/')[-1]
100         if uid:
101             uids.append(uid)
102     return uids
103 
104 def delete_all(uids, url, request_params):
105     for uid in uids:
106         delete_url = '{}/{}'.format(url, uid)
107         requests.delete(delete_url, **request_params)
108 
109 def put_all_vevents_as_vcalendars(vevents, calendar_put_url, request_params):
110     for vevent in vevents:
111         put_url = '{}/{}.ics'.format(calendar_put_url, vevent.uid)
112         data = vevent.get_as_vcal_for_put()
113         requests.put(put_url, data=data, **request_params)
114 
115 
116 if __name__ == '__main__':
117     # Global variables
118     VEVENT_END = re.compile('^END:VEVENT$')
119     SOCIOLOGIE_ORGANISATION = re.compile('.*Sociologie des Organisations.*')
120     EDT_FETCH_URLS = {'omis-org': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=804&projectId=15&calType=ical&firstDate=2015-01-06&lastDate=2015-06-30',
121     'tc-electif': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=1043&projectId=15&calType=ical&firstDate=2014-11-24&lastDate=2015-04-30',
122     'ENT': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=208&projectId=15&calType=ical&firstDate=2014-10-20&lastDate=2015-05-31',
123     'omis-gB': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=507&projectId=15&calType=ical&firstDate=2014-09-15&lastDate=2015-05-31',
124     'tc2-td3': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=194&projectId=15&calType=ical&firstDate=2014-09-01&lastDate=2015-06-30',
125     'omis-i': 'https://ade6.centrale-marseille.fr/jsp/custom/modules/plannings/anonymous_cal.jsp?resources=814&projectId=15&calType=ical&firstDate=2014-09-01&lastDate=2015-05-31'}
126 
127     owncloud_omis_url, login, password = get_login()
128 
129     request_params = {'verify': False, 'auth': HTTPBasicAuth(login, password)}
130     fetched_vevents = fetch_all_vevents(EDT_FETCH_URLS, {'omis-org': SOCIOLOGIE_ORGANISATION})
131     delete_all_removed_vevents(fetched_vevents, owncloud_omis_url, request_params)
132     put_all_vevents_as_vcalendars(fetched_vevents, owncloud_omis_url, request_params)

A basic shell for pelican

2015-01-09T00:00:00+01:00

My blog uses Pelican a static website generator. All my articles are under version control. To generate after each modification the new html version of the pages, you must use the make regenerate. To launch the test web server, you use make serve. Finally to synchronize the content on the server, you use make rsync_upload. Each time I do this, I also push my modifications to my server and to Bitbucket in order to have a backup.

Until now, I managed all that in a standard terminal. I already had a bash function to switch to the proper directory and enable python's venv. However, I did not find that very practical. make serve and make regenerate must always stay in background, I have to push to two different repositories and synchronize with the server. So I decided to create a script to improve that.

Since I need to launch the same commands in the same folder without leaving the venv and since I want to easily interact with the background commands, I decided to create a small shell. I am very happy with it.

Some explanations on the script (available below):

Everything is in an infinite loop to simulate a true shell. echo -ne "> " print a prompt and let the cursor on the same line.
exec 3< <(make serve)
- <(make serve) is replaced by a path to a virtual file connected to the standard output of the command.
- Since we want the process to be in background, we launch it with exec.
- Finally, since we want to fetch the output of the command, we connect the file to a new output named 3. We then read its content with cat <&3. While the file is not closed, this command won't stop.
- Got to this thread for more details.
Additional output: I tried to store the name of the ouputs in variables. That would have allowed me to refactor and clarify the code. However, that makes exec "${var}"< <(make serve) and cat <&"${var}" crash.
[ -n "${serve_pid}" ] && has_died "${serve_pid}": has_died must not be surrounded by brackets (that means it must not be pass to the test command) in order for the script to work properly.
echo ${pid}: you can only send error code with return. For everything else, you must use echo.

You will find the whole script below. You can also download it ou le see it on github if you want the latest version

  1 #!/usr/bin/bash
  2 
  3 stop_command() {
  4     if [ "$1" ]; then
  5         kill "$1"
  6     fi
  7 }
  8 
  9 has_died() {
 10     if kill -0 "$1" > /dev/null 2>&1; then
 11         return 1
 12     else
 13         return 0
 14     fi
 15 }
 16 
 17 find_pid() {
 18     echo $(ps -elf | grep "$1" | grep -v '&&' | grep -v 'grep' | awk '{print $4}')
 19 }
 20 
 21 get_pid() {
 22     pid=$(find_pid "$1")
 23     until [ -n "${pid}" ]; do
 24         pid=$(find_pid "$1")
 25     done
 26     echo "${pid}"
 27 }
 28 
 29 # Activate venv
 30 cd ~/server/blog/
 31 source bin/activate
 32 cd pelican/jujens.eu
 33 
 34 regenerate_pid=''
 35 serve_pid=''
 36 # These variables cannot be used. If you try to use them, you will get an error like 3< not found.
 37 #regenerate_output=3
 38 #serve_output=4
 39 
 40 while true; do
 41     # Print errors for regenerate
 42     if [ -n "${regenerate_pid}" ] && has_died "${regenerate_pid}"; then
 43         echo -e "Regenerate has died with ouput:\n"
 44         cat <&3
 45     fi
 46     # Print errors for serve
 47     if [ -n "${serve_pid}" ] && has_died "${serve_pid}"; then
 48         echo -e "Serve has died with ouput:\n"
 49         cat <&4
 50     fi
 51 
 52     echo -en "(blog) > "
 53     read command
 54 
 55     case "${command}" in
 56         deploy)
 57             hg push > /dev/null
 58             hg push bitbucket >/dev/null
 59             stop_command "${serve_pid}"
 60             stop_command "${regenerate_pid}"
 61             serve_pid=''
 62             regenerate_pid=''
 63             cat <&3 > /dev/null 2>&1
 64             cat <&4 > /dev/null 2>&1
 65             make rsync_upload > /dev/null
 66             ;;
 67         push)
 68             hg push > /dev/null
 69             hg push bitbucket > /dev/null
 70             ;;
 71         st|status)
 72             hg st
 73             ;;
 74         add)
 75             echo "Enter the filename to add (. for all files)"
 76             read file_name
 77             hg add "${file_name}"
 78             ;;
 79         ci|commit)
 80             echo "Please enter the commit message:"
 81             read commit_msg
 82             hg ci -m "${commit_msg}"
 83             ;;
 84         serve)
 85             if [ -n "${serve_pid}" ] && ! has_died "${serve_pid}"; then
 86                 echo "Serve is already running."
 87             else
 88                 exec 4< <(make serve 2>&1)
 89                 serve_pid=$(get_pid 'python3 -m pelican.server')
 90             fi
 91             ;;
 92         regenerate)
 93             if [ -n "${regenerate_pid}" ] && ! has_died "${regenerate_pid}"; then
 94                 echo "Regenerate is already running."
 95             else
 96                 exec 3< <(make regenerate 2>&1)
 97                 regenerate_pid=$(get_pid 'make regenerate')
 98             fi
 99             ;;
100         "stop serve")
101             stop_command "${serve_pid}" > /dev/null
102             serve_pid=''
103             cat <&4 > /dev/null
104             ;;
105         "stop regenerate")
106             stop_command "${regenerate_pid}" > /dev/null
107             regenerate_pid=''
108             cat <&3 > /dev/null
109             ;;
110         stop)
111             echo "Stop requires an argument: serve or regenerate"
112             ;;
113         quit)
114             break
115             ;;
116         help)
117             echo "Available commands:"
118             echo -e "\tdeploy"
119             echo -e "\tpush"
120             echo -e "\tserve"
121             echo -e "\tregenerate"
122             echo -e "\tstop serve"
123             echo -e "\tstop regenerate"
124             echo -e "\thelp"
125             ;;
126         *)
127             if [ -n "${command}" ]; then
128                 echo -e "${command} is invalid."
129             fi
130     esac
131 done
132 
133 echo "Done"

Use trap in Bash to trap signals

2015-01-09T00:00:00+01:00

Update 2017-09-96: add example to use the name of the signals. Thanks to Thomas and Lokta.

Bash can intercept signals send by some special keys like Ctrl-C and change their default behaviour. In order to do that, you must use the trap command. This command take as argument the command you want to execute when the signal is emitted and the list of signals you want to trap.

For instance:

trap "echo yollo" 2 3

will print yollo when Ctrl-C is pressed.

To get the list of all signals, with their numbers and their names, you can use kill.

kill -l
1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL
5) SIGTRAP      6) SIGABRT      7) SIGEMT       8) SIGFPE
9)SIGKILL     10) SIGBUS      11) SIGSEGV     12) SIGSYS

To use the names of the signal instead, use:

trap "echo 'sig INT caught' " INT

You can also use it to kill the children of a process with:

trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT

Utiliser un service webdav avec curl

2015-01-07T00:00:00+01:00

Il est tout à fait possible (et c'est même facile) d'utiliser curl pour communiquer avec un service webdav. Attention toute fois, tous les événements sont contenus dans des fichiers séparés qui doivent contenir les lignes BEGIN:VCALENDAR et END:VCALENDAR en plus des lignes décrivant l'évènement en lui même.

Pour recevoir la liste au format xml des événements, il suffit de faire :

curl -k --user "${login}:${password}" -X PROPFIND "https://owncloud.jujens.eu/remote.php/caldav/calendars/jujens/defaultcalendar"

Pour poster un nouvel événement ou mettre à jour un événement existant, il suffit de faire une requête put à l'url du service complétée par un nom de fichier. Si le fichier existe, on met à jour l'événement correspondant. Sinon on en crée un nouveau.

curl -k --user "${login}:${password}" -X PUT -H "Content-Type: text/calendar; charset=utf-8" -d "$a" "https://owncloud.jujens.eu/remote.php/caldav/calendars/jujens/omis/<mon-fichier.ics>"

Idem pour la suppression :

curl -k --user "${login}:${password}" -X DELETE -H "Content-Type: text/calendar; charset=utf-8" "https://owncloud.jujens.eu/remote.php/caldav/calendars/jujens/omis/<mon-fichier.ics>"

Use a webdav service with curl

2015-01-07T00:00:00+01:00

It is really easy to use curl to communicate with a webdav service. You just have to be aware that all events are contained in separate files and that the lines describing the event must be surrounded by BEGIN:VCALENDAR and END:VCALENDAR lines.

To get the list of all events as an xml answer :

curl -k --user "${login}:${password}" -X PROPFIND "https://owncloud.jujens.eu/remote.php/caldav/calendars/jujens/defaultcalendar"

To post a new event or to update an existing event, simply do a put request to your service url concatenated with a file name. If the file name is not already used, you create a new event. Otherwise, you update the given event.

curl -k --user "${login}:${password}" -X PUT -H "Content-Type: text/calendar; charset=utf-8" -d "$a" "https://owncloud.jujens.eu/remote.php/caldav/calendars/jujens/omis/<mon-fichier.ics>"

It is the same for the deletion (with a delete request):

curl -k --user "${login}:${password}" -X DELETE -H "Content-Type: text/calendar; charset=utf-8" "https://owncloud.jujens.eu/remote.php/caldav/calendars/jujens/omis/<mon-fichier.ics>"

Des modèles de business model canvas en français pour LibreOffice Impress

2014-11-22T00:00:00+01:00

Je viens de traduire en français un modèle de business modèle canvas pour LibreOffice Impress. Il est au format A0. Pour l'inclure dans un autre document, je vous conseille de l'exporter au format SVG puis le l'inclure en tant qu'image. J'ai tenté d'utiliser l'import OLE mais le document est toujours trop gros.

Les modèles dont le nom de fichier contient edit ne contiennent pas les descriptifs des sections et sont donc prêts à être remplis par vos soins.

Le modèle original en anglais vient de http://thoughtsof.matthiasjuette.de/business-model-canvas-template/. Ils sont tous publiés sous la licence Creative Commons Share Alike.

Utiliser AngularJS et Django pour faire des requêtes AJAX

2014-10-25T00:00:00+02:00

Dans ce petit tuto, je vais vous expliquer comment faire pour poster un formulaire Django via une requête AJAX en utilisant le framework JavaScript AngularJS. Tout d'abord, il faut un peu de préparation. Voici le modèle que nous allons utiliser (les imports ne sont pas précisés et je suppose que vous savez où mettre ces fichiers) :

class Card(models.Model):
    content = models.TextField()
    date = models.DateTimeField(auto_now_add=True, auto_now=True)
    category = models.ForeignKey('Category')

    def __str__(self):
        return self.content

Le formulaire :

class CardForm(forms.ModelForm):
    class Meta:
        model = Card

Le code de la vue :

def add_card(request):
saved = False

if request.method == 'POST':
    form = CardForm(request.POST, request.FILES)
    if form.is_valid():
        card = Card()
        card.content = form.cleaned_data['content']
        card.category = form.cleaned_data['category']
        saved = True
        card.save()
else:
    form = CardForm()

return render(request, 'board/add_card.html', locals())

Et enfin le template :

<!DOCTYPE html>
{% load staticfiles %}
<html ng-app="ajax">
<head>
  <meta charset="UTF-8" />
   <script src={% static "angular.js" %} type="text/javascript" ></script>
   <script type="text/javascript" src={% static "angular-cookies.js" %}></script>
   <script src={% static "controllers.js" %} type="text/javascript" ></script>
 </head>
 <body>
<form action="{% url "board.views.add_card" %}" ng-controller="MyFormCtrl"
       method="post" name="form">{% csrf_token %}
       <p>
         <label for="id_content">Content:</label>
         <textarea cols="40" id="id_content" name="content" rows="10" ng-model="card.content"></textarea>
       </p>
         <p><label for="id_category">Category:</label> <select ng-model="card.category" id="id_category" name="category">
         <option value="" selected="selected">---------</option>
         <option value="1">Backlog</option>
         <option value="2">Done</option>
         <option value="3">In progress</option>
         </select>
         <button ng-click="submit($event)">Submit</button>
       </p>
 </form>
 </body>
 </html>

Comme toute application Angular, il contient la balise ng-app pour signaler au framework qu'il s'agit une portion de HTML géré par Angular et une balise ng-controller pour faire le lien avec le contrôleur de Angular. J'ai également ajouté {% load staticfiles %} pour utiliser la directive static de django.

Le formulaire doit être présent en intégralité dans le template (on ne peut pas utiliser {{ form.as_p }}) pour pouvoir faire correctement les liens avec la partie Angular :

L'aire de texte avec ng-model="card.content".
La liste d'option avec ng-model="card.category".
Le bouton de soumission avec ng-click="submit($event)". Le $event est nécessaire pour éviter le rechargement de la page (voir le code JavaScript).

Je vous laisse vous renseigner sur le fonctionnement d'Angular si vous n'êtes pas familier avec celui-ci.

Et pour finir, le code Javascript.

var nameSpace = angular.module("trello", ['ngCookies']);

nameSpace.controller("MyFormCtrl", ['$scope', '$http', '$cookies',
function ($scope, $http, $cookies) {
    $http.defaults.headers.post['Content-Type'] = 'application/x-www-form-urlencoded';
    // Pour envoyer le code csrf.
    $http.defaults.headers.post['X-CSRFToken'] = $cookies.csrftoken;

    // Cette fonction est appelée quand on soumet le formulaire.
    $scope.submit = function ($event) {
        // On évite le rechargement de la page.
        $event.preventDefault();
        // On prépare l'envoie des données.
        var in_data = jQuery.param({'content': $scope.card.content, 'category': $scope.card.category, 'csrfmiddlewaretoken': $cookies.csrftoken});
        $http.post('/add', in_data)
          .success(function(out_data) {
            // Remet le formulaire à zéro si tout c'est bien passé.
            $scope.card = angular.copy({});
        });
    }
 }]);

Normalement le code est assez commenté pour que vous le compreniez. Au besoin, postez un commentaire. Je signale juste que pour utiliser correctement le code csrf (qui est stocké dans un cookie) il faut utiliser le plugin ngCookies.

Ce code dépend de jQuery. Je n'ai pas trouvé de méthode pour passer correctement et facilement les paramètres de la requête avec uniquement Angular. La seule méthode que j'ai est :

var in_data = 'content=' + $scope.card.content + '&category=' + $scope.card.category + '&csrfmiddlewaretoken=' + $cookies.csrftoken;

Note : la requête AJAX peut aussi se mettre sous la forme plus longue suivante :

$http({
    url: '/add',
    method: 'POST',
    data: in_data
}).success(function(out_data) {
    $scope.card = angular.copy({});
});

AJAX requests with Django and AngularJS

2014-10-25T00:00:00+02:00

In this tutorial, I will explain how to submit a Django form with an AJAX request thanks to AngularJS. First, we need to prepare our app - I don't precise imports and I assume that you know in which file to write the code. First, the model:

class Card(models.Model):
    content = models.TextField()
    date = models.DateTimeField(auto_now_add=True, auto_now=True)
    category = models.ForeignKey('Category')

    def __str__(self):
        return self.content

The form:

class CardForm(forms.ModelForm):
    class Meta:
        model = Card

The view:

def add_card(request):
saved = False

if request.method == 'POST':
    form = CardForm(request.POST, request.FILES)
    if form.is_valid():
        card = Card()
        card.content = form.cleaned_data['content']
        card.category = form.cleaned_data['category']
        saved = True
        card.save()
else:
    form = CardForm()

return render(request, 'board/add_card.html', locals())

And the template:

<!DOCTYPE html>
{% load staticfiles %}
<html ng-app="ajax">
<head>
  <meta charset="UTF-8" />
   <script src={% static "angular.js" %} type="text/javascript" ></script>
   <script type="text/javascript" src={% static "angular-cookies.js" %}></script>
   <script src={% static "controllers.js" %} type="text/javascript" ></script>
 </head>
 <body>
<form action="{% url "board.views.add_card" %}" ng-controller="MyFormCtrl"
       method="post" name="form">{% csrf_token %}
       <p>
         <label for="id_content">Content:</label>
         <textarea cols="40" id="id_content" name="content" rows="10" ng-model="card.content"></textarea>
       </p>
         <p><label for="id_category">Category:</label> <select ng-model="card.category" id="id_category" name="category">
         <option value="" selected="selected">---------</option>
         <option value="1">Backlog</option>
         <option value="2">Done</option>
         <option value="3">In progress</option>
         </select>
         <button ng-click="submit($event)">Submit</button>
       </p>
 </form>
 </body>
 </html>

As any Angular app, it contains a ng-app and a ng-controllers tags to link your page with the JS code. I also added {% load staticfile %} to use django's static directive.

The form must be completely present in the template - you can't use {{ form.as_p }} - in order to correctly map its element with the model in Angular:

The Textaera with ng-model="card.content".
The options with ng-model="card.category".
The submit button with ng-click="submit($event)". $event is required to avoid a page reload when the form is submitted.

And now, the JS code:

var nameSpace = angular.module("trello", ['ngCookies']);

nameSpace.controller("MyFormCtrl", ['$scope', '$http', '$cookies',
function ($scope, $http, $cookies) {
    $http.defaults.headers.post['Content-Type'] = 'application/x-www-form-urlencoded';
    // To send the csrf code.
    $http.defaults.headers.post['X-CSRFToken'] = $cookies.csrftoken;

    // This function is called when the form is submitted.
    $scope.submit = function ($event) {
        // Prevent page reload.
        $event.preventDefault();
        // Send the data.
        var in_data = jQuery.param({'content': $scope.card.content, 'category': $scope.card.category, 'csrfmiddlewaretoken': $cookies.csrftoken});
        $http.post('/add', in_data)
          .success(function(out_data) {
            // Reset the form in case of success.
            $scope.card = angular.copy({});
        });
    }
 }]);

In order to use the csrf code - which is stored in a cookie, you need to use the ngCookie plugin. You should be able to understand the rest of the code. In case of trouble, leave a comment.

This code requires jQuery. I have not found a way to correctly pass the argument with just Angular. The only method I have is:

var in_data = 'content=' + $scope.card.content + '&category=' + $scope.card.category + '&csrfmiddlewaretoken=' + $cookies.csrftoken;

Note: The AJAX request can also be written like this:

$http({
    url: '/add',
    method: 'POST',
    data: in_data
}).success(function(out_data) {
    $scope.card = angular.copy({});
});

Installer des paquets npm sans les droits root

2014-10-24T00:00:00+02:00

npm peut soit installer un paquet dans le dossier du projet courant soit de façon globale pour tous les projets avec l'option -g. Par défaut dans ce mode, npm installe les paquets dans /usr/lib/node_module ce qui nécessite les droits root. En dehors du désagrément que cela procure, cela peut être dangereux si on installe un paquet depuis une source inconnue.

Heureusement, npm peut aussi installer les paquets dans le home de l'utilisateur courant. Pour cela, il faut tout d'abord créer le dossier qui contiendra les paquets :

mkdir -p ~/.npm-packages

Reste à configurer le préfixe de npm (c'est là qu'il installe ces paquets) avec la variable prefix dans le .npmrc. Ajoutez donc prefix = /home/username/.npm-packages à votre npmrc (~/.npmrc).

Il faut ensuite que votre shell ait les bonnes variables d'environnement. Pour cela, ajoutez à votre .zshrc ou .bashrc :

# NPM
NPM_PACKAGES="$HOME/.npm-packages" # Pour que npm trouve ces paquets.
PATH="$NPM_PACKAGES/bin:$PATH" # Pour que les scripts puissent être lancés directement depuis le terminal.
unset MANPATH
MANPATH="$NPM_PACKAGES/share/man:$(manpath)" # Pour que man trouve les manuels des commandes.
NODE_PATH="$NPM_PACKAGES/lib/node_modules:$NODE_PATH" # Pour que node trouve les paquets.
export NPM_PACKAGES PATH MANPATH NODE_PATH

C'est fini. Maintenant, npm install -g bower installera bower dans votre home sans les droits root.

Install npm packages with non root user

2014-10-24T00:00:00+02:00

npm can install packages either in the current folder or globally with the -g option. By default, in this mode, npm installs packages in /usr/lib/node_mudole. You must be root to do this and it can be very dangerous if you are installing an untrusted package.

However, npm can also install packages in your home. To do this, you must first create the folder which will contain all your packages :

mkdir -p ~/.npm-packages

You must then configure the prefix - where npm installs its packages - with the prefix variable in ~/.npmrc. So add prefix = /home/username/.npm-packages ~/.npmrc.

Your shell must then have the proper environment variables. So add to your.zshrc or .bashrc :

# NPM
NPM_PACKAGES="$HOME/.npm-packages" # So that npm finds its packages.
PATH="$NPM_PACKAGES/bin:$PATH" # To launch script directly from the shell.
unset MANPATH
MANPATH="$NPM_PACKAGES/share/man:$(manpath)" # So that man finds the manpages.
NODE_PATH="$NPM_PACKAGES/lib/node_modules:$NODE_PATH" # So that node finds these packages.
export NPM_PACKAGES PATH MANPATH NODE_PATH

That's all. Now npm install -g bower will install bower in your home without the need to be root!

Utiliser git pour gérer les fichiers de configuration de son HOME

2014-09-09T00:00:00+02:00

Depuis pas mal de temps, je gère les fichiers de conf de mon home avec git, notamment ma conf emacs et mon dossier bin. L'intérêt étant à la fois de pouvoir partager les configurations entre plusieurs ordinateurs, de pouvoir revenir en arrière en cas de besoin et de voir l'historique (surtout pour emacs).

Je faisais ça à la main, avec un dépôt git tout simple. Puis, suite à lecture de cet article, j'ai décidé de passer à vcsh et mr.

VCSH – Version Control System for $HOME

L'intérêt de ce logiciel est qu'il vous permet de faire cohabiter plusieurs dépôts git dans un même dossier. Ainsi vous pouvez facilement avoir un dépôt git par catégorie de logiciel : un pour emacs, un pour votre bin, un autre pour zsh, …

Par défaut, tous les dépôts sont dans ~/.config/vcsh/repos.d. Évidemment ça se paramètre, voir la page du projet sur github.

Normalement, vous pouvez installer vcsh depuis les dépôts de votre distribution.

mr — myrepo

mr est le compagnon de vcsh puisqu'il vous permet en une seule commande d'agir sur plusieurs dépôts. Ainsi, pour pusher ou puller les modifications de tous les dépôts sur un serveur il suffit de faire mr push ou mr checkout. Il permet aussi de gérer la configuration des dépôts (sur quels url pusher les modifications, sur votre serveur perso et sur github par exemple).

Une fois de plus, le logiciel devrait être dans les dépôts de votre distribution. Plus d'infos ici.

Mettre en place vcsh

Vous devez tout d'abord identifier comment vous désirez organiser vos dépôts. Personnellement, je me suis organisé comme suis (les liens pointent sur github) :

dvcs pour les configurations de git, mercurial et mr
emacs pour la configuration d'Emacs
shell pour mon bin/ et les aliases + fonction
zsh pour la configuration de zsh

Pour initialiser un nouveau dépôt, rien de très compliqué :

# Initialisation du dépôt
vcsh init emacs
# Création du gitignore pour éviter d'avoir des erreures sur les dossiers non #suivi.
# Met simplement * dans le fichier.
vcsh write-gitignore emacs
# Ajout des fichiers
vcsh emacs add -f ~/.emacs ~/.emacs.d ~/.gitignore.d/emacs
vcsh emacs commit -am 'intial commit'
vcsh emacs remote set-url --add origin git@github.com:Jenselme/dot-files-emacs.git

Il peut être intéressant de personnaliser son .gitignore (qui de base est un peu violent), surtout pour des configurations auxquelles on ajoute des fichiers (comme celle d'emacs). Pour cela, il faut utiliser le ! pour inclure un sous chemin exclu. Par exemple avec mon ~/.gitignore.d/emacs (le / représente la racine du dépôt) :

/*
!/.emacs
!/.emacs.d
*.elc
*~
/.emacs.d/session*
/.emacs.d/tramp
.emacs.d/url
.emacs.d/semanticdb/*
\#*
.emacs.d/*.dat
.emacs.d/*.lst

Je signale que si vous omettez le nom du dépôt pour une commande vcsh, alors elle s'appliquera à tous. Particulièrement utile si vous voulez connaître le statut de tous vos dépôts.

Configurer mr

La configuration de mr se place dans ~/.mrconfig. Ma configuration permet de pusher mes dépôts à la fois sur mon serveur perso et sur github (pour partager). Je la présente en intégralité et je vous laisse vous en inspirer pour la vôtre.

[DEFAULT]
git_gc = git gc "$@"

[$HOME/.config/vcsh/repo.d/emacs.git]
checkout =
  vcsh clone --recursive ssh://git@jujens.eu/home/git/emacs.git emacs
  vcsh emacs remote add github git@github.com:Jenselme/dot-files-emacs.git

push =
  vcsh emacs push
  vcsh emacs push github

[$HOME/.config/vcsh/repo.d/zsh.git]
checkout =
  vcsh clone ssh://git@jujens.eu/home/git/zsh.git zsh
  vcsh zsh remote add github git@github.com:Jenselme/dot-files-zsh.git

push =
  vcsh zsh push
  vcsh zsh push github

[$HOME/.config/vcsh/repo.d/dcvs.git]
checkout =
  vcsh clone ssh://git@jujens.eu/home/git/dcvs.git dcvs
  vcsh dcvs remote add github git@github.com:Jenselme/dot-files-dcvs.git

push =
  vcsh dcvs push
  vcsh dcvs push github

[$HOME/.config/vcsh/repo.d/shell.git]
checkout =
  vcsh clone ssh://git@jujens.eu/home/git/shell.git shell
  vcsh shell remote add github git@github.com:Jenselme/dot-files-shell.git

push =
  vcsh shell push
  vcsh shell push github

Récupérer la configuration

Si vous arrivez sur une nouvelle machine, vous n'avez plus qu'à installer mr et vcsh et cloner le dépôt contenant la configuration de mr.

vcsh clone ssh://git@jujens.eu/home/git/dcvs.git dcvs
mr checkout

Pour mettre à jour tous ces dépôts, un petit mr update suffit.

Conclusion

Le duo vcsh + mr est simple et efficace pour partager ses configurations entre plusieurs ordinateurs.

Divers astuces en Shell

2014-09-07T00:00:00+02:00

Liste de commandes pour des actions spécifiques. Pas/peu de détails/explications, prérequis de shell supposés connus. Si une commande est donnée pour bash uniquement, cela signifie qu’elle n’est pas compatible avec le shell POSIX ou d’autres implémentations basiques mais est vraisemblablement disponible dans les autres implémentations évoluées telles que zsh.

J'ai initialement publié ce document ici. Ce document se veut un peu plus complet et ne contient pas de rappels de commandes élémentaires comme wc. J'ai également écrit un tutoriel sur le shell.

Sommaire

Corriger le caractère de fin de ligne
Enlever les shebang de plusieurs fichiers
Opérations avancées sur les variables
- Valeurs par défaut
- Travailler avec les chemins
Manipulation avancée de chaînes de caractères
Générer des nombres aléatoires
Savoir si une chaîne est contenue dans une autre
Renommer les fichiers en fonction d’une expression régulière
Savoir si un script est lancé dans un shell interactif
Chmod a+X
Différences entre les implémentations de certaines commandes entre GNU et BSD
- sed

Corriger le caractère de fin de ligne

Pour changer les caractères de fin de ligne d'un fichier de \r\n (windows) à \n (Unix), (l'option -i fait en sorte que la modification se fasse dans le même fichier).

sed -i 's/\r$//' file

Enlever les shebang de plusieurs fichiers

# Remove shebang
for lib in $RPM_BUILD_ROOT/svgwrite/*.py; do
  sed -i '1{\@^#!/usr/bin/env python@d}' $lib
done

Opérations avancées sur les variables

Valeurs par défaut

Il est tout à fait possible de donner une valeur par défaut à une variable en bash. Ainsi, le code suivant : FOO=${FOO:-coucou} signifie que :

Si la variable FOO est définie, alors ne rien faire
Si FOO n’est pas définie, alors FOO vaut coucou

Cela fonctionne aussi avec une autre variable : FOO=${toto:-coucou}. Cette fois, FOO prendra la valeur coucou si toto n’est pas définie. Cette forme laisse toto inchangée. Pour affecter coucou a foo et toto : FOO=${toto:=coucou}.

Travailler avec les chemins

Le shell dispose de moyen de travailler avec les chemins de façon facile et agréable :

Supprimer le dernier / : ${a%/}
Supprimer le premier / : ${#*/}
Récupérer le dossier courant : ${a##*/} (si a ne se termine pas par /)
Récupérer le dossier parent : ${a%/*}

En vérité, vous pouvez remplacer le / par n'importe quel autre caractère (un point par exemple). Voir supprimer une partie de chaîne pour plus de détails.

Manipulation avancée de chaînes de caractères

Extraire une sous-chaîne (bash uniquement)

Simplement avec la syntaxe ${chaine:début:longueur}.

chaine="Trucs et astuces en shell"
echo ${chaine:0:16} # Affiche "Trucs et astuces"

Ou avec cut :

cut -cN-M # N : position du début, M : position de la fin

Remplacer une sous-chaîne par une autre (bash uniquement)

On peut remplace la première occurrence uniquement :

chaine_de_base="J'aime les pommes et les framboises"
remplacer_pomme_par="fraises"
chaine_corrigee=${chaine_de_base/pommes/$remplacer_pomme_par}

On peut aussi remplacer toutes les occurrences avec :

chaine_corrigee=${chaine_de_base//pommes/$remplacer_pomme_par}

On peut aussi remplacer le début ou la fin :

fichier="/root/script.py"

echo ${fichier/#\/root/\/tmp}
echo ${fichier/%.*/.sh}

Ce qui donne :

/tmp/script.py
/root/script.sh

Passer du texte en majuscule/minuscule (bash uniquement)

a=COUCOU
echo ${a,,} # affiche "coucou"

a=coucou
echo ${a^^} # affiche "COUCOU"

On peut aussi passer juste la première lettre en majuscule/minuscule :

a=coucou
echo ${a^} # affiche "Coucou"

a=Coucou
echo ${a,} # affiche "coucou"

Supprimer une partie de chaîne

Pour supprimer la plus courte partie d'une string en partant du début, il suffit d'utiliser la syntaxe ${chaine#sous-chaine}. Pour partir de la fin : ${chaine%sous-chaine}.

fichier="/var/tmp/"

echo ${fichier#*/} # Affiche "var/tmp/"
echo ${fichier%/*} # Affiche "/var/tmp"

Pour supprimer la plus longue partie possible d'une chaîne, il suffit de doubler le caractère :

fichier="/var/tmp/toto"
echo ${fichier##*/} # Affiche toto

fichier=var/tmp/toto
echo  ${fichier%%/*} # Affiche var

Générer des nombres aléatoires

Tire un nombre aléatoirement entre 0 - 32767 (bash uniquement) :
```
$RANDOM
```
Si RANDOM n’est pas disponible, la commande fortune qui tire aléatoirement un proverbe l’est peut être. On en prend la somme pour obtenir un nombre.
```
fortune | cksum | cut -f1 -d" "
```

Utilise la date où le PID du dernier processus.

seed=$(( echo $$ ; time ps ; w ; date ) | cksum | cut -f1 -d" ")

Utilise l’entrée d’entropie du noyau. C’est la meilleur méthode, la plus robuste pour générer de l’aléatoire.
```
dd if=/dev/urandom count=1 2> /dev/null | cksum | cut -f1 -d" "
```

Savoir si une chaîne est contenue dans une autre

Avec la fonction test (bash uniquement)
```
[[ reg_exp =~ contenant ]]
```
Avec grep
```
grep -q "reg_exp" chaine
```

Renommer les fichiers en fonction d’une expression régulière

for file in `ls` ; do
  new_name=`echo $file | sed 's/\(.*\)\.dump\(.*\)\.sql/\2.\1.sql/'`
  mv $file $new_name
done

On peut également utiliser Emacs.

Savoir si un script est lancé dans un shell interactif

La commande tty -s renvoie 0 si le script est lancé dans un terminal interactif et 1 sinon.

Chmod a+X

chmod a+X va donner les droits d'exécution à un fichier uniquement si les droits d'exécution sont présents pour l'utilisateur, le groupe ou tous. Par exemple :

$ mkdir test
$ mkdir test/folder
$ mkdir -m 600 test/weired_folder
$ touch test/script
$ chmod +x test/script
$ touch test/file

Les droits sont donc :

-rw-r--r--.  1 jenselme jenselme   0 Feb 20 23:02 file
drwxr-xr-x.  2 jenselme jenselme  40 Feb 20 23:01 folder
-rwxr-xr-x.  1 jenselme jenselme   0 Feb 20 23:02 script
drw-------.  2 jenselme jenselme  40 Feb 20 23:02 weired_folder

$ chmod -R a+X

Maintenant les droits sont :

-rw-r--r--.  1 jenselme jenselme   0 Feb 20 23:02 file
drwxr-xr-x.  2 jenselme jenselme  40 Feb 20 23:01 folder
-rwxr-xr-x.  1 jenselme jenselme   0 Feb 20 23:02 script
drwx--x--x.  2 jenselme jenselme  40 Feb 20 23:02 weired_folder

Différences entre les implémentations de certaines commandes entre GNU et BSD

sed

La commande sed de BSD ne connaît pas certains métacaractères tels que \s.

Les opérations dans un fichier se font :

sed -i'' 's/toto/tata/g' fichier sous GNU mais
sed -i '' 's/toto/tata/g' fichier sous BSD. La commande
sed -i -e s/toto/tata/g' fichier est compatible avec les deux versions.

Petit précis de shell

2014-09-07T00:00:00+02:00

Le shell (POSIX, ie sans les améliorations de Bash ou zsh) est un langage assez particulier qu’on ne connaît pas forcément. Le but de ce document est de rappeler (ou donner) quelques bases du langage, de rappeler les différences notables avec bash. Je propose dans un autre article quelques solutions standards afin de vous aider dans vos scripts. Ce document ne se veut en aucun cas un cours/tuto complet et exhaustif. Des bases de shell et d’un autre langage de programmation peuvent s’avérer utile. Voir le site du zéros pour un tuto orienté vers les débutants.

Initialement publié ici. Ce document est une version améliorée et corrigée.

Sommaire

Quelques rappels d’Unix

Le langage shell est le langage de script qui vient par défaut avec tous les Unix. En effet, même si désormais d’autres interpréteurs plus modernes sont désormais répandus (bash, csh, zsh, etc.), ils ont tous conservé la compatibilité avec le shell.

Comme pour la plupart des langages de script, il existe deux façons d’exécuter des instructions shell :

directement dans l’interpréteur
dans un script shell

Pour lancer un interpréteur shell, rien de plus simple : lancer un terminal (graphique ou tty). Et oui, le shell comprend toutes les commandes Unix. Vous pouvez donc les réutiliser telles quelles dans vos scripts et utiliser un simple terminal pour faire des boucles, des conditions et j’en passe. C'est un des très gros intérêts du langage d'ailleurs.

Les bases du shell

Il est très important de comprendre et de garder à l’esprit qu’en shell tout est :

chaîne de caractères (y compris les nombres) ! Entrez echo 1 + 1 dans le terminal pour vous en convaincre.
commande et que donc elles peuvent prendre des arguments (cela s’éclaircira plus tard)

Syntaxe de base

Les commandes s’écrivent soit :

les unes à la suite des autres séparées par ; (peu recommandé)
séparées les unes des autres par un saut de ligne

Chaque commande peut prendre des options (qui modifient la façon dont la commande se comporte comme affiche de l’aide, modifier la verbosité de la sortie, etc.) de deux types :

les options courtes (l, r, h pour ls par exemple) qui sont passés comme suit : CMD -ARG
les options longues (recursive pour rsync par exemple) qui se passent comme ceci : CMD --ARGUMENT

Il est évidement possible de passer plusieurs options à une même commande.

Certains paramètres existent sous une forme courte et une forme longue. Consulter le manuel de la commande pour plus de détails. Le manuel contient également la liste complète des arguments supportés par une commande.

Certains commandes ne respectent pas la convention énoncée ci-dessus. Leurs arguments long se passent avec un seul - (find en est un exemple).

Les commandes peuvent aussi prendre des arguments positionnels. Ils se placent après les paramètres.

Exemple : ls -R ~/Documents. Par défaut, ls ne liste que les fichiers et dossiers dans le dossier courant. L’option -R permet de descendre dans tous les dossiers. L’argument ~/Documents permet d’exécuter la commande dans le dossier ~/Documents sans s’y déplacer.

Pensez à la commande man qui prend comme argument une autre commande et affiche son manuel.

Les variables

Les variables se déclarent comme ceci : var="ping". Notez bien l'absence d'espaces entre var, = et "ping". Si vous ajoutez des espaces, le shell tentera de faire un test. Voir Conditions if … else ….

Ensuite, pour utiliser une variable, il faut mettre un $ devant le nom de la variable. Par exemple : echo $var affichera le contenu de la variable var. La variable est aussi remplacée par sa valeur dans une chaîne si elle est encadrée par des guillemets mais pas si elle est encadrée par des apostrophes. Par exemple :

echo "$var pong" affichera ping pong.
echo '$var pong' affichera $var pong.

Valeurs de retour des commandes et exceptions

Une fois qu’une commande s’est exécutée, elle renvoie une valeur de retour afin d’informer l’utilisateur. Cette valeur permet en effet de savoir si la commande s’est exécutée correctement. Voici les valeurs de retour courantes et leur signification :

0 : tout va bien
1 : erreur
2 : erreur grave

Vous pouvez vous aussi utiliser ces valeurs de retour. Par défaut, un script qui se complète correctement retourne 0. Mais vous pouvez (par exemple si un utilisateur tente d’exécuter un script qui nécessite un argument sans) retourner un autre code d’erreur avec la commande exit. Il suffit de lui passer le code qu’elle doit retourner. Votre script s’arrêtera alors avec le code d’erreur spécifié.

Vous pouvez connaître la valeur de retour de la dernière commande exécutée avec la variable $?.

Les valeurs de retour ne sont pas limitées à 0, 1, 2. Par convention, c'est n'importe quel entier positif. À chaque entier est associé une signification, ce qui permet de comprendre ce qui s'est passé si une commande échoue.

Exécuter une commande

S’il est facile dans les cas simples d’exécuter une commande, dès lors qu’en shell tout est chaîne de caractères, si vous voulez affecter la sortie d’une commande à une variable, vous ne pouvez pas simplement faire var=CMD car var va alors valoir la chaîne CMD.

Pour obtenir le résultat souhaité vous devez placer CMS entre $(…). Par exemple : var=$(ls).

Vous pouvez aussi mettre la commande entre backquote `. Mais cette syntaxe ` est plus ancienne et ne doit plus être utilisée. La syntaxe $(…) présente l’avantage de pouvoir imbriquer les commandes sans ambiguïté.

Il est également tout à fait possible de stocker une commande dans une variable pour l'exécuter ensuite (une fois de plus, tout est chaîne de caractère). Une commande est en effet une chaîne de caractère située en position 0 sur une ligne.

mon_cd=cd
$mon_cd

Conditions et itérations

Conditions if … else …

La structure générale d’une condition est la suivante :

if QQC
then
    CMDS
else
    CMDS
fi

Le else est facultatif. Il est aussi possible de regrouper if et then en une seule ligne comme ceci : if QQC ; then. On peut également utiliser des elif.

La question que vous devriez avoir est que mettre à la place de QQC. Il y a deux possibilités :

la fonction test
une commande

La fonction test

Dans toute la suite, il faudra faire très attention aux espaces

La fonction test s’utilise en général comme suit : if [ ARGS ] (Notez les espaces).

La syntaxe if [[ ARGS ]] n’est valide qu’avec bash.

Pour faire un test, il suffit ensuite de passer les bons arguments à la commande. Par exemple, pour tester si une chaîne est vide : if [ -z "$chaine" ]. Si l’argument a besoin de deux paramètres pour fonctionner, mettre un paramètre de chaque côté de celui-ci. Par exemple, pour faire un test d’égalité de chaîne de caractères : CHAINE1 = CHAINE2.

On peut aussi combiner les arguments avec des ET et des OU avec les options -a et -o. Le caractère “!” permet de faire une négation.

On peut également faire des ET avec && et des OU avec ||. Ces opérateurs ne sont pas spécifiques à la fonction test, ils font parti du langage. Il est donc tout à fait possible de les utiliser sans la fonction test.

Voir ci-dessous pour la liste complète.

En shell, tout est chaîne de caractère. Bien faire attention au type que l’on veut tester (chaîne ou nombre).

Tests sur les chaînes de caractères

Argument	Signification
=	égalité
-z	chaîne vide
-n	chaîne non vide

Lors des tests de chaîne de caractères, entourez la variable de guillemets. Sinon, si la chaîne est vide, le test ne pourra être correctement effectué.

Tests sur les nombres

Argument	Signification
-eq	égalité
-ne	différent
-lt	strictement plus petit
-gt	strictement plus grand
-ge	plus grand ou égal
-le	plus petit ou égal

Test avec une commande

Comme indiqué précédemment, une commande qui s’exécute correctement est considérée comme vraie. Ainsi, il est tout a fait possible, par exemple, pour savoir si on arrive à se connecter à un serveur mysql de faire simplement : if mysql -h HOST -u asso -pTATA.

Parfois vous pourrez rencontrer des problèmes. Pensez alors à donner cette commande en argument à la fonction test.

Le shell contient deux fonctions utiles: true et false. La première renvoie toujours 0 et la seconde toujours 1. Vous pouvez les trouver utiles si vous avez besoin de manipuler des booléens.

Boucles while/until

La structure générale est la suivante :

while QQC
do
    CMD
done

Il est possible de regrouper while QQC et le do en while QQC ; do. Le QQC peut être remplacer par exactement les mêmes choses que pour la condition. Se référer à cette section pour les précisions.

Le shell propose également le mot clé until QQC qui fait une action jusqu’à ce que QQC soit réalisé (équivalent de while ! QQC).

Boucle for

L’utilisation de la boucle for en shell ressemble à celle de python. La structure générale est la suivante :

for var in `CMD`
do
   CMD
done

La variable var va alors prendre une à une les valeurs données par CMD. Par exemple, for file in $(ls) la variable var va prendre tour à tour le nom de tous les fichiers et dossiers donnés par la commande ls.

Vous pouvez également utiliser for pour boucler d’un nombre à un autre avec la syntaxe : for i in $(seq [first [incr]] last)

Les tableaux

Un tableau se crée comme ceci : tab=(un deux trois). On accède à un élément grâce à son index (les index commencent à zéros) :

tab=(un deux trois)
echo ${tab[0]}
echo ${tab[1]}
echo ${tab[42]} # Ne renvoie rien, index en dehors de la plage défini.

Idem pour l'assignation :

tab[0]=zero
echo ${tab[0]} # Renvoie zero.

Les index ne sont pas obligatoirement continus :

tab[42]=42
echo ${tab[42]} # Renvoie 42.
echo ${tab[9]} # Ne renvoie rien.

On peut facilement connaître le nombre d'élément du tableau avec ${#tab[@]}. Il est également facile de parcourir tous les éléments :

for elt in ${tab[@]}; do
    echo $elt
done

Va afficher :

12
deux
trois
42

On peut également afficher tout le tableau avec ${tab[@]}.

Paramètres de scripts

Généralités

Un script peut prendre des paramètres qui ont le même but que les arguments d’une fonction : lui passer des informations. Un paramètre peut être :

une variable
une chaîne de caractère (donc un nombre, en shell on ne fait pas la distinction). Si la chaîne à passer en paramètre contient plusieurs mots séparés par des espaces, ne pas oublier de la mettre entre ‘ ou “.

Si une variable coucou contient la chaîne ‘salut’, alors ‘$coucou toi’ sera compris $coucou toi tandis que $coucou toi sera interprétée en salut toi
Les paramètres se passent à un script comme ceux d’une commande. Ils sont ensuite disponibles dans l’ordre avec des numéros :
- le premier : $1
- le deuxième : $2
- et ainsi de suite
Le shell ne supporte que 9 paramètres.

Les paramètres spéciaux

$0 : contient le nom du script
$# : contient le nombre d’arguments passés au script

Le paramètre $0 est toujours passé au script. $# est donc toujours supérieur ou égal à 1.
$? : le code de retour de la dernière commande invoquée
$$ : le PID du shell qui exécute le script
$! : le PID du dernier processus lancé en arrière plan
$* : l’ensemble des paramètres en un seul argument
$@ : L’ensemble des arguments, un argument par paramètre

Pour bien voir la différence entre $* et $@, il suffit de regarder la sortie du script suivant :

echo 'Avec $*'
for param in "$*" ; do
  echo $param
done

echo 'Avec $@'
for param in "$@" ; do
  echo $param
done

Les commandes shift et set

La commande set permet d’affecter les paramètres. Ainsi set bonjour salut bonsoir va initialiser $1 à bonjour, $2 à salut et $3 à bonsoir. Les paramètres spéciaux $#, $* et $@ sont mis à jour.

La commande shift permet de décaler les variables. Ainsi, si après shift $1 prend la valeur de $2 et ainsi de suite. shift n décale les arguments de n.

Les fonctions

Il est tout à fait possible de définir en shell des fonctions que ce soit dans un script ou un terminal. La syntaxe est la même.

nom_de_la_fonction () {
 CMDS
}

ou encore

function nom_de_la_fonction {
 CMDS
}

Les fonctions ainsi créées s’utilisent comme les commandes classiques et leurs arguments se manipulent exactement comme ceux d’un script. Voir la section dédiée. Il faut néanmoins faire attention à deux points :

la portée des variables
la valeur de retour

Par défaut, les variables définies dans la fonction resteront accessibles une fois la fonction exécutée. De même les variables définies auparavant restent accessibles dans la fonction. Ces variables sont donc globales par défaut.

Pour qu’une variable soit locale, il faut utiliser le mot clé local lors de sa définition. Par exemple : local nom=clubdrupal (NB : local est une commande qui peut prendre des options).

Pour qu’une fonction en bash retourne une valeur comme vous pouvez en avoir l’habitude dans d’autres langages, il faut utiliser la commande echo`.

Il faut alors faire très attention car si la sortie n'est pas capturée dans une variable, elle s'affiche dans le terminal. Par exemple avec la fonction suivante :

x_files () {
  top_secret=`dd if=/dev/urandom count=1 2> /dev/null | cksum | cut -f1 -d" "`
  echo $top_secret
}

Le mot clé return

Comme la plupart des langages, le shell dispose d'un mot clé return. Lorsque ce mot clé est exécuté dans une fonction, cette fonction est exécuté jusqu'à ce mot clé, puis l'exécution du programme reprend à l'instruction suivante. Ce mot clé est particulièrement utile pour que des fonctions renvoient des codes d'erreur sans pour autant arrêter le script (ce que exit aurait fait).

Les redirections de flux

Les flux sortant

Les flux représentent les sorties générées par les commandes. Par défaut, il existe deux flux :

le flux standard
le flux d’erreur

Par défaut, il s’affiche sur la sortie standard (votre écran pour être bref). Il peut s’avérer intéressant d’envoyer ces flux ailleurs (logs, le néant, etc.). Pour cela, on va les rediriger. Par exemple pour rediriger la sortie de ls dans un fichier nommé toto, on fait :

ls > toto
ou
ls >> toto

La première solution efface le contenu du fichier puis écrit dedans. La seconde ajoute la sortie à la fin du fichier. On a ici redirigé le flux standard. Pour rediriger les flux d’erreurs, on utilise les symboles 2> ou 2>>. On peut évidemment combiner les deux : ls -R / > mes_fichiers.txt 2> errors.log avec toutes les variantes possibles.

Pour rediriger l’erreur au même endroit que l’entrée, on peut faire ls > toto 2> toto ou plus simplement ls > toto 2>&1. Pour rediriger une sortie vers le néant, on l’envoie dans /dev/null.

Les flux entrant

Il est également possible de passer en paramètre le contenu d’un fichier. Pour cela, on utilise le symbole <.

Importer une configuration

Il est tout à fait possible d’écrire un fichier de configuration contenant les variables et les fonctions indispensables à d’autres scripts et les réutiliser facilement dans ceux-ci. Pour cela, respecter la syntaxe shell dans le fichier puis au début du script qui en a besoin, placer la ligne : . config-file.sh pour l’importer.

Mode debug

Lorsqu’un de vos scripts shell bug il peut être difficile de savoir d’où vient le problème. Heureusement, le shell propose un mode débug qui vous dit pour chaque ligne comment elle est exécuté, avec quels paramètres (les variables sont remplacées par leur contenu). Il suffit de faire : sh -x SCRIPT

Parser des arguments

Il existe deux commandes pour parser des arguments en shell : getopts, qui est une commande ”builtin" qui ne supporte que les options courtes et getopt, qui est une commande à part, pas forcément présente mais qui supporte les arguments courts et longs. Nous n’étudierons ici que getopts qui a l’avantage d’être présent partout.

La commande s’utilise comme suit :

while getopts "options" opt; do
  case "$opt" in
    option1) action;;
  esac
done

Ainsi, pour utiliser l’option -o :

while getopts "o" opt; do
  case "$opt" in
    o) echo $opt;;
  esac
done

script -o affera : o. Si on le lance avec l’option -a, il affichera :

illegal option -- a

Il est possible d’utiliser getopts en mode erreurs silencieuses en ajoutant : au début de la chaîne des options.

Si votre option a besoin d’un argument, il suffit de placer : après son nom :

while getopts "ob:" opt; do
  case "$opt" in
    o) echo $opt;;
    b) echo "$opt used with option $OPTARG";;
  esac
done

Une fois vos arguments parser, vous pouvez mettre le premier argument positionel de votre script dans $1 avec :

shift $((OPTIND-1))

Vous pouvez aussi utiliser dans le case ? pour afficher un message si l’utilisateur passe une option inconnue et : pour afficher un message si une option qui requiert un argument ne l’a pas eu.

Exemple

while getopts ":h:u:p:P" opt
do
    case "$opt" in
        h)
            host=$OPTARG; hflag=true;;
        u)
            user=$OPTARG; uflag=true;;
        p)
            passwd=$OPTARG; pflag=true;;
        P)
            Pflag=true;;
        :)
            echo "Option -$OPTARG requires an argument." >&2
            usage; exit 1;;
        \?)
            usage; exit 0;;
    esac
done

shift $((OPTIND-1)) # To get the 1st positional argument with $1

Différences notables avec bash

Variables disponibles uniquement en Bash

RANDOM (pour la génération de nombre aléatoire). Voir les trucs et astuces pour plus de détails.

Syntaxe

&> et |&. Permettent de rediriger tous les flux vers un fichier ou de passer tous les flux à une commande (pipe généralisé)
{2..10} pour générer des séquences de nombres
[[ ARGS ]] : permet de faire des tests de façon plus facile ou agréable. Exemple : [[ chaine == chaine2 ]]
=~ : s'utilise conjointement avec [[ ]] pour tester si la chaîne de gauche est contenu dans celle de droite
$((ARGS)) est un raccourcis pour la fonction let

Autres

Le nombre de paramètres en bash n’est pas limité à 9. Les paramètres positionnels de numéros supérieurs à 10 s’appellent comme suit : ${num}

Divers

Différences entre la sortie de ls et de find

ls renvoie simplement la liste des fichiers.
find renvoie un chemin absolu si l’argument donnée est un chemin absolu et relatif (de la forme ./fichier) si l’argument est .

ET et OU dans le langage

L’opérateur && permet de réaliser un ET paresseux entre deux commandes. Ainsi, par exemple : cmd1 && cmd2. cmd2 ne sera exécuté que si cmd1 a pu s’exécuter correctement. En outre le code de retour de l’ensemble ne sera 0 que si les deux commandes ont pu s’exécuter correctement.

L’opérateur || permet de réaliser un OU paresseux entre deux commandes. Ainsi, par exemple : cmd1 || cmd2. Si cmd1 s’exécute correctement cmd2 ne sera pas exécutée. Si cmd1 retourne une erreur, cmd2 sera exécutée. Le code de retour de l’ensemble correspond à un OU entre les deux codes de retour.

Ceci peut s’utiliser dans les conditions.

Des fonctionnalités analogues existes dans d’autres langages inspirés du shell comme Perl ou PHP.

Pour exécuter simplement deux commandes à la suite des autres sur une seule ligne, il suffit de séparer les commandes par un point virgule.

Sources et liens externes

Quelques règles de style. Certains points ne s’appliquent qu’à bash : http://google-styleguide.googlecode.com/svn/trunk/shell.xml
http://www.siteduzero.com/informatique/tutoriels/reprenez-le-controle-a-l-aide-de-linux
http://www.commentcamarche.net/faq/5444-bash-les-parametres

Zsh, c'est trop cool

2014-09-06T00:00:00+02:00

Depuis plusieurs mois maintenant, j'utilise zsh comme interpréteur par défaut. L'un des principaux avantages de zsh par rapport à bash est la possibilité de complétion avancée. Sous bash, lorsqu'on appuie sur <TAB> pour obtenir la complétion d'une commande, bash se contente de lister les possibilités. Sous zsh, les pressions suivantes sur <TAB> passe en revue les autres. Cela fonctionne également pour les arguments des commandes.

Mais ce n'est pas tout. Avec bash, le raccourcis ^w supprime l'intégralité de la chaîne juste à gauche du curseur. Si la chaîne est un chemin, zsh va simplement supprimer les caractères jusqu'au '/' le plus proche.

Zsh est aussi capable de proposer des corrections de commande. Ainsi, si je tape systemctlw au lieu de systemd et que cette commande n'existe pas, zsh va me proposer la correction :

zsh: correct 'systemctlw' to 'systemctl' [nyae]?

Autre point très intéressant pour un développeur, si le dossier courant est géré par git, zsh est capable d'afficher diverses informations telles que la branche, s'il y a des fichiers modifiés mais non commités ou encore le nombre de jours depuis le dernier commit. Cette fonctionnalité est aussi disponible pour mercurial mais dans une moindre mesure.

J'espère vous avoir convaincu que la puissance de zsh. Si vous avez très certainement déjà un paquet zsh dans les dépôts de votre distribution, je vous conseille d'utiliser le projet Oh my zsh!. Le but de ce projet est de fournir un framework de configuration pour zsh. Il contient de nombreux plugins (plus de 150 d'après le site officiel) et de nombreux thèmes (plus de 100 d'après le site).

La plupart des options courantes (comme l'activation de l'autocorrection) sont activables via des variables dans le fichier .zshrc. Le système de plugins va vous permettre de facilement utiliser de nouveaux alias (gp pour git push, gup pour git pull --rebase), les informations fournies par git, la possibilité d'ajouter sudo devant une commande en pressant deux fois ESC, de faire automatiquement appel à sudo pour les commandes systemd qui en ont besoin et bien plus !

Pour finir, voilà une petite capture de mon terminal. Il affiche des informations sur l'état d'un dépôt git, un cercle rouge si une commande à un code de retour différent de 0. Il s'agit du thème avit légèrement modifié.

Réaliser des diagramme FAST en LaTeX

2014-08-13T00:00:00+02:00

J'ai récemment eu besoin d'insérer des diagrammes FAST dans un rapport en LaTeX. J'ai d'abord pensé les réaliser sur un logiciel de diagramme comme dia pour l'insérer comme une image et puis j'ai découvert le paquet fast-diagram qui définit un environnent et des commandes LaTeX pour réaliser un joli diagramme FAST facilement.

C'est très simple d'utilisation, on importe le paquet avec \usepackage[raccourcis]{fast-digram} et ensuite :

\begin{fast}{Fonction de service}
       \FT{FT1}
       {
         \FT{FT2}
         {
           \ST{Solution technique}
         }
       }
       \FT{}
       {
         \FT{FT3}
         {
           \ST{ST2}
         }
       }
       \FT{FT4}
       {
         \ST{ST3}
       }
\end{fast}
\fastReset

ce qui donne :

On peut également modifier la largeur des boîtes avec \renewcommand*{\fastLargeurBoite}{8em} par exemple pour une largeur de 8em. Idem pour la hauteur avec \renewcommand*{\fastHauteurBoite}{3em}.

C'est facile et très complet. La documentation complète est là : http://www.tug.org/texlive//devsrc/Master/texmf-dist/doc/latex/fast-diagram/help.pdf

Rédiger ses rapports avec rst et sphinx

2014-08-13T00:00:00+02:00

Comme je l'ai déjà évoqué ici et là, j'adore les langages à balisage léger. Du coup, j'essaie de les utiliser partout où je peux. Mais de là à rédiger un rapport avec table des matières, pages de garde et glossaire, il y a un pas. Surtout face à la puissance de LaTeX.

J'avais déjà évoqué org-mode ici et pourquoi j'envisageais de l'utiliser en lieu et place de LaTeX, à savoir :

Bonne intégration avec Emacs
Syntaxe légère
Possibilité d'export en LaTeX
Possibilité d'insérer du code LaTeX

Mais comme je n'ai pas trouvé de bon support pour org-mode dans Pelican, je m'en suis un peu détourné (je l'utilise toujours pour mes notes) au profit le Restructured Text. Comme ce langage a été conçu pour remplacer LaTeX lors pour la génération HTML de la documentation de python, il est particulièrement puissant. J'ai donc décidé de l'utiliser pour rédiger mon rapport de stage.

Je vais vous présenter tout d'abord Restructured Text (rst) puis sphinx (qui est utilisé pour générer la documentation à partir de fichiers rst) et pourquoi je conseille son utilisation pour un rapport plutôt que juste rst.

Restructured Text

Présentation

La syntaxe est simple. Un bloc de texte constitue un paragraphe. On sépare les paragraphes avec un saut de ligne. On peut écrire en italique et en gras :

*italique* et en **gras**

On peut faire des listes numérotées ou non (attention, les tirets et les dièses doivent être correctement aligné et les sous listes doivent être séparées de la liste par un saut de ligne) :

- item 1
- item 2

  #. Sous-liste numérotée
  #. Un deuxième élément de la sous-liste numérotée

On peut inclure des images :

.. image:: /images/diagramme-fast-LaTeX.png
           :alt: Un diagramme fast réalisé en LaTeX
           :width: 150%

du code :

.. code:: python

          print('Hello World')

Il est tout à fait possible de découper son fichier en plusieurs fichiers plus petits, puis de les inclure avec :

.. include:: fichier.rst

On peut également insérer des fichiers LaTeX avec la directive :

.. raw:: latex
         :file: fast.tex

La syntaxe complète est disponible en ligne. Elle peut paraître déroutante au début, mais on s'y fait et dans l'ensemble, je la trouve très bien faite (de même que la documentation d'ailleurs).

On peut ensuite exporter son fichier en HTML ou LaTeX avec respectivement rst2html et rst2latex.

Rédiger son rapport

Rien de bien fantastique, on entre son texte en respectant la syntaxe, on insère la table des matière avec .. contents:: et on exporte le tout avec rst2latex. On compile ensuite ce fichier avec latexmk. On obtient un fichier Pdf tout ce qu'il y a de plus joli (normal me direz-vous, c'est LaTeX qui l'a fait).

Mais cela ne résout pas tout, comme :

Comment fait-on une page de garde ?
Comment utilise-t-on une autre police ?
Comment insère-t-on un glossaire ?

La réponse aux deux premières questions tient en un mot : template ! En effet, rst2latex dispose d'un argument --template au quel on peut passer un fichier contenant du code LaTeX et des variables rst pour personnaliser le fichier pdf.

Le fichier template de base s'appelle default.tex et se trouve ici : /usr/lib/python3.3/site-packages/docutils/writers/latex2e/default.tex (sur mon PC du moins). Il est assez basique :

$head_prefix% generated by Docutils <http://docutils.sourceforge.net/>
\usepackage{fixltx2e} % LaTeX patches, \textsubscript
\usepackage{cmap} % fix search and cut-and-paste in Acrobat
$requirements
%%% Custom LaTeX preamble
$latex_preamble
%%% User specified packages and stylesheets
$stylesheet
%%% Fallback definitions for Docutils-specific commands
$fallbacks$pdfsetup
$titledata
%%% Body
\begin{document}
$body_pre_docinfo$docinfo$dedication$abstract$body
\end{document}

Il n'y plus qu'à le personnaliser avec du code LaTeX pour qu'il réponde à nos besoins :

$head_prefix% generated by Docutils <http://docutils.sourceforge.net/>
\usepackage{fixltx2e} % LaTeX patches, \textsubscript
\usepackage{cmap} % fix search and cut-and-paste in Acrobat
\usepackage[raccourcis]{fast-diagram}
\usepackage[french]{babel}
\usepackage{titlesec}
\usepackage[a4paper,left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}
$requirements
%%% Custom LaTeX preamble
$latex_preamble
%%% User specified packages and stylesheets
$stylesheet
%%% Fallback definitions for Docutils-specific commands
$fallbacks$pdfsetup
$titledata
%%% Custom font
\usepackage{libertine}
%%% Set numeration
\setcounter{secnumdepth}{3}
%%% Body
\begin{document}
$body_pre_docinfo$docinfo$dedication$abstract$body
\end{document}

Je précise que l'ordre avec lequel vous insérer votre code et les variables est important. Si on met la ligne \usepackage{libertine} (qui change la police) juste après les autres usepackage le choix de ma police sera écrasée par celle de Rst.

Je propose un exemple plus complet en téléchargement (avec notamment une page de garde personnalisée).

On peut également utiliser les options --document-class et --document-options pour personnaliser le document. Il existe bien d'autres options, je vous laisse regarder la sortie de rst2latex -h pour la liste complète.

On générera le fichier .tex avec :

rst2latex
rapport.rst --template=template.tex --documentclass=report --documentoptions=12pt

Et pour le glossaire ? Ce n'est pas possible, faute de syntaxe approprié, directement avec rst. Ça viendra peut être un jour, c'est sur la todo list des dévoleppeurs. C'est là que sphinx entre en jeu.

Sphinx

Présentation

sphinx est un générateur de documentation pour python. Il utilise la syntaxe rst et peut produire des documents HTML, pdf, des pages man, etc. Il propose également des extensions à la syntaxe de base comme term et glossary pour un glossaire et quelques divergences. Par exemple, la table des matières ne se génère plus avec .. contents:: mais avec .. toctree:: et les inclusions de fichier se font avec .. raw::. Typiquement, on utilise un Makefile pour automatiser la génération, Makefile que sphinx est capable de créer tout seul.

Mais rassurez-vous, dans l'ensemble tout fonctionne quasiment comme avec rst. Lorsque je suis passé de rst brut à sphinx, mon seul problème a été l'utilisation des mes entêtes LaTeX personnalisées.

Rédiger son rapport

Il faut tout d'abord initialiser sphinx, avec sphinx-quickstart. Le programme va alors vous poser plusieurs questions pour générer une configuration de base.

Le rapport en lui même se rédige comme sous rst avec les mêmes possibilités d'inclusion de fichiers. Attention toute fois, l'inclusion est relative au Makefile alors qu'aupapravant, elle était relative au fichier d'inclusion.

Lors de mon passage à sphinx, peu importe l'endroit où j'insérais la directive .. toctree:: la table des matières se retrouvait en début de fichier alors que je la voulais après l'introduction et le résumé. Qu'à cela ne tienne, la directive .. raw:: m'a permit d'insérer la commande LaTeX \tableofcontents où je voulais.

.. raw:: latex

         \tableofcontents

Et pour le glossaire alors ? Rien de plus simple :

J'ulitise le :term:`CMS` :term:`Drupal`

.. glossary::

   CMS
     Acronyme de *Content Management System*, soit système de gestion de
     contenu.

   Drupal
     Un des :term:`CMS` les plus connus et réputé. Il est utilisé par de
     nombreuses entreprises et institutions à travers le monde.

Le seul point qui m'a posé problème c'est la configuration. Elle est quasi intégralement géré pour l'export LaTeX par le dictionnaire latex_elements. Par exemple, pour moi :

preamble = r'''\usepackage{fixltx2e} % LaTeX patches, \textsubscript
\usepackage{cmap} % fix search and cut-and-paste in Acrobat
\usepackage[raccourcis]{fast-diagram}
\usepackage{titlesec}
\usepackage[a4paper,left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}
%%% Redifined titleformat
\setlength{\parindent}{0cm}
\setlength{\parskip}{1ex plus 0.5ex minus 0.2ex}
\newcommand{\hsp}{\hspace{20pt}}
\newcommand{\HRule}{\rule{\linewidth}{0.5mm}}
\titleformat{\chapter}[hang]{\Huge\bfseries\sffamily}{\thechapter\hsp}{0pt}{\Huge\bfseries\sffamily}
%%% Custom font
\usepackage{libertine}
%%% Set numeration
\setcounter{secnumdepth}{3}
'''
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
'papersize': 'a4paper',

# The font size ('10pt', '11pt' or '12pt').
'pointsize': '12pt',

# Babel for french
'babel': '\\usepackage[french]{babel}',

# Additional stuff for the LaTeX preamble.
'preamble': preamble,

# No default title
'maketitle': '',

# No default toc
'tableofcontents': '',
}

La classe du document se spécifie dans la variable latex_documents :

latex_documents = [
  ('rapport', 'Rapport.tex', 'Rapport de stage 2A',
  'Julien Enselme', 'manual'),
]

Les classes habituelles de LaTeX ne sont pas compatibles avec sphinx. Il faut soit utiliser celles par défaut (manual pour un rapport ou how-to pour un article) soit les personnaliser.

Je pense qu'à partir de cet exemple, vous pourrez subvenir à vos besoins facilement. Sinon, vous pouvez toujours poster un commentaire.

Vous trouverez dans la documentation complète toutes des options que je n'ai pas pu vous présenter ici.

Je vous propose également un un fichier de configuration complet

Conclusion

J'espère vous avoir convaincu d'utiliser la puissance et la légèreté de rst et sphinx pour rédiger vos rapports, avec un peu de LaTeX brut par endroit (faute de mieux) en lieu et place des WYSIWYG.

Même si abandonner LaTeX peut faire un peu peur et que la configuration de sphinx est un peu ardue sans exemple complet, je ne regrette pas mon choix et suis bien content d'utiliser rst (même si je dois reconnaître qu'il est dommage que sphinx ne puisse utiliser les classes de base de LaTeX) !

Comment supprimer toutes les tables mysql qui contiennent un motif

2014-07-30T00:00:00+02:00

Récemment, j'ai eu besoin de faire du ménage dans les tables d'une base de données. Toutes les tables à supprimer commençaient par le même motif. Malheureusement, mysql ne permet pas d'utiliser le jocker % dans une requête DROP.

Heureusement la solution est simple. Il suffit de lister toutes les tables ayant ce motif grâce à la requête SHOW TABLES LIKE 'motif%'. Reste à boucler sur ces tables pour les supprimer. D'où le code suivant :

motif=del_
for table in $(mysql database -NBe "SHOW TABLES LIKE '$motif%'"); do
  mysql database "DROP TABLE $table"
done

L'option -N de mysql masque les entêtes de colonnes et l'option -B permet d'avoir un résultat brut, exploitable dans la boucle.

On peut également faire le script plus générique suivant :

#!/usr/bin/bash

help="mysql_drop_table_like [options] database pattern

Drop all the mysql tables in database that starts with pattern.

-h: host
-u: user
-pPASSWD: give password
-P: prompt for password
"

usage() {
    echo -e $help
}

pflag=false
Pflag=false
uflag=false
hflag=false

while getopts ":h:u:p:P" opt
do
    case "$opt" in
        h)
            host=$OPTARG; hflag=true;;
        u)
            user=$OPTARG; uflag=true;;
        p)
            passwd=$OPTARG; pflag=true;;
        P)
            Pflag=true;;
        :)
            echo "Option -$OPTARG requires an argument." >&2
            usage; exit 1;;
        \?)
            usage; exit 0;;
    esac
done
shift $((OPTIND-1)) # To get the 1st positional argument with $1

# Check that we have at least 2 positional arguments
if [ $# -lt 2 ] ; then
    echo "Number of positional arguments insuffisant."
    usage
    exit 1
fi

# Cannot provide and ask for password
if $Pflag && $pflag; then
    echo "Cannot provide and ask for password."
    usage
    exit 1
fi

if $hflag; then
    HOST="-h $host"
fi
if $pflag; then
    PASSWD="-p$passwd"
fi
if $Pflag; then
    PASSWD="-p"
fi
if $uflag; then
    USER="-u $user"
fi

mysql="mysql $HOST $PASSWD $USER"
for table in $($mysql $1 -NBe "SHOW TABLES LIKE '$2%'"); do
    $mysql $1 -e "DROP TABLE $table"
done

Créer des pages d'erreurs personnalisées pour son blog pelican

2014-06-29T00:00:00+02:00

Rien de très compliqué. Par exemple, pour les erreurs 404, il suffit :

de créer une page 404.rst avec le status hidden :

:title: 404
:status: hidden

La page demandée n'existe pas. Vous pouvez revenir à `l'accueil <http://www.jujens.eu>`_.

d'ajouter une ligne dans sa configuration nginx :

error_page 404 /pages/404.html;

Et voilà ! Vous pouvez évidement faire de même pour les autres erreurs, mais je doute qu'elle soit aussi courante.

Faire du https avec nginx

2014-06-29T00:00:00+02:00

Sommaire

Générer les certificats
Générer la liste de révocation de certificats (CRL)
Configurer nginx
Importer le certificat au niveau système
- Linux
- Windows
Source

Dans cet article, je ne vais pas du tout rentrer dans les détails de la cryptographie ou de SSL, TLS, https, … Je présente juste comment faire rapidement du https avec nginx. Si vous voulez plus d'informations (ce qui est une bonne idée), je vous invite à consulter le web.

Pour faire du https, nous avons besoin de certificats et de modifier la configuration du serveur web. Nous allons générer nos certificats. Ils seront auto-signés. Quand vous vous y connecterez, vous aurez donc un message d'erreur vous disant que la connection n'est pas de confiance (sauf si vous ajoutez manuellement le certificat à ceux de confiances au niveau système). Ce type de certificat est donc à réserver à son propre usage.

Générer les certificats

Préparation

Pour générer les certificats, nous avons besoin d'OpenSSL. Ce logiciel est présent dans toutes les distributions GNU/Linux dignes de ce nom. Installez le de la façon habituelle.

Nous allons placer les certificats dans un dossier dédié. Ce dossier contiendra 3 sous dossiers :

private pour les clés de notre CA (Certificate Authority)
certs pour les certificats
newcerts pour les nouveaux cerfiticats
crl pour le Ceritificate Revocation List (crl)

On ajoute les deux fichiers :

index.txt qui est utilisé par OpenSSL pour stocker les informations sur les certificats
serial qui contint le numéro des certificats crées. Il est automatiquement incrémenté.

Par exemple :

mkdir -p /root/certs/{private,certs,newcerts,crl}
cd /root/certs
touch index.txt
echo "01" > serial

Toutes les commandes seront à exécuter depuis ce dossier.

On crée le fichier de configuration d'OpenSSL afin de faciliter la génération des clés de la CA dans ca.cnf :

[ ca ]
default_ca      = CA_default            # The default ca section


[ CA_default ]

dir = . # Where everything is kept
certs = $dir/certs # Where the issued certs are kept
crl_dir = $dir/crl # Where the issued crl are kept
database = $dir/index.txt # database index file.
#unique_subject = no # Set to 'no' to allow creation of
# several ctificates with same subject.
new_certs_dir = $dir/newcerts # default place for new certs.

certificate = $dir/certs/ca.crt # The CA certificate
serial = $dir/serial # The current serial number
#crlnumber = $dir/crlnumber # the current crl number
# must be commented out to leave a V1 CRL
crl = $dir/crl.pem # The current CRL
private_key = $dir/private/ca.key # The private key
default_days = 365 # how long to certify for
default_crl_days= 365 # how long before next CRL
default_md = sha512 # use SHA-512 by default
x509_extensions = usr_cert


[ req ]
distinguished_name = req_distinguished_name


[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = FR
countryName_min = 2
countryName_max = 2

stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = PACA

localityName = Locality Name (eg, city)
localityName_default = MARSEILLE

0.organizationName = Organization Name (eg, company)
0.organizationName_default = jujens.eu

organizationalUnitName = Organizational Unit Name (eg, section)
#organizationalUnitName_default =

commonName = Common Name (eg, YOUR name)
commonName_max = 64

emailAddress = Email Address
emailAddress_max = 64
default_days = 365 # how long to certify for


[ policy_anything ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional


[ usr_cert ]
basicConstraints = CA:TRUE
crlDistributionPoints = URI:http://dl.jujens.eu/jujens-eu.crl


[ v3_ca ]


# Extensions for a typical CA


# PKIX recommendation.

subjectKeyIdentifier=hash

authorityKeyIdentifier=keyid:always,issuer

# This is what PKIX recommends but some broken software chokes on critical
# extensions.
#basicConstraints = critical,CA:true
# So we do this instead.
basicConstraints = CA:true

# Key usage: this is typical for a CA certificate. However since it will
# prevent it being used as an test self-signed certificate it is best
# left out by default.
# keyUsage = cRLSign, keyCertSign

# Some might want this also
# nsCertType = sslCA, emailCA

# Include email address in subject alt name: another PKIX recommendation
# subjectAltName=email:copy
# Copy issuer details
# issuerAltName=issuer:copy

# DER hex encoding of an extension: beware experts only!
# obj=DER:02:03
# Where 'obj' is a standard or added object
# You can even override a supported extension:
# basicConstraints= critical, DER:30:03:01:01:FF

Note : le champ commonName de la section policy_anything est indispensable, il ne peut donc pas avoir la valeur optional. Vous pouvez rendre obligatoire n'importe quel autre champ à la création d'un certificat.

Création de l'autorité

Nous créons le certificat de l'autorité qui servira à signer les autres certificats et la clé associé. L'option -nodes permet de ne pas chiffrer la clé privée.

openssl req -new -x509 -extensions v3_ca -newkey rsa:4096 -keyout private/ca.key -out certs/ca.crt -config ca.conf -days 730

Generating a 4096 bit RSA private key
............................................................................................................................++
....................................................................................................................................................................................................................................................................................................................................................++
writing new private key to 'private/ca.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [FR]:
State or Province Name (full name) [PACA]:
Locality Name (eg, city) [MARSEILLE]:
Organization Name (eg, company) [jujens.eu]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:jujens.eu
Email Address []:admin@jujens.eu

req est la commande de génération de certificat
-new pour créer un nouveau certificat
-x509 pour créer un certificat auto signé ce qui permet de créer le certificat d'une CA (ou de faire un certificat de test).
-extensions v3_ca pour utiliser l'extension CA.
-newkey rsa:4096 pour générer une clé rsa de 4096 bits
-keyout private/ca.key dans quel fichier écrire la clé
-out certs/ca.crt dans quel fichier écrire le certificat
-config ca.conf pour utiliser notre fichier de configuration

Ne perdez pas la passphrase (que vous pouvez choisir comme vous voulez).

L'option -pasout pass:x permet de ne pas avoir à entrer de passphrase, openssl va utiliser la passphrase passée en paramètre.

Générer le CSR (Certificate Signing Request)

Puisque le certificat va être utilisé par nginx, il ne faut pas qu'il soit protégé par une passphrase. On utilise presque la même commande que tout à l'heure.

openssl req -new -nodes -newkey rsa:4096 -keyout private/jujens-eu.key -out certs/jujens-eu.csr -config ca.conf -set_serial 10

Generating a 4096 bit RSA private key
.++
...............................................................................................++
writing new private key to 'private/jujens-eu.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [FR]:
State or Province Name (full name) [PACA]:
Locality Name (eg, city) [MARSEILLE]:
Organization Name (eg, company) [jujens.eu]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:*.jujens.eu
Email Address []:admin@jujens.eu

Le champ Common Name doit correspondre au nom de domaine du serveur. Wildcard autorisé.

Retirer la passphrase de la clé

L'inconvénient que l'on a avec cette clé chiffrée avec une passphrase est qu'il faut entrer cette passphrase à chaque fois que l'on démarre le serveur. Ce qui n'est ni pratique ni forcément possible (reboot, crash, …). On peut retirer la passphrase de la clé. Cela peut présenter des risques de sécurité. Cette version de la clé ne doit donc être accessible qu'à l'utilisateur root.

[jenselme@fastolfe tmp]% cp server.key server.key.org

[jenselme@fastolfe tmp]% openssl rsa -in server.key.org -out server.key
Enter pass phrase for server.key.org:
writing RSA key

Vous pouvez également utiliser l'option -passin pass:x pour passer la clé à la commande.

Générer le certificat auto-signé

On génère enfin le certificat auto-signé.

openssl ca -extensions usr_cert -config ca.conf -policy policy_anything -out certs/jujens-eu.crt -infiles certs/jujens-eu.csr

Using configuration from ca.conf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'FR'
stateOrProvinceName   :ASN.1 12:'PACA'
localityName          :ASN.1 12:'MARSEILLE'
organizationName      :ASN.1 12:'jujens.eu'
commonName            :ASN.1 12:'*.jujens.eu'
emailAddress          :IA5STRING:'admin@jujens.eu'
Certificate is to be certified until Aug 20 18:24:46 2016 GMT (365 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

Vous pouvez vérifier votre certificat avec : openssl x509 -noout -text -in server.crt.

Générer la liste de révocation de certificats (CRL)

Cette liste contient des informations sur quels certificats ont été révoqué. Certaines applications (comme les navigateurs web) vont périodiquement télécharger ce fichier pour savoir si un certificat est toujours valide. Si ce téléchargement échoue, le certificat sera considéré comme étant invalide.

La période au bout de laquelle la liste est téléchargée est définie par l'option default_crl_days de la section CA_default dans la configuration. L'adresse où le CRL est disponible est elle définie dans la section usr_cert avec la clé crlDistributionPoints.

Si vous ne voulez pas gérer ce paramètre, configurer simplement la durée du crl comme étant égale ou supérieure à la durée du certificat.

openssl ca -config ca.conf -gencrl -out crl/jujens-eu.crl

Configurer nginx

Pour rediriger le trafic vers https, on utilise une rewrite rule :

server {
       listen 80;
       server_name ttrss.jujens.eu;
       root /var/www/framanews;

       location / {
           rewrite ^ https://$server_name$request_uri? permanent;
       }
}

On ajoute ensuite une entrée serveur qui écoute sur le port 443 et on donne les chemins des certificats. Le reste est identique à la configuration http "standard".

server {
       listen 443 ssl;
       server_name ttrss.jujens.eu;
       root /var/www/framanews;
       index index.php;

       ssl_prefer_server_ciphers on;
       ssl_session_cache shared:SSL:10m;
       ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
       ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
       ssl_certificate /root/certs/certs/jujens-eu.crt;
       ssl_certificate_key /root/certs/private/jujens-eu.key;

       location ~* \.php {
           try_files $uri =404;
           fastcgi_pass unix:/var/run/php5-fpm.sock;
           fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
           include fastcgi_params;
       }
}

L'option ssl_prefer_server_ciphers force l'utilisation d'un algorithme donné par le serveur (par défaut, c'est la préférence du client qui est importante).

Importer le certificat au niveau système

Linux

Récupérer le certificat de votre autorité (ca.crt) et lancer :

trust anchor Downloads/ca.crt
update-ca-trust

Windows

Exportez le certificat puis double cliquez sur celui-ci pour lancer la procédure d'installation. Laissez toutes les options avec les valeurs par défaut.

Source

Certificats (en anglais) : http://www.akadia.com/services/ssh_test_certificate.html
CA (en français) : http://jeyg.info/des-certificats-signes-par-votre-autorite-de-certification-avec-openssl/

Comment sauvegarder une base de données PostgreSQL

2014-06-29T00:00:00+02:00

Je vous propose une solution pour vous aider à sauvegarder votre base de données PostgreSQL. La commande est pg_dump. Elle prend comme argument (au minimum) :

Le nom d'utilisateur
Le mot de passe
Le nom de la base de données
L'adresse IP du serveur

Donc la commande de base pour sauvegarder une base de données est :

pg_dump -h 127.0.0.1 -U owncloud -WMDP owncloud

Le problème est que l'on doit fournir le mot de passe à chaque fois. Heureusement, il y a une autre solution : le fichier .pgpass. Dans votre home, créer un fichier .pgpass.

Chaque ligne de ce fichier contient les informations sur une base de données sous la forme :

IP:Port:Database:Username:Password

Soit par exemple :

127.0.0.1:5432:owncloud:owncloud:MPD

La sauvegarde se lance ensuite avec :

pg_dump -h 127.0.0.1 -U owncloud owncloud

Reste à mettre la commande dans le crontab.

Comment installer framanews sur son serveur avec PostgreSQL et nginx

2014-06-14T00:00:00+02:00

Framanews est un fork de ttrss, un lecteur de flux RSS en ligne. Il vous permet donc de consulter vos flux, de vous y abonner, … directement depuis un navigateur, où que vous soyez. Il dispose aussi d'une application Android. Il est évidement possible d'importer un fichier opml existant pour garder tous ses flux.

Vous pouvez utiliser Framanews sur des serveurs de Framasoft ou sur votre serveur. C'est ce que nous allons faire ici.

Installer les bons paquets

Pour faire fonctionner framanews, nous avons besoin de :

git pour récupérer framanews et ses plugins ou ses thèmes.
PostgreSQL comme système de base de données.
php et de ses extensions pour fonctionner avec PostgreSQL.
nginx (serveur web)

Sur un système basé sur debian, nous pouvons installer tous ça avec :

sudo apt-get install php5 php5-pgsql php5-fpm php-apc php5-curl php5-cli
sudo apt-get install git postgresql nginx

Sur un système basé sur CentOS/RHEL :

yum install postgresql postgresql-libs postgresql-server
yum install php-fpm php-gd php-mbstring php-pgsql php-pdo
yum install git nginx

Reste à activer les bons services (nous les lancerons un peu plus tard) :

service nginx enable
service postgresql enable
service php-fpm enable

Récupérer framanews

Framanews est disponible sur github. Pour le récupérer, placez vous dans le dossier où vous voulez le placer (/var/www/ pour moi) et lancez :

git clone https://github.com/framasoft/framanews_ttrss.git

Framanews sera alors placé dans /var/www/framanews_ttrss ce que je ne trouve pas parlant. Je propose de renommer ce dossier en framanews avec :

mv framanews_ttrss framanews

Ensuite, nous devons donner les droits au serveur pour qu'il puisse faire son travail correctement. La commande peut varier en fonction de votre système. Sous debian et affiliée, l'utilisateur s'appelle www-data et mais apache sous CentOS/RHEL et affiliée. On fera donc au choix :

chown -R www-data:root framanews/

ou :

chown -R apache:root framanews/

Configurer nginx

Rien de très compliqué. Copiez/collez simplement les lignes suivantes dans :

/etc/nginx/conf.d/ttrss.conf sur CentOS/RHEL
/etc/nginx/sites-available/ttrss.conf sur Debian

server {
   listen 80;
   server_name VOTRE_SERVEUR; # ttrss.jujens.eu pour moi
   root /var/www/framanews; # Changer ce chemin au besoin
   index index.php;

   location ~ \.php$ {
       try_files $uri =404;
       fastcgi_pass unix:/var/run/php5-fpm.sock;
       fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
       include fastcgi_params;
   }

Il ne reste plus qu'à lancer nginx et php-fpm :

service nginx start
service php-fpm start

Vous pourrez trouver des informations complémentaires utiles dans mes notes sur nginx.

Avec cette configuration, le trafic (dont votre mot de passe) est en clair. Pour faire du https avec nginx et des certificats auto-signés, je vous propose ce post.

Configurer la base de données

Autoriser la connexion

PostgreSQL possède de nombreux mécanismes d'authentification. Nous allons vérifier que le mécanisme configuré est le bon. Pour cela :

Sous debian, éditez :
```
/etc/postgresql/9.1/main/pg_hba.conf
```
Sous CentOS, éditez :
```
/var/lib/pgsql/data/pg_hba.conf
```

Je vous propose la configuration suivante :

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
# "local" is for Unix domain socket connections only
local        all         all                               ident
# Allow auth with password
#local       all         all                               password
host         all         all         127.0.0.1/32          password
host         all         all         ::1/128               password

Elle autorise :

La connexion par socket local
La connexion à toutes les bases de données pour tous les utilisateurs (utilisateurs de la base de données, pas les utilisateurs du système) sur les adresses 127.0.0.1/32 avec mot de passe.
Idem pour les adresses ipv6

Créer la base de données

Nous allons maintenant créer la base de données et son utilisateur à l'aide des commandes ci-dessous (gardez le nom de la base de données, le nom de l'utilisateur et son mot de passe, nous allons en avoir besoin dans pas longtemps).

root@jujens:/home/jenselme# su - -c "psql" postgres
psql (9.1.13)
Saisissez « help » pour l'aide.

postgres=# DROP USER ttrss;
DROP ROLE
postgres=# CREATE USER ttrss WITH PASSWORD 'SUPER_MDP';
CREATE ROLE
postgres=# CREATE DATABASE ttrss OWNER ttrss ENCODING 'UTF8';
CREATE DATABASE
postgres=# GRANT ALL PRIVILEGES ON DATABASE ttrss TO ttrss;
GRANT

Reste à lancer PostgreSQL :

service postgresql start

Pour sauvegarder, je vous invite à lire cet article.

Finaliser l'installation

Allez sur l'adresse que vous avez choisie pour Framanews (ttrss.jujens.eu pour moi). Il reste à remplir avec les informations demandées (à adapter à votre cas) :

Type de base de données : PostgreSQL
Username : ttrss
Password : SUPER_MDP
Database name : ttrss
Port : 5432

On teste la configuration :

Et on sauvegarde le fichier de conf :

Et voilà !

Je vous laisse faire un tour dans les options pour adapter Framanews à vos besoins.

Mise à jour des flux

Par défaut, les flux ne se mettent pas à jour via l'interface web. La solution recommandée est d'entrer la ligne suivante dans un terminal :

su www-data -c './update.php --daemon& > /dev/null&'

Malheureusement, dès que vous quitterez le terminal, le programme sortira. Je propose donc plutôt d'utiliser le crontab. Pour éditer le crontab, tapez (en tant que root) :

crontab -e

et ajouter la ligne suivante pour une mise à jour toutes les 30 minutes (le chemin est à adapter) :

*/30 * * * * su www-data -c '/usr/bin/php    /var/www/framanews/update.php --feeds --quiet'

Pour information : cette ligne lance en tant que l'utilisateur www-data le script update.php.

Si vous n'avez pas de crontab, vous pouvez utiliser la mise à jour en arrière plan. Framanews va alors tenter de mettre à jour les flux tant qu'il est ouvert dans votre navigateur. Pour cela, dans /var/www/framanews/config.php mettre SIMPLE_UPDATE_MODE à TRUE.

Plugins et thèmes

ttrss (et donc Framanews) peut être étendu grâce à des plugins et son pas très joli thème de base peut être changé. La liste des plugins est disponible ici et celle des thèmes là. Installer un thème ou un plugin n'est pas compliqué.

Plugins

Par exemple, pour le plugin framarticle_toolbar, il suffit de :

Se placer dans le dossier plugin de votre installation de Framanews (/var/www/framanews/plugin pour moi.

Lancer :

git clone https://github.com/framasoft/framarticle_toolbar.git
chown -R www-data:root framarticle_toolbar/

On l'active dans Actions > Préférences > Préférences > Plugin.

Thème

On procède comme pour un plugin, mais en se plaçant dans le dossier theme. On active le thème dans Actions > Préférences directement dans l'onglet préférence.

Sources

Le site officiel de framanews : https://framanews.org/install.php
Le site officiel de Tiny Tiny RSS : http://tt-rss.org
How To Install TTRSS with Nginx for Debian 7 on a VPS : https://www.digitalocean.com/community/tutorials/how-to-install-ttrss-with-nginx-for-debian-7-on-a-vps
Tiny Tiny RSS on CentOS with nginx and MYSQL : https://www.zufallsheld.de/2013/11/03/tiny-tiny-rss-centos-nginx-mysql/
Changing the tt-rss database from MySQL to Postgresql : https://www.zufallsheld.de/2014/02/25/changing-the-tt-rss-database-from-mysql-to-postgresql/

Utiliser isso pour avoir des commentaires auto-hébergés

2014-06-08T00:00:00+02:00

Lorsque j'ai lancé ce blog, j'utilisais un plugin de pelican pour avoir un système de commentaires statiques. Je ne suis en effet pas un grand fan des commentaires hébergés chez des prestataires externes comme disqus pour des questions de vie privée. Ce plugin me semblait un bon compromis : il ajoute un formulaire grâce à du javascript dans lequel on peut entrer son commentaire qui est ensuite envoyé par mail à l'administrateur du site. Celui-ci doit ensuite copier/coller le contenu du commentaire dans un fichier dédié.

Cela présente l'avantage d'être simple et respectueux de la vie privée bien qu'un peu fastidieux pour l'administrateur. Mais quand on a pas beaucoup de commentaire, ce n'est pas très grave et ça permet de les modérer.

Je me disais que c'était très bien et puis récemment, je suis tombé sur cet article qui présente issos. Isso est un petit logiciel qui vous permet de stocker les commentaires dans une base de données sqlite et de les enregistrer/afficher grâce à un peu de javascript. Cela vous permet de combiner les avantages de l'auto-hébergement et de l'aspect dynamique d'un service comme disqus. Petit plus sympa, isso peut vous envoyer un mail à chaque commentaire ajouté avec la possibilité de les modérer.

C'est simple, efficace et d'après mes tests ça marche plutôt bien. Évidement, ce n'est pas utilisable si vous avez vraiment beaucoup de commentaires, ça reste du sqlite ;-). Il manque également une vraie interface d'administration pour gérer les commentaires.

Voilà un exemple basique de configuration.

[general]

# File location to the SQLite3 database, highly recommended
# to change this location to a non-temporary location!
dbpath = /var/lib/isso/comments.db

# URL to your website. When you start Isso, it will probe
# your website with a simple GET / request to see if it can
# reach the webserver. If this fails, Isso may not be able
# check if a web page exists, thus fails to accept new
# comments.
# You can supply more than one host.
host = http://www.jujens.eu

notify = smtp

[server]
listen = http://localhost:8080/

[moderation]

# Enable comment moderation queue. This option only affects
# new comments. Comments in modertion queue are not visible
# to other users until you activate them.
enabled = false

Pour lancer isso en arrière plan avec sa configuration :

/usr/local/bin/isso -c /etc/isso.conf run &

Pour en savoir plus

** L'article cité plus haut : http://hackriculture.fr/isso-commentaires-auto-heberges-pour-pelican-et-autres-sites-statiques.html
** Le site officiel d'isso : http://posativ.org/isso/

Markdown vs creole vs Restructured Text vs org-mode

2014-05-10T00:00:00+02:00

Comme je l'ai déjà détaillé ici, je ne suis pas un grand fan des syntaxes verbeuses à la LaTeX ou HTML et je préfère les langages à balisages légers (et oui, je ne suis pas non plus fan des WYSIWYG). Comme j'ai eu l'occasion d'en tester plusieurs ces derniers temps, je vous propose une petite comparaison.

Markdown

Peut être le plus connus et le plus utilisé. On le retrouve sur Github, Stackoverflow, pelican et dans les commentaires de ce blog :D. La syntaxe est simple ce qui doit expliquer sa popularité.

Malheureusement, le markdown de base n'est pas très complet. Il ne gère pas les tableaux par exemple. Certes, on peut insérer du HTML dans du code markdown pour palier ce problème mais bon, le but c'est justement de pouvoir utiliser quelque chose de plus léger que le HTML. Cela a donc assez logiquement entraîné la création de diverses extensions comme le markdow à la sauce github qui ne sont évidement pas toujours compatibles entre elle.

Ce langage contient aussi un point que je trouve particulièrement mal fait : pour insérer un saut de ligne, on peut ajouter deux espaces à la fin de la dite ligne. Pour moi c'est très mauvais : on ne voit pas si on a modifié ces espaces et tous mes éditeurs de texte sont paramétrés pour enlever les espaces de fin de ligne.

Mais ce qui m'a fait fuir ce langage c'est la possibilité d'ajouter du code que je trouve vraiment pourrie. On peut ajouter du code, soit en l'entourant avec ~~~ soit en ajoutant deux tabulations devant. J'avais déjà testé la syntaxe avec les tildes et ça marchait plutôt bien. Puis en voulant ajouter du code sur ce blog j'ai eu des problèmes pour activer la coloration syntaxique si l'extension table des matières est active. La syntaxe avec les tildes ne fonctionnait plus et j'ai bien galéré avec l'autre. Du coup, j'ai regardé d'autres syntaxes.

Restructured Text

L'autre syntaxe bien supportée par Pelican est le Restructured Text (rst en abrégé). C'est beaucoup plus complet que markdown et ça supporte beaucoup de chose : ajouter des propriétés aux images insérées, les références, sommaire et index… Il faut dire que le langage a été conçu pour remplacer LaTeX pour la documentation de python. Donc, mécaniquement, il faut pas mal d'options. Et je dois dire que le résultat, du point de vue des fonctionnalités offertes en est un concurrent sérieux pour l'export en html. LaTeX garde l'avantage pour l'export pdf car c'est aussi un système de mise en page très pointu (ce dont on a pas forcément besoin pour autre chose que la page de garde cela dit).

Pour être franc, mes premiers tests avec rst n'ont pas été concluant du tout. Premièrement, le fait qu'il soit nécessaire de souligner ces titres avec =, -, ou + suivant le niveau ne me plaît pas particulièrement, je trouve ça un peu lourd. Ici c'est d'autant plus lourd que le soulignement doit faire exactement la taille du titre, sinon rst génère une erreur.

Ensuite, les sous-listes doivent être séparées de la liste parente par un saut de ligne et la puce doit être indentée précisément sous le premier caractère du texte de la liste parente. Le calvaire des indentations en python mais en pire !

Enfin, les tableaux doivent être fait intégralement à la main.

Et puis j'ai trouvé cet article (en) qui m'a réconcilié avec le format. J'utilise désormais le rst-mode pour emacs qui permet de placer le bon nombre de caractères sous les titres avec ^u = , fait l'indentation tout seul. J'ai aussi appris que emacs est capable sans plugin supplémentaire de créer des tableaux ascii compatible avec rst. J'ai donc retenté l'expérience et ce fut concluant. Maintenant, j'admets, j'aime bien et j'ai pris plaisir à écrire mon tuto sur la communication série en arduino avec. Par contre, compte tenu de sa relative verbosité, je le réserve pour les longs articles ou les tutos.

Creole

Encore un autre langage à balise. Il a été créé dans le but d'unifier les langages existants. Il est léger et plutôt complet. Voir par là pour la syntaxe. J'admets avoir été séduit par ce langage simple et efficace. J'ai même fait le plugin pelican pour l'utiliser. Je l'utilise pour les articles de blog assez courts qui n'ont pas besoin de la puissance de rst.

Org-mode

Je n'ai pas grand chose de plus à dire par rapport à mon dernier article : de LaTeX à org-mode. C'est complet et bien intégré avec emacs. Si je m'en suis un peu détourné c'est à cause du manque d'implémentation dans pelican. Il faut impérativement passer par emacs et on perd la possibilité d'utiliser le système de cache de pelican.

Conclusion

Le moins que l'on puisse dire c'est qu'on a énormément de choix. Je suis personnellement très déçu par mardown et j'ai du mal à comprendre pourquoi il est si populaire alors que d'autres langages comme creole sont vraiment bien.

Je pense également utiliser rst pour mes prochains rapport plutôt que LaTeX ou org-mode. Je trouve ce format mieux fait que org-mode bien qu'un peu plus verbeux. Mais je vais certainement continuer à utiliser org-mode pour mes notes.

Ajouter un dépôt git distant

2014-05-09T00:00:00+02:00

Un cas d'utilisation typique est lorsque vous avez forké un dépôt sur github et que vous voulez le synchroniser avec le dépôt initial. Commençons par lister les sources disponibles et ajouter la nouvelle source avec la commande git remote add NOM URL.

[jenselme@fastolfe SimpleCV]% git remote -v
origin        git@github.com:Jenselme/SimpleCV.git (fetch)
origin        git@github.com:Jenselme/SimpleCV.git (push)

[jenselme@fastolfe SimpleCV]% git remote add upstream https://github.com/sightmachine/SimpleCV.git

[jenselme@fastolfe SimpleCV]% git remote -v          [git][SimpleCV/.][develop]
origin        https://github.com/Jenselme/SimpleCV.git (fetch)
origin        https://github.com/Jenselme/SimpleCV.git (push)
upstream      https://github.com/sightmachine/SimpleCV.git (fetch)
upstream      https://github.com/sightmachine/SimpleCV.git (push)

Il ne reste plus qu'à mettre à jour le dépôt.

fetch

jenselme@fastolfe SimpleCV]% git fetch upstream
remote: Counting objects: 155, done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 155 (delta 102), reused 130 (delta 83)
Receiving objects: 100% (155/155), 208.82 KiB | 237.00 KiB/s, done.
Resolving deltas: 100% (102/102), done.
From https://github.com/sightmachine/SimpleCV
* [new branch]      AVT-timing -> upstream/AVT-timing
* [new branch]      SCV-1      -> upstream/SCV-1
* [new branch]      develop    -> upstream/develop
* [new branch]      gsoc       -> upstream/gsoc
* [new branch]      master     -> upstream/master
* [new branch]      njotask/JNAPC-502 -> upstream/njotask/JNAPC-502
* [new branch]      task/DEMOS-65 -> upstream/task/DEMOS-65
* [new branch]      task/JNAPC-520 -> upstream/task/JNAPC-520

merge

[jenselme@fastolfe SimpleCV]% git branch         [git][SimpleCV/.][develop]
* develop

[jenselme@fastolfe SimpleCV]% git checkout develop

[jenselme@fastolfe SimpleCV]% git merge upstream/develop
Merge made by the 'recursive' strategy.
README.markdown        | 32 +++++++++++++++++++++++++-------
SimpleCV/ImageClass.py |  2 +-
2 files changed, 26 insertions(+), 8 deletions(-)

Source : https://help.github.com/articles/syncing-a-fork

Une petite sélection de jeu libre pour se détendre

2014-05-07T00:00:00+02:00

Durant les dernières vacances, j'ai voulu me détendre avec quelques jeux
libres. J'en ai retenu 3 qui sont simples et distrayants. J'ai donc exclu 0ad et
Battle for Wesnoth qui sont des jeux de stratégie un peu trop « prise
de tête », pas assez simples (mais très bons quand même).

SuperTuxKart

C'est un jeu de course qui ressemble au célèbre Mario Kart. On y joue seul ou à
plusieurs (pas pu tester faute de manette). On a le choix entre plusieurs karts
qui s'inspirent tous d'un projet libre. On retrouve donc Tux, Beastie,
ElePHPant, … Le jeu dispose d'un mode histoire : le but étant de gagner les
courses proposées pour libérer GNU. Le scénario n'est donc pas très profond,
mais ce n'est pas ce qu'on lui demande.

Le jeu contient tous les éléments qui font qu'un jeu de course comme ça est bien
: nitro, trucs à lancer sur les adversaires (ballon de basket, cake explosif,…),
plutôt bons graphismes et de bonnes musiques qui s'accordent bien avec la course.

Captures

Liens et téléchargements

Site officiel : http://supertuxkart.sourceforge.net/
Téléchargement (Disponible dans les dépôts de la plupart des distributions) : http://supertuxkart.sourceforge.net/Downloads

Funguloïds

C'est un jeu assez étrange. Vous êtes une sorte de comète et vous devez
collecter des champignons de couleur pour les ramener sur leur platforme tout en
évitant des astéroïdes. Vous gagnez alors des points et changer de carte. Cela
étant dit, le jeu est vraiment très beau graphiquement et sa musique calme vous
permettra de vous reposer tout en vous amusant.

Captures

Liens et téléchargements

Site officiel : http://funguloids.sourceforge.net/
Téléchargement (Disponible dans les dépôts de la plupart des distributions) : http://funguloids.sourceforge.net/download.html

SuperTux 2

Pour faire simple : c'est Tux qui se prend pour Mario sur la banquise. C'est un
bon jeu de platforme, simple mais efficace.

Captures

Liens et téléchargements

Site officiel : http://supertux.lethargik.org/
Téléchargement (Disponible dans les dépôts de la plupart des distributions) : http://supertux.lethargik.org/download.html

La communication série avec arduino

2014-05-05T00:00:00+02:00

La communication série est indispensable pour dialoguer avec votre Arduino puisque c'est le mode de transmission utilisé pour communiquer avec la carte. Dans ce tuto je vais vous expliquer comment ça fonctionne avec des exemples pratiques pour faire communiquer une carte arduino (le modèle n'importe pas) et votre PC grâce à une connexion USB. Nous utiliserons dans un premier temps l'IDE Arduino puis du code Python. Je suppose que vous avez quelques bases en programmation. Les exemples seront suffisamment simples pour que la connaissance préalable de python ne soit pas nécessaire.

Le tuto peut se réaliser indifféremment sous Windows, GNU/Linux, Mac ou autre. Néanmoins, les exemples fonctionnent tel quel uniquement sous linux. Sous d'autres systèmes, vous pouvez être amené à changer les chemins des ports. Le reste devrait rester inchangé.

Tous les codes sont disponibles en téléchargement à la fin de ce document sous licence Apache v2.

Sommaire

Présentation

En communication série, on découpe l'information à transmettre en petits blocs de taille fixe avant de la transmettre. La taille des blocs correspond au nombre des lignes disponibles pour la transmission des données.

Ce type de communication s'oppose à la communication parallèle. En communication parallèle, il y a une ligne par bits à transmettre. Tous les bits sont donc transmis en même temps. Pour une même fréquence de communication, la communication parallèle est donc plus rapide.

L'avantage de la communication série sur la communication parallèle est qu'elle nécessite moins de lignes, donc moins de broches, donc moins de composants. Son coût est donc plus faible.

Les protocoles de communication série les plus connus sont :

Le protocole USB
Le protocole I2C
Le protocole PCI Express

Pré-requis

Logiciels et bibliothèques à installer

Arduino IDE : l'IDE officiel arduino. Le moyen le plus simple d'écrire et de télécharger les programmes sur la carte. Il est disponible sur le site officiel d'Arduino : http://arduino.cc/en/Main/Software#toc1
Python3 : indispensable pour exécuter les certains programmes d'exemple (uniquement pour la partie communication entre un programme et la carte). Il se trouve dans les dépôts de votre distribution linux ou par là : Python
Pyserial : c'est la bibliothèque python qui permet de faire de la communication série. Elle est normalement dans les dépôts de votre distribution linux. Elle peut aussi être installée via pip : pip install pyserial.

Vocabulaire

Commençons par définir quelques termes de vocabulaire avec lesquels vous pouvez ne pas être familier et que nous allons rencontrer tout au long de ce tuto :

Ascii : c'est l'acronyme de American Standard Code for Information Exchange, soit code américain pour l'échange d'information. Ce code permet de représenter avec des nombres compris entre 0 et 127 les caractères utilisés dans l'alphabet américain soit :
- Les lettres, minuscules et majuscules
- Les symboles de ponctuation : . , ;
- Des caractères spéciaux : $ # * +
- Pas les accents
Baud (Bd) : Unité de mesure du nombre de symboles transmissibles par seconde. Il ne faut pas le confondre avec le nombre de bits par seconde (bps). Par exemple, considérons que l'on souhaite transmettre le symbole ascii a en 1 seconde exactement. D'après la table ascii a s'écrit en binaire 01100001. Pour transmettre ce symbole en 1 seconde, il faut communiquer avec une vitesse de 1 Bd mais de 8 bps.
Bytes (en python3) : Petite précision de vocabulaire spécifique à python 3. C'est un type de base du langage qui représente une chaîne d'octets. C'est ce type de donnée que nous manipulerons dans communication entre un programme et la carte

Communiquer avec le moniteur série

Avant de rentrer vraiment dans la communication, je vous signale que deux diodes présentes sur la arduino, peuvent être utiles :

TX : s'allume lors d'une transmission
RX : s'allume lors d'une réception

Communication la plus simple possible

Nous allons aborder notre premier exemple. Il consiste à envoyer des données sur le port série de la arduino de d'utiliser l'outil intégré à l'IDE pour voir les données envoyées. Lancez l'éditeur arduino et entrez le code ci-dessous :

void setup() {
  Serial.begin(9600);
}

void loop() {
  Serial.println("Coucou");
}

Détaillons le :

Dans la fonction setup, on initialise simplement le port série en donnant sa vitesse avec Serial.begin(9600). En effet, pour que la carte et l'ordinateur communique correctement, ils doivent « parler » à la même vitesse. Une vitesse différente est une erreur fréquente au début. Pensez à la vérifier si vous rencontrez des problèmes. On choisit ici 9600 Bd ce qui est une valeur classique. Nous verrons plus loin l'influence du nombre de bauds.
Dans la fonction loop, on se contente d'envoyer "Coucou" avec Serial.println("Coucou").

Vérifiez dans Outils > Carte que la bonne carte est sélectionnée et dans Outils > Port série que le bon port est choisi. Si tout est bon, utilisez le bouton Téléchargements pour envoyer le code sur la carte.

Ouvrez le moniteur série avec Outils > Moniteur série. Vérifiez en bas à droite que la valeur est bien à 9600 Bd. Si tout va bien, vous devriez voir le moniteur se remplir de Coucou.

Si ce n'est pas le cas, reprenez les étapes précédentes, vous avez raté quelque chose. Vous pouvez aussi regarder du côté des quelques problèmes fréquents.

Echo

Après cet exemple d'autant moins intéressant qu'il ne fonctionne que dans un sens, faisons un programme d'écho : la arduino va nous renvoyer exactement ce qu'on lui envoie. Si elle ne reçoit rien, elle envoie Rien. Voilà le code de la arduino :

void setup() {
  Serial.begin(9600);
}

void loop() {
  if ( Serial.available() ) {
    int lu = Serial.read();
    Serial.println(lu);
  } else {
    Serial.println("Rien");
  }
  delay(2000);
}

Regardons le d'un peu plus près. Pas de changement du côté de setup. Dans loop :

Nous commençons par vérifier si des données en attente de lecture sur le port avec : Serial.available(). Cette fonction renvoie true si telle est le cas.
Si on a des données, on définit une variable de type int dans laquelle on place une donnée en attente. Puis on affiche cette donnée.
Sinon, on affiche Rien.
Enfin, on attend 2 secondes histoire d'avoir le temps de voir ce qui se passe.

Testez ce code. Pour envoyer une donnée à la carte, tapez du texte dans la barre du haut puis presser Entrée ou appuyez sur Envoyer. Vous devriez obtenir quelque chose comme ça (j'ai tapé coucou) :

Mais ce n'est pas ce que j'ai tapé ! En effet, Serial.read() lit un octet de données sur le port série. Ni plus, ni moins. Et c'est la seule manière de lire les données. Mais rassurez-vous, vous pouvez facilement donner un sens à ce qui s'affiche : les données sont encodées en ascii. Par conséquent, le numéro qui s'affiche est la version décimale du code ascii. Vous pouvez consulter la table pour convertir ce code dans le caractère standard.

Lire des entiers

Dans cette partie, nous allons utiliser la table ascii pour convertir le nombre reçu en quelque chose d'exploitable. Nous ferons tout d'abord un programme qui lit et affiche correctement un nombre puis un second programme qui fera des opérations sur deux entiers que vous choisirez.

Lecture basique d'un entier

Ce n'est pas très compliqué. Il suffit de prendre la table ascii et de constater que les chiffres sont codés en décimal entre 48 et 57 puis de décaler.

int byte_read = 0; ///< The current byte
int recieved_integer = 0; ///< The current recieved integer

void setup() {
  Serial.begin(9600);
}

boolean is_a_number(int n)
{
  return n >= 48 && n <= 57;
}

int ascii2int(int n, int byte_read)
{
  return n*10 + (byte_read - 48);
}

void loop() {
  recieved_integer = 0;
  while ( Serial.available() ) {
    byte_read = Serial.read();
    if ( is_a_number(byte_read) ) {
          recieved_integer = ascii2int(recieved_integer, byte_read);
    }
  }
  Serial.println( recieved_integer );
  delay(1000);
}

Détaillons ce programme :

On définit les variables globales en début de programme. Ça ne sert à rien de les redéfinir à chaque tour de boucle.
Ensuite on définit deux fonctions :
- boolean is_a_number(int n) qui renvoie vrai si on lui donne un nombre ascii codé en décimal.
- int ascii2int(int n, int byte_read) qui renvoie le nombre sous la forme d'un entier. Elle prend en paramètre l'entier déjà calculé ainsi que l'octet à ajouter.
Dans loop : on fait la conversion si un entier est disponible.

Je vous laisse constater que ça marche bien.

Faire des maths avec deux entiers

Maintenant on va faire un peu mieux : lire deux entiers séparés par une virgule et les additionner. Le code est ci-dessous :

const int baudrate = 9600;
int byte_read = 0; ///< The current byte read.
int coords[2]; ///< Contains x and y.
int index = 0; ///< 0: reading x, 1: reading y.
int separator = 44; ///< The separator between the integers (44: ,)

void setup()
{
    Serial.begin(baudrate);
}


boolean is_a_number(int n)
{
  return n >= 48 && n <= 57;
}


int ascii2int(int n, int byte_read)
{
  return n*10 + (byte_read - 48);
}


void loop()
{
  coords[0] = 0;
  coords[1] = 0;
  index = 0;
  while ( Serial.available() > 0 )
    {
      byte_read = Serial.read();
      if ( is_a_number(byte_read) )
        {
          coords[index] = ascii2int( coords[index], byte_read );
        }
      else if ( byte_read == separator )
        {
          ++index;
        }
    }

  if ( index )
    {
      Serial.print("x + y = ");
      Serial.println(coords[0] + coords[1]);

      Serial.print("x * y = ");
      Serial.println(coords[0] * coords[1]);
    }
  else
    {
      Serial.println();
    }
  delay(1000);
}

Décortiquons :

Concernant les variables :
- Nos deux nombres seront stockés dans un tableau d'entier.
- index nous permet de savoir si on lit x ou y.
Dans loop :
- On prend soin de remettre à 0 les variables pour éviter les problèmes.
- Puis suivant si l'octet lut représente un nombre ou le séparateur, on lit un nombre ou on incrémente l'index.
- Concernant les calculs : on vérifie que index n'est pas à 0 et donc qu'on a bien lut deux nombres. Si tel est le cas, on fait l'addition et la multiplication de x et y et on affiche le résultat. La fonction print affiche ses paramètres sans revenir à la ligne comme println.

Vous pouvez vous amuser à écrire une calculatrice plus complète si vous voulez. Vous pouvez même envoyer l'opération à effectuer si vous voulez. Utilisez éventuellement l'instruction switch.

Communication entre un programme et la carte

Cette partie reprend en partie ce qui a déjà été évoqué ici mais de façon plus détaillée et un peu moins complète.

Lecture

Nous allons maintenant faire communiquer un programme de votre ordinateur (autre que l'IDE Arduino) et la carte. Le code sera écrit directement dans l'interpréteur, mais vous pouvez sans problème le mettre dans un fichier puis l'exécuter. Il sera écrit en python, mais les principes restent les mêmes dans les autres langages. Reprenons le code « echo » pour la carte :

void setup() {
  Serial.begin(9600);
}

void loop() {
  if ( Serial.available() ) {
    int lu = Serial.read();
    Serial.println(lu);
  } else {
    Serial.println("Rien");
  }
  delay(2000);
}

Lancez l'interpréteur python sur votre PC et importer la bibliothèque pyserial :

Python 3.3.2 (default, Mar  5 2014, 08:21:05)
[GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from serial import Serial

Créez la communication série avec (en remplaçant /dev/ttyACM0 par le nom de votre port) :

>>> serial_port = Serial(port='/dev/ttyACM0', baudrate=9600)

Pour lire des données, rien de plus simple :

>>> serial_port.read()
b'R'
>>> serial_port.readline()
b'Rien\r\n'

Les résultats sont de type bytes. Si vous comptez faire des manipulations par la suite (hors transmission série), je vous conseille de passer dans le type string plus adapté (ou un autre type qui représente correctement vos données) :

>>> lu = serial_port.readline()
>>> lu
b'Rien\r\n'
>>> type(lu)
<class 'bytes'>
>>> chaine = lu.decode('ascii')
>>> chaine
'Rien\r\n'
>>> type(chaine)
<class 'str'>

Écriture

Pour écrire des données c'est aussi très simple :

>>> while True:
...     nombre = input('Entrez un chiffre : ')
...     serial_port.write(nombre.encode('ascii'))
...     serial_port.readline()
...
Entrez un nombre : 78
1
b'55\r\n'

Le nombre 1 est la valeur de retour de serial_port.write qui renvoie la longueur de la chaîne écrite.

Influence du nombre de bauds

Comme vous avez dû le comprendre en lisant vocabulaire, plus le nombre de bauds est élevé, plus on peut transmettre de données dans un temps court. Nous allons illustrer ceci dans cette partie. Télécharger tout d'abord le fichier donnee.txt qui contient le texte à envoyer. Enregistrez de façon à pouvoir l'ouvrir facilement avec l'interpréteur python. Programmez votre arduino avec le code suivant :

void setup() {
  Serial.begin(117500);
}

void loop() {
  while ( Serial.available() ) {
    Serial.read();
  }
}

Puis dans l'interpréteur python, tapez :

>>> port = Serial(port='/dev/ttyACM0', baudrate=117500)
>>> f = open('donnees.txt', 'r')
>>> data = f.read()
>>> a.write(data.encode('ascii'))

L'interpréteur devrait répondre rapidement 8000. Remplacez maintenant 117500 par 4800 dans les deux codes. C'est beaucoup plus long, non ?

Le principal est d'utiliser la bonne vitesse de transmission pour son application. Avec une grande vitesse, vous transmettez plus d'informations mais cela peut poser des problèmes sur les grandes distances. À vous de tester et de choisir la bonne, sachant qu'en général sur des cas simples, 9600 Bd est suffisant.

Conclusion

À partir de ces informations, vous devriez pouvoir faire ce que vous voulez tant que ce n'est pas trop compliqué. Je publierais sûrement d'ici quelques temps un tuto plus complet sur pyserial. En attendant, vous pouvez aller consulter la documentation.

En résumé

Choisir la bonne vitesse de communication.
Que les deux programmes qui communiquent le fasse à la même vitesse.
Se souvenir que l'arduino lit les données en code ascii.
Ne pas désespérer si ça ne fonctionne pas.

Quelques problèmes fréquents

Problème de téléchargement

Ils sont malheureusement courants et je n'ai pas de méthode miracle pour les régler. Vous pouvez :

Vérifiez le port de communication
Débranchez/Re-branchez la carte
Relancez l'éditeur

Lecture de données de taille indéterminées

Lors de tous nos tests de lecture, nous avons utilisé soit des int soit des tableaux de taille fixe de int. Vous ne pouvez pas lire des données de taille indéterminée avec la arduino. Par exemple, faire :

String data = Serial.read();

En effet, contrairement à votre ordinateur qui a beaucoup de mémoire, une arduino est beaucoup plus limitée : pas plus de quelques kilos octets. La lecture de données sous cette forme empêche le compilateur de connaître la taille nécessaire pour la donnée et va donc refuser de compiler le code.

Dans tous les cas, souvenez-vous que la mémoire est très limitée et que vous ne pourrez donc pas tout traiter.

Téléchargements

Tout est sous licence Apache v2.

L'archive avec tous les codes : codes.zip

Mon passage à Pelican

2014-04-22T00:00:00+02:00

Présentation

Pelican est un générateur de site statique écrit en python. Un générateur de site statique est simplement un programme qui transforme des fichiers textes en un site web avec thème, catégorie, … Les plus connus sont jekyll qui est écrit en ruby, middleman en ruby également, octopress qui est basé sur jekyll et nanoc. Mais il y en a beaucoup d'autres.

L'avantage de ce genre de site est qu'ils sont simples, qu'ils n'ont pas besoin de bases de données ou de langages particuliers comme PHP sur votre serveur. Il faut juste un serveur web qui délivre des fichiers. Et ça, ils sont tous capables de le faire. J'ajouterais également la simplicité de rédaction : pas d'éditeur WYSIWYG, juste du texte brut et une syntaxe légère pour y donner du sens. Cela implique également que l'on peut utiliser un gestionnaire de version comme git ou mercurial pour gérer les fichiers (et pour en faire des sauvegardes facilement).

Avant de passer à Pelican, j'ai pu tester Jekyll. Les deux sont très proches, très complets et extensibles via des plugins. Si j'ai préféré Pelican c'est parce que :

Il est en python ce qui me permettra de faire des modifications ou des plugins au besoin (je ne connais pas suffisament ruby pour ça).
Il a un système de cache ce qui évite de devoir recompiller tout les fichiers à chaque fois.

L’installation

Tout est très bien détaillé dans la documentation officielle. J'ai choisi la méthode recommandé : dans un enviromment virtuel, en python 3 (pour être moderne) et à partir de la version git (le plugin des commentaires statiques ne fonctionne pas bien sur la dernière version publiée).

Attention toute fois, en python 3 la commande virtualenv ne fonctionne pas comme il faut. J'ai utilisé la commande python3 -m venv blog pour créer cet environement dans le dossier blog. Le module venv vient par défaut avec python 3. Reste à installer pelican et toutes les bibliothèques avec pip install <bibliothèque> . Pelican a nottament besoin de Markdown. Les autres sont installées comme dépendance par pip. Le support d'Asciidoc demande un peu plus de travail. Allez voir la doc pour plus d'informations.

Ensuite, la commande pelican-quickstart va vous aider à générer la configuration du site. Vous allez devoir répondre à plusieurs questions pour que le site vous corresponde. Point intéressant, cette commande va aussi créer un Makefile pour vous aider à automatiser le processus de création de site. Plus d'infos ci-dessous.

Le site

Par défaut, Pelican supporte les articles écrits en markdown, RestructuredText ou dans une moindre mesure Asciidoc. On écrit simplement l'article dans le dossier content (si vous avez lancez pelican-quickstart ) et on l'enregistre avec la bonne extension (.md pour du markdown ou .rst pour RestructuredText).

Une fois l'article écrit, on lance la commande pelican chemin/vers/votre/article et pelican va générer votre site statique dans un dossier nommé output. Si vous avez un Makefile, un simple make html suffira. Pour visualiser votre site sur votre ordinateur, make serve va lancer un serveur sur http://localhost:8000 . Simple et efficace.

Les plugins et les thèmes

Un point intéressant : les fonctionnalités de Pelican peuvent être étendu via des plugins. Cela va de l'ajout d'un système de commentaires statiques au rendu des équations LaTeX en passant par une gallerie. Allez par là pour en savoir plus.

J'ai d'ailleur écrit mon propre plugin : creole-reader qui ajoute le support de la syntaxe creole dans pelican. Je ne suis en effet pas un grand fan de Markdown et la syntaxe Rest ne me plait que moyennement. Je crérais peut être aussi un plugin pour org-mode si je n'arrive pas à me faire à Rest (creole est beaucoup moins complet et trop limité pour des long tutos).

Si vous pouvez créer facilement des thèmes à partir du moteur de template Jinja2, vous pouvez aussi prendre un des nombreux thèmes disponibles ici. J'ai choisi dev-random2 (que j'ai un peu adapté).

Conclusion

Mon passage à Pelican est récent mais je suis vraiment séduit par l'outil. Auparavant, mon blog était en Drupal. Cela peut paraître étrange, mais je gère une installation multisite de Drupal, donc ce n'était pas un problème. Par contre, Drupal manque de souplesse pour un blog. D'autant que sur notre installation, c'est soit du texte brut soit du HTML (éventuellement avec CKEditor). C'est d'autant plus pénible que CKeditor supporte très mal le code et à tendance à enlever certaines balise. Peu pratique, idem pour la gestion des révisions c'est pas souple même si ça marche très bien.

Avec pelican, je suis très content de pouvoir utiliser du balisage léger et mercurial pour les révisions. Plus qu'à importer tout mes articles ;-).

Détecter une architecture avec CMake

2014-04-06T00:00:00+02:00

Il est tout à fait possible de détecter l'architecture de l'hôte avec CMake et d'adapter votre Makefile avec cette info. Il suffit d'utiliser la variable CMAKE_SYSTEM_PROCESSOR. Ci dessous un petit exemple :

if(${CMAKE_SYSTEM_PROCESSOR} MATCHES armv6l)
    include_directories(${OpenCV_INCLUDE_DIRS} ${Boost_INCLUDE_DIRS} ~/git/robidouille/raspicam_cv)
    link_directories(${OpenCV_LIB_DIR} ~/git/raspberrypi/userland/build/lib ~/git/robidouille/raspicam_cv)
    target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS} ${Boost_LIBRARIES} raspicamcv mmal_core mmal mmal_util vcos bcm_host X11 Xext rt stdc++)
else()
    include_directories(${OpenCV_INCLUDE_DIRS} ${Boost_INCLUDE_DIRS})
    target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS} ${Boost_LIBRARIES})
endif()

Émuler une Raspberry Pi sous Linux avec Qemu

2014-04-06T00:00:00+02:00

Qemu est un logiciel de virtualisation qui présente l'avantage de pouvoir émuler de façon logicielle une architecture donnée. Il vous permet donc de simuler une architecture que vous n'avez pas, par exemple l'architecture arm sur x86. Qemu existe aussi sous windows, mais seule la méthode sous linux sera évoquée ici. Vous pouvez vous référer à ce document pour la version windows. Je signale au passage que virt-manager ne supporte pas a priori l'architecture arm1176. Voir par là pour plus d'infos.

Installer Qemu

Tout d'abord, il faut installer qemu. Normalement, c'est présent dans les dépôts de votre distribution. Sous Fedora, j'ai installé les paquets suivants : qemu-system-arm et qemu-user. Vous devez avoir les commades qemu-arm et qemu-system-arm une fois installé. Vérifiez que l'achitecture arm1176 est bien supportée avec :

qemu-arm -cpu ? | grep arm

Vous devriez voir l'architecture correctement listée.

Récupérer une distribution et un noyau

Récupérez une image de la distribution de votre choix sur le site qui va bien : http://www.raspberrypi.org/downloads

Pour booter, vous allez aussi avoir besoin d'un noyau. Récuprérez le avec :

wget http://xecdesign.com/downloads/linux-qemu/kernel-qemu

Augmenter la taille de l'image

Par défaut, les images des distributions sont très petites. Vous pourrez faire quelques tests mais pas l'utiliser comme un vrai système émulé. Pour cela, il faut augmenter la taille de l'image avec :

qemu-img resize 2014-01-07-wheezy-raspbian.img +6G

Démarrer avec un shell root (pour régler des détails)

Lancez la commande ci-dessous pour démarrer. Vous aurez un clavier qwerty. Ne changez pas la valeur après -m, qui représente la mémoire du système. Il semble qu'elle soit hardcodée quelque part et une autre valeur vous empêchera de démarrer.

qemu-system-arm -kernel kernel-qemu -cpu arm1176 -m 256 -M versatilepb -append "root=/dev/sda2 panic=1 init=/bin/sh rw" -hda 2014-01-07-wheezy-raspbian.img

Éditez le fichiers /etc/ld.so.preload et commentez la première ligne (maj + 3 pour faire un #) :

nano /etc/ld.so.preload

Puis Ctlr-O pour sauvegarder et Ctrl-X pour quitter. Ce changement évite le chargement de libcofi_rpi.so ce qui évite une boucle inifinie au boot.

Ensuite, on prépare les disques pour une utilisation future en créant /etc/udev/rules.d/90-qemu.rules. Entrez les lignes suivantes dans ce fichier :

KERNEL=="sda", SYMLINK+="mmcblk0"
KERNEL=="sda?", SYMLINK+="mmcblk0p%n",

Terminez avec la commande sync pour être sûr que tout est écrit sur l'image. Éditez également votre /etc/fstab : changez mmcblk0p1 et mmcblk0p2en respectivement sda1 et sda2.

Démarrage normal

Entrez :

qemu-system-arm -kernel kernel-qemu -cpu arm1176 -m 256 -M versatilepb -append "root=/dev/sda2 panic=1" -hda 2014-01-07-wheezy-raspbian.img

Si qemu vous demande de lancer fsck, entrez :

fsck /dev/sda2 && shutdown -r now

Qemu devrait démarrer sur raspiconfig. Sélectionnez directement Finish. Nous allons maintenant agrandir les partitions. Avec

sudo cfdisk /dev/sdb

Supprimez la deuxième partition et recréez là avec tout l'espace disponible. Ensuite, lancez (si les commandes ratent, redémarrez et relancez les) :

sudo resize2fs /dev/sdb2
sudo fsck -f /dev/sdb2
sudo halt

C'est fini.

CMake et drapeaux de compilation

2014-03-30T00:00:00+01:00

Pour ajouter des options à gcc, il suffit d'utiliser l'option set comme ci dessous :

# Compiler flags
if(CMAKE_COMPILER_IS_GNUCXX)
    set(CMAKE_CXX_FLAGS "-O2")        ## Optimize
    set(CMAKE_EXE_LINKER_FLAGS "-s")  ## Strip binary
endif()

Les bases de CMake

2014-03-29T00:00:00+01:00

CMake est un utilitaire qui génère des MakeFile. Son principal intérêt est d'avoir une syntaxe plus légère que le MakeFile de base. Les bibliothèques les plus connues (Boost, OpenCV, …) sont compatibles avec ce système. Ci dessous un petit exemple pour vous aider à démarrer. Il faut enregistrer le contenu du fichier dans CMakeLists.txt puis dans le dossier où vous voulez produire vos binaires, entrez : cmake VERS_CMakeLists.txt

project(image)
cmake_minimum_required(VERSION 2.8)
FIND_PACKAGE(OpenCV REQUIRED)
FIND_PACKAGE(Boost COMPONENTS system filesystem REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS} ${Boost_INCLUDE_DIRS})
link_directories(${OpenCV_LIB_DIR} ${Boost_LIBRARY_DIR})
aux_source_directory(. SRC_LIST)
add_executable(${PROJECT_NAME} ${SRC_LIST})
target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS} ${Boost_LIBRARIES})

Et voilà un CMakeLists.txt basique. À vous de creuser pour en faire des plus complexes.

Compilation conditionnelle avec CMake

2014-03-29T00:00:00+01:00

Lorsqu'on fait de gros programme C ou C++, il peut être intéressant de ne compiler une partie du programme que si une option est passée en argument. Nous allons voir comment le faire en utilisant CMake, un générateur de MakeFile très connu et très puissant. Voir notre article sur CMake pour les bases de CMake.

Considérons pour cet article, que le programme doit afficher coucou si l'option est présente et rien sinon. Entourons remplaçons simplement la ligne `` std::cout << "coucou" << std::endl; `` par :

#if defined(COUCOU)
    cout << "coucou" << endl;
#endif

Il est tout à fait possible d'utiliser #ifdef à la place de #if defined mais on ne peut alors imbriquer les conditions : #if ( defined(COUCOU) && defined(SALUT) )

Il n'y a plus qu'à ajouter dans le CMakeLists.txt :

option(COUCOU "Active coucou" OFF)
if(COUCOU)
    MESSAGE( STATUS "COUCOU SUPPORT")
    add_definitions(-DCOUCOU)
endif()

Pour compiler, utilisez simplement :

Pour activer COUCOU : cmake .. -DCOUCOU=on && make
Pour ne pas activer COUCOU : cmake .. -DCOUCOU=off && make

Utiliser ipython pour manipuler des espaces en shell

2014-03-09T00:00:00+01:00

Bash est assez pénible quand à la gestion des espaces. Par défaut, la boucle for itère sur ce qui est séparé par un saut de ligne ou un espace. Je n'ai pas encore trouvé de solution satisfaisante en bash pur. Mais python (ou plutôt ipthon peut venir à la rescousse). Ipython est un shell interactif pour python. Vous pouvez y appeler n'importe quelle commande bash directement via la syntaxe !cmd. Avec a = !cmd, la sortie de cmd est stocké dans une liste constitué de chaque ligne de la sortie. Pour voir ce qu'aurait contenu a sans l'affectée, il suffit d'utiliser la syntaxe : !!cmd.

Les présentations étant faites, voici un petit script pour renommer des fichiers avec des espaces en ipython. Il s'adapte faciliment à d'autres situations :

#!/usr/bin/env ipython3

a = !ls
# Les variables contiennent des espaces, on les entoure donc avec des guillements
for file in a:
    !mv "$file" tmp/

Drupal : un an d'utilisation, mes impressions

2014-02-01T00:00:00+01:00

Je suis utilisateur de Drupal depuis environ 1 an maintenant et j’ai donc décidé de partager ce que j’ai appris avec cet outil que j’ai découvert à Centrale Marseille (ECM) grâce au projet multi-assos (qui héberge cet humble blog). Aujourd’hui, j’admets avoir pris goût à l’outil et je suis le webmaster du forum foceen un des plus gros sites de l’installation (et un des plus compliqué aussi).

Avant

Avant d’entrer à l’ECM, je n’avais jamais fait de vrai site web. J’avais appris quelques bases de php et de mysql sur le site du zéro (openclassrooms maintenant). J’avais un peu entendu parler de framework et de CMS mais je n’en pensais pas grand chose de bien. Sans trop caricatirer, je pensais : « Non mais un site ça se code à la main. Les CMS et les framework c’est des machins d’assistés ». Et puis j'ai découvert Drupal.

Drupal

Si vous ne connaissez pas encore Drupal, permettez que je vous présente l'outil. C'est un des CMS les plus utilisés et les plus populaires. Drupal accueille de gros sites (comme celui de la maison blanche) et des plus modestes (comme celui-ci) et globalement de nombreux sites d'entreprises ou d'institutions.

Un des gros avantages de Drupal est sa modularité. Sa communauté très active maintient de très nombreux modules de qualité qui vous permettent de transformer votre Drupal en site de e-commerce, en forum ou plus simplement de lui ajouter des fonctionnalités intéressantes comme un éditeur WYSIWYG. Le tout sans coder une seule ligne. De plus, ces modules viennent avec des interfaces puissantes mais bien faîtes pour les configurer suivant vos besoins. Le tout juste avec votre souris. Ce qui signifie que vous n'avez pas besoin d'être un geek pour être webmaster d'un site. Tout ce fait très bien via l'interface (sauf pour des besoins très spécifiques qui demandent du dévelloppement).

La communauté crée aussi pleins de thèmes et certains sont mêmes adaptatifs. Pas besoin d'être un très bon designer pour avoir un joli site. Plutôt cool non ?

Ce sont toutes ces raisons qui font que Drupal est bien adapté pour les associations de l'école. On est pas des dévelopeurs et cela permet de péréniser les sites web. Car c'est dur de faire un site web (du moins correctement).

Administrer l'installation

En tant que membre du club Drupal, je fais des sites web (et aide ceux qui en font) et administre l'installation. C'est à mon avis la partie la plus intéressante.

Faire des scripts Shell pour automatiser des tâches, faire du git, monter un serveur en local, bosser sur les migartions de drupal,… c'est intéressant et formateur (surtout quand il ne faut rien casser).

C'est dur de faire un site web

Contrairement à ce que peut laisser penser un cours de PHP ce n'est pas simple de faire un site web correctement. Ma (certes assez petite) expérience de webmaster me l'a bien montrée. Sur un site fait par quelqu'un qui est reconnu comme plutôt bon et qui sait coder, il peut y avoir de nombreux problèmes.

Ceux que j'ai vu et qui m'ont posés le plus de problèmes sont :

du code non indenté
les mots de passe stockés en clair dans la base de données (si si, il y en a qui le font)
des accès base de données sans passer par PDO (donc pas de requêtes préparées et des injections SQL faciles à faire)
des données encodées n'importe comment dans une base de données
fonction_1, fonction_2 et fonction_3 qui font la même chose (en tout cas je n'ai pas vu la différence). Et la gestion de version ça sert à quoi ?

Conclusion

Drupal c'est bien ;-). Vous vous prendrez la tête sur l'interface au lieu de vous prendre la tête sur du code mais bon, vous surviverez. Vous tomberez aussi sur des gens qui pense que comme tout se fait à la souris c'est facile et ça ne s'apprend pas. Ça c'est faux. Il vont faire des pages entièrement codé en html sans utiliser les champs et les vues. Ils vous dirons que Drupal c'est pourri et qu'en PHP ils auraient pu le faire facilement. On ne peut pas tout résoudre avec un outil, aussi bien soit-il.

Mettre sous git sous Drupal

2014-02-01T00:00:00+01:00

Récemment j'ai mis l'install Drupal du projet multi assos sous git (vous pouvez aussi utiliser un autre DVCS si vous voulez). Voilà mon petit compte rendu.

Intérêt

La première question que vous vous poserez sûrement est pourquoi placer Drupal sous git ? Parce que c'est utile. Cela :

vous permet de suivre les évolutions de l'installation : mise à jour, installation/suppression de module.
vous fournit un moyen simple de revenir en arrière en cas de problèmes (mauvaise mise à jour)
vous permet de comparer les différentes versions de vos settings.php et .htaccess
savoir qu'un fichier utile (comme un .htaccess) a disparu
faire des tests en local facilement et les pusher sur le seveur s'ils sont concluants.

Préparations

La question qui doit nous guider lors de cette étape est : que devons placer sous git ? En effet, il ne faut pas faire un brutal git add drupal.

Que contient une installation de Drupal ?

Drupal en lui même
Les modules et thèmes
Les dossiers des sites qui contiennent
- un settings.php
- les fichiers du sites

Nous voulons tous suivre sauf les fichiers du site (parce qu'ils changent tous de temps et n'ont rien à faire dans le dépôt) et les données de connexion à la base de données pour chacun des sites.

Nous devons modifier la façon dont Drupal gère les settings.php pour que les mots de passes de la base de données ne se retrouvent pas dans le dépôt. Pour ce faire :

créez dans le dossier sites un fichier settings.global.php. Ce fichier sera commun à tous les sites. Déplacez-y toutes les informations génériques (commentaires, les fonctions qui permettent de configurer PHP) et tout ce qui touche à la sécurité.
créez dans chaque dossier de site un fichier settings.local.php et placez-y les informations spécifiques au site (accès base de données, base url,…)
remplacez le contenu du settings.php par ceci :

<?php
// Load the global settings file.
require dirname(__FILE__) . '/../settings.global.php';

// Load the local settings file with databases connexion.
require dirname(__FILE__) . '/settings.local.php';

Ceci étant fait, préparons le .gitignore. Nous devons y ajouter les dossiers des sites. (default et assos.centrale-marseille.fr.* pour nous). Apdaptez à votre installation.

Reste également vos scripts éventuels. De notre côté plusieurs ont dû être modifiés. La liste est là (par ordre chronologique) :

Le passage

Maintenant que nous sommes prêts, il n'y a plus qu'à faire un petit git add et de faire votre premier commit. Pensez à commiter dès que vous faîtes un changement dans votre installation (avec des scripts au besoin).

Conclusion

On sait enfin ce qui se passe et on est certain de pouvoir revenir en arrière facilement au besoin. Vous pouvez évidement mettre vos scripts, template, … dans le dépôt. De notre côté, on a aussi un script pour synchroniser un site et faire des tests dessus en local. Il est ici (et sa dépendance ici).

Relier sa Raspberry Pi à son PC directement en Ethernet

2014-01-12T00:00:00+01:00

Dans cet article, nous allons étudier comment connecter son Raspberry Pi directement à son PC pour

pouvoir s'y connecter en SSH
qu'il ait une connection internet (via le wifi de notre PC)

Sommaire

Configurer le réseau
- Méthode graphique avec NetworkManager
- En ligne de commande
Configurer le pare feu
Configurons la Raspberry Pi pour le réseau
Et voilà
En plus
Sources

Configurer le réseau

Le Raspberry Pi et le PC doivent être sur la même plage d'adresse IP pour pouvoir communiquer mais pas sur la même plage d'adresse que l'interface wifi (sinon il y a conflit). Dans la suite, je vais configurer les interfaces éthernet sur 192.168.1.* le wifi étant géré automatiquement par votre gestionnaire habituel (NetworkManager chez moi). Je ne détaillerai que la connexion en IP fixe qui fonctionne très bien mais qui peux être lourde si vous avez beaucoup de machine à configurer. Il vous faudra alors regarder comment installer un serveur DHCP par exemple en suivant les instructions données ici.

Méthode graphique avec NetworkManager

NetworkManager permet de gérer les réseaux plutôt facilement. Ouvrez la fenêtre de configuration et cliquez sur réseaux filaires. Choisissez Ajoutez un profil. Dans la section Identité, donnez un nom au profil et cochez Connecter automatiquement. Dans la section IPV4, changez Automatic (DHCP) en Manuel. Entrez l'adresse IP que vous désirez associer à cette interface. Dans mon cas : 192.168.1.1, le masque de sous réseau : 255.255.255 et la route par défaut : 192.168.1.1.

Cochez Use this connection only for resources on its network.

Enregistrez et activez cette interface.

En ligne de commande

sudo ip addr add 192.168.1.1/24 dev eth0

Configurer le pare feu

Il faut maintenant autoriser la connexion et rediriger le traffic de la raspberry pi vers l'interface wifi. Pour cela, on entre les commandes suivantes :

sudo iptables -A FORWARD -o wlan0 -i eth0 -s 192.168.1.0/24 -m conntrack --ctstate NEW -j ACCEPT
sudo iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A POSTROUTING -t nat -j MASQUERADE

Ou sous Fedora (ou tout autre système utilisant firewalld)

firewall-cmd --zone=public --add-masquerade

À titre d'information : sous Fedora, les trois commandes précédentes permettent de se connecter en SSH à la Raspberry Pi, de pinger une IP depuis la Raspberry Pi mais pas un nom de domaine. Après un certain temps à chercher du côté du resolv.conf, c'est grâce à la commande dig centrale-marseille.fr @8.8.8.8 que j'ai pu voir que le problème était d'un autre ordre. Étrangement cette commande ne vient pas par défaut sous Raspbian. Elle est en revanche présente sous Pidora.

Si vous rencontrez des problèmes, sachez que la commande iptables -F enlève toutes les règles du parefeu.

Pensez également à vérifier que votre système autorise le transfert d'IP

sysctl -a | grep ip_forward

Si la réponse est :

net.ipv4.ip_forward = 0

faîtes, pour autoriser ce paramétrage :

sysctl net.ipv4.ip_forward = 1

Configurons la Raspberry Pi pour le réseau

Il y a deux méthodes pour avoir du réseau : l'IP fixe qui implique de configurer la Raspberry Pi et l'IP attribuée par un serveur DHCP qui tourne sur votre machine (ce qui implique de le configurer mais permet d'installer sa rapsberry pi sans écran).

Avec un serveur DHCP

Installez le paquet dhcp qui devrait être présent dans les dépôts de votre distribution. Éditez ensuite le fichier /etc/dhcp/dhcpd.conf. Il doit ressembler à :

#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp/dhcpd.conf.example
#   see dhcpd.conf(5) man page
#
subnet 192.168.1.0 netmask 255.255.255.0 {
         option routers 192.168.1.1;            # passerelle par défaut
        option domain-name-servers 8.8.8.8;# serveurs DNS
        range 192.168.1.2 192.168.1.100;        # plage d’adresse
}

Pour donner une adresse fixe à un hôte (l'adresse mac est requise) :

subnet 192.168.1.0 netmask 255.255.255.0 {
         option routers 192.168.1.1;            # passerelle par défaut
        option domain-name-servers 8.8.8.8;# serveurs DNS
        host daneel.egab {
            hardware ethernet b8:27:eb:44:ca:ee;
            fixed-address 192.168.1.3;
        }
        range 192.168.1.2 192.168.1.100;        # plage d’adresse
}

Le bloc host qui donne l'ip fixe doit être avant la directive range

Il n'y a plus qu'à lancer le serveur dhcp avec la commande systemctl start dhcpd. Pour savoir quelle est l'adresse atribuée à la Raspberry, consultez les log : journalctl -f /usr/sbin/dhcpd ou tail -f /var/log/messages si vous n'utilisez pas journald.

Pour une IP fixe

Mainteant, au tour de la Raspberry Pi de recevoir sa conf.

Pidora

Ouvrez /etc/sysconfig/network-scripts/ifcfg-eth0 et entrez (remplacez les instructions déjà présentes au besoin) :

DEVICE=eth0
BOOTPROTO=none
BROADCAST=192.168.1.255
ONBOOT=yes
USERCTL=no
DNS=8.8.8.8
PEERDNS=yes
IPV6INIT=no
NM_CONTROLLED=no
NETMASK=255.255.255.0
IPADDR=192.168.1.2
GATEWAY=192.168.1.1
TYPE=Ethernet

Raspbian

Ouvrez /etc/network/interfaces et entrez (en ne modifiant que les lignes concernant l'interface eth0) :

auto eth0
iface eth0 inet static
        address 192.0.2.7
        netmask 255.255.255.0
        gateway 192.0.2.254
        dns-nameservers 8.8.8.8

Et voilà

Normalement tout fonctionne. J'espère que cet article vous a été utile. Si vous avez des questions ou des remarques, n'hésitez pas ;-).

Si vous avez des problèmes avec la résolutions des noms de domaine, entrez dans /etc/resolv.conf

nameserver 8.8.8.8

En plus

Se connecter en wifi à une raspberry pi

Sources

Communication série facile avec python

2014-01-11T00:00:00+01:00

Python permet de réaliser facilement une liaison via USB entre un PC et une carte arduino. Nous allons l'illustrer par un exemple qui lit ce qu'une Arduino Uno envoie et un autre qui envoie un nombre à la Arduino. La diode de celle-ci doit alors clignoter du nombre envoyé.

Il faut uploader le code suivant sur votre carte Arduino :

#define LED_PIN 13
void setup()
{
  pinMode(ledPin, OUTPUT);
  Serial.begin(9600);
}

void loop()
{
    Serial.println("Bonjour");
    Serial.println("C'est la carte Arduino qui parle");

    while(Serial.available()) {
         int lu = Serial.read();
        flash(lu);
    }
    delay(1000);
}

void flash(int n)
{
  for (int i = 0; i < n; i++)
  {
    digitalWrite(ledPin, HIGH);
    delay(1000);
    digitalWrite(ledPin, LOW);
    delay(1000);
  }
}

Pour la lecture de ce qu'envoie la Arduino :

# Module de lecture/ecriture du port série
from serial import *
# Port série ttyACM0
# Vitesse de baud : 9600
# Timeout en lecture : 1 sec
# Timeout en écriture : 1 sec
with Serial(port="/dev/ttyACM0", baudrate=9600, timeout=1, writeTimeout=1) as port_serie:
    if port_serie.isOpen():
        while True:
            ligne = port_serie.read_line()
            print ligne

Pour l'envoie de données vers la Arduino, en python2 :

from serial import *
nombre = raw_input("Entrez un nombre : ")
port_serie.write([nombre])

Notez qu'il est possible d'envoyer un plusieurs nombres à la suite au port série, puisque la fonction "serial.write()" en Python prend en argument un tableau d'entiers.

Pareil en python3 :

from serial import *
nombre = input("Entrez un nombre : ")
port_serie.write(nombre.encode('ascii'))

La différence entre le code python2 et python3 vient du fait qu'en python3, les communications réseaux et série doivent se faire avec des objets de type bytes (chaîne d'octet). De même, les données reçues précédemment sont de type bytes en python3 mais de type str (chaîne de caractère) en python2. Voir par là pour plus d'infos.

Serial commincation made easy with Python

2014-01-11T00:00:00+01:00

With Python, it is easy to communicate with USB from your computer to your Arduino card. I'll illustrate this with a small example: the card will send some text to the computer and then we are going to send a number, X, to the Arduino whose LED will blink X times.

Upload the code below to your Arduino card:

#define LED_PIN 13
void setup()
{
  pinMode(ledPin, OUTPUT);
  Serial.begin(9600);
}

void loop()
{
    Serial.println("Hi");
    Serial.println("I am your Arduino card");

    while(Serial.available()) {
         int lu = Serial.read();
        flash(lu);
    }
    delay(1000);
}

void flash(int n)
{
  for (int i = 0; i < n; i++)
  {
    digitalWrite(ledPin, HIGH);
    delay(1000);
    digitalWrite(ledPin, LOW);
    delay(1000);
  }
}

To read the data from the Arduino:

# Module to read/write on a serial port
from serial import *
# Sereial port: ACM0
# Speed: 9600
# Read timeout: 1 sec
# Write timeout: 1 sec
with Serial(port="/dev/ttyACM0", baudrate=9600, timeout=1, writeTimeout=1) as port_serie:
    if port_serie.isOpen():
        while True:
            ligne = port_serie.read_line()
            print ligne

To send data to the Arduino, in Python 2:

nombre = raw_input("Entrez un nombre : ")
port_serie.write([nombre])

Note: it is possible to send one or more numbers in a row trough the serial port since the function serial.write() takes a list of ints.

In Python 3:

nombre = input("Entrez un nombre : ")
port_serie.write(nombre.encode('ascii'))

The difference between Python 2 and Python 3 comes from the fact that in Python 3, network and serial transfer must be done with bytes. Likewise, the data you receive are of type bytes in Python 3 but of type str in Python 2.

Tutoriel d’utilisation basique de LibreOffice Writer

2013-12-02T00:00:00+01:00

Vous pouvez télécharger la version odt ici et la version pdf là.

But de ce document

Vous pensez savoir utiliser LibreOffice (ou Word®) car c’est facile et qu’apprendre à l’utiliser ne sert à rien ? Ce n’est pas mon expérience. Elle prouve au contraire que la plupart des utilisateurs n’utilisent pas cet outil comme il faut, alors qu’il est très facile de gagner en efficacité avec quelques notions de bases.

En effet, lors de l’utilisation d’outils comme celui-ci (qualifiés de WYSIWYG¹) la tentation de mélanger contenu et forme est très vive. Par exemple, lorsque vous écrivez un titre, élément de structure, vous le mettez immédiatement en gros et en gras (forme). Ce faisant, votre document n’est pas structuré pour l’ordinateur.

Un autre exemple : vous serez amené à rédiger des rapports parfois très longs. Eh bien un document réalisé avec quelques fonctionnalités très basiques peut vous être extrêmement utile : changer tous les titres en 2-3 clics, faire un sommaire automatique avec renvoi, les titres qui se numérotent automatiquement…

Pour la rédaction de longs rapports scientifiques contenant plein de formules et plein d’autres choses, vous pouvez également vous tourner vers LaTeX. Ce logiciel est conçu pour l’édition de formules mathématiques et propose un rendu très professionnel pour celles-ci ainsi que des symboles mathématiques très peu courants. Il propose également des outils très puissants pour la bibliographie². En outre, il vous force à structurer votre document.

Ce document ne couvre que les fonctionnalités basiques (mais qui vous éviterons de nombreuses frustrations par la suite). Un autre document couvre des fonctionnalités plus avancées. Vous trouverez plus d’informations en fin de document. Tout ce que vous lirez est valable pour LibreOffice 3.6.x et 4.0.x.

Mais pourquoi LibreOffice Writer et pas…

Word

Word est un autre logiciel de traitement de texte édité par Microsoft. Pendant longtemps il a fait la promotion de son propre format qui fonctionne très difficilement avec les outils concurrents. Le support de l’OpenDocument (qui est un standard) est beaucoup plus récent. En outre, ce logiciel n’est ni multiplateforme (comment on le fait marcher sous Linux ?), ni gratuit. Vous pouvez vous procurer LibreOffice librement et le distribuer à vos amis sans problème.

OpenOffice.org

LibreOffice est en fait le descendant d’OpenOffice.org et est basé sur le code d’OpenOffice.org. Vous trouverez plus de précision sur Internet.

Google Documents et affilié

Bien que Google Documents puisse être utile pour le travail collaboratif, il est toujours très loin d’être complet. Je dirais même qu’il lui manque de nombreuses fonctions essentielles présentées dans ce document. De plus, vous perdez le contrôle sur vos données et vous êtes tributaires de Google (et de votre connexion à Internet).

Sommaire

1 Structurer un document

1.1 Les titres

1.2 Numéroter ses titres

2 Insérer des images

3 Les numéros de page

4 Rupture de page

5 Faire une page de garde

6 Le sommaire automatique

7 La correction orthographique et grammaticale

7.1 La correction orthographique

7.2 La correction grammaticale

8 Divers

8.1 Puces et numérotation

8.2 Les divers collages

9 Citations

10 Auteur et licences

1 Structurer un document

Structurer votre document c’est ce qui vous permet de faire la différence entre les titres, le corps de texte… Vous allez me dire en sélectionnant du texte et en le mettant en gras et en gros, j’ai fait un titre. Eh bien non. Vous avez structuré le document pour l’impression, pas pour l’édition informatique. L’idée de la structure c’est de « dire » à votre ordinateur que ce texte sélectionné est un titre. Il pourra alors le manipuler comme tel et le mettre dans un sommaire, vous proposer des options de navigation entre vos titres…

1.1 Les titres

Comment faire un vrai titre me direz-vous. Rien de plus simple. Cliquez sur la flèche à droite de Corps de texte, puis sélectionnez Titre principal. Si ce titre n’apparaît pas dans la liste, sélectionnez Corps de texte et tapez Titre principal.

N
B : le Titre principal est suivi du style Sous titre.

Voilà, vous venez de donner un titre à votre document. Vous pouvez également mettre un titre à vos sections/chapitres en sélectionnant Titre 1 pour le premier niveau puis Titre 2 pour le second niveau et ainsi de suite.

Titre est une sorte de super-titre. En le modifiant, vous appliquez des modifications à tous les titres !

Et maintenant, modifions tous nos titres sans en oublier un ! Cliquez sur Styles et formatage en haut à gauche de l’écran.

Puis faites un clic droit sur l’élément que vous voulez modifier et cliquez sur modifier (ici Titre principal).

Vous pouvez alors choisir en navigant dans les différents onglets l’alignement du texte (centré, justifié, aligné à droite, aligné à gauche), la police, la taille, la fonte, la couleur… Une fois que les réglages vous conviennent, appuyez sur OK pour les appliquer. Voilà, plus besoin de parcourir tout le document pour changer les titres. De plus, un titre suivit par un texte ne sera jamais seul sur une page.

Vous pouvez également ajouter vos propres styles en faisant un clic droit dans cette fenêtre et en choisissant Nouveau.

Ne modifiez jamais au grand jamais l’apparence des éléments de structure via la barre de formatage. Vous encourrez des problèmes à traiter à la main.

Vous pouvez naviguer entre vos titres via le menu de navigation. Pour cela, allez dans Affichage > Navigateur ou appuyez sur F5. Vous pouvez aussi naviguer entre vos images, index…

1.2 Numéroter ses titres

Un autre intérêt à faire des titres de section c’est que LibreOffice peut gérer leur numérotation tout seul. Et si vous ajoutez un titre entre deux autres, ils seront renumérotés automatiquement ! Comment fait-on ? Allez dans la barre d’outils, sélectionnez Outils > Numérotation des chapitres.

Dans la fenêtre qui s’ouvre, sélectionnez le type de numéro que vous voulez pour chaque hauteur de titre. Le champ afficher les niveaux permet de faire des plans de ce type 1., 1.1,… Appuyez sur OK pour valider les changements. Vos titres se numérotent sans votre aide !

L
es champs Devant et Derrière vous permettent d’ajouter ce que vous voulez avant et après vos numéros de titres. Si vous voulez entourer les numéros de vos titres par Salut et Adieu, c’est ici que ça ce passe. Je vous conseille de mettre un espace dans Derrière (vous comprendrez en lisant la section sur le sommaire automatique).

2 Insérer des images

Vous pouvez insérer des images à partir d’un fichier via le menu Insertion > Image > À partir d’un fichier.

Lorsque vous sélectionnez cette image, vous pouvez choisir l’adaptation de cette image. Des options supplémentaires sont disponibles sous l’onglet Adaptation du texte. Fiez-vous au dessin, ils sont très parlants. Vous pouvez également modifier le style Images via Styles et formatage. Ce faisant, vous pouvez mettre toutes vos images avec la même adaptation.

En double cliquant sur une image, une boîte de dialogue s’ouvre. Elle vous permet notamment de rogner l’image, de choisir la bordure et la couleur de fond.

Dans l’onglet Type de la boîte de dialogue vous pouvez choisir l’ancrage de votre image. Vous avez quatre types d’ancrage possible : à la page, au paragraphe, au caractère et comme caractère. Cela vous permet de savoir par rapport à quoi le positionnement de l’image sera fait. Par exemple, si l’image est ancrée à la page et que vous sélectionnez un positionnement vertical Haut et horizontal Centré, votre image sera toujours en haut de la page et centrée horizontalement.

Le gros avantage de l’ancrage comme caractère est que l’image se comportera comme un caractère et restera donc là où vous l’avez placé.

Afin de réduire la taille de votre document, je vous invite à rogner vos images en dehors de LibreOffice avec un logiciel de manipulation d’images tel que Gimp. En effet, lorsque vous rognez une image, LibreOffice conserve l’image de départ dans le fichier. Ainsi, votre fichier odt va grossir très rapidement avec des informations inutiles. Il en est de même pour le pdf généré à partir de ce fichier.

Si vous rognez vos images, ces fichiers seront plus petits et donc, par exemple, pourront être envoyés plus facilement par courriel. Vous pourrez alors exporter le pdf sans compression pour avoir un maximum de qualité, sans avoir un fichier trop gros. De plus, ces logiciels étant dédiés au traitement d’images, il est souvent plus facile de redimensionner correctement celles-ci.

3 Les numéros de page

Vous pouvez numéroter vos pages en :

Insérant un pied de page
Y insérant un numéro de page

Pour insérer le pied de page, cliquez simplement en base de la page. Une boîte comme ci-dessous apparaît. Cliquez dessus pour ajouter le pied de page.

NB : cela fonctionne aussi pour les entêtes.

Pour ajouter le numéro, allez dans la barre d’outils, puis dans Champs > Numéro de page.

Pour que vos pages aient des numéros de la forme : numéro de la page/nombre de page, ajoutez le caractère '/' après le numéro de la page puis insérez le champ Nombre de pages.

4 Rupture de page

Si vous désirez qu’un texte commence toujours sur une nouvelle page, par exemple le texte qui suit la page de garde ou le sommaire, il faut insérer un saut de page avant ce texte. Allez dans Insertion > Saut manuel… Vous pouvez également insérer des sauts de paragraphes et de colonnes.

Si vous voulez que vos titres commencent impérativement sur une nouvelle page, allez dans le panneau de style, puis dans l’onglet Enchaînement de votre titre. Cochez la case Insérer et laissez les autres champs aux valeurs par défaut.

5 Faire une page de garde

Pour faire une vraie page de garde, avec du texte centré sur la page, des entêtes et pieds de page spécifiques, c’est assez simple. Placez-vous sur la page à convertir. Allez dans Format > Page de titre… Sélectionnez Convertir les pages existantes en page de titre.

Vous pouvez choisir si la page doit être numérotée et à quel chiffre la numérotation doit reprendre après cette page.

Pour mettre un texte (le titre de votre document par exemple) bien au centre de la page, allez dans Insérer > Cadre… Sélectionnez la largeur comme relative et à 100 %. Dans la section position, choisissez Centré pour les alignements verticaux et horizontaux. Dans l’onglet Bordures, vous pouvez supprimer la bordure. Ceci est aussi valable pour les autres pages.

6 Le sommaire automatique

Maintenant que vous savez faire des titres, vous aimeriez bien les intégrer dans votre sommaire. Eh bien LibreOffice va le faire pour vous. Dans la barre d’outils, allez dans Insertion > Index et tables > Index et tables…

Validez avec OK. Vous avez un sommaire automatique ! Libre à vous de parcourir toutes les options proposées. Par défaut, cette table s’appelle Table des matières. Vous pouvez la renommer en modifiant son champ Titre comme ci-dessus. Je vous conseille fortement de laisser cochée la case Protégé contre toutes modifications manuelles. Pensez également à faire un clic droit puis Actualiser l’index/la table de façon à ce que LibreOffice prenne en compte les modifications que vous avez apportées aux titres (ajouts, suppressions, déplacements à une page différente…). Vous pouvez également aller dans Outils > Actualiser > Tout actualiser.

Pour modifier l’apparence du contenu de la table des matières, faites un clic droit sur une entrée puis sélectionnez éditer le style de paragraphe. Vous pouvez également passer par l’écran de style en sélectionnant une des entrées Table des matières. Vous avez 10 niveaux de numérotation à votre disposition.

Sommaire ou table des matières ? Le sommaire est plus synthétique (3-4 niveau de titre maximum) que la table des matières et est placé en début de document.

7 La correction orthographique et grammaticale

7.1 La correction orthographique

Dans l’écran de style, vous pouvez également sélectionner une langue pour chaque entrée. Comme vous pouvez ajouter des entrées, vous pouvez créer un corps de texte en français et un autre en anglais, ce qui vous simplifiera la vie.

Pour des modifications de langues plus ponctuelles, faites un clic gauche sur la langue dans la barre du bas et sélectionnez plus. Dans l’onglet police, sélectionnez la langue que vous désirez. Vous pouvez également choisir aucune pour n’avoir aucune correction (car vous êtes vraiment bon en orthographe, ce paragraphe contient du code…)

Si vous voyez la langue avec un trait bleu à côté, cela veut dire quelle est installée, sinon c’est à vous de l’ajouter. Sous Linux, consultez votre gestionnaire de paquets, sous Windows, allez voir http://extensions.libreoffice.org/extension-center/dictionnaires-francais.

7.2 La correction grammaticale

LibreOffice ne propose pas de correcteur grammatical par défaut. Heureusement, la communauté propose des extensions pour pallier ce problème. Vous pouvez installer LanguageTools qui supporte de nombreuses langues et grammalecte qui ne supporte que le français mais renvoie moins de faux positifs.

http://www.languagetool.org/

http://www.dicollecte.org/grammalecte/

8 Divers

8.1 Puces et numérotation

Pour construire une liste, allez dans le menu Format > Puces et numérota tion. De nombreux formats de puces et de numérotation sont disponibles. Et même des plans ! Faites votre choix.

8.2 Les divers collages

Lorsque vous collez un texte, par défaut, LibreOffice conserve le formatage. Néanmoins, parfois vous désirez vous en débarrasser. Pour cela, utilisez le raccourcis clavier Ctrl + Maj + V et sélectionnez texte non formaté. Les autres options de texte garde une partie du formatage.

9 Citations

« Note that underlining for emphasis is considered bad practice in the publishing world. Underlining is only used when the output device can’t do highlighting in another way — for example, when using a typewriter. »

Michel Goossens et al. in the LATEX Companion

« Perhaps poets who wish to speak in a still small voice will cause future books to make use of frequent font variations, but nowadays it’s only an occasional font freak (like the author of this manual) who likes such experiments. »

Donald Knuth in the TEXbook

10 Auteur et licences

Ce document a été écrit par Julien Enselme, élève à l’école Centrale Marseille pour l’usage de tous ceux qui veulent. Vous pouvez envoyer vos suggestions à jenselme@centrale-marseille.fr et trouver une version actualisée de ce document (et peut-être des tutos sur les autres logiciels de la suite bureautique LibreOffice) :

Ce document est distribué sous Creative Commons BY-SA v3 (Parité, partage à l’identique). Pour plus d’information, voir :

https://creativecommons.org/licenses/by-sa/3.0/legalcode

11 Remerciements

Je remercie chaleureusement tous ceux qui m’ont conseillé et m’ont signalé les coquilles, imprécisions anciennement contenues dans ce document. Mes remerciements vont en particulier aux membres de la liste Ginfo de l’école pour leur nombreux conseils et corrections, ainsi qu’au secrétaire de l’association la mouette.

1What You See is What You Get, ce que vous voyez est ce que vous obtenez. À opposer à WYSIWYM — What You See Is What You Mean, ce que vous voyez est ce que vous pensez.

2LibreOffice propose également de tels outils. Se reporter à la partie avancée.

De drupal 6 à drupal 7 : drush sup

2013-10-30T00:00:00+01:00

Pour le projet multi-assos j'ai dû migrer trois sites webs en drupal 6 vers drupal 7. Voilà mon petit retour d'expérience (un autre suivra bientôt sur le module migrate_d2d qui m’a permi de migrer deux des trois sites). Ici c'est la méthode utilisée pour le forum. Ce site a beaucoup de nodes (plus de 16 000) mais reste simple dans sa construction.

Pourquoi drush sup

Il existe plusieurs outils pour transférer du contenu d'un drupal vers un autre (node export p. ex.) mais pour migrer de drupal 6 et 7 ce qui me semble le mieux c’est drush sup. Il migre le contenu, télécharge les modules (dont les thèmes) et les active, copie les fichiers,… sur le site drupal 7 ce qui est plutôt sympa. Bref, en théorie une fois que vous avez finis la migration, votre site d7 est une copie conforme du d6 (enfin en théorie).

Préparation

Étape 1 : télécharger le module drush sup avec drush sup

Étape 2 : faire une copie de test des sites (il serait dommage de casser le vrai site).

Étape 3 : mettre en place les alias drush (pour le drupal 6 et le drupal 7) en éditant le fichier ~/.drush/aliases.drushrc.php comme suit (à adapter à votre configuration) :

<?php
$aliases['toto'] = array(
    'uri' = 'toto.local',
    'root' => '/var/www/local',
);

Étape 4 : vérifier la disponibilité des modules et thèmes en drupal 7. Drush sup est très fort mais il ne fait pas de miracle non plus ;-)

Vous êtes prêt. Il vous suffit de lancer : drush @d6 sup @d7 et de suivre les instructions.

La migration

Normalement, le script de migration détaille bien les étapes et les questions posées sont claires. Il n’y a pas de raison majeure que ça se passe mal.

Il est bon de savoir que drush fait des sauvegardes et peut donc rétablir le site d7 a un point antérieur de la migration (tout refaire, juste après la migration des modules,…).

Le point qui m’a posé problème est en général la migration du contenu qui bloque. Parfois en retentant ça passe. Pas compris pourquoi.

Un point bizarre que j’ai eu sur un site : drush sup refusait de migrer mes modules via l’option qui permet de tout migrer en un coup. Mais en les prenant un à un, c’est passé.

Conclusion

Pour les sites relativement simples tels que ceux que l’on a sur assos la migration ne doit pas faire peur. Drupal 7 est suffisament mature, avec suffisament de modules pour que tout se passe bien. La migration demande quand même pas mal de tests au préalable pour régler les petits problèmes tels que les blocs qui disparaissent ou le thème qui casse. Le nombre de node ne doit pas faire peur. Par contre, il faut se méfier des sites avec des modules étranges.

Concernant les deux sites pour lesquels je n’ai pas pu faire du drush sup : le premier avait une base de données pourrie (un de mes prédécesseurs avait tenté une migration directement sur le serveur ce qui a pourri la base de données, même en supprimant les tables de drupal 7, il restait des problèmes). Pour le second, les très nombreuses images indispensables au site ne sont pas migrées et il se trouve que c’est un peu plus simple de rattraper ça à partir du résultat de migrate_d2d.

Voilà. Si vous avez des questions ou des remarques, les commentaires sont là.

De drupal 6 à drupal 7 : migrate_d2d

2013-10-30T00:00:00+01:00

Après la première partie sur drush sup, voilà la deuxième et dernière partie de mon retour d'expérience sur les migrations Drupal. Aujourd’hui c'est le module migrate_d2d (pour migrate drupal to drupal) qui est à l’honneur. Je ne parlerai ici que de la version 2.1 (encore en développement à l'heure où j'écris ces lignes) car c'est avec elle que le projet introduit migrate_d2d_ui une joli interface pour vous simplifier la tâche.

Ce module très sympa vous permet de repartir sur de bonnes bases, dans un site Drupal 7 tout neuf. Il m'a été très utile pour migrer le site des annales (accessible uniquement pour les élèves de l'école) (dont la base de données était corrompue ce qui empêchait drush sup de fonctionner) et pour le blog des expats dont la version drupal 7 est en cours de validation de notre côté pour lequel drush sup ne migre pas correctement les images.

Préparation et migration

Ce coup ci, votre site drupal 7 ne va pas se créer tout seul. Il faut donc tout d'abord créer un site drupal 7 neuf, comme d'habitude. Activer tout les modules nécessaires, créer vos types de contenu avec les bons champs, les rôles et les vocabulaires de taxonomie.

Une fois que le site drupal 7 est prêt, il faut télécharger et activer les modules migrate et migrate_d2d (pour le moment, prendre lesversions de développement des deux modules, sinon vous n’aurez pas droit à l'interface).

Direction Contenu > Migrate > Import from drupal. Après l'interface détaille bien les étapes. On commence par remplir les infos sur la base de données, puis on choisit pour chaque type de contenu du drupal 6 le type de contenu d'arrivée dans le drupal 7. Idem pour les rôles. Pour les fichiers, il suffit de lui donner l'emplacement sur le serveur. On sauvegarde.

Ensuite, retour sur le dashboard, on sélectionne ce qu’on veut importer et c'est parti !

Les problèmes

J'ai eu quelques problèmes avec les préfix de bases de données qui n'étaient apparament pas pris en compte. J’ai retesté hier et c'est passé. Le problème a dû être corrigé en amont.

Les formats de texte n’étaient également pas toujours bien pris en compte. Mais ça, c'est facile à régler avec du SQL.

Le gros problème est que si la table file_managed était bien remplie, les tables file_usage et celles qui spécifient dans quel champ le fichier est utilisé sont restées désespérément vides. De même pour la taxonomie. Les termes du vocabulaire sont bien créé mais ils ne sont pas associé à chaque contenu. Du coup j’ai dû faire un script pour récupérer les données dans le Drupal 6 pour remplir les tables du Drupal 7.

Conclusion

Cela conclue mes migrations de Drupal 6 à 7. En résumé, quand drush sup fonctionne, c’est génial. On a alors vraiment l'impression que tout est magique et automatique. Par contre, sans ça devient galère, surtout s'il faut aller dans la base de données pour finir la migration. Ça allonge considérablement le temps nécessaire pour terminer. D'autant plus que je n'avais aucune expérience sur les bases de données de Drupal avant. Mais bon, c'est l'occasion d'apprendre ;-)

Les migrations vers drupal 7 c'est bientôt fini pour nous. Les derniers Drupal 6 qui restent sont en train d'être refait depuis zéro par leur webmaster respectif. En tout cas, je comprends pourquoi sur drupal.org il aura fallu plus d’un an pour faire la migration.

Tutoriel d’utilisation avancé de LibreOffice Writer

2013-10-25T00:00:00+02:00

Ce tuto est en cours de rédaction. Toute aide est la bienvenue : comments [at] jujens [dot] eu

Vous pouvez télécharger la version odt ici et la version pdf là.

Sommaire

1 La table des matières : fonctionnalités avancées

2 Créer ses styles personnalisés

3 Les commentaires

4 Les champs

5 Renvois

6 Les formules et les équations (mathématiques et chimiques)

6.1 Forme générale

6.2 Exemples les plus courants

6.3 Mise en forme

6.4 Numéroter ses équation

7 Bibliographie

7.1 Constitution de la bibliographie

7.2 Affichage de la bibliographie

8 Notes de bas de page

9 Format de page

10 Les diagrammes et objets OLE

10.1 Diagrammes

10.2 Autres objets OLE

11 Illustrations

11.1 Créer une illustration

11.2 Table des illustrations

16 Suivis des modifications

17 Les bonus

17.1 Faire des dessins

17.2 Exporter son document

17.3 Les PDF hybrides

17.4 LaTeX et LibreOffice

17.4.1 L’import

17.4.2 L’export

17.5 Envoyer des documents sur les serveurs de l’école facilement

18 Auteurs et licence

19 Pour aller plus loin

1 La table des matières : fonctionnalités avancées

Vous savez désormais insérer des tables des matières. Voyons maintenant comment personnaliser cette table pour la rendre conforme à vos désirs.

Tout d’abord, vous n’êtes pas obligés de faire apparaître tous vos niveaux de titres dans la table, surtout si vous en avez de nombreux. Dans le champ Évaluer jusqu’au niveau entrer le nombre qui correspond au niveau maximum de titres souhaités. Par défaut ce chiffre est de 10, ce qui correspond au nombreux de niveau de titres par défaut.

Ensuite, vous pouvez changer l’ordre du texte de la table, en supprimer, choisir la partie du texte cliquable qui vous envoie vers votre chapitre… Pour cela, allez dans l’onglet Entrées. Sélectionner le niveau de titre que vous souhaitez modifier. Par exemple pour ne placer le lien hypertexte que sur le numéro du chapitre :

Cliquez sur FH (fin de l’hypertexte)
Cliquez dans la case vide à droite de E# (numéro de chapitre)
Cliquez sur le bouton hyperlien

Et c’est tout. De la même manière, vous pouvez déplacer le numéro de page, supprimer le numéro de titre…

2 Créer ses styles personnalisés

Ouvrez l’écran Style et formatage (F11). Faîtes un clic droit et cliquez sur Nouveau…

Entrez un nom pour ce style. Le Style de suite est le style de texte qui sera appliqué lorsque vous appuierez sur la touche Entrée. Le champ Lié à permet de faire jouer l’héritage des propriétés du style parent. Si vous effectuez une modification à ce style, elle sera appliquée à votre nouveau style. Le champ Catégorie permet de classer les styles. Ce classement est accessible via le menu déroulant en bas de l’écran de style.

Concernant les autres onglets, ils vont vous permettent de personnaliser votre style. Signalons tout de même dans l’onglet Enchaînement, l’option qui vous active la césure automatique.

Si vous rencontrez des problèmes liés au style, sélectionner le texte concerné puis clique droit et cliquez sur Effacez le formatage direct.

3 Les commentaires

Vous pouvez ajouter des commentaires à un endroit du texte. Pour cela, allez dans Barre d’outil : Insertions > Commentaires. Dans la boîte qui apparaît à droite, vous pouvez entrer votre texte. Les commentaires contiennent le nom du commentateur et la date. À côté de ces informations, la flèche vous permet de supprimer ce commentaire.

4 Les champs

5 Renvois

6 Les formules et les équations (mathématiques et chimiques)

Même si LibreOffice n’est pas LaTeX et propose moins de symboles, on peut faire de « belles » équations avec !

6.1 Forme générale

Pour insérer une équation allez dans Barre d’outils : Insertion > Objet > Formule. Un cadre apparaît en bas de l’écran dans lequel vous pouvez taper votre formule. Si vous ne connaissez pas les commandes, une fenêtre apparaît également avec les dessins de relations, opérations… En cliquant sur ceux-ci, du texte est inséré dans le cadre d’édition de votre formule. Bien pratique si vous ne connaissez pas le texte permettant d’insérer ce symbole !

NB : si cette fenêtre n’apparaît pas, sélectionnez Éléments dans Barre d’outils : Affichage.

Ce texte est composé de deux éléments distincts :

<?> à remplacer par votre texte (le plus souvent ce sera des nombres)
symbole qui schématise l’emplacement d’un symbole.

Par défaut, seul un chiffre ou groupe de chiffre sans espace est concerné par le symbole. Pour que celui-ci affecte toute une expression, mettez celle-ci entre accolades. Vous pouvez également utiliser des parenthèses, mais celles-ci apparaîtront dans la formule.

6.2 Exemples les plus courants

Les fractions :

1 over {x + 2} 1 over (x + 2)
La puissance :

x^2
Les indices :

x_0
La somme :

sum from n = 1 to 100 {1 over n^2}
L’intégration :

int from - infty to 0 {x + 2 dx}
Les lettres grecques :

%alpha + %bêta + %gamma
Les vecteurs :

widevec {E_1}
Le saut de ligne : newline
Les systèmes d’équations :

left lbrace stack {x + y + 2 = 0 # x^2 + y + 7 = 0 # 3*x + y = 9} right none
Les matrices :

left [ matrix{1#0##0#1} right ] nitalic{ et }

left [matrix{A_11#A_12#dotsaxis#A_{1n}##A_21#{} #{}#A_{2n}##dotsvert#{}#{}#dotsvert##A_{n1}#A_{n2}#dotsaxis#A_nn} right ]
La chimie :

nitalic{ H_{3}O^{-{}} + H_{2}O %Ux21CC 2H_2 O}
Les autres caractères spéciaux : vous disposez d’un catalogue de symbole à votre disposition via le bouton . Si la liste ne vous convient pas, vous pouvez ajouter d’autres caractères (dans le jeu de symboles Spécial de préférence) via le bouton insérer. Vous pouvez également utiliser directement le code UTF-8 si vous le connaissez.

6.3 Mise en forme

Il est tout à fait possible de changer la taille et la fonte d’écriture de votre équation. Pour cela, cliquez sur l’équation à modifier et allez dans Barre d’outil : Format et choisissez le sous menu que vous désirez (Polices, Tailles de police). Les modifications seront localisées à l’équation courante à moins que vous ne cliquiez sur Par défaut.

Vous pouvez également apporter des modifications de style locales. Pour cela, il faut utiliser la commande size. Vous pouvez indiquer une taille absolue ou relative :

x + 3 size 8{y}

%alpha + size *1.5{%BÊTA} + size /1.5{%igamma}

Vous pouvez également mettre de la couleur avec la commande color. Vous pouvez choisir parmi les couleurs suivantes : white, black, cyan, magenta, red, blue, green, yellow.

color green AAUIE {color red bépo}

6.4 Numéroter ses équation

C’est sans conteste une fonctionnalité très appréciable mais cachée. Démarrez une nouvelle ligne. Tapez nf (en environnement français) ou fn (en environnement anglais) puis appuyez sur la touche F3. Le cadre ci-dessous va alors apparaître.

	(1)

Vous pouvez évidemment faire un renvoi vers cette équation. Allez dans Barre d’outil : Insertion > Renvoi… Dans Type de champ, sélectionnez Texte. Dans Sélection, sélectionnez le numéro de votre équation. Dans Insérer une référence à, choisissez Référence.

7 Bibliographie

La bibliographie se gère de deux manières différentes pour ce qui est du stockage de l'information mais se gère de la même manière que la table des matières pour l’affichage.

7.1 Constitution de la bibliographie

Vous avez deux manières de stocker vos informations bibliographiques :

Soit dans une base de données LibreOffice, pour cela utilisez LibreOffice Base, ainsi vos données seront réutilisables dans tous les documents LibreOffice manipulés sur votre ordinateur. À vérifier, la possibilité de réaliser une base de données LibreOffice accessible par le réseau et donc utilisable à plusieurs en simultanément.
Soit dans le document sur lequel vous êtes en train de travailler, pour que quelqu’un d’autre puisse le manipuler sans devoir accéder à la base de données, impossible de réutiliser les données dans un autre document.

Nous allons voir la deuxième possibilité. Pour créer une entrée de bibliographie :

Cliquez sur Insertion > Index et Tables > Entrées de bibliographie
Choisissez à partir du contenu du document (l’autre option si vous utilisez une base de données)
Cliquez sur Nouveau
Choisissez un type et remplissez les données, puis cliquez sur OK
Votre entrée de bibliographie est prête à être insérée n’importe où dans le document, autant de fois que souhaité en cliquant sur Insérer.

7.2 Affichage de la bibliographie

Cliquez sur Insertion > Index et Tables > Index et tables
Dans type, choisir Bibliographie
Comme pour la table des matières, vous avez accès à tout un tas d'options (que vous pourrez modifier par la suite),
Cliquez sur OK pour finir.

Votre bibliographie est prête. Elle se mettra à jour en allant sur Outils > Actualiser > l'option que vous souhaitez.

8 Notes de bas de page

9 Format de page

10 Les diagrammes et objets OLE

10.1 Diagrammes

Vous pouvez insérer des diagrammes (histogramme, nuage de points) dans vos documents LibreOffice sans utiliser des images. Allez dans Barre d’outils : Insertions > Objet > Diagramme…

10.2 Autres objets OLE

11 Illustrations

11.1 Créer une illustration

Vous savez insérer des images dans votre fichier. C’est bien. Mais vous voulez peut-être mettre une légende à cette image, ou faire une table à la fin de votre rapport rappelant toutes les images insérées dans votre document.

C’est très simple : faîtes un clic droit sur votre image et choisissez Légende… Dans la boîte légende, tapez votre légende. Par défaut, votre légende sera précédée de Illustration. Dans le champ Catégorie, vous pouvez choisir <Aucun(e)> pour supprimer ce mot. Par contre, dans ce cas vous ne pourrez faire apparaître votre illustration dans la table. Vous pouvez en revanche choisir votre propre nom : sélectionnez le texte dans Catégorie et entrez votre texte.

Vous pouvez choisir votre numérotation, séparateur…

11.2 Table des illustrations

Pour créer une table des illustrations, il faut procéder de façon analogue à la création de table des matières. Allez dans le menu Insertion > Index et tables > Index et tables… Dans le type d’index, choisissez Index des illustrations.

12 Métadonnées

Vous pouvez ajouter des informations supplémentaires à votre document, tel que auteur, la licence, des mots clés… Lors de l’export en pdf par exemple, ces informations seront disponibles via la fenêtre de propriété du document.

Pour ajouter les métadonnées, allez dans Barre d’outils : Fichier > Propriétés… Vous n’avez plus qu’à remplir les champs. Les données d’identités (principalement utiles dans un contexte professionnel) se trouve dans le menu Outils > Options…, puis dans la section LibreOffice > Données d’identités. Vous n’avez plus qu’à compléter avec votre nom, votre adresse…

13 Formulaires

Vous pouvez générer des formulaires avec LibreOffice, ce qui vous permettra d’envoyer des documents par exemple au format PDF, et de les recevoir remplis. Plus besoin que les personnes les impriment pour les remplir et les scanner pour renvoi.

Pour cela, aller dans Affichage > Barres d’outils > Contrôles de formulaire.

Cliquer sur « activer le mode Ébauche », les boutons permettant de placer des éléments de formulaire vont devenir disponibles.

Il faudra aller dans les options de chaque élément pour configurer les valeurs possibles pour une liste de sélection, l’étiquette de l’élément, etc.

14 Publipostage

Le publipostage sert à générer des documents personnalisés à partir d’une base de données. Par exemple, générer des badges à partir de nom et prénom dans la base de données.

Le principe est de connecter le document à une base de données LibreOffice (qui peut être une base de données MySQL, LibreOffice, un fichier CSV, etc.) afin de placer dans le document des balises représentant les colonnes de cette base de données, puis de faire l’export durant lequel LibreOffice va remplacer les balises par les valeurs de la base de données.

Pour ce faire :

Édition > Changer de base de données...
Sélectionner la base de données qui vous intéresse.
LibreOffice va afficher les colonnes détectées.
Il suffit de faire un glisser-déposer des entêtes des colonnes pour positionner les balises dans le document.
Une fois la mise en page faite, Outils > Assistant Mailing...
Suivre les étapes.

Note : je conseille de faire une copie du document LibreOffice au cas où le document modèle ne soit écrasé par celui contenant le publipostage.

15 Références

16 Suivis des modifications

17 Les bonus

17.1 Faire des dessins

17.2 Exporter son document

17.3 Les PDF hybrides

Vous avez peut-être déjà été confronté au cas suivant : vous voulez envoyer un document par e-mail et qu’il reste modifiable mais vous ne savez pas si votre correspondant dispose de LibreOffice. Eh bien, vous pouvez au moins lui permettre de le lire. Pour cela, nous allons créer des PDF hybrides.

Qu’est-ce qu’un PDF hybride ? C’est en apparence un PDF normal qui s’ouvre comme n’importe quel autre PDF. Mais il contient également une copie de votre document texte. Si vous ouvrez un tel PDF avec Writer, vous croirez éditer un odt.

Pour créer un tel PDF, aller dans Barre d’outil : Fichier > Exporter au format PDF… Dans la fenêtre qui s’ouvre sélectionnez Incorporer le fichier OpenDocument.

17.4 LaTeX et LibreOffice

17.4.1 L’import

17.4.2 L’export

17.5 Envoyer des documents sur les serveurs de l’école facilement

18 Auteurs et licence

Ce tutoriel a été écrit par Julien Enselme (jenselme@ec-m.fr). Il est publié sous licence CC-BY-SA. Vous pouvez trouver une version actualisée de ce document :

Liste des contributeurs :

Florent Torregrosa (bibliographie, formulaire, publipostage).

19 Pour aller plus loin

La documentation officielle de LibreOffice : https://www.libreoffice.org/get-help/documentation/
Le publipostage : http://ia71.ac-dijon.fr/chalon2/tutoriels/oo/Publipostage_OOo.pdf

LaTeX : annexe

2013-10-20T00:00:00+02:00

Comment faire des annexes (numéroté par des lettres plutôt que par des chiffres) en LaTeX ? C’est très facile avec le paquet appendix. Il suffit juste de l’utiliser : \usepackage{appendix}. Ensuite, placé toutes vos annexes dans l’environnement appendix comme ci-dessous.

\begin{appendix}
    \chapter{Les micromodules}
    \label{a_micro}

    \begin{center}
        \includegraphics[scale=0.4, angle=90]{./img/micro.jpg}
    \end{center}

    \chapter{L’encartage bancaire}
    \label{a_encartage}

    \begin{center}
        \includegraphics[scale=0.28, angle=90]{./img/encartage.jpg}
    \end{center}

    \chapter{La personnalisation}
    \label{a_perso}

    \begin{center}
        \includegraphics[scale=0.28, angle=90]{./img/perso.jpg}
    \end{center}

    \chapter{Les différentes étapes de la production}
    \label{a_prod}

    \begin{center}
        \includegraphics[scale=0.5, angle=0]{./img/stage_prod.png}
    \end{center}
\end{appendix}

LaTeX : glossaire

2013-10-20T00:00:00+02:00

Pour mon rapport de stage, j’ai eu besoin de faire un glossaire en LaTeX. Contrairement à ce que l’on pourrait croire, c’est assez simple grâce au paquet glossaries. L’option nonumberlist permet de ne pas avoir les numéros de pages pour lesquelles le terme est utilisé à côté du texte explicatif.

\documentclass[12pt]{report}
\usepackage[frenchb]{babel}
\usepackage{libertine}
\usepackage[nonumberlist]{glossaries}Cela fait un moment que je me demande comment changer l’apparence des titres de chapitre sous LaTeX. Parce que avoir un gros Chapitre 1 ce n’est pas toujours ce que je veux.

Heureusement, le paquet titlesec vient à la rescousse. Il ajoute la commande titleformat Ainsi, pour avoir un titre qui ressemble à ça :

makeglossaries

begin{document} input{includes/glossaire.tex} end{document}

Et dans le fichier glossaire.tex :

\newglossaryentry{waffer}{name={waffer}, description={Un waffer est la plaque sur laquelle se trouve les puces. C’est sous cette forme que Gemalto reçoit les puces de son fournisseur}}

LaTeX : une jolie page de garde

2013-10-20T00:00:00+02:00

LaTeX sait faire des pages de titre avec la commande maketitle si on a au préalable renseigné les champs \author{Moi}, \date{\today} et \title{Mon titre}. Néanmoins, elle fait un peu austère. Heureusement, l’environement titlepage vient à la rescousse. Avec le code ci-dessous, on obtient un très beau résultat.

\documentclass[12pt, openany]{report}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[a4paper,left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}
\usepackage[frenchb]{babel}
\usepackage{libertine}
\usepackage[pdftex]{graphicx}

\setlength{\parindent}{0cm}
\setlength{\parskip}{1ex plus 0.5ex minus 0.2ex}
\newcommand{\hsp}{\hspace{20pt}}
\newcommand{\HRule}{\rule{\linewidth}{0.5mm}}

\begin{document}

\begin{titlepage}
  \begin{sffamily}
  \begin{center}

    % Upper part of the page. The '~' is needed because \\
    % only works if a paragraph has started.
    \includegraphics[scale=0.04]{img1.JPG}~\\[1.5cm]

    \textsc{\LARGE École Centrale Marseille}\\[2cm]

    \textsc{\Large Rapport de stage 1A}\\[1.5cm]

    % Title
    \HRule \\[0.4cm]
    { \huge \bfseries Opérateur expédition produits finis\\[0.4cm] }

    \HRule \\[2cm]
    \includegraphics[scale=0.2]{img2.JPG}
    \\[2cm]

    % Author and supervisor
    \begin{minipage}{0.4\textwidth}
      \begin{flushleft} \large
        Moi \textsc{Même}\\
        Promo 2015\\
      \end{flushleft}
    \end{minipage}
    \begin{minipage}{0.4\textwidth}
      \begin{flushright} \large
        \emph{Tuteur :} M. Le \textsc{Tuteur}\\
        \emph{Chef d'équipe : } M. Chef \textsc{D’Équipe}
      \end{flushright}
    \end{minipage}

    \vfill

    % Bottom of the page
    {\large 1\ier{} Juillet 2013 — 30 Août 2013}

  \end{center}
  \end{sffamily}
\end{titlepage}
\end{document}

LaTeX : de jolis titres

2013-10-20T00:00:00+02:00

Cela fait un moment que je me demande comment changer l’apparence des titres de chapitre sous LaTeX. Parce que avoir un gros Chapitre 1 ce n’est pas toujours ce que je veux.

Heureusement, le paquet titlesec vient à la rescousse. Il ajoute la commande titleformat Ainsi, pour avoir un titre qui ressemble à ça :

Utilisez simplement :

\documentclass[12pt, openany]{report}
\usepackage[frenchb]{babel}
\usepackage{libertine}
\usepackage{titlesec}

\newcommand{\hsp}{\hspace{20pt}}

\titleformat{\chapter}[hang]{\Huge\bfseries\sffamily}{\thechapter\hsp}{0pt}{\Huge\bfseries\sffamily}
\begin{document}
\chapter{Introduction}
\end{document}

Cela enlève à la fois le chapter et modifie la police pour avoir celle qui me convient.

Citons également le paquet fncychap et le type de document memoire (documentclass{memoir}) qui changent également l’apparence des titres.

Source

Pour voir encore plus de titre : http://texblog.org/2012/07/03/fancy-latex-chapter-styles/

LaTeX: a nice front page

2013-10-20T00:00:00+02:00

LaTeX can make front pages with the \maketitle command if the fields \author{Me}, \date{\today} and \title{My title} are filled. However, it is quite dry. Luckily, the titlepage environment comes to the rescue: with the code below, you can get a very nice result!

\documentclass[12pt, openany]{report}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[a4paper,left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}
\usepackage[frenchb]{babel}
\usepackage{libertine}
\usepackage[pdftex]{graphicx}

\setlength{\parindent}{0cm}
\setlength{\parskip}{1ex plus 0.5ex minus 0.2ex}
\newcommand{\hsp}{\hspace{20pt}}
\newcommand{\HRule}{\rule{\linewidth}{0.5mm}}

\begin{document}

\begin{titlepage}
  \begin{sffamily}
  \begin{center}

    % Upper part of the page. The '~' is needed because \\
    % only works if a paragraph has started.
    \includegraphics[scale=0.04]{img1.JPG}~\\[1.5cm]

    \textsc{\LARGE École Centrale Marseille}\\[2cm]

    \textsc{\Large Rapport de stage 1A}\\[1.5cm]

    % Title
    \HRule \\[0.4cm]
    { \huge \bfseries Opérateur expédition produits finis\\[0.4cm] }

    \HRule \\[2cm]
    \includegraphics[scale=0.2]{img2.JPG}
    \\[2cm]

    % Author and supervisor
    \begin{minipage}{0.4\textwidth}
      \begin{flushleft} \large
        Moi \textsc{Même}\\
        Promo 2015\\
      \end{flushleft}
    \end{minipage}
    \begin{minipage}{0.4\textwidth}
      \begin{flushright} \large
        \emph{Tuteur :} M. Le \textsc{Tuteur}\\
        \emph{Chef d'équipe : } M. Chef \textsc{D’Équipe}
      \end{flushright}
    \end{minipage}

    \vfill

    % Bottom of the page
    {\large 1\ier{} Juillet 2013 — 30 Août 2013}

  \end{center}
  \end{sffamily}
\end{titlepage}
\end{document}

Pourquoi j'abandonne LaTeX et je passe à org-mode

2013-08-23T00:00:00+02:00

Oui, LaTeX est surpuissant et permet de faire de très belles choses étrange assez facilement. Même si je lui préfère LibreOffice pour certaines tâches (tout ce qui contient beaucoup d'images notamment), j'aime LaTeX (et j'ai même fait mon CV en LaTeX ;-)). LaTeX nous force à organiser le contenu, tout est un dans fichier texte (ce qui est à mon avis pérenne) et pouvoir tout faire au clavier est un gros plus (on ne se perd pas dans l'interface).

Pour contre, soyons franc parfois ce n'est pas évident d'avoir le résultat souhaité et il faut chercher sur pleins de forum et mettre les mains dans le camboui.

Il y a quelques temps déjà, je cherchais un logiciel pour prendre des notes. Je suis tombé sur pas mal de logiciels graphiques avant de découvrir org-mode. C'est un mode d'Emacs intégralement dédié à la prise de note et il est présent de façon standard depuis les dernières versions d'Emacs. Et en plus de pouvoir remplacer LaTeX (on peut faire de l'export vers LaTeX, html, pdf) ce logiciel fait bien plus. Et c'est pourquoi j'abonne LaTeX pour passer à org-mode. Voyons tout de suite pourquoi.

La syntaxe

LaTeX a une syntaxe très verbeuse, tellement verbeuse qu'elle en devient lourde (un peu comme le HTML à mon avis) et qui utilise lourdement le backslash pas toujours accessible facilement. À mon avis, pour l'édition de fichier texte, la syntaxe des divers wiki est vraiment plus performante.

Et voici Org-mode. La syntaxe d'org-mode est simple et élégante. En LaTeX pour créer un titre, il faut \title{Mon titre} soit 8 caractères qui n'ont pas d'utilité pour le lecteur. Et pour les sous-titres le nombre de caractères explose. En Org-mode : * Mon titre. Et c'est tout ! Pour faire un sous-titre ** Sous titre. Et on peut en mettre autant que l'on veut ! Bon seulement 6 si on veut faire de l’export html.

Autre point très intéressant, Emacs va plier le texte de ces section tout seul. Et on peut déplier facilement, en appuyant sur TAB.

Pour mettre en gras, il suffit de mettre le texte entre * et pour l'italique, entre /. Pour faire des listes, il suffit de commencer la ligne par un - ou un nombre.

Pas mal hein ? Mais on peut faire bien plus.

Les TODO listes

Comme je l'ai dis en introduction Org-mode est fait pour prendre des notes. Mais il fait aussi des TODO list. Placez-vous sur une ligne et tapez TODO devant celle -ci (ou utilisez le racourcis clavier ^-c ^-e). Lors que l'item est fait, réutiliser le raccourcis clavier. Et hop, TODO devient DONE. Mais ce n'est pas tout. Considérons la liste ci-dessous, que l'on écrit telle quelle dans Emacs :

* TODO remplir mon blog [0%]
** TODO Faire des articles
- [ ] Sur Org-mode
- [ ] Sur autre chose
** TODO Finir les tutos

Et bien Lorsqu'on aura coché toutes les cases de Faire des articles (avec ^-c ^-e), cette item passera en DONE automatiquement et le pourcentage de complétion de Remplir mon blog passera à 50% (si ce pourcentage est faux, forcer sa réévaluation avec ^-c ^-c).

* TODO remplir mon blog [50%]
** DONE Faire des articles
- [X] Sur Org-mode
- [X] Sur autre chose
** TODO Finir les tutos

Plutôt pas mal, non ?

Ils sont beaux mes tableaux, ils sont beaux

Autre point très lourd en LaTeX : les tableaux. La syntaxe est lourde et on ne sait pas forcément quel sera le résultat. En Org-mode c'est simple et efficace (vous vous en seriez doutez). Entrez :

| Produits | Prix |
|-

Appuyez sur TAB et hop :

| Produits | Prix |
|----------+------|
|          |      |

Et on continue à modifier en appuyant sur TAB pour passer d'une case à l'autre (Emacs va les ajouter tout seul) et les redimensionner comme un grand.

Et on peut même le transformer en tableur, mais là je vous revoie vers la doc.

L'export

Mais un joli fichier texte n'est pas adapté pour tout les usages. Un petit pdf (très joli car généré par LaTeX) ou un fichier html peut être plus approprié. Org-mode peut évidemment exporter votre TODO liste. Il existe plein de raccourcis clavier complexe (c'est un peu le problème d'Emacs) mais vous pouvez aussi passer par les menus (c'est un peu l'un des avantages d'Emacs sur vim). Ou alors :

M-x org-mode-export-TAB

Et choisissez l'export qui vous convient.

On peut faire du LaTeX avec

Les syntaxes légères c'est bien joli mais parfois ce n'est pas suffisant. Si je veux faire des équations très compliquées avec pleins de symboles étranges ça ne suffit pas. Heureusement, avec Org-mode on peut insérer du code LaTeX directement, sans modification. Il sera conservé lors de l'export en LaTeX et interprété lors de la compilation en pdf (même si vous exportez directement en pdf). Je peux donc faire :

* Equation de Jenselme
La voilà, avec $h = \frac{h}{2}$ :
$$\frac{p}{2m}|\Psi> + \sum_{i = 1}^{N} x^i = 0$$

Et on obtient après compilation :

Conclusion

Voilà pourquoi je passe à Org-mode. On peut évidemment aussi faire des présentations en beamer avec (avec une jolie syntaxe toute simple et très claire), mettre du code source, le transformer en calendrier à partir des TODO list en mettant une date de fin,… Je vous laisse regarder la doc pour ça.

Emacs : se connecter en ssh

2013-08-08T00:00:00+02:00

Emacs n’est pas installé sur le seveur où vous devez modifier des fichiers ? Vous détestez vi et trouvez nano trop limité ? Eh bien vous pouvez utiliser Emacs avec votre configuration habituelle !

Pour que Emacs se connecte au serveur, tapez simplement : ^x ^f /ssh2:USER@SEVER. Emacs va alors se connecter en ssh au serveur et vous pourrez naviguer dans le dossier home du serveur comme si vous étiez en local. Vous pouvez évidememt directement préciser quel dossier/fichier vous désirez ouvrir.

C’est beau non ?

Emacs : édition multiple

2013-08-08T00:00:00+02:00

J’ai récemment eu besoin d’éditer plusieurs fichiers contenant la même chaîne de caractères pour la remplacer. Je pensais tout d’abord utiliser sed avant de regarder ce que pouvais faire Emacs. Eh bien non seulement il peut le faire, mais il peut le faire facilement en vous laissant un certain "contrôle" lors du remplacement.

Ouvrons nos fichiers

Étape 1 : trouvez tout les fichiers à éditer. Peu importe leur localisation, on va utiliser la commande find qui s’intègre très bien avec Emacs. Lancez Emacs. Tapez ensuite :

M-x find-dired

Et ouvrez le répertoire dans lequel vous voulez lancer votre recherche. Emacs vous demande alors les options à passer à find. Allez voir le man de find pour savoir tout ce que vous pouvez donner comme argument. Par exemple :

-type f -name settings.php -maxdepth 3

ce qui nous retournera tous les fichiers nommés settings.php situé à une profondeur maximale de 3. Libre à vous de faire plus compliqué suivant vous besoin. (Ne vous prenez pas trop la tête non plus).

Sélectionnons les fichiers à éditer

Une fois la recherche terminée, Emacs vous affiche tous les résultats. Et c’est à vous de les sélectionner. Nul besoin de trop se prendre la tête avec find à l’étape précédente. Aller sur la ligne/le nom du fichier à éditer et appuyez sur la touche t. Le nom du fichier change de couleur. Il est sélectionné ! Pour sélectionner tous les fichiers, tapez m.

Cherchons et remplaçons

Tapez

M-x dired-do-query-replace-regexp

pour procéder à la recherche. Emacs vous demande tout d’abord l’expression régulière à chercher, puis la chaîne par laquelle il faut la remplacer. Emacs va alors vous proposer fichier par fichier et occurrence par occurrence le remplacement. Tapez "y" pour le faire, "n" pour passer à l’occurrence suivante. "!" remplace toutes les occurrences du buffer courant et "Y" toutes les occurrences dans tous les fichiers. "?" affiche la liste complète des commandes disponibles.

Sauvegardons

Pour sauvegarder tous vos fichiers en une seule fois, tapez

M-x ibuffer

Sélectionnez les fichers non sauvegardés en tapant * U. Sauvegarder les avec S. Et voilà, c’est fini ;-).

Conclusion

Nous venons facilement de remplacer une chaîne présente dans plusieurs fichiers. Il est évidement possible de le faire sur des fichiers présents sur un serveur. Vous avez également accès à des fonctions de sélection de fichiers plus avancées, comme sélectionner tous les fichiers contenant une expression régulière et à d’autres fonctions de manipulation de fichier (comme les renommer).

Maintenant, vous savez quel outil prendre pour éditer vos fichiers !

Emacs : regexp builder

2013-08-08T00:00:00+02:00

Perdu avec une expression régulière ? Emacs peut vous aider ! Il propose en effet un outil intitulé regexp-builder qui vérifie la syntaxe de la regexp durant la frappe et cherche une correspondance dans le buffer courant.

Pour le lancer, tapez simplement M-x regexp-builder. Entrez alors simplement votre regexp entre les guillemets.

Attention toute fois, Emacs étant codé en Lisp, par défaut regexp-builder utilise la syntaxe de Lisp pour les regexp qui diffèrent de la syntaxe habitulle (pour ignorer un caractère, il faut utiliser \\). Heureusement, on peut changer le mode d’édition ! Tapez ^-c TAB et choisissez string pour avoir la syntaxe habituelle.

Connaître le niveau d'encre de son imprimante epson

2013-01-12T00:00:00+01:00

Il suffit de lancer en root :

sudo escputil -i -r /dev/usb/lp0