Jujens' blog - Trucs et astuces//www.jujens.eu/2023-12-02T00:00:00+01:00Ignore commits in git blame2023-12-02T00:00:00+01:002023-12-02T00:00:00+01:00Julien Enselmetag:www.jujens.eu,2023-12-02:/posts/en/2023/Dec/02/test/<p>Commits that change formatting are a pain in git because they make using <tt class="docutils literal">git blame</tt> much harder.
It turns out, you can ask git to ignore these commits when running git blame with the <tt class="docutils literal"><span class="pre">--ignore-rev</span></tt> option and passing it a commit hash.
You can even do things better and create …</p><p>Commits that change formatting are a pain in git because they make using <tt class="docutils literal">git blame</tt> much harder.
It turns out, you can ask git to ignore these commits when running git blame with the <tt class="docutils literal"><span class="pre">--ignore-rev</span></tt> option and passing it a commit hash.
You can even do things better and create a <tt class="docutils literal"><span class="pre">.git-blame-ignore-revs</span></tt> listing the full commit hashes you want to ignore and pass it to git with <tt class="docutils literal"><span class="pre">--ignore-revs-file</span></tt>.</p>
<p>To make things even more simple, you can configure git to use this file automatically with:</p>
<pre class="literal-block">
git config blame.ignoreRevsFile .git-blame-ignore-revs
</pre>
<p>It even <a class="reference external" href="https://docs.github.com/en/repositories/working-with-files/using-files/viewing-a-file#ignore-commits-in-the-blame-view">works natively on GitHub</a>!</p>
<p>Initially seen on <a class="reference external" href="https://black.readthedocs.io/en/stable/guides/introducing_black_to_your_project.html#avoiding-ruining-git-blame">the Black documentation</a>.</p>
Some cool type tips in TypeScript2023-08-17T00:00:00+02:002023-08-17T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2023-08-17:/posts/en/2023/Aug/17/ts-type-tips/<p>Here is an unsorted type tips I find useful in TypeScript.
I may update it when I find new ones.
Don't hesitate to share yours in the comments!</p>
<p>Another good start, is to read <a class="reference external" href="https://www.typescriptlang.org/docs/handbook/utility-types.html">this page</a> from the handbook which list various builtin utility types.
If you are really motivated …</p><p>Here is an unsorted type tips I find useful in TypeScript.
I may update it when I find new ones.
Don't hesitate to share yours in the comments!</p>
<p>Another good start, is to read <a class="reference external" href="https://www.typescriptlang.org/docs/handbook/utility-types.html">this page</a> from the handbook which list various builtin utility types.
If you are really motivated and want to check your skills, you can try <a class="reference external" href="https://github.com/type-challenges/type-challenges">these challenges</a>: I stopped after some intermediate ones, but even the easy ones will help you practice and learn new things (I sure did!).</p>
<div class="contents topic" id="sommaire">
<p class="topic-title">Sommaire</p>
<ul class="simple">
<li><a class="reference internal" href="#functions-related" id="toc-entry-1">Functions related</a></li>
<li><a class="reference internal" href="#array-related" id="toc-entry-2">Array related</a></li>
<li><a class="reference internal" href="#object-related" id="toc-entry-3">Object related</a></li>
<li><a class="reference internal" href="#misc" id="toc-entry-4">Misc</a></li>
<li><a class="reference internal" href="#advanced" id="toc-entry-5">Advanced</a></li>
</ul>
</div>
<div class="section" id="functions-related">
<h2><a class="toc-backref" href="#toc-entry-1">Functions related</a></h2>
<p>You can use the builtin <tt class="docutils literal">ReturnType</tt> type to get the return type of a function:</p>
<pre class="code typescript literal-block">
<span class="kd">const</span><span class="w"> </span><span class="nx">toMyString</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="nx">obj</span><span class="o">:</span><span class="w"> </span><span class="kt">object</span><span class="p">)</span><span class="o">:</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">obj</span><span class="p">.</span><span class="nx">toString</span><span class="p">()</span><span class="w">
</span><span class="c1">// ToString will be string.</span><span class="w">
</span><span class="kr">type</span><span class="w"> </span><span class="nx">ToString</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">ReturnType</span><span class="o"><</span><span class="ow">typeof</span><span class="w"> </span><span class="nx">toMyString</span><span class="o">></span>
</pre>
<p>Similarly, you can use the builtin <tt class="docutils literal">Parameters</tt> type to get the types of the parameters of a function:</p>
<pre class="code typescript literal-block">
<span class="kd">const</span><span class="w"> </span><span class="nx">toMyString</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="nx">obj</span><span class="o">:</span><span class="w"> </span><span class="kt">object</span><span class="p">)</span><span class="o">:</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">obj</span><span class="p">.</span><span class="nx">toString</span><span class="p">()</span><span class="w">
</span><span class="c1">// ToStringParams will be [obj: object]</span><span class="w">
</span><span class="kr">type</span><span class="w"> </span><span class="nx">ToStringParams</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">Parameters</span><span class="o"><</span><span class="ow">typeof</span><span class="w"> </span><span class="nx">toMyString</span><span class="o">></span>
</pre>
</div>
<div class="section" id="array-related">
<h2><a class="toc-backref" href="#toc-entry-2">Array related</a></h2>
<p>You can define tuples (that is arrays with a fixed number of elements of a given type) like this:</p>
<pre class="code typescript literal-block">
<span class="kr">type</span><span class="w"> </span><span class="nx">Coords</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="kt">number</span><span class="p">,</span><span class="w"> </span><span class="kt">number</span><span class="p">]</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">a</span><span class="o">:</span><span class="w"> </span><span class="kt">Coords</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mf">0</span><span class="p">,</span><span class="w"> </span><span class="mf">0</span><span class="p">]</span><span class="w">
</span><span class="c1">// This is a type error.</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">b</span><span class="o">:</span><span class="w"> </span><span class="kt">Coords</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mf">0</span><span class="p">,</span><span class="w"> </span><span class="mf">0</span><span class="p">,</span><span class="w"> </span><span class="mf">0</span><span class="p">]</span>
</pre>
<p>You can access array properties like this:</p>
<pre class="code typescript literal-block">
<span class="kr">type</span><span class="w"> </span><span class="nx">Length</span><span class="o"><</span><span class="nx">T</span><span class="w"> </span><span class="k">extends</span><span class="w"> </span><span class="nx">unknown</span><span class="p">[]</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">T</span><span class="p">[</span><span class="s1">'length'</span><span class="p">]</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">a</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">string</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s1">'Hello'</span><span class="p">,</span><span class="w"> </span><span class="s1">'World'</span><span class="p">]</span><span class="w">
</span><span class="c1">// CustomLength will be 2. If a were an array, CustomLength would be a number.</span><span class="w">
</span><span class="kr">type</span><span class="w"> </span><span class="nx">CustomLength</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">Length</span><span class="o"><</span><span class="ow">typeof</span><span class="w"> </span><span class="nx">a</span><span class="o">></span>
</pre>
<p>You can use array types as array in types:</p>
<pre class="code typescript literal-block">
<span class="kr">type</span><span class="w"> </span><span class="nx">Concat</span><span class="o"><</span><span class="nx">T</span><span class="w"> </span><span class="k">extends</span><span class="w"> </span><span class="nx">any</span><span class="p">[],</span><span class="w"> </span><span class="nx">U</span><span class="w"> </span><span class="k">extends</span><span class="w"> </span><span class="nx">any</span><span class="p">[]</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[...</span><span class="nx">T</span><span class="p">,</span><span class="w"> </span><span class="p">...</span><span class="nx">U</span><span class="p">];</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">a</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">string</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s1">'Hello'</span><span class="p">,</span><span class="w"> </span><span class="s1">'World'</span><span class="p">]</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">b</span><span class="o">:</span><span class="w"> </span><span class="kt">number</span><span class="p">[]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mf">1</span><span class="p">,</span><span class="w"> </span><span class="mf">100</span><span class="p">]</span><span class="w">
</span><span class="c1">// c will be of type [string, string, ...number[]]. If a were an array, c would be of type (string | number)[].</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">c</span><span class="o">:</span><span class="w"> </span><span class="kt">Concat</span><span class="o"><</span><span class="ow">typeof</span><span class="w"> </span><span class="nx">a</span><span class="p">,</span><span class="w"> </span><span class="ow">typeof</span><span class="w"> </span><span class="nx">b</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[...</span><span class="nx">a</span><span class="p">,</span><span class="w"> </span><span class="p">...</span><span class="nx">b</span><span class="p">]</span>
</pre>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">See the <a class="reference internal" href="#advanced">Advanced</a> section below for more on extends.</p>
</div>
</div>
<div class="section" id="object-related">
<h2><a class="toc-backref" href="#toc-entry-3">Object related</a></h2>
<ul class="simple">
<li><tt class="docutils literal">Partial<Type></tt> will make all properties of a object optional.</li>
<li><tt class="docutils literal">Required<Type></tt> will make all properties of an object required.</li>
<li><tt class="docutils literal">Readonly<Type></tt> will make all properties of an object read only.</li>
<li><tt class="docutils literal">Pick<Type, Keys></tt> allows you to define a new type with only the selected properties.</li>
<li><tt class="docutils literal">Omit<Type, Keys></tt> allows you to define a new type without the selected properties.</li>
</ul>
</div>
<div class="section" id="misc">
<h2><a class="toc-backref" href="#toc-entry-4">Misc</a></h2>
<ul class="simple">
<li><tt class="docutils literal">typeof</tt> allows you to get the type of a variable.</li>
<li><tt class="docutils literal">Exclude<UnionType, ExcludedMembers></tt> is like <tt class="docutils literal">Omit</tt> but for union types.</li>
<li><tt class="docutils literal">Extract<Type, <span class="pre">Union>``constructs</span> a type by extracting from ``Type</tt> all union members that are assignable to <tt class="docutils literal">Union</tt>.</li>
<li><tt class="docutils literal">keyof</tt> allows you to build a union type from the keys of an object.</li>
<li><tt class="docutils literal">extends</tt> allows you to put constraints on a type. See examples in the <a class="reference internal" href="#advanced">Advanced</a> section.</li>
</ul>
</div>
<div class="section" id="advanced">
<h2><a class="toc-backref" href="#toc-entry-5">Advanced</a></h2>
<p>You can use <tt class="docutils literal">infer</tt> to infer types to build more complex ones:</p>
<pre class="code typescript literal-block">
<span class="kr">type</span><span class="w"> </span><span class="nx">MyAwaitable</span><span class="o"><</span><span class="nx">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">T</span><span class="w"> </span><span class="k">extends</span><span class="w"> </span><span class="nb">Promise</span><span class="o"><</span><span class="nx">infer</span><span class="w"> </span><span class="nx">Value</span><span class="o">></span><span class="w"> </span><span class="o">?</span><span class="w"> </span><span class="nx">Value</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="kt">never</span><span class="p">;</span><span class="w">
</span><span class="c1">// MyValue is a string.</span><span class="w">
</span><span class="kr">type</span><span class="w"> </span><span class="nx">MyValue</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">MyAwaitable</span><span class="o"><</span><span class="nb">Promise</span><span class="o"><</span><span class="kt">string</span><span class="o">>></span>
</pre>
<pre class="code typescript literal-block">
<span class="kr">type</span><span class="w"> </span><span class="nx">First</span><span class="o"><</span><span class="nx">T</span><span class="w"> </span><span class="k">extends</span><span class="w"> </span><span class="nx">any</span><span class="p">[]</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">T</span><span class="w"> </span><span class="k">extends</span><span class="w"> </span><span class="p">[</span><span class="nx">infer</span><span class="w"> </span><span class="nx">P</span><span class="p">,</span><span class="w"> </span><span class="p">...</span><span class="nx">any</span><span class="p">[]]</span><span class="w"> </span><span class="o">?</span><span class="w"> </span><span class="nx">P</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="kt">never</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">a</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">number</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s1">'Hello'</span><span class="p">,</span><span class="w"> </span><span class="mf">1</span><span class="p">]</span><span class="w">
</span><span class="c1">// Value is a string.</span><span class="w">
</span><span class="kr">type</span><span class="w"> </span><span class="nx">Value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">First</span><span class="o"><</span><span class="ow">typeof</span><span class="w"> </span><span class="nx">a</span><span class="o">></span>
</pre>
<pre class="code typescript literal-block">
<span class="kr">type</span><span class="w"> </span><span class="nx">SavedData</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nx">options</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="nx">isEnabled</span><span class="o">:</span><span class="w"> </span><span class="kt">boolean</span><span class="p">};</span><span class="w">
</span><span class="nx">authentication</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="nx">username</span><span class="o">:</span><span class="w"> </span><span class="kt">string</span><span class="p">};</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">saveData</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o"><</span><span class="nx">Key</span><span class="w"> </span><span class="k">extends</span><span class="w"> </span><span class="nx">keyof</span><span class="w"> </span><span class="nx">SavedData</span><span class="o">></span><span class="p">(</span><span class="nx">key</span><span class="o">:</span><span class="w"> </span><span class="kt">Key</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="o">:</span><span class="w"> </span><span class="kt">SavedData</span><span class="p">[</span><span class="nx">Key</span><span class="p">])</span><span class="o">:</span><span class="w"> </span><span class="ow">void</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nx">localStorage</span><span class="p">.</span><span class="nx">setItem</span><span class="p">(</span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nb">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">(</span><span class="nx">data</span><span class="p">));</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="nx">saveData</span><span class="p">(</span><span class="s1">'options'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">isEnabled</span><span class="o">:</span><span class="w"> </span><span class="kt">false</span><span class="p">})</span><span class="w">
</span><span class="nx">saveData</span><span class="p">(</span><span class="s1">'authentication'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">username</span><span class="o">:</span><span class="w"> </span><span class="s1">'Jujens'</span><span class="p">})</span><span class="w">
</span><span class="c1">// Those will be reported as errors.</span><span class="w">
</span><span class="nx">saveData</span><span class="p">(</span><span class="s1">'options'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">username</span><span class="o">:</span><span class="w"> </span><span class="s1">'Jujens'</span><span class="p">})</span><span class="w">
</span><span class="nx">saveData</span><span class="p">(</span><span class="s1">'authentication'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">isEnabled</span><span class="o">:</span><span class="w"> </span><span class="kt">false</span><span class="p">})</span>
</pre>
<p>You can define types with string interpolation to force part of a string to be of a certain type:</p>
<pre class="code typescript literal-block">
<span class="kr">type</span><span class="w"> </span><span class="nx">Width</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sb">`</span><span class="si">${</span><span class="kt">number</span><span class="si">}</span><span class="sb">px`</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">a</span><span class="o">:</span><span class="w"> </span><span class="kt">Width</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'18px'</span><span class="w">
</span><span class="c1">// All these will be reported as errors.</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">b</span><span class="o">:</span><span class="w"> </span><span class="kt">Width</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">18</span><span class="p">;</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">c</span><span class="o">:</span><span class="w"> </span><span class="kt">Width</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'18'</span><span class="p">;</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="nx">d</span><span class="o">:</span><span class="w"> </span><span class="kt">Width</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'totopx'</span>
</pre>
</div>
Use the same function as context manager and decorator2022-09-25T00:00:00+02:002022-09-25T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2022-09-25:/posts/en/2022/Sep/25/context-manager-decorators/<p>I recently learned that context managers created with <tt class="docutils literal">@contextmanager</tt> can be used either as a context manager or a decorator:</p>
<pre class="code python literal-block">
<span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">contextmanager</span>
<span class="nd">@contextmanager</span>
<span class="k">def</span> <span class="nf">test_context</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Entering'</span><span class="p">)</span>
<span class="k">yield</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Leaving'</span><span class="p">)</span>
<span class="k">with</span> <span class="n">test_context</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Inside'</span><span class="p">)</span>
<span class="nd">@test_context</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">test_decorated</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Decorated'</span><span class="p">)</span>
<span class="n">test_decorated</span><span class="p">()</span>
</pre>
<p>We will yield:</p>
<pre class="literal-block">
Entering
Inside
Leaving
Entering
Decorated …</pre><p>I recently learned that context managers created with <tt class="docutils literal">@contextmanager</tt> can be used either as a context manager or a decorator:</p>
<pre class="code python literal-block">
<span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">contextmanager</span>
<span class="nd">@contextmanager</span>
<span class="k">def</span> <span class="nf">test_context</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Entering'</span><span class="p">)</span>
<span class="k">yield</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Leaving'</span><span class="p">)</span>
<span class="k">with</span> <span class="n">test_context</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Inside'</span><span class="p">)</span>
<span class="nd">@test_context</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">test_decorated</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Decorated'</span><span class="p">)</span>
<span class="n">test_decorated</span><span class="p">()</span>
</pre>
<p>We will yield:</p>
<pre class="literal-block">
Entering
Inside
Leaving
Entering
Decorated
Leaving
</pre>
<p>In both cases, we can pass argument to our context manager/decorator like this:</p>
<pre class="code python literal-block">
<span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">contextmanager</span>
<span class="nd">@contextmanager</span>
<span class="k">def</span> <span class="nf">test_context</span><span class="p">(</span><span class="n">value</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Entering:'</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="k">yield</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Leaving'</span><span class="p">)</span>
<span class="k">with</span> <span class="n">test_context</span><span class="p">(</span><span class="s1">'Hello there!'</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Inside'</span><span class="p">)</span>
<span class="nd">@test_context</span><span class="p">(</span><span class="s1">'Hello there!'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_decotared</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Decorated'</span><span class="p">)</span>
</pre>
<p>But what if we need to access the arguments of the function inside the decorator?
When used as a context manager, we have to pass them, but we cannot access them directly when we use it as a decorator.
To do this, we need to take inspiration from <tt class="docutils literal">contextlib.ContextDecorator</tt> (what <tt class="docutils literal">contextlib.contextmanager</tt> is using) to to some work ourselves:</p>
<pre class="code python literal-block">
<span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">wraps</span>
<span class="k">class</span> <span class="nc">test_context</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">option_value</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">function_value</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_option_value</span> <span class="o">=</span> <span class="n">option_value</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_function_value</span> <span class="o">=</span> <span class="n">function_value</span>
<span class="k">def</span> <span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># Called when entering the ctx manager.</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Entering:'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_option_value</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_function_value</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span>
<span class="k">def</span> <span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">exc</span><span class="p">):</span>
<span class="c1"># Called when leaving the ctx manager.</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Leaving'</span><span class="p">)</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">func</span><span class="p">):</span>
<span class="c1"># Called when used as a decorator.</span>
<span class="nd">@wraps</span><span class="p">(</span><span class="n">func</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="n">function_value</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_function_value</span> <span class="o">=</span> <span class="n">function_value</span>
<span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">_recreate_cm</span><span class="p">():</span>
<span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="n">function_value</span><span class="p">)</span>
<span class="k">return</span> <span class="n">wrapper</span>
<span class="k">def</span> <span class="nf">_recreate_cm</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># Taken as is from contextlib.ContextDecorator.</span>
<span class="k">return</span> <span class="bp">self</span>
<span class="k">with</span> <span class="n">test_context</span><span class="p">(</span><span class="s1">'Option value'</span><span class="p">,</span> <span class="s1">'Function value'</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Inside'</span><span class="p">)</span>
<span class="nd">@test_context</span><span class="p">(</span><span class="s1">'Option value'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_decorated</span><span class="p">(</span><span class="n">value</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Decorated'</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="n">test_decorated</span><span class="p">(</span><span class="s1">'Function value'</span><span class="p">)</span>
</pre>
<p>This will yield:</p>
<pre class="literal-block">
Entering: Option value Function value
Inside
Leaving
Entering: Option value Function value
Decorated Function value
Leaving
</pre>
React hook to load fonts2022-06-11T00:00:00+02:002023-08-01T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2022-06-11:/posts/en/2022/Jun/11/use-fonts-hook/<p>If you need to load a specific font in a component, you can use this hook.
It will return <tt class="docutils literal">true</tt> when the fonts are loaded.
It relies on the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/API/Document/fonts">document.fonts API</a> to load the fonts.
If the fonts are already loaded, the promise will be fulfilled immediately and the …</p><p>If you need to load a specific font in a component, you can use this hook.
It will return <tt class="docutils literal">true</tt> when the fonts are loaded.
It relies on the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/API/Document/fonts">document.fonts API</a> to load the fonts.
If the fonts are already loaded, the promise will be fulfilled immediately and the component should be "immediately" re-rendered.</p>
<pre class="code javascript literal-block">
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">useEffect</span><span class="p">,</span><span class="w"> </span><span class="nx">useState</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"react"</span><span class="p">;</span><span class="w">
</span><span class="k">export</span><span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">useFonts</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(...</span><span class="nx">fontNames</span><span class="o">:</span><span class="w"> </span><span class="nx">string</span><span class="p">[])</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="kd">const</span><span class="w"> </span><span class="p">[</span><span class="nx">isLoaded</span><span class="p">,</span><span class="w"> </span><span class="nx">setIsLoaded</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">useState</span><span class="p">(</span><span class="kc">false</span><span class="p">);</span><span class="w">
</span><span class="nx">useEffect</span><span class="p">(()</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nb">Promise</span><span class="p">.</span><span class="nx">all</span><span class="p">(</span><span class="nx">fontNames</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="w">
</span><span class="p">(</span><span class="nx">fontName</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">fonts</span><span class="p">.</span><span class="nx">load</span><span class="p">(</span><span class="sb">`16px "</span><span class="si">${</span><span class="nx">fontName</span><span class="si">}</span><span class="sb">"`</span><span class="p">)</span><span class="w">
</span><span class="p">)).</span><span class="nx">then</span><span class="p">(()</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nx">setIsLoaded</span><span class="p">(</span><span class="kc">true</span><span class="p">);</span><span class="w">
</span><span class="p">});</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">[</span><span class="nx">fontNames</span><span class="p">]);</span><span class="w">
</span><span class="k">return</span><span class="w"> </span><span class="nx">isLoaded</span><span class="p">;</span><span class="w">
</span><span class="p">};</span>
</pre>
<p>Example usage:</p>
<pre class="code javascript literal-block">
<span class="kd">const</span><span class="w"> </span><span class="nx">areFontsLoaded</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">useFonts</span><span class="p">(</span><span class="s2">"Mistral"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Freestyle Script"</span><span class="p">);</span><span class="w">
</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="nx">areFontsLoaded</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="k">return</span><span class="w"> </span><span class="o"><</span><span class="nx">Loader</span><span class="w"> </span><span class="o">/></span><span class="p">;</span><span class="w">
</span><span class="p">}</span>
</pre>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">We don't need to check that the API is supported with <tt class="docutils literal">document && document.fonts</tt> since all browsers support it nowadays. See <a class="reference external" href="https://caniuse.com/mdn-api_document_fonts">here</a>.</p>
</div>
<p>As pointed out by <a class="reference external" href="https://www.jujens.eu/posts/en/2022/Jun/11/use-fonts-hook/#isso-320">choyeK</a>, for testing you may want to mock this with:</p>
<pre class="code javascript literal-block">
<span class="nb">Object</span><span class="p">.</span><span class="nx">defineProperty</span><span class="p">(</span><span class="nb">document</span><span class="p">,</span><span class="w"> </span><span class="s1">'fonts'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nx">value</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">load</span><span class="o">:</span><span class="w"> </span><span class="nb">Promise</span><span class="p">.</span><span class="nx">resolve</span><span class="p">({})</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">})</span>
</pre>
How to avoid CSRF token issues with Django when running on different sub-domains2022-02-13T00:00:00+01:002022-02-13T00:00:00+01:00Julien Enselmetag:www.jujens.eu,2022-02-13:/posts/en/2022/Feb/13/csrftoken-domains-django/<p>If you deploy multiple Django websites on your infrastructure on various subdomains, you may get issues about invalid CSRF tokens.
This is happening either because:</p>
<ul class="simple">
<li>You erase the cookie by using the same domain.
For instance, if you have your prod API at <tt class="docutils literal">api.jujens.eu</tt> and pre-production one at …</li></ul><p>If you deploy multiple Django websites on your infrastructure on various subdomains, you may get issues about invalid CSRF tokens.
This is happening either because:</p>
<ul class="simple">
<li>You erase the cookie by using the same domain.
For instance, if you have your prod API at <tt class="docutils literal">api.jujens.eu</tt> and pre-production one at <tt class="docutils literal"><span class="pre">preprod-api.jujens.eu</span></tt> and set the CSRF cookie on <tt class="docutils literal">.jujens.eu</tt>.
This is required you have a frontend app on its dedicated domain (let's say <tt class="docutils literal">app.jujens.eu</tt> and <tt class="docutils literal"><span class="pre">preprod-app.jujens.eu</span></tt>) and need it to send the CSRF token in the header as described <a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/csrf/#ajax">here</a>.</li>
<li>You have multiple cookies and thus will send a valid or invalid ones at random.
This can happened when you set your production cookie on <tt class="docutils literal">.jujens.eu</tt> and your pre-production ones on <tt class="docutils literal">.preprod.jujens.eu</tt>.
Your browser, when on <tt class="docutils literal">app.preprod.jujens.eu</tt> will see both cookies since their domain allow it.
As far as I know, there is no way to select the domain when reading cookies with JavaScript, so you are stuck.</li>
</ul>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Auth is not concerned by this, because its cookie will always be linked to a specific domain (eg <tt class="docutils literal">api.jujens.eu</tt>) and thus won't be sent to any other domain. I think that even if you do weird stuff and do things like this <tt class="docutils literal">preprod.api.jujens.eu</tt>, you should be fine because the cookie is strictly tied to the domain and not any subdomain like it would with <tt class="docutils literal">.api.jujens.eu</tt>.</p>
</div>
<p>How do you prevent these issues?
By changing the cookie name.
You can add a different <tt class="docutils literal">CSRF_COOKIE_NAME</tt> in your settings for each Django instances.
For instance: <tt class="docutils literal"><span class="pre">CSRF_COOKIE_NAME=f"csrf-${ENVIRONMENT}"</span></tt>.
You will then need to inject this cookie names in your frontend so it knows which one to read.
See <a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/csrf/#settings">the documentation</a> for more details.</p>
<p>I hope you found it useful.
Please comment if you have a question.</p>
Check domains2021-09-22T00:00:00+02:002021-09-22T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2021-09-22:/posts/en/2021/Sep/22/check-domains/<p>Here is a small script to allow you to easily check that a domain your manage is correct (ie responds correctly, is available with IPv4 and IPv6, only supports TLS 1.2+…).
It even has some color built in!
You can of course adapt it to fit your needs.</p>
<p>You …</p><p>Here is a small script to allow you to easily check that a domain your manage is correct (ie responds correctly, is available with IPv4 and IPv6, only supports TLS 1.2+…).
It even has some color built in!
You can of course adapt it to fit your needs.</p>
<p>You can use it like this:</p>
<pre class="literal-block">
./check-domains.sh www.jujens.eu
</pre>
<p>And you should get something like that:</p>
<pre class="literal-block">
---- Checking domain www.jujens.eu
Redirect to HTTPS: OK
Is reachable with IPv4: OK
Is reachable with IPv6: OK
Check HTTP 1.1: OK
Check HTTP2: OK
Check not reachable with TLS 1.0: OK
Check not reachable with TLS 1.1: OK
Check reachable with TLS 1.2: OK
Check not reachable with TLS 1.3: OK
Check that server responds with 200 or 404: OK
</pre>
<p>The script (<a class="reference external" href="//www.jujens.eu/static/check-domains/check-domains.sh">direct link</a>):</p>
<pre class="code bash literal-block">
<span class="ln"> 1 </span><span class="ch">#!/usr/bin/env bash
</span><span class="ln"> 2 </span><span class="ch"></span>
<span class="ln"> 3 </span><span class="nb">readonly</span> <span class="nv">red</span><span class="o">=</span><span class="s2">"\e[0;91m"</span>
<span class="ln"> 4 </span><span class="nb">readonly</span> <span class="nv">blue</span><span class="o">=</span><span class="s2">"\e[0;94m"</span>
<span class="ln"> 5 </span><span class="nb">readonly</span> <span class="nv">expand_bg</span><span class="o">=</span><span class="s2">"\e[K"</span>
<span class="ln"> 6 </span><span class="nb">readonly</span> <span class="nv">blue_bg</span><span class="o">=</span><span class="s2">"\e[0;104m</span><span class="si">${</span><span class="nv">expand_bg</span><span class="si">}</span><span class="s2">"</span>
<span class="ln"> 7 </span><span class="nb">readonly</span> <span class="nv">red_bg</span><span class="o">=</span><span class="s2">"\e[0;101m</span><span class="si">${</span><span class="nv">expand_bg</span><span class="si">}</span><span class="s2">"</span>
<span class="ln"> 8 </span><span class="nb">readonly</span> <span class="nv">green_bg</span><span class="o">=</span><span class="s2">"\e[0;102m</span><span class="si">${</span><span class="nv">expand_bg</span><span class="si">}</span><span class="s2">"</span>
<span class="ln"> 9 </span><span class="nb">readonly</span> <span class="nv">green</span><span class="o">=</span><span class="s2">"\e[0;92m"</span>
<span class="ln"> 10 </span><span class="nb">readonly</span> <span class="nv">white</span><span class="o">=</span><span class="s2">"\e[0;97m"</span>
<span class="ln"> 11 </span><span class="nb">readonly</span> <span class="nv">bold</span><span class="o">=</span><span class="s2">"\e[1m"</span>
<span class="ln"> 12 </span><span class="nb">readonly</span> <span class="nv">uline</span><span class="o">=</span><span class="s2">"\e[4m"</span>
<span class="ln"> 13 </span><span class="nb">readonly</span> <span class="nv">reset</span><span class="o">=</span><span class="s2">"\e[0m"</span>
<span class="ln"> 14 </span>
<span class="ln"> 15 </span>
<span class="ln"> 16 </span><span class="k">function</span> ok<span class="o">()</span> <span class="o">{</span>
<span class="ln"> 17 </span> <span class="nb">echo</span> -e <span class="s2">"</span><span class="si">${</span><span class="nv">green</span><span class="si">}</span><span class="s2">OK</span><span class="si">${</span><span class="nv">reset</span><span class="si">}</span><span class="s2">"</span>
<span class="ln"> 18 </span><span class="o">}</span>
<span class="ln"> 19 </span>
<span class="ln"> 20 </span><span class="k">function</span> nok<span class="o">()</span> <span class="o">{</span>
<span class="ln"> 21 </span> <span class="nb">echo</span> -e <span class="s2">"</span><span class="si">${</span><span class="nv">red</span><span class="si">}</span><span class="s2">NOK</span><span class="si">${</span><span class="nv">reset</span><span class="si">}</span><span class="s2">"</span>
<span class="ln"> 22 </span><span class="o">}</span>
<span class="ln"> 23 </span>
<span class="ln"> 24 </span><span class="k">function</span> main<span class="o">()</span> <span class="o">{</span>
<span class="ln"> 25 </span> <span class="k">for</span> domain in <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="p">;</span> <span class="k">do</span>
<span class="ln"> 26 </span> <span class="nb">echo</span> -e <span class="s2">"</span><span class="si">${</span><span class="nv">bold</span><span class="si">}</span><span class="s2">---- Checking domain </span><span class="si">${</span><span class="nv">domain</span><span class="si">}${</span><span class="nv">reset</span><span class="si">}</span><span class="s2">"</span>
<span class="ln"> 27 </span>
<span class="ln"> 28 </span> <span class="nb">echo</span> -n <span class="s2">"Redirect to HTTPS: "</span>
<span class="ln"> 29 </span> <span class="k">if</span> curl --head --silent <span class="s2">"http://</span><span class="si">${</span><span class="nv">domain</span><span class="si">}</span><span class="s2">"</span> <span class="p">|</span> grep --silent Location<span class="p">;</span> <span class="k">then</span>
<span class="ln"> 30 </span> ok
<span class="ln"> 31 </span> <span class="k">else</span>
<span class="ln"> 32 </span> nok
<span class="ln"> 33 </span> <span class="k">fi</span>
<span class="ln"> 34 </span>
<span class="ln"> 35 </span> <span class="nb">echo</span> -n <span class="s2">"Is reachable with IPv4: "</span>
<span class="ln"> 36 </span> <span class="k">if</span> curl -4 --head --silent <span class="s2">"https://</span><span class="si">${</span><span class="nv">domain</span><span class="si">}</span><span class="s2">"</span> > /dev/null<span class="p">;</span> <span class="k">then</span>
<span class="ln"> 37 </span> ok
<span class="ln"> 38 </span> <span class="k">else</span>
<span class="ln"> 39 </span> nok
<span class="ln"> 40 </span> <span class="k">fi</span>
<span class="ln"> 41 </span>
<span class="ln"> 42 </span> <span class="nb">echo</span> -n <span class="s2">"Is reachable with IPv6: "</span>
<span class="ln"> 43 </span> <span class="k">if</span> curl -6 --head --silent <span class="s2">"https://</span><span class="si">${</span><span class="nv">domain</span><span class="si">}</span><span class="s2">"</span> > /dev/null<span class="p">;</span> <span class="k">then</span>
<span class="ln"> 44 </span> ok
<span class="ln"> 45 </span> <span class="k">else</span>
<span class="ln"> 46 </span> nok
<span class="ln"> 47 </span> <span class="k">fi</span>
<span class="ln"> 48 </span>
<span class="ln"> 49 </span> <span class="nb">echo</span> -n <span class="s2">"Check HTTP 1.1: "</span>
<span class="ln"> 50 </span> <span class="k">if</span> curl --http1.1 --silent <span class="s2">"https://</span><span class="si">${</span><span class="nv">domain</span><span class="si">}</span><span class="s2">"</span> > /dev/null<span class="p">;</span> <span class="k">then</span>
<span class="ln"> 51 </span> ok
<span class="ln"> 52 </span> <span class="k">else</span>
<span class="ln"> 53 </span> nok
<span class="ln"> 54 </span> <span class="k">fi</span>
<span class="ln"> 55 </span>
<span class="ln"> 56 </span> <span class="nb">echo</span> -n <span class="s2">"Check HTTP2: "</span>
<span class="ln"> 57 </span> <span class="k">if</span> curl --http2 --silent <span class="s2">"https://</span><span class="si">${</span><span class="nv">domain</span><span class="si">}</span><span class="s2">"</span> > /dev/null<span class="p">;</span> <span class="k">then</span>
<span class="ln"> 58 </span> ok
<span class="ln"> 59 </span> <span class="k">else</span>
<span class="ln"> 60 </span> nok
<span class="ln"> 61 </span> <span class="k">fi</span>
<span class="ln"> 62 </span>
<span class="ln"> 63 </span> <span class="nb">echo</span> -n <span class="s2">"Check not reachable with TLS 1.0: "</span>
<span class="ln"> 64 </span> <span class="k">if</span> ! curl --tlsv1.0 --tls-max <span class="m">1</span>.0 --silent <span class="s2">"https://</span><span class="si">${</span><span class="nv">domain</span><span class="si">}</span><span class="s2">"</span> > /dev/null<span class="p">;</span> <span class="k">then</span>
<span class="ln"> 65 </span> ok
<span class="ln"> 66 </span> <span class="k">else</span>
<span class="ln"> 67 </span> nok
<span class="ln"> 68 </span> <span class="k">fi</span>
<span class="ln"> 69 </span>
<span class="ln"> 70 </span> <span class="nb">echo</span> -n <span class="s2">"Check not reachable with TLS 1.1: "</span>
<span class="ln"> 71 </span> <span class="k">if</span> ! curl --tlsv1.1 --tls-max <span class="m">1</span>.1 --silent <span class="s2">"https://</span><span class="si">${</span><span class="nv">domain</span><span class="si">}</span><span class="s2">"</span> > /dev/null<span class="p">;</span> <span class="k">then</span>
<span class="ln"> 72 </span> ok
<span class="ln"> 73 </span> <span class="k">else</span>
<span class="ln"> 74 </span> nok
<span class="ln"> 75 </span> <span class="k">fi</span>
<span class="ln"> 76 </span>
<span class="ln"> 77 </span> <span class="nb">echo</span> -n <span class="s2">"Check reachable with TLS 1.2: "</span>
<span class="ln"> 78 </span> <span class="k">if</span> curl --tlsv1.2 --tls-max <span class="m">1</span>.2 --silent <span class="s2">"https://</span><span class="si">${</span><span class="nv">domain</span><span class="si">}</span><span class="s2">"</span> > /dev/null<span class="p">;</span> <span class="k">then</span>
<span class="ln"> 79 </span> ok
<span class="ln"> 80 </span> <span class="k">else</span>
<span class="ln"> 81 </span> nok
<span class="ln"> 82 </span> <span class="k">fi</span>
<span class="ln"> 83 </span>
<span class="ln"> 84 </span> <span class="nb">echo</span> -n <span class="s2">"Check not reachable with TLS 1.3: "</span>
<span class="ln"> 85 </span> <span class="k">if</span> curl --tlsv1.3 --tls-max <span class="m">1</span>.3 --silent <span class="s2">"https://</span><span class="si">${</span><span class="nv">domain</span><span class="si">}</span><span class="s2">"</span> > /dev/null<span class="p">;</span> <span class="k">then</span>
<span class="ln"> 86 </span> ok
<span class="ln"> 87 </span> <span class="k">else</span>
<span class="ln"> 88 </span> nok
<span class="ln"> 89 </span> <span class="k">fi</span>
<span class="ln"> 90 </span>
<span class="ln"> 91 </span> <span class="nb">echo</span> -n <span class="s2">"Check that server responds with 200 or 404: "</span>
<span class="ln"> 92 </span> <span class="nv">status_code</span><span class="o">=</span><span class="k">$(</span>curl --write <span class="s2">"%{http_code}"</span> --silent --head --output /dev/null <span class="s2">"https://</span><span class="si">${</span><span class="nv">domain</span><span class="si">}</span><span class="s2">"</span><span class="k">)</span>
<span class="ln"> 93 </span> <span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="si">${</span><span class="nv">status_code</span><span class="si">}</span><span class="s2">"</span> <span class="o">==</span> <span class="s1">'200'</span> <span class="o">||</span> <span class="s2">"</span><span class="si">${</span><span class="nv">status_code</span><span class="si">}</span><span class="s2">"</span> <span class="o">==</span> <span class="s1">'404'</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="ln"> 94 </span> ok
<span class="ln"> 95 </span> <span class="k">else</span>
<span class="ln"> 96 </span> nok
<span class="ln"> 97 </span> <span class="k">fi</span>
<span class="ln"> 98 </span>
<span class="ln"> 99 </span> <span class="nb">echo</span> -e <span class="s2">""</span>
<span class="ln">100 </span> <span class="k">done</span>
<span class="ln">101 </span><span class="o">}</span>
<span class="ln">102 </span>
<span class="ln">103 </span>main <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</pre>
Small security checklist for public backend services2021-09-19T00:00:00+02:002021-09-19T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2021-09-19:/posts/en/2021/Sep/19/security-checklist/<p>Here are some security tips to check for backend services.
It's mostly meant so that I can have a check list.
So I don't develop them much but provide extra links where necessary.
I also probably expand this list as time goes one and I learn more about this subject …</p><p>Here are some security tips to check for backend services.
It's mostly meant so that I can have a check list.
So I don't develop them much but provide extra links where necessary.
I also probably expand this list as time goes one and I learn more about this subject.</p>
<div class="contents topic" id="contents">
<p class="topic-title">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#http-headers" id="toc-entry-1">HTTP headers</a></li>
<li><a class="reference internal" href="#cookies" id="toc-entry-2">Cookies</a></li>
<li><a class="reference internal" href="#misc" id="toc-entry-3">Misc</a><ul>
<li><a class="reference internal" href="#django" id="toc-entry-4">Django</a></li>
</ul>
</li>
<li><a class="reference internal" href="#examples" id="toc-entry-5">Examples</a><ul>
<li><a class="reference internal" href="#django-1" id="toc-entry-6">Django</a></li>
<li><a class="reference internal" href="#nextjs" id="toc-entry-7">NextJS</a></li>
<li><a class="reference internal" href="#nginx" id="toc-entry-8">nginx</a></li>
<li><a class="reference internal" href="#csp-html-webpack-plugin" id="toc-entry-9">csp-html-webpack-plugin</a></li>
</ul>
</li>
</ul>
</div>
<div class="section" id="http-headers">
<h2><a class="toc-backref" href="#toc-entry-1">HTTP headers</a></h2>
<ul class="simple">
<li>Make sure your server doesn't respond with its version (and perhaps even its name): with this information, an attacker knows which security holes to use.<ul>
<li>For example, with nginx it's done with <tt class="docutils literal">server_tokens off;</tt>.</li>
<li>Also make sure your framework don't leak this information in cookies.</li>
</ul>
</li>
<li>Enable <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP">content security policy</a> to mitigate XSS and data injection attacks.<ul>
<li>In SPA (with React for instance), you can inject this policy in the HTML instead of relying on a header. It's mostly useful if some CSS or JS must be inline in the HTML. <a class="reference external" href="https://www.npmjs.com/package/csp-html-webpack-plugin">This webpack plugin</a> will help you achieve that and limit inlined CSS and JS to code that matches a hash created at generation time so you are still protected against XSS.</li>
<li>If you use <tt class="docutils literal"><span class="pre">create-react-app</span></tt>, you can set <tt class="docutils literal">INLINE_RUNTIME_CHUNK</tt> to <tt class="docutils literal">false</tt> to disable inline script and use HTTP headers correctly. See <a class="reference external" href="https://create-react-app.dev/docs/advanced-configuration">the documentation</a>.</li>
<li>If you need to add inline scripts outside your build system, you can enable CSP and load the page of Chrome to get the hash of the script to use in your CSP policy.</li>
<li>You can use either <tt class="docutils literal"><span class="pre">frame-ancestors</span> 'none'</tt> with CSP or <tt class="docutils literal"><span class="pre">X-Frame-Options:</span> deny</tt>. See <a class="reference external" href="https://developer.mozilla.org/fr/docs/Web/HTTP/Headers/Content-Security-Policy/frame-ancestors">here</a>.</li>
</ul>
</li>
<li>Enable <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security">strict transport security</a> so you browser always connect to your site with HTTPS (after the first visit).<ul>
<li>With nginx it's done with something like <tt class="docutils literal">add_header <span class="pre">Strict-Transport-Security</span> <span class="pre">"max-age=31536000;</span> includeSubDomains" always;</tt>.</li>
</ul>
</li>
<li>Extra headers:<ul>
<li>Configure a <a class="reference external" href="https://scotthelme.co.uk/a-new-security-header-referrer-policy/">Referrer Policy</a>.</li>
<li>Configure a <a class="reference external" href="https://scotthelme.co.uk/goodbye-feature-policy-and-hello-permissions-policy/">Permissions Policy</a></li>
<li>Other useful headers, mostly to prevent user tracking: <tt class="docutils literal">add_header <span class="pre">Permissions-Policy</span> <span class="pre">"interest-cohort=()"</span> always;</tt>, <tt class="docutils literal"><span class="pre">Cross-Origin-Opener-Policy:</span> <span class="pre">same-origin</span></tt>, <tt class="docutils literal"><span class="pre">Cross-Origin-Resource-Policy:</span> <span class="pre">same-site</span></tt> and <tt class="docutils literal"><span class="pre">Cross-Origin-Embedder-Policy:</span> <span class="pre">unsafe-none</span></tt>. See <a class="reference external" href="https://adamj.eu/tech/2021/05/01/how-to-set-coep-coop-corp-security-headers-in-django/">this article</a> for the rational being this.</li>
</ul>
</li>
</ul>
<div class="admonition tip">
<p class="first admonition-title">Astuce</p>
<p class="last">You can use <a class="reference external" href="https://securityheaders.com/">this tool</a> to check your HTTP headers.</p>
</div>
</div>
<div class="section" id="cookies">
<h2><a class="toc-backref" href="#toc-entry-2">Cookies</a></h2>
<ul class="simple">
<li>Authentication cookies must be created with <tt class="docutils literal">httpOnly=true</tt> and <tt class="docutils literal">secure=true</tt> so they cannot be accessed by javascript (in case of an XSS attack) nor intercepted. They should also probably be restricted to a specific domain and expire in a reasonable amount of time.
- You must enable CSRF protection on all views that performs actions for authenticated user (see <a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/csrf/">here</a> for Django). If you rely on a cookie you can set <tt class="docutils literal">httpOnly=true</tt> if you need to use it with AJAX request (Django's documentation explains this well).
- You must also configure <tt class="docutils literal">SameSite</tt> for your cookies, <tt class="docutils literal">Lax</tt> being a good default in most cases. See <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite">here</a> for more details.</li>
<li>Beware of cookie and <a class="reference external" href="https://medium.com/nextdns/cname-cloaking-the-dangerous-disguise-of-third-party-trackers-195205dc522a">CNAME cloaking</a> (even CNIL, the French data protection authority <a class="reference external" href="https://www.cnil.fr/fr/definition/cname-cloaking-ou-delegation-de-sous-domaine">talks about it</a>): the idea is that some advertising company will ask you to add a CNAME entry to your DNS (eg <tt class="docutils literal">tracking.jujens.eu</tt>) so they can track your user (this technique is hard to detect and block by tools since it can be a legitimate domain). Here is the security risk: if your authentication cookies are sent to all you subdomains, you will send them to the advertising server, allowing them to connect as your users. You can disallow this technique (not always possible) or mitigate this by restricting to which domains your cookies are sent or switching to another authentication method (like tokens <a class="footnote-reference" href="#tokens" id="footnote-reference-1">[1]</a>).</li>
</ul>
</div>
<div class="section" id="misc">
<h2><a class="toc-backref" href="#toc-entry-3">Misc</a></h2>
<ul class="simple">
<li>Add audit for user actions (eg when user connects, changes password) if required.</li>
<li>Enable <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity">Subresource integrity</a> so your browser can verify the resource it fetches are fetched without manipulations.<ul>
<li>If you use webpack, <a class="reference external" href="https://www.npmjs.com/package/webpack-subresource-integrity">this plugin</a> can help.</li>
</ul>
</li>
<li>Never enable your frameworks debug mode in production: this will make you leak sensitive information like configuration.</li>
<li>Make your secure your connection with the database. For Django, you should at least set <tt class="docutils literal">sslmode</tt> to <tt class="docutils literal">prefer</tt> and ideally configure full certificate validation. See <a class="reference external" href="https://www.postgresql.org/docs/9.1/libpq-ssl.html">here</a>.</li>
<li>Don't store credit card data (these data shouldn't even be seen by your system).</li>
<li>Validate and sanitize all incoming data with forms. Escape it if necessary.<ul>
<li>In Python, <a class="reference external" href="https://github.com/mozilla/bleach/">bleach</a> is great for that.</li>
</ul>
</li>
<li>Beware of XML: by default it contains feature that presents security risks.<ul>
<li>In Python, <a class="reference external" href="https://github.com/tiran/defusedxml">defusedxml</a> is a great parser to avoid that.</li>
</ul>
</li>
<li>Allow users to enable 2FA and force admin to use it.</li>
<li>Obfuscate primary keys with UUIDs to resist enumeration attacks.</li>
<li>Don't store passwords in plain text: store your application password hashes instead. Add a random salt as well.</li>
<li>Don't log any sensitive data: filter out the confidential data, such as API keys, before recording them in your log files.</li>
<li>Any secure transaction or login should use SSL: be aware that eavesdroppers in the same network as you could listen to your web traffic if it is not in HTTPS. Ideally, you ought to use HTTPS for the entire site.</li>
<li>Avoid using redirects to user-supplied URLs: If you have redirects such as <a class="reference external" href="http://example.com/r?url=http://evil.com">http://example.com/r?url=http://evil.com</a>, then always check against whitelisted domains.</li>
<li>Check authorization even for authenticated users: Before performing any change with side effects, check whether the logged-in user is allowed to perform it.</li>
<li>Don't keep your backend code in web root: This can lead to an accidental leak of source code if it gets served as plain text.</li>
<li>Use templating libraries with XSS protection built in.</li>
<li>Use an ORM rather than SQL commands: good ORMs offers protection against SQL injection.</li>
<li>Use Django forms with POST input for any action with side effects: It might seem like overkill to use forms for a simple vote button, but do it.</li>
<li>CSRF should be enabled and used.<ul>
<li>In Django, be very careful if you are exempting certain views using the <cite>@csrf_exempt decorator</cite>.</li>
</ul>
</li>
<li>Ensure that Django and all packages are the latest versions. Plan for updates (tools like <a class="reference external" href="https://dependabot.com/">dependabot</a> can help).</li>
<li>Limit the size and type of user-uploaded files.</li>
<li>Run <a class="reference external" href="https://www.qualys.com/forms/freescan/">https://www.qualys.com/forms/freescan/</a>, <a class="reference external" href="https://www.owasp.org/index.php/OWASP_Dependency_Check">https://www.owasp.org/index.php/OWASP_Dependency_Check</a> and <a class="reference external" href="https://observatory.mozilla.org/">https://observatory.mozilla.org/</a> to detect potential issues.</li>
</ul>
<div class="section" id="django">
<h3><a class="toc-backref" href="#toc-entry-4">Django</a></h3>
<ul class="simple">
<li>Never use <tt class="docutils literal">Meta.exclude</tt> because you may include fields in a form by accident. For the same reason, don't use the <tt class="docutils literal">__all__</tt> shortcut in <tt class="docutils literal">fields</tt>.</li>
<li>Django automatically escapes data dynamically inserted in templates. But you still need to quote all HTML attribute. For example, replace <tt class="docutils literal"><a <span class="pre">href={{link}}></span></tt> with <tt class="docutils literal"><a <span class="pre">href="{{link}}"></span></tt>.</li>
<li>Avoid the <tt class="docutils literal">extra</tt> and <tt class="docutils literal">execute</tt> functions of the ORM. The ORM will prevent SQL injection by default but with these you need to do the work manually.</li>
<li>You can use <a class="reference external" href="https://github.com/cakinney/secure.py">secure.py</a> to use standard security headers.</li>
<li>Change the default admin URL to something else so it is harder to find for an attacker.</li>
<li>Don't keep <tt class="docutils literal">SECRET_KEY</tt> in version control. As a best practice, pick <tt class="docutils literal">SECRET_KEY</tt> from the environment. Check out the <a class="reference external" href="https://django-environ.readthedocs.io/en/latest/">django-environ</a> package.</li>
</ul>
</div>
</div>
<div class="section" id="examples">
<h2><a class="toc-backref" href="#toc-entry-5">Examples</a></h2>
<div class="section" id="django-1">
<h3><a class="toc-backref" href="#toc-entry-6">Django</a></h3>
<p>This relies on <a class="reference external" href="https://github.com/adamchainz/django-cors-headers">django-cors-headers</a></p>
<pre class="code python literal-block">
<span class="kn">from</span> <span class="nn">corsheaders.defaults</span> <span class="kn">import</span> <span class="n">default_headers</span>
<span class="c1"># Security</span>
<span class="c1"># See https://github.com/adamchainz/django-cors-headers</span>
<span class="n">CORS_ALLOW_HEADERS</span> <span class="o">=</span> <span class="n">default_headers</span>
<span class="n">CORS_ORIGIN_WHITELIST</span> <span class="o">=</span> <span class="p">[</span><span class="n">FRONTEND_BASE_URL</span><span class="p">]</span>
<span class="n">CORS_ALLOW_CREDENTIALS</span> <span class="o">=</span> <span class="kc">True</span>
<span class="c1"># Both of these must be False so we can correctly use the CSRF token in</span>
<span class="c1"># our AJAX request.</span>
<span class="c1"># See: https://docs.djangoproject.com/en/3.1/ref/csrf/#ajax</span>
<span class="c1"># and https://www.django-rest-framework.org/topics/ajax-csrf-cors/</span>
<span class="n">CSRF_USE_SESSIONS</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">CSRF_COOKIE_HTTPONLY</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">CSRF_TRUSTED_ORIGINS</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">tuple</span><span class="p">(</span><span class="s2">"CSRF_TRUSTED_ORIGINS"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="p">(</span><span class="s2">"localhost"</span><span class="p">,))</span>
<span class="n">CSRF_COOKIE_DOMAIN</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">str</span><span class="p">(</span><span class="s2">"CSRF_COOKIE_DOMAIN"</span><span class="p">,</span> <span class="s2">"localhost"</span><span class="p">)</span>
<span class="c1"># https://docs.djangoproject.com/en/dev/ref/settings/#x-frame-options</span>
<span class="n">X_FRAME_OPTIONS</span> <span class="o">=</span> <span class="s2">"DENY"</span>
<span class="c1"># https://docs.djangoproject.com/en/dev/ref/middleware/#x-content-type-options-nosniff</span>
<span class="n">SECURE_CONTENT_TYPE_NOSNIFF</span> <span class="o">=</span> <span class="kc">True</span>
<span class="c1"># https://docs.djangoproject.com/fr/3.1/ref/settings/#password-reset-timeout</span>
<span class="n">PASSWORD_RESET_TIMEOUT</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span>
<span class="c1"># https://docs.djangoproject.com/en/3.2/ref/middleware/#module-django.middleware.security</span>
<span class="n">SECURE_REFERRER_POLICY</span> <span class="o">=</span> <span class="s2">"strict-origin-when-cross-origin"</span>
<span class="c1"># CSP settings</span>
<span class="c1"># https://django-csp.readthedocs.io/en/latest/configuration.html</span>
<span class="n">CSP_DEFAULT_SRC</span> <span class="o">=</span> <span class="p">(</span><span class="s2">"'self'"</span><span class="p">,)</span>
<span class="n">CSP_CONNECT_SRC</span> <span class="o">=</span> <span class="p">(</span><span class="s2">"'self'"</span><span class="p">,)</span>
<span class="c1"># To use for JS, CSS, images and fonts.</span>
<span class="n">CUSTOM_CSP_STATIC_SRC</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">tuple</span><span class="p">(</span><span class="s2">"CUSTOM_CSP_STATIC_SRC"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="p">(</span><span class="s2">"'self'"</span><span class="p">,))</span>
<span class="n">CSP_SCRIPT_SRC</span> <span class="o">=</span> <span class="n">CUSTOM_CSP_STATIC_SRC</span>
<span class="n">CSP_IMG_SRC</span> <span class="o">=</span> <span class="n">CUSTOM_CSP_STATIC_SRC</span>
<span class="n">CSP_MEDIA_SRC</span> <span class="o">=</span> <span class="n">CUSTOM_CSP_STATIC_SRC</span>
<span class="n">CSP_FONT_SRC</span> <span class="o">=</span> <span class="n">CUSTOM_CSP_STATIC_SRC</span>
<span class="n">CSP_STYLE_SRC</span> <span class="o">=</span> <span class="n">CUSTOM_CSP_STATIC_SRC</span>
<span class="n">CSP_BLOCK_ALL_MIXED_CONTENT</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">CSP_UPGRADE_INSECURE_REQUESTS</span> <span class="o">=</span> <span class="kc">True</span>
<span class="c1"># https://docs.djangoproject.com/en/3.2/ref/settings/#secure-browser-xss-filter</span>
<span class="n">SECURE_BROWSER_XSS_FILTER</span> <span class="o">=</span> <span class="kc">True</span>
<span class="c1"># Secure connection</span>
<span class="n">SECURE_REDIRECT_EXEMPT</span> <span class="o">=</span> <span class="p">[</span><span class="sa">r</span><span class="s2">"/?health/?"</span><span class="p">]</span>
<span class="n">SECURE_SSL_REDIRECT</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">SECURE_PROXY_SSL_HEADER</span> <span class="o">=</span> <span class="p">(</span><span class="s2">"HTTP_X_FORWARDED_PROTO"</span><span class="p">,</span> <span class="s2">"https"</span><span class="p">)</span>
<span class="n">SESSION_COOKIE_SECURE</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">CSRF_COOKIE_SECURE</span> <span class="o">=</span> <span class="kc">True</span>
</pre>
</div>
<div class="section" id="nextjs">
<h3><a class="toc-backref" href="#toc-entry-7">NextJS</a></h3>
<p>You can have the code below in your <tt class="docutils literal">next.config.js</tt>:</p>
<pre class="code javascript literal-block">
<span class="kd">const</span> <span class="nx">production</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">NODE_ENV</span> <span class="o">===</span> <span class="s2">"production"</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">getCsp</span> <span class="o">=</span> <span class="p">()</span> <span class="p">=></span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">csp</span> <span class="o">=</span> <span class="sb">``</span><span class="p">;</span>
<span class="nx">csp</span> <span class="o">+=</span> <span class="sb">`base-uri 'self';`</span><span class="p">;</span>
<span class="nx">csp</span> <span class="o">+=</span> <span class="sb">`form-action 'self';`</span><span class="p">;</span>
<span class="nx">csp</span> <span class="o">+=</span> <span class="sb">`default-src 'self';`</span><span class="p">;</span>
<span class="nx">csp</span> <span class="o">+=</span> <span class="sb">`frame-src DOMAIN;`</span><span class="p">;</span>
<span class="c1">// NextJS requires 'unsafe-eval' in dev (faster source maps)
</span> <span class="c1">// sha256-XXXX is for XX service.
</span> <span class="nx">csp</span> <span class="o">+=</span> <span class="sb">`script-src 'self' </span><span class="si">${</span>
<span class="nx">production</span> <span class="o">?</span> <span class="s2">""</span> <span class="o">:</span> <span class="s2">"'unsafe-eval'"</span>
<span class="si">}</span><span class="sb"> https://maps.googleapis.com 'sha256-XXX'`</span><span class="p">;</span>
<span class="c1">// NextJS requires 'unsafe-inline'. Hash are not supported. Neither are nonce (the style tags are
</span> <span class="c1">// not updated correctly. This can also be limited to our usage of MaterialUI.
</span> <span class="c1">// Furthermore, since nonce must be generated at each request, we could get into issues with
</span> <span class="c1">// caching for these public pages.
</span> <span class="nx">csp</span> <span class="o">+=</span> <span class="sb">`style-src 'self' 'unsafe-inline' data: https://fonts.google.com https://fonts.googleapis.com https://client.crisp.chat;`</span><span class="p">;</span>
<span class="nx">csp</span> <span class="o">+=</span> <span class="sb">`img-src 'self' data: blob: https://maps.googleapis.com https://maps.gstatic.com https://maps.googleapis.com https://storage.googleapis.com </span><span class="si">${</span><span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">NEXT_PUBLIC_BACKEND_API_DOMAIN_URL</span><span class="si">}</span><span class="sb">;`</span><span class="p">;</span>
<span class="c1">// noembed.com is required for ReactPlayer to work correctly.
</span> <span class="nx">csp</span> <span class="o">+=</span> <span class="sb">`connect-src 'self' https://noembed.com wss://client.relay.crisp.chat;`</span><span class="p">;</span>
<span class="nx">csp</span> <span class="o">+=</span> <span class="sb">`font-src 'self' https://fonts.gstatic.com https://client.crisp.chat;`</span><span class="p">;</span>
<span class="nx">csp</span> <span class="o">+=</span> <span class="sb">`media-src 'self' https://storage.googleapis.com`</span><span class="p">;</span>
<span class="k">return</span> <span class="nx">csp</span><span class="p">;</span>
<span class="p">};</span>
<span class="kd">const</span> <span class="nx">headers</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nx">key</span><span class="o">:</span> <span class="s2">"X-Content-Type-Options"</span><span class="p">,</span>
<span class="nx">value</span><span class="o">:</span> <span class="s2">"nosniff"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nx">key</span><span class="o">:</span> <span class="s2">"X-Frame-Options"</span><span class="p">,</span>
<span class="nx">value</span><span class="o">:</span> <span class="s2">"DENY"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nx">key</span><span class="o">:</span> <span class="s2">"X-XSS-Protection"</span><span class="p">,</span>
<span class="nx">value</span><span class="o">:</span> <span class="s2">"1; mode=block"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nx">key</span><span class="o">:</span> <span class="s2">"Content-Security-Policy"</span><span class="p">,</span>
<span class="nx">value</span><span class="o">:</span> <span class="nx">getCsp</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">];</span>
<span class="kd">let</span> <span class="nx">moduleExports</span> <span class="o">=</span> <span class="p">{</span>
<span class="k">async</span> <span class="nx">headers</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nx">source</span><span class="o">:</span> <span class="s2">"/(.*)"</span><span class="p">,</span>
<span class="nx">headers</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nx">source</span><span class="o">:</span> <span class="s2">"/:path*"</span><span class="p">,</span>
<span class="nx">headers</span>
<span class="p">}</span>
<span class="p">];</span>
<span class="p">},</span>
<span class="p">};</span>
</pre>
</div>
<div class="section" id="nginx">
<h3><a class="toc-backref" href="#toc-entry-8">nginx</a></h3>
<p>I suppose that CSP in handled by the app itself (eg with Django's configuration).</p>
<pre class="code nginx literal-block">
<span class="k">add_header</span><span class="w"> </span><span class="s">Strict-Transport-Security</span><span class="w"> </span><span class="s">"max-age=31536000</span><span class="p">;</span><span class="w"> </span><span class="k">includeSubDomains"</span><span class="w"> </span><span class="s">always</span><span class="p">;</span><span class="w">
</span><span class="k">add_header</span><span class="w"> </span><span class="s">Permissions-Policy</span><span class="w"> </span><span class="s">"interest-cohort=()"</span><span class="w"> </span><span class="s">always</span><span class="p">;</span><span class="w">
</span><span class="k">add_header</span><span class="w"> </span><span class="s">Cross-Origin-Opener-Policy</span><span class="w"> </span><span class="s">same-origin</span><span class="w"> </span><span class="s">always</span><span class="p">;</span><span class="w">
</span><span class="k">add_header</span><span class="w"> </span><span class="s">Cross-Origin-Resource-Policy</span><span class="w"> </span><span class="s">same-site</span><span class="w"> </span><span class="s">always</span><span class="p">;</span><span class="w">
</span><span class="k">add_header</span><span class="w"> </span><span class="s">Cross-Origin-Embedder-Policy</span><span class="w"> </span><span class="s">unsafe-none</span><span class="w"> </span><span class="s">always</span><span class="p">;</span>
</pre>
</div>
<div class="section" id="csp-html-webpack-plugin">
<h3><a class="toc-backref" href="#toc-entry-9">csp-html-webpack-plugin</a></h3>
<pre class="code javascript literal-block">
<span class="ow">new</span> <span class="nx">cspHtmlWebpackPlugin</span><span class="p">(</span>
<span class="c1">// We still need to have unsafe-inline to support old browser. Modern browser will just
</span> <span class="c1">// ignore it if nonce or hash is set.
</span> <span class="p">{</span>
<span class="s1">'default-src'</span><span class="o">:</span> <span class="p">[</span><span class="s2">"'self'"</span><span class="p">],</span>
<span class="s1">'connect-src'</span><span class="o">:</span> <span class="p">[</span>
<span class="sb">`</span><span class="si">${</span><span class="nx">apiUrl</span><span class="p">.</span><span class="nx">host</span><span class="si">}</span><span class="sb">`</span><span class="p">,</span>
<span class="s1">'*.sentry.io'</span><span class="p">,</span>
<span class="s1">'o552216.ingest.sentry.io'</span><span class="p">,</span>
<span class="s1">'sentry.io'</span><span class="p">,</span>
<span class="p">],</span>
<span class="s1">'style-src'</span><span class="o">:</span> <span class="p">[</span><span class="s2">"'self'"</span><span class="p">,</span> <span class="s2">"'unsafe-inline'"</span><span class="p">,</span> <span class="s1">'fonts.googleapis.com'</span><span class="p">],</span>
<span class="s1">'font-src'</span><span class="o">:</span> <span class="p">[</span><span class="s2">"'self'"</span><span class="p">,</span> <span class="s1">'fonts.gstatic.com'</span><span class="p">],</span>
<span class="s1">'img-src'</span><span class="o">:</span> <span class="p">[</span>
<span class="s2">"'self'"</span><span class="p">,</span>
<span class="s1">'data:'</span><span class="p">,</span>
<span class="s1">'maps.gstatic.com'</span><span class="p">,</span>
<span class="s1">'storage.googleapis.com'</span><span class="p">,</span>
<span class="sb">`</span><span class="si">${</span><span class="nx">apiUrl</span><span class="p">.</span><span class="nx">host</span><span class="si">}</span><span class="sb">`</span><span class="p">,</span>
<span class="p">],</span>
<span class="s1">'script-src'</span><span class="o">:</span> <span class="p">[</span>
<span class="s2">"'unsafe-inline'"</span><span class="p">,</span>
<span class="s2">"'self'"</span><span class="p">,</span>
<span class="s2">"'unsafe-eval'"</span><span class="p">,</span>
<span class="s1">'o552216.ingest.sentry.io'</span><span class="p">,</span>
<span class="s1">'sentry.io'</span><span class="p">,</span>
<span class="s1">'maps.googleapis.com'</span><span class="p">,</span>
<span class="p">],</span>
<span class="c1">// report-uri is not supported in the meta tag.
</span> <span class="p">},</span>
<span class="c1">// We must disable nonce and hash for styles. It's supported by styled-components
</span> <span class="c1">// (see https://github.com/styled-components/styled-components/issues/887) but not easily
</span> <span class="c1">// by material-ui (see https://material-ui.com/styles/advanced/#how-does-one-implement-csp)
</span> <span class="c1">// Since all this is statically generated anyway, it defeats the purpose on nonce anyway
</span> <span class="c1">// (see https://stackoverflow.com/questions/42922784/what-s-the-purpose-of-the-html-nonce-attribute-for-script-and-style-elements).
</span> <span class="c1">// We leave hash for script since they won't budge after generation.
</span> <span class="p">{</span>
<span class="nx">enabled</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="nx">hashEnabled</span><span class="o">:</span> <span class="p">{</span>
<span class="s1">'script-src'</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="s1">'style-src'</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="p">},</span>
<span class="nx">nonceEnabled</span><span class="o">:</span> <span class="p">{</span>
<span class="s1">'script-src'</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="s1">'style-src'</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">)</span>
</pre>
<table class="docutils footnote" frame="void" id="tokens" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>They can have their own security risk. I'm not export and I don't have a good link to provide right now. It can change in the future. From what I know properly securing cookies is most likely the best thing to do security wise if you can do it.</td></tr>
</tbody>
</table>
</div>
</div>
Manage deployment transitions for static application2021-04-25T00:00:00+02:002021-04-25T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2021-04-25:/posts/en/2021/Apr/25/deployment-transition-static-apps-docker/<p>When you deploy a frontend app, most of the time the name of your assets contains their hash so you can easily cache the files.
So instead of just having <tt class="docutils literal">main.js</tt> you will have something like <tt class="docutils literal">main.1234.js</tt>.
The problem is that your HTML will reference <tt class="docutils literal">main.1234 …</tt></p><p>When you deploy a frontend app, most of the time the name of your assets contains their hash so you can easily cache the files.
So instead of just having <tt class="docutils literal">main.js</tt> you will have something like <tt class="docutils literal">main.1234.js</tt>.
The problem is that your HTML will reference <tt class="docutils literal">main.1234.js</tt>, so on the next deploy, once this file is rebuilt if it was changed, it will be named something like <tt class="docutils literal">main.5678.js</tt>.
If you were to load all you files immediately when the browser opens your <tt class="docutils literal">index.html</tt> this wouldn't be a problem: your user either already has the old file or will load the new one.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Depending on your build process, this can concern JS, CSS, images or video files.</p>
</div>
<p>However, on big JS apps, your application is decomposed in many small chunks that are loaded on demand.
So you could get in a situation where the user starts using the app in the old version that references <tt class="docutils literal">main.1234.js</tt>.
You deploy your application the old <tt class="docutils literal">main.1234.js</tt> file is removed.
Now your user needs to download <tt class="docutils literal">main.1234.js</tt> but it doesn't exist any more.
Ouch.</p>
<p>To avoid that, the idea is to keep serving previous files of your applications.
So instead of removing them right away, we keep them around for a given amount of time.
I think a week is a good compromise between assets to keep (and that takes space) and user session.
Of course, user sessions can be way longer that that.
For these fringe cases, we accept the break.</p>
<p>Depending on how you deploy your application, I think there are two main strategies for that:</p>
<ol class="arabic simple">
<li>You copy build files on a server: in this case, copy the new files and then run <tt class="docutils literal">find . <span class="pre">-mtime</span> +7 <span class="pre">-delete</span></tt> in the folder in which you deployed the files. This will delete all files that haven't been modified since more than one week. Since each time you build a file that hasn't changed it will get the same hash, it will get modified on the copy if it still exists and thus won't be deleted.</li>
<li>You build the files into some archive (a docker image for instance) and you deploy this: you must restore old files from a cache to have them in your next deployment. This is this more complex technique (when coupled with Docker) that I will discuss in more detail further.</li>
</ol>
<p>Given my constraints, I decided to store the previous files into a Docker image and use multi-stage build to copy them after building the new files.
My Dockerfile looks like this:</p>
<pre class="code Dockerfile literal-block">
<span class="ln"> 1 </span><span class="k">FROM</span> <span class="s">mydocker-registry.io/my-image:latest</span> <span class="k">AS</span> <span class="s">previous-assets</span>
<span class="ln"> 2 </span><span class="c"># Create directory if it doesn't exist.</span>
<span class="ln"> 3 </span><span class="c"># This should only happen during the first build when the cache is empty.</span>
<span class="ln"> 4 </span><span class="k">RUN</span> mkdir -p /var/www/frontend-app/
<span class="ln"> 5 </span>
<span class="ln"> 6 </span>
<span class="ln"> 7 </span><span class="c"># Build the application.</span>
<span class="ln"> 8 </span><span class="k">FROM</span> <span class="s">node:14.16.0-slim</span> <span class="k">AS</span> <span class="s">builder</span>
<span class="ln"> 9 </span><span class="k">WORKDIR</span><span class="s"> /app</span>
<span class="ln">10 </span>
<span class="ln">11 </span><span class="k">COPY</span> . ./
<span class="ln">12 </span><span class="c"># Copy the previous assets into a dedicated folder so our new build won't delete</span>
<span class="ln">13 </span><span class="c"># them by accident.</span>
<span class="ln">14 </span><span class="k">COPY</span> --from<span class="o">=</span>previous-assets /var/www/frontend-app/ ./previous-builds/
<span class="ln">15 </span>
<span class="ln">16 </span><span class="k">RUN</span> yarn install --frozen-lockfile
<span class="ln">17 </span><span class="k">RUN</span> yarn build <span class="o">&&</span> <span class="se">\
</span><span class="ln">18 </span><span class="se"></span> # Restore static files from previous builds. Use --no-clobber to avoid overriding files from
<span class="ln">19 </span> # current build <span class="o">(</span>this option is not available from the cp <span class="nb">command</span> in alpine<span class="o">)</span>. This option is
<span class="ln">20 </span> # not available to the COPY command. This way, existing files will have an updated modification
<span class="ln">21 </span> # time.
<span class="ln">22 </span> # Use --archive to be sure to preserve their modification <span class="nb">time</span> so we can correctly delete
<span class="ln">23 </span> # them later.
<span class="ln">24 </span> cp -R --no-clobber --archive ./previous-builds/* ./build/ <span class="o">&&</span> <span class="se">\
</span><span class="ln">25 </span><span class="se"></span> # Delete <span class="nb">source</span> maps.
<span class="ln">26 </span> find build -name <span class="se">\*</span>.js<span class="se">\*</span>.map -type f -delete <span class="o">&&</span> <span class="se">\
</span><span class="ln">27 </span><span class="se"></span> # Delete files older than <span class="m">7</span> days
<span class="ln">28 </span> find build -mtime +7 -type f -delete
<span class="ln">29 </span>
<span class="ln">30 </span>
<span class="ln">31 </span><span class="c"># Run the app in nginx.</span>
<span class="ln">32 </span><span class="k">FROM</span> <span class="s">nginx:latest</span> <span class="k">AS</span> <span class="s">runner</span>
<span class="ln">33 </span><span class="k">RUN</span> mkdir -p /var/www/frontend-app
<span class="ln">34 </span><span class="k">WORKDIR</span><span class="s"> /var/www/frontend-app</span>
<span class="ln">35 </span>
<span class="ln">36 </span><span class="k">COPY</span> --from<span class="o">=</span>builder /app/build /var/www/frontend-app/
</pre>
<p>The build steps are then like this:</p>
<ol class="arabic">
<li><p class="first">Pull or create the asset image with a custom scripts: <tt class="docutils literal"><span class="pre">create-assets-image.sh</span> <span class="pre">mydocker-registry.io/my-image</span></tt>. The script is:</p>
<pre class="code bash literal-block">
<span class="ln"> 1 </span><span class="ch">#!/usr/bin/env bash
</span><span class="ln"> 2 </span><span class="ch"></span>
<span class="ln"> 3 </span><span class="nb">set</span> -e
<span class="ln"> 4 </span><span class="nb">set</span> -u
<span class="ln"> 5 </span><span class="nb">set</span> -o pipefail
<span class="ln"> 6 </span>
<span class="ln"> 7 </span><span class="nv">ASSETS_IMAGE_NAME</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">1</span><span class="k">:-</span><span class="nv">previous</span><span class="p">-assets</span><span class="si">}</span><span class="s2">"</span>
<span class="ln"> 8 </span><span class="nv">ASSETS_IMAGE_TAG</span><span class="o">=</span>latest
<span class="ln"> 9 </span><span class="nb">readonly</span> ASSETS_IMAGE_NAME
<span class="ln">10 </span><span class="nb">readonly</span> ASSETS_IMAGE_TAG
<span class="ln">11 </span>
<span class="ln">12 </span><span class="c1"># If we don't already have the image for previous assets, we create it to be sure to
</span><span class="ln">13 </span><span class="c1"># have this base image.
</span><span class="ln">14 </span><span class="c1"></span><span class="k">if</span> ! docker pull <span class="s2">"</span><span class="si">${</span><span class="nv">ASSETS_IMAGE_NAME</span><span class="si">}</span><span class="s2">:</span><span class="si">${</span><span class="nv">ASSETS_IMAGE_TAG</span><span class="si">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">then</span>
<span class="ln">15 </span> docker pull nginx:latest
<span class="ln">16 </span> docker tag nginx:latest <span class="s2">"</span><span class="si">${</span><span class="nv">ASSETS_IMAGE_NAME</span><span class="si">}</span><span class="s2">:</span><span class="si">${</span><span class="nv">ASSETS_IMAGE_TAG</span><span class="si">}</span><span class="s2">"</span>
<span class="ln">17 </span><span class="k">fi</span>
</pre>
</li>
<li><p class="first">Build the images and tag them: <tt class="docutils literal">docker build <span class="pre">--tag</span> <span class="pre">mydocker-registry.io/my-image:$COMMIT_SHA</span> <span class="pre">--tag</span> <span class="pre">mydocker-registry.io/my-image:latest</span> .</tt>. I need to tag two images with the same content. One will be the production image I deploy and contains the commit hash in its version. The other one will always have the same name so I know which image to use to fetch the previous assets.</p>
</li>
<li><p class="first">Push the image: <tt class="docutils literal">docker push <span class="pre">mydocker-registry.io/my-image:$COMMIT_SHA</span> <span class="pre">mydocker-registry.io/my-image:latest</span></tt></p>
</li>
</ol>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">To avoid issue with the <tt class="docutils literal">latest</tt> tag and always have file from the previous builds, you must wait for a build to end before starting a new one.</p>
</div>
Using DateTimeRangeField in Django2021-04-06T00:00:00+02:002021-04-06T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2021-04-06:/posts/2021/Apr/06/datetime-ranges-django/<p>:tags:Django, Python, PostgreSQL
:lang: en</p>
<div class="section" id="the-basics">
<h2>The basics</h2>
<p>If you need to store a datetime range in Django (for instance from which day to which day a listing is valid), instead of relying on two fields like <tt class="docutils literal">valid_from</tt> and <tt class="docutils literal">valid_to</tt>, you can use a single field <tt class="docutils literal">validity_range</tt> of type <tt class="docutils literal">DateRangeField …</tt></p></div><p>:tags:Django, Python, PostgreSQL
:lang: en</p>
<div class="section" id="the-basics">
<h2>The basics</h2>
<p>If you need to store a datetime range in Django (for instance from which day to which day a listing is valid), instead of relying on two fields like <tt class="docutils literal">valid_from</tt> and <tt class="docutils literal">valid_to</tt>, you can use a single field <tt class="docutils literal">validity_range</tt> of type <tt class="docutils literal">DateRangeField</tt>.
This way, both values are stored together in the same field.
Since <tt class="docutils literal">DateTimeRangeField</tt> is specific to PostgreSQL, you must import it like this <tt class="docutils literal">from django.contrib.postgres.fields import DateRangeField</tt>.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">PostgreSQL provides other range field for integers, decimal and datetimes. See <a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/contrib/postgres/fields/#range-fields">this page</a> in the Django documentation for a full list.</p>
</div>
<p>You can assign a value to this field directly with a tuple:</p>
<pre class="code python literal-block">
<span class="c1"># Create a range from 2021-01-01 to 2021-04-01</span>
<span class="n">instance</span><span class="o">.</span><span class="n">validity_range</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'2021-01-01'</span><span class="p">,</span> <span class="s1">'2021-04-01'</span><span class="p">)</span>
</pre>
<p>You can also leave either bound unbound by setting it to <tt class="docutils literal">None</tt>:</p>
<pre class="code python literal-block">
<span class="c1"># Create a range from 2021-01-01 to the end of times.</span>
<span class="n">instance</span><span class="o">.</span><span class="n">validity_range</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'2021-01-01'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
</pre>
<p>You can also create the range explicitly with <tt class="docutils literal">DateTimeTZRange</tt> (that's what you will get from the database anyway):</p>
<pre class="code python literal-block">
<span class="kn">from</span> <span class="nn">psycopg2.extras</span> <span class="kn">import</span> <span class="n">DateTimeTZRange</span>
<span class="n">instance</span><span class="o">.</span><span class="n">validity_range</span> <span class="o">=</span> <span class="n">DateTimeTZRange</span><span class="p">(</span><span class="n">lower</span><span class="o">=</span><span class="s1">'2021-01-01'</span><span class="p">,</span> <span class="n">upper</span><span class="o">=</span><span class="s1">'2021-03-31'</span><span class="p">)</span>
</pre>
<p>This will also allow you to select the behavior of the boundaries: should they be included or excluded.
By default, the lower boundary is included while the upper one is excluded.
Please refer <a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/contrib/postgres/fields/#range-fields">to the documentation</a> for more details.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">A opened boundary (that is set at <tt class="docutils literal">None</tt>) is always excluded.</p>
</div>
<p>You can them query your models like this:</p>
<pre class="code python literal-block">
<span class="n">MyModel</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">validity_range__contains</span><span class="o">=</span><span class="n">date</span><span class="o">.</span><span class="n">today</span><span class="p">())</span>
<span class="n">MyModel</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">validity_range__endswith__lte</span><span class="o">=</span><span class="n">date</span><span class="o">.</span><span class="n">today</span><span class="p">())</span>
<span class="n">MyModel</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">validity_range__startswith__gt</span><span class="o">=</span><span class="n">date</span><span class="o">.</span><span class="n">today</span><span class="p">())</span>
</pre>
<p>And many more!
See <a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/contrib/postgres/fields/#querying-range-fields">here</a> to view everything that is available.</p>
</div>
<div class="section" id="constraints">
<h2>Constraints</h2>
<p>One very interesting things you can do with these fields is add an exclusion constraint at the database level to prevent ranges to overlap.
You can also prevent overlap with conditions.
For instance, to take back our listing example, you can prevent overlap across all customers with:</p>
<pre class="code python literal-block">
<span class="kn">from</span> <span class="nn">django.contrib.postgres.constraints</span> <span class="kn">import</span> <span class="n">ExclusionConstraint</span>
<span class="kn">from</span> <span class="nn">django.contrib.postgres.fields</span> <span class="kn">import</span> <span class="n">DateTimeRangeField</span><span class="p">,</span> <span class="n">RangeOperators</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">constraints</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">ExclusionConstraint</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="s1">'exclude_overlapping_validity_range'</span><span class="p">,</span>
<span class="n">expressions</span><span class="o">=</span><span class="p">[</span>
<span class="p">(</span><span class="s1">'validity_range'</span><span class="p">,</span> <span class="n">RangeOperators</span><span class="o">.</span><span class="n">OVERLAPS</span><span class="p">),</span>
<span class="p">],</span>
<span class="p">),</span>
<span class="p">]</span>
</pre>
<p>Or only for each customers:</p>
<pre class="code python literal-block">
<span class="kn">from</span> <span class="nn">django.contrib.postgres.constraints</span> <span class="kn">import</span> <span class="n">ExclusionConstraint</span>
<span class="kn">from</span> <span class="nn">django.contrib.postgres.fields</span> <span class="kn">import</span> <span class="n">DateTimeRangeField</span><span class="p">,</span> <span class="n">RangeOperators</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">constraints</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">ExclusionConstraint</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="s1">'exclude_overlapping_validity_range'</span><span class="p">,</span>
<span class="n">expressions</span><span class="o">=</span><span class="p">[</span>
<span class="p">(</span><span class="s1">'validity_range'</span><span class="p">,</span> <span class="n">RangeOperators</span><span class="o">.</span><span class="n">OVERLAPS</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'customer'</span><span class="p">,</span> <span class="n">RangeOperators</span><span class="o">.</span><span class="n">EQUAL</span><span class="p">),</span>
<span class="p">],</span>
<span class="p">),</span>
<span class="p">]</span>
</pre>
<p>See <a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/contrib/postgres/constraints/">the documentation</a> for a more complete example and the list of all options.</p>
</div>
<div class="section" id="discrete-intervals">
<h2>Discrete intervals</h2>
<p>One thing to note is that in this case, ranges cannot share a date. For instance, if a range ends at 2021-01-01, another cannot start at the same date, even if only one range actually includes the value.</p>
<p>To prevent this, for most ranges, you can end the range just before the new one start.
For instance, with integers, you can do this <tt class="docutils literal"><span class="pre">[0-9]</span></tt> and <tt class="docutils literal"><span class="pre">[10-20]</span></tt>: since there is no integer value between 9 and 10, our ranges are contiguous, without holes between them.
The same reasoning can be easily applied to dates.
It can also be applied for datetimes: the minimum time resolution PostgreSQL supports <a class="reference external" href="https://www.postgresql.org/docs/13/datatype-datetime.html">is the micro-second</a>. So, if one range ends one micro-second before the next one start, we are always in one range or the other and never between ranges.
We will be in a <a class="reference external" href="https://www.postgresql.org/docs/13/rangetypes.html#RANGETYPES-DISCRETE">discrete range</a>.</p>
<p>The same logic should apply to decimals when they have a fixed precision and thus cannot allow numbers between two values, if those values are "close enough" (that is separated only by the smallest value you can represent at this precision) you will be in either interval, never in between.
I don't think this can ever work with float though.</p>
</div>
Deploy a React app in kuberentes2021-04-01T00:00:00+02:002021-04-01T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2021-04-01:/posts/en/2021/Apr/01/react-app-k8s/<p>I recently deployed a React app in kubernetes.
It was initially deployed directly in a public bucket: I have an API that is already hosted on kubernetes, but the app itself is made only of static files, so it made sense.
However, requirements for my app changed and I required …</p><p>I recently deployed a React app in kubernetes.
It was initially deployed directly in a public bucket: I have an API that is already hosted on kubernetes, but the app itself is made only of static files, so it made sense.
However, requirements for my app changed and I required to add basic authentication to it, so before accessing the app the browser would ask for a username and a password.
The main idea being to prevent users to access our pre-production app and to mistake it for our production one should a URL leak.</p>
<p>Since we are limited in what we can do with HTTP headers with GCP buckets, I was stuck.
I also wanted to add extra HTTP headers like <tt class="docutils literal"><span class="pre">X-Frame-Options</span></tt> or <tt class="docutils literal"><span class="pre">X-Content-Type-Options</span></tt> which are useful for security reasons.</p>
<p>To do this, we can:</p>
<ul class="simple">
<li>Insert a nginx between the user and the bucket. The nginx server would then act as a reverse proxy for the bucket. This would add extra round-trips to get the files and after some tests, it's hard to configure nginx correctly in this case.</li>
<li>Just put everything into a container and host the result on kubernetes alongside the API. We avoid round trips, and we have a complex but standard nginx configuration for SPAs. This configuration will be more simple and more tested than our weird "bucket proxy" one. Since the built JS, CSS and image files are small, we will get a small Docker image we can easily deploy.</li>
</ul>
<p>To achieve this, I used the nginx configuration below, stored in a <tt class="docutils literal">ConfigMap</tt> and deployed with <a class="reference external" href="https://helm.sh">Helm</a>:</p>
<pre class="code yaml literal-block">
<span class="ln"> 1 </span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">v1</span><span class="w">
</span><span class="ln"> 2 </span><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">ConfigMap</span><span class="w">
</span><span class="ln"> 3 </span><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span><span class="ln"> 4 </span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">frontend-app-nginx</span><span class="w">
</span><span class="ln"> 5 </span><span class="w"></span><span class="nt">data</span><span class="p">:</span><span class="w">
</span><span class="ln"> 6 </span><span class="w"> </span><span class="nt">website.conf</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">|</span><span class="w">
</span><span class="ln"> 7 </span><span class="w"> </span><span class="no">server {</span><span class="w">
</span><span class="ln"> 8 </span><span class="w"> </span><span class="no">listen 80;</span><span class="w">
</span><span class="ln"> 9 </span><span class="w"> </span><span class="no">root /var/www/frontend-app/;</span><span class="w">
</span><span class="ln">10 </span><span class="w"> </span><span class="no">client_max_body_size 1G;</span><span class="w">
</span><span class="ln">11 </span><span class="w">
</span><span class="ln">12 </span><span class="w"> </span><span class="no">gzip on;</span><span class="w">
</span><span class="ln">13 </span><span class="w"> </span><span class="no">index index.html;</span><span class="w">
</span><span class="ln">14 </span><span class="w">
</span><span class="ln">15 </span><span class="w"> </span><span class="no">access_log stdout;</span><span class="w">
</span><span class="ln">16 </span><span class="w"> </span><span class="no">error_log stderr;</span><span class="w">
</span><span class="ln">17 </span><span class="w">
</span><span class="ln">18 </span><span class="w"> </span><span class="no">location / {</span><span class="w">
</span><span class="ln">19 </span><span class="w"> </span><span class="no">{{ if .Values.container.nginx.enableBasicAuth -}}</span><span class="w">
</span><span class="ln">20 </span><span class="w"> </span><span class="no">auth_basic "Pre-Production. Access Restricted";</span><span class="w">
</span><span class="ln">21 </span><span class="w"> </span><span class="no">auth_basic_user_file /etc/nginx/conf.d/.htpasswd;</span><span class="w">
</span><span class="ln">22 </span><span class="w"> </span><span class="no">{{- end }}</span><span class="w">
</span><span class="ln">23 </span><span class="w">
</span><span class="ln">24 </span><span class="w"> </span><span class="no">add_header X-Frame-Options DENY;</span><span class="w">
</span><span class="ln">25 </span><span class="w"> </span><span class="no">add_header X-XSS-Protection "1; mode=block";</span><span class="w">
</span><span class="ln">26 </span><span class="w"> </span><span class="no">add_header X-Content-Type-Options nosniff;</span><span class="w">
</span><span class="ln">27 </span><span class="w">
</span><span class="ln">28 </span><span class="w"> </span><span class="no"># Set a very long cache on scripts and CSS since their URL contains</span><span class="w">
</span><span class="ln">29 </span><span class="w"> </span><span class="no"># their hash making them immutable.</span><span class="w">
</span><span class="ln">30 </span><span class="w"> </span><span class="no"># We can keep them for as long as we want.</span><span class="w">
</span><span class="ln">31 </span><span class="w"> </span><span class="no"># All those files are in a /static folder.</span><span class="w">
</span><span class="ln">32 </span><span class="w"> </span><span class="no">location /static/ {</span><span class="w">
</span><span class="ln">33 </span><span class="w"> </span><span class="no">add_header Cache-Control "public, max-age=31536000, immutable";</span><span class="w">
</span><span class="ln">34 </span><span class="w"> </span><span class="no">try_files $uri /$uri =404;</span><span class="w">
</span><span class="ln">35 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">36 </span><span class="w">
</span><span class="ln">37 </span><span class="w"> </span><span class="no"># Set some cache on unhashed files (that's everything that ends with an extension).</span><span class="w">
</span><span class="ln">38 </span><span class="w"> </span><span class="no">location ~ .*\.[a-z]+$ {</span><span class="w">
</span><span class="ln">39 </span><span class="w"> </span><span class="no">add_header Cache-Control "public, max-age=3600";</span><span class="w">
</span><span class="ln">40 </span><span class="w"> </span><span class="no">try_files $uri /$uri =404;</span><span class="w">
</span><span class="ln">41 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">42 </span><span class="w">
</span><span class="ln">43 </span><span class="w"> </span><span class="no">location /nghealth {</span><span class="w">
</span><span class="ln">44 </span><span class="w"> </span><span class="no">{{ if .Values.container.nginx.enableBasicAuth -}}</span><span class="w">
</span><span class="ln">45 </span><span class="w"> </span><span class="no"># Always disable basic authentication for health routes.</span><span class="w">
</span><span class="ln">46 </span><span class="w"> </span><span class="no"># It makes working with them easier in kubernetes and for load balancers.</span><span class="w">
</span><span class="ln">47 </span><span class="w"> </span><span class="no">auth_basic off;</span><span class="w">
</span><span class="ln">48 </span><span class="w"> </span><span class="no">{{- end }}</span><span class="w">
</span><span class="ln">49 </span><span class="w"> </span><span class="no">return 200;</span><span class="w">
</span><span class="ln">50 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">51 </span><span class="w">
</span><span class="ln">52 </span><span class="w"> </span><span class="no"># If path doesn't exist, the user wants to reach the site.</span><span class="w">
</span><span class="ln">53 </span><span class="w"> </span><span class="no"># So we return the default index.</span><span class="w">
</span><span class="ln">54 </span><span class="w"> </span><span class="no">add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';</span><span class="w">
</span><span class="ln">55 </span><span class="w"> </span><span class="no">try_files /$uri /index.html =404;</span><span class="w">
</span><span class="ln">56 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">57 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">58 </span><span class="w"> </span><span class="c1"># Put the password here: it's base64 encoded so no directly readable and it exists first and foremost</span><span class="w">
</span><span class="ln">59 </span><span class="w"> </span><span class="c1"># to signal users they don't have anything to do here, not to protect access from anything sensitive.</span><span class="w">
</span><span class="ln">60 </span><span class="w"> </span><span class="nt">.htpasswd</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain"><copy your .htpasswd here></span>
</pre>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">You should always disable authentication for your health routes. You can configure probes to send HTTP headers to make it work, but I encountered issues in my setup where the load balancer can talk directly to the pod and required a health route too. It didn't seem to work when it was protected with basic authentication.</p>
</div>
<p>The <tt class="docutils literal">deployment.yaml</tt> file:</p>
<pre class="code yaml literal-block">
<span class="ln"> 1 </span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">apps/v1</span><span class="w">
</span><span class="ln"> 2 </span><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">Deployment</span><span class="w">
</span><span class="ln"> 3 </span><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span><span class="ln"> 4 </span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">include "chart.fullname" .</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 5 </span><span class="w"> </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span><span class="ln"> 6 </span><span class="w"></span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">include "chart.labels" . | indent 4</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 7 </span><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span><span class="ln"> 8 </span><span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w">
</span><span class="ln"> 9 </span><span class="w"> </span><span class="nt">matchLabels</span><span class="p">:</span><span class="w">
</span><span class="ln">10 </span><span class="w"> </span><span class="nt">app.kubernetes.io/name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">include "chart.name" .</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">11 </span><span class="w"> </span><span class="nt">app.kubernetes.io/instance</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Release.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">12 </span><span class="w"> </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span><span class="ln">13 </span><span class="w"> </span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span><span class="ln">14 </span><span class="w"> </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span><span class="ln">15 </span><span class="w"> </span><span class="nt">app.kubernetes.io/name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">include "chart.name" .</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">16 </span><span class="w"> </span><span class="nt">app.kubernetes.io/instance</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Release.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">17 </span><span class="w"> </span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span><span class="ln">18 </span><span class="w"> </span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span><span class="ln">19 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Chart.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">20 </span><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">.Values.container.image.repository</span><span class="nv"> </span><span class="s">}}:{{</span><span class="nv"> </span><span class="s">.Values.container.image.tag</span><span class="nv"> </span><span class="s">}}"</span><span class="w">
</span><span class="ln">21 </span><span class="w"> </span><span class="nt">imagePullPolicy</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.image.pullPolicy</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">22 </span><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span><span class="ln">23 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">http</span><span class="w">
</span><span class="ln">24 </span><span class="w"> </span><span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">25 </span><span class="w"> </span><span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">TCP</span><span class="w">
</span><span class="ln">26 </span><span class="w"> </span><span class="nt">resources</span><span class="p">:</span><span class="w">
</span><span class="ln">27 </span><span class="w"> </span><span class="nt">limits</span><span class="p">:</span><span class="w">
</span><span class="ln">28 </span><span class="w"> </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.resources.limits.memory</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">29 </span><span class="w"> </span><span class="nt">cpu</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.resources.limits.cpu</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">30 </span><span class="w"> </span><span class="nt">requests</span><span class="p">:</span><span class="w">
</span><span class="ln">31 </span><span class="w"> </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.resources.requests.memory</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">32 </span><span class="w"> </span><span class="nt">cpu</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.resources.requests.cpu</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">33 </span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">if .Values.container.probe.enabled -</span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">34 </span><span class="w"> </span><span class="nt">livenessProbe</span><span class="p">:</span><span class="w">
</span><span class="ln">35 </span><span class="w"> </span><span class="nt">httpGet</span><span class="p">:</span><span class="w">
</span><span class="ln">36 </span><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.path</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">37 </span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">38 </span><span class="w"> </span><span class="nt">timeoutSeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.livenessTimeOut</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">39 </span><span class="w"> </span><span class="nt">initialDelaySeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.initialDelaySeconds</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">40 </span><span class="w"> </span><span class="nt">readinessProbe</span><span class="p">:</span><span class="w">
</span><span class="ln">41 </span><span class="w"> </span><span class="nt">httpGet</span><span class="p">:</span><span class="w">
</span><span class="ln">42 </span><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.path</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">43 </span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">44 </span><span class="w"> </span><span class="nt">timeoutSeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.livenessTimeOut</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">45 </span><span class="w"> </span><span class="nt">initialDelaySeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.initialDelaySeconds</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">46 </span><span class="w"> </span><span class="p-Indicator">{{</span><span class="nv">- end</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">47 </span><span class="w"> </span><span class="nt">volumeMounts</span><span class="p">:</span><span class="w">
</span><span class="ln">48 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">nginx-conf</span><span class="w">
</span><span class="ln">49 </span><span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">/etc/nginx/conf.d</span><span class="w">
</span><span class="ln">50 </span><span class="w"> </span><span class="nt">readOnly</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="ln">51 </span><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span><span class="ln">52 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">nginx-conf</span><span class="w">
</span><span class="ln">53 </span><span class="w"> </span><span class="nt">configMap</span><span class="p">:</span><span class="w">
</span><span class="ln">54 </span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">frontend-app-nginx</span>
</pre>
<p>And my <tt class="docutils literal">Dockerfile</tt> which relies on multi-stage builds to reduce the size of the target image.
I use a node image to build the files and then only a standard nginx image to serve them:</p>
<pre class="code Dockerfile literal-block">
<span class="k">FROM</span><span class="w"> </span><span class="s">node:14.16.0-alpine3.13</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">builder</span>
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/app</span>
<span class="k">RUN</span><span class="w"> </span>apk add python3 make gcc g++ libc-dev
<span class="k">ARG</span><span class="w"> </span><span class="nv">REACT_APP_ENV</span><span class="o">=</span>undefined
<span class="k">ENV</span><span class="w"> </span><span class="nv">REACT_APP_ENV</span><span class="o">=</span><span class="nv">$REACT_APP_ENV</span>
<span class="k">COPY</span><span class="w"> </span>. ./
<span class="k">RUN</span><span class="w"> </span>yarn install --frozen-lockfile
<span class="k">RUN</span><span class="w"> </span>yarn build <span class="o">&&</span> <span class="se">\
</span> find -name <span class="se">\*</span>.js<span class="se">\*</span>.map -delete
<span class="c"># Run the app in nginx.</span>
<span class="k">FROM</span><span class="w"> </span><span class="s">nginx:latest</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">runner</span>
<span class="k">RUN</span><span class="w"> </span>mkdir -p /var/www/frontend-app
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/var/www/frontend-app</span>
<span class="k">COPY</span><span class="w"> </span>--from<span class="o">=</span>builder /app/build /var/www/frontend-app/
</pre>
<p>I pass the necessary arguments to docker so it can build the image with something like <tt class="docutils literal"><span class="pre">--build-arg</span> <span class="pre">REACT_APP_ENV=$REACT_APP_ENV</span></tt>.
I then capture it with the <tt class="docutils literal">ARG</tt> command in the <tt class="docutils literal">Dockerfile</tt> and transform it into an environment variable <tt class="docutils literal"><span class="pre">react-app</span></tt> can use.</p>
<p>With all this, you should be able to have a working nginx for your SPA if you need it.
There are some consequences to doing it this way though:</p>
<ul class="simple">
<li>Nginx will respond with a 200 status and the <tt class="docutils literal">index.html</tt> when a 404 would be more appropriate. Sadly, this cannot be solved in a simple manner, and we also have this issue with the bucket. So on that problem, both solutions are even.</li>
<li>You gain more flexibility over what you can/cannot do with headers and authentication, which is very good.</li>
<li>You should be able to keep old versions of static files to prevent issues when we switch from one version of the app to another. I hope I'll be able to make it work soon. If I do, I'll post a link here.</li>
<li>If we need big files (eg videos, many images…), we will need to put them into a bucket to keep the image small. For instance, if you have big images or videos. Since those files shouldn't change often (and most likely not at every build), managing them differently shouldn't be an issue.</li>
</ul>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">To keep things simple, I don't recommend you deploy a SPA into kubernetes unless you need the extra flexibility a dedicated nginx will give you. Bucket are bare bones, but simple and reliable. Use them for as long as you can.</p>
</div>
Manage static files for Django when deployed on kubernetes2021-03-31T00:00:00+02:002021-03-31T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2021-03-31:/posts/en/2021/Mar/31/manage-static-k8s-django/<p>This will be a continuation of my article <a class="reference external" href="//www.jujens.eu/posts/en/2021/Mar/29/deploy-django-kubernetes/">Some tips to deploy Django in kubernetes</a>.
In this previous article, I talked about generic tips you should apply to deploy Django into kuberentes.
Here, I'll focus on static files.</p>
<p>If you don't already know that, <tt class="docutils literal">gunicorn</tt> (or any other app server …</p><p>This will be a continuation of my article <a class="reference external" href="//www.jujens.eu/posts/en/2021/Mar/29/deploy-django-kubernetes/">Some tips to deploy Django in kubernetes</a>.
In this previous article, I talked about generic tips you should apply to deploy Django into kuberentes.
Here, I'll focus on static files.</p>
<p>If you don't already know that, <tt class="docutils literal">gunicorn</tt> (or any other app server) is not designed to serve your static files directly.
You should use something else.</p>
<p>I think you have three main options:</p>
<ul>
<li><p class="first">Rely on <a class="reference external" href="http://whitenoise.evans.io/en/stable/">WhiteNoise</a> and let your application serve the files.</p>
</li>
<li><p class="first">Put all your static files into a S3 bucket or something equivalent. For that, you will need to:</p>
<ul class="simple">
<li>Make this bucket public on the internet.</li>
<li>Configure <tt class="docutils literal">STATIC_URL</tt> in you settings so Django knows where to search for these files.</li>
<li>Collect your static and upload them into the bucket during your deployment process.</li>
</ul>
</li>
<li><p class="first">Configure an nginx sidecar and let it serve the files. For that, you will need to:</p>
<ul>
<li><p class="first">Configure the nginx sidecar as described in <a class="reference external" href="//www.jujens.eu/posts/en/2021/Mar/29/deploy-django-kubernetes/">my previous article</a>.</p>
</li>
<li><p class="first">Collect the static in the Dockerfile with something like this (I updated my <tt class="docutils literal"><span class="pre">setup-django-run-as-non-root.sh</span></tt> script for that):</p>
<pre class="code bash literal-block">
<span class="c1"># Collect static
# Use dummy values just to allow the command to run.
</span><span class="nb">export</span> <span class="s2">"DJANGO_SETTINGS_MODULE=myapp.settings.prod"</span>
<span class="nb">export</span> <span class="s2">"DB_NAME=postgres"</span>
python manage.py collectstatic --on-input
</pre>
</li>
<li><p class="first">Mount a shared folder in both nginx and the Django pod:</p>
<ul>
<li><p class="first">Configure this volume mount in both pods:</p>
<pre class="code yaml literal-block">
<span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">staticfiles</span><span class="w">
</span><span class="l-Scalar-Plain">mountPath</span><span class="p-Indicator">:</span><span class="w"> </span><span class="l-Scalar-Plain">/var/www/api/</span><span class="w">
</span><span class="l-Scalar-Plain"># The API must be able to copy the files to the volume.</span><span class="w">
</span><span class="l-Scalar-Plain">readOnly</span><span class="p-Indicator">:</span><span class="w"> </span><span class="l-Scalar-Plain">false</span>
</pre>
</li>
<li><p class="first">Configure this empty volume:</p>
<pre class="code yaml literal-block">
<span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">staticfiles</span><span class="w">
</span><span class="nt">emptyDir</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{}</span>
</pre>
</li>
</ul>
</li>
<li><p class="first">Use a script to copy the static files and then run the application: an <tt class="docutils literal">emptyDir</tt> is always empty at pod startup even if its mount point was not in the image. So, if you mount it directly to the <tt class="docutils literal">static</tt> folder, it will be emptied. So we need to mount it elsewhere and then run the application. I created <tt class="docutils literal"><span class="pre">run-django-production.sh</span></tt> for that:</p>
<pre class="code bash literal-block">
<span class="ch">#!/bin/bash
</span>
<span class="nb">set</span> -o errexit
<span class="nb">set</span> -o pipefail
<span class="nb">set</span> -o nounset
mkdir -p /var/www/api/
cp -R static /var/www/api/
gunicorn --bind :8000 --workers <span class="m">5</span> myapp.wsgi
</pre>
</li>
<li><p class="first">Configure nginx correctly, for instance with this configuration:</p>
<pre class="code nginx literal-block">
<span class="ln"> 1 </span><span class="k">upstream</span><span class="w"> </span><span class="s">app_server</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="ln"> 2 </span><span class="w"> </span><span class="kn">server</span><span class="w"> </span><span class="mi">127</span><span class="s">.0.0.1:</span><span class="p">{</span><span class="kn">{</span><span class="w"> </span><span class="s">.Values.container.port</span><span class="w"> </span><span class="err">}}</span><span class="w"> </span><span class="s">fail_timeout=0</span><span class="p">;</span><span class="w">
</span><span class="ln"> 3 </span><span class="w"></span><span class="p">}</span><span class="w">
</span><span class="ln"> 4 </span><span class="w">
</span><span class="ln"> 5 </span><span class="w">
</span><span class="ln"> 6 </span><span class="w"></span><span class="kn">server</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="ln"> 7 </span><span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="mi">80</span><span class="p">;</span><span class="w">
</span><span class="ln"> 8 </span><span class="w"> </span><span class="kn">root</span><span class="w"> </span><span class="s">/var/www/api/</span><span class="p">;</span><span class="w">
</span><span class="ln"> 9 </span><span class="w"> </span><span class="kn">client_max_body_size</span><span class="w"> </span><span class="s">1G</span><span class="p">;</span><span class="w">
</span><span class="ln">10 </span><span class="w">
</span><span class="ln">11 </span><span class="w"> </span><span class="kn">access_log</span><span class="w"> </span><span class="s">stdout</span><span class="p">;</span><span class="w">
</span><span class="ln">12 </span><span class="w"> </span><span class="kn">error_log</span><span class="w"> </span><span class="s">stderr</span><span class="p">;</span><span class="w">
</span><span class="ln">13 </span><span class="w">
</span><span class="ln">14 </span><span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="s">/</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="ln">15 </span><span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="s">/static</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="ln">16 </span><span class="w"> </span><span class="kn">add_header</span><span class="w"> </span><span class="s">Access-Control-Allow-Origin</span><span class="w"> </span><span class="s">*</span><span class="p">;</span><span class="w">
</span><span class="ln">17 </span><span class="w"> </span><span class="kn">add_header</span><span class="w"> </span><span class="s">Access-Control-Max-Age</span><span class="w"> </span><span class="mi">3600</span><span class="p">;</span><span class="w">
</span><span class="ln">18 </span><span class="w"> </span><span class="kn">add_header</span><span class="w"> </span><span class="s">Access-Control-Expose-Headers</span><span class="w"> </span><span class="s">Content-Length</span><span class="p">;</span><span class="w">
</span><span class="ln">19 </span><span class="w"> </span><span class="kn">add_header</span><span class="w"> </span><span class="s">Access-Control-Allow-Headers</span><span class="w"> </span><span class="s">Range</span><span class="p">;</span><span class="w">
</span><span class="ln">20 </span><span class="w">
</span><span class="ln">21 </span><span class="w"> </span><span class="kn">if</span><span class="w"> </span><span class="s">(</span><span class="nv">$request_method</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">OPTIONS)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="ln">22 </span><span class="w"> </span><span class="kn">return</span><span class="w"> </span><span class="mi">204</span><span class="p">;</span><span class="w">
</span><span class="ln">23 </span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="ln">24 </span><span class="w">
</span><span class="ln">25 </span><span class="w"> </span><span class="kn">try_files</span><span class="w"> </span><span class="s">/</span><span class="nv">$uri</span><span class="w"> </span><span class="s">@django</span><span class="p">;</span><span class="w">
</span><span class="ln">26 </span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="ln">27 </span><span class="w">
</span><span class="ln">28 </span><span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="s">/nghealth</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="ln">29 </span><span class="w"> </span><span class="kn">return</span><span class="w"> </span><span class="mi">200</span><span class="p">;</span><span class="w">
</span><span class="ln">30 </span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="ln">31 </span><span class="w">
</span><span class="ln">32 </span><span class="w"> </span><span class="kn">try_files</span><span class="w"> </span><span class="nv">$uri</span><span class="w"> </span><span class="s">@django</span><span class="p">;</span><span class="w">
</span><span class="ln">33 </span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="ln">34 </span><span class="w">
</span><span class="ln">35 </span><span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="s">@django</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="ln">36 </span><span class="w"> </span><span class="kn">proxy_connect_timeout</span><span class="w"> </span><span class="mi">30</span><span class="p">;</span><span class="w">
</span><span class="ln">37 </span><span class="w"> </span><span class="kn">proxy_send_timeout</span><span class="w"> </span><span class="mi">30</span><span class="p">;</span><span class="w">
</span><span class="ln">38 </span><span class="w"> </span><span class="kn">proxy_read_timeout</span><span class="w"> </span><span class="mi">30</span><span class="p">;</span><span class="w">
</span><span class="ln">39 </span><span class="w"> </span><span class="kn">send_timeout</span><span class="w"> </span><span class="mi">30</span><span class="p">;</span><span class="w">
</span><span class="ln">40 </span><span class="w"> </span><span class="c1"># We have another proxy in front of this one. It will capture traffic
</span><span class="ln">41 </span><span class="c1"></span><span class="w"> </span><span class="c1"># as HTTPS, so we must not set X-Forwarded-Proto here since it's already
</span><span class="ln">42 </span><span class="c1"></span><span class="w"> </span><span class="c1"># set with the proper value.
</span><span class="ln">43 </span><span class="c1"></span><span class="w"> </span><span class="c1"># proxy_set_header X-Forwarded-Proto $schema;
</span><span class="ln">44 </span><span class="c1"></span><span class="w"> </span><span class="kn">proxy_set_header</span><span class="w"> </span><span class="s">X-Forwarded-For</span><span class="w"> </span><span class="nv">$proxy_add_x_forwarded_for</span><span class="p">;</span><span class="w">
</span><span class="ln">45 </span><span class="w"> </span><span class="kn">proxy_set_header</span><span class="w"> </span><span class="s">Host</span><span class="w"> </span><span class="nv">$http_host</span><span class="p">;</span><span class="w">
</span><span class="ln">46 </span><span class="w"> </span><span class="kn">proxy_redirect</span><span class="w"> </span><span class="no">off</span><span class="p">;</span><span class="w">
</span><span class="ln">47 </span><span class="w"> </span><span class="kn">proxy_pass</span><span class="w"> </span><span class="s">http://app_server</span><span class="p">;</span><span class="w">
</span><span class="ln">48 </span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="ln">49 </span><span class="w"></span><span class="p">}</span>
</pre>
</li>
</ul>
</li>
</ul>
<p>Now that we've seen the options this question remains: which option should I use?
I guess it depends:</p>
<ul class="simple">
<li>If you need a nginx sidecar (to handle file uploads for instance, see my other article), you can rely on nginx if your files are small: you still want to keep the image as small as possible to speed up the deployment of your app. With this method, you'll avoid adding another component to your application. And since we are talking about static files here, and since it's mostly CSS and JS files, you shouldn't have very big files, so this should work great. If you have big static files in your app, you should probably store them outside your repo anyway.</li>
<li>If you have big files or can afford it, use a bucket. It's simple and reliable.</li>
<li>If you are stuck and cannot use anything else, rely on good old WhiteNoise.</li>
</ul>
Some tips to deploy Django in kubernetes2021-03-29T00:00:00+02:002022-03-09T00:00:00+01:00Julien Enselmetag:www.jujens.eu,2021-03-29:/posts/en/2021/Mar/29/deploy-django-kubernetes/<p>I am not going to go into details in this article about how you can deploy Django in kubernetes.
I am just going to highlight the main points you should pay attention to when deploying a Django app.
I expect you to have prior knowledge about how to deploy an …</p><p>I am not going to go into details in this article about how you can deploy Django in kubernetes.
I am just going to highlight the main points you should pay attention to when deploying a Django app.
I expect you to have prior knowledge about how to deploy an application in kubernetes using <a class="reference external" href="https://helm.sh">Helm</a>.
I hope you will still find useful pieces of information in this article.</p>
<div class="contents topic" id="contents">
<p class="topic-title">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#deploying-the-application" id="toc-entry-1">Deploying the application</a><ul>
<li><a class="reference internal" href="#configurations" id="toc-entry-2">Configurations</a><ul>
<li><a class="reference internal" href="#nginx-sidecar-configuration" id="toc-entry-3">Nginx sidecar configuration</a></li>
<li><a class="reference internal" href="#deployment-yaml" id="toc-entry-4">deployment.yaml</a></li>
<li><a class="reference internal" href="#the-dockerfile-and-related-scripts" id="toc-entry-5">The Dockerfile and related scripts</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#handling-commands" id="toc-entry-6">Handling commands</a></li>
<li><a class="reference internal" href="#history" id="toc-entry-7">History</a></li>
</ul>
</div>
<div class="section" id="deploying-the-application">
<h2><a class="toc-backref" href="#toc-entry-1">Deploying the application</a></h2>
<ul class="simple">
<li>Always disable the <tt class="docutils literal">DEBUG</tt> mode with <tt class="docutils literal">DEBUG=False</tt> in your settings. That's the case for all Django deployments not matter how you do it.</li>
<li>Don't use the the Django dev server to launch your application (that's the <tt class="docutils literal">python manage.py server</tt> command), rely on <tt class="docutils literal">gunicorn</tt> or something equivalent instead (like you normally would).</li>
<li>Rely on environment variables to inject configurations into your settings files. You can use <a class="reference external" href="https://django-environ.readthedocs.io/en/latest/">django-environ</a> to help you read, validate and parse them.<ul>
<li>Store secrets into kubernetes secrets. That includes: the <tt class="docutils literal">SECRET_KEY</tt> configuration value, your database connection details, API keys…</li>
<li>Store everything else into a <tt class="docutils literal">ConfigMap</tt> managed by Helm.</li>
</ul>
</li>
<li>Configure <tt class="docutils literal">livenessProbe</tt> to detect issues with your applications and allow kubernetes to correctly restart the pod if needed.</li>
<li>You may want to add a nginx sidecar container to buffer some requests like file uploads. By default, when you deploy Django into kubernetes, the request will hit <tt class="docutils literal">gunicorn</tt> directly. In the case of long file uploads, it means the <tt class="docutils literal">gunicorn</tt> worker that handles this request cannot do anything until the upload is done. This can be a problem and may result in container restarts (because kubernetes cannot check the liveness probe) or request timeouts. A good way to avoid that, is to put a nginx server in front of <tt class="docutils literal">gunicorn</tt> like you would do if you weren't on kubernetes. The sidecar pattern is a common way to do that. Just make sure your service will route traffic to nginx and not to gunicorn. Normally, this can be done by changing the port it must route traffic to to 80.<ul>
<li>If you use async Django, you should already be good without nginx. Sadly, at this time, the ORM doesn't support async yet so it limits where you can apply this pattern, meaning you probably will need nginx.</li>
<li>You could also use gevent workers, but this involves patching the standard library, so I'm not a fan and don't advise it.</li>
<li>You may be able to configure a ngnix ingress at cluster level. However, after some tests, I didn't succeed to correctly configure it. So I decided to use a nginx sidecar which is a much easier pattern to deal with.</li>
</ul>
</li>
<li>Don't run <tt class="docutils literal">gunicorn</tt> as root in the container to limit the surface of attack.</li>
<li>Use an <tt class="docutils literal">initContainer</tt> to run your migrations.</li>
<li>Give your containers resource quotas to avoid any of them using too much resources.</li>
<li>Put the static files into a bucket or let nginx serve them. See <a class="reference external" href="//www.jujens.eu/posts/en/2021/Mar/31/manage-static-k8s-django/">this article</a>.</li>
</ul>
<div class="section" id="configurations">
<h3><a class="toc-backref" href="#toc-entry-2">Configurations</a></h3>
<p>To help you put this into practice, here are some configuration samples.</p>
<div class="section" id="nginx-sidecar-configuration">
<h4><a class="toc-backref" href="#toc-entry-3">Nginx sidecar configuration</a></h4>
<p>It's a very standard reverse proxy configuration.</p>
<pre class="code yaml literal-block">
<span class="ln"> 1 </span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">v1</span><span class="w">
</span><span class="ln"> 2 </span><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">ConfigMap</span><span class="w">
</span><span class="ln"> 3 </span><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span><span class="ln"> 4 </span><span class="w"></span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">backend-api-nginx</span><span class="w">
</span><span class="ln"> 5 </span><span class="w"></span><span class="nt">data</span><span class="p">:</span><span class="w">
</span><span class="ln"> 6 </span><span class="w"></span><span class="nt">api.conf</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">|</span><span class="w">
</span><span class="ln"> 7 </span><span class="w"> </span><span class="no">upstream app_server {</span><span class="w">
</span><span class="ln"> 8 </span><span class="w"> </span><span class="no"># All containers in the same pod are reachable with 127.0.0.1</span><span class="w">
</span><span class="ln"> 9 </span><span class="w"> </span><span class="no">server 127.0.0.1:{{ .Values.container.port }} fail_timeout=0;</span><span class="w">
</span><span class="ln">10 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">11 </span><span class="w">
</span><span class="ln">12 </span><span class="w">
</span><span class="ln">13 </span><span class="w"> </span><span class="no">server {</span><span class="w">
</span><span class="ln">14 </span><span class="w"> </span><span class="no">listen 80;</span><span class="w">
</span><span class="ln">15 </span><span class="w"> </span><span class="no">root /var/www/api/;</span><span class="w">
</span><span class="ln">16 </span><span class="w"> </span><span class="no">client_max_body_size 1G;</span><span class="w">
</span><span class="ln">17 </span><span class="w">
</span><span class="ln">18 </span><span class="w"> </span><span class="no">access_log stdout;</span><span class="w">
</span><span class="ln">19 </span><span class="w"> </span><span class="no">error_log stderr;</span><span class="w">
</span><span class="ln">20 </span><span class="w">
</span><span class="ln">21 </span><span class="w"> </span><span class="no">location / {</span><span class="w">
</span><span class="ln">22 </span><span class="w"> </span><span class="no">location /static {</span><span class="w">
</span><span class="ln">23 </span><span class="w"> </span><span class="no">add_header Access-Control-Allow-Origin *;</span><span class="w">
</span><span class="ln">24 </span><span class="w"> </span><span class="no">add_header Access-Control-Max-Age 3600;</span><span class="w">
</span><span class="ln">25 </span><span class="w"> </span><span class="no">add_header Access-Control-Expose-Headers Content-Length;</span><span class="w">
</span><span class="ln">26 </span><span class="w"> </span><span class="no">add_header Access-Control-Allow-Headers Range;</span><span class="w">
</span><span class="ln">27 </span><span class="w">
</span><span class="ln">28 </span><span class="w"> </span><span class="no">if ($request_method = OPTIONS) {</span><span class="w">
</span><span class="ln">29 </span><span class="w"> </span><span class="no">return 204;</span><span class="w">
</span><span class="ln">30 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">31 </span><span class="w">
</span><span class="ln">32 </span><span class="w"> </span><span class="no">try_files /$uri @django;</span><span class="w">
</span><span class="ln">33 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">34 </span><span class="w">
</span><span class="ln">35 </span><span class="w"> </span><span class="no"># Dedicated route for nginx health to better understand wher problems come from if needed.</span><span class="w">
</span><span class="ln">36 </span><span class="w"> </span><span class="no">location /nghealth {</span><span class="w">
</span><span class="ln">37 </span><span class="w"> </span><span class="no">return 200;</span><span class="w">
</span><span class="ln">38 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">39 </span><span class="w">
</span><span class="ln">40 </span><span class="w"> </span><span class="no">try_files $uri @django;</span><span class="w">
</span><span class="ln">41 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">42 </span><span class="w">
</span><span class="ln">43 </span><span class="w"> </span><span class="no">location @django {</span><span class="w">
</span><span class="ln">44 </span><span class="w"> </span><span class="no">proxy_connect_timeout 30;</span><span class="w">
</span><span class="ln">45 </span><span class="w"> </span><span class="no">proxy_send_timeout 30;</span><span class="w">
</span><span class="ln">46 </span><span class="w"> </span><span class="no">proxy_read_timeout 30;</span><span class="w">
</span><span class="ln">47 </span><span class="w"> </span><span class="no">send_timeout 30;</span><span class="w">
</span><span class="ln">48 </span><span class="w"> </span><span class="no">proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</span><span class="w">
</span><span class="ln">49 </span><span class="w"> </span><span class="no"># We have another proxy in front of this one. It will capture traffic</span><span class="w">
</span><span class="ln">50 </span><span class="w"> </span><span class="no"># as HTTPS, so we must not set X-Forwarded-Proto here since it's already</span><span class="w">
</span><span class="ln">51 </span><span class="w"> </span><span class="no"># set with the proper value.</span><span class="w">
</span><span class="ln">52 </span><span class="w"> </span><span class="no"># proxy_set_header X-Forwarded-Proto $schema;</span><span class="w">
</span><span class="ln">53 </span><span class="w"> </span><span class="no">proxy_set_header Host $http_host;</span><span class="w">
</span><span class="ln">54 </span><span class="w"> </span><span class="no">proxy_redirect off;</span><span class="w">
</span><span class="ln">55 </span><span class="w"> </span><span class="no">proxy_pass http://app_server;</span><span class="w">
</span><span class="ln">56 </span><span class="w"> </span><span class="no">}</span><span class="w">
</span><span class="ln">57 </span><span class="w"> </span><span class="no">}</span>
</pre>
</div>
<div class="section" id="deployment-yaml">
<h4><a class="toc-backref" href="#toc-entry-4">deployment.yaml</a></h4>
<pre class="code yaml literal-block">
<span class="ln"> 1 </span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">apps/v1</span><span class="w">
</span><span class="ln"> 2 </span><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">Deployment</span><span class="w">
</span><span class="ln"> 3 </span><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span><span class="ln"> 4 </span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">include "chart.fullname" .</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 5 </span><span class="w"> </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span><span class="ln"> 6 </span><span class="w"></span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">include "chart.labels" . | indent 4</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 7 </span><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span><span class="ln"> 8 </span><span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w">
</span><span class="ln"> 9 </span><span class="w"> </span><span class="nt">matchLabels</span><span class="p">:</span><span class="w">
</span><span class="ln"> 10 </span><span class="w"> </span><span class="nt">app.kubernetes.io/name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">include "chart.name" .</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 11 </span><span class="w"> </span><span class="nt">app.kubernetes.io/instance</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Release.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 12 </span><span class="w"> </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span><span class="ln"> 13 </span><span class="w"> </span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span><span class="ln"> 14 </span><span class="w"> </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span><span class="ln"> 15 </span><span class="w"> </span><span class="nt">app.kubernetes.io/name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">include "chart.name" .</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 16 </span><span class="w"> </span><span class="nt">app.kubernetes.io/instance</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Release.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 17 </span><span class="w"> </span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span><span class="ln"> 18 </span><span class="w"> </span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span><span class="ln"> 19 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Chart.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 20 </span><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">.Values.container.image.repository</span><span class="nv"> </span><span class="s">}}:{{</span><span class="nv"> </span><span class="s">.Values.container.image.tag</span><span class="nv"> </span><span class="s">}}"</span><span class="w">
</span><span class="ln"> 21 </span><span class="w"> </span><span class="nt">imagePullPolicy</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.image.pullPolicy</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 22 </span><span class="w"> </span><span class="nt">securityContext</span><span class="p">:</span><span class="w">
</span><span class="ln"> 23 </span><span class="w"> </span><span class="nt">privileged</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">false</span><span class="w">
</span><span class="ln"> 24 </span><span class="w"> </span><span class="nt">runAsUser</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">1001</span><span class="w">
</span><span class="ln"> 25 </span><span class="w"> </span><span class="nt">runAsGroup</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">1001</span><span class="w">
</span><span class="ln"> 26 </span><span class="w"> </span><span class="c1"># Required to prevent escalations to root.</span><span class="w">
</span><span class="ln"> 27 </span><span class="w"> </span><span class="nt">allowPrivilegeEscalation</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">false</span><span class="w">
</span><span class="ln"> 28 </span><span class="w"> </span><span class="nt">runAsNonRoot</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="ln"> 29 </span><span class="w"> </span><span class="nt">envFrom</span><span class="p">:</span><span class="w">
</span><span class="ln"> 30 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">configMapRef</span><span class="p">:</span><span class="w">
</span><span class="ln"> 31 </span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Chart.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 32 </span><span class="w"> </span><span class="nt">optional</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="ln"> 33 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">secretRef</span><span class="p">:</span><span class="w">
</span><span class="ln"> 34 </span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Chart.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 35 </span><span class="w"> </span><span class="nt">optional</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="ln"> 36 </span><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span><span class="ln"> 37 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">http</span><span class="w">
</span><span class="ln"> 38 </span><span class="w"> </span><span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 39 </span><span class="w"> </span><span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">TCP</span><span class="w">
</span><span class="ln"> 40 </span><span class="w"> </span><span class="nt">resources</span><span class="p">:</span><span class="w">
</span><span class="ln"> 41 </span><span class="w"> </span><span class="nt">limits</span><span class="p">:</span><span class="w">
</span><span class="ln"> 42 </span><span class="w"> </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.resources.limits.memory</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 43 </span><span class="w"> </span><span class="nt">cpu</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.resources.limits.cpu</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 44 </span><span class="w"> </span><span class="nt">requests</span><span class="p">:</span><span class="w">
</span><span class="ln"> 45 </span><span class="w"> </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.resources.requests.memory</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 46 </span><span class="w"> </span><span class="nt">cpu</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.resources.requests.cpu</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 47 </span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">if .Values.container.probe.enabled -</span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 48 </span><span class="w"> </span><span class="c1"># As soon as this container is alive, it can serve traffic, so no need for a readinessProbe.</span><span class="w">
</span><span class="ln"> 49 </span><span class="w"> </span><span class="c1"># We still need a bit for it to start before trying to consider it alive: gunicorn must</span><span class="w">
</span><span class="ln"> 50 </span><span class="w"> </span><span class="c1"># start its workers and open connections to the database.</span><span class="w">
</span><span class="ln"> 51 </span><span class="w"> </span><span class="nt">livenessProbe</span><span class="p">:</span><span class="w">
</span><span class="ln"> 52 </span><span class="w"> </span><span class="nt">httpGet</span><span class="p">:</span><span class="w">
</span><span class="ln"> 53 </span><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.path</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 54 </span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 55 </span><span class="w"> </span><span class="nt">timeoutSeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.livenessTimeOut</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 56 </span><span class="w"> </span><span class="nt">initialDelaySeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.initialDelaySeconds</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 57 </span><span class="w"> </span><span class="p-Indicator">{{</span><span class="nv">- end</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 58 </span><span class="w"> </span><span class="nt">volumeMounts</span><span class="p">:</span><span class="w">
</span><span class="ln"> 59 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">backend-credentials</span><span class="w">
</span><span class="ln"> 60 </span><span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">/secrets/backend</span><span class="w">
</span><span class="ln"> 61 </span><span class="w"> </span><span class="nt">readOnly</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="ln"> 62 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">staticfiles</span><span class="w">
</span><span class="ln"> 63 </span><span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">/var/www/api/</span><span class="w">
</span><span class="ln"> 64 </span><span class="w"> </span><span class="c1"># The API must be able to copy the files to the volume.</span><span class="w">
</span><span class="ln"> 65 </span><span class="w"> </span><span class="nt">readOnly</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">false</span><span class="w">
</span><span class="ln"> 66 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">nginx-sidecar</span><span class="w">
</span><span class="ln"> 67 </span><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">nginx:stable</span><span class="w">
</span><span class="ln"> 68 </span><span class="w"> </span><span class="nt">imagePullPolicy</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">Always</span><span class="w">
</span><span class="ln"> 69 </span><span class="w"> </span><span class="nt">securityContext</span><span class="p">:</span><span class="w">
</span><span class="ln"> 70 </span><span class="w"> </span><span class="nt">privileged</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">false</span><span class="w">
</span><span class="ln"> 71 </span><span class="w"> </span><span class="c1"># Nginx must start as root to bind the proper port in the container.</span><span class="w">
</span><span class="ln"> 72 </span><span class="w"> </span><span class="nt">allowPrivilegeEscalation</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="ln"> 73 </span><span class="w"> </span><span class="nt">runAsNonRoot</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">false</span><span class="w">
</span><span class="ln"> 74 </span><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span><span class="ln"> 75 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">http</span><span class="w">
</span><span class="ln"> 76 </span><span class="w"> </span><span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.service.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 77 </span><span class="w"> </span><span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">TCP</span><span class="w">
</span><span class="ln"> 78 </span><span class="w"> </span><span class="nt">volumeMounts</span><span class="p">:</span><span class="w">
</span><span class="ln"> 79 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">nginx-conf</span><span class="w">
</span><span class="ln"> 80 </span><span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">/etc/nginx/conf.d</span><span class="w">
</span><span class="ln"> 81 </span><span class="w"> </span><span class="nt">readOnly</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="ln"> 82 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">staticfiles</span><span class="w">
</span><span class="ln"> 83 </span><span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">/var/www/api/</span><span class="w">
</span><span class="ln"> 84 </span><span class="w"> </span><span class="nt">readOnly</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="ln"> 85 </span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">if .Values.sidecar.nginx.probe.enabled -</span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 86 </span><span class="w"> </span><span class="nt">livenessProbe</span><span class="p">:</span><span class="w">
</span><span class="ln"> 87 </span><span class="w"> </span><span class="nt">httpGet</span><span class="p">:</span><span class="w">
</span><span class="ln"> 88 </span><span class="w"> </span><span class="c1"># When we can access this route, nginx is alive, but it is not ready (ie cannot serve</span><span class="w">
</span><span class="ln"> 89 </span><span class="w"> </span><span class="c1"># traffic yet).</span><span class="w">
</span><span class="ln"> 90 </span><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.sidecar.nginx.probe.path</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 91 </span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.service.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 92 </span><span class="w"> </span><span class="nt">timeoutSeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.sidecar.nginx.probe.livenessTimeOut</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln"> 93 </span><span class="w"> </span><span class="nt">readinessProbe</span><span class="p">:</span><span class="w">
</span><span class="ln"> 94 </span><span class="w"> </span><span class="nt">httpGet</span><span class="p">:</span><span class="w">
</span><span class="ln"> 95 </span><span class="w"> </span><span class="c1"># The container cannot be ready (that is accepting traffic) until it can talk to the</span><span class="w">
</span><span class="ln"> 96 </span><span class="w"> </span><span class="c1"># container. So we need to pass through nginx (with the port) to the container (with</span><span class="w">
</span><span class="ln"> 97 </span><span class="w"> </span><span class="c1"># the path) to check this.</span><span class="w">
</span><span class="ln"> 98 </span><span class="w"> </span><span class="c1"># Since it can take a few seconds, we have an initialDelaySeconds.</span><span class="w">
</span><span class="ln"> 99 </span><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.path</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">100 </span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.service.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">101 </span><span class="w"> </span><span class="nt">initialDelaySeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.sidecar.nginx.probe.initialDelaySeconds</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">102 </span><span class="w"> </span><span class="nt">timeoutSeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.sidecar.nginx.probe.livenessTimeOut</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">103 </span><span class="w"> </span><span class="p-Indicator">{{</span><span class="nv">- end</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">104 </span><span class="w"> </span><span class="nt">resources</span><span class="p">:</span><span class="w">
</span><span class="ln">105 </span><span class="w"> </span><span class="nt">limits</span><span class="p">:</span><span class="w">
</span><span class="ln">106 </span><span class="w"> </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.resources.limits.memory</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">107 </span><span class="w"> </span><span class="nt">cpu</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.resources.limits.cpu</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">108 </span><span class="w"> </span><span class="nt">requests</span><span class="p">:</span><span class="w">
</span><span class="ln">109 </span><span class="w"> </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.initContainer.resources.requests.memory</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">110 </span><span class="w"> </span><span class="nt">cpu</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.initContainer.resources.requests.cpu</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">111 </span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">if .Values.initContainer.enabled -</span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">112 </span><span class="w"> </span><span class="nt">initContainers</span><span class="p">:</span><span class="w">
</span><span class="ln">113 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.initContainer.name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">114 </span><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">.Values.container.image.repository</span><span class="nv"> </span><span class="s">}}:{{</span><span class="nv"> </span><span class="s">.Values.container.image.tag</span><span class="nv"> </span><span class="s">}}"</span><span class="w">
</span><span class="ln">115 </span><span class="w"> </span><span class="nt">imagePullPolicy</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.image.pullPolicy</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">116 </span><span class="w"> </span><span class="nt">envFrom</span><span class="p">:</span><span class="w">
</span><span class="ln">117 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">configMapRef</span><span class="p">:</span><span class="w">
</span><span class="ln">118 </span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Chart.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">119 </span><span class="w"> </span><span class="nt">optional</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="ln">120 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">secretRef</span><span class="p">:</span><span class="w">
</span><span class="ln">121 </span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Chart.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">122 </span><span class="w"> </span><span class="nt">optional</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="ln">123 </span><span class="w"> </span><span class="nt">resources</span><span class="p">:</span><span class="w">
</span><span class="ln">124 </span><span class="w"> </span><span class="nt">limits</span><span class="p">:</span><span class="w">
</span><span class="ln">125 </span><span class="w"> </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.initContainer.resources.limits.memory</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">126 </span><span class="w"> </span><span class="nt">cpu</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.initContainer.resources.limits.cpu</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">127 </span><span class="w"> </span><span class="nt">requests</span><span class="p">:</span><span class="w">
</span><span class="ln">128 </span><span class="w"> </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.initContainer.resources.requests.memory</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">129 </span><span class="w"> </span><span class="nt">cpu</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.initContainer.resources.requests.cpu</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">130 </span><span class="w"> </span><span class="p-Indicator">{{</span><span class="nv">- end</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="ln">131 </span><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span><span class="ln">132 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">nginx-conf</span><span class="w">
</span><span class="ln">133 </span><span class="w"> </span><span class="nt">configMap</span><span class="p">:</span><span class="w">
</span><span class="ln">134 </span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">backend-api-nginx</span><span class="w">
</span><span class="ln">135 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">staticfiles</span><span class="w">
</span><span class="ln">136 </span><span class="w"> </span><span class="nt">emptyDir</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{}</span><span class="w">
</span><span class="ln">137 </span><span class="w"> </span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">backend-credentials</span><span class="w">
</span><span class="ln">138 </span><span class="w"> </span><span class="nt">secret</span><span class="p">:</span><span class="w">
</span><span class="ln">139 </span><span class="w"> </span><span class="nt">secretName</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.gcp.backend.credentials.secret</span><span class="w"> </span><span class="p-Indicator">}}</span>
</pre>
</div>
<div class="section" id="the-dockerfile-and-related-scripts">
<h4><a class="toc-backref" href="#toc-entry-5">The Dockerfile and related scripts</a></h4>
<pre class="code Dockerfile literal-block">
<span class="ln"> 1 </span><span class="c"># Don't use alpine based images: Python was designed for glibc and is very slow in them.</span>
<span class="ln"> 2 </span><span class="c"># Always use the -slim images if you can: they are the best compromise between performance and image size.</span>
<span class="ln"> 3 </span><span class="k">FROM</span><span class="w"> </span><span class="s">python:3.8-slim</span>
<span class="ln"> 4 </span><span class="k">ENV</span><span class="w"> </span>PYTHONPATH /code
<span class="ln"> 5 </span><span class="c"># This is to print directly to stdout instead of buffering output</span>
<span class="ln"> 6 </span><span class="k">ENV</span><span class="w"> </span>PYTHONUNBUFFERED <span class="m">1</span>
<span class="ln"> 7 </span><span class="k">ARG</span><span class="w"> </span><span class="nv">BUILD_RELEASE</span><span class="o">=</span>undefined
<span class="ln"> 8 </span><span class="k">ENV</span><span class="w"> </span><span class="nv">RELEASE</span><span class="o">=</span><span class="nv">$BUILD_RELEASE</span>
<span class="ln"> 9 </span><span class="k">RUN</span><span class="w"> </span>pip install pipenv
<span class="ln">10 </span>
<span class="ln">11 </span><span class="k">WORKDIR</span><span class="w"> </span><span class="s">/code</span>
<span class="ln">12 </span>
<span class="ln">13 </span><span class="k">COPY</span><span class="w"> </span>Pipfile ./
<span class="ln">14 </span><span class="k">COPY</span><span class="w"> </span>Pipfile.lock ./
<span class="ln">15 </span><span class="k">COPY</span><span class="w"> </span>scripts/django-entrypoint.sh scripts/django-install.sh scripts/setup-django-run-as-non-root.sh scripts/run-django-production.sh /usr/local/bin/
<span class="ln">16 </span><span class="k">RUN</span><span class="w"> </span>/usr/local/bin/django-install.sh prod
<span class="ln">17 </span><span class="k">RUN</span><span class="w"> </span>pip install dumb-init
<span class="ln">18 </span>
<span class="ln">19 </span><span class="k">COPY</span><span class="w"> </span>myapp/ ./myapp/
<span class="ln">20 </span><span class="k">COPY</span><span class="w"> </span>manage.py .
<span class="ln">21 </span><span class="k">COPY</span><span class="w"> </span>pyproject.toml .
<span class="ln">22 </span><span class="k">COPY</span><span class="w"> </span>tox.ini .
<span class="ln">23 </span>
<span class="ln">24 </span><span class="c"># Create non-root user and configure it for the project to run correctly with it.</span>
<span class="ln">25 </span><span class="k">RUN</span><span class="w"> </span>/usr/local/bin/setup-django-run-as-non-root.sh
<span class="ln">26 </span>
<span class="ln">27 </span><span class="k">ENTRYPOINT</span><span class="w"> </span><span class="p">[</span><span class="s2">"/usr/local/bin/dumb-init"</span><span class="p">,</span><span class="w"> </span><span class="s2">"--"</span><span class="p">]</span>
<span class="ln">28 </span><span class="k">CMD</span><span class="w"> </span><span class="p">[</span><span class="s2">"/usr/local/bin/django-entrypoint.sh"</span><span class="p">,</span><span class="w"> </span><span class="s2">"/usr/local/bin/run-django-production.sh"</span><span class="p">]</span>
</pre>
<p><tt class="docutils literal"><span class="pre">django-install.sh</span></tt>:</p>
<pre class="code bash literal-block">
<span class="ln"> 1 </span><span class="ch">#!/usr/bin/env bash
</span><span class="ln"> 2 </span><span class="ch"></span>
<span class="ln"> 3 </span><span class="nb">set</span> -e
<span class="ln"> 4 </span><span class="nb">set</span> -u
<span class="ln"> 5 </span><span class="nb">set</span> -o pipefail
<span class="ln"> 6 </span>
<span class="ln"> 7 </span><span class="nv">ENV</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">1</span><span class="k">:-</span><span class="nv">prod</span><span class="si">}</span><span class="s2">"</span>
<span class="ln"> 8 </span><span class="nb">readonly</span> ENV
<span class="ln"> 9 </span>
<span class="ln">10 </span><span class="nb">echo</span> <span class="s2">"Installing deps for env </span><span class="si">${</span><span class="nv">ENV</span><span class="si">}</span><span class="s2">"</span>
<span class="ln">11 </span>
<span class="ln">12 </span>apt-get update
<span class="ln">13 </span><span class="c1"># We must install some deps from git, hence the need to install git.
</span><span class="ln">14 </span><span class="c1"># You may not need this and you may need to install extra libs.
</span><span class="ln">15 </span><span class="c1"></span>apt-get install -y git
<span class="ln">16 </span><span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="si">${</span><span class="nv">ENV</span><span class="si">}</span><span class="s2">"</span> <span class="o">==</span> <span class="s1">'prod'</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="ln">17 </span> pipenv install --system --deploy
<span class="ln">18 </span><span class="k">else</span>
<span class="ln">19 </span> pipenv install --dev --system --deploy
<span class="ln">20 </span><span class="k">fi</span>
<span class="ln">21 </span>apt-get auto-remove -y git
<span class="ln">22 </span>apt-get clean
</pre>
<p><tt class="docutils literal"><span class="pre">setup-django-run-as-non-root.sh</span></tt>:</p>
<pre class="code bash literal-block">
<span class="ln"> 1 </span><span class="ch">#!/usr/bin/env bash
</span><span class="ln"> 2 </span><span class="ch"></span>
<span class="ln"> 3 </span><span class="nb">set</span> -e
<span class="ln"> 4 </span><span class="nb">set</span> -u
<span class="ln"> 5 </span><span class="nb">set</span> -o pipefail
<span class="ln"> 6 </span>
<span class="ln"> 7 </span><span class="c1"># In production, we won't start the container as root so we compile the pyc files
</span><span class="ln"> 8 </span><span class="c1"># and to prepare the collect static while we can write files.
</span><span class="ln"> 9 </span><span class="c1"># We wont be able to do this after once we lost write access to code folders.
</span><span class="ln">10 </span><span class="c1"># uuid cannot be 1000 otherwise chown won't work.
</span><span class="ln">11 </span><span class="c1"># This must match what we use in the securityContext of the pod.
</span><span class="ln">12 </span><span class="c1"></span>groupadd --gid <span class="m">1001</span> gunicorn
<span class="ln">13 </span>useradd gunicorn --uid <span class="m">1001</span> --gid <span class="m">1001</span>
<span class="ln">14 </span>mkdir static
<span class="ln">15 </span>chown gunicorn:gunicorn static
<span class="ln">16 </span>python -m compileall myapp
</pre>
<p><tt class="docutils literal"><span class="pre">django-entrypoint.sh</span></tt>:</p>
<pre class="code bash literal-block">
<span class="ch">#!/bin/bash
</span>
<span class="nb">set</span> -o errexit
<span class="nb">set</span> -o pipefail
<span class="nb">set</span> -o nounset
postgres_ready<span class="o">()</span> <span class="o">{</span>
python <span class="s"><< END
import sys
import psycopg2
try:
psycopg2.connect(
dbname="${DB_NAME}",
user="${DB_USER}",
password="${DB_PASSWORD}",
host="${DB_HOST}",
port="${DB_PORT}",
)
except psycopg2.OperationalError:
sys.exit(-1)
sys.exit(0)
END</span>
<span class="o">}</span>
<span class="k">until</span> postgres_ready<span class="p">;</span> <span class="k">do</span>
><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> <span class="s1">'Waiting for PostgreSQL to become available...'</span>
sleep <span class="m">1</span>
<span class="k">done</span>
><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> <span class="s1">'PostgreSQL is available'</span>
<span class="nb">exec</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</pre>
<p><tt class="docutils literal"><span class="pre">run-django-production.sh</span></tt>:</p>
<pre class="code bash literal-block">
<span class="ch">#!/usr/bin/env bash
</span>
<span class="nb">set</span> -o errexit
<span class="nb">set</span> -o pipefail
<span class="nb">set</span> -o nounset
mkdir -p /var/www/api/
cp -R static /var/www/api/
gunicorn --bind :8000 --workers <span class="m">1</span> myapp.wsgi
</pre>
</div>
</div>
</div>
<div class="section" id="handling-commands">
<h2><a class="toc-backref" href="#toc-entry-6">Handling commands</a></h2>
<p>You can run commands at regular intervals with <a class="reference external" href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/">CronJob</a>.
To avoid the need to create one file per CronJob, you can loop over values as described <a class="reference external" href="https://www.padok.fr/en/blog/kubernetes-cronjob-helm-templates">here</a>.
In a nutshell, you can combine this <tt class="docutils literal">cronjobs.yaml</tt> Helm template:</p>
<pre class="code yaml literal-block">
<span class="p-Indicator">{</span><span class="nv">- range $job</span><span class="p-Indicator">,</span><span class="w"> </span><span class="nv">$val</span><span class="w"> </span><span class="p-Indicator">:</span><span class="nv">= .Values.cronjobs</span><span class="w"> </span><span class="p-Indicator">}</span><span class="err">}</span><span class="w">
</span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">batch/v1beta1</span><span class="w">
</span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">CronJob</span><span class="w">
</span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">.name</span><span class="nv"> </span><span class="s">}}"</span><span class="w">
</span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span><span class="nt">schedule</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">.schedule</span><span class="nv"> </span><span class="s">}}"</span><span class="w">
</span><span class="nt">jobTemplate</span><span class="p">:</span><span class="w">
</span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span><span class="nt">template</span><span class="p">:</span><span class="w">
</span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">.name</span><span class="nv"> </span><span class="s">}}"</span><span class="w">
</span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">$.Values.container.image.repository</span><span class="nv"> </span><span class="s">}}:{{</span><span class="nv"> </span><span class="s">$.Values.container.image.tag</span><span class="nv"> </span><span class="s">}}"</span><span class="w">
</span><span class="nt">imagePullPolicy</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">$.Values.container.image.pullPolicy</span><span class="nv"> </span><span class="s">}}"</span><span class="w">
</span><span class="nt">args</span><span class="p">:</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="l-Scalar-Plain">python</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="l-Scalar-Plain">manage.py</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">.djangoCommand</span><span class="nv"> </span><span class="s">}}"</span><span class="w">
</span><span class="nt">envFrom</span><span class="p">:</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">configMapRef</span><span class="p">:</span><span class="w">
</span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">$.Chart.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">optional</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">secretRef</span><span class="p">:</span><span class="w">
</span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">$.Chart.Name</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">optional</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="nt">restartPolicy</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">.restartPolicy</span><span class="nv"> </span><span class="s">}}"</span><span class="w">
</span><span class="nn">---</span><span class="w">
</span><span class="p-Indicator">{{</span><span class="nv">- end</span><span class="p-Indicator">}}</span>
</pre>
<p>With this configuration:</p>
<pre class="code yaml literal-block">
<span class="c1"># We currently assume we run the API Python/Django image for all jobs.</span><span class="w">
</span><span class="nt">cronjobs</span><span class="p">:</span><span class="w">
</span><span class="s">"0"</span><span class="p-Indicator">:</span><span class="w">
</span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">backend-api-clearsessions</span><span class="w">
</span><span class="c1"># This must be in the standard Unix crontab format</span><span class="w">
</span><span class="nt">schedule</span><span class="p">:</span><span class="w"> </span><span class="s">"0</span><span class="nv"> </span><span class="s">23</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*"</span><span class="w">
</span><span class="nt">djangoCommand</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">clearsessions</span><span class="w">
</span><span class="nt">restartPolicy</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">Never</span><span class="w">
</span><span class="s">"1"</span><span class="p-Indicator">:</span><span class="w">
</span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">backend-api-clean-pending-loan-applications</span><span class="w">
</span><span class="nt">schedule</span><span class="p">:</span><span class="w"> </span><span class="s">"0</span><span class="nv"> </span><span class="s">23</span><span class="nv"> </span><span class="s">1</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*"</span><span class="w">
</span><span class="nt">djangoCommand</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">remove_stale_contenttypes</span><span class="w">
</span><span class="nt">restartPolicy</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">Never</span>
</pre>
<p>To create two CronJob in kubernetes: one for <tt class="docutils literal">python manage.py clearsessions</tt> launched every day at 23:00 and one for <tt class="docutils literal">python manage.py remove_stale_contenttypes</tt> launched every fist day of each month at 23:00.</p>
</div>
<div class="section" id="history">
<h2><a class="toc-backref" href="#toc-entry-7">History</a></h2>
<ul class="simple">
<li>9th of March 2022: corrected static files volume mounts and copy of static files to volumes. Thanks to <a class="reference external" href="https://www.jujens.eu/posts/en/2021/Mar/29/deploy-django-kubernetes/#isso-281">RomoSapiens</a> for the catch.</li>
</ul>
</div>
Enable basic authentication to all pages of a NextJS site2021-03-28T00:00:00+01:002021-03-30T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2021-03-28:/posts/en/2021/Mar/28/enable-basic-auth-nextjs/<p>It's not as obvious at it seems.
You can protect your API routes or some pages by following <a class="reference external" href="https://nextjs.org/docs/authentication">the documentation</a>, but nothing to protect everything in one go with basic authentication (to protect your pre-production site from normal user for instance).
Despite NexJS having a server component, I didn't find …</p><p>It's not as obvious at it seems.
You can protect your API routes or some pages by following <a class="reference external" href="https://nextjs.org/docs/authentication">the documentation</a>, but nothing to protect everything in one go with basic authentication (to protect your pre-production site from normal user for instance).
Despite NexJS having a server component, I didn't find a way to do it easily with a middleware.
So I decided to put an nginx in front of NexJS to handle the authentication.</p>
<p>Since this site is deployed in kubernetes, I used the <a class="reference external" href="https://kubernetes.io/docs/concepts/workloads/pods/">sidecar patterns</a> to have a container with nginx next to my NexJS container.</p>
<p>My nginx configuration is like this:</p>
<pre class="code nginx literal-block">
<span class="k">upstream</span><span class="w"> </span><span class="s">app_server</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="kn">server</span><span class="w"> </span><span class="mi">127</span><span class="s">.0.0.1:</span><span class="p">{</span><span class="kn">{</span><span class="w"> </span><span class="s">.Values.container.port</span><span class="w"> </span><span class="err">}}</span><span class="w"> </span><span class="s">fail_timeout=0</span><span class="p">;</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="kn">server</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="kn">listen</span><span class="w"> </span><span class="mi">80</span><span class="p">;</span><span class="w">
</span><span class="kn">root</span><span class="w"> </span><span class="s">/var/www/website/</span><span class="p">;</span><span class="w">
</span><span class="kn">client_max_body_size</span><span class="w"> </span><span class="s">1G</span><span class="p">;</span><span class="w">
</span><span class="kn">access_log</span><span class="w"> </span><span class="s">stdout</span><span class="p">;</span><span class="w">
</span><span class="kn">error_log</span><span class="w"> </span><span class="s">stderr</span><span class="p">;</span><span class="w">
</span><span class="kn">location</span><span class="w"> </span><span class="s">/</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="c1"># Only protect / with authentication and not @nextjs by placing the directives here.
</span><span class="w"> </span><span class="c1"># If you don't, nginx will require you to authenticate for the /api/health route even
</span><span class="w"> </span><span class="c1"># if you disable authentication for it since it's forwarded to @nextjs.
</span><span class="w"> </span><span class="kn">{{</span><span class="w"> </span><span class="s">if</span><span class="w"> </span><span class="s">.Values.sidecar.nginx.enableBasicAuth</span><span class="w"> </span><span class="s">-</span><span class="err">}}</span><span class="w">
</span><span class="s">auth_basic</span><span class="w"> </span><span class="s">"Pre-Production.</span><span class="w"> </span><span class="s">Access</span><span class="w"> </span><span class="s">Restricted"</span><span class="p">;</span><span class="w">
</span><span class="kn">auth_basic_user_file</span><span class="w"> </span><span class="s">/etc/nginx/conf.d/.htpasswd</span><span class="p">;</span><span class="w">
</span><span class="kn">{{-</span><span class="w"> </span><span class="s">end</span><span class="w"> </span><span class="err">}}</span><span class="w">
</span><span class="s">location</span><span class="w"> </span><span class="s">/nghealth</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="kn">{{</span><span class="w"> </span><span class="s">if</span><span class="w"> </span><span class="s">.Values.sidecar.nginx.enableBasicAuth</span><span class="w"> </span><span class="s">-</span><span class="err">}}</span><span class="w">
</span><span class="s">auth_basic</span><span class="w"> </span><span class="no">off</span><span class="p">;</span><span class="w">
</span><span class="kn">{{-</span><span class="w"> </span><span class="s">end</span><span class="w"> </span><span class="err">}}</span><span class="w">
</span><span class="s">return</span><span class="w"> </span><span class="mi">200</span><span class="p">;</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="kn">location</span><span class="w"> </span><span class="s">/api/health</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="kn">{{</span><span class="w"> </span><span class="s">if</span><span class="w"> </span><span class="s">.Values.sidecar.nginx.enableBasicAuth</span><span class="w"> </span><span class="s">-</span><span class="err">}}</span><span class="w">
</span><span class="s">auth_basic</span><span class="w"> </span><span class="no">off</span><span class="p">;</span><span class="w">
</span><span class="kn">{{-</span><span class="w"> </span><span class="s">end</span><span class="w"> </span><span class="err">}}</span><span class="w">
</span><span class="s">try_files</span><span class="w"> </span><span class="nv">$uri</span><span class="w"> </span><span class="s">@nextjs</span><span class="p">;</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="kn">try_files</span><span class="w"> </span><span class="nv">$uri</span><span class="w"> </span><span class="s">@nextjs</span><span class="p">;</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="kn">location</span><span class="w"> </span><span class="s">@nextjs</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="kn">proxy_connect_timeout</span><span class="w"> </span><span class="mi">30</span><span class="p">;</span><span class="w">
</span><span class="kn">proxy_send_timeout</span><span class="w"> </span><span class="mi">30</span><span class="p">;</span><span class="w">
</span><span class="kn">proxy_read_timeout</span><span class="w"> </span><span class="mi">30</span><span class="p">;</span><span class="w">
</span><span class="kn">send_timeout</span><span class="w"> </span><span class="mi">30</span><span class="p">;</span><span class="w">
</span><span class="kn">proxy_set_header</span><span class="w"> </span><span class="s">X-Forwarded-For</span><span class="w"> </span><span class="nv">$proxy_add_x_forwarded_for</span><span class="p">;</span><span class="w">
</span><span class="c1"># We have another proxy in front of this one. It will capture traffic
</span><span class="w"> </span><span class="c1"># as HTTPS, so we must not set X-Forwarded-Proto here since it's already
</span><span class="w"> </span><span class="c1"># set with the proper value.
</span><span class="w"> </span><span class="c1"># proxy_set_header X-Forwarded-Proto $schema;
</span><span class="w"> </span><span class="kn">proxy_set_header</span><span class="w"> </span><span class="s">Host</span><span class="w"> </span><span class="nv">$http_host</span><span class="p">;</span><span class="w">
</span><span class="kn">proxy_redirect</span><span class="w"> </span><span class="no">off</span><span class="p">;</span><span class="w">
</span><span class="kn">proxy_pass</span><span class="w"> </span><span class="s">http://app_server</span><span class="p">;</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span>
</pre>
<div class="admonition tip">
<p class="first admonition-title">Astuce</p>
<p class="last">Always use a route to check that nginx is ok and one to check that your app is ok. This way, in case of failure, it will be easier to spot the faulty component.</p>
</div>
<div class="admonition tip">
<p class="first admonition-title">Astuce</p>
<p class="last">Never protect the health routes with authentication: while you can configure your probes to pass the <tt class="docutils literal">Authorization</tt> header, when I tried I encountered error with my GCP load balancers which also needs to check everything is fine to route traffic correctly directly to the pod.</p>
</div>
<p>As you can guess, I'm using <a class="reference external" href="https://helm.sh">Helm</a> to deploy this. So this configuration file is in a dedicated <tt class="docutils literal">ConfigMap</tt> template like this:</p>
<pre class="code yaml literal-block">
<span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">v1</span><span class="w">
</span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">ConfigMap</span><span class="w">
</span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">website-reverse-proxy</span><span class="w">
</span><span class="nt">data</span><span class="p">:</span><span class="w">
</span><span class="nt">website.conf</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">|</span><span class="w">
</span><span class="no">upstream app_server {</span><span class="w">
</span><span class="no">server 127.0.0.1:{{ .Values.container.port }} fail_timeout=0;</span><span class="w">
</span><span class="no">}</span><span class="w">
</span><span class="no">[Cut for brievety]</span>
</pre>
<p>Since the authentication is only there to prevent people that are not in the company to view the site, I decided to include the content of the <tt class="docutils literal">.htpasswd</tt> file in the <tt class="docutils literal">ConfigMap</tt> above. You probably don't want to do that if it's sensitive and rely on a secret instead. For that, I just created the <tt class="docutils literal">.htpassword</tt> file locally with the <tt class="docutils literal">htpasswd</tt> command and copied its content into my config map.</p>
<p>I can then mount both of these values into the container so it can use them directly in my <tt class="docutils literal">deployment.yaml</tt> template:</p>
<pre class="code yaml literal-block">
<span class="nt">containers</span><span class="p">:</span><span class="w">
</span><span class="p-Indicator">[</span><span class="nv">NextJS ommited for brievety</span><span class="p-Indicator">]</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">nginx-sidecar</span><span class="w">
</span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">nginx:stable</span><span class="w">
</span><span class="nt">imagePullPolicy</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">Always</span><span class="w">
</span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">http</span><span class="w">
</span><span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.service.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">TCP</span><span class="w">
</span><span class="nt">volumeMounts</span><span class="p">:</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">nginx-conf</span><span class="w">
</span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">/etc/nginx/conf.d</span><span class="w">
</span><span class="nt">readOnly</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">if .Values.sidecar.nginx.probe.enabled -</span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">livenessProbe</span><span class="p">:</span><span class="w">
</span><span class="nt">httpGet</span><span class="p">:</span><span class="w">
</span><span class="c1"># When we can access this route, nginx is alive, but it is not ready (ie cannot serve</span><span class="w">
</span><span class="c1"># traffic yet).</span><span class="w">
</span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.sidecar.nginx.probe.path</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.service.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">timeoutSeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.sidecar.nginx.probe.livenessTimeOut</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">readinessProbe</span><span class="p">:</span><span class="w">
</span><span class="nt">httpGet</span><span class="p">:</span><span class="w">
</span><span class="c1"># The container cannot be ready (that is accepting traffic) until it can talk to the</span><span class="w">
</span><span class="c1"># container. So we need to pass through nginx (with the port) to the container (with</span><span class="w">
</span><span class="c1"># the path) to check this.</span><span class="w">
</span><span class="c1"># Since it can take a few seconds, we have an initialDelaySeconds.</span><span class="w">
</span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.container.probe.path</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.service.port</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">initialDelaySeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.sidecar.nginx.probe.initialDelaySeconds</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">timeoutSeconds</span><span class="p">:</span><span class="w"> </span><span class="p-Indicator">{{</span><span class="w"> </span><span class="nv">.Values.sidecar.nginx.probe.livenessTimeOut</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="p-Indicator">{{</span><span class="nv">- end</span><span class="w"> </span><span class="p-Indicator">}}</span><span class="w">
</span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">nginx-conf</span><span class="w">
</span><span class="nt">configMap</span><span class="p">:</span><span class="w">
</span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">website-reverse-proxy</span>
</pre>
<p>I need the two probes:</p>
<ul class="simple">
<li><tt class="docutils literal">livenessProbe</tt> to check that nginx is OK and ready to serve requests.</li>
<li><tt class="docutils literal">readinessProbe</tt> to check that nginx can communicate with NexJS and can serve actual traffic. So in this one, I target the health probe of NextJS through nginx by using its port and not the port of NextJS. Hence the need for these two routes to be accessible without authentication.</li>
</ul>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">I also have both probes in my NexJS deployment and they both target directly the <tt class="docutils literal">/api/health</tt> route. Again, this seems required for GCP load balancers to work correctly.</p>
</div>
<p>Lastly, my NexJS route in <tt class="docutils literal">pages/api/health.ts</tt>:</p>
<pre class="code js literal-block">
<span class="k">import</span> <span class="p">{</span> <span class="nx">NextApiRequest</span><span class="p">,</span> <span class="nx">NextApiResponse</span> <span class="p">}</span> <span class="kr">from</span> <span class="s2">"next"</span><span class="p">;</span>
<span class="k">export</span> <span class="k">default</span> <span class="k">async</span> <span class="p">(</span><span class="nx">req</span><span class="o">:</span> <span class="nx">NextApiRequest</span><span class="p">,</span> <span class="nx">res</span><span class="o">:</span> <span class="nx">NextApiResponse</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mf">200</span><span class="p">).</span><span class="nx">json</span><span class="p">({});</span>
<span class="p">};</span>
</pre>
<div class="section" id="history">
<h2>History</h2>
<ul class="simple">
<li>2021-03-30: I fixed the configurations and added some notes to make it works better. With the previous implementation, some problems could occur as explained in the body of the article. Go <a class="reference external" href="https://gitlab.com/Jenselme/www.jujens.eu/-/commits/master/content/Trucs%20et%20astuces/2021-03-28_enable-basic-auth-nextjs.rst">here</a> to view the changes.</li>
</ul>
</div>
Extract kubectl configmap/secret to .env file2021-03-21T00:00:00+01:002021-03-21T00:00:00+01:00Julien Enselmetag:www.jujens.eu,2021-03-21:/posts/en/2021/Mar/21/kubectl-cfg-to-env/<p>You can extract data from your kubernetes config maps into a <tt class="docutils literal">.env</tt> file with the commands below (requires you to have <a class="reference external" href="https://stedolan.github.io/jq/">jq</a> installed):</p>
<pre class="code bash literal-block">
<span class="c1"># Get the data in JSON.
</span>kubectl get configmap my-map --output json <span class="p">|</span>
<span class="c1"># Extract the data section.
</span> jq <span class="s1">'.data'</span> <span class="p">|</span>
<span class="c1"># Replace each "key": "value" pair with "key=value"
</span> jq -r …</pre><p>You can extract data from your kubernetes config maps into a <tt class="docutils literal">.env</tt> file with the commands below (requires you to have <a class="reference external" href="https://stedolan.github.io/jq/">jq</a> installed):</p>
<pre class="code bash literal-block">
<span class="c1"># Get the data in JSON.
</span>kubectl get configmap my-map --output json <span class="p">|</span>
<span class="c1"># Extract the data section.
</span> jq <span class="s1">'.data'</span> <span class="p">|</span>
<span class="c1"># Replace each "key": "value" pair with "key=value"
</span> jq -r <span class="s1">'to_entries | map(.key + "=" + (.value)) | .[]'</span> >> .env
</pre>
<p>You can also do that with secrets:</p>
<pre class="code bash literal-block">
<span class="c1"># Get the data in JSON.
</span>kubectl get secret my-secret --output json <span class="p">|</span>
<span class="c1"># Extract the data section.
</span> jq <span class="s1">'.data'</span> <span class="p">|</span>
<span class="c1"># Decode the value of each keys.
</span> jq <span class="s1">'map_values(@base64d)'</span> <span class="p">|</span>
<span class="c1"># Replace each "key": "value" pair with "key=value"
</span> jq -r <span class="s1">'to_entries | map(.key + "=" + (.value)) | .[]'</span> >> .env
</pre>
<p>Or with configmap data from helm templates:</p>
<pre class="code bash literal-block">
<span class="c1"># Extract the values with a `awk` script: we print everything starting from the line that contains only `configmap` until the first empty line.
</span>awk <span class="s1">'{if ($0 ~ /^configmap:$/) {triggered=1;}if (triggered) {print; if ($0 ~ /^$/) { exit;}}}'</span> <span class="s2">"./project/values.yaml"</span> <span class="p">|</span>
<span class="c1"># Keep only the indented lines that contains our configuration values.
</span> grep <span class="s1">'^ '</span> <span class="p">|</span>
<span class="c1"># Transform key: value into key=value
</span> sed <span class="s1">'s/ //;s/: /=/'</span> >> .env
</pre>
Server tips2020-09-17T00:00:00+02:002020-09-17T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2020-09-17:/posts/en/2020/Sep/17/server/<div class="contents topic" id="sommaire">
<p class="topic-title">Sommaire</p>
<ul class="simple">
<li><a class="reference internal" href="#start-systemd-services-automatically-for-a-user" id="id1">Start systemd services automatically for a user</a></li>
</ul>
</div>
<div class="section" id="start-systemd-services-automatically-for-a-user">
<h2><a class="toc-backref" href="#id1">Start systemd services automatically for a user</a></h2>
<p>By default sysemtd services linked to a user – created with <tt class="docutils literal">systemctl <span class="pre">--user</span> enable <span class="pre">--now</span> service</tt> for instance – are only started after the first login of the user.
To start them when the server starts, you …</p></div><div class="contents topic" id="sommaire">
<p class="topic-title">Sommaire</p>
<ul class="simple">
<li><a class="reference internal" href="#start-systemd-services-automatically-for-a-user" id="id1">Start systemd services automatically for a user</a></li>
</ul>
</div>
<div class="section" id="start-systemd-services-automatically-for-a-user">
<h2><a class="toc-backref" href="#id1">Start systemd services automatically for a user</a></h2>
<p>By default sysemtd services linked to a user – created with <tt class="docutils literal">systemctl <span class="pre">--user</span> enable <span class="pre">--now</span> service</tt> for instance – are only started after the first login of the user.
To start them when the server starts, you must enable lingering for the user with (as root):</p>
<pre class="literal-block">
loginctl enable-linger USER
</pre>
<p>You can also soo the lingering status for a user with (as root):</p>
<pre class="literal-block">
loginctl show-user USER --property Linger
</pre>
<p>Source: <a class="reference external" href="https://wiki.archlinux.org/index.php/Systemd/User#Automatic_start-up_of_systemd_user_instances">https://wiki.archlinux.org/index.php/Systemd/User#Automatic_start-up_of_systemd_user_instances</a></p>
</div>
CSS tips2019-03-10T00:00:00+01:002019-03-10T00:00:00+01:00Julien Enselmetag:www.jujens.eu,2019-03-10:/posts/en/2019/Mar/10/css/<div class="contents topic" id="sommaire">
<p class="topic-title">Sommaire</p>
<ul class="simple">
<li><a class="reference internal" href="#apply-style-to-ie11-only" id="id1">Apply style to IE11 only</a></li>
</ul>
</div>
<div class="section" id="apply-style-to-ie11-only">
<h2><a class="toc-backref" href="#id1">Apply style to IE11 only</a></h2>
<p>Put the style in this media query:</p>
<pre class="code CSS literal-block">
<span class="p">@</span><span class="k">media</span> <span class="nt">screen</span> <span class="nt">and</span> <span class="o">(</span><span class="nt">-ms-high-contrast</span><span class="o">:</span> <span class="nt">active</span><span class="o">),</span> <span class="o">(</span><span class="nt">-ms-high-contrast</span><span class="o">:</span> <span class="nt">none</span><span class="o">)</span> <span class="p">{</span>
<span class="o">//</span> <span class="nt">Put</span> <span class="nt">CSS</span> <span class="nt">here</span>
<span class="p">}</span>
</pre>
</div>
PyCharm tips2019-03-10T00:00:00+01:002019-03-10T00:00:00+01:00Julien Enselmetag:www.jujens.eu,2019-03-10:/posts/en/2019/Mar/10/pycharm/<div class="contents topic" id="sommaire">
<p class="topic-title">Sommaire</p>
<ul class="simple">
<li><a class="reference internal" href="#view-print-output-immediately" id="id1">View print output immediately</a></li>
<li><a class="reference internal" href="#auto-load-env-file" id="id2">Auto-load .env file</a></li>
<li><a class="reference internal" href="#use-shell-plus-in-the-integrated-console" id="id3">Use shell_plus in the integrated console</a></li>
<li><a class="reference internal" href="#make-sure-commit-hooks-run-in-a-venv" id="id4">Make sure commit hooks run in a venv</a></li>
</ul>
</div>
<div class="section" id="view-print-output-immediately">
<h2><a class="toc-backref" href="#id1">View print output immediately</a></h2>
<p>Add <tt class="docutils literal">PYTHONUNBUFFERED=1</tt> in run config or <tt class="docutils literal">.env</tt> (only use <tt class="docutils literal">.env</tt> if you use the plugin). This is meant to immediately view the output …</p></div><div class="contents topic" id="sommaire">
<p class="topic-title">Sommaire</p>
<ul class="simple">
<li><a class="reference internal" href="#view-print-output-immediately" id="id1">View print output immediately</a></li>
<li><a class="reference internal" href="#auto-load-env-file" id="id2">Auto-load .env file</a></li>
<li><a class="reference internal" href="#use-shell-plus-in-the-integrated-console" id="id3">Use shell_plus in the integrated console</a></li>
<li><a class="reference internal" href="#make-sure-commit-hooks-run-in-a-venv" id="id4">Make sure commit hooks run in a venv</a></li>
</ul>
</div>
<div class="section" id="view-print-output-immediately">
<h2><a class="toc-backref" href="#id1">View print output immediately</a></h2>
<p>Add <tt class="docutils literal">PYTHONUNBUFFERED=1</tt> in run config or <tt class="docutils literal">.env</tt> (only use <tt class="docutils literal">.env</tt> if you use the plugin). This is meant to immediately view the output of the print function without the need for flush=True.</p>
</div>
<div class="section" id="auto-load-env-file">
<h2><a class="toc-backref" href="#id2">Auto-load .env file</a></h2>
<ul>
<li><p class="first">Install the <a class="reference external" href="https://plugins.jetbrains.com/plugin/7861-envfile">EnvFile</a> and <a class="reference external" href="https://plugins.jetbrains.com/plugin/9525--env-files-support">.env file support</a> plugins. Then configure the Django server to run with the custom command <tt class="docutils literal">runserver_plus</tt> (optional) and in the <em>EnvFile</em> tab, tick <em>Enable EnvFile</em> and in the box below, add you <tt class="docutils literal">.env</tt> file. Do the same for pytest.</p>
</li>
<li><p class="first">Use [direnv](<a class="reference external" href="http://direnv.net/">http://direnv.net/</a>) to autoload the <tt class="docutils literal">.env</tt>. To do this, create a <tt class="docutils literal">.envrc</tt> file with the content below and run <tt class="docutils literal">direnv allow .</tt> to allow the <tt class="docutils literal">.envrc</tt> file to be loaded:</p>
<pre class="literal-block">
# Auto load venv.
layout_pipenv
# Auto load .env file
dotenv
</pre>
</li>
<li><p class="first">Add this to the <tt class="docutils literal">activate</tt> script:</p>
</li>
</ul>
<pre class="code bash literal-block">
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="si">${</span><span class="nv">TERMINAL_EMULATOR</span><span class="si">}</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"JetBrains-JediTerm"</span> <span class="o">&&</span> -f .env <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
load-dotenv .env
<span class="k">fi</span>
</pre>
</div>
<div class="section" id="use-shell-plus-in-the-integrated-console">
<h2><a class="toc-backref" href="#id3">Use shell_plus in the integrated console</a></h2>
<p>Use this starting script in <em>Build, Execution, Deployment</em> > <em>Console</em> > <em>Django console</em>:</p>
<pre class="code python literal-block">
<span class="c1"># Load env var</span>
<span class="kn">import</span> <span class="nn">environ</span>
<span class="n">environ</span><span class="o">.</span><span class="n">Env</span><span class="o">.</span><span class="n">read_env</span><span class="p">(</span><span class="s1">'.env'</span><span class="p">)</span>
<span class="kn">import</span> <span class="nn">sys</span><span class="p">;</span> <span class="nb">print</span><span class="p">(</span><span class="s1">'Python </span><span class="si">%s</span><span class="s1"> on </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">version</span><span class="p">,</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="p">))</span>
<span class="kn">import</span> <span class="nn">django</span><span class="p">;</span> <span class="nb">print</span><span class="p">(</span><span class="s1">'Django </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">django</span><span class="o">.</span><span class="n">get_version</span><span class="p">())</span>
<span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span><span class="n">WORKING_DIR_AND_PYTHON_PATHS</span><span class="p">])</span>
<span class="k">if</span> <span class="s1">'setup'</span> <span class="ow">in</span> <span class="nb">dir</span><span class="p">(</span><span class="n">django</span><span class="p">):</span> <span class="n">django</span><span class="o">.</span><span class="n">setup</span><span class="p">()</span>
<span class="kn">import</span> <span class="nn">django_manage_shell</span><span class="p">;</span> <span class="n">django_manage_shell</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">PROJECT_ROOT</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">django_extensions.management</span> <span class="kn">import</span> <span class="n">shells</span>
<span class="kn">from</span> <span class="nn">django.core.management.color</span> <span class="kn">import</span> <span class="n">color_style</span>
<span class="c1"># Default settings for shell_plus</span>
<span class="n">shell_plus_default_settings</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'bpython'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'connection_file'</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span>
<span class="s1">'dont_load'</span><span class="p">:</span> <span class="p">[],</span>
<span class="s1">'ipython'</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
<span class="s1">'kernel'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'no_browser'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'no_color'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'notebook'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'plain'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'print_sql'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'ptipython'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'ptpython'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'pythonpath'</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span>
<span class="s1">'quiet_load'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'settings'</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span>
<span class="s1">'traceback'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'use_pythonrc'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'verbosity'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'vi_mode'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'dont_load'</span><span class="p">:</span> <span class="p">[],</span>
<span class="p">}</span>
<span class="n">g</span> <span class="o">=</span> <span class="nb">globals</span><span class="p">()</span>
<span class="n">objects_to_import</span> <span class="o">=</span> <span class="n">shells</span><span class="o">.</span><span class="n">import_objects</span><span class="p">(</span><span class="n">shell_plus_default_settings</span><span class="p">,</span> <span class="n">color_style</span><span class="p">())</span>
<span class="n">g</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">objects_to_import</span><span class="p">)</span>
</pre>
</div>
<div class="section" id="make-sure-commit-hooks-run-in-a-venv">
<h2><a class="toc-backref" href="#id4">Make sure commit hooks run in a venv</a></h2>
<p>You can use this script:</p>
<pre class="code bash literal-block">
<span class="ch">#!/usr/bin/env bash
</span>
<span class="nb">set</span> -eu
<span class="nb">readonly</span> <span class="nv">INSIDE_DOCKER</span><span class="o">=</span><span class="k">$(</span>grep -q docker /proc/self/cgroup <span class="o">&&</span> <span class="nb">echo</span> <span class="nb">true</span><span class="k">)</span>
<span class="nb">readonly</span> <span class="nv">command</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="c1"># To be sure $@ will only contain the parameters of the command, not the command itself.
</span><span class="nb">shift</span>
<span class="c1"># If we are already in a venv or if the are in docker, run directly.
</span><span class="k">if</span> <span class="o">[[</span> -v VIRTUAL_ENV <span class="o">||</span> <span class="s2">"</span><span class="si">${</span><span class="nv">INSIDE_DOCKER</span><span class="si">}</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"true"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">echo</span> <span class="s2">"Already in venv"</span>
<span class="si">${</span><span class="nv">command</span><span class="si">}</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="nb">exit</span> <span class="nv">$?</span>
<span class="k">else</span>
<span class="c1"># Load bashrc to be sure PATH is correctly set. We can't do this before this it could mess up with the enabled venv.
</span> <span class="c1"># We don't want to fail if the bashrc file contains unbound variables.
</span> <span class="nb">set</span> +u
<span class="nb">source</span> ~/.bashrc <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"Bash RC file not found"</span>
<span class="nb">set</span> -u
<span class="c1"># If not and pipenv is installed and we have a Pipfile, run with pipenv.
</span> <span class="k">if</span> <span class="nb">command</span> -v pipenv > /dev/null <span class="o">&&</span> <span class="o">[[</span> -f Pipfile <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">echo</span> <span class="s2">"Running in venv with pipenv"</span>
pipenv run <span class="s2">"</span><span class="si">${</span><span class="nv">command</span><span class="si">}</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="nb">exit</span> <span class="nv">$?</span>
<span class="c1"># If poetry is installed and we have a pyproject.toml, run with poetry.
</span> <span class="k">elif</span> <span class="nb">command</span> -v poetry > /dev/null <span class="o">&&</span> <span class="o">[[</span> -f pyproject.toml <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">echo</span> <span class="s2">"Running in venv with poetry"</span>
poetry run <span class="s2">"</span><span class="si">${</span><span class="nv">command</span><span class="si">}</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="k">else</span>
<span class="nb">echo</span> <span class="s2">"Not in an virtual env and don't know how to run in one."</span> ><span class="p">&</span><span class="m">2</span>
<span class="nb">exit</span> <span class="m">1</span>
<span class="k">fi</span>
<span class="k">fi</span>
</pre>
<p>Then, if you use <a class="reference external" href="https://pre-commit.com/">pre-commit</a>, you can do something like this:</p>
<pre class="code yaml literal-block">
<span class="p-Indicator">-</span> <span class="nt">id</span><span class="p">:</span> <span class="l-Scalar-Plain">flake8</span>
<span class="nt">name</span><span class="p">:</span> <span class="l-Scalar-Plain">flake8</span>
<span class="nt">entry</span><span class="p">:</span> <span class="l-Scalar-Plain">./bin/ensure-runs-in-venv.sh</span>
<span class="nt">exclude</span><span class="p">:</span> <span class="s">'/snapshots/.*\.py$'</span>
<span class="nt">args</span><span class="p">:</span> <span class="p-Indicator">[</span><span class="nv">flake8</span><span class="p-Indicator">]</span>
<span class="nt">language</span><span class="p">:</span> <span class="l-Scalar-Plain">system</span>
<span class="nt">types</span><span class="p">:</span> <span class="p-Indicator">[</span><span class="nv">python</span><span class="p-Indicator">]</span>
</pre>
</div>
ZSH tips2018-10-09T00:00:00+02:002018-10-09T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2018-10-09:/posts/en/2018/Oct/09/zbell/<div class="contents topic" id="sommaire">
<p class="topic-title">Sommaire</p>
<ul class="simple">
<li><a class="reference internal" href="#zbell" id="id1">ZBell</a></li>
</ul>
</div>
<div class="section" id="zbell">
<h2><a class="toc-backref" href="#id1">ZBell</a></h2>
<p>Useful to have a notification when a long command completes.</p>
<p>To enable it, add <tt class="docutils literal">zbell</tt> to your plugins array if you are using <a class="reference external" href="https://github.com/robbyrussell/oh-my-zsh">oh-my-zsh</a> or source the definition file.</p>
<p>To configure:</p>
<ul class="simple">
<li>The minimum time commands must take for the notification to happen, use: <tt class="docutils literal">ZBELL_DURATION</tt>. For instance <tt class="docutils literal">ZBELL_DURATION …</tt></li></ul></div><div class="contents topic" id="sommaire">
<p class="topic-title">Sommaire</p>
<ul class="simple">
<li><a class="reference internal" href="#zbell" id="id1">ZBell</a></li>
</ul>
</div>
<div class="section" id="zbell">
<h2><a class="toc-backref" href="#id1">ZBell</a></h2>
<p>Useful to have a notification when a long command completes.</p>
<p>To enable it, add <tt class="docutils literal">zbell</tt> to your plugins array if you are using <a class="reference external" href="https://github.com/robbyrussell/oh-my-zsh">oh-my-zsh</a> or source the definition file.</p>
<p>To configure:</p>
<ul class="simple">
<li>The minimum time commands must take for the notification to happen, use: <tt class="docutils literal">ZBELL_DURATION</tt>. For instance <tt class="docutils literal">ZBELL_DURATION=60</tt>.</li>
<li>The command zbell must ignore, use: <tt class="docutils literal">ZBELL_CMD_IGNORE</tt>. For instance: <tt class="docutils literal"><span class="pre">ZBELL_CMD_IGNORE=($EDITOR</span> $PAGER git nano less emacs bash ssh su tmux zsh "pipenv shell" "python manage.py shell_plus")</tt></li>
</ul>
<p>Here is the plugin. It was inspired by <a class="reference external" href="https://gist.github.com/jpouellet/5278239/">https://gist.github.com/jpouellet/5278239/</a> (I made changes to ignore commands made of multiple words, to report duration only once and to use <tt class="docutils literal"><span class="pre">notify-send</span></tt> to send the notification).</p>
<pre class="code zsh literal-block">
<span class="ch">#!/usr/bin/env zsh
</span>
<span class="c1"># From https://gist.github.com/jpouellet/5278239
</span>
<span class="c1"># This script prints a bell character when a command finishes
# if it has been running for longer than $zbell_duration seconds.
# If there are programs that you know run long that you don't
# want to bell after, then add them to $zbell_ignore.
#
# This script uses only zsh builtins so its fast, there's no needless
# forking, and its only dependency is zsh and its standard modules
#
# Written by Jean-Philippe Ouellet <jpo@vt.edu>
# Made available under the ISC license.
</span>
<span class="c1"># only do this if we're in an interactive shell
</span><span class="o">[[</span> -o interactive <span class="o">]]</span> <span class="o">||</span> <span class="k">return</span>
<span class="c1"># get $EPOCHSECONDS. builtins are faster than date(1)
</span>zmodload zsh/datetime <span class="o">||</span> <span class="k">return</span>
<span class="c1"># make sure we can register hooks
</span>autoload -Uz add-zsh-hook <span class="o">||</span> <span class="k">return</span>
<span class="c1"># initialize zbell_duration if not set
</span><span class="o">((</span> <span class="si">${</span><span class="p">+zbell_duration</span><span class="si">}</span> <span class="o">))</span> <span class="o">||</span> <span class="nv">zbell_duration</span><span class="o">=</span><span class="si">${</span><span class="nv">ZBELL_DURATION</span><span class="k">:-</span><span class="nv">15</span><span class="si">}</span>
<span class="c1"># initialize zbell_ignore if not set
</span><span class="k">if</span> <span class="o">[[</span> <span class="si">${#</span><span class="nv">ZBELL_CMD_IGNORE</span><span class="si">}</span> -gt <span class="m">0</span> <span class="o">]]</span> <span class="o">&&</span> <span class="o">((</span> ! <span class="si">${</span><span class="p">+zbell_ignore</span><span class="si">}</span> <span class="o">))</span><span class="p">;</span> <span class="k">then</span>
<span class="nv">zbell_ignore</span><span class="o">=()</span>
<span class="k">for</span> ignore_cmd in <span class="si">${</span><span class="nv">ZBELL_CMD_IGNORE</span><span class="p">[*]</span><span class="si">}</span><span class="p">;</span> <span class="k">do</span>
<span class="nv">zbell_ignore</span><span class="o">+=(</span><span class="s2">"</span><span class="si">${</span><span class="nv">ignore_cmd</span><span class="si">}</span><span class="s2">"</span><span class="o">)</span>
<span class="k">done</span>
<span class="k">fi</span>
<span class="o">((</span> <span class="si">${</span><span class="p">+zbell_ignore</span><span class="si">}</span> <span class="o">))</span> <span class="o">||</span> <span class="nv">zbell_ignore</span><span class="o">=(</span><span class="nv">$EDITOR</span> <span class="nv">$PAGER</span><span class="o">)</span>
<span class="c1"># initialize it because otherwise we compare a date and an empty string
# the first time we see the prompt. it's fine to have lastcmd empty on the
# initial run because it evaluates to an empty string, and splitting an
# empty string just results in an empty array.
</span><span class="nv">zbell_timestamp</span><span class="o">=</span><span class="nv">$EPOCHSECONDS</span>
<span class="c1"># right before we begin to execute something, store the time it started at
</span>zbell_begin<span class="o">()</span> <span class="o">{</span>
<span class="nv">zbell_timestamp</span><span class="o">=</span><span class="nv">$EPOCHSECONDS</span>
<span class="nv">zbell_lastcmd</span><span class="o">=</span><span class="nv">$1</span>
<span class="o">}</span>
<span class="c1"># when it finishes, if it's been running longer than $zbell_duration,
# and we dont have an ignored command in the line, then print a bell.
</span>zbell_end<span class="o">()</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">[[</span> -z <span class="s2">"</span><span class="si">${</span><span class="nv">zbell_lastcmd</span><span class="si">}</span><span class="s2">"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="k">return</span>
<span class="k">fi</span>
<span class="nv">time_run</span><span class="o">=</span><span class="k">$((</span> <span class="nv">$EPOCHSECONDS</span> <span class="o">-</span> <span class="nv">$zbell_timestamp</span> <span class="k">))</span>
<span class="nv">ran_long</span><span class="o">=</span><span class="k">$((</span> <span class="nv">$EPOCHSECONDS</span> <span class="o">-</span> <span class="nv">$zbell_timestamp</span> ><span class="o">=</span> <span class="nv">$zbell_duration</span> <span class="k">))</span>
<span class="nv">has_ignored_cmd</span><span class="o">=</span><span class="m">0</span>
<span class="k">for</span> ignore_cmd in <span class="si">${</span><span class="nv">zbell_ignore</span><span class="p">[*]</span><span class="si">}</span><span class="p">;</span> <span class="k">do</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="si">${</span><span class="nv">zbell_lastcmd</span><span class="si">}</span><span class="s2">"</span> <span class="o">==</span> *<span class="s2">"</span><span class="si">${</span><span class="nv">ignore_cmd</span><span class="si">}</span><span class="s2">"</span>* <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="nv">has_ignored_cmd</span><span class="o">=</span><span class="m">1</span>
<span class="nb">break</span>
<span class="k">fi</span>
<span class="k">done</span>
<span class="k">if</span> <span class="o">((</span> ! <span class="nv">$has_ignored_cmd</span> <span class="o">))</span> <span class="o">&&</span> <span class="o">((</span> ran_long <span class="o">))</span><span class="p">;</span> <span class="k">then</span>
notify-send <span class="s2">"'</span><span class="si">${</span><span class="nv">zbell_lastcmd</span><span class="si">}</span><span class="s2">' completed in </span><span class="si">${</span><span class="nv">time_run</span><span class="si">}</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="nv">zbell_lastcmd</span><span class="o">=</span><span class="s2">""</span>
<span class="o">}</span>
<span class="c1"># register the functions as hooks
</span>add-zsh-hook preexec zbell_begin
add-zsh-hook precmd zbell_end
</pre>
</div>
Bash tricks2018-10-03T00:00:00+02:002018-10-03T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2018-10-03:/posts/en/2018/Oct/03/bash/<div class="contents topic" id="sommaire">
<p class="topic-title">Sommaire</p>
<ul class="simple">
<li><a class="reference internal" href="#scripts" id="id1">Scripts</a></li>
<li><a class="reference internal" href="#signals" id="id2">Signals</a></li>
</ul>
</div>
<div class="section" id="scripts">
<h2><a class="toc-backref" href="#id1">Scripts</a></h2>
<p>Use this at the top of all your Bash scripts to avoid problems:</p>
<pre class="code bash literal-block">
<span class="c1"># Exit on error.
</span><span class="nb">set</span> -e
<span class="c1"># Don't allow undefined variable.
</span><span class="nb">set</span> -u
<span class="c1"># Make pipeline fail if any command in it fail.
</span><span class="nb">set</span> -o pipefail
</pre>
</div>
<div class="section" id="signals">
<h2><a class="toc-backref" href="#id2">Signals</a></h2>
<p>See <a class="reference external" href="//www.jujens.eu/posts/en/2015/Jan/09/utiliser-trap-bash/">this article</a>.</p>
</div>
Heroku tips2018-10-03T00:00:00+02:002018-10-03T00:00:00+02:00Julien Enselmetag:www.jujens.eu,2018-10-03:/posts/en/2018/Oct/03/heroku/<div class="section" id="use-an-editor">
<h2>Use an editor</h2>
<p>By default there is no editor on Heroku. To get one, you can use the <a class="reference external" href="https://github.com/Ehryk/heroku-nano">heroku-nano</a> plugin like this (or by installing as a plugin):</p>
<pre class="code bash literal-block">
mkdir bin
<span class="nb">cd</span> bin
curl https://github.com/Ehryk/heroku-nano/raw/master/heroku-nano-2.5.1/nano.tar.gz --location --silent > nano.tar …</pre></div><div class="section" id="use-an-editor">
<h2>Use an editor</h2>
<p>By default there is no editor on Heroku. To get one, you can use the <a class="reference external" href="https://github.com/Ehryk/heroku-nano">heroku-nano</a> plugin like this (or by installing as a plugin):</p>
<pre class="code bash literal-block">
mkdir bin
<span class="nb">cd</span> bin
curl https://github.com/Ehryk/heroku-nano/raw/master/heroku-nano-2.5.1/nano.tar.gz --location --silent > nano.tar.gz
tar -xvf nano.tar.gz
chmod +x nano
<span class="nb">cd</span> -
<span class="nb">export</span> <span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="nv">$PATH</span><span class="s2">:</span><span class="k">$(</span>realpath bin<span class="k">)</span><span class="s2">"</span>
</pre>
<p>Then use <tt class="docutils literal">nano README.md</tt>.</p>
</div>
<div class="section" id="papertail">
<h2>Papertail</h2>
<div class="section" id="download-logs">
<h3>Download logs</h3>
<p>This requires <a class="reference external" href="https://httpie.org/">httpie</a>. But you can adapt this to use curl instead.</p>
<pre class="code bash literal-block">
<span class="ch">#!/usr/bin/env bash
</span>
<span class="nb">set</span> -eu
<span class="c1"># Change this to start at another date.
</span><span class="nv">start_date</span><span class="o">=</span><span class="m">2019</span>-08-24
<span class="nv">nb_days_to_download</span><span class="o">=</span><span class="m">18</span>
<span class="c1"># Set your paper trail token here.
</span><span class="nv">papertrail_token</span><span class="o">=</span><span class="s1">''</span>
<span class="k">for</span> i in <span class="k">$(</span>seq <span class="m">0</span> <span class="si">${</span><span class="nv">nb_days_to_download</span><span class="si">}</span><span class="k">)</span><span class="p">;</span> <span class="k">do</span>
<span class="nv">log_date</span><span class="o">=</span><span class="k">$(</span>date -I -d <span class="s2">"</span><span class="si">${</span><span class="nv">start_date</span><span class="si">}</span><span class="s2"> +</span><span class="si">${</span><span class="nv">i</span><span class="si">}</span><span class="s2"> days"</span><span class="k">)</span>
mkdir -p <span class="s2">"</span><span class="si">${</span><span class="nv">log_date</span><span class="si">}</span><span class="s2">"</span>
<span class="nb">pushd</span> <span class="s2">"</span><span class="si">${</span><span class="nv">log_date</span><span class="si">}</span><span class="s2">"</span>
<span class="k">for</span> log_hour in <span class="k">$(</span>seq -f <span class="s2">"%02g"</span> <span class="m">0</span> <span class="m">23</span><span class="k">)</span><span class="p">;</span> <span class="k">do</span>
<span class="k">if</span> <span class="o">[[</span> ! -f <span class="s2">"</span><span class="si">${</span><span class="nv">log_date</span><span class="si">}</span><span class="s2">-</span><span class="si">${</span><span class="nv">log_hour</span><span class="si">}</span><span class="s2">.tsv.gz"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
http --follow <span class="s2">"https://papertrailapp.com/api/v1/archives/</span><span class="si">${</span><span class="nv">log_date</span><span class="si">}</span><span class="s2">-</span><span class="si">${</span><span class="nv">log_hour</span><span class="si">}</span><span class="s2">/download"</span> <span class="s2">"X-Papertrail-Token:</span><span class="si">${</span><span class="nv">papertrail_token</span><span class="si">}</span><span class="s2">"</span> > <span class="s2">"</span><span class="si">${</span><span class="nv">log_date</span><span class="si">}</span><span class="s2">-</span><span class="si">${</span><span class="nv">log_hour</span><span class="si">}</span><span class="s2">.tsv.gz"</span>
<span class="k">fi</span>
<span class="k">done</span>
<span class="nb">popd</span>
<span class="k">done</span>
</pre>
</div>
</div>
<div class="section" id="download-files">
<h2>Download files</h2>
<div class="section" id="from-google-drive">
<h3>From google drive</h3>
<p>Define this function:</p>
<pre class="code bash literal-block">
gdrive-download<span class="o">()</span> <span class="o">{</span>
<span class="nv">file_id</span><span class="o">=</span><span class="nv">$1</span>
<span class="nv">file</span><span class="o">=</span><span class="nv">$2</span>
<span class="nv">base_url</span><span class="o">=</span><span class="s2">"https://drive.google.com/uc?export=download&id=</span><span class="si">${</span><span class="nv">file_id</span><span class="si">}</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Downloading </span><span class="si">${</span><span class="nv">base_url</span><span class="si">}</span><span class="s2">"</span>
<span class="nv">CONFIRM</span><span class="o">=</span><span class="k">$(</span>wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate <span class="s2">"</span><span class="si">${</span><span class="nv">base_url</span><span class="si">}</span><span class="s2">"</span> -O- <span class="p">|</span> sed -rn <span class="s1">'s/.*confirm=([0-9A-Za-z_]+).*/\1/p'</span><span class="k">)</span>
wget --load-cookies /tmp/cookies.txt <span class="s2">"https://drive.google.com/uc?export=download&confirm=</span><span class="si">${</span><span class="nv">CONFIRM</span><span class="si">}</span><span class="s2">&id=</span><span class="si">${</span><span class="nv">file_id</span><span class="si">}</span><span class="s2">"</span> -O <span class="s2">"</span><span class="si">${</span><span class="nv">file</span><span class="si">}</span><span class="s2">"</span>
rm -rf /tmp/cookies.txt
<span class="o">}</span>
</pre>
<p>And use it like this: <tt class="docutils literal"><span class="pre">gdrive-download</span> FILE_ID_FROM_THE_SHARE_LINK DEST_FILENAME</tt>.</p>
</div>
</div>