<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>Andrea Veri's Blog - Planet GNOME</title><description>Andrea Veri's Blog</description><link>https://www.dragonsreach.it/categories/planet-gnome</link><atom:link href="https://www.dragonsreach.it/categories/planet-gnome/index.xml" rel="self" type="application/rss+xml"/><language>en</language><docs>https://cyber.harvard.edu/rss/rss.html#requiredChannelElements</docs><lastBuildDate>Tue, 05 May 2026 16:11:01 +0000</lastBuildDate><ttl>60</ttl><item><dc:creator>Andrea Veri</dc:creator><title>SELinux MCS challenges with GitLab Runners</title><description>&lt;h2 id="table-of-contents">Table of Contents&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="#introduction">Introduction&lt;/a>&lt;/li>
&lt;li>&lt;a href="#the-mcs-problem">The MCS problem&lt;/a>&lt;/li>
&lt;li>&lt;a href="#the-test-script">The test script&lt;/a>&lt;/li>
&lt;li>&lt;a href="#gitlabs-official-suggestion-and-why-it-falls-short">GitLab&amp;rsquo;s official suggestion and why it falls short&lt;/a>&lt;/li>
&lt;li>&lt;a href="#how-gnome-currently-handles-this">How GNOME currently handles this&lt;/a>&lt;/li>
&lt;li>&lt;a href="#exploring-libkrun">Exploring libkrun&lt;/a>&lt;/li>
&lt;li>&lt;a href="#firecracker-and-the-custom-executor-path">Firecracker and the custom executor path&lt;/a>&lt;/li>
&lt;li>&lt;a href="#what-comes-next">What comes next&lt;/a>&lt;/li>
&lt;/ul>
&lt;h2 id="introduction">Introduction&lt;/h2>
&lt;p>GNOME&amp;rsquo;s GitLab runners use Podman as the container runtime with SELinux in Enforcing mode on Fedora. The GitLab Runner Docker/Podman executor spawns multiple containers per job: a &lt;strong>helper&lt;/strong> container that clones the repository and handles artifacts, and a &lt;strong>build&lt;/strong> container that runs the actual CI script. Both containers need to share a &lt;code>/builds&lt;/code> volume — and this is where SELinux&amp;rsquo;s &lt;strong>Multi-Category Security (MCS)&lt;/strong> becomes a problem.&lt;/p>
&lt;h2 id="the-mcs-problem">The MCS problem&lt;/h2>
&lt;p>An SELinux label has four fields: &lt;code>user:role:type:level&lt;/code>. For containers the interesting part is the &lt;strong>level&lt;/strong>, also called the MCS field. A level looks like &lt;code>s0:c123,c456&lt;/code> — &lt;code>s0&lt;/code> is the sensitivity (always &lt;code>s0&lt;/code> in targeted policy), and &lt;code>c123,c456&lt;/code> are the &lt;strong>categories&lt;/strong>. A process or file can carry up to two categories.&lt;/p>
&lt;p>MCS access is based on &lt;strong>dominance&lt;/strong>. A subject&amp;rsquo;s label dominates an object&amp;rsquo;s label if the subject&amp;rsquo;s categories are a superset of (or equal to) the object&amp;rsquo;s categories:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Subject&lt;/th>
&lt;th>Object&lt;/th>
&lt;th>Access?&lt;/th>
&lt;th>Why&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>s0:c100,c200&lt;/code>&lt;/td>
&lt;td>&lt;code>s0:c100,c200&lt;/code>&lt;/td>
&lt;td>Yes&lt;/td>
&lt;td>Exact match&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>s0:c100,c200&lt;/code>&lt;/td>
&lt;td>&lt;code>s0:c100&lt;/code>&lt;/td>
&lt;td>Yes&lt;/td>
&lt;td>Subject&amp;rsquo;s categories are a superset&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>s0:c100,c200&lt;/code>&lt;/td>
&lt;td>&lt;code>s0:c100,c300&lt;/code>&lt;/td>
&lt;td>No&lt;/td>
&lt;td>Subject lacks &lt;code>c300&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>s0:c0.c1023&lt;/code>&lt;/td>
&lt;td>&lt;code>s0:c100,c200&lt;/code>&lt;/td>
&lt;td>Yes&lt;/td>
&lt;td>Full range dominates everything&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>s0&lt;/code>&lt;/td>
&lt;td>&lt;code>s0:c100,c200&lt;/code>&lt;/td>
&lt;td>No&lt;/td>
&lt;td>No categories can&amp;rsquo;t dominate any&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>s0&lt;/code>&lt;/td>
&lt;td>&lt;code>s0&lt;/code>&lt;/td>
&lt;td>Yes&lt;/td>
&lt;td>Both have no categories&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>How this applies to the runners:&lt;/p>
&lt;ul>
&lt;li>Container A runs as &lt;code>container_t:s0:c100,c100&lt;/code> — it can only access objects labeled &lt;code>s0:c100,c100&lt;/code> (or &lt;code>s0:c100&lt;/code>, or &lt;code>s0&lt;/code>)&lt;/li>
&lt;li>Container B runs as &lt;code>container_t:s0:c200,c200&lt;/code> — it can only access objects labeled &lt;code>s0:c200,c200&lt;/code> (or &lt;code>s0:c200&lt;/code>, or &lt;code>s0&lt;/code>)&lt;/li>
&lt;li>Container A cannot access Container B&amp;rsquo;s files — &lt;code>c100,c100&lt;/code> doesn&amp;rsquo;t dominate &lt;code>c200,c200&lt;/code>&lt;/li>
&lt;li>Overlay layers labeled &lt;code>s0&lt;/code> (no categories) — accessible by all containers since every category set dominates the empty set&lt;/li>
&lt;li>Podman at &lt;code>container_runtime_t:s0-s0:c0.c1023&lt;/code> — the full range means it dominates every possible category combination, so it can manage all containers&lt;/li>
&lt;/ul>
&lt;p>The &lt;strong>range syntax&lt;/strong> (&lt;code>s0-s0:c0.c1023&lt;/code>) is used for processes that need to operate across multiple levels. It means &amp;ldquo;my low clearance is &lt;code>s0&lt;/code> and my high clearance is &lt;code>s0:c0.c1023&lt;/code>.&amp;rdquo; The process can read objects at any level within that range and create objects at any level within it. This is why Podman needs the full range — it creates containers with different MCS labels and needs to access all of them.&lt;/p>
&lt;p>When Podman starts a container, it picks a random pair of categories (e.g., &lt;code>s0:c512,c768&lt;/code>) from within its allowed range and assigns that as the container&amp;rsquo;s process label. Files created by the container inherit that label. Another container gets a different random pair (e.g., &lt;code>s0:c33,c901&lt;/code>). Since &lt;code>c512,c768&lt;/code> and &lt;code>c33,c901&lt;/code> do not match — neither is a superset of the other — SELinux denies cross-container file access. This is the isolation mechanism, and the root cause of the problem with GitLab Runner&amp;rsquo;s multi-container-per-job architecture.&lt;/p>
&lt;p>The helper container gets one random MCS pair, writes the cloned repo to &lt;code>/builds&lt;/code> labeled with that pair, and the build container gets a different pair. The build container cannot read or write those files. The &lt;code>:Z&lt;/code> volume flag (exclusive relabel) relabels the volume to the mounting container&amp;rsquo;s category, but that only helps the first container — the second one still has a different label.&lt;/p>
&lt;h2 id="the-test-script">The test script&lt;/h2>
&lt;p>I wrote a script that demonstrates the problem with both standard containers (crun) and microVMs (libkrun). The script creates two containers per test — a helper that writes a file to a shared &lt;code>/builds&lt;/code> volume, and a build container that tries to read it — simulating the GitLab Runner workflow:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># Description: SELinux MCS Diagnostic (crun vs krun)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> &lt;span class="o">[&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>getenforce&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> !&lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;Enforcing&amp;#34;&lt;/span> &lt;span class="o">]&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;WARNING: SELinux is not in Enforcing mode. This test requires Enforcing mode.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exit&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">TEST_BASE&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;/tmp/gitlab-runner-mcs-test&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">CRUN_DIR&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$TEST_BASE&lt;/span>&lt;span class="s2">/crun-builds&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">KRUN_DIR&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$TEST_BASE&lt;/span>&lt;span class="s2">/krun-builds&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Cleanup from previous runs&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">rm -rf &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$TEST_BASE&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">mkdir -p &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$CRUN_DIR&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$KRUN_DIR&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;=======================================================&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34; TEST 1: Standard Container Isolation (crun)&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;=======================================================&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 1. CREATE Helper&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman create --name crun-helper -v &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$CRUN_DIR&lt;/span>&lt;span class="s2">:/builds:Z&amp;#34;&lt;/span> fedora bash -c &lt;span class="s2">&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> echo &amp;#39;[crun] -&amp;gt; Helper Process Context (Inside):&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> cat /proc/self/attr/current
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> echo &amp;#39;crun-data&amp;#39; &amp;gt; /builds/artifact.txt
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> echo &amp;#39;[crun] -&amp;gt; File Label INSIDE Helper:&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> ls -Z /builds/artifact.txt
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt; /dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;[crun] Starting Helper Container (applying :Z relabel)...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">HELPER_HOST_LABEL_CRUN&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>podman inspect -f &lt;span class="s1">&amp;#39;{{.ProcessLabel}}&amp;#39;&lt;/span> crun-helper&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;[crun] -&amp;gt; HOST METADATA: Podman assigned process label: &lt;/span>&lt;span class="nv">$HELPER_HOST_LABEL_CRUN&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman start -a crun-helper
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;[crun] -&amp;gt; File Label ON HOST (Notice the specific MCS category):&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ls -Z &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$CRUN_DIR&lt;/span>&lt;span class="s2">/artifact.txt&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 2. CREATE Build Container (The Victim)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman create --name crun-build -v &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$CRUN_DIR&lt;/span>&lt;span class="s2">:/builds&amp;#34;&lt;/span> fedora bash -c &lt;span class="s2">&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> echo &amp;#39; [Build-Internal] Process Context:&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> cat /proc/self/attr/current 2&amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> echo &amp;#39; [Build-Internal] Executing ls -laZ /builds :&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> ls -laZ /builds 2&amp;gt;&amp;amp;1 | sed &amp;#39;s/^/ /&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> echo &amp;#39; [Build-Internal] Executing cat /builds/artifact.txt :&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> cat /builds/artifact.txt 2&amp;gt;&amp;amp;1 | sed &amp;#39;s/^/ /&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt; /dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;[crun] Starting Build Container to inspect shared volume...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">BUILD_HOST_LABEL_CRUN&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>podman inspect -f &lt;span class="s1">&amp;#39;{{.ProcessLabel}}&amp;#39;&lt;/span> crun-build&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;[crun] -&amp;gt; HOST METADATA: Podman assigned process label: &lt;/span>&lt;span class="nv">$BUILD_HOST_LABEL_CRUN&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman start -a crun-build
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman rm -f crun-helper crun-build &amp;gt; /dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;=======================================================&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34; TEST 2: MicroVM Isolation (libkrun / virtio-fs) FIXED&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;=======================================================&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># --- Write the execution scripts to the host to avoid parsing errors ---&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cat &lt;span class="s">&amp;lt;&amp;lt; &amp;#39;EOF&amp;#39; &amp;gt; &amp;#34;$TEST_BASE/krun_helper.sh&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">echo &amp;#39;[krun] -&amp;gt; Helper Process Context (Inside VM):&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">cat /proc/self/attr/current 2&amp;gt;/dev/null || echo &amp;#39; (SELinux disabled/unavailable in guest kernel)&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">echo &amp;#39;krun-data&amp;#39; &amp;gt; /builds/artifact.txt
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">echo &amp;#39;[krun] -&amp;gt; File Label INSIDE Helper VM (Blindspot):&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">ls -laZ /builds/artifact.txt 2&amp;gt;&amp;amp;1 | sed &amp;#39;s/^/ /&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cat &lt;span class="s">&amp;lt;&amp;lt; &amp;#39;EOF&amp;#39; &amp;gt; &amp;#34;$TEST_BASE/krun_build.sh&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">echo &amp;#39; [Build-Internal] Process Context (Inside VM):&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">cat /proc/self/attr/current 2&amp;gt;/dev/null || echo &amp;#39; (SELinux disabled/unavailable in guest kernel)&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">echo &amp;#39; [Build-Internal] Executing ls -laZ /builds :&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">ls -laZ /builds 2&amp;gt;&amp;amp;1 | sed &amp;#39;s/^/ /&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">echo &amp;#39; [Build-Internal] Executing cat /builds/artifact.txt :&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">cat /builds/artifact.txt 2&amp;gt;&amp;amp;1 | sed &amp;#39;s/^/ /&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">chmod +x &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$TEST_BASE&lt;/span>&lt;span class="s2">/krun_helper.sh&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$TEST_BASE&lt;/span>&lt;span class="s2">/krun_build.sh&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ---------------------------------------------------------------------&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 1. CREATE Helper MicroVM&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman create --name krun-helper --runtime krun --memory&lt;span class="o">=&lt;/span>1024m &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -v &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$KRUN_DIR&lt;/span>&lt;span class="s2">:/builds:Z&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -v &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$TEST_BASE&lt;/span>&lt;span class="s2">/krun_helper.sh:/script.sh:ro,Z&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> fedora /script.sh &amp;gt; /dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;[krun] Starting Helper MicroVM (applying :Z relabel)...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">HELPER_HOST_LABEL_KRUN&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>podman inspect -f &lt;span class="s1">&amp;#39;{{.ProcessLabel}}&amp;#39;&lt;/span> krun-helper&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;[krun] -&amp;gt; HOST METADATA: Podman assigned process label: &lt;/span>&lt;span class="nv">$HELPER_HOST_LABEL_KRUN&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman start -a krun-helper
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;[krun] -&amp;gt; File Label ON HOST (Podman applied the helper&amp;#39;s MCS category via :Z):&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ls -Z &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$KRUN_DIR&lt;/span>&lt;span class="s2">/artifact.txt&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 2. CREATE Build MicroVM (The Victim)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman create --name krun-build --runtime krun --memory&lt;span class="o">=&lt;/span>1024m &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -v &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$KRUN_DIR&lt;/span>&lt;span class="s2">:/builds&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -v &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$TEST_BASE&lt;/span>&lt;span class="s2">/krun_build.sh:/script.sh:ro,Z&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> fedora /script.sh &amp;gt; /dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;[krun] Starting Build MicroVM to inspect shared volume...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">BUILD_HOST_LABEL_KRUN&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>podman inspect -f &lt;span class="s1">&amp;#39;{{.ProcessLabel}}&amp;#39;&lt;/span> krun-build&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;[krun] -&amp;gt; HOST METADATA: Podman assigned process label: &lt;/span>&lt;span class="nv">$BUILD_HOST_LABEL_KRUN&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34; *** THE virtiofsd DAEMON ON THE HOST IS TRAPPED IN THIS CONTEXT ***&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman start -a krun-build
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Cleanup&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman rm -f krun-helper krun-build &amp;gt; /dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;=======================================================&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34; Test Complete.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Test 1 (crun)&lt;/strong> creates a helper container that mounts the builds directory with &lt;code>:Z&lt;/code> (exclusive relabel) and writes &lt;code>artifact.txt&lt;/code>. Podman assigns it a random MCS label — in this run it was &lt;code>s0:c20,c540&lt;/code>. The file on disk inherits that label. Then a second container (the build container) mounts the same path without &lt;code>:Z&lt;/code> and gets a different random label (&lt;code>s0:c46,c331&lt;/code>). Since &lt;code>c46,c331&lt;/code> does not dominate &lt;code>c20,c540&lt;/code>, the build container is denied access to the file.&lt;/p>
&lt;p>&lt;strong>Test 2 (krun)&lt;/strong> runs the same scenario but with &lt;code>--runtime krun&lt;/code>, which boots each container inside a lightweight microVM via &lt;a href="https://github.com/containers/libkrun">libkrun&lt;/a>. The helper VM gets &lt;code>container_kvm_t:s0:c823,c999&lt;/code> and the build VM gets &lt;code>container_kvm_t:s0:c309,c405&lt;/code> — same MCS mismatch, same denial. The type changes from &lt;code>container_t&lt;/code> to &lt;code>container_kvm_t&lt;/code>, but the MCS mechanism is identical. On the host side, &lt;code>virtiofsd&lt;/code> — the daemon that serves the volume into the VM via virtio-fs — runs under the MCS label Podman assigned to the VM. The build VM&amp;rsquo;s &lt;code>virtiofsd&lt;/code> is trapped in &lt;code>s0:c309,c405&lt;/code> and cannot access files labeled &lt;code>s0:c823,c999&lt;/code>.&lt;/p>
&lt;p>An interesting detail: inside the libkrun VMs, &lt;code>cat /proc/self/attr/current&lt;/code> returns just &lt;code>kernel&lt;/code> — SELinux is not available in the guest. The VM thinks it has no mandatory access control, but the host-side &lt;code>virtiofsd&lt;/code> is still fully subject to MCS enforcement. This is a blindspot worth being aware of.&lt;/p>
&lt;p>The output from a run on Fedora with SELinux Enforcing and Podman 5.8.2:&lt;/p>
&lt;pre tabindex="0">&lt;code>=======================================================
TEST 1: Standard Container Isolation (crun)
=======================================================
[crun] Starting Helper Container (applying :Z relabel)...
[crun] -&amp;gt; HOST METADATA: Podman assigned process label: system_u:system_r:container_t:s0:c20,c540
[crun] -&amp;gt; Helper Process Context (Inside):
system_u:system_r:container_t:s0:c20,c540 [crun] -&amp;gt; File Label INSIDE Helper:
system_u:object_r:container_file_t:s0:c20,c540 /builds/artifact.txt
[crun] -&amp;gt; File Label ON HOST (Notice the specific MCS category):
system_u:object_r:container_file_t:s0:c20,c540 /tmp/gitlab-runner-mcs-test/crun-builds/artifact.txt
[crun] Starting Build Container to inspect shared volume...
[crun] -&amp;gt; HOST METADATA: Podman assigned process label: system_u:system_r:container_t:s0:c46,c331
*** COMPARE THE cXXX,cYYY ABOVE TO THE FILE LABEL. THIS MISMATCH CAUSES THE DENIAL ***
[Build-Internal] Process Context:
system_u:system_r:container_t:s0:c46,c331 [Build-Internal] Executing ls -laZ /builds :
ls: cannot open directory &amp;#39;/builds&amp;#39;: Permission denied
[Build-Internal] Executing cat /builds/artifact.txt :
cat: /builds/artifact.txt: Permission denied
=======================================================
TEST 2: MicroVM Isolation (libkrun / virtio-fs) FIXED
=======================================================
[krun] Starting Helper MicroVM (applying :Z relabel)...
[krun] -&amp;gt; HOST METADATA: Podman assigned process label: system_u:system_r:container_kvm_t:s0:c823,c999
[krun] -&amp;gt; Helper Process Context (Inside VM):
kernel [krun] -&amp;gt; File Label INSIDE Helper VM (Blindspot):
-rw-r--r--. 1 root root system_u:object_r:container_file_t:s0:c823,c999 10 May 2 2026 /builds/artifact.txt
[krun] -&amp;gt; File Label ON HOST (Podman applied the helper&amp;#39;s MCS category via :Z):
system_u:object_r:container_file_t:s0:c823,c999 /tmp/gitlab-runner-mcs-test/krun-builds/artifact.txt
[krun] Starting Build MicroVM to inspect shared volume...
[krun] -&amp;gt; HOST METADATA: Podman assigned process label: system_u:system_r:container_kvm_t:s0:c309,c405
*** THE virtiofsd DAEMON ON THE HOST IS TRAPPED IN THIS CONTEXT ***
[Build-Internal] Process Context (Inside VM):
kernel [Build-Internal] Executing ls -laZ /builds :
ls: /builds: Permission denied
ls: cannot open directory &amp;#39;/builds&amp;#39;: Permission denied
[Build-Internal] Executing cat /builds/artifact.txt :
cat: /builds/artifact.txt: Permission denied
=======================================================
Test Complete.
&lt;/code>&lt;/pre>&lt;h2 id="gitlabs-official-suggestion-and-why-it-falls-short">GitLab&amp;rsquo;s official suggestion and why it falls short&lt;/h2>
&lt;p>GitLab&amp;rsquo;s documentation on &lt;a href="https://docs.gitlab.com/runner/executors/docker/#configure-selinux-mcs">configuring SELinux MCS&lt;/a> suggests applying the same MCS label to all containers launched by a runner:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-toml" data-lang="toml">&lt;span class="line">&lt;span class="cl">&lt;span class="p">[[&lt;/span>&lt;span class="nx">runners&lt;/span>&lt;span class="p">]]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">[&lt;/span>&lt;span class="nx">runners&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">docker&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">security_opt&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;label=level:s0:c1000,c1000&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This works — all containers get the same category pair, so the helper and build containers can share files. But it collapses MCS isolation between all concurrent jobs on that runner. With &lt;code>concurrent = 4&lt;/code>, four simultaneous jobs all run as &lt;code>s0:c1000,c1000&lt;/code> and can read each other&amp;rsquo;s &lt;code>/builds&lt;/code> content — cloned source code, build artifacts, cached dependencies. On a shared or multi-tenant runner, this is a security regression: it trades MCS isolation for functionality.&lt;/p>
&lt;p>For runners with &lt;code>concurrent = 1&lt;/code> or dedicated single-tenant runners this is an acceptable tradeoff, but it does not generalize to shared infrastructure where multiple untrusted projects run side by side.&lt;/p>
&lt;h2 id="how-gnome-currently-handles-this">How GNOME currently handles this&lt;/h2>
&lt;p>GNOME&amp;rsquo;s runners are managed via an &lt;a href="https://gitlab.gnome.org/GNOME/ansible/-/tree/master/roles/gitlab-runner">Ansible role&lt;/a> that enforces SELinux in Enforcing mode, installs rootless Podman running as a dedicated &lt;code>podman&lt;/code> system user with linger enabled, and deploys custom SELinux policy modules. The Podman service runs under &lt;code>SELinuxContext=system_u:system_r:container_runtime_t:s0-s0:c0.c1023&lt;/code> via a systemd override — the full MCS range (&lt;code>s0-s0:c0.c1023&lt;/code>) gives the container runtime the ability to spawn containers at any MCS level and relabel volumes accordingly, as explained in the dominance rules above.&lt;/p>
&lt;p>Four custom SELinux &lt;code>.te&lt;/code> modules are compiled and loaded on every runner host: &lt;code>pydocuum&lt;/code> (allows the image cleanup daemon to talk to the Podman socket), &lt;code>podman&lt;/code> (grants &lt;code>user_namespace create&lt;/code> and &lt;code>/dev/null&lt;/code> mapping), &lt;code>flatpak&lt;/code> (permits the filesystem mounts flatpak builds need), and &lt;code>gnome_runner&lt;/code> (covers &lt;code>binfmt_misc&lt;/code> access, device nodes, and other permissions GNOME OS builds require).&lt;/p>
&lt;p>For the MCS problem specifically, the runner &lt;code>config.toml&lt;/code> — rendered from a Jinja2 template via per-host Ansible variables — sets a fixed MCS label per runner type. Here&amp;rsquo;s a representative snippet from one of the runner hosts:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-toml" data-lang="toml">&lt;span class="line">&lt;span class="cl">&lt;span class="p">[[&lt;/span>&lt;span class="nx">runners&lt;/span>&lt;span class="p">]]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">name&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s2">&amp;#34;a15948139c78&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">executor&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s2">&amp;#34;docker&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">[&lt;/span>&lt;span class="nx">runners&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">docker&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">image&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s2">&amp;#34;quay.io/fedora/fedora:latest&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">privileged&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="kc">false&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">security_opt&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;label=level:s0:c100,c100&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">devices&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;/dev/kvm&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s2">&amp;#34;/dev/udmabuf&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">cap_add&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;SYS_PTRACE&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s2">&amp;#34;SYS_CHROOT&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">[[&lt;/span>&lt;span class="nx">runners&lt;/span>&lt;span class="p">]]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">name&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s2">&amp;#34;a15948139c78-flatpak&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">executor&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s2">&amp;#34;docker&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">[&lt;/span>&lt;span class="nx">runners&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">docker&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">image&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s2">&amp;#34;quay.io/gnome_infrastructure/gnome-runtime-images:gnome-master&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">privileged&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="kc">false&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">security_opt&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;seccomp:/home/podman/gitlab-runner/flatpak.seccomp.json&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s2">&amp;#34;label=level:s0:c200,c200&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">cap_drop&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;all&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This is the same approach GitLab&amp;rsquo;s documentation suggests, with one refinement: we use &lt;strong>different fixed categories per runner type&lt;/strong> — &lt;code>c100,c100&lt;/code> for untagged runners and &lt;code>c200,c200&lt;/code> for flatpak runners — so that flatpak builds and regular builds remain MCS-isolated from each other, even though builds of the same type share a category.&lt;/p>
&lt;p>This is a pragmatic compromise, not an ideal solution. All concurrent jobs on the same runner type share the same MCS category. With &lt;code>concurrent: 4&lt;/code> on our Hetzner runners, four simultaneous untagged jobs can read each other&amp;rsquo;s &lt;code>/builds&lt;/code> content. For GNOME&amp;rsquo;s use case — a community CI infrastructure where the runners are shared by GNOME project maintainers — this is an acceptable tradeoff. The alternative, leaving MCS labels random, would break every single job. But it is precisely this tradeoff that motivates exploring per-job VM isolation via microVMs.&lt;/p>
&lt;h2 id="exploring-libkrun">Exploring libkrun&lt;/h2>
&lt;p>&lt;a href="https://github.com/containers/libkrun">libkrun&lt;/a> is a lightweight Virtual Machine Monitor (VMM) that integrates with Podman via &lt;code>--runtime krun&lt;/code>, running each container inside a microVM with its own lightweight kernel. The appeal is strong: per-container VM isolation would give each job its own kernel and address space, making the MCS cross-container problem irrelevant inside the VM.&lt;/p>
&lt;p>I tested libkrun on a Fedora system and hit an immediate blocker: &lt;code>Fatal glibc error: rseq registration failed&lt;/code>. The &lt;strong>rseq&lt;/strong> (Restartable Sequences) syscall was introduced in Linux kernel 5.3 and is required by glibc &amp;gt;= 2.35. libkrun uses a custom minimal kernel that does not expose &lt;code>rseq&lt;/code> support. Since the guest images — Fedora in our case — ship modern glibc that expects &lt;code>rseq&lt;/code> to be available, the process aborts at startup before any user code runs.&lt;/p>
&lt;p>The libkrun kernel is compiled into the library itself and cannot be modified or replaced by the user. This is not a configuration issue but a fundamental limitation of the current libkrun release.&lt;/p>
&lt;p>Even if the &lt;code>rseq&lt;/code> issue were resolved, the MCS challenge would still be there — as the test script demonstrates in Test 2. On the host side, Podman assigns MCS labels to the &lt;code>virtiofsd&lt;/code> process that serves the volume into the VM via virtio-fs. Different VMs get different host-side MCS labels, meaning the same &lt;code>:Z&lt;/code> relabel / cross-container access denial applies. The mechanism changes from overlay mounts to virtio-fs, but the SELinux enforcement is identical: &lt;code>virtiofsd&lt;/code> for the build VM runs at &lt;code>container_kvm_t:s0:c309,c405&lt;/code> and cannot access files labeled &lt;code>s0:c823,c999&lt;/code> by the helper VM&amp;rsquo;s &lt;code>virtiofsd&lt;/code>.&lt;/p>
&lt;h2 id="firecracker-and-the-custom-executor-path">Firecracker and the custom executor path&lt;/h2>
&lt;p>&lt;a href="https://firecracker-microvm.github.io/">Firecracker&lt;/a> is another microVM technology, the one behind AWS Lambda and Fly.io, that could provide strong per-job isolation. However, there is no native GitLab Runner executor for Firecracker. The only integration path is the &lt;a href="https://docs.gitlab.com/runner/executors/custom.html">Custom Executor&lt;/a>, which requires implementing &lt;code>prepare&lt;/code>, &lt;code>run&lt;/code>, and &lt;code>cleanup&lt;/code> scripts from scratch.&lt;/p>
&lt;p>The job image is exposed via &lt;code>CUSTOM_ENV_CI_JOB_IMAGE&lt;/code>, but everything else is on the operator: pulling the OCI image, extracting a rootfs, booting a Firecracker VM with the right kernel and network configuration, injecting the build script, mounting or copying the cloned repository into the VM, collecting artifacts and cache after the job finishes, and tearing the VM down. GitLab provides an &lt;a href="https://docs.gitlab.com/runner/executors/custom_examples/lxd/">LXD-based example&lt;/a> that shows the pattern — &lt;code>prepare&lt;/code> creates a container and installs dependencies, &lt;code>run&lt;/code> pipes the job script into it, &lt;code>cleanup&lt;/code> destroys it — but adapting that to microVMs adds the complexity of VM lifecycle management, kernel and rootfs preparation, networking, and storage. This is a significant engineering effort, essentially rebuilding the entire Docker executor workflow from scratch.&lt;/p>
&lt;h2 id="what-comes-next">What comes next&lt;/h2>
&lt;p>MCS is a core SELinux feature. Type enforcement (TE) already confines processes by type — &lt;code>container_t&lt;/code> can only access &lt;code>container_file_t&lt;/code>, not &lt;code>user_home_t&lt;/code> or &lt;code>httpd_sys_content_t&lt;/code> — but TE alone cannot distinguish one &lt;code>container_t&lt;/code> process from another. MCS adds that layer: by assigning each container a unique category pair, the kernel enforces isolation &lt;em>between&lt;/em> processes that share the same type. Container A at &lt;code>s0:c100,c100&lt;/code> and Container B at &lt;code>s0:c200,c200&lt;/code> are both &lt;code>container_t&lt;/code>, but MCS ensures they cannot touch each other&amp;rsquo;s files. The conflict with GitLab Runner&amp;rsquo;s multi-container-per-job architecture is that two containers that &lt;em>need&lt;/em> to share a volume are given different categories by default. The workarounds we deploy today, including the fixed MCS labels on GNOME&amp;rsquo;s runners, trade that inter-container isolation for functionality.&lt;/p>
&lt;p>The most promising direction I&amp;rsquo;ve found so far is the combination of &lt;a href="https://www.cloudhypervisor.org/">Cloud Hypervisor&lt;/a> and the &lt;a href="https://github.com/helmholtzcloud/fleeting-plugin-fleetingd">fleeting-plugin-fleetingd&lt;/a> plugin. Cloud Hypervisor is built on Intel&amp;rsquo;s Rust-VMM crate and is essentially a more capable sibling of Firecracker — it supports CPU and memory hotplugging, VFIO device passthrough, and virtio-fs, features that are often necessary for complex CI tasks like building large binaries or running UI tests and that Firecracker&amp;rsquo;s minimalist design deliberately omits. The fleeting-plugin-fleetingd is a community plugin for GitLab&amp;rsquo;s &lt;strong>Instance Executor&lt;/strong> (the modern evolution of the Custom Executor) that automates the full VM lifecycle: downloading cloud images, creating Copy-on-Write disks, launching Cloud Hypervisor VMs with direct kernel boot, provisioning them via cloud-init, and tearing them down after each build. Each job gets a fresh disposable VM, which is exactly the per-job isolation model we need. The plugin already handles networking via TAP interfaces and nftables SNAT, and supports customization of the VM image through cloud-init commands — so preinstalling Podman or other build tools is straightforward.&lt;/p>
&lt;p>Beyond that, I&amp;rsquo;ll also keep evaluating libkrun (promising Red Hat technology), Firecracker with a hand-rolled custom executor, and QEMU&amp;rsquo;s microvm machine type. The common denominator across all of these — except for the fleeting-plugin-fleetingd path — is that none of them have an existing GitLab Runner integration. Regardless of which microVM technology we settle on, the path forward involves either building a workflow from scratch using the &lt;a href="https://docs.gitlab.com/runner/executors/custom.html">Custom Executor&lt;/a> and its &lt;code>prepare&lt;/code>, &lt;code>run&lt;/code>, &lt;code>cleanup&lt;/code> hooks, or leveraging the fleeting plugin ecosystem that GitLab has been building around the Instance and Docker Autoscaler executors.&lt;/p>
&lt;h3 id="cve-2026-31431">CVE-2026-31431&lt;/h3>
&lt;p>The urgency of per-job VM isolation was underscored by &lt;a href="https://www.helpnetsecurity.com/2026/04/30/copyfail-linux-lpe-vulnerability-cve-2026-31431/">CVE-2026-31431&lt;/a> (&amp;ldquo;Copy Fail&amp;rdquo;), a nine-year-old logic bug in the kernel&amp;rsquo;s &lt;code>algif_aead&lt;/code> cryptographic module disclosed at the end of April. The flaw lets an unprivileged local user write four controlled bytes into the page cache of any readable file — enough to patch a setuid binary like &lt;code>/usr/bin/su&lt;/code> and escalate to root. Unlike Dirty Cow or Dirty Pipe, Copy Fail requires no race condition: the exploit is deterministic, leaves no trace on disk, and — critically — can break out of container isolation. In a shared-runner CI environment, any project that can execute arbitrary code in a job already has exactly the access the exploit needs. Separately, &lt;a href="https://www.computing.co.uk/analysis/2026/claude-mythos-how-ai-broke-out-of-its-sandbox">Claude Mythos&lt;/a> — an Anthropic model trained for cybersecurity research that escaped its own sandbox during a red-team exercise in April — demonstrated that AI-assisted vulnerability discovery and exploitation is no longer theoretical; models can now autonomously find and chain bugs that would take human researchers weeks to exploit. The combination of a reliable, public kernel LPE and AI-augmented offensive tooling makes the case for ephemeral microVMs compelling: when every CI job boots a fresh, disposable VM with its own kernel, a vulnerability like Copy Fail becomes a local-root inside a throwaway guest that is destroyed seconds later, not a stepping stone to the host or adjacent jobs.&lt;/p>
&lt;p>That should be all for today, stay tuned!&lt;/p></description><link>https://www.dragonsreach.it/2026/05/01/selinux-mcs-challenges-gitlab-runners/</link><guid>https://www.dragonsreach.it/2026/05/01/selinux-mcs-challenges-gitlab-runners/</guid><pubDate>Fri, 01 May 2026 21:00:00 -0400</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>GNOME GitLab Git traffic caching</title><description>&lt;h2 id="table-of-contents">Table of Contents&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="#introduction">Introduction&lt;/a>&lt;/li>
&lt;li>&lt;a href="#the-problem">The problem&lt;/a>&lt;/li>
&lt;li>&lt;a href="#architecture-overview">Architecture overview&lt;/a>&lt;/li>
&lt;li>&lt;a href="#the-vcl-layer">The VCL layer&lt;/a>&lt;/li>
&lt;li>&lt;a href="#the-post-to-get-conversion">The POST-to-GET conversion&lt;/a>&lt;/li>
&lt;li>&lt;a href="#protecting-private-repositories">Protecting private repositories&lt;/a>&lt;/li>
&lt;li>&lt;a href="#the-lua-layer">The Lua layer&lt;/a>&lt;/li>
&lt;li>&lt;a href="#debugging-the-rollout">Debugging the rollout&lt;/a>&lt;/li>
&lt;li>&lt;a href="#how-we-got-here">How we got here&lt;/a>&lt;/li>
&lt;li>&lt;a href="#conclusions">Conclusions&lt;/a>&lt;/li>
&lt;/ul>
&lt;h2 id="introduction">Introduction&lt;/h2>
&lt;p>One of the most visible signs that GNOME&amp;rsquo;s infrastructure has grown over the years is the amount of CI traffic that flows through &lt;a href="https://gitlab.gnome.org">gitlab.gnome.org&lt;/a> on any given day. Hundreds of pipelines run in parallel, most of them starting with a &lt;code>git clone&lt;/code> or &lt;code>git fetch&lt;/code> of the same repository, often at the same commit. All that traffic was landing directly on GitLab&amp;rsquo;s webservice pods, generating redundant load for work that was essentially identical.&lt;/p>
&lt;p>GNOME&amp;rsquo;s infrastructure runs on AWS, which generously provides credits to the project. Even so, data transfer is one of the largest cost drivers we face, and we have to operate within a defined budget regardless of those credits. The bandwidth costs associated with this Git traffic grew significant enough that for a period of time we redirected unauthenticated HTTPS Git pulls to our GitHub mirrors as a short-term cost mitigation. That measure bought us some breathing room, but it was never meant to be permanent: sending users to a third-party platform for what is essentially a core infrastructure operation is not a position we wanted to stay in. The goal was always to find a proper solution on our own infrastructure.&lt;/p>
&lt;p>This post documents the caching layer we built to address that problem. The solution sits between the client and GitLab, intercepts Git fetch traffic, and routes it through Fastly&amp;rsquo;s CDN so that repeated fetches of the same content are served from cache rather than generating a fresh pack every time. The design went through several iterations — this post presents the final architecture first, then walks through &lt;a href="#how-we-got-here">how we got here&lt;/a> for readers interested in the evolution.&lt;/p>
&lt;h2 id="the-problem">The problem&lt;/h2>
&lt;p>The Git smart HTTP protocol uses two endpoints: &lt;code>info/refs&lt;/code> for capability advertisement and ref discovery, and &lt;code>git-upload-pack&lt;/code> for the actual pack generation. The second one is the expensive one. When a CI job runs &lt;code>git fetch origin main&lt;/code>, GitLab has to compute and send the entire pack for that fetch negotiation. If ten jobs run the same fetch within a short window, GitLab does that work ten times.&lt;/p>
&lt;p>The tricky part is that &lt;code>git-upload-pack&lt;/code> is a &lt;code>POST&lt;/code> request with a binary body that encodes what the client already has (&lt;code>have&lt;/code> lines) and what it wants (&lt;code>want&lt;/code> lines). Traditional HTTP caches ignore POST bodies entirely. Building a cache that actually understands those bodies and deduplicates identical fetches requires some work at the edge.&lt;/p>
&lt;p>For a fresh clone the body contains only &lt;code>want&lt;/code> lines — one per ref the client is requesting:&lt;/p>
&lt;pre tabindex="0">&lt;code>0032want 7d20e995c3c98644eb1c58a136628b12e9f00a78
0032want 93e944c9f728a4b9da506e622592e4e3688a805c
0032want ef2cbad5843a607236b45e5f50fa4318e0580e04
...
&lt;/code>&lt;/pre>&lt;p>For an incremental fetch the body is a mix of &lt;code>want&lt;/code> lines (what the client needs) and &lt;code>have&lt;/code> lines (commits the client already has locally), which the server uses to compute the smallest possible packfile delta:&lt;/p>
&lt;pre tabindex="0">&lt;code>00a4want 51a117587524cbdd59e43567e6cbd5a76e6a39ff
0000
0032have 8282cff4b31dce12e100d4d6c78d30b1f4689dd3
0032have be83e3dae8265fdc4c91f11d5778b20ceb4e2479
0032have 7d46abdf9c5a3f119f645c8de6d87efffe3889b8
...
&lt;/code>&lt;/pre>&lt;p>The leading four hex characters on each line are the pkt-line length prefix. The server walks back through history from the wanted commits until it finds a common ancestor with the &lt;code>have&lt;/code> set, then packages everything in between into a packfile. Two CI jobs running the same pipeline at the same commit will produce byte-for-byte identical request bodies and therefore identical responses — exactly the property a cache can help with.&lt;/p>
&lt;h2 id="architecture-overview">Architecture overview&lt;/h2>
&lt;p>The current architecture has three components:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Fastly&lt;/strong> as the user-facing CDN for &lt;code>gitlab.gnome.org&lt;/code>, with custom VCL that intercepts &lt;code>git-upload-pack&lt;/code> traffic, hashes the request body, converts the POST to a GET, and caches the response at edge POPs worldwide&lt;/li>
&lt;li>&lt;strong>OpenResty&lt;/strong> (Nginx + LuaJIT) running as the origin server, with a Lua script that restores the original POST, checks a Valkey denylist for private repositories, and signals cacheability back to Fastly&lt;/li>
&lt;li>&lt;strong>Valkey + webhook&lt;/strong> — a small &lt;a href="https://valkey.io/">Valkey&lt;/a> instance stores a denylist of private repository paths, kept in sync by a &lt;a href="https://gitlab.gnome.org/GNOME/gitlab-git-cache-webhook">webhook service&lt;/a> that listens for GitLab project visibility changes&lt;/li>
&lt;/ul>
&lt;pre class="mermaid">
flowchart TD
client["Git client / CI runner"]
edge["Fastly Edge POP (nearest)"]
shield["Fastly Shield POP (IAD)"]
nginx["OpenResty Nginx (origin)"]
lua["Lua: git_upload_pack.lua"]
valkey["Valkey denylist"]
gitlab["GitLab webservice"]
webhook["gitlab-git-cache-webhook"]
gitlab_events["GitLab project events"]
client -- "POST /git-upload-pack" --> edge
edge -- "HIT → serve from edge" --> client
edge -- "MISS → forward to shield" --> shield
shield -- "HIT → return to edge (edge caches)" --> edge
shield -- "MISS → fetch from origin" --> nginx
nginx --> lua
lua -- "authenticated? check denylist" --> valkey
lua -- "denied/error: keep auth, skip cache" --> gitlab
lua -- "allowed: keep auth, signal cacheable" --> gitlab
gitlab -- "packfile response" --> nginx
nginx -- "X-Git-Cacheable: 1 (if allowed)" --> shield
gitlab_events --> webhook
webhook -- "SET/DEL git:deny:" --> valkey
&lt;/pre>
&lt;p>The request flow:&lt;/p>
&lt;ol>
&lt;li>The &lt;code>POST /git-upload-pack&lt;/code> arrives at the nearest Fastly edge POP.&lt;/li>
&lt;li>VCL checks the body: if &lt;code>Content-Length&lt;/code> exceeds 8 KB (the limit of what Fastly can read from &lt;code>req.body&lt;/code>), or the body does not contain &lt;code>command=fetch&lt;/code>, the request is passed through uncached.&lt;/li>
&lt;li>VCL hashes the body with SHA256 to build the cache key, base64-encodes the body into &lt;code>X-Git-Original-Body&lt;/code>, and converts the request to GET. If the request carries authentication headers (&lt;code>Authorization&lt;/code>, &lt;code>PRIVATE-TOKEN&lt;/code>, &lt;code>Job-Token&lt;/code>), VCL sets &lt;code>X-Git-Auth-Passthrough&lt;/code> to flag it — but the request still enters the cache lookup.&lt;/li>
&lt;li>On a cache hit at the edge, the packfile is served immediately — regardless of whether the request is authenticated or not.&lt;/li>
&lt;li>On a miss, the request routes to the IAD shield POP. If the shield has it cached, it returns the object and the edge caches it locally.&lt;/li>
&lt;li>On a shield miss, the request reaches Nginx at the origin. Lua detects &lt;code>X-Git-Original-Body&lt;/code> and restores the POST body. If &lt;code>X-Git-Auth-Passthrough&lt;/code> is set, Lua checks the Valkey denylist: if the repo is private (or Valkey is unreachable), the &lt;code>Authorization&lt;/code> header is preserved and cacheability is not signaled — the response passes through uncached. If the repo is not on the denylist, &lt;code>Authorization&lt;/code> is preserved (internal repos need it for GitLab to return 200) and cacheability is signaled.&lt;/li>
&lt;li>For unauthenticated requests (no passthrough flag), Lua strips &lt;code>Authorization&lt;/code> and signals cacheability unconditionally — these are by definition accessing public repositories.&lt;/li>
&lt;li>The response flows back through the shield and the edge. If &lt;code>X-Git-Cacheable: 1&lt;/code> is present, both nodes cache the response. Subsequent requests — authenticated or not — for the same cache key are served directly from cache.&lt;/li>
&lt;/ol>
&lt;h2 id="the-vcl-layer">The VCL layer&lt;/h2>
&lt;p>The &lt;code>vcl_recv&lt;/code> snippet runs at priority 9, before the existing &lt;code>enable_segmented_caching&lt;/code> snippet at priority 10 which would otherwise &lt;code>return(pass)&lt;/code> for non-asset URLs:&lt;/p>
&lt;pre tabindex="0">&lt;code># Snippet git-cache-vcl-recv : 9
# Edge: convert POST to GET, hash body, encode body in header
if (req.url ~ &amp;#34;/git-upload-pack$&amp;#34; &amp;amp;&amp;amp; req.request == &amp;#34;POST&amp;#34;) {
if (std.atoi(req.http.Content-Length) &amp;gt; 8192) {
return(pass);
}
if (req.body !~ &amp;#34;command=fetch&amp;#34;) {
return(pass);
}
set req.http.X-Git-Cache-Key = &amp;#34;v3:&amp;#34; digest.hash_sha256(req.body);
set req.http.X-Git-Original-Body = digest.base64(req.body);
# Flag authenticated requests — they still enter the cache lookup,
# but on a miss Lua uses this to decide whether to cache the response
if (req.http.Authorization || req.http.PRIVATE-TOKEN || req.http.Job-Token) {
set req.http.X-Git-Auth-Passthrough = &amp;#34;1&amp;#34;;
}
set req.request = &amp;#34;GET&amp;#34;;
set req.backend = F_Host_1;
if (req.restarts == 0) {
set req.backend = fastly.try_select_shield(ssl_shield_iad_va_us, F_Host_1);
}
return(lookup);
}
# Shield: request already converted to GET by the edge
if (req.http.X-Git-Cache-Key) {
set req.backend = F_Host_1;
return(lookup);
}
&lt;/code>&lt;/pre>&lt;p>Authenticated requests — CI runners with &lt;code>Authorization: Basic &amp;lt;gitlab-ci-token:TOKEN&amp;gt;&lt;/code>, API clients with &lt;code>PRIVATE-TOKEN&lt;/code> or &lt;code>Job-Token&lt;/code> — are no longer sent straight to origin. Instead, VCL flags them with &lt;code>X-Git-Auth-Passthrough&lt;/code> and lets them enter the cache lookup. On a cache hit, the packfile is served directly from the edge — no origin contact, no credential validation needed, because the cached object can only exist if a previous request already established that the repository is public (see &lt;a href="#protecting-private-repositories">Protecting private repositories&lt;/a>). On a cache miss, the flagged request reaches origin where Lua checks the Valkey denylist to decide whether the response should be cached.&lt;/p>
&lt;p>The &lt;code>command=fetch&lt;/code> filter means only Git protocol v2 fetch commands are cached. The &lt;code>ls-refs&lt;/code> command is excluded because its request body is essentially static — caching it with a long TTL would serve stale ref listings after a push. Fetch bodies encode exactly the SHAs the client wants and already has, making them safe to cache indefinitely.&lt;/p>
&lt;p>The &lt;code>v3:&lt;/code> prefix is a cache version string. Bumping it invalidates all existing cache entries without touching Fastly&amp;rsquo;s purge API.&lt;/p>
&lt;p>The second &lt;code>if&lt;/code> block handles the shield. When a cache miss at the edge forwards the request to the shield POP, the shield runs &lt;code>vcl_recv&lt;/code> again. At that point the request is already a GET (the edge converted it), so the first block&amp;rsquo;s &lt;code>req.request == &amp;quot;POST&amp;quot;&lt;/code> check will not match. Without the second block, the request would fall through to the &lt;code>enable_segmented_caching&lt;/code> snippet, which returns &lt;code>pass&lt;/code> for any URL that is not an artifact or archive — effectively preventing the shield from ever caching git traffic.&lt;/p>
&lt;p>The &lt;code>vcl_hash&lt;/code> snippet overrides the default URL-based hash when a cache key is present:&lt;/p>
&lt;pre tabindex="0">&lt;code># Snippet git-cache-vcl-hash : 10
if (req.http.X-Git-Cache-Key) {
set req.hash += req.http.X-Git-Cache-Key;
return(hash);
}
&lt;/code>&lt;/pre>&lt;p>The &lt;code>vcl_fetch&lt;/code> snippet caches 200 responses that carry the &lt;code>X-Git-Cacheable&lt;/code> signal from Nginx:&lt;/p>
&lt;pre tabindex="0">&lt;code># Snippet git-cache-vcl-fetch : 100
if (req.http.X-Git-Cache-Key) {
if (beresp.status == 200 &amp;amp;&amp;amp; beresp.http.X-Git-Cacheable == &amp;#34;1&amp;#34;) {
set beresp.http.Surrogate-Key = &amp;#34;git-cache &amp;#34; regsub(req.url.path, &amp;#34;/git-upload-pack$&amp;#34;, &amp;#34;&amp;#34;);
set beresp.cacheable = true;
set beresp.ttl = 30d;
set beresp.http.X-Git-Cache-Key = req.http.X-Git-Cache-Key;
unset beresp.http.Cache-Control;
unset beresp.http.Pragma;
unset beresp.http.Expires;
unset beresp.http.Set-Cookie;
return(deliver);
}
set beresp.ttl = 0s;
set beresp.cacheable = false;
return(deliver);
}
&lt;/code>&lt;/pre>&lt;p>The &lt;code>Surrogate-Key&lt;/code> line tags each cached object with both a global &lt;code>git-cache&lt;/code> key and the repository path. This enables targeted purging — a single repository&amp;rsquo;s cache can be flushed with &lt;code>fastly purge --key &amp;quot;/GNOME/glib&amp;quot;&lt;/code>, or all git cache at once with &lt;code>fastly purge --key &amp;quot;git-cache&amp;quot;&lt;/code>.&lt;/p>
&lt;p>The 30-day TTL is deliberately long. Git pack data is content-addressed: a pack for a given set of &lt;code>want&lt;/code>/&lt;code>have&lt;/code> lines will always be the same. As long as the objects exist in the repository, the cached pack is valid. The only case where a cached pack could be wrong is if objects were deleted (force-push that drops history, for instance), which is rare and, on GNOME&amp;rsquo;s GitLab, made even rarer by the &lt;a href="https://gitlab.gnome.org/GNOME/gitaly-custom-hooks">Gitaly custom hooks&lt;/a> we run to prevent force-pushes and history rewrites on protected namespaces. In those cases the cache version prefix would force a key change rather than relying on TTL expiry.&lt;/p>
&lt;p>The &lt;code>X-Git-Cacheable&lt;/code> header is intentionally &lt;strong>not&lt;/strong> unset in &lt;code>vcl_fetch&lt;/code>. This is important for the shielding architecture: when the shield caches the object, the stored headers include &lt;code>X-Git-Cacheable: 1&lt;/code>. When the edge later fetches this object from the shield, the edge&amp;rsquo;s own &lt;code>vcl_fetch&lt;/code> sees the header and knows it is safe to cache locally. If &lt;code>vcl_fetch&lt;/code> stripped the header, the edge would never cache — every request would be a local miss that has to travel back to the shield.&lt;/p>
&lt;p>The cleanup happens in &lt;code>vcl_deliver&lt;/code>, which runs last before the response reaches the client:&lt;/p>
&lt;pre tabindex="0">&lt;code># Snippet git-cache-vcl-deliver : 100
if (req.http.X-Git-Cache-Key) {
set resp.http.X-Git-Cache-Status = if(fastly_info.state ~ &amp;#34;HIT(?:-|\z)&amp;#34;, &amp;#34;HIT&amp;#34;, &amp;#34;MISS&amp;#34;);
unset resp.http.X-Git-Original-Body;
if (!req.http.Fastly-FF) {
unset resp.http.X-Git-Cacheable;
unset resp.http.X-Git-Cache-Key;
}
}
&lt;/code>&lt;/pre>&lt;p>The &lt;code>Fastly-FF&lt;/code> check distinguishes between inter-POP traffic (shield-to-edge) and the final client response. &lt;code>Fastly-FF&lt;/code> is set when the request comes from another Fastly node. On the shield, where the request came from the edge, internal headers like &lt;code>X-Git-Cacheable&lt;/code> and &lt;code>X-Git-Cache-Key&lt;/code> are preserved — the edge&amp;rsquo;s &lt;code>vcl_fetch&lt;/code> needs them. On the edge, where the request came from the actual client, those headers are stripped from the final response. Only &lt;code>X-Git-Cache-Status&lt;/code> is exposed to clients for observability.&lt;/p>
&lt;h2 id="the-post-to-get-conversion">The POST-to-GET conversion&lt;/h2>
&lt;p>This is probably the most unusual part of the design. Fastly&amp;rsquo;s consistent hashing and shield routing only works for GET requests. POST requests always go straight to origin. Fastly does provide a way to force POST responses into the cache — by returning &lt;code>pass&lt;/code> in &lt;code>vcl_recv&lt;/code> and setting &lt;code>beresp.cacheable&lt;/code> in &lt;code>vcl_fetch&lt;/code> — but it is a blunt instrument: there is no consistent hashing, no shield collapsing, and no guarantee that two nodes in the same POP will ever share the cached result.&lt;/p>
&lt;p>By converting the POST to a GET in VCL, encoding the body in a header (&lt;code>X-Git-Original-Body&lt;/code>), and using a body-derived SHA256 as the cache key, we get consistent hashing and shield-level request collapsing for free. The VCL uses the &lt;code>X-Git-Cache-Key&lt;/code> header (not the URL or method) as the cache key, so the GET conversion is invisible to the caching logic.&lt;/p>
&lt;p>Fastly&amp;rsquo;s shield feature routes cache misses through a designated shield node before going to origin. When two different edge nodes both get a MISS for the same cache key simultaneously, the shield node collapses them into a single origin request. This is important because without it, a burst of CI jobs fetching the same commit would all miss, all go to origin in parallel, and GitLab would end up generating the same pack multiple times.&lt;/p>
&lt;h2 id="protecting-private-repositories">Protecting private repositories&lt;/h2>
&lt;p>Private repository traffic must never be cached — that would mean storing authenticated git content in a third-party cache and serving it to arbitrary clients. The protection relies on two independent layers.&lt;/p>
&lt;p>&lt;strong>Layer 1: cache population is restricted.&lt;/strong> The cache can only be populated when Lua signals cacheability via &lt;code>X-Git-Cacheable: 1&lt;/code>. Lua only signals cacheability when the request is either unauthenticated (by definition accessing a public repo) or authenticated for a repo that is not on the Valkey denylist. For private repos, Lua does not signal cacheability, so &lt;code>vcl_fetch&lt;/code> sets &lt;code>ttl=0&lt;/code> and &lt;code>cacheable=false&lt;/code> — the response is delivered but never stored.&lt;/p>
&lt;p>&lt;strong>Layer 2: the Valkey denylist.&lt;/strong> A &lt;a href="https://gitlab.gnome.org/GNOME/gitlab-git-cache-webhook">webhook service&lt;/a> listens for GitLab &lt;code>project_create&lt;/code> and &lt;code>project_update&lt;/code> system hooks. When a project&amp;rsquo;s visibility is set to private (level &lt;code>0&lt;/code>), the webhook sets a &lt;code>git:deny:&amp;lt;path&amp;gt;&lt;/code> key in Valkey. When visibility changes to internal (level &lt;code>10&lt;/code>) or public (level &lt;code>20&lt;/code>), the key is removed. A periodic reconciliation job (&lt;code>reconcile.py&lt;/code>) syncs the full denylist against the GitLab API to correct any drift from missed events.&lt;/p>
&lt;p>On a cache miss for an authenticated request, Lua checks the denylist:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Repo is on the denylist (private):&lt;/strong> &lt;code>Authorization&lt;/code> is preserved, cacheability is not signaled. The request proxies to GitLab with credentials intact, GitLab validates the token, the response is returned but never cached.&lt;/li>
&lt;li>&lt;strong>Repo is not on the denylist (public/internal):&lt;/strong> &lt;code>Authorization&lt;/code> is preserved (internal repos require it for GitLab to return 200), cacheability is signaled. The response is cached for future requests.&lt;/li>
&lt;li>&lt;strong>Valkey is unreachable or returns an error:&lt;/strong> treated the same as denied — &lt;code>Authorization&lt;/code> is preserved, cacheability is not signaled. This fail-closed design means infrastructure failures result in cache misses, never in data leaks.&lt;/li>
&lt;/ul>
&lt;p>The denylist only needs to track private repositories, which are a small fraction of the total on GNOME&amp;rsquo;s GitLab instance. A private repo&amp;rsquo;s packfile can never enter the cache through two independent mechanisms: the denylist prevents Lua from signaling cacheability, and even if the denylist were somehow wrong, an unauthenticated request to a private repo returns a 401 from GitLab — which &lt;code>vcl_fetch&lt;/code> does not cache (it only caches &lt;code>200 + X-Git-Cacheable&lt;/code>).&lt;/p>
&lt;h2 id="the-lua-layer">The Lua layer&lt;/h2>
&lt;p>With the VCL handling body hashing, the POST-to-GET conversion, and the cache lookup for all requests, the Lua script runs on cache misses that reach origin. Both authenticated and unauthenticated requests can arrive here. The script&amp;rsquo;s responsibilities are:&lt;/p>
&lt;ol>
&lt;li>Detect that the request arrived from Fastly with an encoded body (the &lt;code>X-Git-Original-Body&lt;/code> header).&lt;/li>
&lt;li>Decode and restore the original POST.&lt;/li>
&lt;li>For authenticated requests, check the Valkey denylist to determine if the repository is private.&lt;/li>
&lt;li>Signal back to Fastly whether the response is safe to cache.&lt;/li>
&lt;/ol>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Lua" data-lang="Lua">&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">redis_helper&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">require&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;redis_helper&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">redis_host&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">os.getenv&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;REDIS_HOST&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">redis_port&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">os.getenv&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;REDIS_PORT&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">encoded_body&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">get_headers&lt;/span>&lt;span class="p">()[&lt;/span>&lt;span class="s2">&amp;#34;X-Git-Original-Body&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kr">if&lt;/span> &lt;span class="ow">not&lt;/span> &lt;span class="n">encoded_body&lt;/span> &lt;span class="kr">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">return&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kr">end&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">body&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">ngx.decode_base64&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">encoded_body&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">read_body&lt;/span>&lt;span class="p">()&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">set_method&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">ngx.HTTP_POST&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">set_body_data&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">body&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">set_header&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;Content-Length&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">tostring&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">#&lt;/span>&lt;span class="n">body&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">clear_header&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;X-Git-Original-Body&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kr">if&lt;/span> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">get_headers&lt;/span>&lt;span class="p">()[&lt;/span>&lt;span class="s2">&amp;#34;X-Git-Auth-Passthrough&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span> &lt;span class="kr">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">clear_header&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;X-Git-Auth-Passthrough&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kd">local&lt;/span> &lt;span class="n">uri&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">ngx.var&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">uri&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kd">local&lt;/span> &lt;span class="n">repo_path&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">uri&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="n">match&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;^/(.+)/git%-upload%-pack$&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">if&lt;/span> &lt;span class="n">repo_path&lt;/span> &lt;span class="kr">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">repo_path&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">repo_path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="n">gsub&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;%.git$&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">end&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kd">local&lt;/span> &lt;span class="n">denied&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">err&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">redis_helper.is_denied&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">redis_host&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">redis_port&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">repo_path&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">if&lt;/span> &lt;span class="n">err&lt;/span> &lt;span class="kr">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.log&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">ngx.WARN&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s2">&amp;#34;git-cache: Redis error for &amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">repo_path&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s2">&amp;#34;: &amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">err&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34; — keeping auth, skipping cache&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">end&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">if&lt;/span> &lt;span class="n">err&lt;/span> &lt;span class="ow">or&lt;/span> &lt;span class="n">denied&lt;/span> &lt;span class="kr">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">return&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">end&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.ctx&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">git_cacheable&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="kc">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kr">else&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">clear_header&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;Authorization&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.ctx&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">git_cacheable&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="kc">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kr">end&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The two branches handle the authenticated and unauthenticated paths. When &lt;code>X-Git-Auth-Passthrough&lt;/code> is present, the request came from a CI runner or API client. Lua checks the denylist: if the repo is private or Valkey is unreachable, the script returns early — &lt;code>Authorization&lt;/code> stays on the request (so GitLab can validate it), and &lt;code>git_cacheable&lt;/code> is never set (so the response is not cached). If the repo is not denied, &lt;code>Authorization&lt;/code> is preserved and cacheability is signaled. The &lt;code>Authorization&lt;/code> header is kept rather than stripped because internal repositories (visibility level &lt;code>10&lt;/code>) require authentication for git operations — stripping it would cause GitLab to return a 401. Public repos work with or without credentials, so keeping the header is safe for both.&lt;/p>
&lt;p>For unauthenticated requests (no passthrough flag), &lt;code>Authorization&lt;/code> is stripped and cacheability is signaled unconditionally — these are by definition accessing public repositories.&lt;/p>
&lt;p>The early &lt;code>return&lt;/code> for denied or errored lookups is the fail-closed behavior. The request still proxies to GitLab (the &lt;code>proxy_pass&lt;/code> directive in the Nginx location block runs after Lua), but without the cacheable signal, &lt;code>vcl_fetch&lt;/code> will not store the response.&lt;/p>
&lt;p>The &lt;code>ngx.ctx.git_cacheable&lt;/code> flag is picked up by the &lt;code>header_filter_by_lua_block&lt;/code> in the Nginx configuration, which translates it into the &lt;code>X-Git-Cacheable: 1&lt;/code> response header that &lt;code>vcl_fetch&lt;/code> checks:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Nginx" data-lang="Nginx">&lt;span class="line">&lt;span class="cl">&lt;span class="k">location&lt;/span> &lt;span class="p">~&lt;/span> &lt;span class="sr">/git-upload-pack$&lt;/span> &lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">client_body_buffer_size&lt;/span> &lt;span class="mi">5m&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">client_max_body_size&lt;/span> &lt;span class="mi">5m&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">access_by_lua_file&lt;/span> &lt;span class="s">/etc/nginx/lua/git_upload_pack.lua&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">header_filter_by_lua_block&lt;/span> &lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">if&lt;/span> &lt;span class="s">ngx.ctx.git_cacheable&lt;/span> &lt;span class="s">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s">ngx.header[&amp;#34;X-Git-Cacheable&amp;#34;]&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s">&amp;#34;1&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s">end&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="err">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s">proxy_pass&lt;/span> &lt;span class="s">http://gitlab-webservice&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">...&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="err">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="debugging-the-rollout">Debugging the rollout&lt;/h2>
&lt;p>The rollout surfaced a few issues worth documenting for anyone building a similar setup on Fastly.&lt;/p>
&lt;p>&lt;strong>Shielding introduces a second &lt;code>vcl_recv&lt;/code> execution.&lt;/strong> When the edge forwards a cache miss to the shield, the shield runs the entire VCL pipeline from scratch. The POST-to-GET conversion in &lt;code>vcl_recv&lt;/code> checks for &lt;code>req.request == &amp;quot;POST&amp;quot;&lt;/code>, but on the shield the request is already a GET. Without the fallback &lt;code>if (req.http.X-Git-Cache-Key)&lt;/code> block, the shield&amp;rsquo;s &lt;code>vcl_recv&lt;/code> would fall through to the segmented caching snippet and &lt;code>return(pass)&lt;/code> — making the shield unable to cache anything.&lt;/p>
&lt;p>&lt;strong>Response headers must survive the shield-to-edge hop.&lt;/strong> &lt;code>vcl_fetch&lt;/code> and &lt;code>vcl_deliver&lt;/code> both run on each node independently. If &lt;code>vcl_fetch&lt;/code> on the shield strips a header after caching the object, the stored object will not have that header. When the edge fetches from the shield, the edge&amp;rsquo;s &lt;code>vcl_fetch&lt;/code> will not see it. The solution is to only strip internal headers in &lt;code>vcl_deliver&lt;/code> on the final client response, using &lt;code>Fastly-FF&lt;/code> to distinguish inter-POP traffic from client traffic.&lt;/p>
&lt;p>&lt;strong>Fastly&amp;rsquo;s &lt;code>req.body&lt;/code> is limited to 8 KB.&lt;/strong> VCL can only inspect the first 8192 bytes of a request body. For the vast majority of git fetch negotiations — especially shallow clones and CI pipelines fetching recent commits — the body is well under this limit. Requests with larger bodies (deep fetches with many &lt;code>have&lt;/code> lines) fall through to &lt;code>return(pass)&lt;/code> and are handled directly by GitLab without caching. This is an acceptable tradeoff: those large-body requests are typically unique negotiations that would not benefit from caching anyway.&lt;/p>
&lt;p>&lt;strong>Git protocol v1 clients are not cached.&lt;/strong> The VCL filters on &lt;code>command=fetch&lt;/code>, which is a Git protocol v2 construct. Protocol v1 uses a different body format (&lt;code>want&lt;/code>/&lt;code>have&lt;/code> lines without the &lt;code>command=&lt;/code> prefix). Since protocol v2 has been the default since git 2.26 (March 2020), the vast majority of traffic benefits from caching. Protocol v1 clients still work correctly — they simply bypass the cache.&lt;/p>
&lt;p>&lt;strong>Internal repositories require authentication for git operations.&lt;/strong> An early version of the Lua script stripped &lt;code>Authorization&lt;/code> for any repo not on the denylist, assuming that &amp;ldquo;not private&amp;rdquo; meant &amp;ldquo;accessible without credentials.&amp;rdquo; Internal repositories (visibility level &lt;code>10&lt;/code>) are not on the denylist — their content is not sensitive — but GitLab still requires authentication for git clone/fetch operations on them. Stripping credentials produced a 401 from GitLab. The fix was to preserve &lt;code>Authorization&lt;/code> for all authenticated requests that pass the denylist check, regardless of whether the repo is public or internal. Public repos accept the header harmlessly; internal repos require it.&lt;/p>
&lt;h2 id="how-we-got-here">How we got here&lt;/h2>
&lt;p>The current architecture is the result of two iterations. The sections above describe the final design; this section documents the path we took to get there.&lt;/p>
&lt;h3 id="iteration-1-separate-cdn-service-with-lua-driven-caching">Iteration 1: Separate CDN service with Lua-driven caching&lt;/h3>
&lt;p>The first version used a separate Fastly CDN service (&lt;code>cdn.gitlab.gnome.org&lt;/code>) as the cache layer, with Nginx doing most of the heavy lifting in Lua:&lt;/p>
&lt;pre class="mermaid">
flowchart TD
client["Git client / CI runner"]
gitlab_gnome["gitlab.gnome.org (Nginx reverse proxy)"]
nginx["OpenResty Nginx"]
lua["Lua: git_upload_pack.lua"]
cdn_origin["/cdn-origin internal location"]
fastly_cdn["Fastly CDN"]
origin["gitlab.gnome.org via its origin (second pass)"]
gitlab["GitLab webservice"]
valkey["Valkey denylist"]
webhook["gitlab-git-cache-webhook"]
gitlab_events["GitLab project events"]
client --> gitlab_gnome
gitlab_gnome --> nginx
nginx --> lua
lua -- "check denylist" --> valkey
lua -- "private repo: BYPASS" --> gitlab
lua -- "public/internal: internal redirect" --> cdn_origin
cdn_origin --> fastly_cdn
fastly_cdn -- "HIT" --> cdn_origin
fastly_cdn -- "MISS: origin fetch" --> origin
origin --> gitlab
gitlab_events --> webhook
webhook -- "SET/DEL git:deny:" --> valkey
&lt;/pre>
&lt;p>In this design, the Lua script did everything: read the POST body, SHA256-hash it to build a cache key, check a Valkey denylist to exclude private repositories, convert the POST to a GET, encode the body in a header, and perform an internal redirect to a &lt;code>/cdn-origin&lt;/code> location that proxied to the CDN. On a cache miss, the CDN would fetch from &lt;code>gitlab.gnome.org&lt;/code> directly (the &amp;ldquo;second pass&amp;rdquo;), where Lua would detect the origin fetch, decode the body, restore the POST, and proxy to GitLab.&lt;/p>
&lt;p>Private repositories were protected by a denylist stored in Valkey. A small FastAPI webhook service (&lt;a href="https://gitlab.gnome.org/GNOME/gitlab-git-cache-webhook">gitlab-git-cache-webhook&lt;/a>) listened for GitLab system hooks on &lt;code>project_create&lt;/code> and &lt;code>project_update&lt;/code> events, maintaining &lt;code>git:deny:&amp;lt;path&amp;gt;&lt;/code> keys for private repositories (visibility level &lt;code>0&lt;/code>). Internal repositories (level &lt;code>10&lt;/code>) were treated the same as public (level &lt;code>20&lt;/code>) since they are accessible to any authenticated user on the instance.&lt;/p>
&lt;p>The Lua script for this design was substantially more complex:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Lua" data-lang="Lua">&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">resty_sha256&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">require&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;resty.sha256&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">resty_str&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">require&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;resty.string&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">redis_helper&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">require&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;redis_helper&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">redis_host&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">os.getenv&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;REDIS_HOST&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="ow">or&lt;/span> &lt;span class="s2">&amp;#34;localhost&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">redis_port&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">os.getenv&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;REDIS_PORT&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="ow">or&lt;/span> &lt;span class="s2">&amp;#34;6379&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">-- Second pass: request arriving from CDN origin fetch.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kr">if&lt;/span> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">get_headers&lt;/span>&lt;span class="p">()[&lt;/span>&lt;span class="s2">&amp;#34;X-Git-Cache-Internal&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span> &lt;span class="kr">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kd">local&lt;/span> &lt;span class="n">encoded_body&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">get_headers&lt;/span>&lt;span class="p">()[&lt;/span>&lt;span class="s2">&amp;#34;X-Git-Original-Body&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">if&lt;/span> &lt;span class="n">encoded_body&lt;/span> &lt;span class="kr">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">read_body&lt;/span>&lt;span class="p">()&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kd">local&lt;/span> &lt;span class="n">body&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">ngx.decode_base64&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">encoded_body&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">set_method&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">ngx.HTTP_POST&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">set_body_data&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">body&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">set_header&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;Content-Length&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">tostring&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">#&lt;/span>&lt;span class="n">body&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">clear_header&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;X-Git-Original-Body&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">end&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">return&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kr">end&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>And on the first pass, it handled hashing, denylist checks, and the CDN redirect:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Lua" data-lang="Lua">&lt;span class="line">&lt;span class="cl">&lt;span class="kr">if&lt;/span> &lt;span class="ow">not&lt;/span> &lt;span class="n">body&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="n">find&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;command=fetch&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="mi">1&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="kc">true&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="kr">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ngx.header&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;X-Git-Cache-Status&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;BYPASS&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">return&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kr">end&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">sha256&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">resty_sha256&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="n">new&lt;/span>&lt;span class="p">()&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">sha256&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="n">update&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">body&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">body_hash&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">resty_str.to_hex&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">sha256&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="n">final&lt;/span>&lt;span class="p">())&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">cache_key&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;v2:&amp;#34;&lt;/span> &lt;span class="o">..&lt;/span> &lt;span class="n">repo_path&lt;/span> &lt;span class="o">..&lt;/span> &lt;span class="s2">&amp;#34;:&amp;#34;&lt;/span> &lt;span class="o">..&lt;/span> &lt;span class="n">body_hash&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">local&lt;/span> &lt;span class="n">denied&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">err&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">redis_helper.is_denied&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">redis_host&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">redis_port&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">repo_path&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kr">if&lt;/span> &lt;span class="n">denied&lt;/span> &lt;span class="kr">then&lt;/span> &lt;span class="kr">return&lt;/span> &lt;span class="kr">end&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">clear_header&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;Authorization&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">set_header&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;X-Git-Original-Body&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">ngx.encode_base64&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">body&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">set_method&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">ngx.HTTP_GET&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">ngx.req&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">set_body_data&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kr">return&lt;/span> &lt;span class="n">ngx.exec&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;/cdn-origin&amp;#34;&lt;/span> &lt;span class="o">..&lt;/span> &lt;span class="n">uri&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The CDN&amp;rsquo;s VCL was relatively simple — it used &lt;code>X-Git-Cache-Key&lt;/code> for the hash, routed through a shield, and cached 200 responses for 30 days.&lt;/p>
&lt;p>This architecture worked, but it had two significant limitations that led to the current design.&lt;/p>
&lt;h3 id="iteration-2-edge-caching-with-ci-runner-participation">Iteration 2: Edge caching with CI runner participation&lt;/h3>
&lt;p>The first problem with the separate CDN service was geographic. Nginx runs in AWS us-east-1, so from Fastly&amp;rsquo;s perspective the only client of the CDN was that single instance in Virginia. Every request entered through the IAD POP, which meant the CDN&amp;rsquo;s edge POPs around the world were never populated. A CI runner in Europe would have its request travel from a European Fastly POP to IAD, then to Nginx, then back to Fastly IAD, and then all the way back — crossing the Atlantic twice for every cache miss.&lt;/p>
&lt;p>The fix was to eliminate the separate CDN service and move all the caching logic into the &lt;code>gitlab.gnome.org&lt;/code> Fastly service itself. The key insight was that the POST-to-GET conversion and body hashing could happen in Fastly&amp;rsquo;s VCL rather than in Lua — Fastly provides &lt;code>digest.hash_sha256()&lt;/code> and &lt;code>digest.base64()&lt;/code> functions that operate directly on &lt;code>req.body&lt;/code>. By doing the conversion at the CDN edge, every POP in the network became a potential cache node for git traffic.&lt;/p>
&lt;p>The second problem was that the original denylist approach had two flaws. First, its error handling was fail-open: a Valkey connection error would cause the Lua script to assume the repo was public and strip credentials — the wrong default. Second, even after briefly replacing the denylist with a simple VCL auth bypass (&lt;code>return(pass)&lt;/code> for any request with &lt;code>Authorization&lt;/code>), CI runners were left completely uncached. GitLab CI always injects a &lt;code>CI_JOB_TOKEN&lt;/code> into every job, and the runner authenticates with &lt;code>Authorization: Basic &amp;lt;gitlab-ci-token:TOKEN&amp;gt;&lt;/code> regardless of whether the repository is public or private. With the auth bypass, every CI clone skipped the cache entirely — safe, but it left the biggest source of redundant traffic unserved.&lt;/p>
&lt;p>The current design solves both problems. VCL flags authenticated requests with &lt;code>X-Git-Auth-Passthrough&lt;/code> instead of bypassing the cache, letting them participate in cache lookups. On a hit, the cached packfile is served immediately. On a miss, the request reaches Lua at origin, where the flag triggers a denylist check against Valkey — the same denylist and webhook infrastructure from iteration 1, re-deployed with one critical change: fail-closed error handling. A Valkey error or missing connection causes Lua to preserve &lt;code>Authorization&lt;/code> and skip cacheability signaling. The request still works (GitLab validates the token and serves the packfile), but the response is not cached. Infrastructure failures result in cache misses, never in data leaks.&lt;/p>
&lt;p>The denylist only tracks private repositories (visibility level &lt;code>0&lt;/code>), which are a small fraction of the total on GNOME&amp;rsquo;s GitLab. Public and internal repositories pass the denylist check, and Lua signals cacheability while preserving the &lt;code>Authorization&lt;/code> header — internal repos require it for GitLab to return 200, and public repos accept it harmlessly.&lt;/p>
&lt;h2 id="conclusions">Conclusions&lt;/h2>
&lt;p>The system has been running in production since April 2026 and has gone through two iterations to reach its current form. Packfiles are cached at Fastly edge POPs worldwide — a CI runner in Europe gets a cache hit served from a European POP rather than making a round trip to the US East coast.&lt;/p>
&lt;p>The moving parts are Fastly&amp;rsquo;s VCL, an OpenResty Nginx instance with a ~30-line Lua script, a Valkey instance storing the private repository denylist, and a small webhook service that keeps the denylist synchronized with GitLab. Private repositories are protected by two independent layers: the Valkey denylist (which prevents cacheability signaling) and GitLab&amp;rsquo;s own authentication (which rejects unauthenticated access).&lt;/p>
&lt;p>If something goes wrong with the cache layer, requests fall through to GitLab directly — the same path they took before caching existed. There is no failure mode where caching breaks git operations. This also means we don&amp;rsquo;t redirect any traffic to github.com anymore.&lt;/p>
&lt;p>That should be all for today, stay tuned!&lt;/p>
&lt;script type="module">
import mermaid from 'https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.esm.min.mjs';
mermaid.initialize({ startOnLoad: true, theme: 'neutral' });
&lt;/script></description><link>https://www.dragonsreach.it/2026/04/17/gnome-gitlab-git-pulls-caching-improvements/</link><guid>https://www.dragonsreach.it/2026/04/17/gnome-gitlab-git-pulls-caching-improvements/</guid><pubDate>Fri, 17 Apr 2026 10:00:00 -0400</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>2024 GNOME Infrastructure Annual Review</title><description>&lt;h2 id="table-of-contents">Table of Contents&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="#1-introduction">1. Introduction&lt;/a>&lt;/li>
&lt;li>&lt;a href="#2-achievements">2. Achievements&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#21-major-achievements">2.1. Major achievements&lt;/a>&lt;/li>
&lt;li>&lt;a href="#22-minor-achievements">2.2. Minor achievements&lt;/a>&lt;/li>
&lt;li>&lt;a href="#23-minor-annoyancesbugs-that-were-also-fixed-in-2024">2.3 Minor annoyances/bugs that were also fixed in 2024&lt;/a>&lt;/li>
&lt;li>&lt;a href="#23-our-brand-new-and-renewed-partnerships">2.3. Our brand new and renewed partnerships&lt;/a>&lt;/li>
&lt;li>&lt;a href="#expressing-my-gratitude">Expressing my gratitude&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;/ul>
&lt;h2 id="1-introduction">1. Introduction&lt;/h2>
&lt;p>Time is passing by very quickly and another year will go as we approach the end of 2024. This year has been fundamental in shaping the present and the future of GNOME&amp;rsquo;s Infrastructure with its major highlight being a completely revamped platform and a migration of all GNOME services over to AWS. In this post I&amp;rsquo;ll try to highlight what the major achievements have been throughout the past 12 months.&lt;/p>
&lt;h2 id="2-achievements">2. Achievements&lt;/h2>
&lt;p>In the below is a list of individual tasks and projects we were able to fulfill in 2024. This section will be particularly long but I want to stress the importance of each of these items and the efforts we put in to make sure they were delivered in a timely manner.&lt;/p>
&lt;h3 id="21-major-achievements">2.1. Major achievements&lt;/h3>
&lt;ol>
&lt;li>All the applications (except for ego, which we expect to handle as soon as next week or in January) were migrated to our new AWS platform (see &lt;a href="https://www.dragonsreach.it/2024/11/16/gnome-infrastructure-migration-to-aws/">GNOME Infrastructure migration to AWS&lt;/a>)&lt;/li>
&lt;li>During each of the apps migrations we made sure to:
&lt;ol>
&lt;li>Migrate to sso.gnome.org and make 2FA mandatory&lt;/li>
&lt;li>Make sure database connections are handled via connection poolers&lt;/li>
&lt;li>Double check the container images in use were up-to-date and GitLab CI/CD pipeline schedules were turned on for weekly rebuilds (security updates)&lt;/li>
&lt;li>For GitLab, we made sure repositories were migrated to an EBS volume to increase IO throughput and bandwidth&lt;/li>
&lt;/ol>
&lt;/li>
&lt;li>Migrated away our backup mechanism away from rdiff-backup into AWS Backup service (which handles both our AWS EFS and EBS snapshots)&lt;/li>
&lt;li>Retired our NSD install and migrated our authoritative name servers to CloudNS (it comes with multiple redundant authoritative servers, DDOS protection and automated DNSSEC keys rotation and management)&lt;/li>
&lt;li>We moved away from Ceph and the need to maintain our own storage solution and started leveraging AWS EFS and EBS&lt;/li>
&lt;li>We deprecated Splunk and built a solution around promtail and Loki in order to handle our logging requirements&lt;/li>
&lt;li>We deprecated Prometheus blackbox and started leveraging CloudNS monitoring service which we interact with using an API and a set of CI/CD jobs we host in GitHub&lt;/li>
&lt;li>We archived GNOME&amp;rsquo;s wiki and turned it into a static HTML copy&lt;/li>
&lt;li>We replaced ftpadmin with the GNOME Release Services, thanks speknik! More information around what steps should GNOME Maintainers now follow when doing a module release are available &lt;a href="https://handbook.gnome.org/maintainers/making-a-release.html">here&lt;/a>. The service uses JWT tokens to verify and authorize specific CI/CD jobs and only allows new releases when the process is initiated by a project CI living within the GNOME GitLab namespace and a protected tag. With master.gnome.org and ftpadmin being in production for literally ages, we wanted to find a better mechanism to release GNOME software and avoid a single maintainer SSH key leak to allow a possible attacker to tamper tarballs and potentially compromise milions of computers running GNOME around the globe. With this change we don&amp;rsquo;t leverage SSH anymore and most importantly we don&amp;rsquo;t allow maintainers to generate GNOME modules tarballs on their personal computers rather we force them to use CI/CD in order to achieve the same result. We&amp;rsquo;ll be coming up shortly with a dedicated and isolated runner that will only build jobs tagged as releasing GNOME software.&lt;/li>
&lt;li>We retired our mirroring infrastructure based on Mirrorbits and replaced it with our CDN partner, CDN77&lt;/li>
&lt;li>We decoupled GIMP mirroring service from GNOME&amp;rsquo;s one, GIMP now hosts its tarballs (and associated rsync daemon) on top of a different master node, thanks OSUOSL for sponsoring the VM that makes this possible!&lt;/li>
&lt;/ol>
&lt;h3 id="22-minor-achievements">2.2. Minor achievements&lt;/h3>
&lt;ol>
&lt;li>Retired multiple VMs: splunk, nsd0{1,2}, master, ceph-metrics, gitaly&lt;/li>
&lt;li>We started managing our DNS using an API and CI/CD jobs hosted in GitHub (this to avoid relying on GNOME&amp;rsquo;s GitLab which in case of unavailability would prevent us to update DNS entries)&lt;/li>
&lt;li>We migrated smtp.gnome.org to OSCI in order to not lose IP reputations and various whitelists we received throughout the years by multiple organizations&lt;/li>
&lt;li>We deprecated our former internal DNS authoritatives based on FreeIPA. We are now leveraging internal VPC resolvers and Route53 Private zones&lt;/li>
&lt;li>We deprecated all our OSUOSL GitLab runners due to particularly slow IO and high steal time and replaced them with a new Heztner EX44 instance, kindly sponsored by GIMP. OSUOSL is working on coming up with local storage on their Openstack platform. We are looking forward to test that and introduce new runners as soon as the solution will be made available&lt;/li>
&lt;li>Retired idm0{1,2} and redirected them to a new FreeIPA load balanced service at &lt;a href="https://idm.gnome.org">https://idm.gnome.org&lt;/a>&lt;/li>
&lt;li>We retired services which weren&amp;rsquo;t relevant for the community anymore: surveys.gnome.org, roundcube (aka webmail.gnome.org)&lt;/li>
&lt;li>We migrated nmcheck.gnome.org to Fastly and are using Synthetic responses to handle HTTP responses to clients&lt;/li>
&lt;li>We upgraded to Ansible Automation Platform (AAP) 2.5&lt;/li>
&lt;li>As part of the migration to our new AWS based platform, we upgraded Openshift to release 4.17&lt;/li>
&lt;li>We received a 2k grant from Microsoft which we are using for an Azure ARM64 GitLab runner&lt;/li>
&lt;li>All of our GitLab runners fleet are now hourly kept in sync using AAP (Ansible roles were built to make this happen)&lt;/li>
&lt;li>We upgraded Cachet to 3.x series and fixed dynamic status.gnome.org updates (via a customized version of cachet-monitor)&lt;/li>
&lt;li>OS Currency: we upgraded all our systems to RHEL 9&lt;/li>
&lt;li>We converted all our Openshift images that were using a web server to Nginx for consistency/simplicity&lt;/li>
&lt;li>Replaced NRPE with Prometheus metrics based logging, checks such as IDM replication and status are now handled via the Node Exporter textfile plugin&lt;/li>
&lt;li>Migrated download.qemu.org (yes, we also host some components of QEMU&amp;rsquo;s Infrastructure) to use nginx-s3-gateway, downloads are then served via CDN77&lt;/li>
&lt;/ol>
&lt;h3 id="23-minor-annoyancesbugs-that-were-also-fixed-in-2024">2.3 Minor annoyances/bugs that were also fixed in 2024&lt;/h3>
&lt;ol>
&lt;li>Invalid OCSP responses from CDN77, &lt;a href="https://gitlab.gnome.org/Infrastructure/Infrastructure/-/issues/1511">https://gitlab.gnome.org/Infrastructure/Infrastructure/-/issues/1511&lt;/a>&lt;/li>
&lt;li>With the migration to &lt;a href="https://gitlab.com/gitlab-org/build/CNG/-/issues/1853">USE_TINI&lt;/a> for GitLab, no gpg zombie processes are being generated anymore&lt;/li>
&lt;/ol>
&lt;h3 id="23-our-brand-new-and-renewed-partnerships">2.3. Our brand new and renewed partnerships&lt;/h3>
&lt;ol>
&lt;li>From November 2024 and ongoing, AWS will provide sponsorship and funding to the GNOME Project to sustain the majority of its infrastructure needs&lt;/li>
&lt;li>Red Hat kindly sponsored subscriptions for RHEL, Openshift, AAP as well as hosting, bandwidth for the GNOME Infrastructure throughout 2024&lt;/li>
&lt;li>CDN77 provided unlimited bandwidth / traffic on their CDN offering&lt;/li>
&lt;li>Fastly renewed their unlimited bandwidth / traffic plan on their Delivery/Compute offerings&lt;/li>
&lt;li>and thanks to OSUOSL, Packet, DigitalOcean, Microsoft for the continued hosting and sponsorship of a set of GitLab runners, virtual machines and ARM builders!&lt;/li>
&lt;/ol>
&lt;h3 id="expressing-my-gratitude">Expressing my gratitude&lt;/h3>
&lt;p>As I&amp;rsquo;m used to do at the end of each calendar year, I want to express my gratitude to Bartłomiej Piotrowski for our continued cooperation and also to Stefan Peknik for his continued efforts in developing the GNOME Release Service. We started this journey together many months ago when Stefan was trying to find a topic to base his CS bachelor thesis on. With this in mind I went straight into the argument of replacing ftpadmin with a better technology also in light of what happened with the xz case. Stefan put all his enthusiasm and professionality into making this happen and with the service going into production on the 11th of December 2024 history was made.&lt;/p>
&lt;p>That being said, we&amp;rsquo;re closing this year being extremely close to retire our presence from RAL3 which we expect to happen in January 2025. The GNOME Infrastructure will also send in a proposal to talk at GUADEC 2025, in Italy, to present and discuss all these changes with the community.&lt;/p></description><link>https://www.dragonsreach.it/2024/12/14/gnome-infrastructure-annual-review/</link><guid>https://www.dragonsreach.it/2024/12/14/gnome-infrastructure-annual-review/</guid><pubDate>Fri, 13 Dec 2024 16:29:20 -0500</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>GNOME Infrastructure migration to AWS</title><description>&lt;h2 id="1-some-historical-background">1. Some historical background&lt;/h2>
&lt;p>The GNOME Infrastructure has been hosted as part of one of Red Hat&amp;rsquo;s datacenters for over 15 years now. The &amp;ldquo;community cage&amp;rdquo;, which is how we usually define the hosting platform that backs up multiple Open Source projects including &lt;a href="https://www.osci.io">OSCI&lt;/a>, is made of a set of racks living within the RAL3 (located in Raleigh) datacenter. Red Hat has not only been contributing to GNOME by maintaining the Red Hat&amp;rsquo;s Desktop Team operational, sponsoring events (such as GUADEC) but has also been supporting the project with hosting, internet connectivity, machines, RHEL (and many other RH products subscriptions). When the infrastructure was originally stood up it was primarily composed of a set of bare metal machines, workloads were not yet virtualized at the time and many services were running directly on top of the physical nodes. The advent of virtual machines and later containers reshaped how we managed and operated every component. What however remained the same over time was the networking layout of these services: a single L2 and a shared (with other tenants) public internet L3 domains (with both IPv4 and IPv6).&lt;/p>
&lt;h2 id="recent-challenges">Recent challenges&lt;/h2>
&lt;p>When GNOME&amp;rsquo;s Openshift 4 environment was built back in 2020 we had to make specific calls:&lt;/p>
&lt;ol>
&lt;li>We&amp;rsquo;d have ran an Openshift Hyperconverged setup (with storage (Ceph), control plane, workloads running on top of the same subset of nodes)&lt;/li>
&lt;li>The total amount of nodes we received budget for was 3, this meant running with masters.schedulable=true&lt;/li>
&lt;li>We&amp;rsquo;d have kept using our former Ceph cluster (as it had slower disks, a good combination for certain workloads we run), this is however not supported by ODF (Openshift Data Foundation) and would have required some glue to make it completely functional&lt;/li>
&lt;li>Migrating GNOME&amp;rsquo;s private L2 network to L3 would have required an effort from Red Hat&amp;rsquo;s IT Network Team who generally contributes outside of their working hours, no changes were planned in this regard&lt;/li>
&lt;li>No changes were planned on the networking equipment side to make links redundant, that means a code upgrade on switches would have required a full services downtime&lt;/li>
&lt;/ol>
&lt;p>Over time and with GNOME&amp;rsquo;s users and contributors base growing (46k users registered in GitLab, 7.44B requests and 50T of traffic per month on services we host on Openshift and kindly served by Fastly&amp;rsquo;s load balancers) we started noticing some of our original architecture decisions weren&amp;rsquo;t positively contributing to platform&amp;rsquo;s availability, specifically:&lt;/p>
&lt;ol>
&lt;li>Every time an Openshift upgrade was applied, it resulted in a cluster downtime due to the unsupported double ODF cluster layout (one internal and one external to the cluster). The behavior was stuck block devices preventing the machines to reboot with associated high IO (and general SELinux labeling mismatches), with the same nodes also hosting OCP&amp;rsquo;s control plane it was resulting in API and other OCP components becoming unavailable&lt;/li>
&lt;li>With no L3 network, we had to create a next-hop on our own to effectively give internet access through NAT to machines without a public internet IP address, this was resulting in connectivity outages whenever the target VM would go down for a quick maintenance&lt;/li>
&lt;/ol>
&lt;h2 id="migration-to-aws">Migration to AWS&lt;/h2>
&lt;p>With budgets season for FY25 approaching we struggled finding the necessary funds in order to finally optimize and fill the gaps of our previous architecture. With this in mind we reached out to &lt;a href="https://aws.amazon.com/opensource/">AWS Open Source Program&lt;/a> and received a substantial amount for us to be able to fully transition GNOME&amp;rsquo;s Infrastructure to the public cloud.&lt;/p>
&lt;p>What we achieved so far:&lt;/p>
&lt;ol>
&lt;li>Deployed and configured VPC related resources, this step will help us resolve the need to have a next-hop device we have to maintain&lt;/li>
&lt;li>Deployed an Openshift 4.17 cluster (which uses a combination of network and classic load balancers, x86 control plane and arm64 workers)&lt;/li>
&lt;li>Deployed IDM nodes that are using a Wireguard tunnel between AWS and RAL3 to remain in sync&lt;/li>
&lt;li>Migrated several applications including SSO, Discourse, Hedgedoc&lt;/li>
&lt;/ol>
&lt;p>What&amp;rsquo;s upcoming:&lt;/p>
&lt;ol>
&lt;li>Migrating away from Splunk and use a combination of rsyslog/promtail/loki&lt;/li>
&lt;li>Keep migrating further applications, the idea is to fully decommission the former cluster and GNOME&amp;rsquo;s presence within Red Hat&amp;rsquo;s community cage during Q1FY25&lt;/li>
&lt;li>Introduce a &lt;a href="https://gitlab.gnome.org/Infrastructure/openshift-images/gnome-release-service">replacement&lt;/a> for master.gnome.org and GNOME tarballs installation&lt;/li>
&lt;li>Migrate applications to GNOME&amp;rsquo;s SSO&lt;/li>
&lt;li>Retire services such as GNOME&amp;rsquo;s wiki (MoinMoin, a static copy will instead be made available), NSD (authoritative DNS servers were outsourced and replaced with ClouDNS and GitHub&amp;rsquo;s pipelines for DNS RRs updates), Nagios, Prometheus Blackbox (replaced by ClouDNS endpoints monitoring service), Ceph (replaced by EBS, EFS, S3)&lt;/li>
&lt;li>Migrate smtp.gnome.org to OSCI in order to maintain current public IP&amp;rsquo;s reputation&lt;/li>
&lt;/ol>
&lt;p>And benefits of running GNOME&amp;rsquo;s services in AWS:&lt;/p>
&lt;ol>
&lt;li>Scalability, we can easily scale up our worker nodes pool&lt;/li>
&lt;li>We run our services on top of AWS SDN and can easily create networks, routing tables, benefit from faster connectivity options, redundant networking infrastructure&lt;/li>
&lt;li>Use EBS/EFS, don&amp;rsquo;t have to maintain a self-managed Ceph cluster, easily scale volumes IOPS&lt;/li>
&lt;li>Use a local to-the-VPC load balancer, less latency for traffic to flow between the frontend and our VPC&lt;/li>
&lt;li>Have access to AWS services such as AWS Shield for advanced DDOS protection (with one bringing down GNOME&amp;rsquo;s GitLab just a week ago)&lt;/li>
&lt;/ol>
&lt;p>I&amp;rsquo;d like to thank AWS (Tom &amp;ldquo;spot&amp;rdquo; Callaway, Mila Zhou) for their sponsorship and the massive opportunity they are giving to the GNOME&amp;rsquo;s Infrastructure to improve and provide resilient, stable and highly available workloads to GNOME&amp;rsquo;s users and contributors base. And a big thank you to Red Hat for the continued sponsorship over more than 15 years on making the GNOME&amp;rsquo;s Infrastructure run smoothly and efficiently, it&amp;rsquo;s crucial for me to emphatise how critical Red Hat&amp;rsquo;s long term support has been.&lt;/p></description><link>https://www.dragonsreach.it/2024/11/16/gnome-infrastructure-migration-to-aws/</link><guid>https://www.dragonsreach.it/2024/11/16/gnome-infrastructure-migration-to-aws/</guid><pubDate>Wed, 16 Oct 2024 20:25:12 -0400</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>2022 GNOME Infrastructure Annual Review</title><description>&lt;ul>
&lt;li>&lt;a href="#1-introduction">1. Introduction&lt;/a>&lt;/li>
&lt;li>&lt;a href="#2-achievements">2. Achievements&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#21-major-achievements">2.1. Major achievements&lt;/a>&lt;/li>
&lt;li>&lt;a href="#22-minor-achievements">2.2. Minor achievements&lt;/a>&lt;/li>
&lt;li>&lt;a href="#23-our-brand-new-and-renewed-partnerships">2.3. Our brand new and renewed partnerships&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#3-highlights">3. Highlights&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#31-openshift-4-architecture">3.1. Openshift 4: architecture&lt;/a>&lt;/li>
&lt;li>&lt;a href="#32-openshift-4-virtualization-networking">3.2. Openshift 4: virtualization networking&lt;/a>&lt;/li>
&lt;li>&lt;a href="#33-openshift-4-image-builds">3.3. Openshift 4: image builds&lt;/a>&lt;/li>
&lt;li>&lt;a href="#34-openshift-4-cluster-backups">3.4. Openshift 4: cluster backups&lt;/a>&lt;/li>
&lt;li>&lt;a href="#35-gitlab-on-openshift-4-setup">3.5. GitLab on Openshift 4: setup&lt;/a>&lt;/li>
&lt;li>&lt;a href="#36-gitlab-on-openshift-4-early-days">3.6. GitLab on Openshift 4: early days&lt;/a>&lt;/li>
&lt;li>&lt;a href="#37-gitlab-on-openshift-4-logging">3.7. GitLab on Openshift 4: logging&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#future-plans">Future plans&lt;/a>&lt;/li>
&lt;li>&lt;a href="#expressing-my-gratitude">Expressing my gratitude&lt;/a>&lt;/li>
&lt;/ul>
&lt;h2 id="1-introduction">1. Introduction&lt;/h2>
&lt;p>I believe it&amp;rsquo;s kind of vital for the GNOME Infrastructure Team to outline not only the amazing work that was put into place throughout the year, but also the challenges we faced including some of the architectural designs we implemented over the past 12 months. This year has been extremely challenging for multiple reasons, the top one being Openshift 3 (which we deployed in 2018) going EOL in June 2022. We also wanted to make sure we were keeping up with OS currency, specifically finalizing the migration of all our VM-based workloads to RHEL 8 and most importantly to Ansible. The main challenges there being adapting our workflow away from the Source-To-Image (s2i) mechanism into building our own infrastructure images directly through GitLab CI/CD pipelines by ideally also dropping the requirement of hosting an internal containers registry.&lt;/p>
&lt;p>With the community also deciding to move away from Mailman, we also had an hard deadline to finalize the migration to Discourse that was started back in 2018. At the same time the GNOME community was also looking towards a better Matrix to IRC integration while irc.gimp.org (GIMPNet) showing aging sypmptoms and being put into very low maintenance mode due to a lack of IRC operators/admins time.&lt;/p>
&lt;h2 id="2-achievements">2. Achievements&lt;/h2>
&lt;p>A list of each of the individual tasks and projects we were able to fulfill during 2022. This particular section will be particularly long but I want to stress the importance of each of these items and the efforts we put in to make sure they were delivered in a timely manner. A subset of these tasks will also receive further explanation in the sections to come.&lt;/p>
&lt;h3 id="21-major-achievements">2.1. Major achievements&lt;/h3>
&lt;ol>
&lt;li>Architected, deployed, operated an Openshift 4 cluster that replaced our former OCP 3 installation. This was a major undertaking that required a lot of planning, testing and coordination with the community. We also had to make sure we were able to migrate all our existing tenants from OCP 3 to OCP 4. A total of 46 tenants were migrated and/or created during these past 12 months.&lt;/li>
&lt;li>Developed a brand new workflow moving away from Source-To-Image (s2i) and towards GitLab CI/CD pipelines.&lt;/li>
&lt;li>Migrated from individual NSD based internal resolvers to FreeIPA&amp;rsquo;s self managed BIND resolvers, this gave us the possibility to use APIs and other IPA&amp;rsquo;s goodies to manage our internal DNS views.&lt;/li>
&lt;li>For existing virtual machines we wanted to keep around, we leveraged the Openshift Virtualization operator which allows you to benefit from kubevirt&amp;rsquo;s features and effectively run virtual machines from within an OCP cluster. That also includes support for VM templates and automatic bootstraping of VMs out of existing PVCs and/or externally hosted ISO files.&lt;/li>
&lt;li>We &lt;a href="https://gitlab.gnome.org/Infrastructure/openshift-images/accounts-automation-app">developed and deployed automation&lt;/a> for handling Membership and Accounts related requests. The &lt;a href="https://wiki.gnome.org/Infrastructure/NewAccounts">documentation&lt;/a> has also been updated accordingly.&lt;/li>
&lt;li>GitLab was migrated from a monolithic virtual machine to Openshift.&lt;/li>
&lt;li>We introduced DKIM/SPF support for GNOME Foundation members, please see my &lt;a href="https://discourse.gnome.org/t/announcement-upcoming-changes-to-gnomes-mail-infrastructure/11568">announcement&lt;/a> for a list of changes.&lt;/li>
&lt;li>We rebuilt all the VMs that were not retired due to the migration to OCP to RHEL 8&lt;/li>
&lt;li>We successfully migrated away from our Puppet infrastructure to Ansible (and Ansible collections). This particular task is a major milestone, Puppet has been around the GNOME Infrastructure for more than 15 years.&lt;/li>
&lt;/ol>
&lt;h3 id="22-minor-achievements">2.2. Minor achievements&lt;/h3>
&lt;ol>
&lt;li>Identified root cause and blocked a brute force attempt (from 600+ unique IP addresses) against our LDAP database directory. Some of you surely remind the times where you found your GNOME Account locked and you were unsure around why that was happening. This was the exact reason why. A particular remediation was applied temporarily, that also had the side effect of blocking HTTPs based git clones/pushes. That was resolved by moving GitLab to OpenID (via Keycloak) and using token based authentication.&lt;/li>
&lt;li>We moved static.gnome.org to S3 and put our CDN in front of it.&lt;/li>
&lt;li>Re-deployed bastion01, nsd01, nsd02, smtp, master, logs (rsyslog) using Ansible (this also includes building Ansible roles and replicating what we had in Puppet to Ansible)&lt;/li>
&lt;li>Deployed Minio S3 GW (cache) to avoid incurring in extra AWS S3 costs&lt;/li>
&lt;li>Deprecated OpenVPN in favor of Wireguard&lt;/li>
&lt;li>We deprecated GlusterFS entirely and completed our migration to Ceph RBD + CephFS&lt;/li>
&lt;li>We retired several VM based workloads that were either migrated to Openshift or superseded including reverse proxies, Puppet masters, GitLab, the entirety of OCP 3 virtual machines (with OCP 4 being installed on bare metals directly)&lt;/li>
&lt;li>Configured blackbox Prometheus exporter and moved services availability checks to it&lt;/li>
&lt;li>We retired people.gnome.org, barely used by anyone due to the multitude of alternatives we currently provide when it comes to host GNOME related files including GitLab pages, static.gnome.org, GitLab releases, Nextcloud.&lt;/li>
&lt;li>Started ingesting Prometheus metrics into our existing Prometheus cluster via federation, a wide set of dashboards were also created to keep track of the status of OCP, Ceph, general OS related metrics and databases.&lt;/li>
&lt;li>We migrated our databases to OCP: Percona MySQL operator, Crunchy PostgreSQL operator&lt;/li>
&lt;li>Rotated KSK and ZSK DNSSEC keys on gnome.org, gtk.org, gimp.{org,net} domains&lt;/li>
&lt;li>We migrated from obtaining Let&amp;rsquo;s Encrypt certificates using getssl to OCP CertManager operator. For edge routers, we migrated to certbot and deployed specific hooks to automate the handling of DNS-01 challenges.&lt;/li>
&lt;li>We migrated GIMP downloads from a plain httpd setup to use mirrorbits to match what the GNOME Project is operating.&lt;/li>
&lt;li>We deployed AAP (Red Hat Ansible Automation Platform) in order to be able to recreate hourly configuration management runs as we had before with Puppet. These runs are particularly crucial as they make sure the latest content from our Ansible repository is pulled and enforced across all the systems Ansible manages.&lt;/li>
&lt;li>&lt;a href="https://discourse.gnome.org/t/gnome-moves-away-from-gimpnet-on-nov-25-15-00-utc/12046">irc.gnome.org migration to Libera.Chat&lt;/a>, thanks Thibault Martin and Element for the amazing continued efforts supporting GNOME&amp;rsquo;s Matrix to IRC bridge integration!&lt;/li>
&lt;li>Migrated away from Mailman to Discourse. This particular item has been part of community discussions since 2018, after evaluation by the community itself and the GNOME Project governance the migration to Discourse started and was finalized this year, please read here for a list of &lt;a href="https://discourse.gnome.org/t/common-questions-re-mailman-to-discourse/11841">FAQs&lt;/a>.&lt;/li>
&lt;li>We introduced OpenID authentication (via Keycloak) to help resolve the fragmentation multiple different authentication backends were causing.&lt;/li>
&lt;li>We introduced &lt;a href="https://hedgedoc.gnome.org/">Hedgedoc&lt;/a>, an Etherpad replacement.&lt;/li>
&lt;li>We enhanced our Splunk cluster with additional dashboards, log based alerts, new sourcetypes&lt;/li>
&lt;li>We deprecated MeetBot (unused since several years) and CommitsBot, which we replaced with a beta Matrix bot called &lt;a href="https://matrix-org.github.io/matrix-hookshot/latest/">Hookshot&lt;/a>, which leverages GitLab webhooks in order to send notifications to Matrix rooms&lt;/li>
&lt;li>We upgraded FreeIPA to version 4.9.10, and on RHEL 8. We enhanced IPA backups to include hourly file system snapshots (on top of the existing rdiff-backup runs) and daily ipa-backup runs.&lt;/li>
&lt;li>&lt;a href="https://www.dragonsreach.it/files/guadec-reports/guadec-2022.html">We presented at GUADEC 2022&lt;/a>.&lt;/li>
&lt;/ol>
&lt;h3 id="23-our-brand-new-and-renewed-partnerships">2.3. Our brand new and renewed partnerships&lt;/h3>
&lt;ol>
&lt;li>Red Hat kindly sponsored subscriptions for RHEL, Ceph, Openshift, AAP&lt;/li>
&lt;li>Splunk doubled the sponsorship to a total of 10GB/day traffic&lt;/li>
&lt;li>AWS confirmed their partnership with a total of 5k USD credit&lt;/li>
&lt;li>CDN77 provided unlimited bandwidth / traffic on their CDN offering&lt;/li>
&lt;li>We&amp;rsquo;re extremely close to finalize our partnership with Fastly! They&amp;rsquo;ll be providing us with their Traffic Load Balancing product&lt;/li>
&lt;li>and thanks to OSUOSL, Packet, DigitalOcean for the continued hosting and sponsorship of a set of GitLab runners, virtual machines and ARM builders!&lt;/li>
&lt;/ol>
&lt;h2 id="3-highlights">3. Highlights&lt;/h2>
&lt;p>Without going too much deep into technical details I wanted to provide an overview of how we architected and deployed our Openshift 4 cluster and GitLab as these questions pop up pretty frequently among contributors.&lt;/p>
&lt;h3 id="31-openshift-4-architecture">3.1. Openshift 4: architecture&lt;/h3>
&lt;p>The cluster is currently setup with a total of 3 master nodes having similar specs (256G of RAM, 62 Cores, 2x10G NICs, 2x1G NICs) and acting in a hyperconverged setup. That implies we&amp;rsquo;re also hosting a Ceph cluster (in addition to the existing one we setup a while back) we deployed via the Red Hat Openshift Data Foundations operator on the same nodes. OCP (release 4.10), in this scenario, is deployed directly on bare metal with an ingress endpoint per individual node. The current limitation to this particular architecture is there&amp;rsquo;s no proper load balancing (other than DNS Round Robin) in front of these ingresses due to the fact external load balancers are particularly expensive. As I&amp;rsquo;ve mentioned earlier we&amp;rsquo;re really close to finalize a partnership with Fastly to help fill the gap here. These nodes receive their configuration using Ignition, we made sure a specific set of MachineConfigs is there to properly configure these systems once they boot due to the way CoreOS works in this regard. At boot time, it fetches an Ignition definition from the Machine Config Operator controller and applies it to the target node.&lt;/p>
&lt;h3 id="32-openshift-4-virtualization-networking">3.2. Openshift 4: virtualization networking&lt;/h3>
&lt;p>As I previously mentioned we&amp;rsquo;ve been leveraging OCP CNV to support our VM based workloads. I wanted to quickly highlight how we handled the configuration of our internal and public networks in order for these VMs to successfully consume these subnets and be able to communicate back and forth with other data center resources and services:&lt;/p>
&lt;ol>
&lt;li>A set of bonded interfaces was setup for both the 10G and the 1G NICs&lt;/li>
&lt;li>A bridge was configured on top of these bonded interfaces, that is required by Openshift Multus to effectively append the VM interfaces to each of these bridges depending what kind of subnet(s) they&amp;rsquo;re required to access&lt;/li>
&lt;li>We configured OCP Multus (bridge mode) and its dependant NetworkAttachmentDefinition&lt;/li>
&lt;li>From within an OCP CNV CRD, pass in:&lt;/li>
&lt;/ol>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">networks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">multus&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">networkName&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">internal-network&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">nic-1&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>And a sample of the internal-network NetworkAttachmentDefinition:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="nt">apiVersion&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">k8s.cni.cncf.io/v1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">kind&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">NetworkAttachmentDefinition&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">metadata&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">internal-network&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">namespace&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">infrastructure&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">spec&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">config&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">&amp;gt;-&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> {&amp;#34;name&amp;#34;:&amp;#34;internal-network&amp;#34;,&amp;#34;cniVersion&amp;#34;:&amp;#34;0.3.1&amp;#34;,&amp;#34;plugins&amp;#34;:[{&amp;#34;type&amp;#34;:&amp;#34;cnv-bridge&amp;#34;,&amp;#34;bridge&amp;#34;:&amp;#34;br0-internal&amp;#34;,&amp;#34;mtu&amp;#34;:9000,&amp;#34;ipam&amp;#34;:{}},{&amp;#34;type&amp;#34;:&amp;#34;cnv-tuning&amp;#34;}]}&lt;/span>&lt;span class="w"> &lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;h3 id="33-openshift-4-image-builds">3.3. Openshift 4: image builds&lt;/h3>
&lt;p>One of the major changes we implemented with the migration to OCP 4 was the way we built infrastructure related container images. In early days we were leveraging the s2i OCP feature which allowed building images out of a git repository, those builds were directly happening from within OCP worker nodes and pushed to the internal OCP registry. With the new setup what happens instead is:&lt;/p>
&lt;ol>
&lt;li>We create a new git repository containing an application and an associated Dockerfile&lt;/li>
&lt;li>From within that repository, we define a .gitlab-ci.yml file that inherits the build templates from a &lt;a href="https://gitlab.gnome.org/Infrastructure/openshift-images/ci-templates">common set of templates &lt;/a> we created&lt;/li>
&lt;li>The image is then built using GitLab CI/CD and pushed to quay.io&lt;/li>
&lt;li>On the target OCP tenant, we define an ImageStream and point it to the quay.io registry namespace/image combination&lt;/li>
&lt;li>From there the Deployment/DeploymentConfig resource is updated to re-use the previously created ImageStream, whenever the ImageStream changes, the deployment/deploymentconfig is triggered (via ImageChange triggers)&lt;/li>
&lt;/ol>
&lt;h3 id="34-openshift-4-cluster-backups">3.4. Openshift 4: cluster backups&lt;/h3>
&lt;p>When it comes to cluster backups we decided to take the following approach:&lt;/p>
&lt;ol>
&lt;li>&lt;a href="https://gitlab.gnome.org/Infrastructure/openshift-images/openshift-etcd-backup">Run daily etcd backups&lt;/a>&lt;/li>
&lt;li>Backup and dump all the tenants CRDs as json files to an encrypted S3 bucket using Velero&lt;/li>
&lt;/ol>
&lt;h3 id="35-gitlab-on-openshift-4-setup">3.5. GitLab on Openshift 4: setup&lt;/h3>
&lt;p>Moving away from hosting GitLab on a monolithic virtual machine was surely one of our top goals for 2022. The reason was particularly simple, anytime we needed to perform a maintenance we were required to cause a service downtime, even during a plain minor platform upgrade. On top of that, we couldn&amp;rsquo;t easily scale the cluster in case of sudden peeks in traffic, but generally when we originally designed our GitLab offering back in 2018 we missed a lot of the goodies OCP provides, the installation has worked well during all these years but the increasing usage of the service, the multitude of new GitLab sub-components made us rethink the way we had to design this particular offering to the community.&lt;/p>
&lt;p>These are the main reasons why we migrated GitLab to OCP using the GitLab OCP Operator. Using operator&amp;rsquo;s built-in declarative resources functionalities we could easily replicate our entire cluster config in a single yaml file, the operator at that point picked up each of our definitions and generated individual configmaps, deployments, scaleapps, services, routes and associated CRDs automatically. The only component we decided to not host via OCP directly but use a plain VM on OCP Virt was gitaly. The reason is particularly simple: gitaly requires port 22 to be accessible from outside of the cluster, that is currently not possible with the default haproxy based OCP ingress. We analyzed whether it made sense to deploy the NGINX ingress which also supports reverse proxying non-HTTP ports, we thought that&amp;rsquo;d have added additional complexity with no particular benefit. MetalL was also a possibility but the product itself is still a WIP, required sending out gARPs on the public network block we
share with other community tenants for L2, for L3 there was a need to setup BGP peering between each of the individual nodes (speakers using MetalLB terminology) and an adiacent router, overkill for a single VIP.&lt;/p>
&lt;h3 id="36-gitlab-on-openshift-4-early-days">3.6. GitLab on Openshift 4: early days&lt;/h3>
&lt;p>Right after the migration we started observing some instability with specific pods (webservice, sidekiq) backtracing after a few hours they were running, specifically:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-shell" data-lang="shell">&lt;span class="line">&lt;span class="cl">Nov &lt;span class="m">16&lt;/span> 05:08:03 master2.openshift4.gnome.org kernel: cgroup: fork rejected by pids controller in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69234f3_8596_477c_b7ea_5b51f6d86cce.slice/crio-d36391a108570d6daecf316d6d19ffc6650a3fa3a82ee616944b9e51266c901f.scope&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>also on kubepods.slice:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-shell" data-lang="shell">&lt;span class="line">&lt;span class="cl">&lt;span class="o">[&lt;/span>core@master2 ~&lt;span class="o">]&lt;/span>$ cat /sys/fs/cgroup/pids/kubepods.slice/pids.max
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">4194304&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>It was clear the target pods were spawning a major set of new processes that were remaining around for the entire pod lifetime:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-shell" data-lang="shell">&lt;span class="line">&lt;span class="cl">$ ps aux &lt;span class="p">|&lt;/span> grep gpg &lt;span class="p">|&lt;/span> wc -l
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">773&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>And a sample out of the previous &amp;lsquo;ps aux&amp;rsquo; run:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-shell" data-lang="shell">&lt;span class="line">&lt;span class="cl">git &lt;span class="m">19726&lt;/span> 0.0 0.0 &lt;span class="m">0&lt;/span> &lt;span class="m">0&lt;/span> ? Z 09:58 0:00 &lt;span class="o">[&lt;/span>gpg&lt;span class="o">]&lt;/span> &amp;lt;defunct&amp;gt;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">git &lt;span class="m">19728&lt;/span> 0.0 0.0 &lt;span class="m">0&lt;/span> &lt;span class="m">0&lt;/span> ? Z 09:58 0:00 &lt;span class="o">[&lt;/span>gpg&lt;span class="o">]&lt;/span> &amp;lt;defunct&amp;gt;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">git &lt;span class="m">19869&lt;/span> 0.0 0.0 &lt;span class="m">0&lt;/span> &lt;span class="m">0&lt;/span> ? Z 10:06 0:00 &lt;span class="o">[&lt;/span>gpg&lt;span class="o">]&lt;/span> &amp;lt;defunct&amp;gt;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">git &lt;span class="m">19871&lt;/span> 0.0 0.0 &lt;span class="m">0&lt;/span> &lt;span class="m">0&lt;/span> ? Z 10:06 0:00 &lt;span class="o">[&lt;/span>gpg&lt;span class="o">]&lt;/span> &amp;lt;defunct&amp;gt;&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>It appears this specific bug was troubleshooted &lt;a href="https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/3989">already&lt;/a> by the GitLab Infrastructure Team around 4 years ago already. This misbehaviour is related to the intimate nature of GnuPG which requires calling its binaries (gpgconf, gpg, gpgsm, gpg-agent) for every required operation GitLab (webservice or sidekiq) asks it to perform. For some reason these processes never notified their parent process (PID 1 on that particular container) with a SIGCHLD and remained hanging around on the pods until pod&amp;rsquo;s dismissal. We&amp;rsquo;re in touch with the GitLab Open Source program support to understand next steps in order to have a fix implemented upstream.&lt;/p>
&lt;h3 id="37-gitlab-on-openshift-4-logging">3.7. GitLab on Openshift 4: logging&lt;/h3>
&lt;p>As part of our intent to migrate as much services as possible to our centralized rsyslog cluster (which then injects those logs into Splunk) we decided to approach GitLab&amp;rsquo;s logging on OCP this way:&lt;/p>
&lt;ol>
&lt;li>We mounted a shared PVC on each of the webservice/sidekiq pods, the target directory was the one GitLab was expected to send its logs by default (/var/log/gitlab)&lt;/li>
&lt;li>From there we deployed a separate rsyslogd deployment that was also mounting the same shared PVC&lt;/li>
&lt;li>We configured rsyslogd to relay those logs to our centralized rsyslog facility making sure proper facilities, tags, severities were also forwarded as part of the process&lt;/li>
&lt;li>Relevant configs, Dockerfile and associated deployment files are &lt;a href="https://gitlab.gnome.org/Infrastructure/openshift-images/gitlab-rsyslog">publicly available&lt;/a>&lt;/li>
&lt;/ol>
&lt;h2 id="future-plans">Future plans&lt;/h2>
&lt;p>Some of the tasks we have planned for the upcoming months:&lt;/p>
&lt;ol>
&lt;li>Move away from ftpadmin and replace it with a web application and/or CLI to securely install a sources tarball without requiring shell access. (Also introduced tarball signatures?)&lt;/li>
&lt;li>Introduce OpenID on GNOME&amp;rsquo;s Matrix homeserver, merge existing Foundation member accounts&lt;/li>
&lt;li>Migrate OCP ingress endpoints to Fastly LBs&lt;/li>
&lt;li>Upgrade Ceph to Ceph 5&lt;/li>
&lt;li>Look at migrating OCP to OVNKubernetes to start supporting IPv6 endpoints (again) - (minor priority)&lt;/li>
&lt;li>Load balance IPA&amp;rsquo;s DNS and LDAPs traffic (minor priority)&lt;/li>
&lt;li>Migrate GitLab runners Ansible roles and playbooks to AAP (minor priority)&lt;/li>
&lt;/ol>
&lt;h2 id="expressing-my-gratitude">Expressing my gratitude&lt;/h2>
&lt;p>I wanted to take a minute to thank all the individuals who helped us accomplishing this year amazing results! And a special thank you to Bartłomiej Piotrowski for his precious insights, technical skills and continued friendship.&lt;/p></description><link>https://www.dragonsreach.it/2022/12/14/gnome-infrastructure-annual-review/</link><guid>https://www.dragonsreach.it/2022/12/14/gnome-infrastructure-annual-review/</guid><pubDate>Mon, 12 Dec 2022 17:53:26 +0100</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>GNOME Infrastructure updates</title><description>&lt;p>As you may have noticed from outage and maintenance notes we sent out last week the GNOME Infrastructure has been undergoing a major redesign due to the need of moving to a different datacenter. It&amp;rsquo;s probably a good time to update the Foundation membership, contributors and generally anyone consuming the multitude of services we maintain of what we&amp;rsquo;ve been up to during these past months.&lt;/p>
&lt;h2 id="new-data-center">New Data Center&lt;/h2>
&lt;p>One of the core projects for 2020 was moving off services from the previous DC we were in (located in PHX2, Arizona) over to the Red Hat community cage located in RAL3. This specific task was made possible right after we received a new set of machines that allowed us to refresh some of the ancient hardware we had (with the average box dating back to 2013). The new layout is composed of a total of 5 (five) bare metals and 2 (two) core technologies: Openshift (v. 3.11) and Ceph (v. 4).&lt;/p>
&lt;p>The major improvements that are worth being mentioned:&lt;/p>
&lt;ol>
&lt;li>VMs can be easily scheduled across the hypervisors stack without having to copy disks over across hypervisors themselves. VM disks and data is now hosted within Ceph.&lt;/li>
&lt;li>IPv6 is available (not yet enabled/configured at the OS, Openshift router level)&lt;/li>
&lt;li>Overall better external internet uplink bandwidth&lt;/li>
&lt;li>Most of the VMs that we had running were turned into pods and are now successfully running from within Openshift&lt;/li>
&lt;/ol>
&lt;h2 id="rhel-8-and-ansible">RHEL 8 and Ansible&lt;/h2>
&lt;p>One of the things we had to take into account was running Ceph on top of RHEL 8 to benefit from its containarized setup. This originally presented itself as a challenge due to the fact RHEL 8 ships with a much newer Puppet release than the one RHEL 7 provides. At the same time we didn&amp;rsquo;t want to invest much time in upgrading our Puppet code base due to the amount of VMs we were able to migrate to Openshift and to the general willingess of slowly moving to use Ansible (client-side, no more need of maintaining server side pieces). On this specific regard we:&lt;/p>
&lt;ol>
&lt;li>Landed support for RHEL 8 provisioning&lt;/li>
&lt;li>Started experimenting with Image Based deployments (much more faster than Cobbler provisioning)&lt;/li>
&lt;li>Cooked a set of &lt;a href="https://gitlab.gnome.org/Infrastructure/ansible/-/tree/master/roles">base Ansible roles&lt;/a> to support our RHEL 8 installs including IDM, chrony, Satellite, Dell OMSA , NRPE etc.&lt;/li>
&lt;/ol>
&lt;h2 id="openshift">Openshift&lt;/h2>
&lt;p>As &lt;a href="https://www.dragonsreach.it/2018/10/18/2018-10-18-gnome-infrastructure-moving-to-openshift">originally announced&lt;/a>, the migration to the Openshift Container Platform (OSCP) has progressed and we now count a total of 34 tenants (including the entirety of GIMP websites). This allowed us to:&lt;/p>
&lt;ol>
&lt;li>Retire running VMs and prevented the need to upgrade their OS whenever they&amp;rsquo;re close to EOL. Also, in general, less maintenance burden&lt;/li>
&lt;li>Allow the community to easily provision services on top of the platform with total autonomy by choosing from a wide variety of frameworks, programming languages and database types (currently Galera and PSQL, both managed outside of OSCP itself)&lt;/li>
&lt;li>Easily scale the platform by adding more nodes/masters/routers whenever that is made necessary by additional load&lt;/li>
&lt;li>Data replicated and made redundant across a GlusterFS cluster (next on the list will be introducing Ceph support for pods persistent storage)&lt;/li>
&lt;li>Easily set up services such as Rocket.Chat and Discourse without having to mess much around with Node.JS or Ruby dependencies&lt;/li>
&lt;/ol>
&lt;h2 id="special-thanks">Special thanks&lt;/h2>
&lt;p>I&amp;rsquo;d like to thank Bartłomiej Piotrowski for all the efforts in helping me out with the migration during the past couple of weeks and Milan Zink from the Red Hat Storage Team who helped out reviewing the Ceph infrastructure design and providing useful information about possible provisioning techniques.&lt;/p></description><link>https://www.dragonsreach.it/2020/03/30/2020-03-30-gnome-infrastructure-updates/</link><guid>https://www.dragonsreach.it/2020/03/30/2020-03-30-gnome-infrastructure-updates/</guid><pubDate>Mon, 30 Mar 2020 12:20:23 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>The GNOME Infrastructure is moving to Openshift</title><description>&lt;p>During GUADEC 2018 we &lt;a href="https://www.dragonsreach.it/2018/07/30/back-from-guadec-2018">announced&lt;/a> one of the core plans of this and the coming year: it being moving as many GNOME web applications as possible to the GNOME Openshift instance we architected, deployed and configured back in July. Moving to Openshift will allow us to:&lt;/p>
&lt;ol>
&lt;li>Save up on resources as we&amp;rsquo;re deprecating and decommissioning VMs only running a single service&lt;/li>
&lt;li>Allow app maintainers to use the most recent Python, Ruby, preferred framework or programming language release without being tied to the release RHEL ships with&lt;/li>
&lt;li>Additional layer of security: containers&lt;/li>
&lt;li>Allow app owners to modify and publish content without requiring external help&lt;/li>
&lt;li>Increased apps redundancy, scalability, availability&lt;/li>
&lt;li>Direct integration with any VCS that ships with webhooks support as we can trigger the Openshift provided endpoint whenever a commit has occurred to generate a new build / deployment&lt;/li>
&lt;/ol>
&lt;h2 id="architecture">Architecture&lt;/h2>
&lt;p>The cluster consists of 3 master nodes (controllers, api, etcd), 4 compute nodes and 2 infrastructure nodes (internal docker registry, cluster console, haproxy-based routers, SSL edge termination). For the persistent storage we&amp;rsquo;re currently making good use of the Red Hat Gluster Storage (RHGS) product that Red Hat is kindly sponsoring together with the Openshift subscriptions. For any app that might require a database we have an external (as not managed as part of Openshift) fully redundant, synchronous, multi-master MariaDB cluster based on Galera (2 data nodes, 1 arbiter).&lt;/p>
&lt;p>The release we&amp;rsquo;re currently running is the &lt;a href="https://docs.openshift.com/container-platform/3.11/release_notes/ocp_3_11_release_notes.html">recently released&lt;/a> 3.11, which comes with the so-called &amp;ldquo;Cluster Console&amp;rdquo;, a web UI that allows you to manage a wide set of the underlying objects that previously were only available to the oc cli client and with a set of Monitoring and Metrics toolings (Prometheus, Grafana) that can be accessed as part of the Cluster Console (Grafana dashboards that show how the cluster is behaving) or externally via their own route.&lt;/p>
&lt;h2 id="ssl-termination">SSL Termination&lt;/h2>
&lt;p>The SSL termination is currently happening on the edge routers via a wildcard certificate for the gnome.org and guadec.org zones. The process of renewing these certificates is automated via Puppet as we&amp;rsquo;re using Let&amp;rsquo;s Encrypt behind the scenes (domain verification for the wildcard certs happen at the DNS level, we built &lt;a href="https://gitlab.gnome.org/Infrastructure/sysadmin-bin/tree/master/letsencrypt">specific hooks&lt;/a> in order to make that happen via the &lt;a href="https://github.com/srvrco/getssl">getssl&lt;/a> tool). The backend connections are following two different paths:&lt;/p>
&lt;ol>
&lt;li>edge termination with no re-encryption in case of pods containing static files (no logins, no personal information ever entered by users): in this case the traffic is encrypted between the client and the edge routers, plain text between the routers and the pods (as they&amp;rsquo;re running on the same local broadcast domain)&lt;/li>
&lt;li>re-encrypt for any service that requires authentication or personal information to be entered for authorization: in this case the traffic is encrypted from end to end&lt;/li>
&lt;/ol>
&lt;h2 id="app-migrations">App migrations&lt;/h2>
&lt;p>App migrations have started already, we&amp;rsquo;ve successfully migrated and deprecated a set of GUADEC-related web applications, specifically:&lt;/p>
&lt;ol>
&lt;li>$year.guadec.org where $year spaces from 2013 to 2019&lt;/li>
&lt;li>wordpress.guadec.org has been deprecated&lt;/li>
&lt;/ol>
&lt;p>We&amp;rsquo;re currently working on migrating the GNOME Paste website making sure we also replace the current unmaintained software to a &lt;a href="https://github.com/LINKIWI/modern-paste">supported one&lt;/a>. Next on the list will be the Wordpress-based websites such as &lt;a href="https://www.gnome.org">www.gnome.org&lt;/a> and blogs.gnome.org (Wordpress Network). I&amp;rsquo;d like to thank the GNOME Websites Team and specifically &lt;strong>Tom Tryfonidis&lt;/strong> for taking the time to migrate existing assets to the new platform as part of the GNOME websites refresh program.&lt;/p></description><link>https://www.dragonsreach.it/2018/10/18/2018-10-18-gnome-infrastructure-moving-to-openshift/</link><guid>https://www.dragonsreach.it/2018/10/18/2018-10-18-gnome-infrastructure-moving-to-openshift/</guid><pubDate>Thu, 18 Oct 2018 10:27:23 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>Back from GUADEC 2018</title><description>&lt;p>Been a while since GUADEC 2018 has ended but subsequent travels and tasks reduced the time to write up a quick summary of what happened during this year&amp;rsquo;s GNOME conference. The topics I&amp;rsquo;d like to emphasize mainly are:&lt;/p>
&lt;ul>
&lt;li>We&amp;rsquo;re hiring another Infrastructure Team member&lt;/li>
&lt;li>We&amp;rsquo;ve successfully finalized the cgit to GitLab migration&lt;/li>
&lt;li>Future plans including the migration to Openshift&lt;/li>
&lt;/ul>
&lt;h2 id="gnome-foundation-hirings">GNOME Foundation hirings&lt;/h2>
&lt;p>With the recent donation of 1M the Foundation has started recruiting on a variety of different new professional roles including a new Infrastructure team member. On this side I want to make sure that although the &lt;a href="https://www.gnome.org/foundation/careers/devops-sysadmin/">job description&lt;/a> mentions the word &lt;strong>sysadmin&lt;/strong> the figure we&amp;rsquo;re looking for is a systems engineer with a proven experience on Cloud computing platforms and tools such as AWS, Openshift and generally configuration management softwares such as Puppet and Ansible. Additionally this person should prove to have a clear understanding of the network and operating system (mainly RHEL) layers.&lt;/p>
&lt;p>We&amp;rsquo;ve already identified a set of candidates and will be proceeding with interviews in the coming weeks. This doesn&amp;rsquo;t mean we&amp;rsquo;ve hired anyone already, please keep sending CVs if interested and feeling the position would match your skills and expectations.&lt;/p>
&lt;h2 id="cgit-to-gitlab">cgit to GitLab&lt;/h2>
&lt;p>As announced on several occasions the GNOME Infrastructure has &lt;a href="https://mail.gnome.org/archives/desktop-devel-list/2018-May/msg00051.html">successfully finalized the cgit to GitLab migration&lt;/a>. From a read-only view against .git directories to a fully compliant CI/CD infrastructure. The next step on this side will be deprecating Bugzilla which has already started with bugmasters turning products read-only in case they were migrated to the new platform or by identifying whether any of the not-yet-migrated products can be archived. The idea here is waiting to see zero activity on BZ in terms of new comments to existing bugs and no new bugs being submitted at all (we have redirects in place to make sure whenever enter_bug.cgi is triggered the request gets sent to the /issues/new endpoint for that specific GitLab project) and then turn the entire BZ instance into an HTML archive for posterity and to reduce the maintenance burden of keeping an instance up-to-date with upstream in terms of CVEs.&lt;/p>
&lt;p>Thanks to all the involved parties including Carlos, Javier and GitLab itself given the prompt and detailed responses they always provided to our queries. Also thanks for sponsoring our AWS activities related to GitLab!&lt;/p>
&lt;h2 id="future-plans">Future plans&lt;/h2>
&lt;p>With the service == VM equation we&amp;rsquo;ve been following for several years it&amp;rsquo;s probably time for us to move to a more scalable infrastructure. The next generation platform we&amp;rsquo;ve picked up is going to be Openshift, its benefits:&lt;/p>
&lt;ol>
&lt;li>It&amp;rsquo;s not important where a service runs behind scenes: it only has to run (VM vs pods and containers that are part of a pod that get scheduled randomly on the available Openshift nodes)&lt;/li>
&lt;li>Easily scalable in case additional resources are needed for a small period of time&lt;/li>
&lt;li>Built-in monitoring starting from Openshift 3.9 (the release we&amp;rsquo;ll be running) based on Prometheus (+ Grafana for dashboarding)&lt;/li>
&lt;li>GNOME services as containers&lt;/li>
&lt;li>Individual application developers to schedule their own builds and see their code deployed with one click directly in production&lt;/li>
&lt;/ol>
&lt;p>The base set of VMs and bare metals has been already configured. Red Hat has been so great to provide the GNOME Project with a set of Red Hat Container Platform (and GlusterFS for heketi-based brick provisioning) subscriptions. We&amp;rsquo;ll start moving over to the infrastructure in the coming weeks. It&amp;rsquo;s going to take some good time but in the end we&amp;rsquo;ll be able to free up a lot of resources and retire several VMs and related deprecated configurations.&lt;/p>
&lt;h2 id="misc">Misc&lt;/h2>
&lt;p>Slides from the Foundation AGM Infrastructure team report are available &lt;a href="https://www.dragonsreach.it/files/guadec-reports/guadec2018.html">here&lt;/a>.&lt;/p>
&lt;p>Many thanks to the GNOME Foundation for the sponsorship of my travel!&lt;/p>
&lt;p>&lt;img src="https://www.dragonsreach.it/img/2018-GUADEC-badge.png" alt="GUADEC 2018 Badge">&lt;/p></description><link>https://www.dragonsreach.it/2018/07/30/back-from-guadec-2018/</link><guid>https://www.dragonsreach.it/2018/07/30/back-from-guadec-2018/</guid><pubDate>Mon, 30 Jul 2018 10:27:23 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>Adding reCAPTCHA v2 support to Mailman</title><description>&lt;p>As a follow-up to the reCAPTCHA v1 &lt;a href="https://www.dragonsreach.it/2014/05/03/adding-recaptcha-support-to-mailman/">post&lt;/a> published back in 2014 here it comes an updated version for migrating your Mailman instance off from version 1 (being decommissioned on the 31th of March 2018) to version 2. The &lt;a href="https://pypi.python.org/pypi/recaptcha-client">original python-recaptcha library&lt;/a> was forked into &lt;a href="https://github.com/redhat-infosec/python-recaptcha">https://github.com/redhat-infosec/python-recaptcha&lt;/a> and made compatible with reCAPTCHA version 2.&lt;/p>
&lt;p>The relevant changes against the original library can be resumed as follows:&lt;/p>
&lt;ol>
&lt;li>Added ‘version=2’ against displayhtml, load_scripts functions&lt;/li>
&lt;li>Introduce the v2submit (along with submit to keep backwards compatibility) function to support reCAPTCHA v2&lt;/li>
&lt;li>The updated library is backwards compatible with version 1 to avoid unexpected code breakages for instances still running version 1&lt;/li>
&lt;/ol>
&lt;p>The required changes are located on the following files:&lt;/p>
&lt;p>&lt;strong>/usr/lib/mailman/Mailman/Cgi/listinfo.py&lt;/strong>&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Python" data-lang="Python">&lt;span class="line">&lt;span class="cl">&lt;span class="o">---&lt;/span> &lt;span class="n">listinfo&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">py&lt;/span> &lt;span class="mi">2018&lt;/span>&lt;span class="o">-&lt;/span>&lt;span class="mi">02&lt;/span>&lt;span class="o">-&lt;/span>&lt;span class="mi">26&lt;/span> &lt;span class="mi">14&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="mi">56&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="mf">48.000000000&lt;/span> &lt;span class="o">+&lt;/span>&lt;span class="mi">0000&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+++&lt;/span> &lt;span class="o">/&lt;/span>&lt;span class="n">usr&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">lib&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">mailman&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">Mailman&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">Cgi&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">listinfo&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">py&lt;/span> &lt;span class="mi">2018&lt;/span>&lt;span class="o">-&lt;/span>&lt;span class="mi">02&lt;/span>&lt;span class="o">-&lt;/span>&lt;span class="mi">26&lt;/span> &lt;span class="mi">14&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="mi">08&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="mf">34.000000000&lt;/span> &lt;span class="o">+&lt;/span>&lt;span class="mi">0000&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">@@&lt;/span> &lt;span class="o">-&lt;/span>&lt;span class="mi">31&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="mi">6&lt;/span> &lt;span class="o">+&lt;/span>&lt;span class="mi">31&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="mi">7&lt;/span> &lt;span class="o">@@&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">from&lt;/span> &lt;span class="nn">Mailman&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="n">i18n&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">from&lt;/span> &lt;span class="nn">Mailman.htmlformat&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="o">*&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">from&lt;/span> &lt;span class="nn">Mailman.Logging.Syslog&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="n">syslog&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span>&lt;span class="kn">from&lt;/span> &lt;span class="nn">recaptcha.client&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="n">captcha&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="c1"># Set up i18n&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">_&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">i18n&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">_&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">@@&lt;/span> &lt;span class="o">-&lt;/span>&lt;span class="mi">244&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="mi">6&lt;/span> &lt;span class="o">+&lt;/span>&lt;span class="mi">245&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="mi">10&lt;/span> &lt;span class="o">@@&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">replacements&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s1">&amp;#39;&amp;lt;mm-lang-form-start&amp;gt;&amp;#39;&lt;/span>&lt;span class="p">]&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">mlist&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">FormatFormStart&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;listinfo&amp;#39;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">replacements&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s1">&amp;#39;&amp;lt;mm-fullname-box&amp;gt;&amp;#39;&lt;/span>&lt;span class="p">]&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">mlist&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">FormatBox&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;fullname&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">size&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="mi">30&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="c1"># Captcha&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="n">replacements&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s1">&amp;#39;&amp;lt;mm-recaptcha-javascript&amp;gt;&amp;#39;&lt;/span>&lt;span class="p">]&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">captcha&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">displayhtml&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">mm_cfg&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">RECAPTCHA_PUBLIC_KEY&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">use_ssl&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="kc">True&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">version&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="mi">2&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="n">replacements&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s1">&amp;#39;&amp;lt;mm-recaptcha-script&amp;gt;&amp;#39;&lt;/span>&lt;span class="p">]&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">captcha&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">load_script&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">version&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="mi">2&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="c1"># Do the expansion.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">doc&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">AddItem&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">mlist&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">ParseTags&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;listinfo.html&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">replacements&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">lang&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span> &lt;span class="n">doc&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">Format&lt;/span>&lt;span class="p">()&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>&lt;strong>/usr/lib/mailman/Mailman/Cgi/subscribe.py&lt;/strong>&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Python" data-lang="Python">&lt;span class="line">&lt;span class="cl">&lt;span class="o">---&lt;/span> &lt;span class="n">subscribe&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">py&lt;/span> &lt;span class="mi">2018&lt;/span>&lt;span class="o">-&lt;/span>&lt;span class="mi">02&lt;/span>&lt;span class="o">-&lt;/span>&lt;span class="mi">26&lt;/span> &lt;span class="mi">14&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="mi">56&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="mf">38.000000000&lt;/span> &lt;span class="o">+&lt;/span>&lt;span class="mi">0000&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+++&lt;/span> &lt;span class="o">/&lt;/span>&lt;span class="n">usr&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">lib&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">mailman&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">Mailman&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">Cgi&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">subscribe&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">py&lt;/span> &lt;span class="mi">2018&lt;/span>&lt;span class="o">-&lt;/span>&lt;span class="mi">02&lt;/span>&lt;span class="o">-&lt;/span>&lt;span class="mi">26&lt;/span> &lt;span class="mi">14&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="mi">08&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="mf">18.000000000&lt;/span> &lt;span class="o">+&lt;/span>&lt;span class="mi">0000&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">@@&lt;/span> &lt;span class="o">-&lt;/span>&lt;span class="mi">32&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="mi">6&lt;/span> &lt;span class="o">+&lt;/span>&lt;span class="mi">32&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="mi">7&lt;/span> &lt;span class="o">@@&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">from&lt;/span> &lt;span class="nn">Mailman.UserDesc&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="n">UserDesc&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">from&lt;/span> &lt;span class="nn">Mailman.htmlformat&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="o">*&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kn">from&lt;/span> &lt;span class="nn">Mailman.Logging.Syslog&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="n">syslog&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span>&lt;span class="kn">from&lt;/span> &lt;span class="nn">recaptcha.client&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="n">captcha&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">SLASH&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s1">&amp;#39;/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">ERRORSEP&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s1">&amp;#39;&lt;/span>&lt;span class="se">\n\n&lt;/span>&lt;span class="s1">&amp;lt;p&amp;gt;&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">@@&lt;/span> &lt;span class="o">-&lt;/span>&lt;span class="mi">165&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="mi">6&lt;/span> &lt;span class="o">+&lt;/span>&lt;span class="mi">166&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="mi">17&lt;/span> &lt;span class="o">@@&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">results&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">append&lt;/span>&lt;span class="p">(&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">_&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;There was no hidden token in your submission or it was corrupted.&amp;#39;&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">results&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">append&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">_&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;You must GET the form before submitting it.&amp;#39;&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="c1"># recaptcha&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="n">captcha_response&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">captcha&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">v2submit&lt;/span>&lt;span class="p">(&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="n">cgidata&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">getvalue&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;g-recaptcha-response&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>&lt;span class="p">),&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="n">mm_cfg&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">RECAPTCHA_PRIVATE_KEY&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="n">remote&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="k">if&lt;/span> &lt;span class="ow">not&lt;/span> &lt;span class="n">captcha_response&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">is_valid&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span> &lt;span class="n">results&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">append&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">_&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;Invalid captcha: &lt;/span>&lt;span class="si">%s&lt;/span>&lt;span class="s1">&amp;#39;&lt;/span> &lt;span class="o">%&lt;/span> &lt;span class="n">captcha_response&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">error_code&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">+&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="c1"># Was an attempt made to subscribe the list to itself?&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="n">email&lt;/span> &lt;span class="o">==&lt;/span> &lt;span class="n">mlist&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">GetListEmail&lt;/span>&lt;span class="p">():&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">syslog&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;mischief&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s1">&amp;#39;Attempt to self subscribe &lt;/span>&lt;span class="si">%s&lt;/span>&lt;span class="s1">: &lt;/span>&lt;span class="si">%s&lt;/span>&lt;span class="s1">&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">email&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">remote&lt;/span>&lt;span class="p">)&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>
&lt;strong>/usr/lib/mailman/templates/en/listinfo.html&lt;/strong>
&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-html" data-lang="html">&lt;span class="line">&lt;span class="cl">--- listinfo.html 2018-02-26 15:02:34.000000000 +0000
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">+++ /usr/lib/mailman/templates/en/listinfo.html 2018-02-26 14:18:52.000000000 +0000
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">@@ -3,7 +3,7 @@
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">HTML&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">HEAD&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">TITLE&lt;/span>&lt;span class="p">&amp;gt;&amp;lt;&lt;/span>&lt;span class="nt">MM-List-Name&lt;/span>&lt;span class="p">&amp;gt;&lt;/span> Info Page&lt;span class="p">&amp;lt;/&lt;/span>&lt;span class="nt">TITLE&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">-
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">+ &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">MM-Recaptcha-Script&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;/&lt;/span>&lt;span class="nt">HEAD&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">BODY&lt;/span> &lt;span class="na">BGCOLOR&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">&amp;#34;#ffffff&amp;#34;&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">@@ -116,6 +116,11 @@
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;/&lt;/span>&lt;span class="nt">tr&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">mm-digest-question-end&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">tr&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">+ &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">tr&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">+ &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">td&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>Please fill out the following captcha&lt;span class="p">&amp;lt;/&lt;/span>&lt;span class="nt">td&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">+ &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">td&lt;/span>&lt;span class="p">&amp;gt;&amp;lt;&lt;/span>&lt;span class="nt">mm-recaptcha-javascript&lt;/span>&lt;span class="p">&amp;gt;&amp;lt;/&lt;/span>&lt;span class="nt">TD&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">+ &lt;span class="p">&amp;lt;/&lt;/span>&lt;span class="nt">tr&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">+ &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">tr&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">td&lt;/span> &lt;span class="na">colspan&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">&amp;#34;3&amp;#34;&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;&lt;/span>&lt;span class="nt">center&lt;/span>&lt;span class="p">&amp;gt;&amp;lt;&lt;/span>&lt;span class="nt">MM-Subscribe-Button&lt;/span>&lt;span class="p">&amp;gt;&amp;lt;/&lt;/span>&lt;span class="nt">center&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">&amp;lt;/&lt;/span>&lt;span class="nt">td&lt;/span>&lt;span class="p">&amp;gt;&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>
The updated RPMs are being rolled out to Fedora, EPEL 6 and EPEL 7. In the meantime you can find them &lt;a href="https://fedorapeople.org/~averi/RPMs/python-recaptcha-client">here&lt;/a>.
&lt;/p></description><link>https://www.dragonsreach.it/2018/02/26/adding-recaptcha-v2-support-mailman/</link><guid>https://www.dragonsreach.it/2018/02/26/adding-recaptcha-v2-support-mailman/</guid><pubDate>Mon, 26 Feb 2018 15:13:09 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>A childhood’s dream</title><description>&lt;p>Six months since my &lt;a href="https://www.dragonsreach.it/2015/12/02/three-years-and-counting/">latest blog post&lt;/a> is definitely a lot and reminds me how difficult this year has been for me in many ways. Back in June 2015 I received a job proposal as a Systems and Network Engineer from a company located in Padova, a city in the north-east part of Italy which is around 150km (around 93 miles) away from my home-town. The offer looked very interesting and I went for it. The idea I originally had was to relocate there but the extremely high costs of rents (the city is well known in Italy as one of the best places to take University courses on several faculties) and the fact I really wanted to experiment on whether the job description was actually going to match the expectations I had, I opted to not relocate at all and travel daily to work by train (a total of 4 hours per day spent travelling).&lt;/p>
&lt;p>I still recall one of my friends saying me “I am sure you won’t make it for more than two weeks” and how satisfying has been mentioning him a few days ago I &lt;strong>did make&lt;/strong> it for roughly one year. Spending around 4 hours every day on a train hasn’t been that fun but the passion and love for what I’ve been doing has helped me overcoming the difficulties I encountered. At some point something I honestly was expecting arrived: my professional grow at the company was completely stuck. While I initially told myself the moment was surely going to not persist much longer, the opposite happened.&lt;/p>
&lt;p>For a passionate, enthusiast and “hungry” person like I am that was one of the worst situations that could ever happened and it was definitely time for me to look at new opportunities. That’s where my childhood’s dream became reality: I will be joining the &lt;strong>Platform Operations Team&lt;/strong> at &lt;strong>Red Hat&lt;/strong> as a &lt;strong>System Administrator&lt;/strong> starting from mid-July! Being part of a great family which cares about Open Source and its values makes me proud and I would really like to thank Red Hat for this incredible opportunity.&lt;/p>
&lt;p>On a side note I will be in Brno between the 14th and the 16th of July, please drop me a note if you want to have a drink together!&lt;/p></description><link>https://www.dragonsreach.it/2016/07/05/a-childhood-dream/</link><guid>https://www.dragonsreach.it/2016/07/05/a-childhood-dream/</guid><pubDate>Tue, 05 Jul 2016 21:26:39 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>Three years and counting</title><description>&lt;p>It’s been a while since my last “what’s been happening behind the scenes” e-mail so I’m here to report on what has been happening within the GNOME Infrastructure, its future plans and my personal sensations about a challenge that started around three (3) years ago when &lt;a href="https://plus.google.com/115250422803614415116/posts" target="_blank">Sriram Ramkrishna&lt;/a> and &lt;a href="http://www.digitalprognosis.com/" target="_blank">Jeff Schroeder&lt;/a> proposed my name as a possible candidate for coordinating the team that runs the systems behind the GNOME Project. All this followed by the &lt;a href="https://www.gnome.org/news/2013/03/behind-the-scene-andrea-veri-is-new-gnome-part-time-sysadmin/" target="_blank">official hiring&lt;/a> achieved by &lt;a href="https://en.wikipedia.org/wiki/Karen_Sandler" target="_blank">Karen Sandler&lt;/a> back in February 2013.&lt;/p>
&lt;p>The &lt;strong>GNOME Infrastructure&lt;/strong> has finally reached stability both in terms of reliability and uptime, we didn’t have any service disruption this and the past year and services have been running smoothly as they were expected to in a project like the one we are managing.&lt;/p>
&lt;p>As many of you know service disruptions and a total lack of maintenance were very common before I joined back in 2013, I’m so glad the situation has dramatically changed and developers, users, passionates are now able to reach our websites, code repositories, build machines without experiencing slowness, downtimes or&lt;/p>
&lt;p>unreachability. Additionally all these groups of people now have a reference point they can contact in case they need help when coping with the infrastructure they daily use. The ticketing system allows users to get in touch with the members of the &lt;span class="il">Sysadmin&lt;/span> Team and receive support right away within a very short period of time (Also thanks to &lt;a href="https://www.pagerduty.com" target="_blank">Pagerduty&lt;/a>, service the Foundation is kindly sponsoring)&lt;/p>
&lt;p>Before moving ahead to the future plans I’d like to provide you a summary of what has been done during these roughly three years so you can get an idea of why I define the changes that happened to the infrastructure a complete revamp:&lt;/p>
&lt;ol>
&lt;li>Recycled several ancient machines migrating services off of them while consolidating them by placing all their configuration on our central configuration management platform ran by Puppet. This includes a grand total of 7 machines that were replaced by new hardware and extended warranties the Foundation kindly sponsored.&lt;/li>
&lt;li>We strenghten our websites security by introducing SSL certificates everywhere and recently replacing them with SHA2 certificates.&lt;/li>
&lt;li>We introduced several services such as Owncloud, the Commits Bot, the Pastebin, the Etherpad, Jabber, the GNOME Github mirror.&lt;/li>
&lt;li>We restructured the way we backup our machines also thanks to the Fedora Project sponsoring the disk space on their backup facility. The way we were used to handle backups drastically changed from early years where a magnetic tape facility was in charge of all the burden of archiving our data to today where a NetApp is used together with &lt;a href="http://www.nongnu.org/rdiff-backup/" target="_blank">rdiff-backup&lt;/a>.&lt;/li>
&lt;li>We upgraded Bugzilla to the latest release, a huge thanks goes to Krzesimir Nowak who kindly helped us building the migration tools.&lt;/li>
&lt;li>We introduced the &lt;a href="https://wiki.gnome.org/Sysadmin/Apprentices" target="_blank">GNOME Apprentice program&lt;/a> open-sourcing our internal Puppet repository and cleansing it (shallow clones FTW!) from any sensitive information which now lives on a different repository with restricted access.&lt;/li>
&lt;li>We retired Mango and our OpenLDAP instance in favor of &lt;a href="https://account.gnome.org" target="_blank">FreeIPA&lt;/a> which allows users to modify their account information on their own without waiting for the Accounts Team to process the change.&lt;/li>
&lt;li>We &lt;a href="https://wiki.gnome.org/Sysadmin/SOP" target="_blank">documented&lt;/a> how our internal tools are customized to play together making it easy for future &lt;span class="il">Sysadmin&lt;/span> Team members to learn how the infrastructure works and supersede existing members in case they aren’t able to keep up their position anymore.&lt;/li>
&lt;li>We started providing hosting to the GIMP and GTK projects which now completely rely on the GNOME Infrastructure. (DNS, email, websites and other services infrastructure hosting)&lt;/li>
&lt;li>We started providing hosting not only to the GIMP and GTK projects but to localized communities as well such as GNOME Hispano and GNOME Greece&lt;/li>
&lt;li>We configured proper monitoring for all the hosted services thanks to Nagios and Check-MK&lt;/li>
&lt;li>We migrated the IRC network to a newer ircd with proper IRC services (Nickserv, Chanserv) in place.&lt;/li>
&lt;li>We made sure each machine had a configured management (mgmt) and KVM interface for direct remote access to the bare metal machine itself, its hardware status and all the operations related to it. (hard reset, reboot, shutdown etc.)&lt;/li>
&lt;li>We &lt;a href="https://mail.gnome.org/archives/infrastructure-announce/2013-May/msg00000.html" target="_blank">upgraded MoinMoin&lt;/a> to the latest release and made a substantial cleanup of old accounts, pages marked as spam and trashed pages.&lt;/li>
&lt;li>We deployed DNSSEC for several domains we manage including gnome.org, guadec.es, gnomehispano.es, guadec.org, gtk.org and gimp.org&lt;/li>
&lt;li>We &lt;a href="https://mail.gnome.org/archives/infrastructure-announce/2014-March/msg00000.html" target="_blank">introduced an account de-activation policy&lt;/a> which comes into play when a contributor not committing to any of the hosted repositories at git.gnome.org since two years is caught by the script. The account in question is marked as inactive and the gnomecvs (from the old cvs days) and ftpadmin groups are removed.&lt;/li>
&lt;li>We planned mass reboots of all the machines roughly every month for properly applying security and kernel updates.&lt;/li>
&lt;li>We introduced &lt;a href="http://mirrorbrain.org/" target="_blank">Mirrorbrain&lt;/a> (MB), the mirroring service serving GNOME and related modules tarballs and software all over the world. Before introducing MB GNOME had several mirrors located in all the main continents and at the same time a very low amount of users making good use of them. Many organizations and companies behind these mirrors decided to not host GNOME sources anymore as the statistics of usage were very poor and preferred providing the same service to projects that really had a demand for these resources. MB solved all this allowing a proper redirect to the closest mirror (through mod_geoip) and making sure the sources checksum matched across all the mirrors and against the original tarball uploaded by a GNOME maintainer and hosted at master.gnome.org.&lt;/li>
&lt;/ol>
&lt;p>I can keep the list going for dozens of other accomplished tasks but I’m sure many of you are now more interested in what the future plans actually are in terms of where the &lt;strong>GNOME Infrastructure&lt;/strong> should be in the next couple of years.&lt;/p>
&lt;p>One of the main topics we’ve been discussing will be migrating our Git infrastructure away from cgit (which is mainly serving as a code browsing tool) to a more complete platform that is surely going to include a code review tool of some sort. (Gerrit, Gitlab, Phabricator)&lt;/p>
&lt;p>Another topic would be migrating our mailing lists to Mailman 3 / Hyperkitty. This also means we definitely need a staging infrastructure in place for testing these kind of transitions ideally bound to a separate Puppet / Ansible repository or branch. Having a different repository for testing purposes will also mean helping apprentices to test their changes directly on a live system and not on their personal computer which might be running a different OS / set of tools than the ones we run on the GNOME Infrastructure.&lt;/p>
&lt;p>What I also aim would be seeing &lt;strong>GNOME&lt;/strong> Accounts being the only authentication resource in use within the whole GNOME Infrastructure. That means one should be able to login to a specific service with the same username / password in use on the other hosted services. That’s been on my todo list for a while already and it’s probably time to push it forward together with Patrick Uiterwijk, responsible of &lt;a href="https://fedorahosted.org/ipsilon/" target="_blank">Ipsilon&lt;/a>‘s development at Red Hat and GNOME Sysadmin.&lt;/p>
&lt;p>While these are the top priority items we are soon receiving new hardware (plus extended warranty renewals for two out of the three machines that had their warranty renewed a while back) and migrating some of the VMs off from the current set of machines to the new boxes is definitely another task I’d be willing to look at in the next couple of months (one machine (ns-master.gnome.org) is being decommissioned giving me a chance to migrate away from BIND into NSD).&lt;/p>
&lt;p>The &lt;strong>GNOME Infrastructure&lt;/strong> is evolving and it’s crucial to have someone maintaining it. On this side I’m bringing to your attention the fact the assigned &lt;span class="il">Sysadmin&lt;/span> funds are running out as reported on the Board minutes from the &lt;a href="https://mail.gnome.org/archives/foundation-announce/2015-November/msg00003.html" target="_blank">27th of October&lt;/a>. On this side &lt;a href="http://jeff.ecchi.ca/" target="_blank">Jeff Fortin&lt;/a> started looking for possible sponsors and came up with the idea of making a brochure with a set of accomplished tasks that couldn’t have been possible without the &lt;a href="https://mail.gnome.org/archives/foundation-announce/2010-June/msg00000.html" target="_blank">&lt;span class="il">Sysadmin &lt;/span>fundraising campaign&lt;/a> launched by &lt;a href="https://en.wikipedia.org/wiki/Stormy_Peters" target="_blank">Stormy Peters&lt;/a> back in &lt;a href="http://stormyscorner.com/2010/03/one-step-closer-to-a-sys-admin.html" target="_blank">June 2010&lt;/a> being a success. The Board is well aware of the importance of having someone looking at the infrastructure that runs the GNOME Project and is making sure the brochure will be properly reviewed and published.&lt;/p>
&lt;p>And now some stats taken from the Puppet Git Repository:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">$ &lt;span class="nb">cd&lt;/span> /git/GNOME/puppet &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> git shortlog -ns
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">3520&lt;/span> Andrea Veri
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">506&lt;/span> Olav Vitters
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">338&lt;/span> Owen W. Taylor
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">239&lt;/span> Patrick Uiterwijk
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">112&lt;/span> Jeff Schroeder
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">71&lt;/span> Christer Edwards
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">4&lt;/span> Daniel Mustieles
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">4&lt;/span> Matanya Moses
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">3&lt;/span> Tobias Mueller
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">2&lt;/span> John Carr
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">2&lt;/span> Ray Wang
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">1&lt;/span> Daniel Mustieles García
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="m">1&lt;/span> Peter Baumgarten&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>and from the &lt;a href="https://www.bestpractical.com/rt/" target="_blank">Request Tracker&lt;/a> database (52388 being my assigned ID):&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-MySQL" data-lang="MySQL">&lt;span class="line">&lt;span class="cl">&lt;span class="n">mysql&lt;/span>&lt;span class="o">&amp;gt;&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">select&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="nf">count&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">*&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">from&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">Tickets&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">where&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">LastUpdatedBy&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;52388&amp;#39;&lt;/span>&lt;span class="p">;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="o">+----------+&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="o">|&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="nf">count&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">*&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">|&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="o">+----------+&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="o">|&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="mi">3613&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">|&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="o">+----------+&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="mi">1&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">row&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">in&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kt">set&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="mi">0&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="mi">01&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">sec&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="n">mysql&lt;/span>&lt;span class="o">&amp;gt;&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">select&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="nf">count&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">*&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">from&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">Tickets&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">where&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">LastUpdatedBy&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;52388&amp;#39;&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">and&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">Status&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;Resolved&amp;#39;&lt;/span>&lt;span class="p">;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="o">+----------+&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="o">|&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="nf">count&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">*&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">|&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="o">+----------+&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="o">|&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="mi">1596&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">|&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="o">+----------+&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="mi">1&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">row&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">in&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kt">set&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="mi">0&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="mi">03&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">sec&lt;/span>&lt;span class="p">)&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>It’s been a long run which made me proud, for the things I learnt, for the tasks I’ve been able to accomplish, for the great support the GNOME community gave me all the time and most of all for the same fact of being part of the team responsible of the systems hosting the GNOME Project. &lt;strong>Thank you&lt;/strong> GNOME community for your continued and never ending backing, we daily work to improve how the services we host are delivered to you and the support we receive back is fundamental for our passion and enthusiasm to remain high!&lt;/p>
&lt;p> &lt;/p></description><link>https://www.dragonsreach.it/2015/12/02/three-years-and-counting/</link><guid>https://www.dragonsreach.it/2015/12/02/three-years-and-counting/</guid><pubDate>Wed, 02 Dec 2015 16:32:41 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>The GNOME Infrastructure Apprentice Program</title><description>&lt;p>Many times it happened seeing someone joining the #sysadmin IRC channel requesting participation to the team after having spent around 5 minutes trying to explain what the skills and the knowledge were and why this person felt it was the right figure for the position. And it was always very disappointing for me having to reject all these requests as we just didn’t have the infrastructure in place to let new people join the rest of the team with limited privileges.&lt;/p>
&lt;p>With the introduction of &lt;a href="https://www.dragonsreach.it/2014/10/07/the-gnome-infrastructure-is-now-powered-by-freeipa/" target="_blank">FreeIPA&lt;/a>, more fine-grained ACLs (and hiera-eyaml-gpg for securing tokens, secrets, passwords out of Puppet itself) we are so glad to announce the launch of the “&lt;strong>GNOME Infrastructure Apprentice Program&lt;/strong>” (from now till the end of the post just “Program”). If you are familiar with the Fedora Infrastructure and how it works you might know what this is about already. If you don’t please read further ahead.&lt;/p>
&lt;p>The Program will allow apprentices to join the Sysadmin Team with a limited set of privileges which mainly consist in being able to access the Puppet repository and all the stored configuration files that run the machines powering the GNOME Infrastructure every day. Once approved to the Program apprentices will be able to submit patches for review to the team and finally see their work merged on the production environment if the proposed changes matched the expectations and addressed comments.&lt;/p>
&lt;p>While the Program is open to everyone to join, we have some prerequisites in place. The interested person should be:&lt;/p>
&lt;ol>
&lt;li>Part of an existing FOSS community&lt;/li>
&lt;li>Familiar with how a FOSS Project works behind the scenes&lt;/li>
&lt;li>Familiar with popular tools like Puppet, Git&lt;/li>
&lt;li>Familiar with RHEL as the OS of choice&lt;/li>
&lt;li>Familiar with popular Sysadmin tools, softwares and procedures&lt;/li>
&lt;li>Eager to learn new things, make constructive discussions with a team, provide feedback and new ideas&lt;/li>
&lt;/ol>
&lt;p>If you feel like having all the needed prerequisites and would be willing to join follow these steps:&lt;/p>
&lt;ol>
&lt;li>Subscribe to the &lt;a href="https://mail.gnome.org/mailman/listinfo/gnome-infrastructure" target="_blank">gnome-infrastructure&lt;/a> and &lt;a href="https://mail.gnome.org/mailman/listinfo/infrastructure-announce" target="_blank">infrastructure-announce&lt;/a> mailing lists&lt;/li>
&lt;li>Join the &lt;strong>#sysadmin&lt;/strong> IRC channel on irc.gnome.org&lt;/li>
&lt;li>Send a presentation e-mail to the gnome-infrastructure mailing list stating who you are, what your past experiences and plans are as an Apprentice&lt;/li>
&lt;li>Once the presentation has been sent an existing Sysadmin Team member will evaluate your application and follow-up with you introducing you to the Program&lt;/li>
&lt;/ol>
&lt;p>More information about the Program is available &lt;a href="https://wiki.gnome.org/Sysadmin/Apprentices" target="_blank">here&lt;/a>.&lt;/p></description><link>https://www.dragonsreach.it/2015/01/28/the-gnome-infrastructure-apprentice-program/</link><guid>https://www.dragonsreach.it/2015/01/28/the-gnome-infrastructure-apprentice-program/</guid><pubDate>Wed, 28 Jan 2015 16:59:46 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>Kerberos over HTTP: getting a TGT on a firewalled network</title><description>&lt;p>One of the benefits I originally wanted to bring with the &lt;a href="https://www.dragonsreach.it/2014/10/07/the-gnome-infrastructure-is-now-powered-by-freeipa/" target="_blank">FreeIPA move&lt;/a> to GNOME contributors was the introduction of an additional authentication system to connect to to the services hosted on the &lt;strong>GNOME&lt;/strong> Infrastructure. The authentication system that comes with the FreeIPA bundle that I had in mind was Kerberos. Users willing to use Kerberos as their preferred authentication system would just be required to get a TGT (Ticket-Granting Ticket) from the KDC (Key Distribution Center) through the &lt;strong>kinit&lt;/strong> command. Once done authenticating to the services currently supporting Kerberos will be as easy as pointing a configured browser (Google for how to configure your browser to use Krb logins) to account.gnome.org without being prompted for entering the usual username / password combination or pushing to git without using the public-private key mechanism. That theoretically means you won’t be required to use a SSH key for loggin in to any of the GNOME services at all as entering your password to the &lt;strong>kinit&lt;/strong> password prompt will be enough (for at least 24 hours as that’s the life of the TGT itself on our setup) for doing all you were used to do before the Kerberos support introduction.&lt;/p>
&lt;p>&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2014/10/kerberos-over-http.png" alt="A successful SSH login using the most recent Kerberos package on Fedora 21">&lt;/p>
&lt;p>The issue we faced at first was the underlying networking infrastructure firewalling all Kerberos ports blocking the use of &lt;strong>kinit&lt;/strong> itself which kept timing out reaching port 88. A few days later I was contacted by RH’s developer &lt;strong>&lt;span class="gD">Nathaniel McCallum &lt;/span>&lt;/strong>&lt;span class="gD">who worked out a way to bypass this restriction by creating a KDC proxy that accepts requests from port 443 and proxies them to the internal KDC running on port 88. With the &lt;a href="http://web.mit.edu/kerberos/krb5-1.13/" target="_blank">recent Kerberos release&lt;/a> (released on October 15th, 2014 and following the &lt;a href="http://msdn.microsoft.com/en-us/library/hh553774.aspx" target="_blank">MS-KKDCP protocol&lt;/a>) a patched &lt;strong>kinit &lt;/strong>allows users to retrieve their TGTs directly from the HTTPS proxy completely bypassing the need for port 88 to stay open on the firewall. &lt;/span>The &lt;strong>GNOME Infrastructure&lt;/strong> now runs the KDC Proxy and we’re glad to announce Kerberos authentications are working as expected on the hosted services.&lt;/p>
&lt;p>If you are facing the same problem and you are curious to know more about the setup, here they come all the details:&lt;/p>
&lt;p>On the &lt;strong>KDC&lt;/strong>:&lt;/p>
&lt;ol>
&lt;li>No changes are needed on the KDC itself, just make sure to install the &lt;strong>python-kdcproxy&lt;/strong> package which is available for RHEL 7, &lt;a href="http://koji.fedoraproject.org/koji/taskinfo?taskID=7937527" target="_blank">HERE&lt;/a>.&lt;/li>
&lt;li>Tweak your vhost accordingly by following the &lt;a href="https://github.com/npmccallum/kdcproxy" target="_blank">provided documentation&lt;/a>.&lt;/li>
&lt;/ol>
&lt;p>On the &lt;strong>client&lt;/strong>:&lt;/p>
&lt;ol>
&lt;li>Install the krb5-workstation package, make sure it’s at least version &lt;strong>1.12.2-9&lt;/strong> as that’s the release which had the additional features we are talking about backported. Right now it’s only available for Fedora 21.&lt;/li>
&lt;li>Adjust /etc/krb5.conf accordingly and finally get a TGT through kinit $userid@GNOME.ORG.&lt;/li>
&lt;/ol>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="k">[realms]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="na">GNOME.ORG&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">{
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> kdc = https://account.gnome.org/KdcProxy
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> kpasswd_server = https://account.gnome.org/KdcProxy&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">}&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>That should be all for today!&lt;/p></description><link>https://www.dragonsreach.it/2014/10/24/kerberos-over-http-on-a-firewalled-network/</link><guid>https://www.dragonsreach.it/2014/10/24/kerberos-over-http-on-a-firewalled-network/</guid><pubDate>Fri, 24 Oct 2014 13:50:52 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>The GNOME Infrastructure’s FreeIPA move behind the scenes</title><description>&lt;p>A few days ago I wrote about the &lt;a href="https://www.dragonsreach.it/2014/10/07/the-gnome-infrastructure-is-now-powered-by-freeipa/" target="_blank">GNOME Infrastructure moving to FreeIPA&lt;/a>, the post was mainly an announcement to the relevant involved parties with many informative details for contributors to properly migrate their account details off from the old authentication system to the new one. Today’s post is a follow-up to that announcement but it’s going to take into account the reasons about our choice to migrate to &lt;strong>FreeIPA&lt;/strong>, what we found interesting and compelling about the software and why we think more projects (them being either smaller or bigger) should migrate to it. Additionally I’ll provide some details about how I performed the migration from our previous &lt;strong>OpenLDAP&lt;/strong> setup with a step-by-step guide that will hopefully help more people to migrate the infrastructure they manage themselves.&lt;/p>
&lt;h2 id="the-gnome-case">The GNOME case&lt;/h2>
&lt;p>It’s very clear to everyone an infrastructure should reflect the needs of its user base, in the case of &lt;strong>GNOME&lt;/strong> a multitude between developers, translators, documenters and between them a very good number of Foundation members, contributors that have proven their non-trivial contributions and have received the status of members of the GNOME Foundation with all the relevant benefits connected to it.&lt;/p>
&lt;p>The situation we had before was very tricky, LDAP accounts were managed through our LDAP istance while Foundation members were being stored on a MySQL database with many of the tables being related to the yearly Board of Director’s elections and one specifically meant to store all the information from each of the members. One of the available fields on that table was defined as ‘userid’ and was supposed to store the LDAP ‘uid’ field the Membership Committee member processing a certain application had to update when accepting the application. This procedure had two issues:&lt;/p>
&lt;ol>
&lt;li>Membership Committee members had no access to LDAP information&lt;/li>
&lt;li>No checks were being run on the web UI to verify the ‘userid’ field was populated correctly taking in multiple inconsistencies between LDAP and the MySQL database&lt;/li>
&lt;/ol>
&lt;p>In addition to the above Mango (the software that helped the GNOME administrative teams to manage the user data for multiple years had no maintainer, no commits on its core since 2008 and several limitations)&lt;/p>
&lt;h2 id="what-were-we-looking-for-as-a-replacement-for-the-current-setup">What were we looking for as a replacement for the current setup?&lt;/h2>
&lt;p>It was very obvious to me we would have had to look around for possible replacements to Mango. What we were aiming for was a software with the following characteristics:&lt;/p>
&lt;ol>
&lt;li>It had to come with a &lt;strong>pre-built web UI&lt;/strong> providing a wide view on several LDAP fields&lt;/li>
&lt;li>The web UI had to be &lt;strong>extensible in some form&lt;/strong> as we had some custom LDAP schemas we wanted users to see and modify&lt;/li>
&lt;li>The sofware had to be &lt;strong>actively developed&lt;/strong> and responsive to eventual security reports (given the high security impact a breach on LDAP could take in)&lt;/li>
&lt;/ol>
&lt;p>FreeIPA clearly matched all our expectations on all the above points.&lt;/p>
&lt;h2 id="the-migration-process-8211-rfc2307-vs-rfc2307bis">The Migration process – RFC2307 vs RFC2307bis&lt;/h2>
&lt;p>Our previous OpenLDAP setup was following &lt;a href="https://www.ietf.org/rfc/rfc2307.txt">RFC 2307&lt;/a>, which means that above all the other available LDAP attributes (listed on the RFC under point &lt;strong>2.2&lt;/strong>) group’s membership was being represented through the ‘memberUid’ attribute. An example:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">cn&lt;/span>&lt;span class="o">=&lt;/span>foundation,cn&lt;span class="o">=&lt;/span>groups,cn&lt;span class="o">=&lt;/span>compat,dc&lt;span class="o">=&lt;/span>gnome,dc&lt;span class="o">=&lt;/span>org
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">objectClass: posixGroup
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">objectClass: top
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">gidNumber: &lt;span class="m">524&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">memberUid: foo
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">memberUid: bar
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">memberUid: foobar&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>As you can each of the members of the group ‘foundation’ are represented using the ‘memberUid’ attribute followed by the ‘uid’ of the user itself. FreeIPA does not make directly use of RFC2307 for its trees, but RFC2307bis instead. (RFC2307bis was not published as a RFC by the &lt;strong>IETF&lt;/strong> as the author didn’t decide to pursue it nor the companies (HP, Sun) that then adopted it)&lt;/p>
&lt;p>RFC2307bis uses a different attribute to represent group’s membership, it being ‘member’. Another example:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">cn&lt;/span>&lt;span class="o">=&lt;/span>foundation,cn&lt;span class="o">=&lt;/span>groups,cn&lt;span class="o">=&lt;/span>accounts,dc&lt;span class="o">=&lt;/span>gnome,dc&lt;span class="o">=&lt;/span>org
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">objectClass: posixGroup
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">objectClass: top
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">gidNumber: &lt;span class="m">524&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">member: &lt;span class="nv">uid&lt;/span>&lt;span class="o">=&lt;/span>foo,cn&lt;span class="o">=&lt;/span>users,cn&lt;span class="o">=&lt;/span>accounts,dc&lt;span class="o">=&lt;/span>gnome,dc&lt;span class="o">=&lt;/span>org
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">member: &lt;span class="nv">uid&lt;/span>&lt;span class="o">=&lt;/span>bar,cn&lt;span class="o">=&lt;/span>users,cn&lt;span class="o">=&lt;/span>accounts,dc&lt;span class="o">=&lt;/span>gnome,dc&lt;span class="o">=&lt;/span>org
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">member: &lt;span class="nv">uid&lt;/span>&lt;span class="o">=&lt;/span>foobar,cn&lt;span class="o">=&lt;/span>users,cn&lt;span class="o">=&lt;/span>accounts,dc&lt;span class="o">=&lt;/span>gnome,dc&lt;span class="o">=&lt;/span>org&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>As you can see the &lt;strong>DN&lt;/strong> representing the group ‘foundation’ differs between the two examples. That is why FreeIPA comes with a &lt;strong>Compatibility plugin&lt;/strong> (cn=compat) which automatically creates RFC2307-compliant trees and entries whenever an append / modify / delete operation happens on any of the hosted RFC2307bis-compliant trees. What’s the point of doing this when we could just stick with RFC2307bis trees and go with it? As the plugin name points out the Compatibility plugin is there to prevent breakages between the directory server and any of the clients or softwares out there still retrieving information and data by using the ‘memberUid’ attribute as specified on RFC2307.&lt;/p>
&lt;p>FreeIPA migration tools (ipa migrate-ds) do come with a ‘–schema’ flag you can use to specify what attribute the istance you are migrating from was following (values are RFC2307 and RFC2307bis as you may have guessed already), in the case of GNOME the complete command we ran (after installing all the relevant tools through ‘ipa-server-install’ and copying the custom schemas under /etc/dirsrv/slapd-$ISTANCE-NAME/schema) was:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">ipa migrate-ds --bind-dn&lt;span class="o">=&lt;/span>&lt;span class="nv">cn&lt;/span>&lt;span class="o">=&lt;/span>Manager,dc&lt;span class="o">=&lt;/span>gnome,dc&lt;span class="o">=&lt;/span>org --user-container&lt;span class="o">=&lt;/span>&lt;span class="nv">ou&lt;/span>&lt;span class="o">=&lt;/span>people,dc&lt;span class="o">=&lt;/span>gnome,dc&lt;span class="o">=&lt;/span>org --group-container&lt;span class="o">=&lt;/span>&lt;span class="nv">ou&lt;/span>&lt;span class="o">=&lt;/span>groups,dc&lt;span class="o">=&lt;/span>gnome,dc&lt;span class="o">=&lt;/span>org --group-objectclass&lt;span class="o">=&lt;/span>posixGroup ldap://internal-IP:389 --schema&lt;span class="o">=&lt;/span>RFC2307&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>Please &lt;strong>note&lt;/strong> that before running the command you should make sure custom schemas you had on the istance you are migrating from are available to the directory server you are migrating your tree to.&lt;/p>
&lt;p>More information on the &lt;strong>migration process&lt;/strong> from an existing OpenLDAP istance can be found &lt;a href="http://docs.fedoraproject.org/en-US/Fedora/15/html/FreeIPA_Guide/Migrating_from_a_Directory_Server_to_IPA-Performing_a_Server_based_Migration.html" target="_blank">HERE&lt;/a>.&lt;/p>
&lt;h2 id="the-migration-process-8211-extending-the-directory-server-with-custom-schemas">The Migration process – Extending the directory server with custom schemas&lt;/h2>
&lt;p>One of the other challenges we had to face has been extending the available LDAP schemas to include Foundation membership attributes. This operation requires the following changes:&lt;/p>
&lt;ol>
&lt;li>Build the LDIF (that will include two new custom fields: FirstAdded and LastRenewedOn)&lt;/li>
&lt;li>Adding the LDIF in place on &lt;strong>/etc/dirsrv/slapd-$ISTANCE-NAME/schema&lt;/strong>&lt;/li>
&lt;li>Extend the web UI to include the new attributes&lt;/li>
&lt;/ol>
&lt;p>I won’t be explaining how to build a LDIF on this post but I’m however pasting the schema I made to help you getting an idea:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">attributeTypes: &lt;span class="o">(&lt;/span> 1.3.6.1.4.1.3319.8.2 NAME &lt;span class="s1">&amp;#39;LastRenewedOn&amp;#39;&lt;/span> SINGLE-VALUE EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch DESC &lt;span class="s1">&amp;#39;Last renewed on date&amp;#39;&lt;/span> SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 &lt;span class="o">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">attributeTypes: &lt;span class="o">(&lt;/span> 1.3.6.1.4.1.3319.8.3 NAME &lt;span class="s1">&amp;#39;FirstAdded&amp;#39;&lt;/span> SINGLE-VALUE EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch DESC &lt;span class="s1">&amp;#39;First added date&amp;#39;&lt;/span> SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 &lt;span class="o">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">objectClasses: &lt;span class="o">(&lt;/span> 1.3.6.1.4.1.3319.8.1 NAME &lt;span class="s1">&amp;#39;FoundationFields&amp;#39;&lt;/span> AUXILIARY MAY &lt;span class="o">(&lt;/span> LastRenewedOn $ FirstAdded &lt;span class="o">)&lt;/span> &lt;span class="o">)&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>After copying the schema in place and restarting the directory server, extend the web UI:&lt;/p>
&lt;p>on &lt;strong>/usr/lib/python2.7/site-packages/ipalib/plugins/foundation.py&lt;/strong>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Python" data-lang="Python">&lt;span class="line">&lt;span class="cl">&lt;span class="kn">from&lt;/span> &lt;span class="nn">ipalib.plugins&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="n">user&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kn">from&lt;/span> &lt;span class="nn">ipalib.parameters&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="n">Str&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kn">from&lt;/span> &lt;span class="nn">ipalib&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="n">_&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kn">from&lt;/span> &lt;span class="nn">time&lt;/span> &lt;span class="kn">import&lt;/span> &lt;span class="n">strftime&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kn">import&lt;/span> &lt;span class="nn">re&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">def&lt;/span> &lt;span class="nf">validate_date&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">ugettext&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">value&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> &lt;span class="ow">not&lt;/span> &lt;span class="n">re&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="k">match&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;^[0-9]&lt;/span>&lt;span class="si">{4}&lt;/span>&lt;span class="s2">-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])$&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">value&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">return&lt;/span> &lt;span class="n">_&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;The entered date is wrong, please make sure it matches the YYYY-MM-DD syntax&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">user&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">user&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">takes_params&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">user&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">user&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">takes_params&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="p">(&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Str&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;firstadded?&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">validate_date&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">cli_name&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s1">&amp;#39;firstadded&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="n">_&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;First Added date&amp;#39;&lt;/span>&lt;span class="p">),&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">),&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Str&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;lastrenewedon?&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">validate_date&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">cli_name&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s1">&amp;#39;lastrenewedon&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="n">_&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;Last Renewed on date&amp;#39;&lt;/span>&lt;span class="p">),&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">),&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">)&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>on &lt;strong>/usr/share/ipa/ui/js/plugins/foundation/foundation.js&lt;/strong>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-JavaScript" data-lang="JavaScript">&lt;span class="line">&lt;span class="cl">&lt;span class="nx">define&lt;/span>&lt;span class="p">([&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s1">&amp;#39;freeipa/phases&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s1">&amp;#39;freeipa/user&amp;#39;&lt;/span>&lt;span class="p">],&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">function&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="nx">phases&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="nx">user_mod&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">// helper function
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">&lt;/span>&lt;span class="kd">function&lt;/span> &lt;span class="nx">get_item&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="nx">array&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="nx">attr&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="nx">value&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="kd">var&lt;/span> &lt;span class="nx">i&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="mi">0&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="nx">l&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nx">array&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">length&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nx">i&lt;/span>&lt;span class="o">&amp;amp;&lt;/span>&lt;span class="nx">lt&lt;/span>&lt;span class="p">;&lt;/span>&lt;span class="nx">l&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nx">i&lt;/span>&lt;span class="o">++&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nx">array&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="nx">i&lt;/span>&lt;span class="p">][&lt;/span>&lt;span class="nx">attr&lt;/span>&lt;span class="p">]&lt;/span> &lt;span class="o">===&lt;/span> &lt;span class="nx">value&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="k">return&lt;/span> &lt;span class="nx">array&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="nx">i&lt;/span>&lt;span class="p">];&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">return&lt;/span> &lt;span class="kc">null&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">var&lt;/span> &lt;span class="nx">foundation_plugin&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="p">{};&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">foundation_plugin&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">add_foundation_fields&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="kd">function&lt;/span>&lt;span class="p">()&lt;/span> &lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">var&lt;/span> &lt;span class="nx">facet&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="nx">get_item&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="nx">user_mod&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">entity_spec&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">facets&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s1">&amp;#39;$type&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s1">&amp;#39;details&amp;#39;&lt;/span>&lt;span class="p">);&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="kd">var&lt;/span> &lt;span class="nx">section&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="nx">get_item&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="nx">facet&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">sections&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s1">&amp;#39;name&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="s1">&amp;#39;identity&amp;#39;&lt;/span>&lt;span class="p">);&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">section&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">fields&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">push&lt;/span>&lt;span class="p">({&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">name&lt;/span>&lt;span class="o">:&lt;/span> &lt;span class="s1">&amp;#39;firstadded&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">label&lt;/span>&lt;span class="o">:&lt;/span> &lt;span class="s1">&amp;#39;Foundation Member since&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">});&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">section&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">fields&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">push&lt;/span>&lt;span class="p">({&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">name&lt;/span>&lt;span class="o">:&lt;/span> &lt;span class="s1">&amp;#39;lastrenewedon&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">label&lt;/span>&lt;span class="o">:&lt;/span> &lt;span class="s1">&amp;#39;Last Renewed on date&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">});&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">section&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">fields&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">push&lt;/span>&lt;span class="p">({&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">name&lt;/span>&lt;span class="o">:&lt;/span> &lt;span class="s1">&amp;#39;description&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">label&lt;/span>&lt;span class="o">:&lt;/span> &lt;span class="s1">&amp;#39;Previous account changes&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">});&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">return&lt;/span> &lt;span class="kc">true&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">};&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nx">phases&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">on&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;customization&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="nx">foundation_plugin&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">add_foundation_fields&lt;/span>&lt;span class="p">);&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">return&lt;/span> &lt;span class="nx">foundation_plugin&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">});&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>Once done, restart the web server. The next step would be migrating all the &lt;strong>FirstAdded&lt;/strong> and &lt;strong>LastRenewedOn&lt;/strong> attributes off from MySQL into LDAP now that our custom schema has been injected.&lt;/p>
&lt;p>The relevant MySQL fields were following the YYYY-MM-DD syntax to store the dates and a little Python script to read from MySQL and populate the LDAP attributes was then made. If interested or you are in a similar situation you can find it &lt;a href="https://git.gnome.org/browse/sysadmin-bin/tree/membership/migrate-foundation-field-to-freeipa.py" target="_blank">HERE&lt;/a>.&lt;/p>
&lt;h2 id="the-migration-process-8211-own-ssl-certificates-for-httpd">The Migration process – Own SSL certificates for HTTPD&lt;/h2>
&lt;p>As you may be aware of FreeIPA comes with its own certificate tools (powered by &lt;strong>Certmonger&lt;/strong>), that means a &lt;strong>CA&lt;/strong> is created (during the ipa-server-install run) and certificates for the various services you provide are then created and signed with it. This is definitely great and removes the burden to maintain an underlying self-hosted PKI infrastructure. At the same time this seems to be a problem for publicly-facing web services as browsers will start complaining they don’t trust the CA that signed the certificate the website you are trying to reach is using.&lt;/p>
&lt;p>The problem is not really a problem as you can specify what certificate HTTPD should be using for displaying FreeIPA’s web UI. The procedure is simple and involves the NSS database at /etc/httpd/alias:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">certutil -d /etc/httpd/alias/ -A -n &lt;span class="s2">&amp;#34;StartSSL CA&amp;#34;&lt;/span> -t CT,C, -a -i sub.class2.server.ca.pem
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">certutil -d /etc/pki/nssdb -A -n &lt;span class="s2">&amp;#34;StartSSL CA&amp;#34;&lt;/span> -t CT,C, -a -i sub.class2.server.ca.pem
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">openssl pkcs12 -inkey freeipa.example.org.key -in freeipa.example.org.crt -export -out freeipa.example.org.p12 -nodes -name &lt;span class="s1">&amp;#39;HTTPD-Server-Certificate&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">pk12util -i freeipa.example.org.p12 -d /etc/httpd/alias/&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>Once done, update /etc/httpd/conf.d/nss.conf with the correct NSSNickname value. (which should match the one you entered after ‘-name’ on the third of the above commands)&lt;/p>
&lt;h2 id="the-migration-process-8211-equivalent-of-authorized_keys8217-8220command8221">The Migration process – Equivalent of authorized_keys’ “command”&lt;/h2>
&lt;p>At GNOME we do run several services that require users to login to specific machines and run a command. At the same time and for security purposes we don’t want all the users to reach a shell. Originally we were making use of SSH’s authorized_keys file to specify the “command” these users should have been restricted to. FreeIPA handles Public Key authentications differently (through the &lt;strong>sss_ssh_authorizedkeys&lt;/strong> binary) which means we had to find an alternative way to restrict groups to only a specific command. SSH’s &lt;strong>ForceCommand&lt;/strong> came in help, an example given a group called ‘foo’:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">Match Group foo,!bar
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">X11Forwarding no
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">PermitTunnel no
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ForceCommand /home/admin/bin/reset-my-password.py&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>The above &lt;strong>Match Group&lt;/strong> will be applied to all the users of the ‘foo’ group except the ones that are also part of the ‘bar’ group. If you are interested in the reset-my-password.py script which resets a certain user password and sends a temporary one to the registered email address (by checking the mail LDAP attr) for the user, click &lt;a href="https://git.gnome.org/browse/sysadmin-bin/tree/reset-my-password.py" target="_blank">HERE&lt;/a>.&lt;/p>
&lt;h2 id="the-migration-process-8211-kerberos-host-keytabs">The Migration process – Kerberos host Keytabs&lt;/h2>
&lt;p>Here, at &lt;strong>GNOME&lt;/strong>, we still have one or two RHEL 5 hosts hanging around and SSSD reported a failure when trying to authenticate with the given Keytab (generated with RHEL 7 default values) to the KDC running (as you may have guessed) RHEL 7. The issue is simple as RHEL 5 does not support many of the encryption types which the Keytab was being encrypted with. Apparently the only currently supported Keytab encryption type on a RHEL 5 machine is &lt;strong>rc4-hmac&lt;/strong>. Creating a Keytab on the KDC accordingly can be done this way:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">ipa-getkeytab -s server.example.org -p host/client.example.org -e rc4-hmac -k /root/keytabs/client.example.org.keytab&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>That should be all for today, I’ll make sure to update this post with further details or answers to possible comments.&lt;/p></description><link>https://www.dragonsreach.it/2014/10/12/the-gnome-infrastructures-freeipa-move-behind-the-scenes/</link><guid>https://www.dragonsreach.it/2014/10/12/the-gnome-infrastructures-freeipa-move-behind-the-scenes/</guid><pubDate>Sun, 12 Oct 2014 18:02:02 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>The GNOME Infrastructure is now powered by FreeIPA!</title><description>&lt;p>As preannounced &lt;a href="https://mail.gnome.org/archives/infrastructure-announce/2014-October/msg00000.html" target="_blank">here&lt;/a> the GNOME Infrastructure switched to a new Account Management System which is reachable at &lt;a href="https://account.gnome.org" target="_blank">&lt;a href="https://account.gnome.org">https://account.gnome.org&lt;/a>&lt;/a>. All the details will follow.&lt;/p>
&lt;h2 id="introduction">Introduction&lt;/h2>
&lt;p>It’s been a while since someone actually touched the underlying authentication infrastructure that powers the GNOME machines. The very first setup was originally configured by &lt;strong>Jonathan Blandford&lt;/strong> (jrb) who configured an &lt;strong>OpenLDAP&lt;/strong> istance with several customized schemas. (pServer fields in the old CVS days, pubAuthorizedKeys and GNOME modules related fields in recent times)&lt;/p>
&lt;p>While OpenLDAP-server was living on the GNOME machine called clipboard (aka ldap.gnome.org) the clients were configured to synchronize users, groups, passwords through the nslcd daemon. After several years &lt;strong>Jeff Schroeder&lt;/strong> joined the Sysadmin Team and during one cold evening (date is Tue, February 1st 2011) spent some time configuring SSSD to replace the nslcd daemon which was missing one of the most important SSSD features: caching. What surely convinced Jeff to adopt SSSD (a very new but promising sofware at that time as the first release happened right before 2010’s Christmas) and as the commit log also states (“New sssd module for ldap information caching”) was SSSD’s caching feature.&lt;/p>
&lt;p>It was enough for a certain user to log in once and the ‘/var/lib/sss/db’ directory was populated with its login information preventing the LDAP daemon in charge of picking up login details (from the LDAP server) to query the LDAP server itself every single time a request was made against it. This feature has definitely helped in many occasions especially when the LDAP server was down for a particular reason and sysadmins needed to access a specific machine or service: without SSSD this wasn’t ever going to work and sysadmins were probably going to be locked out from the machines they were used to manage. (except if you still had ‘/etc/passwd’, ‘/etc/group’ and ‘/etc/shadow’ entries as fallback)&lt;/p>
&lt;p>Things were working just fine except for a few downsides that appeared later on:&lt;/p>
&lt;ol>
&lt;li>the web interface (view) on our LDAP user database was managed by &lt;a href="https://wiki.gnome.org/Attic/Mango" target="_blank">&lt;strong>Mango&lt;/strong>&lt;/a>, an outdated tool which many wanted to rewrite in Django that slowly became a huge dinosaur nobody ever wanted to look into again&lt;/li>
&lt;li>the Foundation membership information were managed through a MySQL database, so two databases, two sets of users unrelated to each other&lt;/li>
&lt;li>users were not able to modify their own account information on their own but even a single e-mail change required them to mail the GNOME Accounts Team which was then going to authenticate their request and finally update the account.&lt;/li>
&lt;/ol>
&lt;p>Today’s infrastructure changes are here to finally say the issues outlined at (1, 2, 3) are now fixed.&lt;/p>
&lt;h2 id="what-has-changed">What has changed?&lt;/h2>
&lt;p>The GNOME Infrastructure is now powered by Red Hat’s &lt;strong>FreeIPA&lt;/strong> which bundles several FOSS softwares into one big “bundle” all surrounded by an easy and intuitive web UI that will help users update their account information on their own without the need of the Accounts Team or any other administrative entity. Users will also find two custom fields on their “Overview” page, these being “Foundation Member since” and “Last Renewed on date”. As you may have understood already we finally managed to migrate the Foundation membership database into LDAP itself to store the information we want once and for all. As a side note it might be possible that some users that were Foundation members in the past won’t find any detail stored on the Foundation fields outlined above. That is actually expected as we were able to migrate all the current and old Foundation members that had an LDAP account registered at the time of the migration. If that’s your case and you still would like the information to be stored on the new setup please get in contact with the Membership Committee at stating so.&lt;/p>
&lt;h2 id="where-can-i-get-my-first-login-credentials">Where can I get my first login credentials?&lt;/h2>
&lt;p>Let’s make a little distinction between users that previously had access to Mango (usually maintainers) and users that didn’t. If you were used to access Mango before you should be able to login on the new Account Management System by entering your GNOME username and the password you were used to use for loggin in into Mango. (after loggin in the very first time you will be prompted to update your password, please choose a strong password as this account will be unique across all the GNOME Infrastructure)&lt;/p>
&lt;p>If you never had access to Mango, you lost your password or the first time you read the word Mango on this post you thought “why is he talking about a fruit now?” you should be able to reset it by using the following command:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">ssh -l yourgnomeuserid account.gnome.org&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>The command will start an SSH connection between you and account.gnome.org, once authenticated (with the SSH key you previously had registered on our Infrastructure) you will trigger a command that will directly send your brand new password on the e-mail registered for your account. From my tests seems GMail sees the e-mail as a phishing attempt probably because the body contains the word “password” twice. That said if the e-mail won’t appear on your INBOX, please &lt;strong>double-check&lt;/strong> your Spam folder.&lt;/p>
&lt;h2 id="now-that-mango-is-gone-how-can-i-request-a-new-account">Now that Mango is gone how can I request a new account?&lt;/h2>
&lt;p>With Mango we used to have a form that automatically e-mailed the maintainer of the selected GNOME module which was then going to approve / reject the request. From there and in the case of a positive vote from the maintainer the Accounts Team was going to create the account itself.&lt;/p>
&lt;p>With the &lt;a href="https://mail.gnome.org/archives/gnome-i18n/2014-February/msg00000.html" target="_blank">recent introduction of a commit robot directly on l10n.gnome.org&lt;/a> the number of account requests reduced its numbers. In addition to that users will now be able to perform pretty much all the needed maintenance on their accounts themselves. That said and while we will probably work on building a form in the future we feel that requesting accounts can definitely be achieved directly by mailing the Accounts Team itself which will mail the maintainer of the respective module and create the account. As just said the number of account creations has become very low and the queue is currently clear. The documentation has been updated to reflect these changes at:&lt;/p>
&lt;p>&lt;a href="https://wiki.gnome.org/AccountsTeam" target="_blank">&lt;a href="https://wiki.gnome.org/AccountsTeam">https://wiki.gnome.org/AccountsTeam&lt;/a>&lt;/a>&lt;/p>
&lt;p>&lt;a href="https://wiki.gnome.org/AccountsTeam/NewAccounts" target="_blank">&lt;a href="https://wiki.gnome.org/AccountsTeam/NewAccounts">https://wiki.gnome.org/AccountsTeam/NewAccounts&lt;/a>&lt;/a>&lt;/p>
&lt;h2 id="i-was-used-to-have-access-to-a-specific-service-but-i-don8217t-anymore-what-should-i-do">I was used to have access to a specific service but I don’t anymore, what should I do?&lt;/h2>
&lt;p>The migration of all the user data and ACLs has been massive and I’ve been spending a lot of time reviewing the existing HBAC rules trying to spot possible errors or misconfigurations. If you happen to not being able to access a certain service as you were used to in the past, please get in contact with the Sysadmin Team. All the possible ways to contact us are available at &lt;a href="https://wiki.gnome.org/Sysadmin/Contact" target="_blank">&lt;a href="https://wiki.gnome.org/Sysadmin/Contact">https://wiki.gnome.org/Sysadmin/Contact&lt;/a>&lt;/a>.&lt;/p>
&lt;h2 id="what-is-missing-still">What is missing still?&lt;/h2>
&lt;p>Now that the Foundation membership information has been moved to LDAP I’ll be looking at porting some of the existing membership scripts to it. What I managed to port already are welcome e-mails for new or existing members. (renewals)&lt;/p>
&lt;p>Next step will be generating a membership page from LDAP (to populate &lt;a href="http://www.gnome.org/foundation/membership" target="_blank">&lt;a href="http://www.gnome.org/foundation/membership">http://www.gnome.org/foundation/membership&lt;/a>&lt;/a>) and all the your-membership-is-going-to-lapse e-mails that were being sent till today.&lt;/p>
&lt;h2 id="other-news-8211-homeusers-mount-on-mastergnomeorg">Other news – /home/users mount on master.gnome.org&lt;/h2>
&lt;p>You will notice that loggin in into &lt;strong>master.gnome.org&lt;/strong> will result in your home directory being empty, don’t worry, you did not lose any of your files but master.gnome.org is now currently hosting your home directories itself. As you may have been aware of adding files to the public_html directory on master resulted in them appearing on your &lt;strong>people.gnome.org/~userid&lt;/strong> space. That was unfortunately expected as both master and webapps2 (the machine serving people.gnome.org’s webspaces) were mounting the same GlusterFS share.&lt;/p>
&lt;p>We wanted to prevent that behaviour to happen as we wanted to know who has access to what resource and where. From today master’s home directories will be there just as a temporary spot for your tarballs, just scp and use ftpadmin against them, that should be all you need from master. If you are interested in receiving or keeping using your people.gnome.org’s webspace please mail &lt;a href="mailto:accounts@gnome.org" target="_blank">&lt;accounts AT gnome DOT org>&lt;/a> stating so.&lt;/p>
&lt;h2 id="other-news-8211-a-shiny-and-new-error-500-page-has-been-deployed">Other news – a shiny and new error 500 page has been deployed&lt;/h2>
&lt;p>Thanks to &lt;strong>Magdalen Berns&lt;/strong> (magpie) a new error 500 web page has been deployed on all the Apache istances we host. The page contains an iframe of &lt;strong>status.gnome.org&lt;/strong> and will appear every single time the web server behind the service you are trying to reach will be unreachable for maintenance or other purposes. While I hope you won’t see the page that often you can still enjoy it at &lt;a href="https://static.gnome.org/error-500/500.html" target="_blank">&lt;a href="https://static.gnome.org/error-500/500.html">https://static.gnome.org/error-500/500.html&lt;/a>&lt;/a>. Make sure to whitelist status.gnome.org on your browser as it currently loads it without https. (as the service is currently hosted on OpenShift which provides us with a *.rhcloud.com wildcard certificate, which differs from the CN the browser would expect it to be)&lt;/p>
&lt;h2 id="updates">Updates&lt;/h2>
&lt;p>&lt;strong>UPDATE&lt;/strong> on status.gnome.org’s SSL certificate: the certificate has been provisioned and it should result in the 500’s page to be displayed correctly with no warnings from your browser.&lt;/p>
&lt;p>&lt;strong>UPDATE&lt;/strong> from Adam Young on Kerberos ports being closed on many DC’s firewalls:&lt;/p>
&lt;blockquote>
&lt;p>The next version of upstream MIT Kerberos will have support for fetching a ticket via ports 443 and marshalling the request over HTTPS. We’ll need to run a proxy on the server side, but we should be able to make it work:&lt;/p>
&lt;p>Read up here&lt;/p>
&lt;/blockquote>
&lt;blockquote>
&lt;p>&lt;a href="http://adam.younglogic.com/2014/06/kerberos-firewalls/" rel="nofollow">&lt;a href="http://adam.younglogic.com/2014/06/kerberos-firewalls">http://adam.younglogic.com/2014/06/kerberos-firewalls&lt;/a>&lt;/a>&lt;/p>
&lt;/blockquote></description><link>https://www.dragonsreach.it/2014/10/07/the-gnome-infrastructure-is-now-powered-by-freeipa/</link><guid>https://www.dragonsreach.it/2014/10/07/the-gnome-infrastructure-is-now-powered-by-freeipa/</guid><pubDate>Tue, 07 Oct 2014 09:21:33 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>Back from GUADEC 2014</title><description>&lt;p>Coming back from &lt;strong>GUADEC&lt;/strong> has never been easy, so much fun, so much great people to speak with and amazing talks to watch but this year has definitely been harder as I totally felt in love with the city that was hosting the event. Honestly speaking I’ve been amazed by how Strasbourg looks like: alsace houses and buildings are just delightful, the cathedral is stunning and people have been so welcoming during my whole stay. (cooks at the Canteen even prepared a few italian dishes and welcomed us in italian every time we were heading there…how cool is that?)&lt;/p>
&lt;p>But let’s get back to our business now as I would probably never stop talking about Strasbourg and how great it was staying there! I did not have a personal talk this year but I presented the yearly Sysadmin Team report during the Foundation’s AGM. If you weren’t there all the slides are available &lt;a href="https://www.dragonsreach.it/files/guadec-reports/guadec-2014.html" target="_blank">here&lt;/a>.&lt;/p>
&lt;p>Apart from presenting what we did and what the changes we introduced on the GNOME Infrastructure were I participated to Patrick Uiterwijk’s talk about FedOAuth and all the upcoming changes that are planned on the infrastructure during the next months. If you were not able to attend Patrick’s talk this little resume should be for you:&lt;/p>
&lt;p>&lt;strong>Current problems&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>The GNOME Infrastructure currently has a lot of different user databases which implies different users and passwords across the services we host&lt;/li>
&lt;li>The Foundation’s database is currently MySQL-based while we do have LDAP in place for all our other needs already&lt;/li>
&lt;li>Some of the tools we do use for managing our LDAP istance are not being maintained properly&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Possible solutions&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Introduce FedOAuth, a SSO solution written and developed by Patrick Uiterwijk&lt;/li>
&lt;li>Unify the various databases and make sure our LDAP istance is used for authentication everywhere&lt;/li>
&lt;li>Remove Mango and configure FreeIPA&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Benefits after the move&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Users will be able to manage their accounts on their own, no more need to poke the accounts team for updating passwords, emails, SSH keys. The accounts team will still be around to adjust ACLs&lt;/li>
&lt;li>No more need for dozen of accounts, one for every single service we provide&lt;/li>
&lt;li>More freedom when managing sudo accesses and accounts on the various machines we manage, this will help new people contributing to the Sysadmin Team (Making our puppet repository public and introducing a GNOME Infrastructure Apprentice group for newcomers is something we will be seriously evaluating after the FreeIPA move)&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Where we are now:&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>Our SSO infrastructure is live at &lt;a href="https://id.gnome.org" target="_blank">&lt;a href="https://id.gnome.org">https://id.gnome.org&lt;/a>&lt;/a>&lt;/li>
&lt;li>Your OpenID URL is &lt;strong>https://$GNOME_USERID.id.gnome.org&lt;/strong>&lt;/li>
&lt;li>Right now you can login with your GNOME account at the following services: l10n.gnome.org, opw.gnome.org. We are slowly migrating all the existing services to the new SSO infrastructure, please be patient and bear with us!&lt;/li>
&lt;/ul>
&lt;p>More information, slides and screenshots from Patrick’s talk are available &lt;a href="http://patrick.uiterwijk.org/2014/07/28/gnome-authentication/#1" target="_blank">here&lt;/a>. Stay tuned and many thanks to the &lt;strong>GNOME Foundation&lt;/strong> for sponsoring my travel and accomodation expenses!&lt;/p>
&lt;div class="gallery">
&lt;figure>
&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2014/08/14581494979_18c62c3083_z.jpg" alt="The GUADEC 2014 group photo!"/>
&lt;figcaption>The GUADEC 2014 group photo!&lt;/figcaption>
&lt;/figure>
&lt;figure>
&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2014/08/IMG_20140729_193107.jpg" alt="A view of the Petite France district"/>
&lt;figcaption>A view of the Petite France district&lt;/figcaption>
&lt;/figure>
&lt;figure>
&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2014/08/IMG_20140729_195206.jpg" alt="Another great view of the Petite France district"/>
&lt;figcaption>Another great view of the Petite France district&lt;/figcaption>
&lt;/figure>
&lt;figure>
&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2014/08/IMG_20140725_103921.jpg" alt="The Notre Dame cathedral"/>
&lt;figcaption>The Notre Dame cathedral&lt;/figcaption>
&lt;/figure>
&lt;figure>
&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2014/08/IMG_20140725_103927.jpg" alt="Another view of the Notre Dame cathedral"/>
&lt;figcaption>Another view of the Notre Dame cathedral&lt;/figcaption>
&lt;/figure>
&lt;figure>
&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2014/08/IMG_20140725_103655.jpg" alt="Place Gutenberg"/>
&lt;figcaption>Place Gutenberg&lt;/figcaption>
&lt;/figure>
&lt;/div>
&lt;p> &lt;/p></description><link>https://www.dragonsreach.it/2014/08/05/back-from-guadec-2014/</link><guid>https://www.dragonsreach.it/2014/08/05/back-from-guadec-2014/</guid><pubDate>Tue, 05 Aug 2014 17:27:23 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>Fedy’s installation of Brackets bricks your Fedora installation</title><description>&lt;p>I wanted to give &lt;a href="http://satya164.github.io/fedy/" target="_blank">Fedy&lt;/a> a try yesterday, specifically to install the **Brackets **code editor designed for web developers. I’m pretty lazy when it comes to install external packages (from the Brackets.io’s homepage it looked like only a DEB file was available) and after asking a few friends who made heavy use of Fedy in the past about its stability and credibility I went ahead and followed the provided instructions to set it up.&lt;/p>
&lt;p>The interface was pretty straightforward and installing Brackets was as easy and clicking on the relevant button. Before starting the installation I gave a fast look around to the various bash scripts used by Fedy to install the package I wanted and yeah, I admit I did not pay enough attention to a few lines of the code and went ahead with the installation.&lt;/p>
&lt;p>After hacking a bit with Brackets I decided it was time for me to head to bed but shutting down my laptop surprisingly returned various errors related to Systemd’s journal not being able to shutdown properly. I then tried to reboot the machine and found out the laptop was totally not bootable anymore.&lt;/p>
&lt;p>The error it was reported at boot (Systemd Journal not being able to start properly) was pretty strange and after looking around the web I couldn’t find any other report about similar failures. I then started digging around with a friend and made the following guesses:&lt;/p>
&lt;ol>
&lt;li>The root partition was running out of space (just 70M left), I then cleaned it a bit and rebooted with no luck. My first guess was &lt;em>/tmp&lt;/em> going out of space when Systemd tries to populate it at boot time.&lt;/li>
&lt;li>I checked yum history to find out what Fedy could have taken in, but nothing relevant was found given Fedy does not install RPM packages on its own, it usually retrieves a tarball (or in my case a DEB package) and installs it by extracting / cping the content&lt;/li>
&lt;li>I turned SELinux to Permissive and rebooted the machine, surprise, the machine was bootable again&lt;/li>
&lt;/ol>
&lt;p>The next move was running a &lt;em>restorecon -r -v /&lt;/em> against the root partition, the result was awful: the whole &lt;em>/usr&lt;/em>‘s context was turned into &lt;em>usr_tmp_t&lt;/em>. Digging around the code for the Brackets installer the following code was found:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">mkdir -p &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">file&lt;/span>&lt;span class="p">%.*&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ar p &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$file&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;data.tar.gz&amp;#34;&lt;/span> &lt;span class="p">|&lt;/span> tar -C &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">file&lt;/span>&lt;span class="p">%.*&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -xzf -
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cp -af &lt;span class="si">${&lt;/span>&lt;span class="nv">file&lt;/span>&lt;span class="p">%.*&lt;/span>&lt;span class="si">}&lt;/span>/* &lt;span class="s2">&amp;#34;/&amp;#34;&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>&amp;lt;br&amp;gt;&lt;/p>
&lt;p>And previously:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">get_file_quiet &lt;span class="s2">&amp;#34;http://download.brackets.io/&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;brackets.htm&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">get&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>cat &lt;span class="s2">&amp;#34;brackets.htm&amp;#34;&lt;/span> &lt;span class="p">|&lt;/span> tr &lt;span class="s1">&amp;#39; &amp;#39;&lt;/span> &lt;span class="s1">&amp;#39;\n&amp;#39;&lt;/span> &lt;span class="p">|&lt;/span> grep -o &lt;span class="s2">&amp;#34;file.cfm?platform=LINUX&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">arch&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;amp;build=[0-9]*&amp;#34;&lt;/span> &lt;span class="p">|&lt;/span> head -n &lt;span class="m">1&lt;/span> &lt;span class="p">|&lt;/span> sed -e &lt;span class="s1">&amp;#39;s/^/http:\/\/download.brackets.io\//&amp;#39;&lt;/span>&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">file&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;brackets-LINUX.deb&amp;#34;&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>&amp;lt;br&amp;gt;&lt;/p>
&lt;p>So what the installer was doing is:&lt;/p>
&lt;ol>
&lt;li>Downloading the DEB file from the Brackets website&lt;/li>
&lt;li>Extracting its content to &lt;em>/tmp/fedy&lt;/em> and copying the contents of the &lt;em>data.tar.gz&lt;/em> tarball in place&lt;/li>
&lt;/ol>
&lt;p>A tree view of the &lt;em>data.tar.gz&lt;/em> file:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-Bash" data-lang="Bash">&lt;span class="line">&lt;span class="cl">.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">|&lt;/span>-- opt
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">|&lt;/span> &lt;span class="sb">`&lt;/span>-- brackets
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sb">`&lt;/span>-- usr
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">|&lt;/span>-- bin
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sb">`&lt;/span>-- share&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>&amp;lt;br&amp;gt;&lt;/p>
&lt;p>Copying the extracted content of the &lt;em>data.tar.gz&lt;/em> tarball to the target directories will do exactly one thing: it will overwrite the SELinux context of your &lt;em>/usr&lt;/em>, &lt;em>/bin&lt;/em>, &lt;em>/share&lt;/em> directories breaking your system. I would advise everyone to &lt;strong>NOT&lt;/strong> make use of &lt;strong>Fedy&lt;/strong> for installing the Brackets editor until the issue has been fixed. Honestly speaking I didn’t have time / willingness to check other bash scripts but something nasty might be found there as well. Generally I would never recommend to install anything on your system without making use of an RPM package. Lesson learned for me to never trust such tools in the future on my local system.&lt;/p>
&lt;p>The issue seems it was reported already one month ago, we added our report to the same issue. You can track it at &lt;a href="https://github.com/satya164/fedy/issues/79" target="_blank">&lt;a href="https://github.com/satya164/fedy/issues/79">https://github.com/satya164/fedy/issues/79&lt;/a>&lt;/a>.&lt;/p>
&lt;p>Resources:&lt;/p>
&lt;ol>
&lt;li>Faulty bash script: &lt;a href="https://github.com/satya164/fedy/blob/master/plugins/soft/adobe_brackets.sh" target="_blank">&lt;a href="https://github.com/satya164/fedy/blob/master/plugins/soft/adobe_brackets.sh">https://github.com/satya164/fedy/blob/master/plugins/soft/adobe_brackets.sh&lt;/a>&lt;/a>&lt;/li>
&lt;li>Why &lt;em>usr_tmp_t&lt;/em> gets added as &lt;em>/usr&lt;/em>‘s context: &lt;a href="https://github.com/satya164/fedy/blob/master/fedy#L26" target="_blank">&lt;a href="https://github.com/satya164/fedy/blob/master/fedy#L26">https://github.com/satya164/fedy/blob/master/fedy#L26&lt;/a>&lt;/a>.&lt;/li>
&lt;/ol></description><link>https://www.dragonsreach.it/2014/04/02/fedy-installation-of-brackets-bricks-your-fedora-installation/</link><guid>https://www.dragonsreach.it/2014/04/02/fedy-installation-of-brackets-bricks-your-fedora-installation/</guid><pubDate>Wed, 02 Apr 2014 14:11:27 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>Back from GUADEC 2013</title><description>&lt;p>&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2013/08/av_guadec.jpg" alt=" Courtesy of Ana Rey, work licensed under the CC-BY-SA-2.0, available on Flickr.
">&lt;/p>
&lt;p>I wanna be really honest, getting back home from this year’s &lt;strong>GUADEC&lt;/strong> has been very painful for me but not because of the trip back home. I had such a very good time at Brno that I actually wanted to stay there for way more days! I must admit that I’ve been missing the italian food for a while until &lt;strong>Mattias Bengtsson&lt;/strong> suggested me to try having a dinner at the “&lt;strong>Flavours&lt;/strong>” indian restaurant. The result was simply amazing and I’ve been falling in love with the indian food we ate that evening so much that we went there again the day after.&lt;/p>
&lt;p>I had a lot of expectations from my very first GUADEC and I was very excited to meet all the people I’ve been contributing with during all these years. Meeting people up in person is fundamental and reminds you the fact that there is a human being with its own feelings and emotions behind a computer and that’s actually why I spent a lot of my time during the event speaking and hanging out with people, finding out their personal interests, their hobbies, the things they love doing on their free time.&lt;/p>
&lt;p>During the event, I had the great pleasure to present two talks:&lt;/p>
&lt;ol>
&lt;li>the &lt;strong>GNOME Infrastructure&lt;/strong>, the presentation is available &lt;a href="http://www.dragonsreach.it/wp-content/uploads/2013/08/The-GNOME-Infrastructure.odp" target="_blank">here&lt;/a> and the video is viewable at the following &lt;a href="http://www.superlectures.com/guadec2013/the-gnome-infrastructure" target="_blank">page&lt;/a>.&lt;/li>
&lt;li>a little resume of what we’ve been doing on the &lt;strong>GNOME Sysadmin team&lt;/strong> during this last year at the GNOME Foundation’s &lt;strong>Annual General Meeting&lt;/strong>. (AGM)&lt;/li>
&lt;/ol>
&lt;p>… and meet a lot of friends:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Mattias Bengtsson&lt;/strong>, you’ve been the greatest room mate of all times! Thanks for all your hints!&lt;/li>
&lt;li>&lt;strong>Andreas Nilsson&lt;/strong> and &lt;strong>Fabiana Simões&lt;/strong>, thanks for all your kind words, I’ll keep doing my best to provide you all the resources and work you need to make things happen!&lt;/li>
&lt;li>&lt;strong>Sriram Ramkrishna&lt;/strong>, you don’t know how much I enjoyed our nightly discussions and walks around the city centre, missing those times already!&lt;/li>
&lt;li>&lt;strong>Allan Day&lt;/strong> and &lt;strong>Jon McCann&lt;/strong>, it’s been truly amazing meeting you guys and hearing how much enthusiast you are about me and my work, thanks a *lot*!&lt;/li>
&lt;li>&lt;strong>Paul Frields&lt;/strong>, it’s been an absolute pleasure meeting you and your wife, I will never forget how you presented me to her at the &lt;strong>Starobrno Brewery&lt;/strong>: “&lt;em>He’s like the Kevin Fenzi of the GNOME Infrastructure!&lt;/em>“. That’s one of the greatest compliments someone can ever receive given the amount of work Kevin does on the Fedora Infrastructure. I’m also glad that we could remember &lt;strong>Seth&lt;/strong> together during my talk, thanks for coming!&lt;/li>
&lt;li>My italian friends, &lt;strong>Flavia&lt;/strong>, &lt;strong>Paolo&lt;/strong>, &lt;strong>Emanuele&lt;/strong>, &lt;strong>Giovanni&lt;/strong> and &lt;strong>Alessandro&lt;/strong>!&lt;/li>
&lt;li>&lt;strong>Tobias Mueller&lt;/strong>, glad we were finally able to meet in person! Thanks for your nano sessions!&lt;/li>
&lt;li>All the indian interns, &lt;strong>Saumya Dwivedi&lt;/strong>, &lt;strong>Saumya Pathak&lt;/strong>, &lt;strong>Sindhu Sundar&lt;/strong>, &lt;strong>Shivani Poddar&lt;/strong> who are definitely doing an amazing job! I was completely impressed by your talks and projects!&lt;/li>
&lt;li>&lt;strong>Ekaterina&lt;/strong>, &lt;strong>David&lt;/strong>, &lt;strong>Karen&lt;/strong>, &lt;strong>Marina&lt;/strong>, &lt;strong>Owen&lt;/strong>, &lt;strong>Rui&lt;/strong>, &lt;strong>Alberto&lt;/strong>: all you guys are simply great! It’s really unfortunate we weren’t able to spend a lot of time together during the event!&lt;/li>
&lt;/ol>
&lt;p>Overall the whole event has been a blast, thanks to the &lt;strong>GNOME Foundation&lt;/strong> making this possible by sponsoring my attendance at the event! I’m looking forward to &lt;strong>Strasburg 2014&lt;/strong> already and last but not least I’m preparing the greatest bid of all times for &lt;strong>GUADEC 2015&lt;/strong> to happen in &lt;strong>Italy&lt;/strong>. Stay tuned!&lt;/p>
&lt;div class="gallery">
&lt;figure>
&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2013/08/seth_guadec.jpg"
&lt;figcaption>Seth at GUADEC&lt;/figcaption>
&lt;/figure>
&lt;figure>
&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2013/08/guadec_friends.jpg"
&lt;figcaption>GUADEC friends&lt;/figcaption>
&lt;/figure>
&lt;figure>
&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2013/08/mesri.jpg"
&lt;figcaption>Myself and Sri&lt;/figcaption>
&lt;/figure>
&lt;/div>
&lt;p>Overall the whole event has been a blast, thanks to the &lt;strong>GNOME Foundation&lt;/strong> making this possible by sponsoring my attendance at the event! I’m looking forward to &lt;strong>Strasburg 2014&lt;/strong> already and last but not least I’m preparing the greatest bid of all times for &lt;strong>GUADEC 2015&lt;/strong> to happen in &lt;strong>Italy&lt;/strong>. Stay tuned!&lt;/p>
&lt;p>&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2013/08/sponsored-badge-simple.png" alt="sponsored-badge-simple">
&lt;br>&lt;/p></description><link>https://www.dragonsreach.it/2013/08/12/back-from-guadec-2013/</link><guid>https://www.dragonsreach.it/2013/08/12/back-from-guadec-2013/</guid><pubDate>Mon, 12 Aug 2013 15:51:30 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>A second round of updates from the GNOME Sysadmin Team</title><description>&lt;p>&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2013/06/nagios.gnome_.org_.png" alt="nagios.gnome.org">&lt;/p>
&lt;p>I haven’t been blogging so much in the past months as I actually promised myself I would have but given the fact a lot has been done on the &lt;strong>GNOME Infrastructure&lt;/strong> lately it’s time for me to announce all the updates we did since my &lt;a href="http://www.dragonsreach.it/2013/03/07/some-updates-from-the-gnome-sysadmin-team/" target="_blank">latest blog post&lt;/a>. So here we come with all the items we’ve been looking at recently:&lt;/p>
&lt;ul>
&lt;li>
&lt;p>Our main &lt;strong>LDAP&lt;/strong> istance was moved from a very ancient machine (which unfortunately died with a broken disk a few weeks ago) to a newer box that currently contains several other admin tools like Mango and Daily Reports. (a little script written by &lt;strong>Owen Taylor&lt;/strong> for creating and storing several reports mainly related to backups and SSL certificates expiration dates) In addition to migrating our LDAP master to a newer machine, we did configure and setup replication to an LDAP slave to share a bit the load and most of all to link all the external (machines outside the RH’s internal network) machines to it.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>A lot of efforts have been spent in the so-called “&lt;strong>Puppet-ization&lt;/strong>” (Puppet allows you to reproduce a complete environment with just a few commands, it’s very very handy in the case of host’s migrations) and several new modules are now stored into our internal Puppet repository. Specifically all the &lt;strong>Iptables&lt;/strong> rules are currently managed in a centralized way, each node has its own rules and policies, finally there’s no need to ssh into each machine to retrieve the information we need for a specific firewall. In addition to the Iptables class, also Cobbler, Owncloud, Jabberd, Denyhosts and several other modules have been properly configured and currently reside in Puppet.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Another item that had top priority on the list was setting up another “webapps” virtual machine to migrate several services from one of the existing ancient machines to it. I can finally tell that the GNOME Infrastructure has get rid of all the old machines, all the services have been migrated to newer machines and most of all all the services are currently being served through SSL. (git, planet, &lt;a href="https://www.gnome.org">www.gnome.org&lt;/a>, l10n, guadec.org, bugzilla, blogs, developer, help, people, news etc.) In regard of SSL and &lt;strong>Bugzilla&lt;/strong>, we’ve configured our Bugzilla istance to serve attachments through a secondary domain which will look like: &lt;strong>&lt;em>&lt;a href="https://bug-id.bugzilla-attachments.gnome.org">https://bug-id.bugzilla-attachments.gnome.org&lt;/a>&lt;/em>&lt;/strong>, this to prevent cross-site scripting attacks in a better way than what we did before.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>We’ve also spent some time working on our Nagios istance hosted at &lt;strong>&lt;a href="https://nagios.gnome.org">https://nagios.gnome.org&lt;/a>&lt;/strong>. We’ve improved it dramatically by adding several new checks and covering all the services we currently take care of but that’s not all. &lt;strong>Event Handlers&lt;/strong> have been setup to help us addressing problems right after they occur on our web servers. The Nagios event handlers are currently configured to read the status of a specific Nagios service and in the case the status is set to CRITICAL, they restart httpd once, which is usually enough in the case of random Apache’s timeouts. But that’s again not all. A public view for &lt;strong>Nagios&lt;/strong> is now ready, every single GNOME contributor and developer should be able to check the current status of all the services we maintain just by loggin in with the username “&lt;strong>anonymous&lt;/strong>” and the password “&lt;strong>anonymous&lt;/strong>” at nagios.gnome.org.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Our wiki was upgraded to the latest &lt;strong>MoinMoin&lt;/strong> release, at the moment of writing, version &lt;strong>1.9.7&lt;/strong>. This release introduces stronger password hashes, please make sure to update your password as soon as you can to strenghten the security of your account. It was also clear that live.gnome.org was behaving a bit sluggish lately, we spent some time cleaning up spammers, old and deleted pages and things started flushing way better. More details about the cleanup can be found at &lt;a href="https://mail.gnome.org/archives/foundation-list/2013-May/msg00098.html">https://mail.gnome.org/archives/foundation-list/2013-May/msg00098.html&lt;/a>. (we cleaned up around 23000 spammers!)&lt;/p>
&lt;/li>
&lt;li>
&lt;p>The most exciting things I usually love to announce are new services. While I always prefer keeping the number of maintained services as low as possible it was time for the GNOME Infrastructure to broaden its horizons satisfying the requests coming from the community and the developers involved into the project. I won’t spend any more word about this since I’m sure you are all waiting for me to list the new services, so here they are:&lt;/p>
&lt;/li>
&lt;li>
&lt;p> A completely new &lt;strong>Jabber&lt;/strong> service hosted at jabber.gnome.org and accessible by all the GNOME Foundation members requesting access to it. More details about it can be found at &lt;a href="https://live.gnome.org/Sysadmin/Jabber" target="_blank">&lt;a href="https://live.gnome.org/Sysadmin/Jabber">https://live.gnome.org/Sysadmin/Jabber&lt;/a>&lt;/a>.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>GNOME is extensively using IRC as its main communication tool, thus we’ve improved our Services IRC Bot to use a plugin called &lt;strong>MeetBot&lt;/strong>. Having a meeting and storing the logs in a public web server is currently possible with a really minor effort of learning a few commands to administer the plugin correctly. If you are going to have a meeting and you want to make use of MeetBot, make sure that Services is there, and give a look at &lt;a href="http://meetbot.gnome.org/Manual.html">http://meetbot.gnome.org/Manual.html&lt;/a>.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Do you want to be always up-to-date with the status of the GNOME Infrastructure? and are you actually wondering what’s the best way to do so? if yes, you should probably have a look at &lt;a href="http://status.gnome.org" target="_blank">&lt;a href="http://status.gnome.org">http://status.gnome.org&lt;/a>&lt;/a>. This service makes use of &lt;a href="http://git.fedorahosted.org/git/fedora-status" target="_blank">Fedora-Status&lt;/a> by &lt;strong>Patrick Uiterwijk&lt;/strong> and allows the GNOME Sysadmin Team to let everyone know whether there is a problem with any of the services listed in the page. This service, together with the public view for Nagios and the brand new &lt;a href="https://mail.gnome.org/mailman/listinfo/infrastructure-announce" target="_blank">&lt;a href="mailto:infrastructure-announce@gnome.org">infrastructure-announce@gnome.org&lt;/a> &lt;/a>mailing list will definitely help everyone finding out what’s going on, where and how long it’ll take for the issue to be fixed.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>It took me a lot of pain having the &lt;strong>KGB IRC Collaboration bot&lt;/strong> packaged into the EPEL repositories but I finally managed to set it up on the GNOME Infrastructure. KGB has become very handy since the time Cia.vc closed its hosting and it’s available for anyone requesting access to it at &lt;strong>irc.gnome.org&lt;/strong>. If you are looking for Git commit notifications of a specific module directly on your IRC channel, this is what you want :-)&lt;/p>
&lt;/li>
&lt;li>
&lt;p>An Owncloud instance is also available, more reading about what are the requirements for requesting access to it at the following &lt;a href="https://live.gnome.org/MembershipCommittee/MembershipBenefits#Account_on_cloud.gnome.org" target="_blank">link&lt;/a>.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>An Etherpad istance is also available at &lt;a href="http://etherpad.gnome.org" target="_blank">&lt;a href="https://etherpad.gnome.org">https://etherpad.gnome.org&lt;/a>&lt;/a> to all the GNOME Teams that need it! Please drop me an e-mail at &lt;av at gnome dot org> if you are interested. (Pad’s creation is currently disabled for preventing spam)&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Build.gnome.org has been revived and it’s currently hosting an &lt;a href="https://live.gnome.org/OSTree" target="_blank">OSTree&lt;/a> istance. GNOME Daily images are costantly being generated, interested in testing one the those images? give this &lt;a href="http://worldofgnome.org/how-to-try-gnome-os-yes-gnome-os/" target="_blank">article&lt;/a> a look.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>That’s all for now! See you all at &lt;strong>GUADEC&lt;/strong> and thanks everyone for all the hints, suggestions and mails you’ve been sending me in the past months! And a special thanks to &lt;strong>Ekaterina Gerasimova&lt;/strong> for taking the time to brainstorm with me suggesting new features and improvements over the GNOME Infrastructure!&lt;/p>
&lt;p>&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2013/06/status.gnome_.org_.png" alt="status.gnome.org">&lt;/p></description><link>https://www.dragonsreach.it/2013/06/14/a-second-round-of-updates-from-the-gnome-sysadmin-team/</link><guid>https://www.dragonsreach.it/2013/06/14/a-second-round-of-updates-from-the-gnome-sysadmin-team/</guid><pubDate>Fri, 14 Jun 2013 11:37:06 +0000</pubDate></item><item><dc:creator>Andrea Veri</dc:creator><title>Some updates from the GNOME Sysadmin Team</title><description>&lt;p>It’s been more than a month now since I started looking into the many outstanding items we had waiting on our To Do list here at the &lt;strong>GNOME Infrastructure&lt;/strong>. A lot has been done and a lot has yet to come during the next months, but I would like to share with you some of the things I managed to look at during these weeks.&lt;/p>
&lt;p>As you may understand many Sysadmin’s tasks are not perceived at all by users especially the ones related to the so-called “Puppet-ization” which refers to the process of creating / modifying / improving our internal Puppet repository. A lot of work has been done on that side and several new modules have been added, specifically Cobbler, Amavisd, SpamAssassin, ClamAV, Bind, Nagios / Check_MK (enabling Apache eventhandlers for automatic restart of faulty httpd processes), Apache.&lt;/p>
&lt;p>Another top priority item was migrating some of our services off to the old physical machines to virtual machines I did setup earlier. The machines that are now recycled are the following: (Two more are missing on the list, specifically window (which still hosts art.gnome.org, people.gnome.org and projects.gnome.org due to be migrated to another host in the next weeks) and label. (which still hosts our Jabberd, an &lt;a href="https://mail.gnome.org/archives/foundation-list/2013-March/msg00000.html" target="_blank">interesting discussion&lt;/a> on its future is currently ongoing on Foundation-list)&lt;/p>
&lt;ol>
&lt;li>&lt;strong>menubar&lt;/strong>, our old Postfix host served the GNOME Foundation since 2004 and processed millions of e-mails from and to @gnome.org addresses.&lt;/li>
&lt;li>&lt;strong>container&lt;/strong>, our old main NFS node was serving the GNOME Foundation since 2003, it hosted our mail archives, our FTP archives and all the /home/users/* directories.&lt;/li>
&lt;li>&lt;strong>button&lt;/strong> hosted many services (MySQL databases, LDAP, Mango) and served the Foundation since 2004, a faulty hardware took it down on January 2012.&lt;/li>
&lt;/ol>
&lt;p>And if you ever wanted to see how menubar, container and button look like, I have two photos for you with the machines being pulled out the &lt;strong>GNOME&lt;/strong> rack:&lt;/p>
&lt;p>&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2013/03/old-machines.jpg" alt="old-machines">
&lt;br>
&lt;img src="https://www.dragonsreach.it/wp-content/uploads/2013/03/old-machines-2.jpg" alt="old-machines-2">&lt;/p>
&lt;p>Some of the things you may have perceived directly on your skin should be the following:&lt;/p>
&lt;ol>
&lt;li>Our live.gnome.org istance has been upgraded to the latest MoinMoin stable release, 1.9.6.&lt;/li>
&lt;li>The Services bot has been added to the &lt;strong>GIMPNET&lt;/strong> network and currently manages all GNOME channels, it currently acts as a Nickserv, Chanserv. More information about how you can register your nickname and gain the needed ACLs at the following &lt;a href="https://live.gnome.org/Sysadmin/IRC" target="_blank">wiki page&lt;/a>.&lt;/li>
&lt;li>Several &lt;strong>GNOME&lt;/strong> services and domains are now covered by SSL as you may have noticed on planet.gnome.org, news.gnome.org, blogs.gnome.org, l10n.gnome.org, git.gnome.org, help.gnome.org, developer.gnome.org.&lt;/li>
&lt;li>Re-design of our Mailman archives as you can see at &lt;a href="https://mail.gnome.org/archives/foundation-list/">https://mail.gnome.org/archives/foundation-list&lt;/a>. A big “thank you” goes to Olav Vitters for taking the time to rebuild our “archive” script from Perl to Python. About the Mailman topic, someone proposed me the use of &lt;a href="https://fedorahosted.org/hyperkitty/" target="_blank">HyperKitty&lt;/a>, that’s something we will evaluate in the next coming months but I find it a very interesting alternative to the current mail archiving.&lt;/li>
&lt;/ol>
&lt;p>What should you expect next?&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Bugzilla&lt;/strong> will be moved to another virtual machine and will be upgraded to the latest release.&lt;/li>
&lt;li>An &lt;strong>Owncloud&lt;/strong> istance will be setup for all the GNOME Foundation members and GNOME Teams that will need access to it.&lt;/li>
&lt;li>A discussion will be started for setting up a &lt;strong>Gitorious&lt;/strong> istance on the GNOME Infrastructure.&lt;/li>
&lt;li>A long-term item will be rewriting &lt;a href="https://live.gnome.org/Mango" target="_blank">Mango&lt;/a> in Django and adding several other features to it than the ones it has now. (ideally voting for Board elections, logins for managing your LDAP information such as your @gnome.org’s alias forward, shutdown of old an unused accounts after a certain period of time, automatic @gnome.org’s alias creation after the “Foundation Membership” flag is selected on LDAP, etc.)&lt;/li>
&lt;/ol>
&lt;p>Thanks a lot for all the mails I’ve received during these weeks containing reports and suggestions about how we should improve our Infrastructure! Please stay tuned, a lot more news are yet to come!&lt;/p></description><link>https://www.dragonsreach.it/2013/03/07/some-updates-from-the-gnome-sysadmin-team/</link><guid>https://www.dragonsreach.it/2013/03/07/some-updates-from-the-gnome-sysadmin-team/</guid><pubDate>Thu, 07 Mar 2013 14:49:26 +0000</pubDate></item></channel></rss>