Andrea Veri's Blog

AWP - The Awesome Weekend Project

Andrea Veri — Sat, 17 May 2025 11:39:39 -0400

Table of Contents
Introduction
MVP
The learning resources
Preliminary steps
The code
A quick peek at the web UI
License
The future

Introduction

One of those things that make Red Hat a special place are the so called “Learning days”, working days where you can primarily focus on improving your skills on subjects that then can have an impact on your team or organization and at the same time get you up to speed with newer technologies and trends. A few months back during one of these days I started getting my hands dirty with Rust. In the past I’ve been writing code in Go, due to its predominance in the k8s/Openshift world, so this really wasn’t my first experience with statically typed programming languages but the increased adoption Rust is having in multiple organizations and Open Source communities (with GNOME being one of them), its memory safety guardrails and the potential it has to slowly replace C in the very long run has hyped me to think it was a good idea for me to spend some time learning more about its internals.

The best way for me to get more proficient with a specific programming language is to write code with it, that also means I had to find a project of some sort to work on. In the back of my brain I always wanted to build my own cloud infrastructure with ability to spawn multi-arch VMs, have them configured using cloud-init and finally have them be able to communicate with the internet in one way or another (NAT? VLAN trunks?). At the same time I wanted a nice web UI to show the state of the VMs, be able to schedule new ones and perform specific actions on them. This is what encouraged me at some point in the development process to also learn a bit of ReactJS in order to be able to make the web UI look fancier with dynamic state updates.

Before actually starting any coding, I started thinking on what lower level technologies I was going to be using. The answer was OVN/OVS for the networking stack and libvirt/qemu for the virtualization layer. While Rust ships with a libvirt library I couldn’t find one for OVN so I had to code my own using the XML-RPC interface OVN ships with.

While this started as a Red Hat Learning Day project, it slowly became something I started working on at late night and/or during weekends. This is the reason why I decided to call it the AWP or Awesome Weekend Project.

MVP

It was important for me to clearly define what the MVP (Minimal Viable Product) of this private cloud solution could potentially be. I drafted the following requirements and decided to release version 0.0.1 only when these were met:

Ability to schedule a multi-arch VM using the API and the web UI
Ability to select multiple different operating systems
Ability for the system to boot using cloud-init and define a default password, also have a working networking stack
Ability to connect to the internet with trunked vlans and provider networks
Ability to configure a L2 network and have two VMs communicate at L2
Ability for VMs using a flat network to receive an IP address via OVN’s DHCP feature
Ability to display specs, status of a VM and other tenant components within the web UI
Have a working agent that would run on each hypervisor, collect specific metrics and send them to the controller to be able to then apply specific scheduler logic based on those data points
Have a basic scheduler in place
Have a deployment mechanism that could help me speed up the coding to testing phase, ideally using Ansible

The learning resources

I’d also like to share the resources I’ve used to learn Rust, associated libraries and ReactJS/TailwindCSS:

The Rust Book - https://doc.rust-lang.org/book
Easy Rust - https://dhghomon.github.io/easy_rust/Chapter_1.html
ReactJS Course - https://scrimba.com/learn-react-c0e
TailwindCSS Course - https://scrimba.com/learn-tailwind-css-c010

The Rust libraries I’ve used the most:

and the VSCode extensions I’ve interacted with:

Preliminary steps

My cluster consists of three systems:

Raspberry Pi 5 (aarch64), hostname: sweetrevenge
Beelink EQR6 (AMD Ryzen 9 6900HX, 24G of RAM), hostname: flumina
A virtual machine hosted on the Beelink that serves as the OVN and AWP controlplane, hostname: ovn-control-plane

Configuration on ovn-control-plane

[root@ovn-control-plane ~]# subscription-manager list
+-------------------------------------------+
 Installed Product Status
+-------------------------------------------+
Product Name: Red Hat Enterprise Linux Fast Datapath
Product ID: 329
Version: 9
Arch: x86_64

Product Name: Red Hat Enterprise Linux for x86_64
Product ID: 479
Version: 9.5
Arch: x86_64

dnf install openvswitch3.5 ovn24.09-host ovn24.09-vtep ovn24.09-central

systemctl enable --now openvswitch
systemctl enable --now ovn-controller

echo 'OVN_NORTHD_OPTS="--db-nb-addr=ovn-control-plane --db-nb-create-insecure-remote=yes --db-sb-addr=ovn-control-plane --db-sb-create-insecure-remote=yes --db-nb-cluster-local-addr=ovn-control-plane --db-sb-cluster-local-addr=ovn-control-plane --ovn-northd-nb-db=tcp:ovn-control-plane:6641 --ovn-northd-sb-db=tcp:ovn-control-plane:6642"' >> /etc/sysconfig/ovn
systemctl enable --now ovn-northd

ovs-vsctl set open . external-ids:ovn-remote=tcp:ovn-control-plane:6642
ovs-vsctl set open . external-ids:ovn-encap-type=geneve
ovs-vsctl set open . external-ids:ovn-encap-ip=192.168.1.15
ovs-vsctl set open . external-ids:ovn-bridge=br-int

ovs-vsctl show
ovn-sbctl show

Configuration on sweetrevenge

apt install ovn-central ovn-host

systemctl enable --now openvswitch-switch.service

ovs-vsctl set open . external-ids:ovn-remote=tcp:ovn-control-plane:6642
ovs-vsctl set open . external-ids:ovn-encap-ip=192.168.1.12
ovs-vsctl set open . external-ids:ovn-encap-type=geneve
ovs-vsctl set open . external-ids:ovn-bridge=br-int

ovs-vsctl show
ovn-sbctl show

Configuration on flumina

apt install ovn-central ovn-host

systemctl enable --now openvswitch-switch.service

ovs-vsctl set open . external-ids:ovn-remote=tcp:ovn-control-plane:6642
ovs-vsctl set open . external-ids:ovn-encap-ip=192.168.1.17
ovs-vsctl set open . external-ids:ovn-encap-type=geneve
ovs-vsctl set open . external-ids:ovn-bridge=br-int

ovs-vsctl show
ovn-sbctl show

Connectivity test

ovn-control-plane:

ovn-nbctl ls-add s0
ovn-nbctl lsp-add s0 port01
ovn-nbctl lsp-set-addresses port01 00:00:00:00:00:01
ovn-nbctl lsp-add s0 port02
ovn-nbctl lsp-set-addresses port02 00:00:00:00:00:02

ip link add name veth01 type veth peer name port01
ip netns add ns0
ip link set dev veth01 netns ns0
ip netns exec ns0 ip link set dev lo up
ip netns exec ns0 ip link set dev veth01 up
ip netns exec ns0 ip link set veth01 address 00:00:00:00:00:01
ip netns exec ns0 ip address add 10.0.0.1/24 dev veth01
ip link set dev port01 up
ovs-vsctl add-port br-int port01
ovs-vsctl set Interface port01 external_ids:iface-id=port01

firewall-cmd --permanent --zone=public --add-port=6642/tcp
firewall-cmd --permanent --zone=public --add-port=6081/udp

sweetrevenge:

The RPi also requires building the geneve kernel module when using Raspberry Pi OS as follows:

apt install bc bison flex libssl-dev make

git clone --branch rpi-6.6.y https://github.com/raspberrypi/linux.git

dpkg -l | grep raspi-firmware
ii raspi-firmware 1:1.20250305-1 all Raspberry Pi family GPU firmware and bootloaders

# Click on revision --> Browse files
https://github.com/raspberrypi/firmware/releases/tag/1.20250305

curl https://raw.githubusercontent.com/raspberrypi/firmware/f9ff9c8f22a148a555a2c090af9649ad84709dc4/extra/git_hash -o /root/git_hash

cd linux
git checkout $(cat /root/git_hash)

export ARCH=arm64
KERNEL=kernel_2712

make bcm2712_defconfig

sed -i 's/# CONFIG_GENEVE is not set/CONFIG_GENEVE=m/' .config

# Commit ID is taken from the matching kernel version as taken from uname -r
curl https://raw.githubusercontent.com/raspberrypi/firmware/f9ff9c8f22a148a555a2c090af9649ad84709dc4/extra/Module_2712.symvers -o Module.symvers

rm /lib/modules/$(uname -r)/build
ln -s /root/linux /lib/modules/$(uname -r)/build

make -j6 Image.gz modules dtbs

cp net/openvswitch/vport-geneve.ko /lib/modules/$(uname -r)/kernel/net/openvswitch/
cp drivers/net/geneve.ko /lib/modules/$(uname -r)/kernel/drivers/net/

depmod -a
modprobe geneve
update-initramfs -u

rm /lib/modules/$(uname -r)/build
ln -s /usr/src/linux-headers-$(uname -r) /lib/modules/$(uname -r)/build

# If something fails or the module cannot be loaded use:
# make mrproper
# this will clean up build generated objects

dkms status
modprobe geneve

ip link add name veth02 type veth peer name port02
ip netns add ns0
ip link set dev veth02 netns ns0
ip netns exec ns0 ip link set dev lo up
ip netns exec ns0 ip link set dev veth02 up
ip netns exec ns0 ip link set veth02 address 00:00:00:00:00:02
ip netns exec ns0 ip address add 10.0.0.2/24 dev veth02
ip link set dev port02 up
ovs-vsctl add-port br-int port02 external_ids:iface-id=port02
ovs-vsctl set Interface port02 external_ids:iface-id=port02

Libvirt

On every hypervisor run:

dnf install qemu-kvm libvirt-daemon qemu-img virsh libvirt-devel xorriso -y

On RPis:

apt install qemu-system-arm qemu-efi-aarch64 libvirt-daemon-system xorriso libvirt-dev libvirt-clients -y

Once the packages have been installed, make sure /var/lib/libvirt/images/RHEL9-base.qcow2 and generally any other OS image via $OS-base.qcow2 points to a Cloud image, this is a specific image built for cloud platforms and comes with cloud-init out of the box, that means you won’t have to build your own via Image Builder and composer-cli, which is also unfortunately limited to be able to only compose images for the same RHEL/CentOS release the system is running on (i.e you cannot build a RHEL 10 image on RHEL 9).

Rust

# For the aarch64 binary use:
# https://static.rust-lang.org/rustup/dist/aarch64-unknown-linux-gnu/rustup-init
curl https://static.rust-lang.org/rustup/dist/x86_64-unknown-linux-gnu/rustup-init -o rustup-init
./rustup-init

From there you can rustup to manage your Rust installations and switch between them based on your needs.

The code

As I promised myself when I started developing this project, as soon as all the requirements I had in mind for the MVP were met, I’d have published the code on GitHub for general consumption and to potentially find more people interested in the idea and be willing to contribute to it further in case they found having a private cloud platform, that could serve the needs of an homelab and be 100% Open Source and based on Open Source technologies and at the same time being incredibly easy to setup, interesting.

The AWP project code can be found at https://github.com/averi/AWP, both the backend and frontend are stored within the same repository.

A quick peek at the web UI

What post would this be without some screenshots?!

AWP Homepage

Tenant view and new VM creation

License

I’ve chosen to release AWP under the GNU General Public License version 3 (GPLv3). I picked this license because of two main reasons:

Keep AWP Open Source: The core principle behind GPLv3 is the so-called “copyleft.” This means if you take AWP’s code and make something new with it, you need to share your new creation under the same license and terms. This ensures that AWP and any projects based on it will always remain open-source for the whole community to benefit from. It prevents the project from being turned into a closed, proprietary product.
Attribution: The GPLv3 requires that the original copyright notices are kept in the code. So, if someone uses or changes AWP, they must preserve the attribution (the Copyright header on each source file, that is) to the original authors and contributors. This helps recognizing the efforts of the original project author(s) who have put time, passion and dedication into code development and maintenance.

The future

While AWP began as a personal learning activity, I believe it has the potential to evolve into something more concrete. Looking ahead, I can envision several enhancements that would significantly expand its capabilities. For instance, implementing a third networking mode in order to support VMs that are configured with a flat network and want to be able to connect to the internet through NAT, this can be easily achieved by leveraging existing OVN features such as Logical Routers.

My hope, as I mentioned earlier, is that AWP won’t remain a one-man show. If you see value in AWP and are excited by the idea of shaping an open-source, easy-to-manage private cloud platform, I wholeheartedly encourage you to jump in. Whether it’s by contributing code, coming up with some initial documentation or by just sharing ideas, your involvement could be key to developing AWP further.

2024 GNOME Infrastructure Annual Review

Andrea Veri — Fri, 13 Dec 2024 16:29:20 -0500

Table of Contents
1. Introduction
2. Achievements

1. Introduction

Time is passing by very quickly and another year will go as we approach the end of 2024. This year has been fundamental in shaping the present and the future of GNOME’s Infrastructure with its major highlight being a completely revamped platform and a migration of all GNOME services over to AWS. In this post I’ll try to highlight what the major achievements have been throughout the past 12 months.

2. Achievements

In the below is a list of individual tasks and projects we were able to fulfill in 2024. This section will be particularly long but I want to stress the importance of each of these items and the efforts we put in to make sure they were delivered in a timely manner.

2.1. Major achievements

All the applications (except for ego, which we expect to handle as soon as next week or in January) were migrated to our new AWS platform (see GNOME Infrastructure migration to AWS)
During each of the apps migrations we made sure to:
1. Migrate to sso.gnome.org and make 2FA mandatory
2. Make sure database connections are handled via connection poolers
3. Double check the container images in use were up-to-date and GitLab CI/CD pipeline schedules were turned on for weekly rebuilds (security updates)
4. For GitLab, we made sure repositories were migrated to an EBS volume to increase IO throughput and bandwidth
Migrated away our backup mechanism away from rdiff-backup into AWS Backup service (which handles both our AWS EFS and EBS snapshots)
Retired our NSD install and migrated our authoritative name servers to CloudNS (it comes with multiple redundant authoritative servers, DDOS protection and automated DNSSEC keys rotation and management)
We moved away from Ceph and the need to maintain our own storage solution and started leveraging AWS EFS and EBS
We deprecated Splunk and built a solution around promtail and Loki in order to handle our logging requirements
We deprecated Prometheus blackbox and started leveraging CloudNS monitoring service which we interact with using an API and a set of CI/CD jobs we host in GitHub
We archived GNOME’s wiki and turned it into a static HTML copy
We replaced ftpadmin with the GNOME Release Services, thanks speknik! More information around what steps should GNOME Maintainers now follow when doing a module release are available here. The service uses JWT tokens to verify and authorize specific CI/CD jobs and only allows new releases when the process is initiated by a project CI living within the GNOME GitLab namespace and a protected tag. With master.gnome.org and ftpadmin being in production for literally ages, we wanted to find a better mechanism to release GNOME software and avoid a single maintainer SSH key leak to allow a possible attacker to tamper tarballs and potentially compromise milions of computers running GNOME around the globe. With this change we don’t leverage SSH anymore and most importantly we don’t allow maintainers to generate GNOME modules tarballs on their personal computers rather we force them to use CI/CD in order to achieve the same result. We’ll be coming up shortly with a dedicated and isolated runner that will only build jobs tagged as releasing GNOME software.
We retired our mirroring infrastructure based on Mirrorbits and replaced it with our CDN partner, CDN77
We decoupled GIMP mirroring service from GNOME’s one, GIMP now hosts its tarballs (and associated rsync daemon) on top of a different master node, thanks OSUOSL for sponsoring the VM that makes this possible!

2.2. Minor achievements

Retired multiple VMs: splunk, nsd0{1,2}, master, ceph-metrics, gitaly
We started managing our DNS using an API and CI/CD jobs hosted in GitHub (this to avoid relying on GNOME’s GitLab which in case of unavailability would prevent us to update DNS entries)
We migrated smtp.gnome.org to OSCI in order to not lose IP reputations and various whitelists we received throughout the years by multiple organizations
We deprecated our former internal DNS authoritatives based on FreeIPA. We are now leveraging internal VPC resolvers and Route53 Private zones
We deprecated all our OSUOSL GitLab runners due to particularly slow IO and high steal time and replaced them with a new Heztner EX44 instance, kindly sponsored by GIMP. OSUOSL is working on coming up with local storage on their Openstack platform. We are looking forward to test that and introduce new runners as soon as the solution will be made available
Retired idm0{1,2} and redirected them to a new FreeIPA load balanced service at https://idm.gnome.org
We retired services which weren’t relevant for the community anymore: surveys.gnome.org, roundcube (aka webmail.gnome.org)
We migrated nmcheck.gnome.org to Fastly and are using Synthetic responses to handle HTTP responses to clients
We upgraded to Ansible Automation Platform (AAP) 2.5
As part of the migration to our new AWS based platform, we upgraded Openshift to release 4.17
We received a 2k grant from Microsoft which we are using for an Azure ARM64 GitLab runner
All of our GitLab runners fleet are now hourly kept in sync using AAP (Ansible roles were built to make this happen)
We upgraded Cachet to 3.x series and fixed dynamic status.gnome.org updates (via a customized version of cachet-monitor)
OS Currency: we upgraded all our systems to RHEL 9
We converted all our Openshift images that were using a web server to Nginx for consistency/simplicity
Replaced NRPE with Prometheus metrics based logging, checks such as IDM replication and status are now handled via the Node Exporter textfile plugin
Migrated download.qemu.org (yes, we also host some components of QEMU’s Infrastructure) to use nginx-s3-gateway, downloads are then served via CDN77

2.3 Minor annoyances/bugs that were also fixed in 2024

Invalid OCSP responses from CDN77, https://gitlab.gnome.org/Infrastructure/Infrastructure/-/issues/1511
With the migration to USE_TINI for GitLab, no gpg zombie processes are being generated anymore

2.3. Our brand new and renewed partnerships

From November 2024 and ongoing, AWS will provide sponsorship and funding to the GNOME Project to sustain the majority of its infrastructure needs
Red Hat kindly sponsored subscriptions for RHEL, Openshift, AAP as well as hosting, bandwidth for the GNOME Infrastructure throughout 2024
CDN77 provided unlimited bandwidth / traffic on their CDN offering
Fastly renewed their unlimited bandwidth / traffic plan on their Delivery/Compute offerings
and thanks to OSUOSL, Packet, DigitalOcean, Microsoft for the continued hosting and sponsorship of a set of GitLab runners, virtual machines and ARM builders!

Expressing my gratitude

As I’m used to do at the end of each calendar year, I want to express my gratitude to Bartłomiej Piotrowski for our continued cooperation and also to Stefan Peknik for his continued efforts in developing the GNOME Release Service. We started this journey together many months ago when Stefan was trying to find a topic to base his CS bachelor thesis on. With this in mind I went straight into the argument of replacing ftpadmin with a better technology also in light of what happened with the xz case. Stefan put all his enthusiasm and professionality into making this happen and with the service going into production on the 11th of December 2024 history was made.

That being said, we’re closing this year being extremely close to retire our presence from RAL3 which we expect to happen in January 2025. The GNOME Infrastructure will also send in a proposal to talk at GUADEC 2025, in Italy, to present and discuss all these changes with the community.

GNOME Infrastructure migration to AWS

Andrea Veri — Wed, 16 Oct 2024 20:25:12 -0400

1. Some historical background

The GNOME Infrastructure has been hosted as part of one of Red Hat’s datacenters for over 15 years now. The “community cage”, which is how we usually define the hosting platform that backs up multiple Open Source projects including OSCI, is made of a set of racks living within the RAL3 (located in Raleigh) datacenter. Red Hat has not only been contributing to GNOME by maintaining the Red Hat’s Desktop Team operational, sponsoring events (such as GUADEC) but has also been supporting the project with hosting, internet connectivity, machines, RHEL (and many other RH products subscriptions). When the infrastructure was originally stood up it was primarily composed of a set of bare metal machines, workloads were not yet virtualized at the time and many services were running directly on top of the physical nodes. The advent of virtual machines and later containers reshaped how we managed and operated every component. What however remained the same over time was the networking layout of these services: a single L2 and a shared (with other tenants) public internet L3 domains (with both IPv4 and IPv6).

Recent challenges

When GNOME’s Openshift 4 environment was built back in 2020 we had to make specific calls:

We’d have ran an Openshift Hyperconverged setup (with storage (Ceph), control plane, workloads running on top of the same subset of nodes)
The total amount of nodes we received budget for was 3, this meant running with masters.schedulable=true
We’d have kept using our former Ceph cluster (as it had slower disks, a good combination for certain workloads we run), this is however not supported by ODF (Openshift Data Foundation) and would have required some glue to make it completely functional
Migrating GNOME’s private L2 network to L3 would have required an effort from Red Hat’s IT Network Team who generally contributes outside of their working hours, no changes were planned in this regard
No changes were planned on the networking equipment side to make links redundant, that means a code upgrade on switches would have required a full services downtime

Over time and with GNOME’s users and contributors base growing (46k users registered in GitLab, 7.44B requests and 50T of traffic per month on services we host on Openshift and kindly served by Fastly’s load balancers) we started noticing some of our original architecture decisions weren’t positively contributing to platform’s availability, specifically:

Every time an Openshift upgrade was applied, it resulted in a cluster downtime due to the unsupported double ODF cluster layout (one internal and one external to the cluster). The behavior was stuck block devices preventing the machines to reboot with associated high IO (and general SELinux labeling mismatches), with the same nodes also hosting OCP’s control plane it was resulting in API and other OCP components becoming unavailable
With no L3 network, we had to create a next-hop on our own to effectively give internet access through NAT to machines without a public internet IP address, this was resulting in connectivity outages whenever the target VM would go down for a quick maintenance

Migration to AWS

With budgets season for FY25 approaching we struggled finding the necessary funds in order to finally optimize and fill the gaps of our previous architecture. With this in mind we reached out to AWS Open Source Program and received a substantial amount for us to be able to fully transition GNOME’s Infrastructure to the public cloud.

What we achieved so far:

Deployed and configured VPC related resources, this step will help us resolve the need to have a next-hop device we have to maintain
Deployed an Openshift 4.17 cluster (which uses a combination of network and classic load balancers, x86 control plane and arm64 workers)
Deployed IDM nodes that are using a Wireguard tunnel between AWS and RAL3 to remain in sync
Migrated several applications including SSO, Discourse, Hedgedoc

What’s upcoming:

Migrating away from Splunk and use a combination of rsyslog/promtail/loki
Keep migrating further applications, the idea is to fully decommission the former cluster and GNOME’s presence within Red Hat’s community cage during Q1FY25
Introduce a replacement for master.gnome.org and GNOME tarballs installation
Migrate applications to GNOME’s SSO
Retire services such as GNOME’s wiki (MoinMoin, a static copy will instead be made available), NSD (authoritative DNS servers were outsourced and replaced with ClouDNS and GitHub’s pipelines for DNS RRs updates), Nagios, Prometheus Blackbox (replaced by ClouDNS endpoints monitoring service), Ceph (replaced by EBS, EFS, S3)
Migrate smtp.gnome.org to OSCI in order to maintain current public IP’s reputation

And benefits of running GNOME’s services in AWS:

Scalability, we can easily scale up our worker nodes pool
We run our services on top of AWS SDN and can easily create networks, routing tables, benefit from faster connectivity options, redundant networking infrastructure
Use EBS/EFS, don’t have to maintain a self-managed Ceph cluster, easily scale volumes IOPS
Use a local to-the-VPC load balancer, less latency for traffic to flow between the frontend and our VPC
Have access to AWS services such as AWS Shield for advanced DDOS protection (with one bringing down GNOME’s GitLab just a week ago)

I’d like to thank AWS (Tom “spot” Callaway, Mila Zhou) for their sponsorship and the massive opportunity they are giving to the GNOME’s Infrastructure to improve and provide resilient, stable and highly available workloads to GNOME’s users and contributors base. And a big thank you to Red Hat for the continued sponsorship over more than 15 years on making the GNOME’s Infrastructure run smoothly and efficiently, it’s crucial for me to emphatise how critical Red Hat’s long term support has been.

2022 GNOME Infrastructure Annual Review

Andrea Veri — Mon, 12 Dec 2022 17:53:26 +0100

1. Introduction
2. Achievements
3. Highlights
Future plans
Expressing my gratitude

1. Introduction

I believe it’s kind of vital for the GNOME Infrastructure Team to outline not only the amazing work that was put into place throughout the year, but also the challenges we faced including some of the architectural designs we implemented over the past 12 months. This year has been extremely challenging for multiple reasons, the top one being Openshift 3 (which we deployed in 2018) going EOL in June 2022. We also wanted to make sure we were keeping up with OS currency, specifically finalizing the migration of all our VM-based workloads to RHEL 8 and most importantly to Ansible. The main challenges there being adapting our workflow away from the Source-To-Image (s2i) mechanism into building our own infrastructure images directly through GitLab CI/CD pipelines by ideally also dropping the requirement of hosting an internal containers registry.

With the community also deciding to move away from Mailman, we also had an hard deadline to finalize the migration to Discourse that was started back in 2018. At the same time the GNOME community was also looking towards a better Matrix to IRC integration while irc.gimp.org (GIMPNet) showing aging sypmptoms and being put into very low maintenance mode due to a lack of IRC operators/admins time.

2. Achievements

A list of each of the individual tasks and projects we were able to fulfill during 2022. This particular section will be particularly long but I want to stress the importance of each of these items and the efforts we put in to make sure they were delivered in a timely manner. A subset of these tasks will also receive further explanation in the sections to come.

2.1. Major achievements

Architected, deployed, operated an Openshift 4 cluster that replaced our former OCP 3 installation. This was a major undertaking that required a lot of planning, testing and coordination with the community. We also had to make sure we were able to migrate all our existing tenants from OCP 3 to OCP 4. A total of 46 tenants were migrated and/or created during these past 12 months.
Developed a brand new workflow moving away from Source-To-Image (s2i) and towards GitLab CI/CD pipelines.
Migrated from individual NSD based internal resolvers to FreeIPA’s self managed BIND resolvers, this gave us the possibility to use APIs and other IPA’s goodies to manage our internal DNS views.
For existing virtual machines we wanted to keep around, we leveraged the Openshift Virtualization operator which allows you to benefit from kubevirt’s features and effectively run virtual machines from within an OCP cluster. That also includes support for VM templates and automatic bootstraping of VMs out of existing PVCs and/or externally hosted ISO files.
We developed and deployed automation for handling Membership and Accounts related requests. The documentation has also been updated accordingly.
GitLab was migrated from a monolithic virtual machine to Openshift.
We introduced DKIM/SPF support for GNOME Foundation members, please see my announcement for a list of changes.
We rebuilt all the VMs that were not retired due to the migration to OCP to RHEL 8
We successfully migrated away from our Puppet infrastructure to Ansible (and Ansible collections). This particular task is a major milestone, Puppet has been around the GNOME Infrastructure for more than 15 years.

2.2. Minor achievements

Identified root cause and blocked a brute force attempt (from 600+ unique IP addresses) against our LDAP database directory. Some of you surely remind the times where you found your GNOME Account locked and you were unsure around why that was happening. This was the exact reason why. A particular remediation was applied temporarily, that also had the side effect of blocking HTTPs based git clones/pushes. That was resolved by moving GitLab to OpenID (via Keycloak) and using token based authentication.
We moved static.gnome.org to S3 and put our CDN in front of it.
Re-deployed bastion01, nsd01, nsd02, smtp, master, logs (rsyslog) using Ansible (this also includes building Ansible roles and replicating what we had in Puppet to Ansible)
Deployed Minio S3 GW (cache) to avoid incurring in extra AWS S3 costs
Deprecated OpenVPN in favor of Wireguard
We deprecated GlusterFS entirely and completed our migration to Ceph RBD + CephFS
We retired several VM based workloads that were either migrated to Openshift or superseded including reverse proxies, Puppet masters, GitLab, the entirety of OCP 3 virtual machines (with OCP 4 being installed on bare metals directly)
Configured blackbox Prometheus exporter and moved services availability checks to it
We retired people.gnome.org, barely used by anyone due to the multitude of alternatives we currently provide when it comes to host GNOME related files including GitLab pages, static.gnome.org, GitLab releases, Nextcloud.
Started ingesting Prometheus metrics into our existing Prometheus cluster via federation, a wide set of dashboards were also created to keep track of the status of OCP, Ceph, general OS related metrics and databases.
We migrated our databases to OCP: Percona MySQL operator, Crunchy PostgreSQL operator
Rotated KSK and ZSK DNSSEC keys on gnome.org, gtk.org, gimp.{org,net} domains
We migrated from obtaining Let’s Encrypt certificates using getssl to OCP CertManager operator. For edge routers, we migrated to certbot and deployed specific hooks to automate the handling of DNS-01 challenges.
We migrated GIMP downloads from a plain httpd setup to use mirrorbits to match what the GNOME Project is operating.
We deployed AAP (Red Hat Ansible Automation Platform) in order to be able to recreate hourly configuration management runs as we had before with Puppet. These runs are particularly crucial as they make sure the latest content from our Ansible repository is pulled and enforced across all the systems Ansible manages.
irc.gnome.org migration to Libera.Chat, thanks Thibault Martin and Element for the amazing continued efforts supporting GNOME’s Matrix to IRC bridge integration!
Migrated away from Mailman to Discourse. This particular item has been part of community discussions since 2018, after evaluation by the community itself and the GNOME Project governance the migration to Discourse started and was finalized this year, please read here for a list of FAQs.
We introduced OpenID authentication (via Keycloak) to help resolve the fragmentation multiple different authentication backends were causing.
We introduced Hedgedoc, an Etherpad replacement.
We enhanced our Splunk cluster with additional dashboards, log based alerts, new sourcetypes
We deprecated MeetBot (unused since several years) and CommitsBot, which we replaced with a beta Matrix bot called Hookshot, which leverages GitLab webhooks in order to send notifications to Matrix rooms
We upgraded FreeIPA to version 4.9.10, and on RHEL 8. We enhanced IPA backups to include hourly file system snapshots (on top of the existing rdiff-backup runs) and daily ipa-backup runs.
We presented at GUADEC 2022.

2.3. Our brand new and renewed partnerships

Red Hat kindly sponsored subscriptions for RHEL, Ceph, Openshift, AAP
Splunk doubled the sponsorship to a total of 10GB/day traffic
AWS confirmed their partnership with a total of 5k USD credit
CDN77 provided unlimited bandwidth / traffic on their CDN offering
We’re extremely close to finalize our partnership with Fastly! They’ll be providing us with their Traffic Load Balancing product
and thanks to OSUOSL, Packet, DigitalOcean for the continued hosting and sponsorship of a set of GitLab runners, virtual machines and ARM builders!

3. Highlights

Without going too much deep into technical details I wanted to provide an overview of how we architected and deployed our Openshift 4 cluster and GitLab as these questions pop up pretty frequently among contributors.

3.1. Openshift 4: architecture

The cluster is currently setup with a total of 3 master nodes having similar specs (256G of RAM, 62 Cores, 2x10G NICs, 2x1G NICs) and acting in a hyperconverged setup. That implies we’re also hosting a Ceph cluster (in addition to the existing one we setup a while back) we deployed via the Red Hat Openshift Data Foundations operator on the same nodes. OCP (release 4.10), in this scenario, is deployed directly on bare metal with an ingress endpoint per individual node. The current limitation to this particular architecture is there’s no proper load balancing (other than DNS Round Robin) in front of these ingresses due to the fact external load balancers are particularly expensive. As I’ve mentioned earlier we’re really close to finalize a partnership with Fastly to help fill the gap here. These nodes receive their configuration using Ignition, we made sure a specific set of MachineConfigs is there to properly configure these systems once they boot due to the way CoreOS works in this regard. At boot time, it fetches an Ignition definition from the Machine Config Operator controller and applies it to the target node.

3.2. Openshift 4: virtualization networking

As I previously mentioned we’ve been leveraging OCP CNV to support our VM based workloads. I wanted to quickly highlight how we handled the configuration of our internal and public networks in order for these VMs to successfully consume these subnets and be able to communicate back and forth with other data center resources and services:

A set of bonded interfaces was setup for both the 10G and the 1G NICs
A bridge was configured on top of these bonded interfaces, that is required by Openshift Multus to effectively append the VM interfaces to each of these bridges depending what kind of subnet(s) they’re required to access
We configured OCP Multus (bridge mode) and its dependant NetworkAttachmentDefinition
From within an OCP CNV CRD, pass in:

 networks:
 - multus:
 networkName: internal-network
 name: nic-1

And a sample of the internal-network NetworkAttachmentDefinition:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
 name: internal-network
 namespace: infrastructure
spec:
 config: >-
 {"name":"internal-network","cniVersion":"0.3.1","plugins":[{"type":"cnv-bridge","bridge":"br0-internal","mtu":9000,"ipam":{}},{"type":"cnv-tuning"}]}

3.3. Openshift 4: image builds

One of the major changes we implemented with the migration to OCP 4 was the way we built infrastructure related container images. In early days we were leveraging the s2i OCP feature which allowed building images out of a git repository, those builds were directly happening from within OCP worker nodes and pushed to the internal OCP registry. With the new setup what happens instead is:

We create a new git repository containing an application and an associated Dockerfile
From within that repository, we define a .gitlab-ci.yml file that inherits the build templates from a common set of templates we created
The image is then built using GitLab CI/CD and pushed to quay.io
On the target OCP tenant, we define an ImageStream and point it to the quay.io registry namespace/image combination
From there the Deployment/DeploymentConfig resource is updated to re-use the previously created ImageStream, whenever the ImageStream changes, the deployment/deploymentconfig is triggered (via ImageChange triggers)

3.4. Openshift 4: cluster backups

When it comes to cluster backups we decided to take the following approach:

Run daily etcd backups
Backup and dump all the tenants CRDs as json files to an encrypted S3 bucket using Velero

3.5. GitLab on Openshift 4: setup

Moving away from hosting GitLab on a monolithic virtual machine was surely one of our top goals for 2022. The reason was particularly simple, anytime we needed to perform a maintenance we were required to cause a service downtime, even during a plain minor platform upgrade. On top of that, we couldn’t easily scale the cluster in case of sudden peeks in traffic, but generally when we originally designed our GitLab offering back in 2018 we missed a lot of the goodies OCP provides, the installation has worked well during all these years but the increasing usage of the service, the multitude of new GitLab sub-components made us rethink the way we had to design this particular offering to the community.

These are the main reasons why we migrated GitLab to OCP using the GitLab OCP Operator. Using operator’s built-in declarative resources functionalities we could easily replicate our entire cluster config in a single yaml file, the operator at that point picked up each of our definitions and generated individual configmaps, deployments, scaleapps, services, routes and associated CRDs automatically. The only component we decided to not host via OCP directly but use a plain VM on OCP Virt was gitaly. The reason is particularly simple: gitaly requires port 22 to be accessible from outside of the cluster, that is currently not possible with the default haproxy based OCP ingress. We analyzed whether it made sense to deploy the NGINX ingress which also supports reverse proxying non-HTTP ports, we thought that’d have added additional complexity with no particular benefit. MetalL was also a possibility but the product itself is still a WIP, required sending out gARPs on the public network block we share with other community tenants for L2, for L3 there was a need to setup BGP peering between each of the individual nodes (speakers using MetalLB terminology) and an adiacent router, overkill for a single VIP.

3.6. GitLab on Openshift 4: early days

Right after the migration we started observing some instability with specific pods (webservice, sidekiq) backtracing after a few hours they were running, specifically:

Nov 16 05:08:03 master2.openshift4.gnome.org kernel: cgroup: fork rejected by pids controller in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69234f3_8596_477c_b7ea_5b51f6d86cce.slice/crio-d36391a108570d6daecf316d6d19ffc6650a3fa3a82ee616944b9e51266c901f.scope

also on kubepods.slice:

[core@master2 ~]$ cat /sys/fs/cgroup/pids/kubepods.slice/pids.max
4194304

It was clear the target pods were spawning a major set of new processes that were remaining around for the entire pod lifetime:

$ ps aux | grep gpg | wc -l
773

And a sample out of the previous ‘ps aux’ run:

git 19726 0.0 0.0 0 0 ? Z 09:58 0:00 [gpg] <defunct>
git 19728 0.0 0.0 0 0 ? Z 09:58 0:00 [gpg] <defunct>
git 19869 0.0 0.0 0 0 ? Z 10:06 0:00 [gpg] <defunct>
git 19871 0.0 0.0 0 0 ? Z 10:06 0:00 [gpg] <defunct>

It appears this specific bug was troubleshooted already by the GitLab Infrastructure Team around 4 years ago already. This misbehaviour is related to the intimate nature of GnuPG which requires calling its binaries (gpgconf, gpg, gpgsm, gpg-agent) for every required operation GitLab (webservice or sidekiq) asks it to perform. For some reason these processes never notified their parent process (PID 1 on that particular container) with a SIGCHLD and remained hanging around on the pods until pod’s dismissal. We’re in touch with the GitLab Open Source program support to understand next steps in order to have a fix implemented upstream.

3.7. GitLab on Openshift 4: logging

As part of our intent to migrate as much services as possible to our centralized rsyslog cluster (which then injects those logs into Splunk) we decided to approach GitLab’s logging on OCP this way:

We mounted a shared PVC on each of the webservice/sidekiq pods, the target directory was the one GitLab was expected to send its logs by default (/var/log/gitlab)
From there we deployed a separate rsyslogd deployment that was also mounting the same shared PVC
We configured rsyslogd to relay those logs to our centralized rsyslog facility making sure proper facilities, tags, severities were also forwarded as part of the process
Relevant configs, Dockerfile and associated deployment files are publicly available

Future plans

Some of the tasks we have planned for the upcoming months:

Move away from ftpadmin and replace it with a web application and/or CLI to securely install a sources tarball without requiring shell access. (Also introduced tarball signatures?)
Introduce OpenID on GNOME’s Matrix homeserver, merge existing Foundation member accounts
Migrate OCP ingress endpoints to Fastly LBs
Upgrade Ceph to Ceph 5
Look at migrating OCP to OVNKubernetes to start supporting IPv6 endpoints (again) - (minor priority)
Load balance IPA’s DNS and LDAPs traffic (minor priority)
Migrate GitLab runners Ansible roles and playbooks to AAP (minor priority)

Expressing my gratitude

I wanted to take a minute to thank all the individuals who helped us accomplishing this year amazing results! And a special thank you to Bartłomiej Piotrowski for his precious insights, technical skills and continued friendship.

GNOME Infrastructure updates

Andrea Veri — Mon, 30 Mar 2020 12:20:23 +0000

As you may have noticed from outage and maintenance notes we sent out last week the GNOME Infrastructure has been undergoing a major redesign due to the need of moving to a different datacenter. It’s probably a good time to update the Foundation membership, contributors and generally anyone consuming the multitude of services we maintain of what we’ve been up to during these past months.

New Data Center

One of the core projects for 2020 was moving off services from the previous DC we were in (located in PHX2, Arizona) over to the Red Hat community cage located in RAL3. This specific task was made possible right after we received a new set of machines that allowed us to refresh some of the ancient hardware we had (with the average box dating back to 2013). The new layout is composed of a total of 5 (five) bare metals and 2 (two) core technologies: Openshift (v. 3.11) and Ceph (v. 4).

The major improvements that are worth being mentioned:

VMs can be easily scheduled across the hypervisors stack without having to copy disks over across hypervisors themselves. VM disks and data is now hosted within Ceph.
IPv6 is available (not yet enabled/configured at the OS, Openshift router level)
Overall better external internet uplink bandwidth
Most of the VMs that we had running were turned into pods and are now successfully running from within Openshift

RHEL 8 and Ansible

One of the things we had to take into account was running Ceph on top of RHEL 8 to benefit from its containarized setup. This originally presented itself as a challenge due to the fact RHEL 8 ships with a much newer Puppet release than the one RHEL 7 provides. At the same time we didn’t want to invest much time in upgrading our Puppet code base due to the amount of VMs we were able to migrate to Openshift and to the general willingess of slowly moving to use Ansible (client-side, no more need of maintaining server side pieces). On this specific regard we:

Landed support for RHEL 8 provisioning
Started experimenting with Image Based deployments (much more faster than Cobbler provisioning)
Cooked a set of base Ansible roles to support our RHEL 8 installs including IDM, chrony, Satellite, Dell OMSA , NRPE etc.

Openshift

As originally announced, the migration to the Openshift Container Platform (OSCP) has progressed and we now count a total of 34 tenants (including the entirety of GIMP websites). This allowed us to:

Retire running VMs and prevented the need to upgrade their OS whenever they’re close to EOL. Also, in general, less maintenance burden
Allow the community to easily provision services on top of the platform with total autonomy by choosing from a wide variety of frameworks, programming languages and database types (currently Galera and PSQL, both managed outside of OSCP itself)
Easily scale the platform by adding more nodes/masters/routers whenever that is made necessary by additional load
Data replicated and made redundant across a GlusterFS cluster (next on the list will be introducing Ceph support for pods persistent storage)
Easily set up services such as Rocket.Chat and Discourse without having to mess much around with Node.JS or Ruby dependencies

Special thanks

I’d like to thank Bartłomiej Piotrowski for all the efforts in helping me out with the migration during the past couple of weeks and Milan Zink from the Red Hat Storage Team who helped out reviewing the Ceph infrastructure design and providing useful information about possible provisioning techniques.

The GNOME Infrastructure is moving to Openshift

Andrea Veri — Thu, 18 Oct 2018 10:27:23 +0000

During GUADEC 2018 we announced one of the core plans of this and the coming year: it being moving as many GNOME web applications as possible to the GNOME Openshift instance we architected, deployed and configured back in July. Moving to Openshift will allow us to:

Save up on resources as we’re deprecating and decommissioning VMs only running a single service
Allow app maintainers to use the most recent Python, Ruby, preferred framework or programming language release without being tied to the release RHEL ships with
Additional layer of security: containers
Allow app owners to modify and publish content without requiring external help
Increased apps redundancy, scalability, availability
Direct integration with any VCS that ships with webhooks support as we can trigger the Openshift provided endpoint whenever a commit has occurred to generate a new build / deployment

Architecture

The cluster consists of 3 master nodes (controllers, api, etcd), 4 compute nodes and 2 infrastructure nodes (internal docker registry, cluster console, haproxy-based routers, SSL edge termination). For the persistent storage we’re currently making good use of the Red Hat Gluster Storage (RHGS) product that Red Hat is kindly sponsoring together with the Openshift subscriptions. For any app that might require a database we have an external (as not managed as part of Openshift) fully redundant, synchronous, multi-master MariaDB cluster based on Galera (2 data nodes, 1 arbiter).

The release we’re currently running is the recently released 3.11, which comes with the so-called “Cluster Console”, a web UI that allows you to manage a wide set of the underlying objects that previously were only available to the oc cli client and with a set of Monitoring and Metrics toolings (Prometheus, Grafana) that can be accessed as part of the Cluster Console (Grafana dashboards that show how the cluster is behaving) or externally via their own route.

SSL Termination

The SSL termination is currently happening on the edge routers via a wildcard certificate for the gnome.org and guadec.org zones. The process of renewing these certificates is automated via Puppet as we’re using Let’s Encrypt behind the scenes (domain verification for the wildcard certs happen at the DNS level, we built specific hooks in order to make that happen via the getssl tool). The backend connections are following two different paths:

edge termination with no re-encryption in case of pods containing static files (no logins, no personal information ever entered by users): in this case the traffic is encrypted between the client and the edge routers, plain text between the routers and the pods (as they’re running on the same local broadcast domain)
re-encrypt for any service that requires authentication or personal information to be entered for authorization: in this case the traffic is encrypted from end to end

App migrations

App migrations have started already, we’ve successfully migrated and deprecated a set of GUADEC-related web applications, specifically:

$year.guadec.org where $year spaces from 2013 to 2019
wordpress.guadec.org has been deprecated

We’re currently working on migrating the GNOME Paste website making sure we also replace the current unmaintained software to a supported one. Next on the list will be the Wordpress-based websites such as www.gnome.org and blogs.gnome.org (Wordpress Network). I’d like to thank the GNOME Websites Team and specifically Tom Tryfonidis for taking the time to migrate existing assets to the new platform as part of the GNOME websites refresh program.

Back from GUADEC 2018

Andrea Veri — Mon, 30 Jul 2018 10:27:23 +0000

Been a while since GUADEC 2018 has ended but subsequent travels and tasks reduced the time to write up a quick summary of what happened during this year’s GNOME conference. The topics I’d like to emphasize mainly are:

We’re hiring another Infrastructure Team member
We’ve successfully finalized the cgit to GitLab migration
Future plans including the migration to Openshift

GNOME Foundation hirings

With the recent donation of 1M the Foundation has started recruiting on a variety of different new professional roles including a new Infrastructure team member. On this side I want to make sure that although the job description mentions the word sysadmin the figure we’re looking for is a systems engineer with a proven experience on Cloud computing platforms and tools such as AWS, Openshift and generally configuration management softwares such as Puppet and Ansible. Additionally this person should prove to have a clear understanding of the network and operating system (mainly RHEL) layers.

We’ve already identified a set of candidates and will be proceeding with interviews in the coming weeks. This doesn’t mean we’ve hired anyone already, please keep sending CVs if interested and feeling the position would match your skills and expectations.

cgit to GitLab

As announced on several occasions the GNOME Infrastructure has successfully finalized the cgit to GitLab migration. From a read-only view against .git directories to a fully compliant CI/CD infrastructure. The next step on this side will be deprecating Bugzilla which has already started with bugmasters turning products read-only in case they were migrated to the new platform or by identifying whether any of the not-yet-migrated products can be archived. The idea here is waiting to see zero activity on BZ in terms of new comments to existing bugs and no new bugs being submitted at all (we have redirects in place to make sure whenever enter_bug.cgi is triggered the request gets sent to the /issues/new endpoint for that specific GitLab project) and then turn the entire BZ instance into an HTML archive for posterity and to reduce the maintenance burden of keeping an instance up-to-date with upstream in terms of CVEs.

Thanks to all the involved parties including Carlos, Javier and GitLab itself given the prompt and detailed responses they always provided to our queries. Also thanks for sponsoring our AWS activities related to GitLab!

Future plans

With the service == VM equation we’ve been following for several years it’s probably time for us to move to a more scalable infrastructure. The next generation platform we’ve picked up is going to be Openshift, its benefits:

It’s not important where a service runs behind scenes: it only has to run (VM vs pods and containers that are part of a pod that get scheduled randomly on the available Openshift nodes)
Easily scalable in case additional resources are needed for a small period of time
Built-in monitoring starting from Openshift 3.9 (the release we’ll be running) based on Prometheus (+ Grafana for dashboarding)
GNOME services as containers
Individual application developers to schedule their own builds and see their code deployed with one click directly in production

The base set of VMs and bare metals has been already configured. Red Hat has been so great to provide the GNOME Project with a set of Red Hat Container Platform (and GlusterFS for heketi-based brick provisioning) subscriptions. We’ll start moving over to the infrastructure in the coming weeks. It’s going to take some good time but in the end we’ll be able to free up a lot of resources and retire several VMs and related deprecated configurations.

Misc

Slides from the Foundation AGM Infrastructure team report are available here.

Many thanks to the GNOME Foundation for the sponsorship of my travel!

Adding reCAPTCHA v2 support to Mailman

Andrea Veri — Mon, 26 Feb 2018 15:13:09 +0000

As a follow-up to the reCAPTCHA v1 post published back in 2014 here it comes an updated version for migrating your Mailman instance off from version 1 (being decommissioned on the 31th of March 2018) to version 2. The original python-recaptcha library was forked into https://github.com/redhat-infosec/python-recaptcha and made compatible with reCAPTCHA version 2.

The relevant changes against the original library can be resumed as follows:

Added ‘version=2’ against displayhtml, load_scripts functions
Introduce the v2submit (along with submit to keep backwards compatibility) function to support reCAPTCHA v2
The updated library is backwards compatible with version 1 to avoid unexpected code breakages for instances still running version 1

The required changes are located on the following files:

/usr/lib/mailman/Mailman/Cgi/listinfo.py

--- listinfo.py 2018-02-26 14:56:48.000000000 +0000
+++ /usr/lib/mailman/Mailman/Cgi/listinfo.py 2018-02-26 14:08:34.000000000 +0000
@@ -31,6 +31,7 @@
 from Mailman import i18n
 from Mailman.htmlformat import *
 from Mailman.Logging.Syslog import syslog
+from recaptcha.client import captcha

 # Set up i18n
 _ = i18n._
@@ -244,6 +245,10 @@
 replacements['<mm-lang-form-start>'] = mlist.FormatFormStart('listinfo')
 replacements['<mm-fullname-box>'] = mlist.FormatBox('fullname', size=30)

+ # Captcha
+ replacements['<mm-recaptcha-javascript>'] = captcha.displayhtml(mm_cfg.RECAPTCHA_PUBLIC_KEY, use_ssl=True, version=2)
+ replacements['<mm-recaptcha-script>'] = captcha.load_script(version=2)
+
 # Do the expansion.
 doc.AddItem(mlist.ParseTags('listinfo.html', replacements, lang))
 print doc.Format()

/usr/lib/mailman/Mailman/Cgi/subscribe.py

--- subscribe.py 2018-02-26 14:56:38.000000000 +0000
+++ /usr/lib/mailman/Mailman/Cgi/subscribe.py 2018-02-26 14:08:18.000000000 +0000
@@ -32,6 +32,7 @@
 from Mailman.UserDesc import UserDesc
 from Mailman.htmlformat import *
 from Mailman.Logging.Syslog import syslog
+from recaptcha.client import captcha

 SLASH = '/'
 ERRORSEP = '\n\n<p>'
@@ -165,6 +166,17 @@
 results.append(
 _('There was no hidden token in your submission or it was corrupted.'))
 results.append(_('You must GET the form before submitting it.'))
+
+ # recaptcha
+ captcha_response = captcha.v2submit(
+ cgidata.getvalue('g-recaptcha-response', ""),
+ mm_cfg.RECAPTCHA_PRIVATE_KEY,
+ remote,
+ )
+
+ if not captcha_response.is_valid:
+ results.append(_('Invalid captcha: %s' % captcha_response.error_code))
+
 # Was an attempt made to subscribe the list to itself?
 if email == mlist.GetListEmail():
 syslog('mischief', 'Attempt to self subscribe %s: %s', email, remote)

/usr/lib/mailman/templates/en/listinfo.html

--- listinfo.html 2018-02-26 15:02:34.000000000 +0000
+++ /usr/lib/mailman/templates/en/listinfo.html 2018-02-26 14:18:52.000000000 +0000
@@ -3,7 +3,7 @@
 <HTML>
 <HEAD>
 <TITLE><MM-List-Name> Info Page</TITLE>
-
+ <MM-Recaptcha-Script>
 </HEAD>
 <BODY BGCOLOR="#ffffff">

@@ -116,6 +116,11 @@
 </tr>
 <mm-digest-question-end>
 <tr>
+ <tr>
+ <td>Please fill out the following captcha</td>
+ <td><mm-recaptcha-javascript></TD>
+ </tr>
+ <tr>
 <td colspan="3">
 <center><MM-Subscribe-Button></center>
 </td>

The updated RPMs are being rolled out to Fedora, EPEL 6 and EPEL 7. In the meantime you can find them here.

A childhood’s dream

Andrea Veri — Tue, 05 Jul 2016 21:26:39 +0000

Six months since my latest blog post is definitely a lot and reminds me how difficult this year has been for me in many ways. Back in June 2015 I received a job proposal as a Systems and Network Engineer from a company located in Padova, a city in the north-east part of Italy which is around 150km (around 93 miles) away from my home-town. The offer looked very interesting and I went for it. The idea I originally had was to relocate there but the extremely high costs of rents (the city is well known in Italy as one of the best places to take University courses on several faculties) and the fact I really wanted to experiment on whether the job description was actually going to match the expectations I had, I opted to not relocate at all and travel daily to work by train (a total of 4 hours per day spent travelling).

I still recall one of my friends saying me “I am sure you won’t make it for more than two weeks” and how satisfying has been mentioning him a few days ago I did make it for roughly one year. Spending around 4 hours every day on a train hasn’t been that fun but the passion and love for what I’ve been doing has helped me overcoming the difficulties I encountered. At some point something I honestly was expecting arrived: my professional grow at the company was completely stuck. While I initially told myself the moment was surely going to not persist much longer, the opposite happened.

For a passionate, enthusiast and “hungry” person like I am that was one of the worst situations that could ever happened and it was definitely time for me to look at new opportunities. That’s where my childhood’s dream became reality: I will be joining the Platform Operations Team at Red Hat as a System Administrator starting from mid-July! Being part of a great family which cares about Open Source and its values makes me proud and I would really like to thank Red Hat for this incredible opportunity.

On a side note I will be in Brno between the 14th and the 16th of July, please drop me a note if you want to have a drink together!

Three years and counting

Andrea Veri — Wed, 02 Dec 2015 16:32:41 +0000

It’s been a while since my last “what’s been happening behind the scenes” e-mail so I’m here to report on what has been happening within the GNOME Infrastructure, its future plans and my personal sensations about a challenge that started around three (3) years ago when Sriram Ramkrishna and Jeff Schroeder proposed my name as a possible candidate for coordinating the team that runs the systems behind the GNOME Project. All this followed by the official hiring achieved by Karen Sandler back in February 2013.

The GNOME Infrastructure has finally reached stability both in terms of reliability and uptime, we didn’t have any service disruption this and the past year and services have been running smoothly as they were expected to in a project like the one we are managing.

As many of you know service disruptions and a total lack of maintenance were very common before I joined back in 2013, I’m so glad the situation has dramatically changed and developers, users, passionates are now able to reach our websites, code repositories, build machines without experiencing slowness, downtimes or

unreachability. Additionally all these groups of people now have a reference point they can contact in case they need help when coping with the infrastructure they daily use. The ticketing system allows users to get in touch with the members of the Sysadmin Team and receive support right away within a very short period of time (Also thanks to Pagerduty, service the Foundation is kindly sponsoring)

Before moving ahead to the future plans I’d like to provide you a summary of what has been done during these roughly three years so you can get an idea of why I define the changes that happened to the infrastructure a complete revamp:

Recycled several ancient machines migrating services off of them while consolidating them by placing all their configuration on our central configuration management platform ran by Puppet. This includes a grand total of 7 machines that were replaced by new hardware and extended warranties the Foundation kindly sponsored.
We strenghten our websites security by introducing SSL certificates everywhere and recently replacing them with SHA2 certificates.
We introduced several services such as Owncloud, the Commits Bot, the Pastebin, the Etherpad, Jabber, the GNOME Github mirror.
We restructured the way we backup our machines also thanks to the Fedora Project sponsoring the disk space on their backup facility. The way we were used to handle backups drastically changed from early years where a magnetic tape facility was in charge of all the burden of archiving our data to today where a NetApp is used together with rdiff-backup.
We upgraded Bugzilla to the latest release, a huge thanks goes to Krzesimir Nowak who kindly helped us building the migration tools.
We introduced the GNOME Apprentice program open-sourcing our internal Puppet repository and cleansing it (shallow clones FTW!) from any sensitive information which now lives on a different repository with restricted access.
We retired Mango and our OpenLDAP instance in favor of FreeIPA which allows users to modify their account information on their own without waiting for the Accounts Team to process the change.
We documented how our internal tools are customized to play together making it easy for future Sysadmin Team members to learn how the infrastructure works and supersede existing members in case they aren’t able to keep up their position anymore.
We started providing hosting to the GIMP and GTK projects which now completely rely on the GNOME Infrastructure. (DNS, email, websites and other services infrastructure hosting)
We started providing hosting not only to the GIMP and GTK projects but to localized communities as well such as GNOME Hispano and GNOME Greece
We configured proper monitoring for all the hosted services thanks to Nagios and Check-MK
We migrated the IRC network to a newer ircd with proper IRC services (Nickserv, Chanserv) in place.
We made sure each machine had a configured management (mgmt) and KVM interface for direct remote access to the bare metal machine itself, its hardware status and all the operations related to it. (hard reset, reboot, shutdown etc.)
We upgraded MoinMoin to the latest release and made a substantial cleanup of old accounts, pages marked as spam and trashed pages.
We deployed DNSSEC for several domains we manage including gnome.org, guadec.es, gnomehispano.es, guadec.org, gtk.org and gimp.org
We introduced an account de-activation policy which comes into play when a contributor not committing to any of the hosted repositories at git.gnome.org since two years is caught by the script. The account in question is marked as inactive and the gnomecvs (from the old cvs days) and ftpadmin groups are removed.
We planned mass reboots of all the machines roughly every month for properly applying security and kernel updates.
We introduced Mirrorbrain (MB), the mirroring service serving GNOME and related modules tarballs and software all over the world. Before introducing MB GNOME had several mirrors located in all the main continents and at the same time a very low amount of users making good use of them. Many organizations and companies behind these mirrors decided to not host GNOME sources anymore as the statistics of usage were very poor and preferred providing the same service to projects that really had a demand for these resources. MB solved all this allowing a proper redirect to the closest mirror (through mod_geoip) and making sure the sources checksum matched across all the mirrors and against the original tarball uploaded by a GNOME maintainer and hosted at master.gnome.org.

I can keep the list going for dozens of other accomplished tasks but I’m sure many of you are now more interested in what the future plans actually are in terms of where the GNOME Infrastructure should be in the next couple of years.

One of the main topics we’ve been discussing will be migrating our Git infrastructure away from cgit (which is mainly serving as a code browsing tool) to a more complete platform that is surely going to include a code review tool of some sort. (Gerrit, Gitlab, Phabricator)

Another topic would be migrating our mailing lists to Mailman 3 / Hyperkitty. This also means we definitely need a staging infrastructure in place for testing these kind of transitions ideally bound to a separate Puppet / Ansible repository or branch. Having a different repository for testing purposes will also mean helping apprentices to test their changes directly on a live system and not on their personal computer which might be running a different OS / set of tools than the ones we run on the GNOME Infrastructure.

What I also aim would be seeing GNOME Accounts being the only authentication resource in use within the whole GNOME Infrastructure. That means one should be able to login to a specific service with the same username / password in use on the other hosted services. That’s been on my todo list for a while already and it’s probably time to push it forward together with Patrick Uiterwijk, responsible of Ipsilon‘s development at Red Hat and GNOME Sysadmin.

While these are the top priority items we are soon receiving new hardware (plus extended warranty renewals for two out of the three machines that had their warranty renewed a while back) and migrating some of the VMs off from the current set of machines to the new boxes is definitely another task I’d be willing to look at in the next couple of months (one machine (ns-master.gnome.org) is being decommissioned giving me a chance to migrate away from BIND into NSD).

The GNOME Infrastructure is evolving and it’s crucial to have someone maintaining it. On this side I’m bringing to your attention the fact the assigned Sysadmin funds are running out as reported on the Board minutes from the 27th of October. On this side Jeff Fortin started looking for possible sponsors and came up with the idea of making a brochure with a set of accomplished tasks that couldn’t have been possible without the Sysadmin fundraising campaign launched by Stormy Peters back in June 2010 being a success. The Board is well aware of the importance of having someone looking at the infrastructure that runs the GNOME Project and is making sure the brochure will be properly reviewed and published.

And now some stats taken from the Puppet Git Repository:

$ cd /git/GNOME/puppet && git shortlog -ns

3520 Andrea Veri
506 Olav Vitters
338 Owen W. Taylor
239 Patrick Uiterwijk
112 Jeff Schroeder
71 Christer Edwards
4 Daniel Mustieles
4 Matanya Moses
3 Tobias Mueller
2 John Carr
2 Ray Wang
1 Daniel Mustieles García
1 Peter Baumgarten

and from the Request Tracker database (52388 being my assigned ID):

mysql> select count(*) from Tickets where LastUpdatedBy = '52388';
+----------+
| count(*) |
+----------+
| 3613 |
+----------+
1 row in set (0.01 sec)

mysql> select count(*) from Tickets where LastUpdatedBy = '52388' and Status = 'Resolved';
+----------+
| count(*) |
+----------+
| 1596 |
+----------+
1 row in set (0.03 sec)

It’s been a long run which made me proud, for the things I learnt, for the tasks I’ve been able to accomplish, for the great support the GNOME community gave me all the time and most of all for the same fact of being part of the team responsible of the systems hosting the GNOME Project. Thank you GNOME community for your continued and never ending backing, we daily work to improve how the services we host are delivered to you and the support we receive back is fundamental for our passion and enthusiasm to remain high!

The GNOME Infrastructure Apprentice Program

Andrea Veri — Wed, 28 Jan 2015 16:59:46 +0000

Many times it happened seeing someone joining the #sysadmin IRC channel requesting participation to the team after having spent around 5 minutes trying to explain what the skills and the knowledge were and why this person felt it was the right figure for the position. And it was always very disappointing for me having to reject all these requests as we just didn’t have the infrastructure in place to let new people join the rest of the team with limited privileges.

With the introduction of FreeIPA, more fine-grained ACLs (and hiera-eyaml-gpg for securing tokens, secrets, passwords out of Puppet itself) we are so glad to announce the launch of the “GNOME Infrastructure Apprentice Program” (from now till the end of the post just “Program”). If you are familiar with the Fedora Infrastructure and how it works you might know what this is about already. If you don’t please read further ahead.

The Program will allow apprentices to join the Sysadmin Team with a limited set of privileges which mainly consist in being able to access the Puppet repository and all the stored configuration files that run the machines powering the GNOME Infrastructure every day. Once approved to the Program apprentices will be able to submit patches for review to the team and finally see their work merged on the production environment if the proposed changes matched the expectations and addressed comments.

While the Program is open to everyone to join, we have some prerequisites in place. The interested person should be:

Part of an existing FOSS community
Familiar with how a FOSS Project works behind the scenes
Familiar with popular tools like Puppet, Git
Familiar with RHEL as the OS of choice
Familiar with popular Sysadmin tools, softwares and procedures
Eager to learn new things, make constructive discussions with a team, provide feedback and new ideas

If you feel like having all the needed prerequisites and would be willing to join follow these steps:

Subscribe to the gnome-infrastructure and infrastructure-announce mailing lists
Join the #sysadmin IRC channel on irc.gnome.org
Send a presentation e-mail to the gnome-infrastructure mailing list stating who you are, what your past experiences and plans are as an Apprentice
Once the presentation has been sent an existing Sysadmin Team member will evaluate your application and follow-up with you introducing you to the Program

More information about the Program is available here.

Kerberos over HTTP: getting a TGT on a firewalled network

Andrea Veri — Fri, 24 Oct 2014 13:50:52 +0000

One of the benefits I originally wanted to bring with the FreeIPA move to GNOME contributors was the introduction of an additional authentication system to connect to to the services hosted on the GNOME Infrastructure. The authentication system that comes with the FreeIPA bundle that I had in mind was Kerberos. Users willing to use Kerberos as their preferred authentication system would just be required to get a TGT (Ticket-Granting Ticket) from the KDC (Key Distribution Center) through the kinit command. Once done authenticating to the services currently supporting Kerberos will be as easy as pointing a configured browser (Google for how to configure your browser to use Krb logins) to account.gnome.org without being prompted for entering the usual username / password combination or pushing to git without using the public-private key mechanism. That theoretically means you won’t be required to use a SSH key for loggin in to any of the GNOME services at all as entering your password to the kinit password prompt will be enough (for at least 24 hours as that’s the life of the TGT itself on our setup) for doing all you were used to do before the Kerberos support introduction.

The issue we faced at first was the underlying networking infrastructure firewalling all Kerberos ports blocking the use of kinit itself which kept timing out reaching port 88. A few days later I was contacted by RH’s developer Nathaniel McCallum who worked out a way to bypass this restriction by creating a KDC proxy that accepts requests from port 443 and proxies them to the internal KDC running on port 88. With the recent Kerberos release (released on October 15th, 2014 and following the MS-KKDCP protocol) a patched kinit allows users to retrieve their TGTs directly from the HTTPS proxy completely bypassing the need for port 88 to stay open on the firewall. The GNOME Infrastructure now runs the KDC Proxy and we’re glad to announce Kerberos authentications are working as expected on the hosted services.

If you are facing the same problem and you are curious to know more about the setup, here they come all the details:

On the KDC:

No changes are needed on the KDC itself, just make sure to install the python-kdcproxy package which is available for RHEL 7, HERE.
Tweak your vhost accordingly by following the provided documentation.

On the client:

Install the krb5-workstation package, make sure it’s at least version 1.12.2-9 as that’s the release which had the additional features we are talking about backported. Right now it’s only available for Fedora 21.
Adjust /etc/krb5.conf accordingly and finally get a TGT through kinit $userid@GNOME.ORG.

[realms]
 GNOME.ORG = {
 kdc = https://account.gnome.org/KdcProxy
 kpasswd_server = https://account.gnome.org/KdcProxy
}

That should be all for today!

The GNOME Infrastructure’s FreeIPA move behind the scenes

Andrea Veri — Sun, 12 Oct 2014 18:02:02 +0000

A few days ago I wrote about the GNOME Infrastructure moving to FreeIPA, the post was mainly an announcement to the relevant involved parties with many informative details for contributors to properly migrate their account details off from the old authentication system to the new one. Today’s post is a follow-up to that announcement but it’s going to take into account the reasons about our choice to migrate to FreeIPA, what we found interesting and compelling about the software and why we think more projects (them being either smaller or bigger) should migrate to it. Additionally I’ll provide some details about how I performed the migration from our previous OpenLDAP setup with a step-by-step guide that will hopefully help more people to migrate the infrastructure they manage themselves.

The GNOME case

It’s very clear to everyone an infrastructure should reflect the needs of its user base, in the case of GNOME a multitude between developers, translators, documenters and between them a very good number of Foundation members, contributors that have proven their non-trivial contributions and have received the status of members of the GNOME Foundation with all the relevant benefits connected to it.

The situation we had before was very tricky, LDAP accounts were managed through our LDAP istance while Foundation members were being stored on a MySQL database with many of the tables being related to the yearly Board of Director’s elections and one specifically meant to store all the information from each of the members. One of the available fields on that table was defined as ‘userid’ and was supposed to store the LDAP ‘uid’ field the Membership Committee member processing a certain application had to update when accepting the application. This procedure had two issues:

Membership Committee members had no access to LDAP information
No checks were being run on the web UI to verify the ‘userid’ field was populated correctly taking in multiple inconsistencies between LDAP and the MySQL database

In addition to the above Mango (the software that helped the GNOME administrative teams to manage the user data for multiple years had no maintainer, no commits on its core since 2008 and several limitations)

What were we looking for as a replacement for the current setup?

It was very obvious to me we would have had to look around for possible replacements to Mango. What we were aiming for was a software with the following characteristics:

It had to come with a pre-built web UI providing a wide view on several LDAP fields
The web UI had to be extensible in some form as we had some custom LDAP schemas we wanted users to see and modify
The sofware had to be actively developed and responsive to eventual security reports (given the high security impact a breach on LDAP could take in)

FreeIPA clearly matched all our expectations on all the above points.

The Migration process – RFC2307 vs RFC2307bis

Our previous OpenLDAP setup was following RFC 2307, which means that above all the other available LDAP attributes (listed on the RFC under point 2.2) group’s membership was being represented through the ‘memberUid’ attribute. An example:

cn=foundation,cn=groups,cn=compat,dc=gnome,dc=org
objectClass: posixGroup
objectClass: top
gidNumber: 524
memberUid: foo
memberUid: bar
memberUid: foobar

As you can each of the members of the group ‘foundation’ are represented using the ‘memberUid’ attribute followed by the ‘uid’ of the user itself. FreeIPA does not make directly use of RFC2307 for its trees, but RFC2307bis instead. (RFC2307bis was not published as a RFC by the IETF as the author didn’t decide to pursue it nor the companies (HP, Sun) that then adopted it)

RFC2307bis uses a different attribute to represent group’s membership, it being ‘member’. Another example:

cn=foundation,cn=groups,cn=accounts,dc=gnome,dc=org
objectClass: posixGroup
objectClass: top
gidNumber: 524
member: uid=foo,cn=users,cn=accounts,dc=gnome,dc=org
member: uid=bar,cn=users,cn=accounts,dc=gnome,dc=org
member: uid=foobar,cn=users,cn=accounts,dc=gnome,dc=org

As you can see the DN representing the group ‘foundation’ differs between the two examples. That is why FreeIPA comes with a Compatibility plugin (cn=compat) which automatically creates RFC2307-compliant trees and entries whenever an append / modify / delete operation happens on any of the hosted RFC2307bis-compliant trees. What’s the point of doing this when we could just stick with RFC2307bis trees and go with it? As the plugin name points out the Compatibility plugin is there to prevent breakages between the directory server and any of the clients or softwares out there still retrieving information and data by using the ‘memberUid’ attribute as specified on RFC2307.

FreeIPA migration tools (ipa migrate-ds) do come with a ‘–schema’ flag you can use to specify what attribute the istance you are migrating from was following (values are RFC2307 and RFC2307bis as you may have guessed already), in the case of GNOME the complete command we ran (after installing all the relevant tools through ‘ipa-server-install’ and copying the custom schemas under /etc/dirsrv/slapd-$ISTANCE-NAME/schema) was:

ipa migrate-ds --bind-dn=cn=Manager,dc=gnome,dc=org --user-container=ou=people,dc=gnome,dc=org --group-container=ou=groups,dc=gnome,dc=org --group-objectclass=posixGroup ldap://internal-IP:389 --schema=RFC2307

Please note that before running the command you should make sure custom schemas you had on the istance you are migrating from are available to the directory server you are migrating your tree to.

More information on the migration process from an existing OpenLDAP istance can be found HERE.

The Migration process – Extending the directory server with custom schemas

One of the other challenges we had to face has been extending the available LDAP schemas to include Foundation membership attributes. This operation requires the following changes:

Build the LDIF (that will include two new custom fields: FirstAdded and LastRenewedOn)
Adding the LDIF in place on /etc/dirsrv/slapd-$ISTANCE-NAME/schema
Extend the web UI to include the new attributes

I won’t be explaining how to build a LDIF on this post but I’m however pasting the schema I made to help you getting an idea:

attributeTypes: ( 1.3.6.1.4.1.3319.8.2 NAME 'LastRenewedOn' SINGLE-VALUE EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch DESC 'Last renewed on date' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )
attributeTypes: ( 1.3.6.1.4.1.3319.8.3 NAME 'FirstAdded' SINGLE-VALUE EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch DESC 'First added date' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )
objectClasses: ( 1.3.6.1.4.1.3319.8.1 NAME 'FoundationFields' AUXILIARY MAY ( LastRenewedOn $ FirstAdded ) )

After copying the schema in place and restarting the directory server, extend the web UI:

on /usr/lib/python2.7/site-packages/ipalib/plugins/foundation.py:

from ipalib.plugins import user
from ipalib.parameters import Str
from ipalib import _
from time import strftime
import re

def validate_date(ugettext, value):
if not re.match("^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])$", value):
return _("The entered date is wrong, please make sure it matches the YYYY-MM-DD syntax")

user.user.takes_params = user.user.takes_params + (
Str('firstadded?', validate_date,
cli_name='firstadded',
label=_('First Added date'),
),
Str('lastrenewedon?', validate_date,
cli_name='lastrenewedon',
label=_('Last Renewed on date'),
),
)

on /usr/share/ipa/ui/js/plugins/foundation/foundation.js:

define([
'freeipa/phases',
'freeipa/user'],
function(phases, user_mod) {

// helper function
function get_item(array, attr, value) {
for (var i=0,l=array.length; i&lt;l; i++) {
if (array[i][attr] === value) return array[i];
}

return null;
}

var foundation_plugin = {};

foundation_plugin.add_foundation_fields = function() {

var facet = get_item(user_mod.entity_spec.facets, '$type', 'details');
var section = get_item(facet.sections, 'name', 'identity');
section.fields.push({
name: 'firstadded',
label: 'Foundation Member since'
});

section.fields.push({
name: 'lastrenewedon',
label: 'Last Renewed on date'
});

section.fields.push({
name: 'description',
label: 'Previous account changes'
});

return true;

};

phases.on('customization', foundation_plugin.add_foundation_fields);
return foundation_plugin;
});

Once done, restart the web server. The next step would be migrating all the FirstAdded and LastRenewedOn attributes off from MySQL into LDAP now that our custom schema has been injected.

The relevant MySQL fields were following the YYYY-MM-DD syntax to store the dates and a little Python script to read from MySQL and populate the LDAP attributes was then made. If interested or you are in a similar situation you can find it HERE.

The Migration process – Own SSL certificates for HTTPD

As you may be aware of FreeIPA comes with its own certificate tools (powered by Certmonger), that means a CA is created (during the ipa-server-install run) and certificates for the various services you provide are then created and signed with it. This is definitely great and removes the burden to maintain an underlying self-hosted PKI infrastructure. At the same time this seems to be a problem for publicly-facing web services as browsers will start complaining they don’t trust the CA that signed the certificate the website you are trying to reach is using.

The problem is not really a problem as you can specify what certificate HTTPD should be using for displaying FreeIPA’s web UI. The procedure is simple and involves the NSS database at /etc/httpd/alias:

certutil -d /etc/httpd/alias/ -A -n "StartSSL CA" -t CT,C, -a -i sub.class2.server.ca.pem
certutil -d /etc/pki/nssdb -A -n "StartSSL CA" -t CT,C, -a -i sub.class2.server.ca.pem
openssl pkcs12 -inkey freeipa.example.org.key -in freeipa.example.org.crt -export -out freeipa.example.org.p12 -nodes -name 'HTTPD-Server-Certificate'
pk12util -i freeipa.example.org.p12 -d /etc/httpd/alias/

Once done, update /etc/httpd/conf.d/nss.conf with the correct NSSNickname value. (which should match the one you entered after ‘-name’ on the third of the above commands)

The Migration process – Equivalent of authorized_keys’ “command”

At GNOME we do run several services that require users to login to specific machines and run a command. At the same time and for security purposes we don’t want all the users to reach a shell. Originally we were making use of SSH’s authorized_keys file to specify the “command” these users should have been restricted to. FreeIPA handles Public Key authentications differently (through the sss_ssh_authorizedkeys binary) which means we had to find an alternative way to restrict groups to only a specific command. SSH’s ForceCommand came in help, an example given a group called ‘foo’:

Match Group foo,!bar
X11Forwarding no
PermitTunnel no
ForceCommand /home/admin/bin/reset-my-password.py

The above Match Group will be applied to all the users of the ‘foo’ group except the ones that are also part of the ‘bar’ group. If you are interested in the reset-my-password.py script which resets a certain user password and sends a temporary one to the registered email address (by checking the mail LDAP attr) for the user, click HERE.

The Migration process – Kerberos host Keytabs

Here, at GNOME, we still have one or two RHEL 5 hosts hanging around and SSSD reported a failure when trying to authenticate with the given Keytab (generated with RHEL 7 default values) to the KDC running (as you may have guessed) RHEL 7. The issue is simple as RHEL 5 does not support many of the encryption types which the Keytab was being encrypted with. Apparently the only currently supported Keytab encryption type on a RHEL 5 machine is rc4-hmac. Creating a Keytab on the KDC accordingly can be done this way:

ipa-getkeytab -s server.example.org -p host/client.example.org -e rc4-hmac -k /root/keytabs/client.example.org.keytab

That should be all for today, I’ll make sure to update this post with further details or answers to possible comments.

The GNOME Infrastructure is now powered by FreeIPA!

Andrea Veri — Tue, 07 Oct 2014 09:21:33 +0000

As preannounced here the GNOME Infrastructure switched to a new Account Management System which is reachable at https://account.gnome.org. All the details will follow.

Introduction

It’s been a while since someone actually touched the underlying authentication infrastructure that powers the GNOME machines. The very first setup was originally configured by Jonathan Blandford (jrb) who configured an OpenLDAP istance with several customized schemas. (pServer fields in the old CVS days, pubAuthorizedKeys and GNOME modules related fields in recent times)

While OpenLDAP-server was living on the GNOME machine called clipboard (aka ldap.gnome.org) the clients were configured to synchronize users, groups, passwords through the nslcd daemon. After several years Jeff Schroeder joined the Sysadmin Team and during one cold evening (date is Tue, February 1st 2011) spent some time configuring SSSD to replace the nslcd daemon which was missing one of the most important SSSD features: caching. What surely convinced Jeff to adopt SSSD (a very new but promising sofware at that time as the first release happened right before 2010’s Christmas) and as the commit log also states (“New sssd module for ldap information caching”) was SSSD’s caching feature.

It was enough for a certain user to log in once and the ‘/var/lib/sss/db’ directory was populated with its login information preventing the LDAP daemon in charge of picking up login details (from the LDAP server) to query the LDAP server itself every single time a request was made against it. This feature has definitely helped in many occasions especially when the LDAP server was down for a particular reason and sysadmins needed to access a specific machine or service: without SSSD this wasn’t ever going to work and sysadmins were probably going to be locked out from the machines they were used to manage. (except if you still had ‘/etc/passwd’, ‘/etc/group’ and ‘/etc/shadow’ entries as fallback)

Things were working just fine except for a few downsides that appeared later on:

the web interface (view) on our LDAP user database was managed by Mango, an outdated tool which many wanted to rewrite in Django that slowly became a huge dinosaur nobody ever wanted to look into again
the Foundation membership information were managed through a MySQL database, so two databases, two sets of users unrelated to each other
users were not able to modify their own account information on their own but even a single e-mail change required them to mail the GNOME Accounts Team which was then going to authenticate their request and finally update the account.

Today’s infrastructure changes are here to finally say the issues outlined at (1, 2, 3) are now fixed.

What has changed?

The GNOME Infrastructure is now powered by Red Hat’s FreeIPA which bundles several FOSS softwares into one big “bundle” all surrounded by an easy and intuitive web UI that will help users update their account information on their own without the need of the Accounts Team or any other administrative entity. Users will also find two custom fields on their “Overview” page, these being “Foundation Member since” and “Last Renewed on date”. As you may have understood already we finally managed to migrate the Foundation membership database into LDAP itself to store the information we want once and for all. As a side note it might be possible that some users that were Foundation members in the past won’t find any detail stored on the Foundation fields outlined above. That is actually expected as we were able to migrate all the current and old Foundation members that had an LDAP account registered at the time of the migration. If that’s your case and you still would like the information to be stored on the new setup please get in contact with the Membership Committee at stating so.

Let’s make a little distinction between users that previously had access to Mango (usually maintainers) and users that didn’t. If you were used to access Mango before you should be able to login on the new Account Management System by entering your GNOME username and the password you were used to use for loggin in into Mango. (after loggin in the very first time you will be prompted to update your password, please choose a strong password as this account will be unique across all the GNOME Infrastructure)

If you never had access to Mango, you lost your password or the first time you read the word Mango on this post you thought “why is he talking about a fruit now?” you should be able to reset it by using the following command:

ssh -l yourgnomeuserid account.gnome.org

The command will start an SSH connection between you and account.gnome.org, once authenticated (with the SSH key you previously had registered on our Infrastructure) you will trigger a command that will directly send your brand new password on the e-mail registered for your account. From my tests seems GMail sees the e-mail as a phishing attempt probably because the body contains the word “password” twice. That said if the e-mail won’t appear on your INBOX, please double-check your Spam folder.

Now that Mango is gone how can I request a new account?

With Mango we used to have a form that automatically e-mailed the maintainer of the selected GNOME module which was then going to approve / reject the request. From there and in the case of a positive vote from the maintainer the Accounts Team was going to create the account itself.

With the recent introduction of a commit robot directly on l10n.gnome.org the number of account requests reduced its numbers. In addition to that users will now be able to perform pretty much all the needed maintenance on their accounts themselves. That said and while we will probably work on building a form in the future we feel that requesting accounts can definitely be achieved directly by mailing the Accounts Team itself which will mail the maintainer of the respective module and create the account. As just said the number of account creations has become very low and the queue is currently clear. The documentation has been updated to reflect these changes at:

https://wiki.gnome.org/AccountsTeam

https://wiki.gnome.org/AccountsTeam/NewAccounts

I was used to have access to a specific service but I don’t anymore, what should I do?

The migration of all the user data and ACLs has been massive and I’ve been spending a lot of time reviewing the existing HBAC rules trying to spot possible errors or misconfigurations. If you happen to not being able to access a certain service as you were used to in the past, please get in contact with the Sysadmin Team. All the possible ways to contact us are available at https://wiki.gnome.org/Sysadmin/Contact.

What is missing still?

Now that the Foundation membership information has been moved to LDAP I’ll be looking at porting some of the existing membership scripts to it. What I managed to port already are welcome e-mails for new or existing members. (renewals)

Next step will be generating a membership page from LDAP (to populate http://www.gnome.org/foundation/membership) and all the your-membership-is-going-to-lapse e-mails that were being sent till today.

Other news – /home/users mount on master.gnome.org

You will notice that loggin in into master.gnome.org will result in your home directory being empty, don’t worry, you did not lose any of your files but master.gnome.org is now currently hosting your home directories itself. As you may have been aware of adding files to the public_html directory on master resulted in them appearing on your people.gnome.org/~userid space. That was unfortunately expected as both master and webapps2 (the machine serving people.gnome.org’s webspaces) were mounting the same GlusterFS share.

We wanted to prevent that behaviour to happen as we wanted to know who has access to what resource and where. From today master’s home directories will be there just as a temporary spot for your tarballs, just scp and use ftpadmin against them, that should be all you need from master. If you are interested in receiving or keeping using your people.gnome.org’s webspace please mail stating so.

Other news – a shiny and new error 500 page has been deployed

Thanks to Magdalen Berns (magpie) a new error 500 web page has been deployed on all the Apache istances we host. The page contains an iframe of status.gnome.org and will appear every single time the web server behind the service you are trying to reach will be unreachable for maintenance or other purposes. While I hope you won’t see the page that often you can still enjoy it at https://static.gnome.org/error-500/500.html. Make sure to whitelist status.gnome.org on your browser as it currently loads it without https. (as the service is currently hosted on OpenShift which provides us with a *.rhcloud.com wildcard certificate, which differs from the CN the browser would expect it to be)

Updates

UPDATE on status.gnome.org’s SSL certificate: the certificate has been provisioned and it should result in the 500’s page to be displayed correctly with no warnings from your browser.

UPDATE from Adam Young on Kerberos ports being closed on many DC’s firewalls:

The next version of upstream MIT Kerberos will have support for fetching a ticket via ports 443 and marshalling the request over HTTPS. We’ll need to run a proxy on the server side, but we should be able to make it work:

Read up here

http://adam.younglogic.com/2014/06/kerberos-firewalls

Back from GUADEC 2014

Andrea Veri — Tue, 05 Aug 2014 17:27:23 +0000

Coming back from GUADEC has never been easy, so much fun, so much great people to speak with and amazing talks to watch but this year has definitely been harder as I totally felt in love with the city that was hosting the event. Honestly speaking I’ve been amazed by how Strasbourg looks like: alsace houses and buildings are just delightful, the cathedral is stunning and people have been so welcoming during my whole stay. (cooks at the Canteen even prepared a few italian dishes and welcomed us in italian every time we were heading there…how cool is that?)

But let’s get back to our business now as I would probably never stop talking about Strasbourg and how great it was staying there! I did not have a personal talk this year but I presented the yearly Sysadmin Team report during the Foundation’s AGM. If you weren’t there all the slides are available here.

Apart from presenting what we did and what the changes we introduced on the GNOME Infrastructure were I participated to Patrick Uiterwijk’s talk about FedOAuth and all the upcoming changes that are planned on the infrastructure during the next months. If you were not able to attend Patrick’s talk this little resume should be for you:

Current problems:

The GNOME Infrastructure currently has a lot of different user databases which implies different users and passwords across the services we host
The Foundation’s database is currently MySQL-based while we do have LDAP in place for all our other needs already
Some of the tools we do use for managing our LDAP istance are not being maintained properly

Possible solutions:

Introduce FedOAuth, a SSO solution written and developed by Patrick Uiterwijk
Unify the various databases and make sure our LDAP istance is used for authentication everywhere
Remove Mango and configure FreeIPA

Benefits after the move:

Users will be able to manage their accounts on their own, no more need to poke the accounts team for updating passwords, emails, SSH keys. The accounts team will still be around to adjust ACLs
No more need for dozen of accounts, one for every single service we provide
More freedom when managing sudo accesses and accounts on the various machines we manage, this will help new people contributing to the Sysadmin Team (Making our puppet repository public and introducing a GNOME Infrastructure Apprentice group for newcomers is something we will be seriously evaluating after the FreeIPA move)

Where we are now:

Our SSO infrastructure is live at https://id.gnome.org
Your OpenID URL is https://$GNOME_USERID.id.gnome.org
Right now you can login with your GNOME account at the following services: l10n.gnome.org, opw.gnome.org. We are slowly migrating all the existing services to the new SSO infrastructure, please be patient and bear with us!

More information, slides and screenshots from Patrick’s talk are available here. Stay tuned and many thanks to the GNOME Foundation for sponsoring my travel and accomodation expenses!

The GUADEC 2014 group photo!

A view of the Petite France district

Another great view of the Petite France district

The Notre Dame cathedral

Another view of the Notre Dame cathedral

Place Gutenberg

Adding reCAPTCHA support to Mailman

Andrea Veri — Sat, 03 May 2014 09:37:23 +0000

The GNOME and many other infrastructures have been recently attacked by an huge amount of subscription-based spam against their Mailman istances. What the attackers were doing was simply launching a GET call against a specific REST API URL passing all the parameters it needed for a subscription request (and confirmation) to be sent out. Understanding it becomes very easy when you look at the following example taken from our apache.log:

May 3 04:14:38 restaurant apache: 81.17.17.90, 127.0.0.1 - - [03/May/2014:04:14:38 +0000] "GET /mailman/subscribe/banshee-list?email=example@me.com&fullname=&pw=123456789&pw-conf=123456789&language=en&digest=0&email-button=Subscribe HTTP/1.1" 403 313 "http://spam/index2.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"

As you can the see attackers were sending all the relevant details needed for the subscription to go forward (and specifically the full name, the email, the digest option and the password for the target list). At first we tried to either stop the spam by banning the subnets where the requests were coming from, then when it was obvious that more subnets were being used and manual intervention was needed we tried banning their User-Agents. Again no luck, the spammers were smart enough to change it every now and then making it to match an existing browser User-Agent. (with a good percentage to have a lot of false-positives)

Now you might be wondering why such an attack caused a lot of issues and pain, well, the attackers made use of addresses found around the web for their malicius subscription requests. That means we received a lot of emails from people that have never heard about the GNOME mailing lists but received around 10k subscription requests that were seemingly being sent by themselves.

It was obvious we needed to look at a backup solution and luckily someone on our support channel suggested the freedesktop.org sysadmins recently added CAPTCHAs support to Mailman. I’m now sharing the patch and providing a few more details on how to properly set it up on either DEB or RPM based distributions. Credits for the patch should be given to Debian Developer Tollef Fog Heen, who has been so kind to share it with us.

Before patching your installation make sure to install the python-recaptcha package (tested on Debian with Mailman 2.1.15) on DEB based distributions and python-recaptcha-client on RPM based distributions. (I personally tested it against Mailman release 2.1.15, RHEL 6)

The Patch

diff --git a/Mailman/Cgi/listinfo.py b/Mailman/Cgi/listinfo.py
index 4a54517..d6417ca 100644
--- a/Mailman/Cgi/listinfo.py
+++ b/Mailman/Cgi/listinfo.py
@@ -30,6 +31,8 @@ from Mailman import Errors
 from Mailman import i18n
 from Mailman.htmlformat import *
 from Mailman.Logging.Syslog import syslog
+from recaptcha.client import captcha

 # Set up i18n
 _ = i18n._
@@ -200,6 +203,9 @@ def list_listinfo(mlist, lang):
 replacements[''] = mlist.FormatFormStart('listinfo')
 replacements[''] = mlist.FormatBox('fullname', size=30)

+ # Captcha
+ replacements['mm-recaptcha-javascript'] = captcha.displayhtml(mm_cfg.RECAPTCHA_PUBLIC_KEY, use_ssl=False)
+
 # Do the expansion.
 doc.AddItem(mlist.ParseTags('listinfo.html', replacements, lang))
 print doc.Format()
diff --git a/Mailman/Cgi/subscribe.py b/Mailman/Cgi/subscribe.py
index 7b0b0e4..c1c7b8c 100644
--- a/Mailman/Cgi/subscribe.py
+++ b/Mailman/Cgi/subscribe.py
@@ -21,6 +21,8 @@ import sys
 import os
 import cgi
 import signal
+from recaptcha.client import captcha

 from Mailman import mm_cfg
 from Mailman import Utils
@@ -132,6 +130,17 @@ def process_form(mlist, doc, cgidata, lang):
 remote = os.environ.get('REMOTE_HOST',
 os.environ.get('REMOTE_ADDR',
 'unidentified origin'))
+
+ # recaptcha
+ captcha_response = captcha.submit(
+ cgidata.getvalue('recaptcha_challenge_field', ""),
+ cgidata.getvalue('recaptcha_response_field', ""),
+ mm_cfg.RECAPTCHA_PRIVATE_KEY,
+ remote,
+ )
+ if not captcha_response.is_valid:
+ results.append(_('Invalid captcha'))
+
 # Was an attempt made to subscribe the list to itself?
 if email == mlist.GetListEmail():
 syslog('mischief', 'Attempt to self subscribe %s: %s', email, remote)

Additional setup

Then on the /var/lib/mailman/templates/en/listinfo.html template (right below ) add:

<tr>
 <td>
 Please fill out the following captcha
 </td>
 <td>
 <mm-recaptcha-javascript>
 </TD>
</tr>

Make also sure to generate a public and private key at https://www.google.com/recaptcha and add the following paramaters on your mm_cfg.py file:

RECAPTCHA_PRIVATE_KEY
RECAPTCHA_PUBLIC_KEY

Loading reCAPTCHAs images from a trusted HTTPS source can be done by changing the following line:

replacements['<mm-recaptcha-javascript>'] = captcha.displayhtml(mm_cfg.RECAPTCHA_PUBLIC_KEY, use_ssl=False)

replacements['<mm-recaptcha-javascript>'] = captcha.displayhtml(mm_cfg.RECAPTCHA_PUBLIC_KEY, use_ssl=True)

A few additional details should be provided in case you are setting this up against a RHEL 6 host: (or any other machine using the EPEL 6 package python-recaptcha-client-1.0.5-3.1.el6)

Importing the recaptcha.client module will fail for some strange reason, importing it correctly can be done this way:

ln -s /usr/lib/python2.6/site-packages/recaptcha/client /usr/lib/mailman/pythonlib/recaptcha

and then fix the imports also making sure sys.path.append(“/usr/share/pyshared”) is not there:

from recaptcha import captcha

That’s not all, the package still won’t work as expected given the API_SSL_SERVER, API_SERVER and VERIFY_SERVER variables on captcha.py are outdated (filed as bug #1093855), substitute them with the following ones:

API_SSL_SERVER="https://www.google.com/recaptcha/api"
API_SERVER="http://www.google.com/recaptcha/api"
VERIFY_SERVER="www.google.com"

And then on line 76:

url = "https://%s/recaptcha/api/verify" % VERIFY_SERVER,

reCAPTCHA v2

Google’s reCAPTCHA v1 will be deactivated starting from the 31th of March 2018, read more on how to migrate your Mailman install to version 2 here.

That should be all! Enjoy!

Fedy’s installation of Brackets bricks your Fedora installation

Andrea Veri — Wed, 02 Apr 2014 14:11:27 +0000

I wanted to give Fedy a try yesterday, specifically to install the **Brackets **code editor designed for web developers. I’m pretty lazy when it comes to install external packages (from the Brackets.io’s homepage it looked like only a DEB file was available) and after asking a few friends who made heavy use of Fedy in the past about its stability and credibility I went ahead and followed the provided instructions to set it up.

The interface was pretty straightforward and installing Brackets was as easy and clicking on the relevant button. Before starting the installation I gave a fast look around to the various bash scripts used by Fedy to install the package I wanted and yeah, I admit I did not pay enough attention to a few lines of the code and went ahead with the installation.

After hacking a bit with Brackets I decided it was time for me to head to bed but shutting down my laptop surprisingly returned various errors related to Systemd’s journal not being able to shutdown properly. I then tried to reboot the machine and found out the laptop was totally not bootable anymore.

The error it was reported at boot (Systemd Journal not being able to start properly) was pretty strange and after looking around the web I couldn’t find any other report about similar failures. I then started digging around with a friend and made the following guesses:

The root partition was running out of space (just 70M left), I then cleaned it a bit and rebooted with no luck. My first guess was /tmp going out of space when Systemd tries to populate it at boot time.
I checked yum history to find out what Fedy could have taken in, but nothing relevant was found given Fedy does not install RPM packages on its own, it usually retrieves a tarball (or in my case a DEB package) and installs it by extracting / cping the content
I turned SELinux to Permissive and rebooted the machine, surprise, the machine was bootable again

The next move was running a restorecon -r -v / against the root partition, the result was awful: the whole /usr‘s context was turned into usr_tmp_t. Digging around the code for the Brackets installer the following code was found:

mkdir -p "${file%.*}"
ar p "$file" "data.tar.gz" | tar -C "${file%.*}" -xzf -
cp -af ${file%.*}/* "/"

<br>

And previously:

get_file_quiet "http://download.brackets.io/" "brackets.htm"
get=$(cat "brackets.htm" | tr ' ' '\n' | grep -o "file.cfm?platform=LINUX${arch}&build=[0-9]*" | head -n 1 | sed -e 's/^/http:\/\/download.brackets.io\//')
file="brackets-LINUX.deb"

<br>

So what the installer was doing is:

Downloading the DEB file from the Brackets website
Extracting its content to /tmp/fedy and copying the contents of the data.tar.gz tarball in place

A tree view of the data.tar.gz file:

.
|-- opt
| `-- brackets
`-- usr
|-- bin
`-- share

<br>

Copying the extracted content of the data.tar.gz tarball to the target directories will do exactly one thing: it will overwrite the SELinux context of your /usr, /bin, /share directories breaking your system. I would advise everyone to NOT make use of Fedy for installing the Brackets editor until the issue has been fixed. Honestly speaking I didn’t have time / willingness to check other bash scripts but something nasty might be found there as well. Generally I would never recommend to install anything on your system without making use of an RPM package. Lesson learned for me to never trust such tools in the future on my local system.

The issue seems it was reported already one month ago, we added our report to the same issue. You can track it at https://github.com/satya164/fedy/issues/79.

Resources:

Faulty bash script: https://github.com/satya164/fedy/blob/master/plugins/soft/adobe_brackets.sh
Why usr_tmp_t gets added as /usr‘s context: https://github.com/satya164/fedy/blob/master/fedy#L26.

Fedora 20 on a Samsung Chronos Series 7

Andrea Veri — Thu, 20 Mar 2014 15:36:38 +0000

It’s been a while now since the very first time I posed my hands on this shiny new Samsung Chronos Series 7 laptop and oh dear… how much pain did my metallic-grey fellow take me in order to figure out how properly have every single piece of the hardware working as expected?

What I did right after unboxing it was dropping Windows 8 with a copy of Fedora 20 (yeah, stupid me, I could have booted Windows 8 at least once to check for UEFI / firmware updates) and setting everything up as usual. Right after booting the machine I disabled Windows 8’s Secure Boot, configured the laptop to boot from the USB Key I plugged in and restarted it to finally perform the real OS installation.

The laptop booted back (with UEFI mode marked as on) and the installation started. The Chronos Series 7 came with an iSSD of 16G in size, not much but definitely enough for keeping the root partition, swap and home directory. (I don’t need an huge home dir given all the various data is stored and mounted through NFS directly from the NAS)

The installation went just fine and no issues arised at all until I booted into the system. The laptop since the beginning started to reach high CPU and graphics card temperatures (sticking around 75-78 C for the CPU and 60 C for the NVIDIA card), the fans were constantly on and I could feel the heat of the machine while typing. But that’s not all, the backlit keyboard had its lights always set as on (even in the case of a locked screen) and the battery life was sticking around one hour and an half.

I’m now going to list all my findings and solutions for the above issues after spending some time debugging and trying hard.

UEFI vs CSM (Legacy BIOS Compatibility mode)

Installing Fedora 20 with UEFI will result in your laptop not loading the samsung-laptop kernel module at all. That is the result of a known bug (with a good chance to brick your laptop) on Samsung laptops and UEFI boots (mainly related to the incompatibility between the samsung-laptop kernel module and the Samsung’s UEFI firmware and its corruption when the module is loaded) . More details here and here.

That said, before starting the installation, press Fn+F2 right after powering up your laptop, you will be then prompted with the laptop’s Setup configuration. Switch to the Boot tab and disable Secure Boot making sure CSM is selected on the dropdown window. (before moving on make also sure to disable the Fast BIOS Mode on the Advanced tab before proceeding)

When done, boot up a copy of Fedora 20 with your preferred media (I did use the Live CD myself) and make your way through the installation. Make sure to read on before touching the disk partitioning schemas.

iSSD not recognized with CSM mode marked as on

During one of my trials I did try to install the OS directly on the iSSD (in our examples, /dev/sdb) itself. The result was the system being completely un-bootable probably cause the EFI firmware being unable to recognize the iSSD on CSM mode. (the only disk that was getting recognized was the 750G HDD (in our examples, /dev/sda) the laptop has as an additional storage)

While the EFI firmware is not able to recognize the iSSD at all properly (when in CSM mode) it can flawlessly detect the HDD. That means one thing: we can keep the root, home and swap partitions on the iSSD and move the boot, bios boot partitions on the HDD itself and boot up the machine from there.

The installation

We left our installation tutorial right before starting the installation itself through Anaconda. Let’s resume from there by making sure the following partitions are created on the HDD (from now on /dev/sda):

a bios-boot partition (details on how to set it up at here).
a boot partition (ext4, 500M in size)

Once done, perform the installation on the iSSD (from now on /dev/sdb), my setup on the 16G iSSD:

LVM Volume Group with three logical volumes:

5.12 G /home + Luks (5.12 G for an home directory seems not enough but that’s definitely more than enough when you have a local HDD and a NAS with more than 2T of storage)
7.8 G / (I prefer keeping root a big bigger than home to prevent the need to cleanup the yum cache and other packages cruft every now and then)
1.5 G swap space

Another working setup might be:

A 15G / partition on the iSSD, no need for LVM here
An LVM Volume Group that will store the /home and swap space so you can expand it to be more than just 5G / 1.5G. The LVM VolGroup should ideally go on its own partition on /dev/sda

When the system has been installed, mount the /dev/sda boot partition you previously created and install grub:

mount /dev/sda3 /mnt/boot-sda

grub2-install /dev/sda

I did assume /dev/sda3 was your /boot partition, make sure to check that is right for your case as well. (just run fdisk -l /dev/sda to find out)

When done, mount the /dev/sdb1 partition and copy all the files to the previously created mount point /mnt/boot-sda. From there figure out the UUID (ls -l /dev/disk/by-uuid) for the /dev/sda3 partition and modify the relevant entries on the /boot/grub2/grub.cfg file removing the UUID for the /dev/sdb1 (in my case that was the partition containing /boot) partition with the one of /dev/sda3. Save the file and reboot the machine.

You should then be able to unmount the /boot partition from /dev/sdb1 and modify the relevant /etc/fstab entry with /dev/sda3‘s UUID. That way new kernel’s installations will be handled correctly without the need to manually edit /boot/grub2/grub.cfg and moving around the initramfs / vmlinuz images. At this point it should be safe to remove the /dev/sdb1 partition completely.

Things to do after installing Fedora 20

Installing Bumblebee

The NP700Z3C has a NVIDIA GeForce GT 630M that benefits from the NVIDIA Optimus technology which allows the user to gather the maximum performance possible when launching specific high-demand applications (like during gameplay or while watching an HD movie) and fallback to the integrated GPU when the user is performing normal operations like browsing the web, writing emails or text editing.

Luckily the Bumblebee project comes in help about this providing Optimus support for a variety of Linux distributions. More details on how to set it up here. (you will need bumblebee, bumblebee-nvidia and bbswitch)

Installing TLP

From the tlp‘s website:

TLP brings you the benefits of advanced power management for Linux without the need to understand every technical detail. TLP comes with a default configuration already optimized for battery life, so you may just install and forget it. Nevertheless TLP is highly customizable to fulfil your specific requirements.

I can tell you power management for your laptop has never been easier with tlp. Make sure to visit tlp’s homepage for more details on how to set it up.

Samsung Tools

From the samsung-tool‘s homepage:

Samsung Tools is the successor of Samsung Scripts provided by the ‘Linux On My Samsung’ project.

It enables control in a friendly way of the devices available on Samsung laptops (bluetooth, wireless, webcam, backlight, CPU fan, special keys) and the control of various aspects related to power management, like the CPU undervolting (when a PHC-enabled kernel is available).

Given there’s no RPM available for samsung-tools, downloading the tarball and running make as root should suffice for installing it on your laptop.

Kernel flags

What seem to have helped me a lot with the high CPU temperatures (and thus with the noisy fans going on and on) are the following Kernel flags you should pass to Grub through the /etc/sysconfig/grub file on the GRUB_CMDLINE_LINUX line:

pcie_aspm=force i915.i915_enable_rc6=1 i915.i915_enable_fbc=1 i915.lvds_downclock=1 i915.semaphores=1 i915.modeset=1 acpi_osi=Linux rdblacklist=nouveau

Additional notes

The above procedure has been tested with the laptop having the following specs.

The average temperature for the CPU is sticking around 47-51 C, while the discrete GPU at 49-51C. I could also get around 3.5 – 4 hours of battery life!

That should be all! Please leave me a comment in case of questions, troubles with the above setup!

Manage passwords with ‘pass’

Andrea Veri — Tue, 24 Dec 2013 15:51:20 +0000

Fighting with passwords have always been one of my favorite battles in the past and unfortunately the former always won. I never liked using the root user that much for administering a machine and made a massive use of sudo, I won’t list all the benefits of using sudo, but the following wiki page has a pretty nice overview of them.

Said that, when using sudo it’s definitely ideal to combine a strong password that is also easy to remember and type again when prompted. Sadly strong passwords that are also easy to remember can be considered an oxymoron. How hard would it be to recall a 30+ chars long password? Honestly that would be close to impossible for an human being but what if a little software available on the major GNU/Linux distributions could handle that for us? That’s where pass comes handy, but what is pass? from the pass manpage itself:

pass is a very simple password store that keeps passwords inside gpg2(1) encrypted files inside a simple directory tree residing at ~/.password-store. The pass utility provides a series of commands for manipulating the password store, allowing the user to add, remove, edit, synchronize, generate, and manipulate passwords.

I’m sure that a lot of you guys have been looking for a tool like this one for ages: pass allows you to generate very strong passwords with pwgen, GPG encrypt them with your GPG Key, store them safely on your disk and make them available whenever you need them with a single command. But let’s move to the practice, give the following steps a try and enjoy how powerful your pass setup will be.

First setup

Install the software:

yum/apt-get install pass

Generate a GPG Key if you don’t have one already, a detailed guide can be found here. Initialize your passwords storage. (GPGKEYID can be retrieved by running gpg –list-keys and then looking for a line similar to this one: pub 4096R/B3A6223D 2012-06-25)

pass init GPGKEYID

Generate your first password and call it ‘sudo_password’ given you are going to make use of it as your brand new sudo password. (we want it at least 30+ chars long)

pass generate sudo_password 30

(Optional) Create as much passwords as you need and make sure to save them with unique names, that way you will be able to identify what a password is used for easily.

pass generate gmail_password 30

Additional maintenance commands on your password database

Look at the existing passwords on your database.

pass ls

Result:

Password Store
├── gmail_password
├── sudo_password
└── root_password

Manually edit a password.

pass edit password_name

Remove a password from your database.

pass rm password_name

Copy a password on your clipboard and paste it.

pass -c password_name

Are you wondering if pass supports a VCS? Yeah, it does, it currently allows you to manage your passwords database with Git, so that each applied change to the database will be tracked through a VCS so that you won’t forget when and how you updated a specific password.

Configuring DNSSEC on your personal domain

Andrea Veri — Wed, 13 Nov 2013 13:53:31 +0000

Today I’ll be working out how to properly configure DNSSEC on a BIND9 installation, I’ll also make sure to give you all the needed instructions to properly verify if a specific domain is being correctly covered by DNSSEC itself. In addition to that a few more details will be provided about adding the relevant SSHFP‘s entries on your DNS zone files to be able to automatically verify the authenticity of your domain when connecting to it with SSH avoiding any possible MITM attack.

First of all, let’s create the Zone Signing Key (ZSK) which is the key that will be responsible to sign any other record on the zone file which is not a DNSKEY record itself:

dnssec-keygen -a RSASHA1 -b 1024 -n ZONE gnome.org

Note: the dnssec-keygen binary file should be part of bind97 (RHEL 5) or bind (RHEL6) package according to yum whatprovides:

RHEL 5:

32:bind97-9.7.0-17.P2.el5_9.2.x86_64 : The Berkeley Internet Name Domain (BIND) DNS (Domain Name System) server
Repo : rhel-x86_64-server-5
Matched from:
Filename : /usr/sbin/dnssec-keygen

RHEL 6:

32:bind-9.8.2-0.17.rc1.el6.3.x86_64 : The Berkeley Internet Name Domain (BIND) DNS (Domain Name System) server
Repo : rhel-x86_64-server-6
Matched from:
Filename : /usr/sbin/dnssec-keygen

Then, create the Key Signing Key (KSK), which will be used to sign all the DNSKEY records:

dnssec-keygen -a RSASHA1 -b 2048 -n ZONE -f KSK gnome.org

Creating the above keys can take several minutes, when done copy the public keys to the zone file this way:

cat Kgnome.org*.key >> gnome.org

When done you can clean out the useless bits from the zone file and just leave the DNSKEY records (which are not commented out as you will notice)

An additional and cleaner way of accomplishing the above is to use the INCLUDE rule on the zone file itself as follows:

$INCLUDE /srv/dnssec-keys/Kgnome.org+005+12345.key
$INCLUDE /srv/dnssec-keys/Kgnome.org+005+67890.key

Choosing which method to use is really up to you.

Once that is done you can go ahead and sign the zone file. As of myself I’m making use of the do-domain script taken from the Fedora Infrastructure Team’s repositories. If you are going to use it yourself, make sure to adjust all the relevant variables to match your setup, especially keyspath, region_zones, template_zone, signed_zones and AREA. The do-domain script also checks your zone file through named-checkzone before signing it.

If instead you don’t want to use the script above, you can sign the zone file manually in the following way:

dnssec-signzone -K /path/to/your/dnssec/keys -e +3024000 -N INCREMENT gnome.org

By default, the above command will append ‘.signed’ to the file name, you can modify that behaviour by appending the ‘-f’ flag to the dnssec-signzone call. The ‘-N INCREMENT’ will increment the serial number automatically making use of the RFC 1982 arithmetics while the ‘-e’ flag will extend the zone signature end date from the default 30 days to 35. (this way we can safely run a monthly cron job that will sign the zone file automatically)

You can make use of the following script to achieve the above:

#!/bin/sh
SIGNZONE="/usr/sbin/dnssec-signzone"
DNSSEC_KEYS="/srv/dnssec-keys"
NAMEDCHROOT="/var/named/chroot"
ZONEFILES="gnome.org"

cd $NAMEDCHROOT

for ZONE in $ZONEFILES; do
$SIGNZONE -K $DNSSEC_KEYS -e +3024000 -f $ZONE.signed -N INCREMENT $ZONE
done

/sbin/service named reload

Once the zone file has been signed just make sure to include it on named.conf and restart named:

zone "gnome.org" {
file "gnome.org.signed";
};

When you’re done with that we should be moving ahead adding a DS record for our domain at our domain registrar. My example is taken from the known gandi.net registrar.

Select KSK (257) and (RSA/SHA-1) on the dropdown list and paste your public key on the box. You will find the public key you need on one of the Kgnome.org*.key files, you should look for the DNSKEY 257 entry as ‘dig DNSKEY gnome.org‘ shows:

;; ANSWER SECTION:
gnome.org. 888 IN DNSKEY 257 3 5 AwEAAbRD7AymDFuKc2iXta7HXZMleMkUMwjOZTsn4f75ZUp0of8TJdlU DtFtqifEBnFcGJU5r+ZVvkBKQ0qDTTjayL54Nz56XGGoIBj6XxbG8Es+ VbZCg0RsetDk5EsxLst0egrvOXga27jbsJ+7Me3D5Xp1bkBnQMrXEXQ9 C43QfO2KUWJVljo1Bii3fTfnHSLRUsbRn8Puz+orK71qxs3G9mgGR6rm n91brkpfmHKr3S9Rbxq8iDRWDPiCaWkI7qfASdFk4TLV0gSVlA3OxyW9 TCkPZStZ5r/WRW2jhUY/kjHERQd4qX5dHAuYrjJSV99P6FfCFXoJ3ty5 s3fl1RZaTo8=

Once that is done you should have a fully covered DNSSEC domain, you can verify that this way:

dig . DNSKEY | grep -Ev '^($|;)' > root.keys

dig +sigchase +trusted-key=./root.keys gnome.org. A | cat -n

The result:

105 ;; WE HAVE MATERIAL, WE NOW DO VALIDATION
106 ;; VERIFYING DS RRset for org. with DNSKEY:59085: success
107 ;; OK We found DNSKEY (or more) to validate the RRset
108 ;; Ok, find a Trusted Key in the DNSKEY RRset: 19036
109 ;; VERIFYING DNSKEY RRset for . with DNSKEY:19036: success
110
111 ;; Ok this DNSKEY is a Trusted Key, DNSSEC validation is ok: SUCCESS

Bonus content: Adding SSHFP entries for your domain and verifying them

You can retrieve the SSHFP entries for a specific host with the following command:

ssh-keygen -r $(hostname --fqdn) -f /etc/ssh/ssh_host_rsa_key.pub

When retrieved just add the SSHFP entry on the zone file for your domain and verify it:

ssh -oVerifyHostKeyDNS=yes -v subdomain.gnome.org

Or directly add the above parameter into your /etc/ssh/ssh_config file this way:

VerifyHostKeyDNS=yes

And run ‘ssh -v subdomain.gnome.org’, the result you should receive:

debug1: Server host key: RSA 00:39:fd:1a:a4:2c:6b:28:b8:2e:95:31:c2:90:72:03
debug1: found 1 secure fingerprints in DNS
debug1: matching host key fingerprint found in DNS
debug1: ssh_rsa_verify: signature correct

That’s it! Enjoy!

Back from GUADEC 2013

Andrea Veri — Mon, 12 Aug 2013 15:51:30 +0000

I wanna be really honest, getting back home from this year’s GUADEC has been very painful for me but not because of the trip back home. I had such a very good time at Brno that I actually wanted to stay there for way more days! I must admit that I’ve been missing the italian food for a while until Mattias Bengtsson suggested me to try having a dinner at the “Flavours” indian restaurant. The result was simply amazing and I’ve been falling in love with the indian food we ate that evening so much that we went there again the day after.

I had a lot of expectations from my very first GUADEC and I was very excited to meet all the people I’ve been contributing with during all these years. Meeting people up in person is fundamental and reminds you the fact that there is a human being with its own feelings and emotions behind a computer and that’s actually why I spent a lot of my time during the event speaking and hanging out with people, finding out their personal interests, their hobbies, the things they love doing on their free time.

During the event, I had the great pleasure to present two talks:

the GNOME Infrastructure, the presentation is available here and the video is viewable at the following page.
a little resume of what we’ve been doing on the GNOME Sysadmin team during this last year at the GNOME Foundation’s Annual General Meeting. (AGM)

… and meet a lot of friends:

Mattias Bengtsson, you’ve been the greatest room mate of all times! Thanks for all your hints!
Andreas Nilsson and Fabiana Simões, thanks for all your kind words, I’ll keep doing my best to provide you all the resources and work you need to make things happen!
Sriram Ramkrishna, you don’t know how much I enjoyed our nightly discussions and walks around the city centre, missing those times already!
Allan Day and Jon McCann, it’s been truly amazing meeting you guys and hearing how much enthusiast you are about me and my work, thanks a *lot*!
Paul Frields, it’s been an absolute pleasure meeting you and your wife, I will never forget how you presented me to her at the Starobrno Brewery: “He’s like the Kevin Fenzi of the GNOME Infrastructure!“. That’s one of the greatest compliments someone can ever receive given the amount of work Kevin does on the Fedora Infrastructure. I’m also glad that we could remember Seth together during my talk, thanks for coming!
My italian friends, Flavia, Paolo, Emanuele, Giovanni and Alessandro!
Tobias Mueller, glad we were finally able to meet in person! Thanks for your nano sessions!
All the indian interns, Saumya Dwivedi, Saumya Pathak, Sindhu Sundar, Shivani Poddar who are definitely doing an amazing job! I was completely impressed by your talks and projects!
Ekaterina, David, Karen, Marina, Owen, Rui, Alberto: all you guys are simply great! It’s really unfortunate we weren’t able to spend a lot of time together during the event!

Overall the whole event has been a blast, thanks to the GNOME Foundation making this possible by sponsoring my attendance at the event! I’m looking forward to Strasburg 2014 already and last but not least I’m preparing the greatest bid of all times for GUADEC 2015 to happen in Italy. Stay tuned!

Seth at GUADEC

GUADEC friends

Myself and Sri

Two years later: Vim, Tmux and my Linux desktop

Andrea Veri — Thu, 27 Jun 2013 16:59:57 +0000

It’s been two years since my latest blog post about my Linux desktop and many things have changed since then. I completely moved all my machines to GNOME 3, switched my main editor from nano to vim and my terminal multiplexer from screen to tmux. What didn’t change at all except for a tweaks on the theme is my Irssi setup.

Switching from nano to vim has been a pain at first, nano is really a straightforward editor, it does what you actually need from a CLI editor but while it works just fine when modifying configuration or text files, it’s a bit limiting when it comes to programming. Vim on the other hand is highly customizable in every single part also thanks to its huge amount of plugins. Honestly I admit I spent several hours watching videos, reading documentation, trying out key bindings and I’m not completely used to vim to be as much productive I would like to be with it.

What I found to be the common error of vim’s newcomers is their willingness to look around the web for a complete vimrc configuration file, full of key bindings, custom settings and personalizations. That’s definitely something you should avoid when learning to use vim, the perfect vimrc doesn’t objectively exist, each of us should spend some time investigating what is the best configuration for your needs and build a vimrc accordingly time by time.

It will probably take months to have a complex vimrc file matching your needs completely, until then you won’t be able to define your vimrc to be “ultimate”. And that’s actually why wgetting someone else’s vimrc and copying it to your home folder won’t make you an expert of vim, it’ll probably make your life harder when trying a specific action with the stock settings will result in something you wouldn’t have expected thanks to a particular key binding on the vimrc you downloaded.

The other tool that definitely improved my productivity is Tmux given the huge amount of open terminals I had every day during my sysadmin’s duties at GNOME. Each day usually started with one or two open terminals mainly meant for random maintenance issues, after a few hours the amount of open terminals jumped to around 30.

A second round of updates from the GNOME Sysadmin Team

Andrea Veri — Fri, 14 Jun 2013 11:37:06 +0000

I haven’t been blogging so much in the past months as I actually promised myself I would have but given the fact a lot has been done on the GNOME Infrastructure lately it’s time for me to announce all the updates we did since my latest blog post. So here we come with all the items we’ve been looking at recently:

Our main LDAP istance was moved from a very ancient machine (which unfortunately died with a broken disk a few weeks ago) to a newer box that currently contains several other admin tools like Mango and Daily Reports. (a little script written by Owen Taylor for creating and storing several reports mainly related to backups and SSL certificates expiration dates) In addition to migrating our LDAP master to a newer machine, we did configure and setup replication to an LDAP slave to share a bit the load and most of all to link all the external (machines outside the RH’s internal network) machines to it.
A lot of efforts have been spent in the so-called “Puppet-ization” (Puppet allows you to reproduce a complete environment with just a few commands, it’s very very handy in the case of host’s migrations) and several new modules are now stored into our internal Puppet repository. Specifically all the Iptables rules are currently managed in a centralized way, each node has its own rules and policies, finally there’s no need to ssh into each machine to retrieve the information we need for a specific firewall. In addition to the Iptables class, also Cobbler, Owncloud, Jabberd, Denyhosts and several other modules have been properly configured and currently reside in Puppet.
Another item that had top priority on the list was setting up another “webapps” virtual machine to migrate several services from one of the existing ancient machines to it. I can finally tell that the GNOME Infrastructure has get rid of all the old machines, all the services have been migrated to newer machines and most of all all the services are currently being served through SSL. (git, planet, www.gnome.org, l10n, guadec.org, bugzilla, blogs, developer, help, people, news etc.) In regard of SSL and Bugzilla, we’ve configured our Bugzilla istance to serve attachments through a secondary domain which will look like: https://bug-id.bugzilla-attachments.gnome.org, this to prevent cross-site scripting attacks in a better way than what we did before.
We’ve also spent some time working on our Nagios istance hosted at https://nagios.gnome.org. We’ve improved it dramatically by adding several new checks and covering all the services we currently take care of but that’s not all. Event Handlers have been setup to help us addressing problems right after they occur on our web servers. The Nagios event handlers are currently configured to read the status of a specific Nagios service and in the case the status is set to CRITICAL, they restart httpd once, which is usually enough in the case of random Apache’s timeouts. But that’s again not all. A public view for Nagios is now ready, every single GNOME contributor and developer should be able to check the current status of all the services we maintain just by loggin in with the username “anonymous” and the password “anonymous” at nagios.gnome.org.
Our wiki was upgraded to the latest MoinMoin release, at the moment of writing, version 1.9.7. This release introduces stronger password hashes, please make sure to update your password as soon as you can to strenghten the security of your account. It was also clear that live.gnome.org was behaving a bit sluggish lately, we spent some time cleaning up spammers, old and deleted pages and things started flushing way better. More details about the cleanup can be found at https://mail.gnome.org/archives/foundation-list/2013-May/msg00098.html. (we cleaned up around 23000 spammers!)
The most exciting things I usually love to announce are new services. While I always prefer keeping the number of maintained services as low as possible it was time for the GNOME Infrastructure to broaden its horizons satisfying the requests coming from the community and the developers involved into the project. I won’t spend any more word about this since I’m sure you are all waiting for me to list the new services, so here they are:
A completely new Jabber service hosted at jabber.gnome.org and accessible by all the GNOME Foundation members requesting access to it. More details about it can be found at https://live.gnome.org/Sysadmin/Jabber.
GNOME is extensively using IRC as its main communication tool, thus we’ve improved our Services IRC Bot to use a plugin called MeetBot. Having a meeting and storing the logs in a public web server is currently possible with a really minor effort of learning a few commands to administer the plugin correctly. If you are going to have a meeting and you want to make use of MeetBot, make sure that Services is there, and give a look at http://meetbot.gnome.org/Manual.html.
Do you want to be always up-to-date with the status of the GNOME Infrastructure? and are you actually wondering what’s the best way to do so? if yes, you should probably have a look at http://status.gnome.org. This service makes use of Fedora-Status by Patrick Uiterwijk and allows the GNOME Sysadmin Team to let everyone know whether there is a problem with any of the services listed in the page. This service, together with the public view for Nagios and the brand new infrastructure-announce@gnome.org mailing list will definitely help everyone finding out what’s going on, where and how long it’ll take for the issue to be fixed.
It took me a lot of pain having the KGB IRC Collaboration bot packaged into the EPEL repositories but I finally managed to set it up on the GNOME Infrastructure. KGB has become very handy since the time Cia.vc closed its hosting and it’s available for anyone requesting access to it at irc.gnome.org. If you are looking for Git commit notifications of a specific module directly on your IRC channel, this is what you want :-)
An Owncloud instance is also available, more reading about what are the requirements for requesting access to it at the following link.
An Etherpad istance is also available at https://etherpad.gnome.org to all the GNOME Teams that need it! Please drop me an e-mail at if you are interested. (Pad’s creation is currently disabled for preventing spam)
Build.gnome.org has been revived and it’s currently hosting an OSTree istance. GNOME Daily images are costantly being generated, interested in testing one the those images? give this article a look.

That’s all for now! See you all at GUADEC and thanks everyone for all the hints, suggestions and mails you’ve been sending me in the past months! And a special thanks to Ekaterina Gerasimova for taking the time to brainstorm with me suggesting new features and improvements over the GNOME Infrastructure!

Setting up your SSL certificates on OpenLDAP by using a Mozilla NSS database

Andrea Veri — Wed, 27 Mar 2013 12:04:02 +0000

I’ve recently spent some time setting up TLS/SSL encryption (SSSD won’t send a password in clear text when an user will try to authenticate against your LDAP server) on an OpenLDAP istance and as you may know the only way for doing that on a RHEL / CentOS environment is dealing with a Mozilla NSS database (which is, in fact, a SQLite database). I’ve been reading all the man pages of the relevant tools available to manipulate Mozilla NSS databases and I thought I would have shared the whole procedure and commands I used to achieve my goal. Even if you aren’t running an RPM based system you can opt to use a Mozilla NSS database to store your certificates as your preferred setup.

On the LDAP (SLAPD) server

*Re-create .db files

mkdir /etc/openldap/certs
modutil -create -dbdir /etc/openldap/certs

Setup a CA Certificate

certutil -d /etc/openldap/certs -A -n "My CA Certificate" -t TCu,Cu,Tuw -a -i /etc/openldap/cacerts/ca.pem

where ca.pem should be your CA’s certificate file.

Remove the password from the Database

modutil -dbdir /etc/openldap/certs -changepw 'NSS Certificate DB'

Creates the .p12 file and imports it on the Database

openssl pkcs12 -inkey domain.org.key -in domain.org.crt -export -out domain.org.p12 -nodes -name 'LDAP-Certificate'

pk12util -i domain.org.p12 -d /etc/openldap/certs

where **domain.org.key **and **domain.org.crt **are the names of the certificates you previously created at your CA’s website.

List all the certificates on the database and make sure all the informations are correct

certutil -d /etc/openldap/certs -L

Configure /etc/openldap/slapd.conf and make sure the TLSCACertificatePath points to your Mozilla NSS database

TLSCACertificateFile /etc/openldap/cacerts/ca.pem
TLSCACertificatePath /etc/openldap/certs/
TLSCertificateFile LDAP-Certificate

Additional commands

Modify the trust flags if necessary

certutil -d /etc/openldap/certs -M -n "My CA Certificate" -t "TCu,Cu,Tuw"

Delete a certificate from the database

certutil -d /etc/openldap/certs -D -n "My LDAP Certificate"

On the clients (nslcd uses ldap.conf while sssd uses /etc/sssd/sssd.conf)

On /etc/openldap/ldap.conf

BASE dc=domain,dc=org
URI ldaps://ldap.domain.org

TLS_REQCERT demand
TLS_CACERT /etc/openldap/cacerts/ca.pem

On /etc/sssd/sssd.conf

ldap_tls_cacert = /etc/openldap/cacerts/ca.pem
ldap_tls_reqcert = demand
ldap_uri = ldaps://ldap.domain.org

How to test the whole setup

ldapsearch -x -b 'dc=domain,dc=org' -D "cn=Manager,dc=domain,dc=org" '(objectclass=*)' -H ldaps://ldap.domain.org -W -v

Troubleshooting

If anything goes wrong you can run SLAPD with the following args for its debug mode:

/usr/sbin/slapd -d 256 -f /etc/openldap/slapd.conf -h "ldaps:/// ldap:///"

**Possible errors: **

If you happen to see an error similar to this one: “TLS error -8049:Unrecognized Object Identifier.“, try running ldapsearch with its debug mode this way:

ldapsearch -d 1 -x -ZZ -H ldap://ldap.domain.org

Make also sure that the FQDN you are trying to connect to is listed on the trusted FQDN’s list of your domain.org.crt.

Update: as SSSD’s developer Stephen Gallagher correctly pointed out on the comments using ldap_tls_reqcert = allow isn’t a best practice since it may take in Man in the Midle Attacks, adjusting the how to to match his suggestions.

Some updates from the GNOME Sysadmin Team

Andrea Veri — Thu, 07 Mar 2013 14:49:26 +0000

It’s been more than a month now since I started looking into the many outstanding items we had waiting on our To Do list here at the GNOME Infrastructure. A lot has been done and a lot has yet to come during the next months, but I would like to share with you some of the things I managed to look at during these weeks.

As you may understand many Sysadmin’s tasks are not perceived at all by users especially the ones related to the so-called “Puppet-ization” which refers to the process of creating / modifying / improving our internal Puppet repository. A lot of work has been done on that side and several new modules have been added, specifically Cobbler, Amavisd, SpamAssassin, ClamAV, Bind, Nagios / Check_MK (enabling Apache eventhandlers for automatic restart of faulty httpd processes), Apache.

Another top priority item was migrating some of our services off to the old physical machines to virtual machines I did setup earlier. The machines that are now recycled are the following: (Two more are missing on the list, specifically window (which still hosts art.gnome.org, people.gnome.org and projects.gnome.org due to be migrated to another host in the next weeks) and label. (which still hosts our Jabberd, an interesting discussion on its future is currently ongoing on Foundation-list)

menubar, our old Postfix host served the GNOME Foundation since 2004 and processed millions of e-mails from and to @gnome.org addresses.
container, our old main NFS node was serving the GNOME Foundation since 2003, it hosted our mail archives, our FTP archives and all the /home/users/* directories.
button hosted many services (MySQL databases, LDAP, Mango) and served the Foundation since 2004, a faulty hardware took it down on January 2012.

And if you ever wanted to see how menubar, container and button look like, I have two photos for you with the machines being pulled out the GNOME rack:

Some of the things you may have perceived directly on your skin should be the following:

Our live.gnome.org istance has been upgraded to the latest MoinMoin stable release, 1.9.6.
The Services bot has been added to the GIMPNET network and currently manages all GNOME channels, it currently acts as a Nickserv, Chanserv. More information about how you can register your nickname and gain the needed ACLs at the following wiki page.
Several GNOME services and domains are now covered by SSL as you may have noticed on planet.gnome.org, news.gnome.org, blogs.gnome.org, l10n.gnome.org, git.gnome.org, help.gnome.org, developer.gnome.org.
Re-design of our Mailman archives as you can see at https://mail.gnome.org/archives/foundation-list. A big “thank you” goes to Olav Vitters for taking the time to rebuild our “archive” script from Perl to Python. About the Mailman topic, someone proposed me the use of HyperKitty, that’s something we will evaluate in the next coming months but I find it a very interesting alternative to the current mail archiving.

What should you expect next?

Bugzilla will be moved to another virtual machine and will be upgraded to the latest release.
An Owncloud istance will be setup for all the GNOME Foundation members and GNOME Teams that will need access to it.
A discussion will be started for setting up a Gitorious istance on the GNOME Infrastructure.
A long-term item will be rewriting Mango in Django and adding several other features to it than the ones it has now. (ideally voting for Board elections, logins for managing your LDAP information such as your @gnome.org’s alias forward, shutdown of old an unused accounts after a certain period of time, automatic @gnome.org’s alias creation after the “Foundation Membership” flag is selected on LDAP, etc.)

Thanks a lot for all the mails I’ve received during these weeks containing reports and suggestions about how we should improve our Infrastructure! Please stay tuned, a lot more news are yet to come!

IPv6 tunneling with Hurricane Electrics (HE)

Andrea Veri — Fri, 21 Dec 2012 13:08:20 +0000

I’ve been looking around for a possible way to connect to the IPv6 internet for some time now and given the fact my provider didn’t allow me to run IPv6 natively I had to find an alternative solution. Hurricane Electrics (HE) provides (for free) five configurable IPv4-to-IPv6 tunnels together with a free DNS service and an interesting certification program.

Willing to test the latest revision of the Internet Protocol on your Debian, Ubuntu, Fedora machines? Here’s how:

1. Register yourself at Hurricane Electrics by visiting tunnelbroker.net.

2. Create a new tunnel and make sure to use your public IP address as your IPv4 Endpoint.

3. Write down the relevant details of your tunnel, specifically:

Server IPv6 Address: 2001:470:1f0a:a6f::1 /64
Server IPv4 Address: 216.66.84.46 (this actually depends on which server did you choose on the previous step)
Client IPv6 Address: 2001:470:1f0a:a6f::2/64

4. Create a little script that will update your IPv4 tunnel endpoint every time your internet IP changes. (this step is not needed if you have an internet connection with a static IP):

#!/bin/bash
USERNAME=yourHEUsername
PASSWORD=yourHEPassword
TUNNELID=yourHETunnelID
GET "https://$USERNAME:$PASSWORD@ipv4.tunnelbroker.net/ipv4_end.php?tid=$TUNNELID"

5. Create the networking configuration files on your computer:

Debian / Ubuntu, on the /etc/network/interfaces file:

auto he-ipv6
iface he-ipv6 inet6 v4tunnel
address 2001:470:<b>1f0a</b>:a6f::2
netmask 64
endpoint 216.66.80.30
local 192.168.X.X (Your PC's LAN IP address)
ttl 255
gateway 2001:470:<b>1f0a</b>:a6f::1
pre-up /home/user/bin/update_tunnel.sh

Fedora, on the ** /etc/sysconfig/network-scripts/ifcfg-he-ipv6** file:

DEVICE=he-ipv6
TYPE=sit
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6TUNNELIPV4="216.66.80.30"
IPV6TUNNELIPV4LOCAL="192.168.X.X" (Your PC's LAN IP address)
IPV6ADDR="2001:470:<b>1f0a</b>:a6f::2/64"

and on the /etc/sysconfig/network file, add:

NETWORKING_IPV6=yes
IPV6_DEFAULTGW="2001:470:<b>1f0a</b>:a6f::1"
IPV6_DEFAULTDEV="he-ipv6"

You can then set up a little /sbin/ifup-pre-local script to update the IPv4 tunnel endpoint when your dynamic IP changes or simply add the script on the /etc/cron.daily directory and have it executed when you turn up your computer.

6. Change the DNS servers on /etc/resolv.conf:

OpenDNS:

nameserver 2620:0:ccc::2
nameserver 2620:0:ccd::2

Google DNS:

nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

7. Restart your network and enjoy IPv6!

8. If you want to know more about IPv6 take some time for the HE Certification program, you will learn a lot and eventually win a sponsored t-shirt, I just finished mine :-)

EDIT: Be aware of the fact that as soon as the tunnel is up, your computer will be exposed to to the internet without any kind of firewall (the tunnel sets up a direct connection to the internet, even bypassing your router’s firewall), you can secure your machine by using ip6tables. Thanks Michael Zanetti for pointing this out!

The future is Cloudy

Andrea Veri — Thu, 13 Dec 2012 14:22:25 +0000

Have you ever heard someone talking extensively about Cloud Computing or generally Clouds? and have you ever noticed the fact many people (even the ones who present themselves as experts) don’t really understand what a Cloud is at all? That happened to me multiple times and one of the most common misunderstandings is many see the Cloud as something being on the internet. Many companies add a little logo representing a cloud on their frontpage and without a single change on their infrastructure (but surely with a price increment) they start calling their products as being on the Cloud. Given the lack of knowledge about this specific topic people tend to buy the product presented as being on the Cloud without understanding what they really bought.

But what Cloud Computing really means? it took several years and more than fifteen drafts to the National Institute of Standards and Technology‘s (NIST) to find a definition. The final accepted proposal:

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

The above definition requires a few more clarifications specifically when it comes to understand where should we focus on while checking for a Cloud Computing solution. A few key points:

On-demand self-service: every consumer will be able to unilaterally provision multiple computing capabilities like server time, storage, bandwidth, dedicated RAM or CPU without requiring any sort of human interaction from their respective Cloud providers.
Rapid elasticity and scalability: all the computing capabilities outlined above can be elastically provisioned and released depending on how much demand my company will have in a specific period of time. Suppose the X company is launching a new product today and it expects a very large number of customers. The X company will add more resources to their Cloud for the very first days (where they suppose the load to be very high) and then it’ll scale the resources back as they were before. Elasticity and scalability permit the X company to improve and enhance their infrastructure when they need it with an huge saving in monetary terms.
Broad network access: capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).
Measured service: Cloud systems allow maximum transparency between the provider and the consumer, the usage of all the resources is monitored, controlled, and reported. The consumer knows how much will spend, when and in how long.
Resource pooling: each provider’s computing resources are pooled to serve multiple consumers at the same time. The consumer has no control or knownledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
Resources price: when buying a Cloud service make sure the cost for two units of RAM, storage, CPU, bandwidth, server time is exactly the double of the price of one unit of the same capability. An example, if a provider offers you one hour of bandwitdh for 1 Euro, the price of two hours will have to be 2 Euros.

Another common error I usually hear is people feeling Cloud Computing just as a place to put their files online as a backup or for sharing them with co-workers and friends. That is just one of the available Cloud “features“, specifically the “Cloud Storage“, where typical examples are companies like Dropbox, Spideroak, Google Drive, iCloud and so on. But let’s make a little note about the other three “features”:

Infrastructure as a Service (IaaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. In this specific case the consumer has still no control or management over the underlying Cloud infrastructure but has control over operating systems, storage, and deployed applications. A customer will be able to add and destroy virtual machines (VMs), install an operating system on them based on custom kickstart files and eventually manage selected networking components like firewalls, hosted domains, accounts.
Platform as a Service (PaaS). the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages, libraries, services, and tools (like Mysql + PHP + PhpMyAdmin or Ruby on Rails) supported by the provider. In this specific case the consumer has still no control or management over the Cloud infrastructure itself (servers, OSs, storage, bandiwitdh etc.) but has control over the deployed applications and configuration settings for the application-hosting environment.
Software as a Service (SaaS): the capability provided to the consumer is to use the provider’s applications running on a Cloud infrastructure. The applications are accessible through various client devices, such as a browser, a mobile phone or a program interface. The consumer doesn’t not manage nor control the Cloud infrastructure (servers, OSs, storage, bandwidth, etc.) that allows the applications to run. Even the provided applications aren’t customizable by the consumer, which should rely on limited configuration settings.

The Cloud Computing technology is reasonably the future but can we trust Cloud providers? Are we sure that no one will ever have access to our files except us? and what about governments interested in acquiring a specific customer data hosted on the Cloud?

I always suggest to read deeply both the Privacy Policy and Terms of Use of a certain service before signing in especially when it comes to choose a Cloud storage provider. Many providers have the same aspect, they seem to provide the same resources, the same amount of storage for the same price but legally they may present different problems, and that is the case of Spideroak vs Dropbox. Quoting from the Dropbox’s Privacy Policy:

Compliance with Laws and Law Enforcement Requests; Protection of DropBox’s Rights. We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of DropBox or its users; or (d) to protect Dropbox’s property rights. If we provide your Dropbox files to a law enforcement agency as set forth above, we will remove Dropbox’s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.

It’s evident that Dropbox employees can access your data or be forced by legal process to turn over your data unencrypted. On the other side, Spideroak on its latest update to its Privacy Policy states that data stored on their Cloud is encrypted and inaccessible without user’s key, which is stored locally on user’s computers.

And what about the latest research paper, titled “Cloud Computing in Higher Education and Research Institutions and the USA Patriot Act” written by the legal experts of the University of Amsterdam’s Institute for Information Law stating the anti-terror Patriot Act could be theoretically used by U.S. law enforcement to bypass strict European privacy laws to acquire citizen data within the European Union without their consensus?

The only requirement for the data acquisition is the provider being an U.S company or an European company conducting systematic business in the U.S. For example an Italian company storing their documents (protected by the European privacy laws and under the general Italian jurisdiction) on a provider based in Europe but conducting systematic business in the United States, could be forced by U.S. law enforcement to transfer data to the U.S. territory for inspection by law enforcement agencies.

Does someone really care about the privacy of companies, consumers and users at all? or better does privacy exists at all for the millions of the people that connect to the internet every day?

The Linux’s perception of my neighbours

Andrea Veri — Thu, 22 Nov 2012 11:45:04 +0000

I live in a little village close to the city and one of the houses close to my property is for rent since more than ten years. A lot of families and people succeeded in that house and every time someone new joined my Linux evangelist hat jumped in my head.

I’ve always presented myself as a Linux geek to my neighbours and it has been nice seeing how the Linux word evolved (with funny and surprising quotes) during the past ten years in their minds. A friend of mine (Aretha Battistutta) made a little comic strip out of the topic and the result is simply amazing.

Enjoy!

Report: FAD Milan 2012

Andrea Veri — Sat, 03 Nov 2012 16:38:28 +0000

Exactly one week ago I was attending the Fedora Activity Day organized in Milan in concomitance with the Linux Day event being organized in several italian cities. Meeting the Fedora italian team has been simply great, we’ve been collaborating remotely since more than an year now and finding out all them being so friendly and pleasant has been a great pleasure.

Each of us presented a specific Fedora-related topic (I personally talked about Fedora and its Infrastructure, my presentation is publicly viewable at the following link) and I must admit everyone did an awesome job, we taught and learnt from each other at the same time and given the fact it was the very first time the team was put together, well, I can say our mission was accomplished.

Unfortunately something didn’t work as expected and the number of visitors that joined our event was very limited. The lack of marketing is apparently one of the most common problems within Linux and its derivative distributions especially when it comes to attending specific events. I feel our event’s concomitance with the Linux Day was one of the main causes, especially when the LuG (Linux User Group) behind the event sponsors a distribution which is not the same as the one you promote. I’m pretty much sure there is a lot of space for improvements but the team is still very young and a lot of work will be put together to improve our next events, that said I would like to thank Gabriele Trombini and Marina Latini for taking the time and efforts for organizing everything in an awesome location like Milan is.

We also had the luck to have a Gnomer sitting in one of the chairs of the Politecnico‘s classroom where we were presenting our Fedora activities: Paolo Borelli, the father of Gedit and one of the most active GNOME developers. His first question right after meeting up? “hey Andrea, given the fact CIA.vc is dead, are you going to set up a KGB istance sooner or later for GNOME? being a sysadmin has never been easy :-)

SSH Tunneling for VNC

Andrea Veri — Sat, 06 Oct 2012 13:50:24 +0000

Logging in into a Linux machine and executing the hundreds commands available is just one of the most common usages of OpenSSH. Another interesting and very useful usage is tunneling some specific (or even all) traffic from your local machine to an external machine you have access to.

Today we’ll analyze how to access a certain virtual machine’s console by tunneling the relevant VNC port locally and accessing it through your favorite VNC client. The scenario:

Machine A is our main virtualization machine and hosts several virtual machines. (VMs)
Each VM has its own VNC port assigned. (usually the port range goes from 5900 to 5910 or even more if the hosted VMs are more than 10)
We’ll be using libvirt, thus virsh.

We first need to find out which port got assigned to the VM we want to have console access to:

sudo virsh

virsh # list
Id Name Status
----------------------------------------------------
5 foo running
6 bar running
7 foobar running

virsh # vncdisplay foobar
:3

We, then, create a tunnel which redirects all the traffic from the main virtualization machine’s port to the port we gonna specify in the next command:

ssh -f -N -L 5910:localhost:5903 user@machine-A.com

A few details about the previous command:

**-N **tells SSH to not execute any command after logging in.
-f tells SSH to hide into the background just before the command gets executed.
-L enables the port forwarding between the local (client) host and the host on the remote side.

And…why did I choose respectively port 5903 and 5910

While you can adjust port 5910 with your own choice (that will just move the tunneled traffic from port 5910 to your favorite port), that won’t work as expected with port 5903 since each VNC port is binded to the number of display virsh assigned to it. (for example, the bar VM may be running on display 5, thus its vncdisplay port will be 5905)

When done, fire up your favorite VNC client and create a new connection with the following details:

Protocol: VNC - Virtual Network Computing
Server: localhost - 127.0.0.1
Port: 5910

The connection will load and you’ll be put in front of your ‘foobar’ VM console.

Me, myself and I

Andrea Veri — Sat, 22 Sep 2012 11:19:47 +0000

I’m an Italian Red Hatter who lives in New York City. While my primary education field has been Law, my passion and dedication for the FOSS world have brought me to volunteer and keep up many activities for GNOME and Fedora for several years.

What I am doing now:

GNOME:
- Being the GNOME Infrastructure Team Coordinator
- Being the GNOME Foundation Membership Committee‘s Chairman.

What I have done in the past:

Maintained a set of RPM packages for the Fedora / EPEL repositories
Designed the old Edubuntu’s Homepage for the Ubuntu Italian LoCo Team as reported on my announcement on the Ubuntu-it’s forums.
Wrote several Wiki pages and their translations for the Ubuntu international and italian communities.
Have been an Ubuntu Developer uploading several packages into the main and universe repositories, full list available at my Launchpad ID.
Sponsored and cleaned up the Ubuntu’s universe queue within the Ubuntu Sponsors Team.
Have helped out the awesome Insight Team setting up a working Drupal website meant to publish and share news and articles related to Fedora.
Been an Application Manager (AM) and helped out new contributors to go through the NM process and hopefully welcome them as Debian Developers.
Have been a Debian Developer and working on my Debian packages available at my QA Page.
Was the secretary of the GNOME Foundation Board of Directors for the 2014-2015 and 2015-2016 terms.
Have been a proud member of the Fedora Infrastructure Team.

Random bits:

My Ubuntu MOTU interview is available at this link.
A brief summary of my Accounts Team work can be found browsing Paul Cutler’s Blog.
Some material from my candidacy for the Fedora Board: Election’s Questionnaire, Town Hall’s meeting log and my nomination.
I made the news at GNOME.org
My interview for World Trademark Review about the GNOME trademark dispute with Groupon is available here (requires registration).
My Bachelor degree thesis having “Legal profiles of the GNOME Desktop Environment: from source code’s freedom to users’ participation in the community” as its title can be consulted here. (italian only)

Presentations

My presentation for the FAD Milan 2012 event can be downloaded here
My presentation for the Open Source Day 2014 in Udine about Puppet is here (italian only)
GNOME Foundation Annual General Meeting (AGM) reports (GNOME Infrastructure)
- GUADEC 2013 slides video
- GUADEC 2014 slides
- GUADEC 2015 slides
- GUADEC 2018 slides
- GUADEC 2022 slides
- GUADEC 2025 slides lightning talks

Contact:

Email:

IRC:

av on irc.gimp.org.
averi on irc.libera.chat.

FAD Milano 2012

Andrea Veri — Wed, 19 Sep 2012 16:17:27 +0000

È con piacere che annuncio la mia presenza al Fedora Activity Day di Milano in data 27 Ottobre 2012 e con altrettanto piacere ringrazio gli organizzatori per avermi dedicato uno spazio di trenta minuti in cui discutere ed approfondire le tematiche derivanti dall’amministrazione di sistemi Linux, nello specifico, dell’amministrazione degli stessi computer che ospitano e rendono disponibili numerosi servizi alle migliaia di utenti Fedora che contribuiscono quotidianamente al progetto.

Ho partecipato fisicamente, purtroppo, a pochissimi eventi Open Source e per questo motivo sono particolarmente emozionato di aver la possibilità di prender parte ad una manifestazione così sentita ed apprezzata come il Linux Day, contesto in cui lo stesso Fedora Activity Day prenderà piede. Le aspettative sono tante e mi impegnerò in modo tale da presentare al pubblico il maggior numero di strumenti e attività inerenti all’amministrazione di sistemi Linux, il tutto con la speranza di invogliare il maggior numero dei presenti a contribuire allo stesso progetto Fedora.

A titolo introduttivo elenco alcuni dei punti su cui mi soffermerò durante il mio talk:

Quali sono i principi fondamentali a cui l’infrastruttura Fedora si ispira.
Chi c’è dietro all’infrastruttura Fedora, quante persone e con quali competenze, quali sono i mezzi di comunicazione.
Quanti e quali Datacenter permettono al progetto Fedora di funzionare.
Quali sono i software utilizzati, come vengono utilizzati e con quali strumenti il team Infrastruttura provvede alla manutenzione dei sistemi fisici e virtuali disponibili, come vengono monitorati i server Fedora e molto altro!

Con la speranza di una partecipazione numerosa, vi rimando alla pagina dedicata all’evento.

Building Debian packages with Deb-o-Matic

Andrea Veri — Mon, 17 Sep 2012 13:07:14 +0000

Today I’ll be telling you about an interesting way to build your Debian packages using Deb-o-Matic, a tool developed and maintained by Luca Falavigna. Some more details about this tool from the package’s description:

Deb-o-Matic is an easy to use build machine for Debian source packages based on pbuilder, written in Python.

It provides a simple tool to automate build of source packages with limited user interaction and a simple configuration. It has some useful features such as automatic update of pbuilder, automatic scan and selection of source packages to build and modules support.

The setup.

Download the package.

apt-get install debomatic

Modify the main configuration file as follows:

[default]
builder: pbuilder
packagedir: /home/john/debomatic # Take note of the following path since we'll need it for later use.
configdir: /etc/debomatic/distributions
pbuilderhooks: /usr/share/debomatic/pbuilderhooks
maxbuilds: 3 # The number of builds you can perform at the same time.
inotify: 1
sleep: 60 # The sleep time between one build and another.
logfile: /var/log/debomatic.log

[gpg]
gpg: 0 # Change to 1 if you want Deb-O-Matic to check the GPG signature of the uploaded packages.
keyring: /etc/debomatic/debomatic.gpg # Add the GPG Keys you want Deb-O-Matic to accept in this keyring.

[modules]
modules: 1 # A list of all the available modules will follow right after.
modulespath: /usr/share/debomatic/modules

[runtime]
alwaysupdate: unstable experimental precise
distblacklist:
modulesblacklist: Lintian Mailer
mapper: {'sid': 'unstable',
'wheezy': 'testing',
'squeeze': 'stable'}

[lintian]
lintopts: -i -I -E --pedantic # Run Lintian in Pedantic mode.

[mailer] # You need an SMTP server running on your machine for the mailer to work. You can have a look at the 'Ssmtp' daemon which is a one-minute-setup MTA, check an example over <a href="https://github.com/averi/config-files/blob/master/backups/offlineimap%20%2B%20ssmtp%20%2B%20imapfilter/ssmtp.conf" target="_blank">here</a>.
fromaddr: debomatic@localhost
smtphost: localhost
smtpport: 25
authrequired: 0
smtpuser: user
smtppass: pass
success: /etc/debomatic/mailer/build_success.mail-template # Update the build success or failure mails as you wish by modifying the relevant files.
failure: /etc/debomatic/mailer/build_failure.mail-template

[internals]
configversion = 010a

The available modules are:

“Contents“, which acts as a ‘dpkg -c’ over the built packages.
“DateStamp“, which displays build start and finish times into a file in the build directory.
“Lintian“, which stores Lintian output on top of the built package in the pool directory.
“Mailer“, which sends a reply to the uploader once the build has finished.
“PrevBuildCleaner“, which deletes all files generated by the previous build.
“Repository“, which generates a local repository of built packages.

Configure ‘dput‘ to upload package’s sources to your local repository, edit the /etc/dput.cf file and add this entry:

[debomatic]
method = local
incoming = /home/john/debomatic

or the following if you are going to upload the files to a different machine through SSH:

[debomatic]
login = john
fqdn = debomatic.example.net
method = scp
incoming = /debomatic

Add a new Virtual Host on Apache and access the repository / built packages directly through your browser:

<VirtualHost *:80>

ServerAdmin john@example.net
ServerName debomatic.example.net
DocumentRoot /home/john/debomatic

<Directory /home/john/debomatic>
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Order allow,deny
allow from all
<Directory>

</VirtualHost>

Start the daemon:

sudo /etc/init.d/debomatic start

(Optional) Add your repository to APT‘s sources.list:

deb http://debomatic.example.net/ unstable main contrib non-free

(Optional) Start Deb-O-Matic at system’s startup by modifying the /etc/init.d/debomatic file at line 21:

- [ -x "$DAEMON" ] || exit 0
- [ "$DEBOMATIC_AUTOSTART" = 0 ] && exit 0

+ [ -x "$DAEMON" ] || exit 0
+ [ "$DEBOMATIC_AUTOSTART" = 1 ] && exit 0

and finally add it to the desired runlevels:

update-rc.d debomatic defaults

Enjoy!

Manage your website through Git

Andrea Veri — Sat, 15 Sep 2012 12:52:08 +0000

Ever wondered how you can update your website (in our case a static website with a bunch of HTML and PHP files) by committing to a Git repository hosted on a different server? if the answer to the previous question is yes, then you are in the right place.

The scenario:

Website hosted on server A.
Git repository hosted on server B.

and a few details about why would you opt for maintaining your website through Git:

You need multiple people to access the static content of your website and you also want to maintain all the history of changes together with all the Git’s magic.
You think using an FTP server is not secure enough.
You think giving out SSH access or more permissions on the server to multiple users it’s not what you want. (also using scp will overwrite the files directly with all its consequences)

the setup, on server A:

Clone the Git repository over the directory that your Web server will serve.

cd /srv/www.domain.com/http && sudo git clone http://git.example.com/git/domain.git

Grab the needed package:

apt-get install fishpolld or yum install fishpolld

Set up a “topic” (called website_update) that will call a ‘git pull‘ each time the repository hosted on server B receives an update. (the file has to be placed into the /etc/fishpoll.d directory)

#!/bin/bash

PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
WEBSITEDIR="/srv/www.domain.com/http"

if [ -d "${WEBSITEDIR}" ]; then
cd "${WEBSITEDIR}"
else
echo "Unable to access theme directory. Failing."
exit 1
fi

git pull

Add a little configuration file that will enable the website_update‘s topic at daemon’s startup. (you should name it website_update.conf)

[fishpoll]
on_start = True

Open the relevant port on Iptables so that server A and server B can communicate as expected, the daemon runs over** port 27527**.

-A INPUT -m state --state NEW -m tcp -p tcp -s <strong>server-B-IP</strong> --dport 27527 -j ACCEPT

Start the daemon. (by default, logs are sent to syslog but you can run the daemon in Debug mode by using the -D flag)

sudo /etc/init.d/fishpolld start or sudo systemctl start fishpolld.service or sudo /usr/sbin/fishpolld -D

and on server B:

Grab the needed package:

apt-get install fishpoke or yum install fishpoke

Configure the relevant Git hook (post-update, located into the hooks directory) and make it executable.

#!/bin/sh

echo "Triggering update of configuration on server A"
fishpoke server-A-IP-or-DNS website_update

Finally test the whole setup by committing to the repository hosted on server B and verify your changes being sent live on your website!

Nagios IRC Notifications

Andrea Veri — Sat, 30 Jun 2012 12:59:28 +0000

Lately (as I earlier pointed out on my blog) I’ve been working on improving GNOME’s infrastructure monitoring services. After configuring XMPP it was time to find out a good way for sending out relevant notifications to our IRC channel hosted on GIMPNET. I achieved that with a nice combo: supybot + supybot-notify, all that mixed up with a few grains of Nagios command definitions.

But here we go with a little step-by-step guide:

Requirements

Install supybot and configure a new installation:

apt-get install supybot or yum install supybot
mkdir /home/$user/nagbot && cd /home/$user/nagbot
supybot-wizard (follow the directions to get the bot initially configured)

Install and load the supybot-notify plugin by doing:

git clone git://git.fedorahosted.org/supybot-notify.git && cd supybot-notify
mkdir -p /home/$user/nagbot/plugins/notify && cp -r * /home/$user/nagbot/plugins/notify

Finally, load the plugin. (this will require you to authenticate to the bot)

Nagios configuration

Add the relevant command definitions to the commands.cfg file:

# 'notify-by-ircbot' command definition
define command{
 command_name notify-by-ircbot
 command_line /usr/bin/printf "%b" "#channel $NOTIFICATIONTYPE$ - $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$: $SERVICEOUTPUT$ ($$(hostname -s))" | nc -w 1 localhost 5050
 }

# 'host-notify-by-ircbot' command definition
define command{
 command_name host-notify-by-ircbot
 command_line /usr/bin/printf "%b" "#channel $NOTIFICATIONTYPE$ - $HOSTALIAS$ is $HOSTSTATE$: $HOSTOUTPUT$ ($$(hostname -s))" | nc -w 1 localhost 5050
 }

* adjust the Netcat’s host and port to your needs, in my case Supybot and Nagios were running on the same host. In the case of a Supybot running on a different host than Nagios, tweak Iptables to allow the desired port:

-A INPUT -m state --state NEW,ESTABLISHED -m tcp -p tcp --dport 5050 -j ACCEPT

Add a new entry on the contacts.cfg file:

define contact{
 contact_name nagbot
 use generic-contact
 alias Nagios IRC Bot
 email example@example.com
 service_notification_commands notify-by-ircbot
 host_notification_commands host-notify-by-ircbot
}

Reload Nagios:

sudo /etc/init.d/nagios3 reload

And finally, enjoy the result:

PROBLEM - $hostalias/load average is CRITICAL: CRITICAL - load average: 30.45, 16.24, 7.16 (nagioshost)
 RECOVERY - $hostalias/load average is OK: OK - load average: 0.06, 0.60, 3.65 (nagioshost)

A few useful Puppet snippets

Andrea Veri — Tue, 31 Jan 2012 18:43:26 +0000

As per Wikipedia:

Puppet is a tool for managing the configuration of Unix-like systems, declaratively. The developer provides puppet templates for describing parts of the system, and, when these templates are deployed, the runtime puts the managed systems into the declared state.

Puppet consists of a custom declarative language to describe system configuration, distributed using the client-server paradigm (using XML-RPC protocol), and a library to realize the configuration. The resource abstraction layer enables administrators to describe the configuration in high-level terms, such as users, services and packages.

I’ve been playing with the aforementioned tool lately both on my home network and within the Fedora’s Infrastructure team and I thought some of the work I did might be useful for anyone out there being stuck with a Puppet’s manifest or an ERB template.

Snippet #1: Make sure the user ‘foo’ is always created with its own home directory, password, shell, and full name.

class users {
 users::add { "foo":
 username => 'foo',
 comment => 'Foo's Full Name',
 shell => '/bin/bash',
 password_hash => 'pwd_hash_as_you_can_see_in_/etc/shadow'
 }

define users::add($username, $comment, $shell, $password_hash) {
 user { $username:
 ensure => 'present',
 home => "/home/${username}",
 comment => $comment,
 shell => $shell,
 managehome => 'true',
 password => $password_hash,
 }
 }
}

Snippet #2: Make sure the user ‘foo’ gets added into /etc/sudoers.

class sudoers {

file { "/etc/sudoers":
 owner => "root",
 group => "root",
 mode => "440",
 }
}

augeas { "addfootosudoers":
 context => "/files/etc/sudoers",
 changes => [
 "set spec[user = 'foo']/user foo",
 "set spec[user = 'foo']/host_group/host ALL",
 "set spec[user = 'foo']/host_group/command ALL",
 "set spec[user = 'foo']/host_group/command/runas_user ALL",
 ],
}

Snippet #3: Make sure that openssh-server is: installed, running on Port 222 and accepting RSA authentications only.

class openssh-server {

 package { "openssh-server": 
 ensure => "installed",
 }

 service { "ssh":
 ensure => running,
 hasstatus => true,
 require => Package["openssh-server"],
 }

augeas { "sshd_config":
 context => "/files/etc/ssh/sshd_config",
 changes => [
 "set PermitRootLogin no",
 "set RSAAuthentication yes",
 "set PubkeyAuthentication yes",
 "set AuthorizedKeysFile %h/.ssh/authorized_keys",
 "set PasswordAuthentication no",
 "set Port 222",
 ],
 }
}

Snippet #4: Don’t apply a specific IPTABLES rule if an host is tagged as ‘staging’ in the relevant node file.

On templates/iptables.erb:

# Allow unlimited traffic on eth0
-A INPUT -i eth0 -j ACCEPT
-A OUTPUT -o eth0 -j ACCEPT

# Allow unlimited traffic from trusted IP addresses
-A INPUT -s 192.168.1.1/24 -j ACCEPT

&lt;% if environment == "production" %>

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

&lt;% unless defined?(staging).nil? %>
-A INPUT -s X.X.X.X -j REJECT --reject-with icmp-host-prohibited
&lt;% end -%>

&lt;% end -%>

On the manifest file:

class iptables {
 package { iptables:
 ensure => installed;
 }

 service { "iptables":
 ensure => running,
 hasstatus => true,
 require => Package["iptables"],
 }

 file { "/etc/sysconfig/iptables":
 owner => "root",
 group => "root",
 mode => 644,
 content => template("iptables/iptables.erb"),
 notify => Service["iptables"],
 }
}

That’s all for now!

A few other additions to my Mutt and Desktop setup!

Andrea Veri — Sun, 11 Sep 2011 18:25:32 +0000

A few days ago I blogged about my main computer’s configuration files and desktop’s appearance and today I managed to add a few little tweaks to those, they are:

Google’s contacts list integrated into Mutt
a cleaner and nicer Login screen

Curious to know how you can easily integrate your Google’s contacts into Mutt? Well, you should be able to achieve that within a few minutes after reading this small HowTo:

1. Download and install goobook as explained here.

2. Setup a .goobookrc file into your Home directory. It should look like this:

machine google.com
login example@gmail.com
password yourpassword

3. Add the relevant configuration bits into your /etc/Muttrc file:

set query_command="goobook query '%s'"
bind editor complete-query
macro index,pager a ";goobook add" "Add sender's address to your Google contacts"

4. Your configuration should be good to go now, so here’s a few examples on goobook’s usage within Mutt:

Use TAB if you want to auto-complete a mail address when specifying the To: field.
Use the A key if you want to add sender’s address to your Google contacts.
Use the Q key for querying your contacts list.

We can now move on on customizing your Login Screen running GDM3. Let’s begin with a screenshoot:

I definitely love it, it’s clear and clean and most of all it has everything I need, no extra toolbars or menus. If you agree with me, open up the /etc/gdm3/greeter.gconf-defaults file and do the needed changes. This is how your greeter.gconf-defaults file should look like:

/desktop/gnome/background/picture_filename /path/to/your/dusty-bg/file # dusty's background can be downloaded <a href="http://gnome-look.org/content/show.php/Dusty?content=94332" target="_blank">here</a>.
/desktop/gnome/interface/gtk_theme Darklooks # this is my main theme, feel free to adapt that to your needs.
/apps/gdm/simple-greeter/logo_icon_name debian-swirl # this is the default on Debian's systems.
/desktop/gnome/sound/event_sounds false # I don't like hearing any sound when when I am prompted to insert my user's details on the Login Screen.
/apps/gdm/simple-greeter/disable_user_list true # users list will be disabled, you won't be able to select your username from a list but you'll have to insert that yourself.
/apps/metacity/general/compositing_manager false # default, no need to change this.
/apps/gnome-power-manager/ui/icon_policy never # default, no need to change this.

We are close to the end but we are missing an important detail: how can you safely remove bottom’s toolbar and menus for a clearer and cleaner Login Screen? Open up the /var/lib/gdm3/.gconf.mandatory/%gconf-tree.xml file, search for the <dir name=“general”> section and apply the following change:

- <entry name="compositing_manager" mtime="1315580582" type="bool" value="false"/>;
+ <entry name="compositing_manager" mtime="1315580582" type="bool" value="true"/>

But what if you prefer keeping the toolbar as it is, but you definitely don’t like seeing the Accessibility icon appearing on your Login Screen? On the same file as above, search for the <dir name=“general”> section and modify the following string as it follows:

- <entry name="enable" mtime="1315580582" type="bool" value="true"/>
+ <entry name="enable" mtime="1315580582" type="bool" value="false"/>

See you on my next blog post and don’t forget to have a look at my GitHub’s repository! Oh…and follow me on Twitter!

New Desktop, Mutt and Irssi setup!

Andrea Veri — Sun, 04 Sep 2011 14:41:10 +0000

I bought a new PC a few weeks ago and I then decided to renew a bit my Desktop, my Mutt and my Irssi setup. I’ve been spending several hours cleaning up old scripts, logs and configuration files but the result definitely seems to reward me the right way. But here they come a few screenshots:

Desktop

Irssi

Mutt

If you liked all the above and would like to reproduce everything yourself, you should consider having a look at my GitHub’s repository. See you on the next blog post!

Backup your Gmail in a few easy steps!

Andrea Veri — Mon, 25 Jul 2011 22:50:57 +0000

I’ve actually spent a few hours searching around for a good backup solution for my mailbox until I decided to stick with getmail. What you’ll be able to achieve after reading this HowTo and deploying the following setup is:

A full backup of your e-mail DATA in the Mbox format. (yes, Gmail’s labels / folders as well)
Prevent getmail to mark all mails as read after delivering them. (this was a pretty bad issue since getmail was marking all my mails as read even if I did not access my e-mail at all)
Keep your backups up-to-date with the latest content from your mailbox. (by default getmail grabs all the DATA from your mailbox and fills up the Mbox / Maildir content keeping deleted mails. So let’s say I deleted a mail two days ago, well it’ll still appear on today’s backups. This behaviour is definitely unwanted)

I’ll now move to explain a few details about my new configuration but before moving to tweak getmail’s main config file, please do the following change on _retrieverbases.py* :

return self._getmsgpartbyid(msgid, '(RFC822)')

return self._getmsgpartbyid(msgid, '(BODY.PEEK[])')

When done grab the following getmailrc and adapt it to your needs**:

[retriever]
type = SimpleIMAPSSLRetriever ## or SimplePOP3SSLRetriever.
server = imap.gmail.com ## or pop.gmail.com for POP3.
username = example@gmail.com
password = password

## so-called Gmail's labels should be listed one by one here for getmail to retrieve mail from them successfully.

mailboxes = ("INBOX", "[Gmail]/Sent mail",
"ubuntu", "gnome/example", "linux/example")

[destination]
type = Mboxrd
path = ~/.getmail/backup.mbox

[options]
delivered_to = false ## No delivered_to header added automatically.
received = false ## No received header added automatically.
verbose = 2 ## getmail will print messages about each of its actions.

When done we should go ahead setting up getmail’s directories and config file:

mkdir $HOME/.getmail
cp $HOME/getmailrc $HOME/.getmail/

Adapt $HOME/getmailrc to whatever dir you put that file into. But…pretty much all the remaining work will be done by a small shell script I wrote:

#!/bin/sh

WORKDIR=$HOME/.getmail
date=`date "+%d-%m-%Y_%H:%M"`

if [ ! -f  $WORKDIR/backup.mbox ]
then
touch $WORKDIR/backup.mbox
fi

getmail > $WORKDIR/getmail.log
OUT=$?
if [ $OUT -eq 0 ]
then
mkdir -p $WORKDIR/backups/ && { mv $WORKDIR/backup.mbox $WORKDIR/backups/backup_$date.mbox ;}
else [ $OUT -eq 1 ]
exit 1
fi

## Cleanup older than 3 days backups
find $WORKDIR/backups/* -mtime +3 -exec rm {} ;
cd $WORKDIR && { rm -rf oldmail-* ;}

This script will:

Run getmail using the getmailrc config file you previously worked on.
If the above command will be successful, it’ll create a_ backups_ dir into $HOME/.getmail and move the latest Mbox file there appending a date and time to its name. (by doing this we are sure next getmail run will happen on an empty backup.mbox file, thus it will just contain the latest content from your mailbox)
It’ll re-create a backup.mbox file on $HOME/.getmail to avoid the next getmail run to fail.
In the end, it’ll clean up older than 3 days backups to avoid a too crowded backups folder. (it removes the oldmail file as well since it is useless in our case)

In the end set up a cronjob that will run the above script and generate the backups for you every one hour:

0 * * * * $HOME/.getmail/getmail_run.sh > /dev/null

Feel free to let me know if you’ve encountered any issue while following the above HowTo. Enjoy!

* /usr/share/getmail4/getmailcore/_retrieverbases.py on line 901.

** More documentation about the getmailrc file and syntax can be found on getmail’s documentation page.

Automatic Gmail’s Trash & Spam folders cleanup

Andrea Veri — Sun, 03 Jul 2011 21:29:35 +0000

Since some time I’ve been thinking about a possible way to delete my Gmail’s Trash & Spam folders content automatically without having to bother doing it manually every single time I wanted to check my mail and clean it up. (I love keeping everything in place and having my Trash&Spam folders empty as they should be makes me pretty happy)

A few years ago when Mutt was my main mail client I had the need to filter my mail through IMAP and while googling around for that I found out a great piece of software: imapfilter. Today while analyzing the above quoted issue I suddenly told myself: “Hey, but why don’t you use your dear and old friend imapfilter to fulfil your needs?”

After a few minutes I came up with a small lua script that was doing exactly what I wanted: my Trash&Spam folders are no longer crowded and I finally don’t have to delete mails twice! But here they come a few details about my script:

options.timeout = 120
options.subscribe = true

account = IMAP {
 server = 'imap.gmail.com',
 username = 'example@gmail.com',
 password = 'password',
 ssl = 'ssl3'
 }

trash = account['[Gmail]/Trash']:is_undeleted()
account['[Gmail]/Trash']:delete_messages(trash)

spam = account['[Gmail]/Spam']:is_unanswered()
account['[Gmail]/Spam']:delete_messages(spam)

The script does two things:

It checks whether a mail is not marked as “deleted” (moving an e-mail into the Trash does not mark it as “to be deleted” automatically) already and removes it.
It checks whether a mail on the Spam folder has been answered (I never had to answer a single e-mail contained into my Spam folder) and if not removes it.

Using the above script is really easy (you should run imapfilter on interactive mode first to generate Gmail’s certificates, do that before having cron to run the script for you or otherwise it’ll just hang), just make sure to have imapfilter installed on your system and then run it through cron every half an hour or less depending on your needs:

crontab -e

*/30 * * * * imapfilter -c /home/user/imapfilter.lua >> /home/user/imapfilter.log

Please also remember to setup appropriate permissions on the config file since it contains your Gmail’s password and most of all make sure that your Spam folder is visible through IMAP (this option can be found on the label menu available under your Gmail’s settings) otherwise imapfilter will just report an error.

Enjoy!

Fedora Board’s Town Hall

Andrea Veri — Mon, 30 May 2011 21:06:14 +0000

Today we had a great Town Hall meeting kindly hosted and moderated by Kevin Fenzi (nirik). We received a lot of interesting and nice questions by the contributors and developers that were attending:

What do you feel needs to be improved in the Fedora Community? How can you’re being on the Board improve the Community?
Do you think that too many issues in Fedora are referred directly to the Board, and if so, how would you like to see this improved?
Tell us something about what you like doing that isn’t computer or fedora related. What do you like doing for fun?
What do you plan to do about the issues of polish? Specifically, shipping with minor issues that with recent releases have been hurting the Fedora name.
What are the plans for mobile devices, such as phones, tablets, ‘pads’, etc.? What are the chances of working on a ‘spin’ for such emerging technologies?
The board has discussed working on “goals” over the next term. (a) do you think these goals should be focused on helping “us” (people already in the community) or our “target audience”? (b) what goals would you like to see fedora achieve?
Is anyone in favor the board doing more of its business in public view? I mean like all of it that actually can be?
What do you plan to do to address operator abuse in #fedora? 2. What penalties will there be for operators when it’s deemed that they are abusing their authority or swaying from Fedora’s values?
What can be done to bring Fedora to lead market share amonsgt Linux desktops? What can be done to take market share from Microsoft Windows?
How do you measure the success of the Fedora Project as a whole?
Recently we’ve seen an influx in new users with questions as well as new volunteers with skills (and no idea where to make use of them). What should we do to better facilitate community engagement?
How do the candidates feel that they are viewed by the general population of new Fedora users as representatives of Fedora, and do they set an example of model behavior? If so how?

If any of the above questions do cover a topic you are interested in, please take a few minutes to read candidates answers and discussions by looking at the meeting’s log. Enjoy!

Fedora Board’s Questionnaire

Andrea Veri — Wed, 25 May 2011 20:45:10 +0000

If you are not subscribed to the fedora-announce mailing list but you are still interested in having a look at my responses about community-asked questions, here they are:

What will you be able to accomplish by being elected, that you would not otherwise be able to do as a contributor?

(As you will notice by reading the list right down here) Being a single contributor makes achieving these points impossible since changing how localized communities should work, improving our CoC and enforcing its rules and re-thinking Board’s role in our community is something that must be discussed and voted within the Board and its members.

What are your top three priorities as a board member?

If elected, I will mainly try to focus on:

Improving Fedora’s localization putting a great effort on introducing a form of formalization for specific localized communities having all the needed requirements to gain the “blessing” of Official local community for a certain country or language / dialect. This means pursuing one main objective, which is making Fedora Ambassadors and contributors not fighting each other but acting together as a community. Having two-three or even four websites / local communities just for the Italian or French langs is simply the wrong way to achieve the result of having a Fedora community together again. Ambassadors and contributors of a specific country or lang should focus on establishing *one* strong and trusted localized community, they should throw away the idea of multiple support websites, we need to put together everyone again, act as a team, Fedora together should be our motto. (the specific requirements to gain the above formalization will be written up by me and presented to the Board for a discussion, so expect more news to come about this point if my candidature will be accepted)
Re-thinking what the Fedora Board should be within the Fedora community. It should represent the community and all its members, if a single or multiple members are having a specific problem, from the bigger to the smaller one, the Board must deal with them to find a valid solution, nothing and no one should be left behind. The Board, in the end, should be the main reference point for everyone wanting to propose a new idea or just willing to costructively complain about something not working in the right way. Discussing problems, respecting everyone’s ideas and opinions and finding a good consensus / common solutions for everyone is alwais the way to go to improve the relationships between community members, contributors and developers.
Improving our Code of Conduct, finding a good way to enforce members respecting it and remembering which values should be found behind a community (respect between members and their ideas, costructive discussions, decisions taken with general consensus etc.) is the latest point but it’s definitely not the less important on my list. As I stated in my candicacy, I’ve been negatively impressed by the behaviour of some community members in two occasions: while introducing JustFedora’s Planet and while working on another Infrastructure duty. Criticizing without valid motivations just for the sake of doing so seemed to be the common rule on both of the above cases. I would like to remember everyone that this is *not* the best behaviour for an Open Source community, we need to act together as a single team, we don’t have to fight each other but we have to cooperate finding common solutions, discussing, criticizing *costructively* and helping our community coming out from the current situation.

Who do you think Fedora is for today? Who should it be for?

Fedora is about innovation, but as you may all know, innovation might take in several problems especially for new comers or people switching from a Microsoft OS. Most of the people I know do have a lot of problems to simply open up a computer, writing a mail or working on a document; I would like to see Fedora (but generally Linux based OSs) available to use to everyone: from developers to complete newbies. I would like to work making the idea of Linux being usable by a restricted circle of people changing and I’m sure the arrival of GNOME 3 will definitely help us out on achieving our goal. (It’s user-friendly but innovative interface it’s simply superb)

Where do you see Fedora in five years? How do you think we’ll get there?

An innovative Fedora but at the same time a distribution easy to use by any of us out there, an awesome cloud service available to everyone, a package manager made easier to learn and understand by newcomers, a GNOME 3 improved, stronger, robust and a gnome-shell completely ready and fully integrated on Fedora is what I would like to see happening within five years.

What will you do to ensure that Fedora remains at the forefront of innovation in the GNU/Linux space?

I will try to do my best to give a warm welcome to new ideas and projects within Fedora, I’ll listen, discuss with as many contributors and developers willing to propose something new and innovative that could benefit our beloved distribution. I would like to link this answer with the third point of the first answer I gave on the questionnaire: new ideas are strictly related to my vision of innovation, everyone should be free to propose something new without having to worry about receiving personal insults or complaints: this is unfortunately missing in our community. (is our community really prepared for new ideas yet?)

Planet Edited? More details to come!

Andrea Veri — Fri, 11 Feb 2011 18:43:24 +0000

Today I had the possibility to announce a new tool for the Fedora Project, a sub-planet called ‘Edited’ mainly focused on Fedora-related posts and announcements.

The scenario we gonna have when Edited will become a known tool by the whole project can be resumed as it follows:

team leaders or whoever will be appointed to, will send out their team’s status and any relevant announcement such as important changes, needs for help directed to any contributor unsure about where and how to start contributing to a specific team. (I heard dozen times new contributors coming into #fedora-admin asking more informations about when and where should they start contributing. I think having a working tool like Edited will make it easier for everyone finding a duty to work at)
a new contributor or a general user willing to know more about how Fedora works behind the scenes will find in Edited the greatest companion ever. No need to cherrypick relevant informations between several posts, but all the information he/she might need directly in a few clicks.

Please remember that Planet Fedora is not and won’t be the same as Edited. They are different things and they will remain separate. While Planet Fedora is open to any general discussion (from talking about Debian or Ubuntu, to any other Free Software topic), we will try to make Edited as much as we can close to Fedora and its development.

Lastly, one of the things I had in mind when setting Edited up was transparency and that’s why all the requests are currently stored on the Fedora Infrastructure’s Trac istance. (I added a link about how the whole process works but looks like someone didnt see it, correct Larry?)

Anyway I’ve been talking with two or three team leaders and they told they will be more than happy to send out announcements through Edited. Let’s make it rock then!

See you on Edited!

Bits from September / October

Andrea Veri — Sat, 10 Oct 2009 22:17:23 +0000

Debian maintainer:

Some days ago I had the great announcement that my Debian Maintainer‘s application was accepted and thanks to Jonathan McDowell my key is finally into the debian-maintainers keyring. (which is now part of the debian-keyring itself thanks to the ftp-masters / keyring maintainer work that made the changes on both DAK and keyring)

I applied on the 22th of August, and thanks to Reinhard’s advocation, everything went on the right side. Many thanks to all the people who made this possible.

Pkg-mozext:

Around 3 weeks ago I started being involved with the pkg-mozext team working on the new policy and to mozilla-devscripts transition for all iceweasel / icedove extensions. The work is going great so far and we managed to update several extensions that are now part of the team itself. For a fast look at our package list, feel free to browse pkg-mozext’s QA page. A lot of work needs to be done, so if anyone wanna jump in, I would suggest to read our guidelines / policy into the team’s Wiki Page.

Pkg-gnome:

GNOME 2.28 is now released, and the pkg-gnome team is now working hard to provide all the needed updates into unstable and experimental. Everything is going great and we are cleaning up everything listed on our schedule.

Ephipany is under transition (gecko –> webkit) so there are still some open issues / bugs, but now it looks fine so far and it can be used without any particular problem.

BehindMOTU: La mia Intervista…

Andrea Veri — Sat, 20 Oct 2007 10:27:00 +0000

Pochi giorni fa ho avuto l’onore di essere intervistato da BehindMOTU, vi riporto l’intera intervista (ovviamente in inglese) augurandovi ,inoltre, una buona lettura! Potete trovare il post originale qua.

Today we are interviewing Andrea Veri, fresh MOTU and eager Ubuntu volunteer.

Age: 18

Location: Udine, Italy

IRC Nick: bluekuja

How long have you used Linux and what was your first distro?

I started using Linux at the end of 2005 using Red Hat and Fedora distros, contributing on writing several pages for Fedora documentation (mostly server docs) but mainly working on some packaging-related activities (introducing ctorrent, gtorrent-viewer and v2strip packages inside Fedora) for more than 3 months until the beginning of March 2006 when I decided to move definitely to Ubuntu after discovering it at a friend’s party. Was love at first sight that made me leaving every Fedora plan and project creating my first personal wiki page on wiki.ubuntu.com some days later.

How long have you been using Ubuntu?

In fact, I started using Ubuntu at the beginning of 2006, firstly getting involved inside the Edubuntu family making real the possibility to have an Edubuntu Italian support and website area inside the current Italian LoCo Team.

When did you get involved with the MOTU team and how?

Right after joining the Ubuntu brigade I started checking out MOTU documentation, mainly packaging guide plus debian new maintainer’s guide, trying to understand every single new word and applying directly to a source package every lesson learned during developer’s world “travel”. After getting introduced and fascinated from an active community, I had to left the project for a while for some small problems, restarting everything on May 2007 with my first sponsored upload inside the archive. My packaging passion increased right after meeting Alexander Sack inside #ubuntu-mozillateam irc channel some days later, deciding to work directly with him as my mentor for both Debian and Ubuntu distributions.

What helped you learning packaging and learning how the Ubuntu teams work?

I started with Debian New Maintainer’s guide and Ubuntu’s packaging guide moving then to package my first applications learning from already-packaged software and asking if needed to Alexander improving and learning every time from him or from other developers a new Ubuntu Team lesson.

Favorite part of working with MOTU?

Introducing a fix making tons of users happy is one of the best things I appreciate of being a MOTU. Mentoring, sponsoring, helping out new contributors or students is something special as well.

Any advice for people wanting to help with MOTU?

I always suggest to start with a package a new contributor cares about personally, that’s useful to improve/fix the package itself during its maintenance.

Reading MOTU and Debian documentation is a great starting point as well to avoid any strange question on our MOTU irc channel.

What packages/areas of Universe are you most interested in?

I’m currently working on a vast area of packages, but I’ll try to focus on p2p (Peer-2-Peer) applications both for Universe and Main. I planned to create a MOTU-p2p team really soon including it inside the existing motu-torrent team, but it will take some months to organize everything up; contributors (testers/packagers) are currently missing.

Any Plans for Hardy Heron?

I’ll keep working on a large number of packages but as I said before I would like to focus on having an updated situation of p2p applications, introducing libtorrent-rasterbar and its related clients like btg or linkage. Creating a working team with interested contributors and developers will be the first step to work on.

Favorite quote?

“As for me, all I know is that I know nothing.” — Socrates

**What do you do in your other spare time?

** I love going around with my motorbike, listening good music, playing basketball and meeting up with friends around the city centre.

Pic of you, your work area, and/or your screen?

Il primo upload ha sempre un sapore speciale!

Andrea Veri — Thu, 13 Sep 2007 18:16:00 +0000

Questo pomeriggio ho eseguito il primo upload nell’archivio di Ubuntu e devo dire che ricevere poco dopo la mail di conferma e di successo dell’operazione è qualcosa di veramente speciale! Mi trovavo in una delle tante cartelle della mia Home e dopo aver controllato innumerevoli volte che tutto fosse corretto, ho lanciato dput e eseguito l’upload del pacchetto in questione, che, dopo il solito controllo della chiave di autorizzazione (GPG-Key) ha provveduto ad inviare i file necessari al server, mandandomi poco dopo l’email che tanto aspettavo: ACCEPTED. Dopo qualche minuto sono ripassato a controllare che i build fossero andati come aspettavo e…..

….e fortunatamente tutto era andato a buon fine! :-)

Una grande soddisfazione!

Andrea Veri — Wed, 12 Sep 2007 18:14:00 +0000

Dopo ben 3 mesi dalla mia presentazione al MOTU Council per la candidatura ufficiale a Master of the Universe è giunta ieri la lieta mail di Matt Zimmerman, chairman della tech board di Ubuntu e responsabile di ogni decisione finale per ogni candidatura presentata al consiglio. La mail che potete trovare nella mailing list di ubuntu-devel annuncia il mio ingresso come developer ufficiale di Ubuntu, con permessi di upload all’interno dell’archivio. È stata per me una grandissima soddisfazione vedere dopo 2 anni di lavoro l’icona di MOTU e di Ubuntu developer nel mio profilo di Launchpad.

Continuerò ora a lavorare e migliorare avendo come primo obiettivo l’ufficializzazione del gruppo “ubuntu-it-dev” nato da qualche mese nella nostra communità. Al prossimo post!

Creazione MOTU Torrent Team

Andrea Veri — Sun, 20 May 2007 19:40:00 +0000

Era da parecchio tempo ormai che l’idea di creare una squadra che si dedicasse solamente alla manutenzione di pacchetti legati al protocollo bittorrent mi percorreva la mente. Da intendere però che per manutenzione non si preveda solo la creazione di pacchetti non presenti negli archivi ma anche di tenere aggiornate le applicazioni già presenti nelle componenti main e universe tramite merges e syncs da debian, creando una collaborazione tra upstream e debian stesso per migliorare i programmi disponibili introducendo magari nuove realtà anche nel mondo delle librerie legate ai torrent. Proprio in questo settore mi sto impegnando ad aggiungere il supporto per la libreria creata da Rasterbar software, chiamata “libtorrent-rasterbar”, usata da numerossimi client che si sono diffusi negli ultimi tempi navigando nei meandri delle applicazioni torrent. Mi era caduto l’occhio soprattutto su linkage (http://zeflunk.googlepages.com/) che richiede tra le altre cose la libreria citata in precedenza, che verrà introdotta appena l’upstream sistemerà un errore legato al soname che coincide purtroppo con quella della libreria già esistenti nei database (libtorrent).
Non mi resta che linkarvi le pagine legate al team, così potrete osservare più da vicino come viene organizzato il lavoro tra i membri:

Launchpad: https://launchpad.net/~motu-torrent
Wiki: https://wiki.ubuntu.com/MOTU/Teams/Torrent
Bug reports: https://bugs.launchpad.net/~motu-torrent/+packagebugs

A presto

Andrea Veri's Blog

AWP - The Awesome Weekend Project

Table of Contents

Introduction

MVP

The learning resources

Preliminary steps

Configuration on ovn-control-plane

Configuration on sweetrevenge

Configuration on flumina

Connectivity test

Libvirt

Rust

The code

A quick peek at the web UI

License

The future

2024 GNOME Infrastructure Annual Review

Table of Contents

1. Introduction

2. Achievements

2.1. Major achievements

2.2. Minor achievements

2.3 Minor annoyances/bugs that were also fixed in 2024

2.3. Our brand new and renewed partnerships

Expressing my gratitude

GNOME Infrastructure migration to AWS

1. Some historical background

Recent challenges

Migration to AWS

2022 GNOME Infrastructure Annual Review

1. Introduction

2. Achievements

2.1. Major achievements

2.2. Minor achievements

2.3. Our brand new and renewed partnerships

3. Highlights

3.1. Openshift 4: architecture

3.2. Openshift 4: virtualization networking

3.3. Openshift 4: image builds

3.4. Openshift 4: cluster backups

3.5. GitLab on Openshift 4: setup

3.6. GitLab on Openshift 4: early days

3.7. GitLab on Openshift 4: logging

Future plans

Expressing my gratitude

GNOME Infrastructure updates

New Data Center

RHEL 8 and Ansible

Openshift

Special thanks

The GNOME Infrastructure is moving to Openshift

Architecture

SSL Termination

App migrations

Back from GUADEC 2018

GNOME Foundation hirings

cgit to GitLab

Future plans

Misc

Adding reCAPTCHA v2 support to Mailman

A childhood’s dream

Three years and counting

The GNOME Infrastructure Apprentice Program

Kerberos over HTTP: getting a TGT on a firewalled network

The GNOME Infrastructure’s FreeIPA move behind the scenes

The GNOME case

What were we looking for as a replacement for the current setup?

The Migration process – RFC2307 vs RFC2307bis

The Migration process – Extending the directory server with custom schemas

The Migration process – Own SSL certificates for HTTPD

The Migration process – Equivalent of authorized_keys’ “command”

The Migration process – Kerberos host Keytabs

The GNOME Infrastructure is now powered by FreeIPA!

Introduction

What has changed?

Where can I get my first login credentials?

Now that Mango is gone how can I request a new account?

I was used to have access to a specific service but I don’t anymore, what should I do?

What is missing still?