Jul 5, 2016
Six months since my latest blog post is definitely a lot and reminds me how difficult this year has been for me in many ways. Back in June 2015 I received a job proposal as a Systems and Network Engineer from a company located in Padova, a city in the north-east part of Italy which is around 150km (around 93 miles) away from my home-town. The offer looked very interesting and I went for it. The idea I originally had was to relocate there but the extremely high costs of rents (the city is well known in Italy as one of the best places to take University courses on several faculties) and the fact I really wanted to experiment on whether the job description was actually going to match the expectations I had, I opted to not relocate at all and travel daily to work by train (a total of 4 hours per day spent travelling).
I still recall one of my friends saying me “I am sure you won’t make it for more than two weeks” and how satisfying has been mentioning him a few days ago I did make it for roughly one year. Spending around 4 hours every day on a train hasn’t been that fun but the passion and love for what I’ve been doing has helped me overcoming the difficulties I encountered. At some point something I honestly was expecting arrived: my professional grow at the company was completely stuck. While I initially told myself the moment was surely going to not persist much longer, the opposite happened.
For a passionate, enthusiast and “hungry” person like I am that was one of the worst situations that could ever happened and it was definitely time for me to look at new opportunities. That’s where my childhood’s dream became reality: I will be joining the Platform Operations Team at Red Hat as a System Administrator starting from mid-July! Being part of a great family which cares about Open Source and its values makes me proud and I would really like to thank Red Hat for this incredible opportunity.
On a side note I will be in Brno between the 14th and the 16th of July, please drop me a note if you want to have a drink together!
Dec 2, 2015
It’s been a while since my last “what’s been happening behind the scenes” e-mail so I’m here to report on what has been happening within the GNOME Infrastructure, its future plans and my personal sensations about a challenge that started around three (3) years ago when Sriram Ramkrishna and Jeff Schroeder proposed my name as a possible candidate for coordinating the team that runs the systems behind the GNOME Project. All this followed by the official hiring achieved by Karen Sandler back in February 2013.
The GNOME Infrastructure has finally reached stability both in terms of reliability and uptime, we didn’t have any service disruption this and the past year and services have been running smoothly as they were expected to in a project like the one we are managing.
As many of you know service disruptions and a total lack of maintenance were very common before I joined back in 2013, I’m so glad the situation has dramatically changed and developers, users, passionates are now able to reach our websites, code repositories, build machines without experiencing slowness, downtimes or
unreachability. Additionally all these groups of people now have a reference point they can contact in case they need help when coping with the infrastructure they daily use. The ticketing system allows users to get in touch with the members of the Sysadmin Team and receive support right away within a very short period of time (Also thanks to Pagerduty, service the Foundation is kindly sponsoring)
Before moving ahead to the future plans I’d like to provide you a summary of what has been done during these roughly three years so you can get an idea of why I define the changes that happened to the infrastructure a complete revamp:
- Recycled several ancient machines migrating services off of them while consolidating them by placing all their configuration on our central configuration management platform ran by Puppet. This includes a grand total of 7 machines that were replaced by new hardware and extended warranties the Foundation kindly sponsored.
- We strenghten our websites security by introducing SSL certificates everywhere and recently replacing them with SHA2 certificates.
- We introduced several services such as Owncloud, the Commits Bot, the Pastebin, the Etherpad, Jabber, the GNOME Github mirror.
- We restructured the way we backup our machines also thanks to the Fedora Project sponsoring the disk space on their backup facility. The way we were used to handle backups drastically changed from early years where a magnetic tape facility was in charge of all the burden of archiving our data to today where a NetApp is used together with rdiff-backup.
- We upgraded Bugzilla to the latest release, a huge thanks goes to Krzesimir Nowak who kindly helped us building the migration tools.
- We introduced the GNOME Apprentice program open-sourcing our internal Puppet repository and cleansing it (shallow clones FTW!) from any sensitive information which now lives on a different repository with restricted access.
- We retired Mango and our OpenLDAP instance in favor of FreeIPA which allows users to modify their account information on their own without waiting for the Accounts Team to process the change.
- We documented how our internal tools are customized to play together making it easy for future Sysadmin Team members to learn how the infrastructure works and supersede existing members in case they aren’t able to keep up their position anymore.
- We started providing hosting to the GIMP and GTK projects which now completely rely on the GNOME Infrastructure. (DNS, email, websites and other services infrastructure hosting)
- We started providing hosting not only to the GIMP and GTK projects but to localized communities as well such as GNOME Hispano and GNOME Greece
- We configured proper monitoring for all the hosted services thanks to Nagios and Check-MK
- We migrated the IRC network to a newer ircd with proper IRC services (Nickserv, Chanserv) in place.
- We made sure each machine had a configured management (mgmt) and KVM interface for direct remote access to the bare metal machine itself, its hardware status and all the operations related to it. (hard reset, reboot, shutdown etc.)
- We upgraded MoinMoin to the latest release and made a substantial cleanup of old accounts, pages marked as spam and trashed pages.
- We deployed DNSSEC for several domains we manage including gnome.org, guadec.es, gnomehispano.es, guadec.org, gtk.org and gimp.org
- We introduced an account de-activation policy which comes into play when a contributor not committing to any of the hosted repositories at git.gnome.org since two years is caught by the script. The account in question is marked as inactive and the gnomecvs (from the old cvs days) and ftpadmin groups are removed.
- We planned mass reboots of all the machines roughly every month for properly applying security and kernel updates.
- We introduced Mirrorbrain (MB), the mirroring service serving GNOME and related modules tarballs and software all over the world. Before introducing MB GNOME had several mirrors located in all the main continents and at the same time a very low amount of users making good use of them. Many organizations and companies behind these mirrors decided to not host GNOME sources anymore as the statistics of usage were very poor and preferred providing the same service to projects that really had a demand for these resources. MB solved all this allowing a proper redirect to the closest mirror (through mod_geoip) and making sure the sources checksum matched across all the mirrors and against the original tarball uploaded by a GNOME maintainer and hosted at master.gnome.org.
I can keep the list going for dozens of other accomplished tasks but I’m sure many of you are now more interested in what the future plans actually are in terms of where the GNOME Infrastructure should be in the next couple of years.
One of the main topics we’ve been discussing will be migrating our Git infrastructure away from cgit (which is mainly serving as a code browsing tool) to a more complete platform that is surely going to include a code review tool of some sort. (Gerrit, Gitlab, Phabricator)
Another topic would be migrating our mailing lists to Mailman 3 / Hyperkitty. This also means we definitely need a staging infrastructure in place for testing these kind of transitions ideally bound to a separate Puppet / Ansible repository or branch. Having a different repository for testing purposes will also mean helping apprentices to test their changes directly on a live system and not on their personal computer which might be running a different OS / set of tools than the ones we run on the GNOME Infrastructure.
What I also aim would be seeing GNOME Accounts being the only authentication resource in use within the whole GNOME Infrastructure. That means one should be able to login to a specific service with the same username / password in use on the other hosted services. That’s been on my todo list for a while already and it’s probably time to push it forward together with Patrick Uiterwijk, responsible of Ipsilon‘s development at Red Hat and GNOME Sysadmin.
While these are the top priority items we are soon receiving new hardware (plus extended warranty renewals for two out of the three machines that had their warranty renewed a while back) and migrating some of the VMs off from the current set of machines to the new boxes is definitely another task I’d be willing to look at in the next couple of months (one machine (ns-master.gnome.org) is being decommissioned giving me a chance to migrate away from BIND into NSD).
The GNOME Infrastructure is evolving and it’s crucial to have someone maintaining it. On this side I’m bringing to your attention the fact the assigned Sysadmin funds are running out as reported on the Board minutes from the 27th of October. On this side Jeff Fortin started looking for possible sponsors and came up with the idea of making a brochure with a set of accomplished tasks that couldn’t have been possible without the Sysadmin fundraising campaign launched by Stormy Peters back in June 2010 being a success. The Board is well aware of the importance of having someone looking at the infrastructure that runs the GNOME Project and is making sure the brochure will be properly reviewed and published.
And now some stats taken from the Puppet Git Repository:
$ cd /git/GNOME/puppet && git shortlog -ns
3520 Andrea Veri
506 Olav Vitters
338 Owen W. Taylor
239 Patrick Uiterwijk
112 Jeff Schroeder
71 Christer Edwards
4 Daniel Mustieles
4 Matanya Moses
3 Tobias Mueller
2 John Carr
2 Ray Wang
1 Daniel Mustieles García
1 Peter Baumgarten
and from the Request Tracker database (52388 being my assigned ID):
mysql> select count(*) from Tickets where LastUpdatedBy = '52388';
+----------+
| count(*) |
+----------+
| 3613 |
+----------+
1 row in set (0.01 sec)
mysql> select count(*) from Tickets where LastUpdatedBy = '52388' and Status = 'Resolved';
+----------+
| count(*) |
+----------+
| 1596 |
+----------+
1 row in set (0.03 sec)
It’s been a long run which made me proud, for the things I learnt, for the tasks I’ve been able to accomplish, for the great support the GNOME community gave me all the time and most of all for the same fact of being part of the team responsible of the systems hosting the GNOME Project. Thank you GNOME community for your continued and never ending backing, we daily work to improve how the services we host are delivered to you and the support we receive back is fundamental for our passion and enthusiasm to remain high!
Jan 28, 2015
Many times it happened seeing someone joining the #sysadmin IRC channel requesting participation to the team after having spent around 5 minutes trying to explain what the skills and the knowledge were and why this person felt it was the right figure for the position. And it was always very disappointing for me having to reject all these requests as we just didn’t have the infrastructure in place to let new people join the rest of the team with limited privileges.
With the introduction of FreeIPA, more fine-grained ACLs (and hiera-eyaml-gpg for securing tokens, secrets, passwords out of Puppet itself) we are so glad to announce the launch of the “GNOME Infrastructure Apprentice Program” (from now till the end of the post just “Program”). If you are familiar with the Fedora Infrastructure and how it works you might know what this is about already. If you don’t please read further ahead.
The Program will allow apprentices to join the Sysadmin Team with a limited set of privileges which mainly consist in being able to access the Puppet repository and all the stored configuration files that run the machines powering the GNOME Infrastructure every day. Once approved to the Program apprentices will be able to submit patches for review to the team and finally see their work merged on the production environment if the proposed changes matched the expectations and addressed comments.
While the Program is open to everyone to join, we have some prerequisites in place. The interested person should be:
- Part of an existing FOSS community
- Familiar with how a FOSS Project works behind the scenes
- Familiar with popular tools like Puppet, Git
- Familiar with RHEL as the OS of choice
- Familiar with popular Sysadmin tools, softwares and procedures
- Eager to learn new things, make constructive discussions with a team, provide feedback and new ideas
If you feel like having all the needed prerequisites and would be willing to join follow these steps:
- Subscribe to the gnome-infrastructure and infrastructure-announce mailing lists
- Join the #sysadmin IRC channel on irc.gnome.org
- Send a presentation e-mail to the gnome-infrastructure mailing list stating who you are, what your past experiences and plans are as an Apprentice
- Once the presentation has been sent an existing Sysadmin Team member will evaluate your application and follow-up with you introducing you to the Program
More information about the Program is available here.
Oct 24, 2014
One of the benefits I originally wanted to bring with the FreeIPA move to GNOME contributors was the introduction of an additional authentication system to connect to to the services hosted on the GNOME Infrastructure. The authentication system that comes with the FreeIPA bundle that I had in mind was Kerberos. Users willing to use Kerberos as their preferred authentication system would just be required to get a TGT (Ticket-Granting Ticket) from the KDC (Key Distribution Center) through the kinit command. Once done authenticating to the services currently supporting Kerberos will be as easy as pointing a configured browser (Google for how to configure your browser to use Krb logins) to account.gnome.org without being prompted for entering the usual username / password combination or pushing to git without using the public-private key mechanism. That theoretically means you won’t be required to use a SSH key for loggin in to any of the GNOME services at all as entering your password to the kinit password prompt will be enough (for at least 24 hours as that’s the life of the TGT itself on our setup) for doing all you were used to do before the Kerberos support introduction.
The issue we faced at first was the underlying networking infrastructure firewalling all Kerberos ports blocking the use of kinit itself which kept timing out reaching port 88. A few days later I was contacted by RH’s developer Nathaniel McCallum who worked out a way to bypass this restriction by creating a KDC proxy that accepts requests from port 443 and proxies them to the internal KDC running on port 88. With the recent Kerberos release (released on October 15th, 2014 and following the MS-KKDCP protocol) a patched kinit allows users to retrieve their TGTs directly from the HTTPS proxy completely bypassing the need for port 88 to stay open on the firewall. The GNOME Infrastructure now runs the KDC Proxy and we’re glad to announce Kerberos authentications are working as expected on the hosted services.
If you are facing the same problem and you are curious to know more about the setup, here they come all the details:
On the KDC:
- No changes are needed on the KDC itself, just make sure to install the python-kdcproxy package which is available for RHEL 7, HERE.
- Tweak your vhost accordingly by following the provided documentation.
On the client:
- Install the krb5-workstation package, make sure it’s at least version 1.12.2-9 as that’s the release which had the additional features we are talking about backported. Right now it’s only available for Fedora 21.
- Adjust /etc/krb5.conf accordingly and finally get a TGT through kinit $userid@GNOME.ORG.
[realms]
GNOME.ORG = {
kdc = https://account.gnome.org/KdcProxy
kpasswd_server = https://account.gnome.org/KdcProxy
}
That should be all for today!
Oct 12, 2014
A few days ago I wrote about the GNOME Infrastructure moving to FreeIPA, the post was mainly an announcement to the relevant involved parties with many informative details for contributors to properly migrate their account details off from the old authentication system to the new one. Today’s post is a follow-up to that announcement but it’s going to take into account the reasons about our choice to migrate to FreeIPA, what we found interesting and compelling about the software and why we think more projects (them being either smaller or bigger) should migrate to it. Additionally I’ll provide some details about how I performed the migration from our previous OpenLDAP setup with a step-by-step guide that will hopefully help more people to migrate the infrastructure they manage themselves.
The GNOME case
It’s very clear to everyone an infrastructure should reflect the needs of its user base, in the case of GNOME a multitude between developers, translators, documenters and between them a very good number of Foundation members, contributors that have proven their non-trivial contributions and have received the status of members of the GNOME Foundation with all the relevant benefits connected to it.
The situation we had before was very tricky, LDAP accounts were managed through our LDAP istance while Foundation members were being stored on a MySQL database with many of the tables being related to the yearly Board of Director’s elections and one specifically meant to store all the information from each of the members. One of the available fields on that table was defined as ‘userid’ and was supposed to store the LDAP ‘uid’ field the Membership Committee member processing a certain application had to update when accepting the application. This procedure had two issues:
- Membership Committee members had no access to LDAP information
- No checks were being run on the web UI to verify the ‘userid’ field was populated correctly taking in multiple inconsistencies between LDAP and the MySQL database
In addition to the above Mango (the software that helped the GNOME administrative teams to manage the user data for multiple years had no maintainer, no commits on its core since 2008 and several limitations)
What were we looking for as a replacement for the current setup?
It was very obvious to me we would have had to look around for possible replacements to Mango. What we were aiming for was a software with the following characteristics:
- It had to come with a pre-built web UI providing a wide view on several LDAP fields
- The web UI had to be extensible in some form as we had some custom LDAP schemas we wanted users to see and modify
- The sofware had to be actively developed and responsive to eventual security reports (given the high security impact a breach on LDAP could take in)
FreeIPA clearly matched all our expectations on all the above points.
The Migration process – RFC2307 vs RFC2307bis
Our previous OpenLDAP setup was following RFC 2307, which means that above all the other available LDAP attributes (listed on the RFC under point 2.2) group’s membership was being represented through the ‘memberUid’ attribute. An example:
cn=foundation,cn=groups,cn=compat,dc=gnome,dc=org
objectClass: posixGroup
objectClass: top
gidNumber: 524
memberUid: foo
memberUid: bar
memberUid: foobar
As you can each of the members of the group ‘foundation’ are represented using the ‘memberUid’ attribute followed by the ‘uid’ of the user itself. FreeIPA does not make directly use of RFC2307 for its trees, but RFC2307bis instead. (RFC2307bis was not published as a RFC by the IETF as the author didn’t decide to pursue it nor the companies (HP, Sun) that then adopted it)
RFC2307bis uses a different attribute to represent group’s membership, it being ‘member’. Another example:
cn=foundation,cn=groups,cn=accounts,dc=gnome,dc=org
objectClass: posixGroup
objectClass: top
gidNumber: 524
member: uid=foo,cn=users,cn=accounts,dc=gnome,dc=org
member: uid=bar,cn=users,cn=accounts,dc=gnome,dc=org
member: uid=foobar,cn=users,cn=accounts,dc=gnome,dc=org
As you can see the DN representing the group ‘foundation’ differs between the two examples. That is why FreeIPA comes with a Compatibility plugin (cn=compat) which automatically creates RFC2307-compliant trees and entries whenever an append / modify / delete operation happens on any of the hosted RFC2307bis-compliant trees. What’s the point of doing this when we could just stick with RFC2307bis trees and go with it? As the plugin name points out the Compatibility plugin is there to prevent breakages between the directory server and any of the clients or softwares out there still retrieving information and data by using the ‘memberUid’ attribute as specified on RFC2307.
FreeIPA migration tools (ipa migrate-ds) do come with a ‘–schema’ flag you can use to specify what attribute the istance you are migrating from was following (values are RFC2307 and RFC2307bis as you may have guessed already), in the case of GNOME the complete command we ran (after installing all the relevant tools through ‘ipa-server-install’ and copying the custom schemas under /etc/dirsrv/slapd-$ISTANCE-NAME/schema) was:
ipa migrate-ds --bind-dn=cn=Manager,dc=gnome,dc=org --user-container=ou=people,dc=gnome,dc=org --group-container=ou=groups,dc=gnome,dc=org --group-objectclass=posixGroup ldap://internal-IP:389 --schema=RFC2307
Please note that before running the command you should make sure custom schemas you had on the istance you are migrating from are available to the directory server you are migrating your tree to.
More information on the migration process from an existing OpenLDAP istance can be found HERE.
The Migration process – Extending the directory server with custom schemas
One of the other challenges we had to face has been extending the available LDAP schemas to include Foundation membership attributes. This operation requires the following changes:
- Build the LDIF (that will include two new custom fields: FirstAdded and LastRenewedOn)
- Adding the LDIF in place on /etc/dirsrv/slapd-$ISTANCE-NAME/schema
- Extend the web UI to include the new attributes
I won’t be explaining how to build a LDIF on this post but I’m however pasting the schema I made to help you getting an idea:
attributeTypes: ( 1.3.6.1.4.1.3319.8.2 NAME 'LastRenewedOn' SINGLE-VALUE EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch DESC 'Last renewed on date' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )
attributeTypes: ( 1.3.6.1.4.1.3319.8.3 NAME 'FirstAdded' SINGLE-VALUE EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch DESC 'First added date' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )
objectClasses: ( 1.3.6.1.4.1.3319.8.1 NAME 'FoundationFields' AUXILIARY MAY ( LastRenewedOn $ FirstAdded ) )
After copying the schema in place and restarting the directory server, extend the web UI:
on /usr/lib/python2.7/site-packages/ipalib/plugins/foundation.py:
from ipalib.plugins import user
from ipalib.parameters import Str
from ipalib import _
from time import strftime
import re
def validate_date(ugettext, value):
if not re.match("^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])$", value):
return _("The entered date is wrong, please make sure it matches the YYYY-MM-DD syntax")
user.user.takes_params = user.user.takes_params + (
Str('firstadded?', validate_date,
cli_name='firstadded',
label=_('First Added date'),
),
Str('lastrenewedon?', validate_date,
cli_name='lastrenewedon',
label=_('Last Renewed on date'),
),
)
on /usr/share/ipa/ui/js/plugins/foundation/foundation.js:
define([
'freeipa/phases',
'freeipa/user'],
function(phases, user_mod) {
// helper function
function get_item(array, attr, value) {
for (var i=0,l=array.length; i<l; i++) {
if (array[i][attr] === value) return array[i];
}
return null;
}
var foundation_plugin = {};
foundation_plugin.add_foundation_fields = function() {
var facet = get_item(user_mod.entity_spec.facets, '$type', 'details');
var section = get_item(facet.sections, 'name', 'identity');
section.fields.push({
name: 'firstadded',
label: 'Foundation Member since'
});
section.fields.push({
name: 'lastrenewedon',
label: 'Last Renewed on date'
});
section.fields.push({
name: 'description',
label: 'Previous account changes'
});
return true;
};
phases.on('customization', foundation_plugin.add_foundation_fields);
return foundation_plugin;
});
Once done, restart the web server. The next step would be migrating all the FirstAdded and LastRenewedOn attributes off from MySQL into LDAP now that our custom schema has been injected.
The relevant MySQL fields were following the YYYY-MM-DD syntax to store the dates and a little Python script to read from MySQL and populate the LDAP attributes was then made. If interested or you are in a similar situation you can find it HERE.
The Migration process – Own SSL certificates for HTTPD
As you may be aware of FreeIPA comes with its own certificate tools (powered by Certmonger), that means a CA is created (during the ipa-server-install run) and certificates for the various services you provide are then created and signed with it. This is definitely great and removes the burden to maintain an underlying self-hosted PKI infrastructure. At the same time this seems to be a problem for publicly-facing web services as browsers will start complaining they don’t trust the CA that signed the certificate the website you are trying to reach is using.
The problem is not really a problem as you can specify what certificate HTTPD should be using for displaying FreeIPA’s web UI. The procedure is simple and involves the NSS database at /etc/httpd/alias:
certutil -d /etc/httpd/alias/ -A -n "StartSSL CA" -t CT,C, -a -i sub.class2.server.ca.pem
certutil -d /etc/pki/nssdb -A -n "StartSSL CA" -t CT,C, -a -i sub.class2.server.ca.pem
openssl pkcs12 -inkey freeipa.example.org.key -in freeipa.example.org.crt -export -out freeipa.example.org.p12 -nodes -name 'HTTPD-Server-Certificate'
pk12util -i freeipa.example.org.p12 -d /etc/httpd/alias/
Once done, update /etc/httpd/conf.d/nss.conf with the correct NSSNickname value. (which should match the one you entered after ‘-name’ on the third of the above commands)
The Migration process – Equivalent of authorized_keys’ “command”
At GNOME we do run several services that require users to login to specific machines and run a command. At the same time and for security purposes we don’t want all the users to reach a shell. Originally we were making use of SSH’s authorized_keys file to specify the “command” these users should have been restricted to. FreeIPA handles Public Key authentications differently (through the sss_ssh_authorizedkeys binary) which means we had to find an alternative way to restrict groups to only a specific command. SSH’s ForceCommand came in help, an example given a group called ‘foo’:
Match Group foo,!bar
X11Forwarding no
PermitTunnel no
ForceCommand /home/admin/bin/reset-my-password.py
The above Match Group will be applied to all the users of the ‘foo’ group except the ones that are also part of the ‘bar’ group. If you are interested in the reset-my-password.py script which resets a certain user password and sends a temporary one to the registered email address (by checking the mail LDAP attr) for the user, click HERE.
The Migration process – Kerberos host Keytabs
Here, at GNOME, we still have one or two RHEL 5 hosts hanging around and SSSD reported a failure when trying to authenticate with the given Keytab (generated with RHEL 7 default values) to the KDC running (as you may have guessed) RHEL 7. The issue is simple as RHEL 5 does not support many of the encryption types which the Keytab was being encrypted with. Apparently the only currently supported Keytab encryption type on a RHEL 5 machine is rc4-hmac. Creating a Keytab on the KDC accordingly can be done this way:
ipa-getkeytab -s server.example.org -p host/client.example.org -e rc4-hmac -k /root/keytabs/client.example.org.keytab
That should be all for today, I’ll make sure to update this post with further details or answers to possible comments.