• Our main LDAP istance was moved from a very ancient machine (which unfortunately died with a broken disk a few weeks ago) to a newer box that currently contains several other admin tools like Mango and Daily Reports. (a little script written by Owen Taylor for creating and storing several reports mainly related to backups and SSL certificates expiration dates) In addition to migrating our LDAP master to a newer machine, we did configure and setup replication to an LDAP slave to share a bit the load and most of all to link all the external (machines outside the RH’s internal network) machines to it.

• A lot of efforts have been spent in the so-called “Puppet-ization” (Puppet allows you to reproduce a complete environment with just a few commands, it’s very very handy in the case of host’s migrations) and several new modules are now stored into our internal Puppet repository. Specifically all the Iptables rules are currently managed in a centralized way, each node has its own rules and policies, finally there’s no need to ssh into each machine to retrieve the information we need for a specific firewall. In addition to the Iptables class, also Cobbler, Owncloud, Jabberd, Denyhosts and several other modules have been properly configured and currently reside in Puppet.

• Another item that had top priority on the list was setting up another “webapps” virtual machine to migrate several services from one of the existing ancient machines to it. I can finally tell that the GNOME Infrastructure has get rid of all the old machines, all the services have been migrated to newer machines and most of all all the services are currently being served through SSL. (git, planet, www.gnome.org, l10n, guadec.org, bugzilla, blogs, developer, help, people, news etc.) In regard of SSL and Bugzilla, we’ve configured our Bugzilla istance to serve attachments through a secondary domain which will look like: https://bug-id.bugzilla-attachments.gnome.org, this to prevent cross-site scripting attacks in a better way than what we did before.

• We’ve also spent some time working on our Nagios istance hosted at https://nagios.gnome.org. We’ve improved it dramatically by adding several new checks and covering all the services we currently take care of but that’s not all. Event Handlers have been setup to help us addressing problems right after they occur on our web servers. The Nagios event handlers are currently configured to read the status of a specific Nagios service and in the case the status is set to CRITICAL, they restart httpd once, which is usually enough in the case of random Apache’s timeouts. But that’s again not all. A public view for Nagios is now ready, every single GNOME contributor and developer should be able to check the current status of all the services we maintain just by loggin in with the username “anonymous” and the password “anonymous” at nagios.gnome.org.

• Our wiki was upgraded to the latest MoinMoin release, at the moment of writing, version 1.9.7. This release introduces stronger password hashes, please make sure to update your password as soon as you can to strenghten the security of your account. It was also clear that live.gnome.org was behaving a bit sluggish lately, we spent some time cleaning up spammers, old and deleted pages and things started flushing way better. More details about the cleanup can be found at https://mail.gnome.org/archives/foundation-list/2013-May/msg00098.html. (we cleaned up around 23000 spammers!)

• The most exciting things I usually love to announce are new services. While I always prefer keeping the number of maintained services as low as possible it was time for the GNOME Infrastructure to broaden its horizons satisfying the requests coming from the community and the developers involved into the project. I won’t spend any more word about this since I’m sure you are all waiting for me to list the new services, so here they are:

1.  A completely new Jabber service hosted at jabber.gnome.org and accessible by all the GNOME Foundation members requesting access to it. More details about it can be found at https://live.gnome.org/Sysadmin/Jabber.
2. GNOME is extensively using IRC as its main communication tool, thus we’ve improved our Services IRC Bot to use a plugin called MeetBot. Having a meeting and storing the logs in a public web server is currently possible with a really minor effort of learning a few commands to administer the plugin correctly. If you are going to have a meeting and you want to make use of MeetBot, make sure that Services is there, and give a look at http://meetbot.gnome.org/Manual.html.
3. Do you want to be always up-to-date with the status of the GNOME Infrastructure? and are you actually wondering what’s the best way to do so? if yes, you should probably have a look at http://status.gnome.org. This service makes use of Fedora-Status by Patrick Uiterwijk and allows the GNOME Sysadmin Team to let everyone know whether there is a problem with any of the services listed in the page. This service, together with the public view for Nagios and the brand new infrastructure-announce@gnome.org mailing list will definitely help everyone finding out what’s going on, where and how long it’ll take for the issue to be fixed.
4. It took me a lot of pain having the KGB IRC Collaboration bot packaged into the EPEL repositories but I finally managed to set it up on the GNOME Infrastructure. KGB has become very handy since the time Cia.vc closed its hosting and it’s available for anyone requesting access to it at irc.gnome.org. If you are looking for Git commit notifications of a specific module directly on your IRC channel, this is what you want :-)
5. An Owncloud instance is also available, more reading about what are the requirements for requesting access to it at the following link.
6. An Etherpad istance is also available at https://etherpad.gnome.org to all the GNOME Teams that need it! Please drop me an e-mail at <av at gnome dot org> if you are interested. (Pad’s creation is currently disabled for preventing spam)
7. Build.gnome.org has been revived and it’s currently hosting an OSTree istance. GNOME Daily images are costantly being generated, interested in testing one the those images? give this article a look.

That’s all for now! See you all at GUADEC and thanks everyone for all the hints, suggestions and mails you’ve been sending me in the past months! And a special thanks to Ekaterina Gerasimova for taking the time to brainstorm with me suggesting new features and improvements over the GNOME Infrastructure!

On the LDAP (SLAPD) server

Re-create *.db files

mkdir /etc/openldap/certs
modutil -create -dbdir /etc/openldap/certs

Setup a CA Certificate

certutil -d /etc/openldap/certs -A -n “My CA Certificate” -t TCu,Cu,Tuw -a -i /etc/openldap/cacerts/ca.pem
where ca.pem should be your CA’s certificate file.

Remove the password from the Database

modutil -dbdir /etc/openldap/certs -changepw ‘NSS Certificate DB’

Creates the .p12 file and imports it on the Database

openssl pkcs12 -inkey domain.org.key -in domain.org.crt -export -out domain.org.p12 -nodes -name ‘LDAP-Certificate’
pk12util -i domain.org.p12 -d /etc/openldap/certs

where domain.org.key and domain.org.crt are the names of the certificates you previously created at your CA’s website.

List all the certificates on the database and make sure all the informations are correct

certutil -d /etc/openldap/certs -L

Configure /etc/openldap/slapd.conf and make sure the TLSCACertificatePath points to your Mozilla NSS database

TLSCACertificateFile /etc/openldap/cacerts/ca.pem
TLSCACertificatePath /etc/openldap/certs/
TLSCertificateFile LDAP-Certificate

Additional commands

Modify the trust flags if necessary

certutil -d /etc/openldap/certs -M -n “My CA Certificate” -t “TCu,Cu,Tuw”

Delete a certificate from the database

certutil -d /etc/openldap/certs -D -n “My LDAP Certificate”

On the clients (nslcd uses ldap.conf while sssd uses /etc/sssd/sssd.conf)

On /etc/openldap/ldap.conf

BASE dc=domain,dc=org
URI ldaps://ldap.domain.org

TLS_REQCERT demand
TLS_CACERT /etc/openldap/cacerts/ca.pem

On /etc/sssd/sssd.conf

ldap_tls_cacert = /etc/openldap/cacerts/ca.pem
ldap_tls_reqcert = demand
ldap_uri = ldaps://ldap.domain.org

How to test the whole setup

ldapsearch -x -b 'dc=domain,dc=org' -D "cn=Manager,dc=domain,dc=org" '(objectclass=*)' -H ldaps://ldap.domain.org -W -v

Troubleshooting

If anything goes wrong you can run SLAPD with the following args for its debug mode:

/usr/sbin/slapd -d 256 -f /etc/openldap/slapd.conf -h “ldaps:/// ldap:///”

Possible errors: 

If you happen to see an error similar to this one: “TLS error -8049:Unrecognized Object Identifier.“, try running ldapsearch with its debug mode this way:

ldapsearch -d 1 -x -ZZ -H ldap://ldap.domain.org

Make also sure that the FQDN you are trying to connect to is listed on the trusted FQDN’s list of your domain.org.crt.

Update: as SSSD’s developer Stephen Gallagher correctly pointed out on the comments using ldap_tls_reqcert = allow isn’t a best practice since it may take in Man in the Midle Attacks, adjusting the how to to match his suggestions.

As you may understand many Sysadmin’s tasks are not perceived at all by users especially the ones related to the so-called “Puppet-ization” which refers to the process of creating / modifying / improving our internal Puppet repository. A lot of work has been done on that side and several new modules have been added, specifically Cobbler, Amavisd, SpamAssassin, ClamAV, Bind, Nagios / Check_MK (enabling Apache eventhandlers for automatic restart of faulty httpd processes), Apache.

Another top priority item was migrating some of our services off to the old physical machines to virtual machines I did setup earlier. The machines that are now recycled are the following: (Two more are missing on the list, specifically window (which still hosts art.gnome.org, people.gnome.org and projects.gnome.org due to be migrated to another host in the next weeks) and label. (which still hosts our Jabberd, an interesting discussion on its future is currently ongoing on Foundation-list)

1. menubar, our old Postfix host served the GNOME Foundation since 2004 and processed millions of e-mails from and to @gnome.org addresses.
2. container, our old main NFS node was serving the GNOME Foundation since 2003, it hosted our mail archives, our FTP archives and all the /home/users/* directories.
3. button hosted many services (MySQL databases, LDAP, Mango) and served the Foundation since 2004, a faulty hardware took it down on January 2012.

And if you ever wanted to see how menubar, container and button look like, I have two photos for you with the machines being pulled out the GNOME rack: (click on them for seeing the photos at full size)

Some of the things you may have perceived directly on your skin should be the following:

1. Our live.gnome.org istance has been upgraded to the latest MoinMoin stable release, 1.9.6.
2. The Services bot has been added to the GIMPNET network and currently manages all GNOME channels, it currently acts as a Nickserv, Chanserv. More information about how you can register your nickname and gain the needed ACLs at the following wiki page.
3. Several GNOME services and domains are now covered by SSL as you may have noticed on planet.gnome.org, news.gnome.org, blogs.gnome.org, l10n.gnome.org, git.gnome.org, help.gnome.org, developer.gnome.org.
4. Re-design of our Mailman archives as you can see at https://mail.gnome.org/archives/foundation-list. A big “thank you” goes to Olav Vitters for taking the time to rebuild our “archive” script from Perl to Python. About the Mailman topic, someone proposed me the use of HyperKitty, that’s something we will evaluate in the next coming months but I find it a very interesting alternative to the current mail archiving.

What should you expect next?

1. Bugzilla will be moved to another virtual machine and will be upgraded to the latest release.
2. An Owncloud istance will be setup for all the GNOME Foundation members and GNOME Teams that will need access to it.
3. A discussion will be started for setting up a Gitorious istance on the GNOME Infrastructure.
4. A long-term item will be rewriting Mango in Django and adding several other features to it than the ones it has now. (ideally voting for Board elections, logins for managing your LDAP information such as your @gnome.org’s alias forward, shutdown of old an unused accounts after a certain period of time, automatic @gnome.org’s alias creation after the “Foundation Membership” flag is selected on LDAP, etc.)

Thanks a lot for all the mails I’ve received during these weeks containing reports and suggestions about how we should improve our Infrastructure! Please stay tuned, a lot more news are yet to come!

Hurricane Electrics IPv6 Certification

Willing to test the latest revision of the Internet Protocol on your Debian, Ubuntu, Fedora machines? Here’s how:

1. Register yourself at Hurricane Electrics by visiting tunnelbroker.net.

2. Create a new tunnel and make sure to use your public IP address as your IPv4 Endpoint.

3. Write down the relevant details of your tunnel, specifically:

1. Server IPv6 Address: 2001:470:1f0a:a6f::1 /64
2. Server IPv4 Address: 216.66.84.46 (this actually depends on which server did you choose on the previous step)
3. Client IPv6 Address: 2001:470:1f0a:a6f::2/64

4. Create a little script that will update your IPv4 tunnel endpoint every time your internet IP changes. (this step is not needed if you have an internet connection with a static IP):

Your Tunnel configuration details at tunnelbroker.net

#!/bin/bash
USERNAME=yourHEUsername
PASSWORD=yourHEPassword
TUNNELID=yourHETunnelID
GET "https://$USERNAME:$PASSWORD@ipv4.tunnelbroker.net/ipv4_end.php?tid=$TUNNELID"

5. Create the networking configuration files on your computer:

Debian / Ubuntu, on the /etc/network/interfaces file:

auto he-ipv6
iface he-ipv6 inet6 v4tunnel
address 2001:470:1f0a:a6f::2
netmask 64
endpoint 216.66.80.30
local 192.168.X.X (Your PC's LAN IP address)
ttl 255
gateway 2001:470:1f0a:a6f::1
pre-up /home/user/bin/update_tunnel.sh

Fedora, on the  /etc/sysconfig/network-scripts/ifcfg-he-ipv6 file:

DEVICE=he-ipv6
TYPE=sit
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6TUNNELIPV4="216.66.80.30"
IPV6TUNNELIPV4LOCAL="192.168.X.X" (Your PC's LAN IP address)
IPV6ADDR="2001:470:1f0a:a6f::2/64"

and on the /etc/sysconfig/network file, add:

NETWORKING_IPV6=yes
IPV6_DEFAULTGW="2001:470:1f0a:a6f::1"
IPV6_DEFAULTDEV="he-ipv6"

You can then set up a little /sbin/ifup-pre-local script to update the IPv4 tunnel endpoint when your dynamic IP changes or simply add the script on the /etc/cron.daily directory and have it executed when you turn up your computer.

6. Change the DNS servers on /etc/resolv.conf:

OpenDNS:

A sample image taken from ipv6-test.com.

nameserver 2620:0:ccc::2
nameserver 2620:0:ccd::2

Google DNS:

nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

7. Restart your network and enjoy IPv6!

8. If you want to know more about IPv6 take some time for the HE Certification program, you will learn a lot and eventually win a sponsored t-shirt, I just finished mine :-)

EDIT: Be aware of the fact that as soon as the tunnel is up, your computer will be exposed to to the internet without any kind of firewall (the tunnel sets up a direct connection to the internet, even bypassing your router’s firewall), you can secure your machine by using ip6tables. Thanks Michael Zanetti for pointing this out!

But what Cloud Computing really means? it took several years and more than fifteen drafts to the National Institute of Standards and Technology‘s (NIST) to find a definition. The final accepted proposal:

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

The above definition requires a few more clarifications specifically when it comes to understand where should we focus on while checking for a Cloud Computing solution. A few key points:

1. On-demand self-service: every consumer will be able to unilaterally provision multiple computing capabilities like server time, storage, bandwidth, dedicated RAM or CPU without requiring any sort of human interaction from their respective Cloud providers.
2. Rapid elasticity and scalability: all the computing capabilities outlined above can be elastically provisioned and released depending on how much demand my company will have in a specific period of time. Suppose the X company is launching a new product today and it expects a very large number of customers. The X company will add more resources to their  Cloud for the very first days (where they suppose the load to be very high) and then it’ll scale the resources back as they were before. Elasticity and scalability permit the X company to improve and enhance their infrastructure when they need it with an huge saving in monetary terms.
3. Broad network access: capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).
4. Measured service: Cloud systems allow maximum transparency between the provider and the consumer, the usage of all the resources is monitored, controlled, and reported. The consumer knows how much will spend, when and in how long.
5. Resource pooling: each provider’s computing resources are pooled to serve multiple consumers at the same time. The consumer has no control or knownledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
6. Resources price: when buying a Cloud service make sure the cost for two units of RAM, storage, CPU, bandwidth, server time is exactly the double of the price of one unit of the same capability. An example, if a provider offers you one hour of bandwitdh for 1 Euro, the price of  two hours will have to be 2 Euros.

The Cloud Computing technology is reasonably the future but can we trust Cloud providers? Are we sure that no one will ever have access to our files except us? and what about governments interested in acquiring a specific customer data hosted on the Cloud?

I always suggest to read deeply both the Privacy Policy and Terms of Use of a certain service before signing in especially when it comes to choose a Cloud storage provider. Many providers have the same aspect, they seem to provide the same resources, the same amount of storage for the same price but legally they may present different problems, and that is the case of Spideroak vs Dropbox. Quoting from the Dropbox’s Privacy Policy:

Compliance with Laws and Law Enforcement Requests; Protection of DropBox’s Rights. We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of DropBox or its users; or (d) to protect Dropbox’s property rights. If we provide your Dropbox files to a law enforcement agency as set forth above, we will remove Dropbox’s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.

It’s evident that Dropbox employees can access your data or be forced by legal process to turn over your data unencrypted. On the other side, Spideroak on its latest update to its Privacy Policy states that data stored on their  Cloud  is encrypted and inaccessible without user’s key, which is stored locally on user’s computers.

And what about the latest research paper, titled ”Cloud Computing in Higher Education and Research Institutions and the USA Patriot Act“ written by the legal experts of the University of Amsterdam’s Institute for Information Law stating the anti-terror Patriot Act could be theoretically used by U.S. law enforcement to bypass strict European privacy laws to acquire citizen data within the European Union without their consensus?

The only requirement for the data acquisition is the provider being an U.S company or an European company conducting systematic business in the U.S. For example an Italian company storing their documents (protected by the European privacy laws and under the general Italian jurisdiction) on a provider based in Europe but conducting systematic business in the United States, could be forced by U.S. law enforcement to transfer data to the U.S. territory for inspection by law enforcement agencies.

Does someone really care about the privacy of companies, consumers and users at all? or better does privacy exists at all for the millions of the people that connect to the internet every day?

• Akismet. This plugin checks your comments against the Akismet web service to see if they look like spam or not and lets you review the spam it catches under your blog's "Comments" admin screen.
• All in one Favicon. The following plugin adds a Favicon to your site and the WordPress admin pages. It currently supports all three Favicon types (ico,png,gif).
• Digg Digg. This plugin adds a floating bar with several share buttons directly on your posts. Readers will be able to share to Google+, Facebook, Twitter, Reddit, Stumbleupon with just one click. A must have.
• Flexi Pages Widget is a highly configurable WordPress sidebar widget to list pages and sub-pages. Also, if you are using a Child theme and your Home-link doesn't appear in your Pages navigation menu without either modifying the functions.php or adding the page manually through the Pages menu, this plugin will do the work for you. You are one click away from fixing many pages-related problems.
• Google Analyticator, it adds the necessary JavaScript code to enable Google Analytics. Includes widgets for Analytics data display.
• Google Authenticator will add a two-step-authentication for your blog. This will require you to own an Android or an iPhone though. After verifying your phone with the plugin your login prompt will have one more field called "Google Authenticator Code", accessing your blog won't be possible without the code generated by the authenticator every 30 seconds.
• Google XML Sitemaps will generate a special XML sitemap which will help search engines to better index your blog.
• Wordpress Backup to Dropbox. Your hosting doesn't provide you with a backup solution or it provides it with an excessive cost? Don't worry, this plugin will keep your valuable WordPress website, its media and database backed up to Dropbox. You can select how often the backup should run, you can exclude huge files from being backed up, everything with one single and easy interface.
• Social broadcasts posts to Twitter and/or Facebook, pull in reactions from each (replies, retweets, comments, "likes") as comments. Social will aggregate the various mentions, retweets, @replies, comments and responses and republish them as WordPress comments.
• SEO Ultimate is a powerful all-in-one SEO plugin, available free for WordPress bloggers. You can take control of your on-page SEO with user-friendly settings and tools for optimizing your titles, meta data, robots tags, canonicalization, autolinks, post slugs, and much more.
• Jetpack adds many features that were available to WordPress.com users but they weren't present on self-hosted WordPress installs. Jetpack is a plugin that connects to WordPress.com and enables awesome features. I simply love the combo of Jetpack and the Follow button based on it. Readers will be able to subscribe to your blog and receive mail notifications whenever a new article will be published. Another handy feature is the optimization that Jetpack provides for Mobile phones. Just enable this feature on the admin menu and browsing your blog through a Mobile device will be a pleasant experience.
• Revision Control will give the user more control over the Revision functionality. The plugin allows the user to set a site-global setting (Settings -> Revisions) for pages/posts to enable/disable/limit the number of revisions which are saved for the page/post.
• Speedy Page Redirection adds a meta box to your page and post screens, you can then enter a new destination URL to which the page will be redirected.
• Twenty Eleven Theme Extensions is an easy-to-use plugin designed for use with the latest default WordPress theme, Twenty Eleven. It adds a set of customizable features for the theme, designed to add more flexibility to the theme's design without having to modify the template files.

#access, #branding .only-search #s, .entry-header .comments-link {
display: none;
}

#branding {
border-top: none;
}

/*** Updates WordPress without FTP credentials. ***/
define('FS_METHOD','direct');

Today we’ll analyze how to access a certain virtual machine’s console by tunneling the relevant VNC port locally and accessing it through your favorite VNC client. The scenario:

1. Machine A is our main virtualization machine and hosts several virtual machines. (VMs)
2. Each VM has its own VNC port assigned. (usually the port range goes from 5900 to 5910 or even more if the hosted VMs are more than 10)
3. We’ll be using libvirt, thus virsh.

We first need to find out which port got assigned to the VM we want to have console access to:

sudo virsh

virsh # list
Id   Name   Status
----------------------------------------------------
5    foo    running
6    bar    running
7    foobar running

virsh # vncdisplay foobar
:3

We, then, create a tunnel which redirects all the traffic from the main virtualization machine’s port to the port we gonna specify in the next command:

ssh -f -N -L 5910:localhost:5903 user@machine-A.com

A few details about the previous command:

1. -N tells SSH to not execute any command after logging in.
2. -f tells SSH to hide into the background just before the command gets executed.
3. -L enables the port forwarding between the local (client) host and the host on the remote side.

And…why did I choose respectively port 5903 and 5910?

While you can adjust port 5910 with your own choice (that will just move the tunneled traffic from port 5910 to your favorite port), that won’t work as expected with port 5903 since each VNC port is binded to the number of display virsh assigned to it. (for example, the bar VM may be running on display 5, thus its vncdisplay port will be 5905)

When done, fire up your favorite VNC client and create a new connection with the following details:

Protocol: VNC - Virtual Network Computing
Server: localhost - 127.0.0.1
Port: 5910

The connection will load and you’ll be put in front of your ‘foobar’ VM console.