When faced with managing dozens of servers, keeping your configuration files up to date (or existent) becomes a real chore. At Booking.com we saw many interesting ways people had solved this problem, but all of them had at least one flaw: a manual step had to be taken. Either configuration files needed to be pulled from an external source or pushed to the new host. When the last hackathon came around we decided there had to be a better way to have your configuration files delivered to the host upon login.
The problem
Once you start working with many servers most of us give up on having any kind of unified experience across all of them. The more persistent ones figure out a way to initially populate a host with configuration files. Some even find a way to keep the systems relatively up to date with each other. Looking at what people had done to solve this we found that they were using a git repository to pull in configuration files or using tools like rsync or scp to copy the files from another server. This was almost always followed by some general issues:
- Network accessibility - some of our servers are in segregated networks and cannot talk to each other
- Permissions of files when using a git repository
- That pesky manual step
- Getting the configuration files to their right places (or linking them)
The goals
Having the above mentioned issues in mind we made a list of goals we wanted to achieve while building the mechanisms which would distribute the configuration files.
- The configuration files should be in git repositories. The benefits are clear here. An additional benefit to us was that the server hosting our git repositories is accessible from (almost) all hosts in the network, at least for read-only (HTTP) access
- The entire mechanism should be opt-in. We don't want to disrupt anyone's current setup or introduce unnecessary steps (more than absolutely necessary)
- The overhead for this should be minimized in order to have the least possible disruption to the login process
- There should be no (or as little as possible) manual steps
The "dotfiles" repositories
Historically the place to keep configuration files and miscellaneous scripts was in our users.git repository. This quickly became troublesome as that repository quickly ballooned to over 800MB. As always, workarounds were found (like using orphan branches) but at this point people's methods for keeping their configurations up to date became a mess of workarounds on top of yet more workarounds.
So a single repository was out of the question. We decided on separate repositories for each user. This made managing them a breeze because each user owned their own repo. When a user opts in to using this mechanism, a fresh repository is cloned from a template on the server. This serves many purposes:
- The template repository contains some sane defaults for ssh and git which will make life easier for new developers and those who don't generally bother with setting things up.
- There is a well defined place where you can take a look at how your colleagues set up their work environment.
- Git experts can manage their environment with sufficient granularity.
With persistence out of the way, we still needed to store preferences and reference those preferences upon login.
LDAP
When we thought about the login process itself we wanted to keep it as light as possible and avoid adding new dependencies. We decided to use LDAP as it's already an integral part of the login process on all our servers. But how were we going to store our preferences into LDAP per user? To accomplish this we (ab)used the labeledURI attribute type.
We settled on some basic options and additional data. The final attribute looks something like this and is a mix of settings and data that facilitates a faster login.
labeledURI:move_files=1,verbose=1,repo=dotfiles/htool.git,uid=htool,sha=32d8e90447687d75e1fe741d33c550e9b28a6cdc,update=1,branch=master,enabled=1
The sha parameter is the magic that makes it all work. To get that parameter
in LDAP, we wrote a simple git
hook which
updates the users LDAP record with the SHA1 of the latest commit.
Finally, we whipped up a new section on the employee profile page in the staff portal to allow people to set these options for themselves.
The login process
The main server side component is a script that lives in /etc/profile.d/. This script does a quick check for a login shell and then executes a Perl script that reads the labeledURI LDAP attribute and acts accordingly. If the user has not opted in, this is where the process ends for them.
When we see that a user has opted in, there are a few main scenarios:
- New host - We need to clone to ~/.dotfiles
- Known host and there are local changes - Raise a warning about the local change
- Known host and LDAP SHA1 matches local dotfiles SHA1 - We are done
- Known host and LDAP SHA1 differs from local dotfiles SHA1 - We need to pull the changes
How do we know that there are changes? That's where the sha contained in the
settings string comes into play. If you have been paying attention, you'll
remember that we populate this with a commit hash after each push to the
repository. At this point all that's needed to test if the repository needs a
pull is to compare the SHA1 from the LDAP attribute to the last local commit.
By doing this we have no need to do a git fetch (or pull) on each login, but
only when there are changes present.
After the git pull or clone, there is a final step of linking all the files to their right places. For this there is a script in the ~/.dotfiles repository itself (remember, it was cloned from a template), which mirrors the files from ~/.dotfiles/dotfiles into the users home directory. If the script finds the same files present in your home directory, they are either ignored or moved to ~/.dotfiles/bit-bucket, depending on your "move_files" setting.
The user is also free to add additional steps to that script (after all, it's in their repository) or source an additional script from there.
And that's all there is to it: a little bit of git, mixed in with a little bit of LDAP and Perl
Usage
Our dotfiles distribution system is currently enjoyed by a large segment of our colleagues who find it incredibly useful to have all of their settings, tools, and other errata available no matter which server gets their attention. It does what it promises to do and it's fast enough to not cause noticeable delay upon login. A ton of people can now be more productive by having their preferred work environment replicated to all hosts in our infrastructure.
This was a big pet peeve that was resolved and a hackathon well spent.