Configuration 101: Using Subversion for Change Management

Whereas SSH Keys come primarily from the sysadmin realm, the next tool I'd like to introduce is most often associated with the programmer's toolkit. Subversion (SVN), or more generally, revision control software, allows you to do away with littering your production systems with tons of backup files, and consolidates your files into one place that's easy to backup, duplicate, and modify. Programmers typically stuff source code, makefiles, and supporting documentation into SVN; I use it not only for that, but for /etc/ files, kickstart configs, shell scripts, and more. Some of my systems -- in particular my puppet server -- pull from subversion via a cron job, others use puppet to push files as needed, and others I push files to manually. Eventually I hope to put a great deal more data in SVN, such as much of our website, but for now, I'm starting out slow.

So, what to do?

1. Find a reliable space to store your repository, which is essentially the authoritative copy of your files. This space needs to be reliable, and should be easy to back up. Subversion (and most other RCS systems) will work with a variety of access types, but primarily I find direct filesystem access over NFS and remote access via SSH to be the most common use cases*. For the purpose of this discussion, we'll call it /export/svn/foo
2. Install subversion on the chosen system. Most *nix systems will have a subversion package; if not, there's a link to their website at the top of this post.
3. Run svnadmin to initialize the repository. 'svnadmin help create' will give you some info on the options available to you, but generally 'svnadmin create /export/svn/foo' is the form of command you want here.
4. Install subversion on your client system; soon I'll be posting information on good Windows-based clients for subversion, but right now I'm going to punt on that and assume we're using *nix.
5. Check out a working copy of your empty repository:
   cd ~
   mkdir svn
   svn co file:///export/svn/foo ~/svn/foo
   -- or over ssh to a server called myhost.com --
   svn co svn+ssh:///export/svn/foo ~/svn/foo
6. Copy files into the working copy, or start creating new files.
7. Add the files into SVN, using svn add ...
8. Commit your changes back to the original repository, using svn commit -m "Log Message Goes Here" ~/svn/foo/

Obviously, there's a lot more to managing source files, but this is a start. Note that if you're using SSH to connect to a remote repository, SSH keys will save you a lot of typing!

* If you've ever used a code.google.com project, you may have noticed that they use SVN. If you have commit access, you'll use authentication over HTTPS to do commits. SVN is capable of storing your password in your working copy in this case, but it's beyond the scope of this particular post; go forth and Google it!

Automation 0.1 (alpha): Using SSH Keys

I'm starting out simple. No, really. I can't imagine that anyone who does system administration isn't already using ssh to replace r[shcplogin], telnet, ftp, and to secure a whole host of other protocols using tunneling, but when I talk to other geeks (or shoulder surf as they work), I've noticed that a lot of them don't seem to realize the flexibility that comes from using keys to log in to ssh. So here's a quick rundown of the benefits (to me):

1. I don't have to type my password in 5000 times a day, as I hop from machine to machine. My laptop and my desktop both have one key on them, and all of my core servers have another key for my account, and between these two, I mostly only enter my password to unlock the desktop or laptop.
2. For systems with equivalent security levels, I can put an ssh key on the root account of one (or all) of the systems, and allow it to log in to all the others via scripts/cron/etc without human interaction. When I say equivalent security levels, allow me to explain: I do this mostly on systems which don't store differing levels of important data, which share the same physical access, and for which I have a reason to have them communicate. For example: a teaching lab with a small compute cluster ~ the head node has root access to all of the compute nodes.
3. With less worries about security, relatively unprivileged service accounts can share a common ssh key to automate routine communication between systems -- I use subversion to distribute a variety of data, and allowing an svn account to do svn+ssh:// pulls from a cron job is very useful. Liekwise, my webservers all have shared keys for the www account, so they can push and pull data directly. If I gave www sudo access to apachectl, then they could also restart each other's httpd processes as needed, but so far I haven't found much use for that.
4. So what if you ARE worried about security, and you have less trusted systems that you want to have access to more trusted ones? Passphrases can help to secure ssh keys, and a great deal of scripting is still possible. Essentially, keys can be encrypted with a passphrase, and unless that phrase is entered, the key is useless. So the key can be stored in an insecure area -- hell, with a good enough passphrase it can be lost -- and the key's access will remain secure at least until someone cracks the passphrase. How do you automate this? With ssh-agent and ssh-add, a pair of utilities that allows a computer to store your ssh key's passphrase as long as your unix session is active. So as an example, I have a backup script that runs for about 5 days at the beginning of every month (lots o' data, not so much bandwidth). In the script, I first check to make sure I'm running within screen, then I load ssh-agent, and use ssh-add to ask for the passphrase. This all happens right at the top of the script. Hours pass as the backups are taken, and then days go by between file transfers, but the files are copied securely and automatically, without me having to be there for the entire process. Most importantly, no one who doesn't have the passphrase can do anything harmful with the script, and as soon as the copies are done, the passphrase is wiped from memory.
   

So, on each one of those points, I really felt a need to put a footnote or two, mostly about security. Basically, though, it boils down to this: unless you're using an one time password system or some sort of hardware security module, you have to trust root on the computer you're using. For that matter, you generally have to trust root a little bit even in those cases. SSH keys can improve security a lot, but they aren't completely unbreakable. The computer you're using could have a keylogger installed to capture your passphrase, and it's pretty trivial for root to grab your ssh key. In short: don't try to do system administration from untrusted systems. Period. I trust my laptop, I trust my desktop, and I trust most of my servers most of the time, but that's about it. If I'm away from work and away from my computers, I prefer to make a phone call and walk a coworker through a task rather than doing it myself from, say, a kiosk. Another option is to use a bootable live-cd, which I've done on rare occasions in the past. Okay, maybe I've convinced you that ssh keys are useful tools in the right circumstances; so how do you get started?

1. First, on your trusted client system, use ssh-keygen to create a key encypt it with a passphrase. Do the passphrase thing -- if you find it to be onerous, you can always take it off later, and you don't have to redeploy the key. If you REALLY don't want a passphrase, just leave it blank. This will create two files, by default ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub. The first of these is the private key; the file that is encrypted with the passphrase, and the one that should be safeguarded.
2. The other file is the public key, which should be appended to the ~/.ssh/authorized_keys file on any system you want to connect to. If you don't already have any ssh keys, you can just rename or copy the file. Otherwise, each key is stored on one (very long) line in the file, and typically has a text field at the end signifying where the key was created to help with identification. For the most part, I leverage our shared home directory structure at work to make sure keys are distrubuted, but in cases where accounts have local home directories, I make other arrangements. For occasional one-off systems, I copy the keys using sftp with (gasp) password authentication. For other situations, I incorporate key management into the rest of my automation. So, for the teaching lab cluster I mentioned above, I distrubute the keys using puppet, another one of my system configuration tools, which uses its own PKI setup to keep everything on the up-and-up.
3. If you're using a passphrase and want to store it, load up ssh-agent. This is a slightly odd utilty because it's designed to run from a variety of environments (X, terminal, etc). Read the docs on the best way to load it for your situation, but from a terminal, "eval `ssh-agent`" will usually do the trick. You should see a single line of output that resembles "Agent pid #####" if you did it corretly. If you see three or four lines of output, you probably did something wrong -- note the backticks above!
4. Finally, run ssh-add which will prompt you for your passphrase. Once you enter it correctly, your key will allow you to log in without prompting to systems which have your public key installed.
5. When you're done, "ssh-agent -k" will kill the session.

Hello, World!

I have a tradition of starting new projects with the phrase "Hello, World!" To any student of programming, the reference will be obvious, but for the benefit of all you non-computer-geeks who may mistakenly stumble across this blog and actually stay for a few minutes, I'll explain. Besides being an appropriately expansive greeting for a new blog, "Hello, World!" is, by convention, used to introduce students of any programming language to the fundamental features of that language. By learning how to write a program that simply displays that phrase (or any other), you can learn a lot about the way programs are constructed in a new language. Over time, Hello World becomes a very simple rosetta stone when trying to pick up a new programming or scripting language.

So, why another 'Hello from me? I'm frankly not a very good blogger; every now and then I write a post on my personal blog, but mostly it sits there, displaying the old stuff day after day. I don't really think that's going to change ~ and frankly, I don't think that "make make" will be all that much better ~ but what I do want to do is give myself another, slightly different, place to put stuff that is less about me and more technical. I am a system administrator who has a lot of training as a programmer, and as such, I find that there's not much of a niche for me in geekdom; most people identify as one or the other, and there is often a hint of animosity between the two groups. Regardless, there is a lot that each side can learn from the other, and my goal is to post the bits and pieces that I've learned about both sysadmin-ing and coding in hopes that someone, somewhere, will find a few of them useful.