make ; make install

My server's hard drive crashed

2010-01-30T00:22:00.000-08:00

This is a test post to make sure publishing is all hunky-dory on my new virtual server

A Simple Django WebAuth Decorator

2010-01-22T23:45:00.000-08:00

Like many universities, my employer uses Stanford's WebAuth Single Sign-On package as one major piece in it's computing account system. WebAuth is an MIT licensed infrastructure that allows decentralized web applications to securely authenticate users without themselves ever handling user credentials. Websites begin by sending unauthenticated users to a trusted WebAuth server which validates the user and provides them with a ticket which is passed back to the application. The application web server then communicates directly with the WebAuth server to validate the ticket it was given. If the ticket is valid, the application is provided with information on the user, and it can proceed without further interaction with the server. As long as the user's WebAuth session remains active, any other application the user visits can authenticate the user behind the scenes.

The benefits to users of a consistent security interface are significant, so I've recently been pushing myself to make use of WebAuth where possible. Unfortunately, there's not a lot of preexisting code for integrating WebAuth into mainstream web frameworks, so I've had to write my own, which -- fortunately for me -- hasn't proven to be all that difficult. Today I'm going to share a snippet of code that I wrote to integrate WebAuth into a Django app:

webauth.py -- Pretty Print HTML

webauth.py -- Raw Code

@webauth_required is a Python decorator that functions similarly to the @login_required decorator that is provided with Django. Importing this decorator and prepending a view function with @webauth_required is all that's needed to force that view to authenticate the user. Once they're authenticated, their username is stored in a Django session for authorization purposes (as 'netid' in this code, since that's the parlance familiar with users and developers on my campus). Obviously the WebAuth endpoint (AUTH_URL) is also specific to my situation, and most WebAuth providers will require registration of client applications to enhance security.

The only thing that's left is to provide a logout mechanism. My logout view (one of the few views that doesn't need the @webauth_required decorator in my app!) simply destroys the session data and provides the user with a link to log out of WebAuth entirely; it's possible for the user to log out of the app but remain logged in to WebAuth, which effectively leaves them logged into the app (since they'll be re-authenticated behind the scenes if they return), but that's how the powers that be have asked client apps to behave, so that's how it is.

Contemplating cost-efficient, highly reliable storage

2010-01-13T09:11:00.000-08:00

I have long been a fan of NetApp's storage products; my office purchased our first NetApp filer shelf back in 2005, and over the years have added an additional five shelves as our capacity needs have grown. Management of these devices has almost always been painless, and NetApp's support has generally been top notch. Having a new drive show up in my mailroom along with a note describing which drive slot it should go in -- because of a predicted potential failure -- without any involvement on my part? As someone who's charged with protecting valuable research data, this level of monitoring helps me sleep at night. And automated snapshots give my users the ability to restore most accidental deletions without ever having to admit to me or our support staff that they've done something careless, which is a win for everyone involved.

In fact, there's really only one downside to NetApp's product line: price. The most recent quote for a full NetApp system I've seen put the total first year cost for storage (with SATA drives) somewhere close to $10/GB, with significantly reduced annual maintenance fees thereafter. Lately, as the NetApp's dispute with Sun over WAFL and ZFS continues to crumble, I've been looking into ZFS and it's feasibility for our premium storage. Sun's lowest end storage servers (with high performance, low capacity SAS drives) have a list price point of about $4/GB, while commodity hardware prices for SATA drive based systems are $0.30/GB; as our total storage needs continue to skyrocket, it makes sense to see if I can replicate most of the critical abilities of NetApp storage at a more reasonable cost.

We will soon be deploying a storage server with somewhere between 32-48 TB of raw disk space, shared via NFS and CIFS. My plan is to install OpenSolaris on this hardware and set it up with ZFS RAIDZ2, ZFS's equivalent to the double-parity RAID-6. In preparation for this, I've been playing with OpenSolaris VMs, trying to get a feel for how management of ZFS is handled, and so far, I've liked what I've seen.

Over time, I'd like to document the critical steps I've had to go through to get a working production system, along with my conclusions along the way about the viability of this platform as a significant piece of our overall storage platform. For starters, here's a few of the things I've come across so far on my test system, which has two 20GB system disks and five 8gb data disks:

At least when using the OpenSolaris LiveCD installer, root mirrors have to be set up after the fact, and mirrored drives must be set up with Solaris slices

The OpenSolaris installer was a pleasure to use, but it's simplicity makes for a lot of additional tuning after the fact to get a usable system, and while ZFS is the default filesystem, you can only install to a single drive. My root partition was c3t0d0s0, so after booting into the system, I set about to add a mirror. The only hitch I ran into was an error about rpool (the Solaris root pool) needing to live only on Solaris slices -- I'd tried to attach an unlabeled c3t1d0 drive to the pool. After labeling the disk and adding a slice 0, I was able to create a mirror with a single command:

pfexec zpool attach -f rpool c3t0d0s0 c3t1d0s0

To watch the progress of data "resilvering" over to the new drive, I used:

zpool status rpool

After a few minutes, this process was done and I moved on with adding a data pool.

RAIDZ pools can't be internally adjusted

Initially I created a RAIDZ2 pool with three disks, the minimum number of devices possible, planning to add the other two later:

pfexec zpool create data raidz2 c3t2d0 c3t3d0 c3t4d0

While I had no problems with this command (creating an 8GB volume from three 8GB disks), adding drives later did not work as I'd hoped:

pfexec zpool add data c3t5d0 c3t6d0
invalid vdev specification
use '-f' to override the following errors:
mismatched replication level: pool uses raidz and new vdev is disk

Sadly, increasing the "width" of a RAIDZ pool is not currently a feature of ZFS, although it would be an important one to have, and is supposedly in the works. This doesn't mean that a RAIDZ pool is forever locked to it's original size, rather it means that to increase the pool's size, you have to add additional RAIDZ "subpools" (this is almost certainly not the correct terminology) which then operate side-by-side with the original RAIDZ logical device, instead of integrating all of the disks into a single logical device. This means that if you plan to build an expandable storage server, you'll need to think carefully about how you want to allocate parity disks, hot spares, and filesystems.

Anyhow, once I discovered this I destroyed the pool and recreated it with with all five disks:

zpool create data raidz2 c3t2d0 c3t3d0 c3t4d0 c3t5d0 c3t6d0

This gave me a 24GB volume made up of five 8GB drives, which is what I'd intended for this test system. Next I set the mountpoint (which defaults to /<pool name>):

zfs set mountpoint=/export/data data

ZFS Deduplication may not be appropriate for many data sets

Next I added a couple of additional configuration options to enable compression and deduplication:

zfs set dedup=on data
zfs set compression=gzip data

Deduplication is very much a new feature in ZFS, and is currently only available in the very latest Release Candidate builds of OpenSolaris. It's not a feature that I would enable in production without carefuly testing not only for this reason, but also because I'm unsure of how it scales to very large filesystems (unlike NetApp, which has a 1-TB volume size limit on their A-SIS dedup technology, ZFS has no such hard limits); Deduplication requires that data block hashes be searchable (and potentially searched often), and at some point a large enough amount of data will require that index to spill out of RAM and onto disk, which could potentially have a very negative affect. Jeff Bonwick of Sun has written an excellent blog post that goes into more detail about dedup in general and ZFS's implementation.

While dedup is yet another feature of NetApp's storage that I find very valuable, ZFS dedup is new enough that there aren't enough field reports on it to have a good idea of how well it performs in extreme cases. Once I have real hardware to test this on (my test system lives on deduped NetApp volumes, so performance testing would be all but meaningless, not to mention cost-prohibitive given the space I would need), I hope to be able to shed some light on the matter of scalability; only time is likely to make me comfortable with the maturity factor.

Quick-n-Dirty Nagios Plugins

2009-11-26T15:14:00.000-08:00

In the spirit of the holidays -- and of spending more time at home, away from the server room -- I'd like to present a sample script from the many that I use to monitor (from home, the beach, or my mythical cabin in the woods) my computers' collective health. Obviously, Nagios itself comes with a rich collection of plugins, and plenty of people have written additional check scripts for most common services, but at some point, a system administrator is likely to be responsible for services which for whatever reason aren't covered by existing scripts, so having a template to whip up your own script can be invaluable. Before I go any further, a couple of disclaimers -- nothing here should be taken as a style guide to writing shippable, enterprise-class Nagios plugins; note the title: "Quick-n-Dirty". The example script I've chosen, which checks the status of a Windows DHCP Server address pool, is production-worthy, but fairly case-specific. There is no --usage flag, the threshhold values are hard coded into the script, and the arguments must be given positionally. Each of these could easily be fixed, but even the cost small of doing so isn't justifiable in my environment.

check_dhcp.txt

Basic requirements:

Python

SSH keyed access to the Windows server -- I use Bitvise WinSSHD, which costs money in a commercial environment, but there are other options, such as FreeSSHd.

Nagios, obviously, is needed to get much value out of this script

Usage:

./check_dhcp <hostname> <subnet>

First, a little bit of python boilerplate code:

#!/usr/bin/env python
import os, sys, re

Next, a few variables to keep track of what we report to nagios:

STATUS = ["OK", "WARNING", "CRITICAL", "UNKNOWN"]
status = "UNKNOWN"
service = "DHCP"

The STATUS array maps the allowable status values to the exit codes that they correspond to, and the initial status is set to UNKNOWN, which is what the script should report if anything goes wrong with the script itself during execution. Finally, I set the service name here, though it's not strictly needed.

warn_level = .8
crit_level = .9

used = free = total = 0
used_percent = 1

The first two variables above are the hard coded threshold values I mentioned earlier, and the rest are the variables I'll use to keep track of the status of the DHCP pool. Note below, where we set the command that we'll actually run to get the status data, that the location of ssh is hard coded, and might need to be adjusted for your environment:

check_cmd = r'/usr/bin/ssh %(server)s "netsh dhcp server show mibinfo"' % {"server": sys.argv[1]}
subnet = sys.argv[2]

The next section parses the output of the check command line by line, looking first for the specified subnet, then for the usage data for that subnet:

ss = None
for ll in os.popen3(check_cmd)[1].readlines():
    rr = re.match(r"\s*Subnet = (?P<ss>[\d\.]+)\.", ll)
    if (rr):
        ss = rr.group("ss")
        if (ss != subnet):
            ss = None
    rr = re.match(r"\s*(?P<kk>[^=]+) = (?P<vv>[\d]+)", ll)
    if (ss and rr):
        if (rr.group("kk") == "No. of Addresses in use"):
            used = int(rr.group("vv"))
        if (rr.group("kk") == "No. of free Addresses"):
            free = int(rr.group("vv"))

Now we calculate the value we're looking for -- the percentage of address space that's currently being leased:

total = used + free
if (total > 0):
    used_percent = float(used) / total
detail = "%d of %d addresses in use; %d%% utilization" % (used, total, used_percent * 100)

Finally, we compare the current status data with the threshholds we've set, and set the output status variable accordingly, build a status message, and look up the return code in the STATUS array:

if (used_percent >= crit_level):
    status = "CRITICAL"
elif (used_percent >= warn_level):
    status = "WARNING"
elif (used_percent == 0):
    status = "UNKNOWN"
else:
    status = "OK"

print "%s %s: (%s) %s" % (service, status, subnet, detail)
sys.exit(STATUS.index(status))

And that's all there is to it. This script returns status messages like the following:

DHCP OK: (150.135.220.0) 229 of 505 addresses in use: 45% utilization

As you've probably noticed, about half of this script is fairly standard structural stuff which translates pretty well to most other Nagios check scripts; the DHCP-specific code all happens in the middle of the script. Below is a script for checking that a Solaris LDAP server is working that uses pretty much the same template, but is stripped down even further:

#!/usr/bin/env python
import os, sys, re

STATUS = ["OK", "WARNING", "CRITICAL", "UNKNOWN"]
status = "OK"
service = "LDAP"

check_cmd = r'/usr/bin/ldaplist passwd nagios'
detail = os.popen3(check_cmd)[1].readline().rstrip()

if (detail.find("uid=nagios") >= 0):
        status = "OK"
else:
        status = "CRITICAL"

print "%s %s: %s" % (service, status, detail)
sys.exit(STATUS.index(status))

Lastly, here's a script (which I'm pretty sure must have been written by one of my coworkers) that uses python's subprocess module to pipe commands together, which allows for some more complex processing of status information:

#!/usr/bin/env python
import subprocess, sys

STATUS = ["OK", "WARNING", "CRITICAL", "UNKNOWN"]
status = "OK"
service = "Mailman"

p1 = subprocess.Popen(["/usr/bin/ps", "-eaf"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "[m]ailmanctl"], stdin=p1.stdout, stdout=subprocess.PIPE)
return_code = p2.wait()

if (return_code == 0):
        status = "OK"
        detail = "Master daemon found."
else:
        status = "CRITICAL"
        detail = "Master daemon not found."

print "%s %s: %s" % (service, status, detail)
sys.exit(STATUS.index(status))

Hopefully, this sampling of scripts gives you a good starting point for whipping up some more scripts that are helpful in your own environment; if you come up with anything particularly innovative or interesting, I'd love so hear about it!

Simple, reliable, rate-limited backups

2009-10-28T21:39:00.000-07:00

Wow, once again it's been a long time since I've written anything. The series on marrying Active Directory with *nix systems is by no means dead, I've just been forced to spend all of my research time tinkering with aspects of AD integration that don't really fit in well with the series, so until I get past that, it'll remain on hold.

In the meantime, I've been meaning to write up a brief post about a backup script that I've been using to back up a number of servers that I manage. Briefly, the script archives a local copy of the previous backup, creates tarballs of specific directories for the current backup, then transfers those to a remote backup server. That gives me at least one good local backup at any given time (two when the script is not actually running) and as many remote backups as I choose to keep -- I've long considered implementing a `find` cron job to clean up old backups, but disk space is cheap, so I usually let them sit as long as possible and manually clean them up a few times a year.

The script doesn't rely on much that you wouldn't expect to have on a typical *nix server -- tar, gz, scp are the minimum requirements, and screen is highly recommended if you choose to run the script manually. So, without further ado... actually, a disclaimer is in order -- the fundamental concepts of this script I came across while lending a hand to a friend of mine who needed someone to keep an eye on his server while away on an extended trip around the world. I've polished it up, added some sanity checks, and combined a two step process into a single script, but the script below is definitely a derivative work, so credit is due to Trevor for his original work.

backupscript.txt

So, let's break it down:


#!/bin/bash

Bash is definitely a requirement, as we'll use it's string manipulation abilities to munge multilevel paths into sane filenames.


backup_dirs="/etc/backup_dirs"
backup_path="/export/backup"
backup_location="backup@backup.makemakeinstall.com:/export/backup/makemakeinstall.com/"
scp_opts="-l 4096"

The variables set here are:

$backup_dirs is a text file containing a list of directories to back up, one per line, and I'm pretty sure trailing slashes are discouraged, though adding a line to make them merely irrelevant would be trivial.

$backup_path is the directory in which the local backups will be kept. It shouldn't initially contain anything other than a directory called old.

$backup_location is the remote path that the backups will be transferred to via scp. This definitely works best if you have SSH keys set up

$scp_opts is a list of additional options that can be passed to scp for the remote transfer -- in this example, it's used to rate limit the connection to 4Mbps


# check for screen
if [[ $TERM != 'screen' ]]  
then
  echo "ERROR: seriously, run me inside screen"
  exit 1
fi

This is a completely optional check to make sure the script isn't being run from a terminal session that you might not want to rely on keeping open -- starting a screen session first will spare you some headaches if you're running the script manually. If you plan to use the script both from cron and for manual backups, prepending `TERM=screen` to the cron job will allow you to keep the check in place but still easily trigger backups via cron.


# make sure we have list of directories to backup
if [[ ! -f $backup_dirs ]]
then
  echo "ERROR: $backup_dirs not found; nothing to backup!"
  exit 1
else
  for dir in `cat $backup_dirs` ; 
  do
    if [[ ! -d $dir ]]
    then
      echo "ERROR: $dir is not a valid directory!"
      exit 1 
    fi
  done
fi

This goes through the file $backup_dirs and makes sure that the listed items are in fact directories that exist on the filesystem. If either the file or any of its entries are not found, the backup will not continue.


# get passphrase
eval `ssh-agent -s`
ssh-add

Another optional element -- I highly recommend passphrases on keys, and if you don't use a utility such as keychain to manage your passphrases, this snippet will allow you to enter the passphrase at the beginning of the script, so that it will be stored when the script is ready to start the uploads.


# run local backup
echo backup started: `date`
now=`date +%Y%m%d`
nice rm $backup_path/old/*.gz
nice mv $backup_path/*.gz $backup_path/old
for dir in `cat $backup_dirs` ; do
  echo $dir
  dirname=${dir:1}
  nice tar -zcf $backup_path/${dirname//\//_}-$now.tar.gz $dir
  chmod 0600 $backup_path/${dirname//\//_}-$now.tar.gz
done
ls -l $backup_path/*.tar.gz
echo backup complete: `date`

Here we get the current date, saved as a variable to keep the backup set together in case the tar processing time overlaps multiple days. If you expect to take multiple backups in a single day (I choose to do weekly backups, but I've set this script up for daily/monthly backups as well at times) you'll want to tweak the `date` command to include a more detailed timestamp. Next we remove the oldest backup from $backup_path/old, move the most recent backup into $backup_path/old, and start a new backup. The manipulations of $dir -> $dirname are intended to strip away the leading slash and to replace any internal slashes with underscores, so that a backup for /var/www (on 31-Oct-09) would be saved as var_www.tgz-20091031.


# upload
echo upload started: `date`
scp $scp_opts $backup_path/*.tar.gz $backup_location
echo upload complete: `date`

Lastly, we start the upload. This is almost anti-climactic. If you get charged for straight bandwidth usage (or don't have to pay for bandwidth at all) then rate-limiting via the $scp_opts variable is a concern only if you need to reserve bandwidth or processing time (encryption can be expensive at higher speeds) for other tasks. But if you pay for a burstable connection, you'll want to make sure that the rate limiting options are tuned to keep the bandwidth usage (of the backup plus any baseline/expected traffic) below your trigger rate.

In conclusion -- this is definitely not the only way to do backups. It's certainly not the most network or space efficient. It is, however, quite reliable and simple enough to be easily grasped, an important factor if you're concerned about making sure that you actually know how to recover from your backups -- and you definitely should be.

My First PowerShell Script: Duplicating NTFS ACLs between file trees

2009-06-01T14:58:00.000-07:00

I wrote my first PowerShell script this morning; it wasn't something I'd planned to do today -- or for that matter, at any point in the near future -- because while I have nothing against PowerShell, I really haven't had enough interest in it to make spending time with it a priority.

Over the weekend, though, I migrated a couple of CIFS shares from one server to another, and while I'd copied ACLs on all of the files and subdirectories, the permissions on the top-level folders housing each shares weren't set correctly. Once users came in this morning, they were able to access their existing files, but not browse the shares. One of my coworkers beat me in to work today, and due to a difference in the default behavior of Vista and older versions of Windows, he managed to clobber all of the permissions on the file shares, replacing them with a single set of ACLs for all files in each share.

Once I arrived, we kicked back and forth a couple of ideas on how best to fix the problem without disrupting the users' work, and finally decided that my best bet was to write a script to traverse the original filesystems, check the ACLs on each file/directory, and apply those to any existing files in the new filesystems that had the same name. I was vaguely familiar with using perl and setacl.exe to do this type of thing, and wasn't looking forward to the headache, so we started looking at alternatives, and PowerShell came up as an option; its Get-Acl and Set-Acl cmdlets (offtopic: who the hell thought "cmdlet" was a good name!?) are naturally suited to duplicating ACLs. The only problem? My aforementioned unfamiliarity with PowerShell.

Actually, it turns out that I had another problem -- PowerShell wasn't even installed on the machine that I was using to connect to all of the relevant shares. After installing it -- along with its requisite specific version of .Net Framework -- I was ready to go... and had no idea how to start.

I won't bore you with the details of how I figured out all the little bits of syntax that went into making my script, other than to say that Google yet again saved my bacon. I wasn't able to find a script that did exactly what I wanted, but since walking though file trees and setting ACLs are both fairly common tasks, it wasn't too hard to come up with this first version of the 'tree-copy-acls.ps1' script:

# get paths

$src = $args[0]

$dest = $args[1]

# walk the source file tree and copy acls to files in dest

Dir $src -recurse | ForEach-Object {

$path = $_.FullName.Replace($src, $dest)

get-acl $_.FullName | set-acl -path $path

}

invoking this as "& .\tree-copy-acls.ps1 <SOURCE> <DEST>" spewed out errors on nearly every file -- changing the ownership of the files is part of the set-acl process, and by default your choices for the new owner are limited to the user account making the change and the "Administrators" group. Processes running with the "SeRestorePrivilege" security token, however, have no such limitation. Looking for a way to get myself of one of these tokens, I headed back to Google, and discovered the "PowerShell Community Extensions" or PSCX, a set of PowerShell helpers that allows (among many, many other things) a script -- running with appropriate access, of course -- to manipulate security tokens. Evidently, this specific task is one that PSCX gets used for quite a bit, because their website states that they're planning to add a one-liner that allows access to modify ownership; the process for the current version, however, isn't too bad:

$SeRestorePrivilege = New-Object -typename \

PSCX.Interop.TokenPrivilege -argumentlist \

"SeRestorePrivilege", $true

Set-Privilege $SeRestorePrivilege

Note: I have no idea whether the "\" convention I've used above to note line continuation works in PowerShell -- so unless you're more experienced with PowerShell than I and know the answer, just make sure that the first three lines of the above snippet are actually a single line, with no "\" characters in it.

Installing PSCX and adding this into the script made everything work, so 'tree-copy-acls.ps1' version 0.2 looks like this:

# get paths

$src = $args[0]

$dest = $args[1]

# allow script to set ownership

$SeRestorePrivilege = New-Object -typename \ PSCX.Interop.TokenPrivilege -argumentlist \

"SeRestorePrivilege", $true

Set-Privilege $SeRestorePrivilege

# walk the source file tree and copy acls to files in dest

Dir $src -recurse | ForEach-Object {

$path = $_.FullName.Replace($src, $dest)

get-acl $_.FullName | set-acl -path $path

}

Obviously, there are a few tweaks that could make the script cleaner -- checking for proper input, printing a usage message in case of invalid arguments, and sanity checking each file during the ForEach loop to avoid trying to set ACLs on nonexistent files in the destination tree would make the script a little more user friendly, but these should be fairly straightforward tasks for another day.

Installing and Configuring Microsoft System Center Configuration Manager

2009-04-25T23:10:00.000-07:00

I know that I've left the Active Directory series with barely enough info to get you going with Linux, but real life priorities are intervening at work, and I don't have the time right now to document the next step in the Linux integration, which will be getting service level authentication working with Kerberos for stuff like NFS. It's coming soon, I promise.

In the meantime, I'm setting up another Microsoft System Center Configuration Manager server -- for obvious reasons, I won't use the full title again, and will call it ConfigMgr or SCCM as the mood strikes me -- and I thought I'd do a quick write-up on that while I'm at it; while this installation is going smoothly (fingers crossed) I think I reinstalled my original ConfigMgr server four to six times before getting it right.

ConfigMgr Server Setup

To keep things simple, I've kept all of my ConfigMgr-related features on one server -- the Certificate Server, SQL Server, IIS, and all of ConfigMgr's own functions are on one box. It seems to handle the load just fine, though the admin interface can be a little sluggish at times when dealing with heavy database interaction.

First of, there's a question of the version of Windows to use. While I'm almost always inclined to use the latest version of software that is available, my experiences a year or so ago with ConfigMgr on Windows Server 2008 were less than stellar. I spent quite a lot of time running into inexplicable dead ends, and time to waste is not a luxury I have right now. So I've decided to continue with Windows Server 2003 R2, which is the next best thing. Once all my new systems are up in production, I'll go back to the lab and see if the situation with ConfigMgr on 2008 has improved.

The edition of 2003 also matters -- you need at least one Enterprise Edition server in your domain running AD Certificate Services, or you won't be able to even install ConfigMgr without a lot of ugly hacks -- it requires a custom certificate template for the SCCM site servers, and Standard Edition won't let you use custom templates. Also, though I installed 2003 R2, I didn't ever actually go through the R2 specific features; I didn't need them on this particular server.

One the system is built, joined to a domain, and patched, I installed IIS. BITS and WebDAV are important requirements, so make sure to install them. Once these were installed, I went back into the Windows Component wizard and installed Certificate Services. I set it up as an enterprise root CA -- if you're deeply concerned about the security of your PKI, you probably don't want the root CA to be on a production system, but rather on a box which stays offline and physically secure. However, the only purpose of this particular PKI is to provide security for my ConfigMgr setup, so I didn't see much point in investing significant resources in making it more secure than the overall Active Directory domain which it plays a comparatively small part in. Next I configured the CA and enrolled the ConfigMgr server for its Site Server certificate. This is a bit of an arbitrary and tedious process, so I'll just point you to the KB article that describes the process. The IIS server should also be configured with a certificate to use for https:// communications -- I used a commercial certificate, but you can also use your local PKI to create one. If you choose the latter, domain clients will automatically trust the cert, but non-domain machines will not. This is probably not a big deal, since any non-domain ConfigMgr clients will have to have the PKI root certificate installed anyway, but it's worth noting in case you're using standalone systems with the web interface.

The next thing to install is SQL Server; I used 2005, again primarily because of a lack of testing time; when I first installed ConfigMgr, 2005 was the latest version, so I know that it works. Service Pack 2 is definitely required, and there's a hotfix that I needed; your mileage may vary. WSUS must also be installed, although it doesn't have to be used for ConfigMgr to deploy patches; I installed it along with SP1, and then took a break from software installations to catch up on patches through Windows Update; a couple of reboots later I was ready to keep forging on ahead, and we were finally done with prerequisites and ready to tackle ConfigMgr itself.

Well, almost ready; technically, extending the AD schema to enhance ConfigMgr support is not required, but it does improve the efficiency of your site, so I decided to go for it; Microsoft has several TechNet articles with more information on why and how to extend the schema -- it was completely painless in my case. There wasn't much to the actual SCCM installer once I'd gotten all of the prereqs out of the way -- I chose to do a native mode installation, enabled all agents except for Network Access Protection (again, a feature I haven't had time to experiment with in a lab), and that was pretty much it. once it had the information it needed, the installer took quite a while to run, so I suggest locking the console and stepping out for a bit of your preferred caffeine source at this point.

After the install itself, I again ran Windows Update, and rebooted, then started up the New Site Roles wizard. A couple of things to keep in mind -- the FQDN for inter- and intra- site communication needs to match the Active Directory domain; this might not be an issue for you, but in my environment most of our servers have internet-facing DNS addresses which are shorter that their fully-qualified AD DNS domain. Since the local PKI will be securing the communications, the AD domain name is the one which will be used, and mismatches definitely won't be accepted. I also instructed the wizard to "use this server as the active software update point", and checked the boxes for the following roles, keeping with my "one server to rule them all" philosophy:

Server locator point
Reporting point
Software update point
Fallback status point

The next configuration step was to enable inter- and intra- site communication on both the Management Point and Distribution Point. I also created two domain accounts -- one user account with minimal access, which ConfigMgr uses to authenticate when pulling data from the server (the Network Access Account) and one with domain admin which is used to do automated installs; strictly speaking, this account might only need local admin on all of your systems, but it seemed easier to me just to go with domain admin and not worry about the one-off problems that might occur with systems not getting the memo about the account's access. I also added the ConfigMgr server's computer account (SERVERNAME$) to the domain admins group -- this allows the server, which runs as LocalSystem, to do pretty much anything it wants. This may seem scary, but again, keep in mind that as long as you practice the same level of physical and electronic security measures on this server as you do on your domain controllers, you're really not introducing a new level of vulnerability, you're just confronting it from a different angle. Like your domain controllers, this server now holds the keys to your electronic kingdom, so treat it with respect and care! The last set of ConfigMgr settings I had to muck with was under the Software Update Point properties:

Active software point on site server
Allow both inter- and intra- communication
Checkmark "All Classifications"
Enable synchronization on a schedule
Uncheck all languages except English

Obviously the last item was specific to my locale; adjust accordingly. At this point, if you've enabled client certificate enrollment (as described in the KB article above) and you enable Active Directory discovery and some form of client installation (I do scheduled pushes through ConfigMgr) you should start to get domain clients joining your system.

Manual Client Installation

Manual client installation is a bit of a hassle, but it's great for supporting users who don't live on your domain, whether they be home users, laptops, or for little pockets of machines that have to be managed but just can't be in your primary Active Directory hierarchy.

Before you try to sign up clients, you'll need to get them into your PKI; the normal Computer certificate isn't ideal, since it doesn't allow you to specify the client's name. Mimic the steps required to create the ConfigMgr Site Server certificate template, but don't make any changes to the template other than setting the Subject Name to be supplied in the request.

The first thing to do on each client is to get the PKI root certificate into your computer's local certificate store. If you use the local PKI for IIS on the ConfigMgr server, then simply browse to https://[fqdn]/certsrv, examine the certificate chain that's presented, and install the Root CA certificate. Once this is done, log in to the address you just visited with a domain admin account, and create a certificate using the template you just set up above. Make sure that the subject name is unique -- it doesn't have to be the FQDN of the system, nor even to be formatted as a DNS name, but if two systems get certificates with the same subject name, only the most recent system will be able to talk to ConfigMgr. Anyhow, once the certificate is installed, get access to the \SMS_SITE\Client folder -- via CD/USB drive, network share, whatever, open a command prompt in the folder, and run the following command:

ccmsetup /forcereboot /noservice /native:FALLBACK SMSSITECODE=[sitecode] CCMHOSTNAME=[server.fqdn] FSP=[server.fqdn]

Be prepared for the system to reboot itself with no warning -- if this is unacceptable, omit the /forcereboot switch, but remember that the ConfigMgr client probably won't run until the system is rebooted.

Part 3: Windows Server 2008 Active Directory Kerberos and LDAP Integration -- Linux User Account Setup

2009-04-20T11:29:00.000-07:00

This is the third post in a series on AD integration; the previous post deals with Active Directory installation and configuration

The Linux client I've chosen for the test lab is a 64-bit Fedora 10 installation. This distribution was chosen for its compatibility with my deployed systems. Like the Windows 2008 server, this machine is a VM, and the installation was a pretty generic GUI install. My deployed systems are installed via kickstart and managed with Puppet, but to maintain control over the environment, all test lab installation and configuration was done manually.

DHCP and DNS

Once the initial installation was done, I logged in as root and began to configure the system. Although most of my client systems are static IPs, I've decided to try using DHCP. The first configuration change I've made is to set the hostname in /etc/sysconfig/network, adding a line:

HOSTNAME=<client>

and to ask the system to send it's hostname with DHCP requests in /etc/sysconfig/network-scripts/ifcfg-eth0:

DHCP_HOSTNAME=<client>

After setting both of these, I rebooted the system to have it pick up the new name. Once the system was back up, I logged in to verify it's hostname, and checked on the DNS server to verify that it was registered in

Network Time Protocol

Kerberos requires clients to be reasonably closely synchronized to the server; NTP is the obvious choice for this function, as it's enabled by default in Windows server, and can be set up on fedora clients with the following commands:

echo 'server <ad>' > /etc/ntp.conf
chkconfig ntpd on
service ntpd start

Shortly after turning ntpd on, notifications should appear in /var/log/messages regarding the state of time synchronization.

In the test lab, I found that the NTP clients didn't respond well to NTP synchronization; this is a known issue with VMware virtual machines, so I've disabled NTP and rely on VMware tools to keep the time sync'd.

OpenLDAP

OpenLDAP configuration is handled by /etc/ldap.conf, which so far has been the trickiest bit of configuration to figure out, so mention of a couple of troubleshooting steps is in order. First, I found it useful to install the openldap-clients package, which includes the ldapsearch utility that can be used for making arbitrary queries. Secondly, appending the line

debug 1

to /etc/ldap.conf provides a wealth of information that has been useful in troubleshooting connections -- the output is sent to your terminal when using 'finger', 'id', or 'getent'. Higher debug levels can be chosen, but for my purposes 1 was usually sufficient.

But back to /etc/ldap.conf; mine looks like this:

uri ldaps://<ad server>:636/
base dc=fqdn,dc=com

ssl start_tls
ssl on
tls_cacertdir /etc/openldap/cacerts

nss_initgroups_ignoreusers root,ldap,named,\
avahi,haldaemon,dbus,radvd,tomcat,radiusd,\
news,mailman,nscd,gdm,polkituser

nss_base_password ou=people,?sub
nss_base_shadow ou=people,?sub
nss_base_group ou=groups,?sub?&(objectCategory=group)(gidnumber=*)
nss_map_objectclass posixAccount User
nss_map_objectclass shadowAccount User
nss_map_attribute uid sAMAccountName
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_attribute shadowLastChange pwdLastSet
nss_map_objectclass posixGroup group
nss_map_attribute uniqueMember member

#debug 1

The 'nss_initgroups_ignoreusers' line is quite important -- without it, Fedora 10 will fail to boot properly into GUI mode. Note also the reference to /etc/openldap/cacerts -- this directory needs to have a copy of the root certificate for your PKI, whether it's internal or third-party.

Name Service Switch

Configuration of NSS is pretty straightforward; /etc/nsswitch.conf should have the passwd, shadow, and group lines modified:

passwd: files ldap
shadow: files ldap
group: files ldap

At this point, you should be able to list user accounts -- try 'getent passwd' and look for your Active Directory accounts at the bottom of the list; if you don't see them, then uncomment the debug directive in ldap.conf, and see if the additional info gives you any clues. 'ldapsearch -x' may also provide some clues.

Kerberos 5

Kerberos deals with realms, which are kerberos administrative regions, and domains, which are DNS domains. By convention, realms are referred to in uppercase,

/etc/krb5.conf:

[libdefaults]
default_realm = FQDN.COM
ticket_lifetime = 24h
forwardable = yes

[realms]
FQDN.COM = {
kdc = <ad server>:88
}

[domain_realm]
fqdn.com = FQDN.COM
.fqdn = FQDN.COM

[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}

Pluggable Authentication Modules

/etc/pam.d/system-auth:

auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet
auth sufficient pam_krb5.so use_first_pass
auth required pam_deny.so

account required pam_unix.so broken_shadow
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid <>
account [default=bad success=ok user_unknown=ignore] pam_krb5.so
account required pam_permit.so

password requisite pam_cracklib.so try_first_pass retry=3
password sufficient pam_unix.so sha512 shadow \
nullok try_first_pass use_authtok
password sufficient pam_krb5.so use_authtok
password required pam_deny.so

session optional pam_keyinit.so revoke
session required pam_limits.so
session [success=1 default=ignore] pam_succeed_if.so \
service in crond quiet use_uid
session required pam_unix.so
session optional pam_krb5.so

Once all of these changes have been made, Active Directory users should be able to authenticate on the new system. Since I use auto-mounted home directories, I don't have PAM creating home directories for new users, but that's not difficult to set up; look into the mkhomedir.so module if this sounds useful in your environment.

Part 2: Windows Server 2008 Active Directory Kerberos and LDAP Integration -- Active Directory Server Setup

2009-04-16T11:52:00.000-07:00

This is the second post in a series on AD integration; the previous post provides an overview of the series; the next post deals with configuring user account authentication on Linux

Basic Installation

For my initial Active Directory server, I chose to use 64-bit Windows Server 2008 Enterprise. The decision to use Enterprise was guided by my intention of eventually supporting an internal PKI for Windows clients, rather than for anything related to the core Active Directory functionality used in this project. As an aside, this initial domain controller which will serve as the primary DC, is a virtual machine running under VMware ESXi. After the initial install reboot, I took care of housekeeping tasks including naming the server, setting a static IP address, installing VMware tools, setting the administrator password, and configuring Windows Updates.

Once this was done, I began the process of installing Active Directory using the dcpromo.exe command. Some of the choices I made:

Don't use 'advanced mode' installation
create a new forest
set the forest functional level to 2008
install DNS
reboot upon completion

After the reboot, I also configured the server as a DHCP server for its subnet. This particular system in is a subnet that is contained to a single server room, and most of the IPs on the network are statically assigned. In environments such as this with good physical security, I like to have a small (5-10 IP) range of addresses available via DHCP to simplify the process of setting up new systems and appliances; and in this case, test lab clients. To support DNS resolution for the test client systems, I set the DHCP scope options for DNS to always request DNS records, and to request DNS updates even if clients do not ask for it; these two settings will allow me to use DHCP clients during testing. Finally to authorize the DHCP service to make DNS updates, I created a service account, opened an administrative command prompt, and ran the following command:

netsh dhcp server set dnscredentials *

the '*' indicates that I wanted to provide the password interactively -- it can be replaced by the password itself for scripted use.

Identity Management for UNIX

Once this was done I moved on with configuring the server for integration. Using the Role Services pane of Server Manager, I installed 'Identity Management for UNIX', which makes some configuration changes allowing required LDAP attributes (such as uid, home directory, etc) to be set using the standard ADUC tools; some resources I've seen on the web indicate that the Active Directory schema is RFC 2307 compliant without the 'Identity Management for UNIX' addition, but I've never tested it, and don't think it's worthwhile to experiment with unless you never plan to use the Active Directory Users and Computers tools. After installation, another reboot is required.

Anonymous Binds

The next step was to set up OUs for accounts and groups. I chose to stick with UNIX conventions and create an OU called People for user accounts, and one called Groups for groups. The purpose for creating separate groups (ie, not using the CN=Users container) was because I've found it valuable to allow anonymous binding for account information, and doing this makes it easier to restrict this access to the most restrictive settings possible.

Enabling anonymous binds requires two steps. First, open Active Directory Users and Computers, check the Advanced Features option on the View menu, right-click properties for the forest root, and select the Security tab. Add the 'ANONYMOUS LOGON' user and give it read access, then click the Advanced button, select the 'ANONYMOUS LOGON' user and choose Edit; the 'Apply To' drop-down box needs to be set to 'This object and all descendant objects'.

Even after the ANONYMOUS LOGON account has been given access, Windows 2003 and 2008 will by policy still deny anonymous binding. Changing this setting requires the use of adsiedit.msc, which is included by default in Windows Server 2008. After launching adsiedit.msc from the Start Menu, right-click the 'ADSI Edit' icon and choose 'Connect To...' then from the 'Select a well known Naming Context' drop-down, choose Configuration. Drill down to 'CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration' and open the properties dialog for that object. Locate the dSHeuristics attribute and note what it's set to. If it is not set -- on a freshly installed machine it shouldn't be set -- it should be the 7-digit string '0000002'. If it is already set, the last digit should be set to '2' and all others should be left alone.

Secure Sockets Layer

Although I intend to run an internal PKI from the Active Directory server, for now I plan to set up a commercial third-party SSL certificate to secure LDAP. So far I've identified this as a potential problem down the road -- Microsoft does not specify any way to choose which of multiple certificates a server will use for LDAPS, so once I add the PKI I'm not sure that LDAPS will continue to use the third-party certificate -- if this turns out to be the case, in production I'll have to create the internal CA at this point and make it's root certificate available to clients, rather than downloading the commercial CA root certificate.

But for now, installing the third-party certificate can be done using the Certificates MMC snap-in. Open mmc.exe and add Certificates; choose 'Computer Account', then 'Local Computer' and click Finish. Expand the 'Certificates (Local Computer)' tree, right-click on Personal, and choose 'All Tasks'->'Import...'. Walk through the wizard to implement your certificate (you'll need it in a PKCS#12 format). After the certificate has been imported, it will be visible under the Personal certificates pane. Rebooting the server is all that is needed to enable LDAPS.

Users and Groups

At this point the domain controller is pretty much all set up for basic authentication. The next step is to populate at least a minimum set of user and group accounts. The user accounts should be normal users created in the OU=People, and the groups should be global security groups created in OU=Groups. Note that due to Active Directory naming restrictions, this won't allow you to use the RedHat user-group convention, in which each user's primary group is the same as the user's name, and any additional memberships are secondary. Since this model isn't used in my environment it barely has any impact on my setup, but if you're used to using this group model, you'll need to plan around this. After creating a new user or group, open the properties dialog for the object, select the UNIX Attributes tab, select the "NIS domain" corresponding to your domain name, and set the required attributes for the object.

Conclusion

That's pretty much it as far as Active Directory configuration goes. I anticipate adding a section on setting up Kerberos service and host principals once I get NFS working, but next up will be the scoop on configuring Linux clients to allow user authentication.

Part 1: Windows Server 2008 Active Directory Kerberos and LDAP Integration -- Preface

2009-04-16T11:48:00.000-07:00

This is the first post in a series on AD integration; the next post covers installing and configuring Active Directory

Systems administrators who work in heterogeneous environments understand both the value and the challenges of integrating user account information for the variety of platforms that they manage into a single system. My goal is to provide a series of technical notes detailing the processes I'm using to provide a single identity management system based on Active Directory that supports authentication for Linux, Solaris, and Mac systems -- in addition to Windows clients, of course. I'm hoping also to provide information on our use of CIFS and NFS to provide home directories to all of these platforms as well.

To start out with a bit of background, my employer has been maintaining dual account systems for many years for Solaris and Windows clients. Initially we used separate NIS and Active Directory accounts, both accounts had to be set up separately, and users were responsible for remembering both passwords, or for manually synchronizing them if desired. Shortly after I was hired, a coworker and myself were asked to look into tying together these two systems. At the time, Sun had just released version 1.0 of their Identity Synchronization product, which tied together Active Directory with Sun Directory Server. After a lot of communication with Sun's engineers we were able to put together a system that allowed us to create accounts in LDAP, and have them automatically populated to Active Directory, and to have password changes populate immediately in either direction. This solved most of our problems, most importantly simplifying our users' situation -- from their perspective, there was only one account that they had to deal with. This system matured over the years, going through various upgrades of the products involved, and expanding to support Linux systems.

Behind the scenes, however, there were still two account systems. The illusion of a single account would be occasionally shattered when a user's password didn't get synchronized properly; the system was pretty robust, but subtle differences in password strength and expiration policies, or unnoticed PKI issues would crop up, causing headaches for users and for me. As these problems continued to crop up at infrequent intervals, I began to do some research into what would be required to take the next step in evolving our network infrastructure, by truly consolidating to a single user directory.

I was initially encouraged by the number Google results that came up when I started looking at the problem -- dozens of HOWTOs, forum threads, and blog posting describing the process of authenticating all of the platforms I used against Active Directory. When I actually started setting up a test lab, however, reality hit me hard. As is often the case in this kind of situation, things that 'just worked' for other people 'just didn't' for me -- I tried many different scenarios, tweaking the suggestions in HOWTO and attempting to mix in fixes from the forum posts. When I finally made some progress, I promised myself that rather than sticking to my usual documentation procedure of writing a cookbook that worked for my exact situation and limiting it to our internal documentation, I'd try to write my own heuristic HOWTO, with enough depth and detail to hopefully be useful to others who are trying to accomplish similar goals. So, my first post is going to detail the setup of the Active Directory server, then I'll talk about setting up the various clients systems. I'll consider posting some info on ancillary processes such as account provisioning, home directory management, etc.

Using rsync for Windows backups

2009-02-21T22:02:00.000-08:00

OK, so this is a little less about crossing over between programming and system administration, and more about applying *nix tools to a Windows environment. The problem? We recently switched to a new backup system at work, and no longer have Windows client licenses available to us. Since most of the data that our Windows users produce lives on network attached storage rather than the Windows systems themselves, this isn't a huge deal, but we do want to backup our Active Directory server, as well as a few one-off systems. Rather than shelling out the cash for some extra clients, I decided to look into using rsync to get the data from Windows onto the NAS.

I've looked at rsync+win32 in the past, and usually came away disappointed. There are a lot of rsync and rsync-like products for win32, and some of them are quite good. However, my requirements were:

It must use the rsync protocol, not an imitation or 'improved' protocol
the rsync conversation must be tunnelled over ssh
authentication must be done using ssh keys
backups should run on a scheduled basis, whether users are logged in or not
status emails must be sent after each backup

This narrowed the field quite a bit; ssh support was rare, and key-based authentication effectively limited me to using a cygwin-based solution. I almost gave up at this point and started evaluating other ways move move the data, because I've never been a big fan of maintaining cygwin installations; no matter how elegant a cygwin solution is, it always ends up feeling like a hack to me. However, after giving it some more thought, I decided to see how far I could strip down a cygwin installation, down to just the binaries and dlls needed to actually perform a backup. The key point for me was the last requirement I had -- only one of the rsync-like systems I'd come across offered status emails, whereas I figured using a micro-cygwin installation would allow me to include things like email pretty easily. This didn't end up working exactly as I had expected, because cygwin's SMTP support isn't as good as I'd expected it to be, but even so, it was enough to convince me to keep going.

Actually working out the nuts and bolts of the script took me a bit, but wasn't terribly difficult; I did a cygwin install, then started copying out into a new directory the executables I'd need (bash, ssh, rsync, etc) and tried running them. Then I actually tried to run them; one after another they started complaining about dlls, but once I got the half dozen or so required dlls, things started falling into place. I generated an ssh key, prepopulated a known_hosts file for ssh, and started rsyncing files up to the server. Then I tried to email out my log file, and ran into a brick wall; if there's a lightweight way to do SMTP from cygwin, I haven't found it. My solution was to cheat; since I was already using ssh connections to a full-fledges *nix server, I just made one more connection once I was done and piped the log file to mail.

backup.sh

Configuration 101: Using Subversion for Change Management

2008-08-18T15:31:00.000-07:00

Whereas SSH Keys come primarily from the sysadmin realm, the next tool I'd like to introduce is most often associated with the programmer's toolkit. Subversion (SVN), or more generally, revision control software, allows you to do away with littering your production systems with tons of backup files, and consolidates your files into one place that's easy to backup, duplicate, and modify. Programmers typically stuff source code, makefiles, and supporting documentation into SVN; I use it not only for that, but for /etc/ files, kickstart configs, shell scripts, and more. Some of my systems -- in particular my puppet server -- pull from subversion via a cron job, others use puppet to push files as needed, and others I push files to manually. Eventually I hope to put a great deal more data in SVN, such as much of our website, but for now, I'm starting out slow.

So, what to do?

Find a reliable space to store your repository, which is essentially the authoritative copy of your files. This space needs to be reliable, and should be easy to back up. Subversion (and most other RCS systems) will work with a variety of access types, but primarily I find direct filesystem access over NFS and remote access via SSH to be the most common use cases*. For the purpose of this discussion, we'll call it /export/svn/foo
Install subversion on the chosen system. Most *nix systems will have a subversion package; if not, there's a link to their website at the top of this post.
Run svnadmin to initialize the repository. 'svnadmin help create' will give you some info on the options available to you, but generally 'svnadmin create /export/svn/foo' is the form of command you want here.
Install subversion on your client system; soon I'll be posting information on good Windows-based clients for subversion, but right now I'm going to punt on that and assume we're using *nix.
Check out a working copy of your empty repository:
cd ~
mkdir svn
svn co file:///export/svn/foo ~/svn/foo
-- or over ssh to a server called myhost.com --
svn co svn+ssh:///export/svn/foo ~/svn/foo
Copy files into the working copy, or start creating new files.
Add the files into SVN, using svn add ...
Commit your changes back to the original repository, using svn commit -m "Log Message Goes Here" ~/svn/foo/

Obviously, there's a lot more to managing source files, but this is a start. Note that if you're using SSH to connect to a remote repository, SSH keys will save you a lot of typing!

* If you've ever used a code.google.com project, you may have noticed that they use SVN. If you have commit access, you'll use authentication over HTTPS to do commits. SVN is capable of storing your password in your working copy in this case, but it's beyond the scope of this particular post; go forth and Google it!

Automation 0.1 (alpha): Using SSH Keys

2008-08-11T08:31:00.000-07:00

I'm starting out simple. No, really. I can't imagine that anyone who does system administration isn't already using ssh to replace r[shcplogin], telnet, ftp, and to secure a whole host of other protocols using tunneling, but when I talk to other geeks (or shoulder surf as they work), I've noticed that a lot of them don't seem to realize the flexibility that comes from using keys to log in to ssh. So here's a quick rundown of the benefits (to me):

I don't have to type my password in 5000 times a day, as I hop from machine to machine. My laptop and my desktop both have one key on them, and all of my core servers have another key for my account, and between these two, I mostly only enter my password to unlock the desktop or laptop.
For systems with equivalent security levels, I can put an ssh key on the root account of one (or all) of the systems, and allow it to log in to all the others via scripts/cron/etc without human interaction. When I say equivalent security levels, allow me to explain: I do this mostly on systems which don't store differing levels of important data, which share the same physical access, and for which I have a reason to have them communicate. For example: a teaching lab with a small compute cluster ~ the head node has root access to all of the compute nodes.
With less worries about security, relatively unprivileged service accounts can share a common ssh key to automate routine communication between systems -- I use subversion to distribute a variety of data, and allowing an svn account to do svn+ssh:// pulls from a cron job is very useful. Liekwise, my webservers all have shared keys for the www account, so they can push and pull data directly. If I gave www sudo access to apachectl, then they could also restart each other's httpd processes as needed, but so far I haven't found much use for that.
So what if you ARE worried about security, and you have less trusted systems that you want to have access to more trusted ones? Passphrases can help to secure ssh keys, and a great deal of scripting is still possible. Essentially, keys can be encrypted with a passphrase, and unless that phrase is entered, the key is useless. So the key can be stored in an insecure area -- hell, with a good enough passphrase it can be lost -- and the key's access will remain secure at least until someone cracks the passphrase. How do you automate this? With ssh-agent and ssh-add, a pair of utilities that allows a computer to store your ssh key's passphrase as long as your unix session is active. So as an example, I have a backup script that runs for about 5 days at the beginning of every month (lots o' data, not so much bandwidth). In the script, I first check to make sure I'm running within screen, then I load ssh-agent, and use ssh-add to ask for the passphrase. This all happens right at the top of the script. Hours pass as the backups are taken, and then days go by between file transfers, but the files are copied securely and automatically, without me having to be there for the entire process. Most importantly, no one who doesn't have the passphrase can do anything harmful with the script, and as soon as the copies are done, the passphrase is wiped from memory.

So, on each one of those points, I really felt a need to put a footnote or two, mostly about security. Basically, though, it boils down to this: unless you're using an one time password system or some sort of hardware security module, you have to trust root on the computer you're using. For that matter, you generally have to trust root a little bit even in those cases. SSH keys can improve security a lot, but they aren't completely unbreakable. The computer you're using could have a keylogger installed to capture your passphrase, and it's pretty trivial for root to grab your ssh key. In short: don't try to do system administration from untrusted systems. Period. I trust my laptop, I trust my desktop, and I trust most of my servers most of the time, but that's about it. If I'm away from work and away from my computers, I prefer to make a phone call and walk a coworker through a task rather than doing it myself from, say, a kiosk. Another option is to use a bootable live-cd, which I've done on rare occasions in the past. Okay, maybe I've convinced you that ssh keys are useful tools in the right circumstances; so how do you get started?

First, on your trusted client system, use ssh-keygen to create a key encypt it with a passphrase. Do the passphrase thing -- if you find it to be onerous, you can always take it off later, and you don't have to redeploy the key. If you REALLY don't want a passphrase, just leave it blank. This will create two files, by default ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub. The first of these is the private key; the file that is encrypted with the passphrase, and the one that should be safeguarded.
The other file is the public key, which should be appended to the ~/.ssh/authorized_keys file on any system you want to connect to. If you don't already have any ssh keys, you can just rename or copy the file. Otherwise, each key is stored on one (very long) line in the file, and typically has a text field at the end signifying where the key was created to help with identification. For the most part, I leverage our shared home directory structure at work to make sure keys are distrubuted, but in cases where accounts have local home directories, I make other arrangements. For occasional one-off systems, I copy the keys using sftp with (gasp) password authentication. For other situations, I incorporate key management into the rest of my automation. So, for the teaching lab cluster I mentioned above, I distrubute the keys using puppet, another one of my system configuration tools, which uses its own PKI setup to keep everything on the up-and-up.
If you're using a passphrase and want to store it, load up ssh-agent. This is a slightly odd utilty because it's designed to run from a variety of environments (X, terminal, etc). Read the docs on the best way to load it for your situation, but from a terminal, "eval `ssh-agent`" will usually do the trick. You should see a single line of output that resembles "Agent pid #####" if you did it corretly. If you see three or four lines of output, you probably did something wrong -- note the backticks above!
Finally, run ssh-add which will prompt you for your passphrase. Once you enter it correctly, your key will allow you to log in without prompting to systems which have your public key installed.
When you're done, "ssh-agent -k" will kill the session.

Hello, World!

2008-08-08T12:19:00.000-07:00

I have a tradition of starting new projects with the phrase "Hello, World!" To any student of programming, the reference will be obvious, but for the benefit of all you non-computer-geeks who may mistakenly stumble across this blog and actually stay for a few minutes, I'll explain. Besides being an appropriately expansive greeting for a new blog, "Hello, World!" is, by convention, used to introduce students of any programming language to the fundamental features of that language. By learning how to write a program that simply displays that phrase (or any other), you can learn a lot about the way programs are constructed in a new language. Over time, Hello World becomes a very simple rosetta stone when trying to pick up a new programming or scripting language.

So, why another 'Hello from me? I'm frankly not a very good blogger; every now and then I write a post on my personal blog, but mostly it sits there, displaying the old stuff day after day. I don't really think that's going to change ~ and frankly, I don't think that "make make" will be all that much better ~ but what I do want to do is give myself another, slightly different, place to put stuff that is less about me and more technical. I am a system administrator who has a lot of training as a programmer, and as such, I find that there's not much of a niche for me in geekdom; most people identify as one or the other, and there is often a hint of animosity between the two groups. Regardless, there is a lot that each side can learn from the other, and my goal is to post the bits and pieces that I've learned about both sysadmin-ing and coding in hopes that someone, somewhere, will find a few of them useful.