The brain of my computer is a “vintage” Pentium 4 (Prescott) 2.8 GHz processor, from an era before there were “cores”, and the only measure that mattered was the “Gigahertz”. As the cliched story goes, it was “top of the line” when I got it, but it’s time to upgrade.

Actually, while I was tempted to get a “top of the line” at the time, I reasoned that I was better getting “second place”, which is what I got. In terms of life, I’m surprised it’s lasted me this long. Moore’s Law is often cited to mean that computers double in processing power every 18 months (or so). I had always thought I would end up doing full upgrades every 2 years. At work, we require servers to be upgraded after 3 years. Instead, this old rig has lasted me four years! The purchase date was easy to remember, too, as I upgraded back then for a very specific reason: to play Final Fantasy XI to its best looking.

Since then, upgrades have been minor. The hard drives had to be replaced early this year due to harassment by overweight felines. More RAM was added. The DVD burner replaced. And perhaps most key to my workstations’ longetivity, I gave up PC games and switched to Linux full-time; the former reduced my computer resource demands, the latter applied that available computing power in a more efficient manner.

While a craving for new toys certainly fuels my desire for a brand new workstation, what really sparked it was downloading via bittorrent and watching videos. Running bittornado for only 5 torrents consumes most of the CPU time. Watching an HD video is jerky. Coincidentally, the driver for my last major upgrade was similar: jerky video when playing FFXI. I’ve very particularly about my computer’s responsiveness to me; control issues perhaps.

Thus I’m forced into upgrading my computer. All of it. To the “top of the line”. Didn’t really want to, but … yeah, I’m not fooling myself either. I need to polish my credit card.

Once upon a time, there was ICQ.

Then came Yahoo Messenger, which could work through the university firewall, and linked into stock quotes.

Briefly, a fling with AOL Instant Messenger, which was regionally popular during my sojourn to Seattle.

MSN Messenger then became another, as nouveau “Netizens” used the instant messaging client that was preinstalled on Windows.

When Trillian appeared, I was overjoyed to consolidate all my messaging clients into one.

On and off, I toyed with other clients, like Miranda, but it wasn’t until the appearance of Gaim that I switched again. Gaim, with its simple interface and resource friendly ways, appealed to me much more than the flashy excesses of Trillian.

Also around the time of GAIM, I got second accounts for Yahoo and MSN: one for work contacts, one for personal contacts.

Then came the AJAX revolution, and I discovered Meebo. At last, everything I wanted: instant messaging for all networks, no installation required, centralize all my chat logs!

Except now… lost messages! I don’t know if it’s Meebo’s fault or the MSN network, but I’ve been sending messages that I later discovered went unreceived. So time to backtrack to the last known good state (Gaim, now called Pidgin) and see what else has changed besides the name.

Initial impressions are good. Smooth install, logical interface, and certainly more responsive than using a web-based IM. Since I still want all my log files in one place for searchability, and Pidgin doesn’t support remote logging yet, I put together a script to commit the local log files (plain text files) to a revision control system every night.

I’ve got the “all-green” to go to San Francisco for the 2008 TIBCO User Conference on behalf of my employer. The conference runs from April 29th to May 2nd, 2008. Naturally, I’ve extended my time in San Francisco to spend some R&R with west-coast friends, catching up and the usual.

Thie time, however, instead of staying at the San Francisco Mariott, where the event is being held, I’ve invited myself over to Averal’s and Firion’s place. I heard the offered accomodations exceed those of any four-star hotel, and I’m breathless with anticipation.

I’ll see if there are any other technical conferences during that time period. I’m often envious of the breadth and depth of conferences in Silicon Valley;Toronto just doesn’t have the same oomph.

I’m really looking forward to talking shop with people who seem to share the same passion as I do. It was pretty fun last time, and this time I have experiences of my own with TIBCO’s software to share.

Naturally, after being hacked, my thoughts turned to prevention and detection tools. My experience with such tools, though, was fraught with false positives and frequent emails that I would simply skim and delete. Considering that the attack vector was successful due to a very weak password, I was looking for simple techniques to act as a safety net that weren’t too noisy on a regular basis.

There’s a lot of information and tools out there, more of them and more mature than the last time I looked a few years ago. Rather than go over what I looked into, it’s probably better to simply consider what I decided to use.

BTW, my server runs on Debian (testing); my workstations run on Xubuntu. The reason I switched to the Debian line is because of the superior administration, operations, and maintenace model ( i.e. apt-get). Similarly, I limited my tool use to what was available in the Debian and Ubuntu archives (which are substantial anyway).

denyhosts is pretty simple, but effective. The server maintains a /var/log/auth.log that notes all attempts to authenticate, in particular all attempts to use ssh to connect to the box. denyhosts periodically analyses that log, and adds repeat offenders to the /etc/hosts.deny file. That file is read used by services like sshd and inetd to block certain IP addresses before they are allowed to exchange a single byte with a service.

denyhosts is an ideal balance of convenience, flexibility, and accuracy. As a periodic script, it has very low resource requirements. Its configuration file is quite flexible in defining what constitutes denial (and whether the denial is permanent or temporary). The rate of false positives is quite low; I’m unlikely to forget my password five times in a row. Lastly, it direclty addresses the most common attack vector on my server: brute force dictionary attempts on the SSH port. As a bonus, my logfiles aren’t filled with denial attempts now.

This completely disables the ability to log in via SSH with a password. I rarely log in with passwords anyway. Instead, I create a private SSH key on each computer that initiates a connection with my server, and add the associated public key to my ~/.ssh/authorized_keys file. This is a very common technique. There’s a chance I’ll be temporarily inconveniened if I’m using a foreign computer or if a workstation gets wiped, but in practice, I can’t recall that ever happening to me. There’s always root access via the remote console through my ISP, anyway. This completely prevents the automated dictionary attacks I fell prey to.

TIGER, or the ‘tiger’ scripts, is a set of Bourne shell scripts, C programs and data files which are used to perform a security audit of UNIX systems. TIGER has one primary goal: report ways ‘root’ can be compromised.

I find Tiger pretty painless. It has spurious warnings, but it remembers which ones it saw last time, and it won’t email me about old ones - only new ones. It’s also easy to configure rules to filter out what you don’t care about, too. I rebooted my server at one point, and on Tiger’s next run I was informed that my firewall was off. I forgot to enable it on boot - oops!

I wrote a simple shell script to read the output of ifconfig and capture the number of bytes sent and received since the last run. I run the script every 24 hours, and have it set to email me if the number of bytes exceeds a threshold. I figure, if my server gets compromised, it’s for the bandwidth that it has, so I’ll notice quickly if my bandwidth usage spikes.

To that end, I also installed bandwidthd, which generates a webpage of daily/weekly/monthly reports on where my bandwidth is going, i.e. who my server is talking to. It was originally a bid memory/cpu hungry, but after reading the documentation, I learned that it was much more reasonable if I turned off the graphs.

passwdqc is a replacement for the default passwd related programs. It can be configured to strictly (or leniently) enforce various rules on password length, complexity, and composition. Most usefully, I find, is that it will suggest complex passphrases to you when setting a new password. This alleviates the temptation to use insecure passwords for lack of imagination. Here’s an example:

You can now choose the new password or passphrase.

A valid password should be a mix of upper and lower case letters, digits, and other characters. You can use a 6 character long password with characters from at least 3 of these 4 classes, or a 5 character long password containing characters from all the classes. An upper case letter that begins the password and a digit that ends it do not count towards the number of character classes used.

A passphrase should be of at least 3 words, 8 to 40 characters long and contain enough different characters.

Alternatively, if noone else can see your terminal now, you can pick this as your password: “grade:sadly!hearty”.

These were all simple steps. The work was in discovering what was out there. I also took some other minor precautions to protect my data that I have on my server - encrypting files, restricting home directory permissions, etc. Anyone who hacks my server is unlikely to be interested in my personal files, but I find it an uncomfortable thought anwyay.

Now that I’m done all the work restoring and securing my server, I’m not completely unhappy that I was hacked; at least I learned a few things useful for the future.

Not so much particular to git, but a nice property of distributed source control systems is that every participant has a copy of the entire repository. While it may sound inefficient, copying the repository is a one-time cost, and at times like these where one’s central server gets hacked, reimaged, and otherwise lost, it’s handy.

Strictly speaking, git doesn’t have a central repository, but I use an account on my server where I push the changes from where I’m working to. After my server was hacked, I could’ve copied from the backup, but it was more interesting to get it from the clients.

My normal workflow on my workstation is:

1. Make edits.
2. git-commit to commit the changes to my local copy of the repository.
3. git-push to push all the new changesets to my server.

I use SSH as the transport, and SSH keys to avoid the use of passwords.

To recover:

1. git-init to create an empty repository on my server.
2. git-push to push all the changesets to my server again.

The first time I tried to recover I pulled it off a workstation with a much older copy of the repository, inadvertently simulating the scenario where different workstations have different, non-intersecting sets of changes. No problem. I repeated the same steps against my laptop, and got the rest of the changes. Since git stores everything as changesets and thinks of branches in a first-class way, there was no opportunity for conflicts.