A client had a new hire start this week, let’s call him (or his login) ‘ted’.
Ted was issued a new account on production servers with a simple password, and was told to login, upload a public key, and change his password. Ted did this, but didn’t change his password for a few hours. Within this few hour window, every server with his account was compromised and an IRC server was installed.
The rest of the server was locked down so no other accounts or data were accessed, but the resource consumption from IRC clients connecting to the machine caused serious problems with the server for a number of hours.
9 hours – Total downtime.
4 hours – Time from hack to Rackspace noticing the servers were down, even though the client pays for URL monitoring.
Lessons learned
1. DO NOT use password auth on public servers, or setup a strong password policy that root can’t get around – someone can get lazy.
2. DO NOT trust Rackspace URL monitoring to alert you to a downtime problem – setup your own monitoring suite and alert/escalation plan. Use Rackspace monitoring as a backup only.
3. Remember that just having access to a user-account with no sudo powers is enough to bring down a server if some aspect of server resources (file handles, ports, memory, disk space, etc) are over-utilized.