RubyRescue has limited space available starting August 1st – a common area with a desk and multiple internet connections, and a cozy private office. We’re a group of developers, mostly focused on the same technologies, all in one space, near Parque Las Heras. A lot of ideas, projects, and opportunities turn up by working near each other – if you are interested, let us know. Contact martina@inakanetworks.com or 15.6940.7097…
Take the time to disable SSH password auth while you are reading this.
Posted by chad
on July 19, 2010
A client had a new hire start this week, let’s call him (or his login) ‘ted’.
Ted was issued a new account on production servers with a simple password, and was told to login, upload a public key, and change his password. Ted did this, but didn’t change his password for a few hours. Within this few hour window, every server with his account was compromised and an IRC server was installed.
The rest of the server was locked down so no other accounts or data were accessed, but the resource consumption from IRC clients connecting to the machine caused serious problems with the server for a number of hours.
9 hours – Total downtime.
4 hours – Time from hack to Rackspace noticing the servers were down, even though the client pays for URL monitoring.
Lessons learned
1. DO NOT use password auth on public servers, or setup a strong password policy that root can’t get around – someone can get lazy.
2. DO NOT trust Rackspace URL monitoring to alert you to a downtime problem – setup your own monitoring suite and alert/escalation plan. Use Rackspace monitoring as a backup only.
3. Remember that just having access to a user-account with no sudo powers is enough to bring down a server if some aspect of server resources (file handles, ports, memory, disk space, etc) are over-utilized.