Ted was issued a new account on production servers with a simple password, and was told to login, upload a public key, and change his password. Ted did this, but didn’t change his password for a few hours. Within this few hour window, every server with his account was compromised and an IRC server was installed.

The rest of the server was locked down so no other accounts or data were accessed, but the resource consumption from IRC clients connecting to the machine caused serious problems with the server for a number of hours.

9 hours – Total downtime.
4 hours – Time from hack to Rackspace noticing the servers were down, even though the client pays for URL monitoring.

Lessons learned

1. DO NOT use password auth on public servers, or setup a strong password policy that root can’t get around – someone can get lazy.
2. DO NOT trust Rackspace URL monitoring to alert you to a downtime problem – setup your own monitoring suite and alert/escalation plan. Use Rackspace monitoring as a backup only.
3. Remember that just having access to a user-account with no sudo powers is enough to bring down a server if some aspect of server resources (file handles, ports, memory, disk space, etc) are over-utilized.

There’s a good article by Ryan Tomayko on Unicorn and the reasons why it’s “Unixyness” is a feature. If you’re a long-time unix user, you won’t find much to be revolutionary. If you’re a web-developer without a CS-background, he covers how unix fundamentals make Unicorn more reliable.

Before Unicorn we were running Passenger and apache, and we decided to switch to unicorn and nginx at the same time. It takes a bit more work to setup unicorn than passenger, but we’ve found the reliability to be worth it.

• gem install unicorn
• create a unicorn.rb file like this

rails_env = ENV['RAILS_ENV'] || 'production'

worker_processes 3

preload_app true

timeout 75

socket_path = '/var/www/APPLICATION/shared/sockets/unicorn.sock'
pid_path = '/var/www/APPLICATION/shared/pids/unicorn.pid'

listen socket_path, :backlog => 2048
pid pid_path

# http://www.rubyenterpriseedition.com/faq.html#adapt_apps_for_cow
if GC.respond_to?(:copy_on_write_friendly=)
  GC.copy_on_write_friendly = true
end

before_fork do |server, worker|
  old_pid = RAILS_ROOT + '/tmp/pids/unicorn.pid.oldbin'
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
    end
  end
end

after_fork do |server, worker|
  ActiveRecord::Base.establish_connection

  begin
    uid, gid = Process.euid, Process.egid
    user, group = 'admin', 'admin'
    target_uid = Etc.getpwnam(user).uid
    target_gid = Etc.getgrnam(group).gid
    worker.tmp.chown(target_uid, target_gid)
    if uid != target_uid || gid != target_gid
      Process.initgroups(user, target_gid)
      Process::GID.change_privilege(target_gid)
      Process::UID.change_privilege(target_uid)
    end
  rescue => e
    raise e
  end
end

• Your deploy.rb will get a bit more complicated as it needs to send the proper signal to the Unicorn master to signal it to reload the workers.

namespace :deploy do
  task :stop, :roles => :app do
  run "cd #{current_path} && kill -QUIT `cat tmp/pids/unicorn.pid`"
end 

task :start, :roles => :app do
  run "cd #{current_path} && /opt/ruby/bin/unicorn_rails -c config/unicorn.rb -E production -D"
end

desc "restart unicorn"
task :restart, :roles => :web do
  run "cd #{current_path}; [ -f tmp/pids/unicorn.pid ] && kill -USR2 `cat tmp/pids/unicorn.pid` || /opt/ruby/bin/unicorn_rails -c config/unicorn.rb -E production -D"
end

Tango Rails is meeting for some drinks and to chat about Ruby on Rails, Sinatra or other interesting things happening with web frameworks and Ruby these days. It’s a very informal after office and a great place to meet people in person working on interesting projects.

Link to the Facebook Invite

The server has the whole stack – Apache, Passenger, MySQL, however once we migrated to Rails we simply had to start thinking about performance in a way we haven’t before. I’m not going to argue that this is a reason not to use Rails – I believe that if we were to have moved to Symfony or another PHP framework we would have been dealing with the same issues – but there are more moving parts in the migrated application and the complexity of deployment means more time is spent thinking about the server than we would want.

When the server was purchased, we asked for a 2-Proc RAID 5 box with 4 GB of memory. The client gets a lot of traffic and is further sensitive to the perceived overhead of multiple box management vs ‘one big box’. That’s fine with me, we setup one box and let it roll. However we didn’t exactly request a 4 CPU box – we actually started with a 2 CPU box with 2GB of memory and then decided to upgrade to bigger hardware after only a few weeks with the smaller box.

No problem – Rackspace took the machine down, swapped the motherboard, added the memory and we were good.

Fast forward to the Rails upgrade 1.5 years later. The box performance has generally been fine but in the back of my mind something wasn’t quite right about the actual server. Should i have benchmarked and solved this nagging feeling? Probably, but i’m not as familiar with Redhat as Ubuntu/Debian and so part of my feeling that it wasn’t performing as fast as it should i blamed on something related to RHEL4. But it’s a 4 CPU box and it has lots of memory, Rails should be fine with Apache and Passenger.

But it wasn’t.

We had some long running requests – Ruby seemed to get backed up behind – something? No problem, let’s make sure we’re doing everything we can to enable page caching – ok page level caching pretty much solved any performance issues. But still – deployments using capistrano clear the cache and the process of rebuilding up to 5,000 pages in the filesystem wasn’t fast. Deployments at particular times are out. Performance is good but not great, but not enough of a problem to cause anyone to question the configuration of the server. But that “PHP Feeling” of being able to deploy a change and have it instantly available just wasn’t there. This wouldn’t be a problem if the team wasn’t used to this flexibility of making changes at any moment, and I wasn’t happy with having to make an excuse for moving to Rails and ending up with a less flexible operation.

Our concerns, and experiences with other deployments, lead us to think that Apache/Passenger was the source of some of the slowness, so we decided to move to unicorn and nginx. The converstion from Apache to Nginx freed up a lot of memory, and unicorn’s graceful restart capability smoothed out performance during deployments. In part 2 we’ll cover in detail this conversion. It was a total success – overall throughput went up by a factor of 3, and deployments are now much smoother.

However, were things as fast as they could be?

In investigating an unrelated issue, we followed up with Rackspace on a Kernel patch that couldn’t be applied to our server. One of the technicians immediately realized why – we were not running the SMP kernel. For almost two years our 4 CPU Racehorse has been hopping on one leg around the track. Let’s look at the numbers: 9 am this morning, we switched to the proper kernel:

Every request is now almost twice as fast. Apache Bench says our throughput is 3.5 times greater. That stings a little. It feels a bit like i just piloted the Shenandoah 22,000 miles around the globe, winning every battle, only to find out that the team lost the war a few months before…

What should we have done differently? None of us are truly full-time ‘operations guys’ – but we’re all really good developers and ‘good enough’ at operations. We have no problem with configuring linux init files, syslogs, virtual ethernet interfaces, iptables, benchmarking tools, netcat, you name it – we’re pretty confident with the toolset. So why did we never cat /proc/cpuinfo ? And why did it take Rackspace 22 months to figure out that we were running a misconfigured server? Why did I have a spidey-sense (or is that spidy (spidie?) -sense?) that something was wrong with this server? And why didn’t I track it down as thoroughly as I could have? Why did I focus on the software vs hardware? But shouldn’t I be able to trust that Rackspace has that covered? What is the proper remedy for this failure from Rackspace? (They refunded the difference in price between the requested and ‘actual’ hardware without hesitation.) Would we have made the same software changes if we were getting ‘good enough’ performance from an actual 4 CPU system? Would we just be happily running on Apache/Passenger?

Answers to some of those questions in part two…

mysqldump -uroot -p rubyrescue | sed
's#rubyrescue.com#rubyrescue.com/blog#g' > temp.sql
mysql -uroot -p rubyrescue < temp.sql

The only other item was dealing with permalinks. The error handler for lighttpd was set to look for index.php in the root. I had to change that to look for it in /blog.

One concern i have is that without a static IP address, if you have a highly SEO-sensitive application it’s probably not the best choice, because a static IP is just one of many factors in your pagerank, but an important one.

1. This OpenIdAuthentication error:

rake aborted!
uninitialized constant Rails::Plugin::OpenIdAuthentication
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:105:in `const_missing'
/home/chadd/foo/vendor/plugins/open_id_authentication/init.rb:16:in `evaluate_init_rb'

The solution is to comment out line 16 in vendor/plugins/open_id_authentication/init.rb when running rake db:migrate.

Updated: Better solution in the comments, thanks!

chadd@ubuntu:~/foo$ rake db:migrate --trace
(in /home/chadd/foo)
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:migrate
==  BortMigration: migrating ==================================================
-- create_table(:sessions)
   -> 0.0017s
-- add_index(:sessions, :session_id)
   -> 0.0004s
-- add_index(:sessions, :updated_at)
   -> 0.0003s
-- create_table(:open_id_authentication_associations, {:force=>true})
   -> 0.0021s
-- create_table(:open_id_authentication_nonces, {:force=>true})
   -> 0.0016s
-- create_table(:users)
   -> 0.0027s
-- add_index(:users, :login, {:unique=>true})
   -> 0.0006s
-- create_table(:passwords)
   -> 0.0017s
-- create_table(:roles)
   -> 0.0007s
-- create_table(:roles_users, {:id=>false})
   -> 0.0007s
==  BortMigration: migrated (0.5497s) =========================================

** Invoke db:schema:dump (first_time)
** Invoke environment
** Execute db:schema:dump
chadd@ubuntu:~/foo$

2. application.rb was renamed to application_controller.rb in Rails 2.3 and if you don’t rename it you get this error:

Loading development environment (Rails 2.3.2)
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:443:in `load_missing_constant':NameError: uninitialized constant ApplicationController