Ted was issued a new account on production servers with a simple password, and was told to login, upload a public key, and change his password. Ted did this, but didn’t change his password for a few hours. Within this few hour window, every server with his account was compromised and an IRC server was installed.

The rest of the server was locked down so no other accounts or data were accessed, but the resource consumption from IRC clients connecting to the machine caused serious problems with the server for a number of hours.

9 hours – Total downtime.
4 hours – Time from hack to Rackspace noticing the servers were down, even though the client pays for URL monitoring.

Lessons learned

1. DO NOT use password auth on public servers, or setup a strong password policy that root can’t get around – someone can get lazy.
2. DO NOT trust Rackspace URL monitoring to alert you to a downtime problem – setup your own monitoring suite and alert/escalation plan. Use Rackspace monitoring as a backup only.
3. Remember that just having access to a user-account with no sudo powers is enough to bring down a server if some aspect of server resources (file handles, ports, memory, disk space, etc) are over-utilized.

There’s a good article by Ryan Tomayko on Unicorn and the reasons why it’s “Unixyness” is a feature. If you’re a long-time unix user, you won’t find much to be revolutionary. If you’re a web-developer without a CS-background, he covers how unix fundamentals make Unicorn more reliable.

Before Unicorn we were running Passenger and apache, and we decided to switch to unicorn and nginx at the same time. It takes a bit more work to setup unicorn than passenger, but we’ve found the reliability to be worth it.

• gem install unicorn
• create a unicorn.rb file like this

rails_env = ENV['RAILS_ENV'] || 'production'

worker_processes 3

preload_app true

timeout 75

socket_path = '/var/www/APPLICATION/shared/sockets/unicorn.sock'
pid_path = '/var/www/APPLICATION/shared/pids/unicorn.pid'

listen socket_path, :backlog => 2048
pid pid_path

# http://www.rubyenterpriseedition.com/faq.html#adapt_apps_for_cow
if GC.respond_to?(:copy_on_write_friendly=)
  GC.copy_on_write_friendly = true
end

before_fork do |server, worker|
  old_pid = RAILS_ROOT + '/tmp/pids/unicorn.pid.oldbin'
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
    end
  end
end

after_fork do |server, worker|
  ActiveRecord::Base.establish_connection

  begin
    uid, gid = Process.euid, Process.egid
    user, group = 'admin', 'admin'
    target_uid = Etc.getpwnam(user).uid
    target_gid = Etc.getgrnam(group).gid
    worker.tmp.chown(target_uid, target_gid)
    if uid != target_uid || gid != target_gid
      Process.initgroups(user, target_gid)
      Process::GID.change_privilege(target_gid)
      Process::UID.change_privilege(target_uid)
    end
  rescue => e
    raise e
  end
end

• Your deploy.rb will get a bit more complicated as it needs to send the proper signal to the Unicorn master to signal it to reload the workers.

namespace :deploy do
  task :stop, :roles => :app do
  run "cd #{current_path} && kill -QUIT `cat tmp/pids/unicorn.pid`"
end 

task :start, :roles => :app do
  run "cd #{current_path} && /opt/ruby/bin/unicorn_rails -c config/unicorn.rb -E production -D"
end

desc "restart unicorn"
task :restart, :roles => :web do
  run "cd #{current_path}; [ -f tmp/pids/unicorn.pid ] && kill -USR2 `cat tmp/pids/unicorn.pid` || /opt/ruby/bin/unicorn_rails -c config/unicorn.rb -E production -D"
end

Tango Rails is meeting for some drinks and to chat about Ruby on Rails, Sinatra or other interesting things happening with web frameworks and Ruby these days. It’s a very informal after office and a great place to meet people in person working on interesting projects.

Link to the Facebook Invite

mysqldump -uroot -p rubyrescue | sed
's#rubyrescue.com#rubyrescue.com/blog#g' > temp.sql
mysql -uroot -p rubyrescue < temp.sql

The only other item was dealing with permalinks. The error handler for lighttpd was set to look for index.php in the root. I had to change that to look for it in /blog.

One concern i have is that without a static IP address, if you have a highly SEO-sensitive application it’s probably not the best choice, because a static IP is just one of many factors in your pagerank, but an important one.

1. This OpenIdAuthentication error:

rake aborted!
uninitialized constant Rails::Plugin::OpenIdAuthentication
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:105:in `const_missing'
/home/chadd/foo/vendor/plugins/open_id_authentication/init.rb:16:in `evaluate_init_rb'

The solution is to comment out line 16 in vendor/plugins/open_id_authentication/init.rb when running rake db:migrate.

Updated: Better solution in the comments, thanks!

chadd@ubuntu:~/foo$ rake db:migrate --trace
(in /home/chadd/foo)
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:migrate
==  BortMigration: migrating ==================================================
-- create_table(:sessions)
   -> 0.0017s
-- add_index(:sessions, :session_id)
   -> 0.0004s
-- add_index(:sessions, :updated_at)
   -> 0.0003s
-- create_table(:open_id_authentication_associations, {:force=>true})
   -> 0.0021s
-- create_table(:open_id_authentication_nonces, {:force=>true})
   -> 0.0016s
-- create_table(:users)
   -> 0.0027s
-- add_index(:users, :login, {:unique=>true})
   -> 0.0006s
-- create_table(:passwords)
   -> 0.0017s
-- create_table(:roles)
   -> 0.0007s
-- create_table(:roles_users, {:id=>false})
   -> 0.0007s
==  BortMigration: migrated (0.5497s) =========================================

** Invoke db:schema:dump (first_time)
** Invoke environment
** Execute db:schema:dump
chadd@ubuntu:~/foo$

2. application.rb was renamed to application_controller.rb in Rails 2.3 and if you don’t rename it you get this error:

Loading development environment (Rails 2.3.2)
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:443:in `load_missing_constant':NameError: uninitialized constant ApplicationController

If you pass in Scrap.friendsandfollowers(‘rubyrescue’) you’ll get about 6000 ids. Anything better than a depth of ’1′ and you’ll need an un-rate-limited account w/Twitter as it will take more than 100 requests and they limit requests to 100/hour.

require 'twitter'
require 'net/http'
require 'json'

class Scrap

  def self.friendsandfollowers(users,depth=0,type = :all)
    users = [users] unless users.is_a?(Array)

    puts "at level #{depth}"

    newusers = []
    [:friends,:followers].each do |t|
      if [:all,t].include?(type)
        users.each() do |u|
          puts "looking for #{t.to_s.pluralize} of #{u}"
          newusers += JSON.parse(Net::HTTP.get(URI.parse("http://twitter.com/#{t}/ids/#{u}.json")))
        end
      end
    end

    newusers.uniq!

    if depth == 0
      return newusers
    else
      newusers.each() do |u|
        users += self.friendsandfollowers(u,depth-1,type)
      end
      return users.uniq
    end

  end

end