RubyRescue

RubyRescue has limited space available starting August 1st – a common area with a desk and multiple internet connections, and a cozy private office. We’re a group of developers, mostly focused on the same technologies, all in one space, near Parque Las Heras. A lot of ideas, projects, and opportunities turn up by working near each other – if you are interested, let us know. Contact martina@inakanetworks.com or 15.6940.7097…

A client had a new hire start this week, let’s call him (or his login) ‘ted’.

Ted was issued a new account on production servers with a simple password, and was told to login, upload a public key, and change his password. Ted did this, but didn’t change his password for a few hours. Within this few hour window, every server with his account was compromised and an IRC server was installed.

The rest of the server was locked down so no other accounts or data were accessed, but the resource consumption from IRC clients connecting to the machine caused serious problems with the server for a number of hours.

9 hours – Total downtime.
4 hours – Time from hack to Rackspace noticing the servers were down, even though the client pays for URL monitoring.

Lessons learned

1. DO NOT use password auth on public servers, or setup a strong password policy that root can’t get around – someone can get lazy.
2. DO NOT trust Rackspace URL monitoring to alert you to a downtime problem – setup your own monitoring suite and alert/escalation plan. Use Rackspace monitoring as a backup only.
3. Remember that just having access to a user-account with no sudo powers is enough to bring down a server if some aspect of server resources (file handles, ports, memory, disk space, etc) are over-utilized.

Ruby Rescue uses unicorn for all of our deployed Ruby applications. Chris Wanstrath promoted it in a blog post on github, and we decided soon after to give it a shot. Because it’s based on mongrel, it takes a reliable and well-tested app server and bolts a ruby layer on top that handles restarts, adding and removing workers, and uses unix sockets to wait for requests.

There’s a good article by Ryan Tomayko on Unicorn and the reasons why it’s “Unixyness” is a feature. If you’re a long-time unix user, you won’t find much to be revolutionary. If you’re a web-developer without a CS-background, he covers how unix fundamentals make Unicorn more reliable.

Before Unicorn we were running Passenger and apache, and we decided to switch to unicorn and nginx at the same time. It takes a bit more work to setup unicorn than passenger, but we’ve found the reliability to be worth it.

• gem install unicorn
• create a unicorn.rb file like this

rails_env = ENV['RAILS_ENV'] || 'production'

worker_processes 3

preload_app true

timeout 75

socket_path = '/var/www/APPLICATION/shared/sockets/unicorn.sock'
pid_path = '/var/www/APPLICATION/shared/pids/unicorn.pid'

listen socket_path, :backlog => 2048
pid pid_path

# http://www.rubyenterpriseedition.com/faq.html#adapt_apps_for_cow
if GC.respond_to?(:copy_on_write_friendly=)
  GC.copy_on_write_friendly = true
end

before_fork do |server, worker|
  old_pid = RAILS_ROOT + '/tmp/pids/unicorn.pid.oldbin'
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
    end
  end
end

after_fork do |server, worker|
  ActiveRecord::Base.establish_connection

  begin
    uid, gid = Process.euid, Process.egid
    user, group = 'admin', 'admin'
    target_uid = Etc.getpwnam(user).uid
    target_gid = Etc.getgrnam(group).gid
    worker.tmp.chown(target_uid, target_gid)
    if uid != target_uid || gid != target_gid
      Process.initgroups(user, target_gid)
      Process::GID.change_privilege(target_gid)
      Process::UID.change_privilege(target_uid)
    end
  rescue => e
    raise e
  end
end

• Your deploy.rb will get a bit more complicated as it needs to send the proper signal to the Unicorn master to signal it to reload the workers.

namespace :deploy do
  task :stop, :roles => :app do
  run "cd #{current_path} && kill -QUIT `cat tmp/pids/unicorn.pid`"
end 

task :start, :roles => :app do
  run "cd #{current_path} && /opt/ruby/bin/unicorn_rails -c config/unicorn.rb -E production -D"
end

desc "restart unicorn"
task :restart, :roles => :web do
  run "cd #{current_path}; [ -f tmp/pids/unicorn.pid ] && kill -USR2 `cat tmp/pids/unicorn.pid` || /opt/ruby/bin/unicorn_rails -c config/unicorn.rb -E production -D"
end

Ruby Rescue is looking for a full-time developer who knows Ruby on Rails. If you have experience with Ruby on Rails, want to work on interesting projects, don’t mind learning a bit of Erlang, and have at least intermediate English language skills, we would like to talk with you. We’re looking only for people who can work full-time in our offices in Palermo. Send us your CV to martina@rubyrescue.com or tweet @rubyrescue.

RubyRescue, creator of high-traffic, user-centered websites, is looking for a web designer to join our team. If you know Photoshop, HTML, CSS, and you can prove it with a solid portfolio, contact us. We’re a small group of hacker/entrepreneurs working on technology projects and we need good, responsible and motivated people to join our team. Most of our projects are based on Ruby on Rails, our team is from all over the world, and you would help shape the look and feel of our projects. Please send us an email with a CV, background and portfolio samples. Contact: martina@rubyrescue.com

Tango Rails esta juntando para tomar unas cervezas y charlar sobre Ruby on Rails, Sinatra y mas… Es un after muy informal y bueno para conocer gente jugando con las mismas tecnologías.

Tango Rails is meeting for some drinks and to chat about Ruby on Rails, Sinatra or other interesting things happening with web frameworks and Ruby these days. It’s a very informal after office and a great place to meet people in person working on interesting projects.

Link to the Facebook Invite

Here’s how I did it:

mysqldump -uroot -p rubyrescue | sed
's#rubyrescue.com#rubyrescue.com/blog#g' > temp.sql
mysql -uroot -p rubyrescue < temp.sql

The only other item was dealing with permalinks. The error handler for lighttpd was set to look for index.php in the root. I had to change that to look for it in /blog.

this article nicely summarizes how great heroku is. i deployed a small personal project with it and LOVE how easy it was. incredible.

One concern i have is that without a static IP address, if you have a highly SEO-sensitive application it’s probably not the best choice, because a static IP is just one of many factors in your pagerank, but an important one.

In attempting to upgrade a Bort app with Rails 2.3.2, i’ve found two errors so far:

1. This OpenIdAuthentication error:

rake aborted!
uninitialized constant Rails::Plugin::OpenIdAuthentication
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:105:in `const_missing'
/home/chadd/foo/vendor/plugins/open_id_authentication/init.rb:16:in `evaluate_init_rb'

The solution is to comment out line 16 in vendor/plugins/open_id_authentication/init.rb when running rake db:migrate.

Updated: Better solution in the comments, thanks!

chadd@ubuntu:~/foo$ rake db:migrate --trace
(in /home/chadd/foo)
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:migrate
==  BortMigration: migrating ==================================================
-- create_table(:sessions)
   -> 0.0017s
-- add_index(:sessions, :session_id)
   -> 0.0004s
-- add_index(:sessions, :updated_at)
   -> 0.0003s
-- create_table(:open_id_authentication_associations, {:force=>true})
   -> 0.0021s
-- create_table(:open_id_authentication_nonces, {:force=>true})
   -> 0.0016s
-- create_table(:users)
   -> 0.0027s
-- add_index(:users, :login, {:unique=>true})
   -> 0.0006s
-- create_table(:passwords)
   -> 0.0017s
-- create_table(:roles)
   -> 0.0007s
-- create_table(:roles_users, {:id=>false})
   -> 0.0007s
==  BortMigration: migrated (0.5497s) =========================================

** Invoke db:schema:dump (first_time)
** Invoke environment
** Execute db:schema:dump
chadd@ubuntu:~/foo$

2. application.rb was renamed to application_controller.rb in Rails 2.3 and if you don’t rename it you get this error:

Loading development environment (Rails 2.3.2)
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:443:in `load_missing_constant':NameError: uninitialized constant ApplicationController

The twitter gem doesn’t do a good job of allowing unauthenticated queries, so I have to make a specific request to the twitter API to do this.

If you pass in Scrap.friendsandfollowers(‘rubyrescue’) you’ll get about 6000 ids. Anything better than a depth of ’1′ and you’ll need an un-rate-limited account w/Twitter as it will take more than 100 requests and they limit requests to 100/hour.

require 'twitter'
require 'net/http'
require 'json'

class Scrap

  def self.friendsandfollowers(users,depth=0,type = :all)
    users = [users] unless users.is_a?(Array)

    puts "at level #{depth}"

    newusers = []
    [:friends,:followers].each do |t|
      if [:all,t].include?(type)
        users.each() do |u|
          puts "looking for #{t.to_s.pluralize} of #{u}"
          newusers += JSON.parse(Net::HTTP.get(URI.parse("http://twitter.com/#{t}/ids/#{u}.json")))
        end
      end
    end

    newusers.uniq!

    if depth == 0
      return newusers
    else
      newusers.each() do |u|
        users += self.friendsandfollowers(u,depth-1,type)
      end
      return users.uniq
    end

  end

end