flouri.sh

Have you ever wanted to make a website that's just a re-skin or re-layout of another website? I always find myself wanting to do this, especially when there's no mobile version of a site and no API.

Enter mechanzied_session

All the mechanized_session gem provides is a way for the programmer to easily define remote actions that require an authenticated session. By providing these actions and an implementation for how to login, mechanized_session makes your life very easy.

View mechanized_session on GitHub

Acts_as_other_website on rails

MechanizedSession requires work to integrate with a frontend, so if you're building a rails app that just a skin of another website, acts_as_other_website is for you. It provides a default login page, handles all the normal exceptions that MechanizedSession raises, and provides the glue that stores your remote session data in your local session hash.

Example using EngineYard's cloud website

class EySession < MechanizedSession
  action :login do |session, options|
    session.get("https://login.engineyard.com/login") do |page|
      next_page = page.form_with(:action =>"/login") do |form|
        form["email"] = options[:username]
        form["password"] = options[:password]
      end.click_button
    end
    true  # tells MechanizedSession that login was successful.  EY returns a 401 exception if not successful
  end

  action :list_environments do |session|
    envs = []
    session.get("https://cloud.engineyard.com/dashboard") do |page|
      page.parser.css("div.environment").each do |env|
        envs << env.css('h3').first["title"]
      end
    end
    envs
  end
end

Then invoke acts_as_other_website in ApplicationController

class ApplicationController
  acts_as_other_website :using => EySession
end

Finally create actions that call the remote server using the @mechanized_session object like so:

class EnvironmentsController < ApplicationController
  def index
    @environments = @mechanized_session.list_environments
  end
end

The plugin will take care of ensuring that there's always an authenticated session when you do actions that require it, and will redirect users to a provided login page to establish that session when necessary.

View this example online using your iPhone or Safari: http://eycloud.heroku.com

Visit acts_as_other_website on GitHub

The JRuby Sandbox is simply a rewrite of why's original sandbox gem in JRuby. It's much less of a hack than the C implementation, and generally considered to be more safe. Here's how I set it up:

1. Download and install the latest JRuby binaries from CodeHaus (I tested with 1.1.5).
2. Download the source of the javasand jruby gem from the JRuby addons project
   svn checkout http://jruby-extras.rubyforge.org/svn/trunk/javasand
3. Compile the gem:
   ant
   BUILD SUCCESSFUL
   If the build fails, it might be because it can't find the JRuby classes. You'll need to find jruby.jar and then add a line to build.xml inside the "build.classpath" path:
   <fileset dir="/path/to/jruby/jars" includes="*.jar" />
4. Package up the gem:
   jgem build javasand.gemspec
5. Install the gem:
   sudo jgem install javasand-0.0.2.gem
6. Test the sandbox with jirb -rubygems

   require "sandbox"
   Sandbox.safe.eval("2+2")
   # yields 4
   

As you can see above, I had to compile the gem from source. The binary gem of javasand from rubyforge failed with the following exception:

irb(main):001:0> require "sandbox"
=> true
irb(main):002:0> Sandbox.safe
org.jruby.ext.sandbox.Sandkit:714:in `removeMethods': java.lang.NoSuchMethodError: org.jruby.RubyModule.removeMethod(Ljava/lang/String;)V

There's very little recent work on the MRI ruby sandbox, so here's a quick guide to getting the sandbox installed and running. Unfortunately, the sandbox requires a patched ruby, but luckily it's not that hard to set up.

1. Download the latest version of ruby 1.8.6 from ftp://ftp.ruby-lang.org/pub/ruby/1.8 (does not work with 1.8.7 or 1.9, sorry)
2. Download the sandbox gem source from git://github.com/why/sandbox.git
3. Patch ruby:
   patch -p1 < ../sandbox_gem/patch/ruby-1.8.6-sandbox_needs.patch
   patching file error.c
4. Compile and install the patched ruby:
   ./configure
make
sudo make install
5. Download and install rubygems from RubyForge
6. Install the sandbox gem:
   cd sandbox_gem && sudo ruby setup.rb
7. Test the sandbox:
   

   require "sandbox"
   Sandbox.safe.eval("2+2")
   # yields 4
   

Now that you've got the sandbox running, read more about it in my article on Advanced Sandboxing, or my Sandbox Introduction.

The title of my talk is: Playing with Fire: Running untrusted code in a sandbox. I expect to be posting some more articles related to the sandbox before my talk on April 17th.

If you've ever suffered from duplicate table aliasing problems in rails, there's a new feature in 2.2 that will help some of these situations go away. This seems to come up mostly in named_scope, where you want to define a list of scopes that can work by themselves or with any other scopes. This often means having identical or similar joins in multiple scopes. Take this example:

class User
  has_one :profile
  named_scope :male, {
    :join => "INNER JOIN profiles ON profiles.user_id = users.id",
    :conditions => "profiles.gender = 'male'" 
  }
  named_scope :recently_updated, {
    :join => "INNER JOIN profiles ON profiles.user_id = users.id",
    :conditions => ["profiles.updated_at > ?", 1.week.ago]	
  }
  named_scope :admin, {
    :join => "INNER JOIN profiles ON profiles.user_id = users.id AND profiles.admin = 1
              INNER JOIN emails ON emails.profile_id = profiles.id"
  }
  named_scope :with_profiles, {
    :join => :profile
  }
end

Before 2.2, calling User.male.recently_updated results in a table aliasing problem, because rails joins in the profiles table twice. Three features in 2.2 make this better:

• :join with hash/symbol syntax merges hashes when combining scope (Andrew White)
• :join now can take an array of strings (David Stevenson & Joseph Palermo)
• string identical joins are uniqued when combining scopes (David Stevenson & Joseph Palermo).

After 2.2, you can call User.male.recently_updated because there are two string identical joins combined in different scopes. You can't call User.male.admin without making some modifications, because the two joins involved are not string identical. Here's how I'd modify the :admin scope:

named_scope :admin, {
  :join => ["INNER JOIN profiles ON profiles.user_id = users.id",
            "INNER JOIN emails ON emails.profile_id = profiles.id"],
  :conditions => "profiles.admin = 1"
}

By using the array of strings :join syntax, I can let rails know that those are two separate joins, each of which can be merged if an identical join comes up in another scope. I also moved the extra join condition (profiles.admin = 1)to :conditions, so that the INNER JOIN statement would not have any specific logic for that particular named_scope in it.

I still can't call User.male.with_profiles because the string representation of the join from with_profiles probably doesn't match the join from male. It will be off due to the way rails generates join using different whitespace and escape characters than I originally wrote. This can be easily fixed by copying the exact string rails generates from :join => :profileand pasting it into male.

Developers finally set up master repo for acts_as_solr

After a long time of no development, a new acts_as_solr plugin has emerged on GitHub. Luke Francl & Mathias Meyer have adding a lot to the old plugin and created a new "master". Mathias has merged several branches on github including changes from David Palm, kengruven, and myself. This is great news for people running/modifying acts_as_solr, as Mathias is a real person who will actually respond to your pull requests. Some new features include:

• SOLR 1.3 installed
• Ruby 1.9 support
• LibXML 0.7+ support
• Store types as string not text fields in the index
• JVM options in solr.yml
• Completely rewritten test suite
• New solr:reindex task that automatically finds and indexes your solr models

I've been using query reviewer exensively at pivotal, and we've noticed that it is ridiculously slow when you have a lot of queries. This is due to the large number of partials rendered (5 partials) for each query in the reviewer box.

A solution to this problem, that also brings a huge advantage, was to report aggregate statistics for near-identical queries. Similar to way mysqlslowdump groups near-identical queries, query_reviewer now also reports only 1 line for X identical queries (identical means the same structure and same stack trace). Consequently, it's a lot easier to find places where you're missing includes . Here's what it looks like now:

The time reported for a group is the total time of all the queries in that group. In addition, I add up all the times for all the queries and report that number in the header. From this, I've noticed that a lot of time is spent in rails outside of waiting for mysql to return result. I wonder where that time goes...

query_reviewer can be downloaded from github!

Thanks to a nice suggestion by the community which I pretty much used verbatim. Still works with rails 2.0 and 1.2.3. Check out the project homepage.

http://code.google.com/p/query-reviewer/

Kevin Hall and I have released a new version of the query_reviewer plugin. You should start by looking at my first post, to see what the basic premise is before reading this article. The single largest improvment is the ability to analyze the database requests of AJAX requests, which is accomplished by piggy backing javascript or HTML into the ends of AJAX responses. Here's what it currently looks like, analyzing my project on a page that does lots of database requests:

The improvements are:

• View query analysis for AJAX requests
• Take into account the duration of a query (if production data)
• Show PROFILE information from mysql
• Warnings for long key lengths (which can be bad even when you hit an index)

Lots of information is now available for a single slow query:

I'm really excited that this plugin is getting attention, and welcome feedback, suggestions, and bug reports on the Google Code project homepage.

To install it, simply visit the project homepage.

Things started going very badly for flouri.sh when my home server wouldn't boot up. I removed almost all the hardware, but could not even get a POST screen or any beeps. I even reseated the RAM and CPU, but no luck. Finally, I grabbed an old media PC shuttle cube and attempted the craziest idea:

The 4 x RAID5 disks are actually being powered by my old server, but the PCI raid card is inserted into my old media PC. The shuttle doesn't have enough power or room to run all those disks so I'm using the old server as a lifeboat with 4 IDE cables running between them!

Using ActiveRecord can be tricky, here's how

Data migrations are a major headache in every rails project I've worked on. Developers typically write straight SQL migrations which take much longer to create and test, or they use ActiveRecord and run into problems. I recent wrote a data migration which took 7 existing tables and compressed them into 4. Some of the new 4 had the same names as the existing models, so I finally figured out how to do this safely with ActiveRecord. Here's my advice:

• When changing the schema of an existing set of tables, create new tables and then rename them.
• Include active record fragments inside the migration class at the top of your migrations.
• Put your data migrations inside transactions.
• Make your data migrations completely reversible (which is a lot easier when you follow the first rule)

Here's an example of an ActiveRecord class fragment at the top of a migration:

class MarkPrimaryBits < ActiveRecord::Migration
  class LessonVersion < ActiveRecord::Base
    belongs_to :lesson
  end

  class Lesson < ActiveRecord::Base
    has_many :lesson_versions
  end
 
  def self.up
    ...

By putting the Lesson and LessonVersion classes at the top of the migration, I'm allowing myself to use those classes inside my migration and be completely independent of any changes made to the real model from then on (including the deletion of the class itself). Furthermore, I can have another migration which uses those same class names with completely different meanings and they won't conflict with each other.

How to build custom methods on associations that aren't slow

The "has_many do" syntax has been widely adopted in rails, but I often see it going wrong. Consider the following association extension:

has_many :versions do
  def primary
    find(:first, :conditions => {:primary => true})
  end
end

What's wrong with this code is that the finder has to execute a database request every time it's invoked. True, rails has a query cache, but it results in more database requests then one should need. We can make it better by caching the result in an instance variable attached to the association:

has_many :versions do
  def primary
    @primary ||= find(:first, :conditions => {:primary => true})
  end
end

But what if you already have the association loaded? Inside the block, you can access various methods of the AssociationCollection and AssociationProxy classes. Notable methods are the following:

• proxy_owner - the module that contains the association
• proxy_reflection - the Reflection object that contains the association options (FK, :dependent, etc)
• proxy_target - the cached association data, if the association has been loaded
• loaded? - returns true if the association has been loaded
• reset - delete the cached association data and forget it has been loaded

Using these methods, we can rewrite our method to be the most efficient possible, by making use of loaded association data when present:

has_many :versions do
  def primary
    if loaded?
      @primary ||= proxy_target.detect { |ver| ver.primary? }
    else
      @primary ||= find(:first, :conditions => {:primary => true})
    end
  end
end

Now our method is guarenteed to make only 1 database call, no matter how many times invoked. It also will make zero database calls if the entire association has already been loaded. The only downside is that "how to determine if primary" logic has to be written once in rails_sql and once in pure ruby.

query_analyzer on steriods!

Features

QueryReviewer is an advanced SQL query analyzer. It accomplishes the following goals:

• View all EXPLAIN output for all SELECT queries to generate a page
• Rate a page's SQL usage into one of three categories: OK, WARNING, CRITICAL
• Attach meaningful warnings to individual queries
• Find out where the query was executed with a stack trace
• Display advanced interactive summary on page

It accomplishes this by injecting an absolutely positioned div into your HTML markup before it's sent out of rails. View injection can be turned off, if necessary and replaced with <%= query_review_output %> somewhere in your view.

Screenshot

Project homepage

http://code.google.com/p/query-reviewer/

Installing

svn export http://query-reviewer.googlecode.com/svn/trunk/ vendor/plugins/query_reviewer

Then optionally run "rake query_reviewer:setup" and edit config/query_reviewer.yml to change the default settings.