Effectif Development - Home

Notes from the Ruby Manor (part 2)

2008-11-23T01:34:00Z

So this is Ruby Manor part 2. After the Ruby Manor morning session we had a tasty pub lunch, rocked up a little late, and settled down to an interesting afternoon of Ruby talks…

Again, the errors and omissions are mine.

Jonathan Conway – Neo4J

In the beginning we had flat files. They worked well. Then we got relational databases. Great for ad-hoc queries and they allow you to define relational constraints. Brilliant. However, there are problems…

When you try and map an object into a relational table you can run into issues. Jonathan pointed out other issues too, but I think I missed them. My mistake (I’d just had lunch and was slightly snoozy)…

There are plenty of ORMs to choose from, such as ActiveRecord, Datamapper, Sequel and Og. When we use an ORM we move the constraints out of the database and into the model. It’s a nice way to work, but you’re no longer using the relational database in the way that it was designed to be used. You end up with semi-structured data (data that has very few mandatory fields but many optional fields) and sparse tables. You then find that you need to partition your tables sooner than you would do if you had properly structured data.

Maybe the relational database isn’t a one size fits all solution. There are other possibilites:

Column oriented databases (e.g. lucid).
Document oriented databases (such as CouchDB).
Graph oriented databases (e.g. Neo4J).

Neo4J uses a graph as the underlying data structure. It’s based on Java, but has a small memory footprint. It’s also very fast, a benefit of being embedded in the JVM.

There’s a Ruby wrapper – Neo4J.rb. It’s available on github.

Queries are written using Lucene. All nodes in the graph – and the relationships between them – can have properties defined on them. They’re exposed as simple hashes in Ruby.

Jonathan doesn’t think that Neo4J.rb is production ready yet, but Neo4J itself is. Also see http://neotechnology.com. It sounds pretty interesting – well worth a look if you’re interested in database technology.

Martin Loughran – APICache

Martin got up and did a lightning talk on his APICache project; a useful little library for improving the resilience of sites that rely on (sometimes flaky) third party APIs.

Alex McCaw – Acts as Recommendable

Alex works at Made By Many where he developed the Acts as Recommendable plugin. It allows you to add a recommendations system to your Rails app, and was largely developed using O’Reilly’s Programming Collective Intelligence. There are three basic approaches to recommendations:

Content based (e.g. bayesian filtering).
User based.
Item based.

Acts as Recommendable uses item based recommendations.

To use the plugin you need to define a has_many :through relationship between your models, then call acts_as_recommendable:

class User < ActiveRecord::Base
  has_many :user_books
  has_many :books, :through => :user_books
  acts_as_recommendable :books, :through => :user_books
end

You will then have these extra methods on your models:

User#similar_users
User#recommended_books
Book#similar_books

In this example the second two methods are the really useful ones.

You can obviously store things on the join table, such as a score (if you’re interested in making a rating system you can add a rating column to your join table).

You can choose from the following algorightms: Manhattan distance, Euclidean distance, Cosine, Pearson correlation coefficient, Jaccard, Levenshtein.

Alex described some of the issues that they ran into whilst developing Acts as Recommendable. They had to work around scaling issues (such as queries that pulled every row from their tables), caching issues, problems with Ruby (it couldn’t cope with data sets so large), and performance problems with the algorithm. They solved all the issues (though I didn’t quite grasp the full details of how they went about it), split the cache up, and re-implemented slow code (the Pearson algorithm) in C.

Alex’s advice when implementing acts_as_recommendable:

You don’t want to have too many relationships.
You don’t want to have too many items (but it’s been tested with 100,000).
Setup the similarity relationships for items, not users.

To scale it further you can use K Means clustering or split your clusters by category (which is the way Amazon does it). You need to start splitting your clusters once you get to approximately 100,000 items. The plugin can probably cope with up to 200,000 items, but you can always partition your data and use Nanite or map reduce to scale it further.

This is clearly pretty clever stuff. I’ve missed many of the details from Alex’s slides; hopefully he’ll upload his slides to his blog. The code is in github (for a change). There’s also a Collective Intelligence video from Goruco 2008 that’s worth watching.

Sean O’Halpin – Unobtrusive Metaprogramming

What does unobtrusive mean? In this context it’s “polite, considerate”. About being nice. And metaprogramming? “Programs that write programs are the happiest programs in the world.” (Andrew Hume)

Before you load ActiveSupport you have 75 methods on a Ruby object. After you load ActiveSupport you have 173. Sean’s point is that Rails messes with stuff. By defining methods on Object rails can affect what you’re doing in your own namespace. For example, Rails defines an inspect on class that hits the database. Would you expect that? More to the point, people who think that’s okay frighten the crap out of me…

So (assuming you’re not also slightly scared by these people) what’s the problem?

Namespace pollution: You can’t use the names you want, your code is not safe, and you get clashes that can be hard to debug (we’ve all been there).
Documentation pollution: rubydoc documents all methods on your object, so all this extra cruft ends up in your docs.
Cognitive pollution: What does an object do? You can’t tell by looking at its capabilities.

So it’s about pollution then. Stinky.

Why does this happen? The obvious answer is that it’s easy. But it’s one thing to monkey patch classes/objects in your own application. It’s another thing entirely to do it in a reusable library; you affect everybody.

How should we solve these problems?

Use functions that accept an object and return a modified version (i.e. you don’t need to monkey patch another method onto an object – functions are okay).
Use namespaces (i.e. modules).
Hide your mess.

Sean then showed us some rather clever code for “hiding your mess”. Hopefully he’ll post the code online. If he does, check out the KittySafe module.

And as Sean pointed out, we each paid 23p (8 minutes out of 8 hours) to hear his talk. Bargain. It would have been even better value if somebody hadn’t been heckling halfway through (cheers for your input Piers, but have you considered being slightly more unobtrusive yourself by saving your quibbles until the end of the presentation?).

James Darling – Government Hack Day

James is organising a government hack day. He’s realised that it’s difficult to persuade the powers that be to open up information behind freely available APIs. Most people just don’t appreciate the benefit of mashups (release the data and goodness is bound to follow) until you show them.

The plan is to get some data together (he’ll be launching a wiki to collaborate on data gathering), get some interested coders together, then we can all spend 24 hours producing some incredible goodness. The results will (at the very least) help to demonstrate to those who hold the keys just how much could be done if the government adopted the same approach.

Interested parties can sign up on the UK Government Hack Day site. So sign up!

David Black – Ruby 1.9

This is a talk about the changes in Ruby 1.9. Ruby 1.8.7 has a bunch of features that were backported from 1.9, but David is looking at the changes between 1.8.6 and 1.9.

Method and block argument semantics

Required arguments can now come after optional arguments. If you call the method below with two arguments they’ll get bound to a and c (i.e. required arguments get serviced first):

def m(a, b = 1, c)

Required args can now come after wildcard aguments:

def m(a, *b, c)

Arguments can be grouped with parentheses for dealing with arguments passed as arrays. If you call the method below with [1, 2], 3 then a will be 1, b 2 and c 3. If you call it with 1, 2 then a will be 1, b will be nil and c will be 2.

def m((a, b), c)

Block and method argument semantics have now effectively been merged, though they were different in previous versions of Ruby. This means you can pass optional arguments to blocks, use the parentheses above, etc.

Block variable scoping rules

In Ruby 1.8:

a = 1
Proc.new { |a| }.call(30)
a #=> 30

In Ruby 1.9:

a = 1
Proc.new { |a| }.call(30)
a #=> 1

However if the variable you update isn’t the block argument then it’ll still get changed in Ruby 1.9:

a = 1; { a = 2 }  # a is now 2

Unless you specify it after a semicolon, which declares the variable to be a block local variable:

a = 1; { |x; a| a = 2 }  # a is still 1

Enumerators

Enumerator objects have all the methods you get in the Enumerable module, but they don’t know how to find their next item. So you have to tell them. You can teach the object how to do it by passing a block when you instantiate it, or you can get it to borrow each from somewhere else.

Here’s an example:

enum = Enumerator.new do |y|
  y << ["New York", "NY"]
  y << ["Florida", "FL"]
  y << ["Texas", "TX"]
end

enum.each do |state, abbr|
  puts "#{abbr} is short for #{state}"
end

Once your object knows how to handle each you can happily call all the other Enumerable methods such as map. Nice.

David then showed us a neat bit of code for XOR’ing a string (that I think he wrote earlier today in response to Paul pointing out how much easier it is to do in Perl) that took advantage of these new enum capabilities. Sorry, I didn’t capture the details…

Good talk. He’s an entertaining chap.

Wrapping Up

It’s been a great day. Excellent work James and Murray. Why aren’t more conferences like this? I think I may actually have made money out of turning up. No really.

You see, in order to break even (having paid for the room, audio equipment, projector, etc.) the conference only needed 66 attendees. It was fully booked with 130 people. All the spare cash was put behind the bar downstairs. I don’t know about you, but I’ve never been to a conference that cost 12 quid (quid = British pounds) and came with five free pints of beer. We didn’t even manage to drink the bar dry (a poor show, but in our defense it was a subsidised University bar), so it stretched to half a curry for those of us that made it.

Genius, really.

Notes from the Ruby Manor

2008-11-22T13:28:00Z

I’m lucky enough to be at RubyManor today; a Ruby conference organised by Ruby users, for Ruby users, costing the grand total of twelve of your British pounds. A bargain, as you’ll see if you check the lineup of talks and speakers.

I started making notes, but it seems to have evolved into some kind of blog post. Apologies for typos, crap grammar and glaring errors.

Paul Battley – I’m an Evil iPlayer hacker

Paul Battley is a hacker. BBC News said so, so it must be true. Paul started by running through the history of iPlayer’s development. It launched in July 2007 in open beta, but required Windows XP and a custom client with built-in DRM. In December 2007 a flash streaming version was released, which worked on Windows, Mac and Linux. However you could only watch content for seven days.

In March 2008 an iPhone version of the iPlayer was released. The iPhone doesn’t support flash, so the iPhone version streams MPEG4 over HTTP. When first released it worked by sniffing the user agent, and checking for the string “iphone”! Within hours people had released scripted solutions for downloading the videos, but one week later the BBC had plugged the hole.

Wasting no time, Paul discovered that the video is pulled with a different user agent to the page, and an extra HTTP header was required. All was well with the world, but in June the downloaded files stopped working. Comparing old files to the new ones, they discovered that the BBC had XOR’d some data in the middle of the MPEG files. That was easy to work around.

Later in June version 2 of the iPlayer came out with new URLs, but no new counter measures. In September the iPlayer started to serve radio, which can also be downloaded with Paul’s scripts.

So the basic technique for downloading content from the iPlayer is to configure your client to pretend to be an iPhone.

Paul then talked about how to get a programme’s ID from some JavaScript served up with the programme’s web page. This was one of the hardest parts of the job, requiring him to write a JavaScript tokenzer.

The flash version of the iPlayer uses the RTMP protocol, to make it a bit more difficult for people to “steal” the content. The next thing that Paul decided to try and reverse engineer RTMP and write a Ruby library to process RTMP streams. He’s not finished yet, but it looks as though he’s got quite a long way through the job.

Lessons learnt:

Releasing Ruby software to users is hard
Writing GUIs is tedious (but check out the iPlayer grabber)
Reverse engineering is fun and enlightening
Obfuscation doesn’t work
You need a feedback channel (or people will make one)

It was a cracking presentation. More information is available here:

If you want a graphical app for the Mac, iPlayer grabber looks great.

Dan Webb – 8 Minutes on Rack

Dan Webb’s main message was that rack is worth everybody’s time, not just that of people who write web frameworks. But what is it? It’s not a framework. It’s not a library. It’s just a convention.

The convention is this. You have a call method on a ruby object that returns an array of three values; the HTTP status code, a hash of headers to send to the client, and the response body. Here’s an example:

app = Proc.new do |env|
  [200, { "Content-Type" => "text/plain" }, "Hello World"]
end
Rack::Handler::Thin.run(app, :Port => 4000)

The response body can be any object that responds to the each method, such as an open file object.

file = File.new("myfile.xml")
[200, { "Content-Type" => "application/xml" }, file]

For binary files you can write your own class to stream anything you like. Here’s an example of an each method that you could use to stream a binary file:

def each
  File.open(@file, "rb") do file
    while part = file.read(8192)
      yield part
    end
    File.delete(@file)
  end
end

Why should you care? Passenger, Mongrel, CGI, SCGI, FastCGI, Thin, Webrick, Fuzed (the list goes on).

Other things that Dan thinks are worth looking into:

The Rack Gem
Middleware (very simple and clver, well worth a look)
Rack and Passenger
rackup

Check http://rack.rubyforge.org and read the (slightly impenetrable) documentation. Dan will be writing some Rack stuff up on his blog, and his slides are available on slideshire.

David Black popped his hand up at the end and pointed out that strings in Ruby 1.9 don’t have an each method. Ooops.

George Palmer – Nanite

George Palmer gave an introduction to [Nanite](. I missed the very beginning because I was chatting in the canteen. Sorry.

There are currently three different broad brush approaches available to running background tasks in Ruby web frameworks:

Fork into background of rails process (e.g. spawn, run_later).
Record a job to a file or database, for handling by a background deamon.
Fork into some kind of queue (e.g. backgroundRB, beanstalk, starling, etc.).

Nanite is great for heavy lifting, and is designed for running jobs on other servers. It’s based on RabbitMQ; a queuing system written in Erlang. As an aside, when installing Erlang on a Mac you need to avoid using the R12B5 of Erlang; this means that you don’t want to use MacPorts, and are better off just downloading the source form the Erlang site.

AMQP stands for Advanced Messaging Queueing Protocol. It’s an enterprise quality protocol, used by (for example) big financial corporations. It’s also language neutral. RabbitMQ implements AMQP 0.8.

To install AMQP for Ruby you need the eventmachine gem, then grab the source from github and install the gem:

$ sudo gem install eventmachine
$ git clone git://github.com/tmm1/amqp.git
$ cd amqp && rake gem && sudo gem install amqp-<version>.gem

Nanite mappers control and track the work. You can run as many mappers as you like; they get updates from the mapper exchange. Mappers are run either inside your Rails/Merb app (on Thin) or via the command line.

Mappers assign work to agents. Agents announce their presence when they start up so that the mappers know that they’re available. Actors know how to do the work. Here’s an example actor:

class Manor < Nanite::Actor
  expose :name

  def name(vars)
    # Do something interesting
    :result => "RubyManor"
  end
end

Nanite:Dispatcher.register(Manor.name)

There were a lot more code examples that I didn’t manage to catch, but George will be uploading the slides to his blog, and I recommend taking a look.

This is what I did catch:

Nanite has no concept of security, but relies on the security in RabbitMQ.
George recommends monitoring your agents with a program like monit or god.
Once a nanite request has been sent to an agent and processed by an actor you can’t check it for status; use the database for that.
You can send files to your agents (e.g. an image that needs resizing).
Nanite allocates work by looking at the currently available agents and sending the job to (by default) the least loaded server. If you prefer you can choose to send a request to all agents, or a randomly selected agent.
You can also define your own metrics (using Ruby) for an agent’s “load”.

Finally, George ran a quick demo. Members of the audience fired up a nanite agent (an example agent that is distributed with the nanite source code) that announced itself to a server running on his laptop, and he used the server to list the gems on everybody’s laptops. A neat demo.

For more information, try these links:

The Nanite docs aren’t currently that great, but the code is very readable.

Rob McKinnon – Rugalytics

Rob wondered if there was a Ruby API for Google Analytics. jnunemaker had written statwhore that gets you some XML, and wrapped it with helper methods such as pageviews and visits.

So that was cool, but Rob wanted more. He wanted to emerge class definitions at runtime from the data. So he forked statwhore, then hacked on it so much that he renamed it to Rugalytics. Then he converted the XML to CSV to make it easier to work with.

Rob is now using Rugalytics to put a list of popular pages on his site, pulling the data direct from Google Analytics. Fantastic stuff.

To get an Analytics report you pass the report name when calling analytics. You can find the available reports by calling the brand new profile.report_names method.

Google have a tendency to tweak the format of the CSV files, and this has been known to break Rugalytics. There are also some language issues – if you’ve got your Google account setup to US english then you end up with a method (dynamically created from the CSV) called pageviews. If you’re Google account is configured for UK english you used to get a method called page_views (but Rob has now hardcoded it to pageviews). In Italian you get pagine_visualizzate. Nice! The README file now advises that you choose US/UK english.

Rob also has a local web server (which will be included in the gem) – similar to a Rack example that Dan talked about earlier – that serves JavaScript to Rob’s Greasemonkey enabled browser. Greasemonkey then annotates the pages on his site (for him, and only for him) showing which keywords people have used to find the page that he’s currently looking at. Pretty impressive.

See Rob’s account on github to get the code. Rugalytics is also available as a gem.

$ sudo gem install rugalytics

Murray Steele and Martin Saddler – GUI Manor Born

Apparrently Ben Griffiths suggested all talks should be named around the phrase “To The Manor Born”. As Murray put it, “at least we tried”.

So Murray and Martin are talking about how to build GUI apps with Ruby. There are plenty of GUI toolkits available to Ruby, such as wxRuby, fxRuby, etc. However for their example app (a Twitter client) Martin used Monkeybars and Murray used Shoes.

Monkeybars

Monkeybars is built using jRuby and produces a swing front end. What you really need (to follow along with Martin’s demo) is JRuby, Netbeans and Ruby gems. To start a new Monkeybars project you run a command that makes you a blank project (much as Rails does):

$ monkeybars myapp
$ cd myapp
$ rawr install  # choose option 2
$ jruby -S rake generate ALL='src/login'

At this point Martin demonstrated (in the style of Blue Peter – “here’s one I made earlier”) how to build his Twitter client using Netbeans. There’s a GUI builder that allows you to drag and drop GUI controls into the user interface. You can then bind the widgets to the code using Netbeans’ normal GUI builder tools.

Monkeybars uses a form of the MVC design pattern. The view binds the widgets defined in the GUI builder to Ruby code that you can access in the controller. The controller and view are pretty much standard stuff.

You can easily create callback methods in your controller for widgets in your UI by taking the name of the widget (e.g. login_button) and creating a method called login_button_action_performed. Seems prety straightforward.

You can use the rawr program to package your app.

The downsides? Not everything works in JRuby at the moment (for example Martin couldn’t use the twitter4r gem). Build paths can get mucked up. Assumes knowledge of Java, there’s a learning curve, etc. But these problems are being worked on.

The good bits? GUI Builder. Interface to mature Java libs. It’s all Ruby. Cross platform distribution. No platform dependency on Ruby.

See tweetobix.com to follow through with Martin’s example.

Shoes

Shoes is a GUI framework written by whytheluckystiff. It was designed as a replacement for an HTML rendering engine for use when teaching people to code, so it’s pretty familiar to people who know a fair bit of HTML.

To install it you don’t install a gem. Shoes is it’s own version of Ruby. Download it and install it.

The code is pretty simple. Murray’s app is called Talon. You start with a block, which makes a window. Then you can do stuff to the window. Here’s the main window (with no widgets in it):

Shoes.app :title => "Talon" do
  background "#fff"
  background "talon.jpg",
             :bottom => 0,
             :right => -20
  flow :width => "100%" do
    background "#df9", :curve => 12
    title "Talon",
          :stroke => "#691FFF",
          :align => "center"
  end
end

The styling is a lot like CSS. Backgrounds can be layered.

To layout widgets you don’t worry about layout managers; you use stacks and flows. Stacks allow you to lay things out vertically (a bit like a vbox in GTK). Flows are similar, but run horizontally (a GTK hbox).

To load a gem you must reference it in the setup block. This will ensure that when somebody runs your app that all required gems will be installed, rather than throwing a backtrace.

Shoes.setup do
  gem "twitter"
end

require "twitter"

Murray showed a lot of exmaple code for laying out and animating widgets. It seems very clear and succinct.

You can ask Shoes to package your app with one click, and it builds a package for Windows, Mac, etc.

For more info you can access the manual:

$ shoes --manual

You should also check out Nobody Knows Shoes, which – Murray assures us – is fantastic.

The code for Murray’s example (Talon) is available on github.

Cracking stuff.

The Afternoon Session

At this point we went off to the pub for lunch, but when we got back I took notes in the afternoon session too. See Notes from the Ruby Manor (part 2).

Optimising PNG images for the web

2008-11-17T23:36:00Z

Stoyan Stefanov has recently written an interesting series of articles on optimising image files for the web. In his latest article he shows how to use pngcrush to safely shrink your files, one file at a time:

$ pngcrush -rem alla -brute -reduce src.png dest.png

This evening I found myself wanting to repeat that command for all the PNG files in my current directory, and then (having reviewed the new images) overwrite the originals. Rather than do it by hand I did what all pragmatic programmers do these days, and wrote a script. Here it is in action:

$ pngcrushall 
Crushing test4012.png ... ok
Crushing test9200a.png ... ok
Crushing test9200b.png ... ok
Overwrite original files? (y/N) y
Overwriting test4012.png ... ok
Overwriting test9200a.png ... ok
Overwriting test9200b.png ... ok

You can get it from github – download pngcrushall.

Obviously you’ll need to install pngcrush too. On a Mac this is pretty easy with MacPorts:

$ sudo port install pngcrush

Integration Testing for Multiple Web Applications – Introducing Integral

2008-11-11T23:20:00Z

I’ve recently been developing a Rails app that relies on a database that is maintained by another team as part of a different application. As you can imagine, unexpected changes to the database schema can bring our Rails app to it’s knees if they’re deployed to the live server before our application is updated.

So far we’ve been lucky, but the issue has been at the back of our minds for a while. So this week I sat down and cooked up a solution, and we’ve called it Integral.

Integral consists of some Thor tasks that:

Run tests on your test server.
Keep track of which versions of your applications work well together.
Can be integrated into your deployment recipes to ensure that deploying an app should never break your live server again…

All the Integral code is on github, and I’ve written some fairly extensive documentation on the Wiki. Please try it out, give us some feedback, fork it, burn it, give us some feedback, etc. etc.

Installing merb with thor

2008-10-05T08:36:00Z

Update (22 October): Things move very fast around here – the merb.thor file has been overhauled and now all you have to do to get the latest edge is download a single .thor file and run one command.

Matt Aimonetti wrote a neat post on installing merb edge with thor. Thor seems to have become the defacto technique for installing merb edge these days, and I’m going to give it a go in preference to installing merb with sake, as I had been doing.

Since Matt wrote his article the merb.thor file underwent a serious overhaul during the sprint at MerbCamp – what you need to do these days is quite a bit simpler:

curl -L http://merbivore.com/merb.thor > merb.thor
sudo thor merb:stack:install --edge

The source code for all the Merb gems will be checked out into a directory called ./src.

To get the full low down on how to use Thor with Merb, head over to the Thor Howto on the excellent new Merb wiki.

Building do_mysql on a 64-bit Mac

2008-10-02T22:24:00Z

Whilst installing merb with edgy I’ve had run into some trouble building the do_mysql gem on my Macbook Pro. It’s a 64-bit machine, so I installed the 64-bit version of MySQL, and that was my first mistake…

The do_mysql gem installed fine. I didn’t realise there was a problem until I tried to run some migrations in a blank database:

$ merb -i
 ~ Loaded DEVELOPMENT Environment...
 ~ loading gem 'dm-validations' ...
 ~ loading gem 'merb_datamapper' ...
 ~ loading gem 'merb-haml' ...
 ~ loading gem 'dm-core' ...
 ~ Merb::Orms::DataMapper::Connect block.
 ~ Connecting to database...
 ~ Checking if we need to use DataMapper sessions
 ~ Merb::Orms::DataMapper::Connect complete
 ~ Parent pid: 9932
 ~ Compiling routes...
>> Post.auto_migrate!
dyld: lazy symbol binding failed: Symbol not found: _mysql_init
  Referenced from: /Library/Ruby/Gems/1.8/gems/do_mysql-0.9.6/lib/do_mysql_ext.bundle
  Expected in: dynamic lookup

dyld: Symbol not found: _mysql_init
  Referenced from: /Library/Ruby/Gems/1.8/gems/do_mysql-0.9.6/lib/do_mysql_ext.bundle
  Expected in: dynamic lookup

Oh dear. I did some Googling, and found a few articles from people having similar problems (e.g. Ben Novakovic, who had problems with the mysql gem).

It turns out that the gems cannot be compiled properly against a 64-bit version of MySQL, as they don’t take proper notice of the output of mysql_config. Chris Cruft investigated the problem and ended up working around it by using the 32-bit version.

Setting the ARCHFLAGS environment variable (as suggested in Chris’s article) to the output of mysql_config --cflags didn’t work for me either, so I’ve gone for the same solution as Chris and installed the i386 build of MySQL. The do_mysql gem now works fine.

If you check the comments on Chris’s post you’ll see that some people have managed to get the mysql gem to build properly against 64-bit MySQL. I’ve opted not to bother trying to shovel lots of command line switches at the problem because I’m installing merb with edgy, and am getting by fine with the 32-bit version for now…

Debugging JavaScript in WebKit

2008-10-02T19:59:00Z

If you do much web development you’re probably a big fan of the Firebug plugin for Firefox. It really is a rare gem in the web developer’s arsenal. When I’m not developing I prefer to use Safari – it seems faster and leaner, so I often find that I’ve got both browsers open at once.

A couple of days ago something rather fantastic happened – Safari got a makeover.

More specifically, the WebKit Web Inspector got a makeover, and now seems to be a viable competitor to Firebug. Here’s a quick screenshot of the debugger in action:

I’m also a big fan of the JavaScript code completion in the console. Here’s an example, with the code completion reminding me of the name of Prototype’s _getElementsByXPath() function (which I rarely manage to type correctly in Firebug, but find very useful for debugging my XPath statements):

You can see from the position of the cursor that I’ve typed document._get, and that Web Inspector is suggesting “ElementsByXPath”. Pressing tab cycles between matching names, and the right cursor key completes the function, moving the cursor to the end of the line. It’s the little things…

You can open the console at any time by pressing Esc (or clicking the second icon from the left at the bottom of the Web Inspector window).

There are loads more features. Check out the WebKit blog post that covers the redesign, or get yourself over to the WebKit nightly build page and download a copy (it doesn’t affect your existing installation of Safari, but can function as a drop in replacement, so there’s no risk and you can flip between them at will).

Configure the Expires header for Rails under nginx

2008-09-13T06:22:00Z

The images, CSS and JavaScript files served up by your Rails application can usually be cached by the web browser, rather than being downloaded (or checked) every time the browser loads a page on your site. For a properly configured site the net effect is faster page load times and a vastly improved user experience. Rails gives you some help here, but it doesn’t happen automatically – you need to configure your web server to set the HTTP Expires header. This article explains how to configure the expires header for a Rails application running behind the nginx HTTP proxy server.

Nginx is popular amongst Rails developers for fronting a cluster of mongrel web servers. I’ll not say much more about it, on the assumption that this article probably won’t be of much interest to you if you’re not already using it.

So how do we speed your Rails site up?

When a web browser downloads a web page for the first time it also retrieves copies of the images, CSS and JavaScript files that are required to display it. On subsequent requests it will typically make a new connection to the web server to verify whether each of these files has been updated. When you know that the files won’t have changed you can gain a significant performance increase by telling the browser when it first downloads the file that it doesn’t need to check it again. More information on the Expires header (and other great performance enhancing tips) can be found amongst Yahoo’s Best Practices for Speeding Up Your Web Site.

Files that don’t change between requests are known in the Rails community as “static assets”. When running in production mode your Rails application will append all URLs for static assets with the file’s timestamp (in seconds since the 1 January 1970). Here’s an example:

http://effectif.com/stylesheets/blog.css?1221178271

If we were to edit the file and redeploy the application the timestamp would be updated, and the browser would realise that it needed to download a new copy of the file.

The Rails documentation for AssetTagHelper covers how to configure a far future expires header under Apache (scroll down to the section entitled “Using asset timestamps”).

The equivalent nginx configuration looks like this:

if ($request_uri ~* ".(ico|css|js|gif|jpe?g|png)\?[0-9]+$") {
    expires max;
    break;
}

Note that we’re matching suitable file types, but only setting the expires header for files that have a timestamp appended.

Here’s the configuration again, in the context of a location directive:

location / {
    root /var/apps/myapp/current/public;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect false;

    if ($request_uri ~* ".(ico|css|js|gif|jpe?g|png)\?[0-9]+$") {
        expires max;
        break;
    }
    if (-f $request_filename) {
        break;
    }
    if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
    }
    if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
    }

    proxy_pass http://mymongrels;
}

Testing your configuration

It’s all well and good sticking the config in nginx and restarting it, but it’s nice to know that the stuff works, right? If you’re not already using it, get a copy of Firefox and install Firebug. Then install Yahoo’s YSlow plugin for Firebug.

I recommend that you read Yahoo’s introduction to YSlow to learn how to use it. You can check whether the expires header is set for all the assets required by your web page using the Components view, a screenshot of which I’ve cheekily purloined here:

Learning Erlang with the Pragmatic Programmer Screencasts

2008-06-18T22:15:00Z

I’ve been meaning to get around to playing with Erlang for a while now. Tonight I set aside 30 minutes to watch the first of the Pragmatic Programmer’s Erlang Screencasts by Kevin Smith: Erlang By Example.

Each screencast is very reasonably priced at $5 a piece. I bought the first one, in which you learn how to build a simple online chat system (see the introductory video for an overview of what’s covered in the entire series). I downloaded it immediately, excited to think that in only 30 minutes time I’d know lots more about Erlang than I did when I woke up this morning.

Installing Erlang

While the video was downloading I kicked off the installation of Erlang on my Mac with MacPorts…

$ sudo port install -d erlang

and grabbed a copy of the Erlang TextMate bundle:

$ cd ~/Library/Application\ Support/TextMate/Bundles
$ svn co https://macromates.com/svn/Bundles/trunk/Bundles/Erlang.tmbundle
$ osascript -e 'tell app "TextMate" to reload bundles'

Following Along

If you’ve never done any Erlang at all (and to all intents and purposes, I hadn’t) then you’ll see quite a bit of new syntax during this screencast. It won’t all be explained to you straight away, but don’t worry about that as it doesn’t detract from what’s going on.

Kevin edits his code in Emacs and compiles it with some magic incantation using an Erlang interpreter that is running in an Emacs buffer. It seems to be assumed that when you start watching the screencast you already know how to compile a code module. Worry not; if you’re trying to run the examples in a terminal, you should be able to get by with something like this:

$ erl
Erlang (BEAM) emulator version 5.6.2 [source] [async-threads:0] [kernel-poll:false]

Eshell V5.6.2  (abort with ^G)
1> c('message_router').
{ok,message_router}
2>

“c” means “compile”. Note the “.” character, which is used as a terminator in Erlang (it’s strange that this should seem noteworthy, given that this is exactly how we use it in English!). Incidentally, if you’re an Emacs fan then Kevin has written up his Emacs configuration on his blog.

Kevin does a good job of giving you a the flavour of concurrent programming in Erlang. Concentrating on the syntax would have distracted from the overall message, without contributing much to your understanding (and if you really must know the syntax, there’s plenty of Erlang documentation that will fill in the gaps).

Once the simple chat system was put together the code was refactored. While recompiling my edits and fixing my typos I picked up some useful tips on using the interactive interpreter.

So in summary, I enjoyed it. I’m a long way from being able to claim that I know Erlang, but I know hell of a lot more about it than I did half an hour ago, and you can’t say fairer than that! For $5 a throw, the screencasts are a bargain.

Useful Links

The Erlang By Example Screencasts.
Getting Started with Erlang (also available as a PDF).
Kevin Smith’s blog.

database.yml should be checked in

2008-05-24T11:05:00Z

It’s generally acknowledged to be good practice to keep your production database.yml file away from your version control system. Why? So that people snooping around your version control system don’t find your database password. There are plenty of examples out there that tell you how to achieve this with Capistrano.

When I say that database.yml should be checked in, I’m talking the version that you use during development.

Teams often resist sharing database config files so they can keep their individual database passwords private and specify operating system specific settings (such as the location of MySQL’s socket file).

Can you share configuration between different developers? A couple of approaches spring to mind.

Don’t store database.yml in version control, but create database.yml.example instead. Force each developer to edit it themselves when they check out the code.
Persuade your team mates to use the same password and check that in.

Neither is satisfactory. When I check my code out I expect it to just work, and I’m sure as hell not going to give anybody who can access the source a free run at my laptop’s database.

Read on for my solution.

So how do you check in database.yml without storing your password in version control? I’m using MySQL at the moment. As we’ve already established, I’m a lazy git, so I hate having to specify a password when I connect to MySQL. I’ve set my local database password in the ~/.my.cnf file:

$ cat ~/.my.cnf
[mysql]
user = root
password = gumo4Gum

It’s a good idea to set the permissions on that file so that only you can read it:

$ chmod 600 ~/.my.cnf

Now I can just connect without any trouble. Neat.

$ mysql myapp_development
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.0.45-Debian_1ubuntu3.3-log Debian etch distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

Once you’ve gone to the trouble to make it easy to login to MySQL, why not let Rails connect using the same settings when it starts up?

This is how I’ve been doing it (“Oh my God! He’s used Python!”):

<%
def get_mysql_password
  script =<<EOF
import ConfigParser
parser = ConfigParser.SafeConfigParser()
parser.read("#{ENV['HOME']}/.my.cnf")
print parser.get("mysql", "password")
EOF
  %x(python -c '#{script}')
end
%>

mysql: &mysql
  adapter: mysql
  username: root
  password: <%= get_mysql_password %>
  host: localhost
  socket: <%= [
    '/var/lib/mysql/mysql.sock',
    '/var/run/mysqld/mysqld.sock',
    '/tmp/mysqld.sock',
    '/tmp/mysql.sock',
  ].detect { |socket| File.exist?(socket) } %>

development:
  database: myapp_development
  <<: *mysql

test: &test
  database: myapp_test
  <<: *mysql

Before I explain the Python, note the sneaky socket trick that allows Rails to try lots of different locations for MySQL’s Unix socket—it allows you to develop the same app on a Mac, or various flavours of Linux/BSD, without editing database.yml.

So then. This Python code. I suspect there’s a rather good way of doing it in Ruby, but when I first started developing with Rails I’d just spent five years doing Python full time, and I knew that the ConfigParser module was designed to read files like ~/.my.cnf. So I used it, and it stuck.

Send me improvements!

How would you do it?

Can you write a neat snippet of Ruby that achieves the same thing as the Python code?

If anybody knows a way to make this trick work with Merb I’d love to hear it. Last time I tried, Merb wasn’t processing the database config file with erb…

Easy git-svn for Rails (or "git-me-up")

2008-04-24T23:13:00Z

I’ve been trying to persuade git-svn to work properly with Rails plugins that have been installed via svn:externals. Whilst working out how to do it I stumbled across several great articles, but I couldn’t get any of the solutions presented to work perfectly. Samuel Tesla’s article is especially informative, but for a long time I couldn’t stop git-svn from trying to commit Git metadata back into my Subversion repository…

After a bit of head scratching I got it sorted, and wrapped the process up in a little utility called git-me-up.

What does git-me-up do?

In a nutshell, it:

Clones the latest version of the source in your Subversion repository into a new local Git repository. It doesn’t import any history for performance reasons.
Recreates empty directories that git-svn ignored (but that your app might need).
Converts any svn:ignore Subversion properties into entries in the .git/info/exclude file.
Checks for Rails plugins installed via svn:externals and clones them into their own git repositories. Each external plugin is then symlinked into vendor/plugins.
Creates a local working branch.

In other words, it takes all the hard work out of creating a local Git repository that points at your Subversion repo.

Installation

You need git-svn installed first. On a Mac, install MacPorts and then type:

$ sudo port install -u git-core +svn +bash_completion

When the package has finished building run these commands to benefit from bash completion:

$ cp /opt/local/etc/bash_completion.d/git ~/.git-bash-completion.sh
$ echo '[ -f ~/.git-bash-completion.sh ] && . ~/.git-bash-completion.sh' \
    >> ~/.bashrc
$ . ~/.bashrc

On Ubuntu Linux:

$ sudo apt-get install git-svn

Installing git-me-up

The source is hosted on GitHub. You can either download the tar ball or pull the source with git:

$ git clone git://github.com/gma/git-me-up.git

The Makefile will install it into /usr/local/bin:

$ cd git-me-up
$ sudo make install

Using git-me-up

This is pretty easy. Let’s start with an empty directory so you can see what gets created:

$ pwd
/home/graham/test
$ git-me-up https://myhost.com/path/to/myproject/trunk myproject
git-me-up: finding latest revision of https://myhost.com/path/to/myproject/trunk
git-me-up: creating git repository in /home/graham/test/myproject
git-me-up: making empty directory: app/views/comments
git-me-up: making empty directory: app/views/pages
git-me-up: making empty directory: components
git-me-up: making empty directory: log
git-me-up: making empty directory: test/mocks/development
git-me-up: making empty directory: test/mocks/test
git-me-up: making empty directory: tmp/cache
git-me-up: making empty directory: tmp/pids
git-me-up: making empty directory: tmp/sessions
git-me-up: making empty directory: tmp/sockets
git-me-up: setting ignored files from subversion (this can take a while)
git-me-up: checking for plugins installed with svn:externals
git-me-up: finding latest revision of https://myhost.com/path/to/plugin/trunk
git-me-up: creating git repository in /home/graham/test/myproject/../plugins/site_theme
git-me-up: setting ignored files from subversion (this can take a while)
git-me-up: symlinking site_theme into myproject/vendor/plugins
git-me-up: creating 'work' branch on /home/graham/test/myproject
Switched to a new branch "work"
$ ls
myproject/  plugins/

If you’re running on a Mac you may see this error message part way through the output:

No such file or directory: PROPFIND request failed on '/app': Could not open the requested SVN filesystem at /opt/local/bin/git-svn line 1872

It appears as though the git svn show-ignore command (using the current version of git from MacPorts–version 1.5.4.4) doesn’t work on my Mac. If you see this too don’t worry about it; it will only mean that Git doesn’t know which files you’ve chosen to ignore in your Subversion repo.

Now we’re ready to use our new Git repo. Note that we’ve got a work branch by default:

$ cd myproject
$ git branch
  master
* work

If you want to call the branch something else, set the BRANCH environment variable before you run git-me-up.

Samuel recommends a good work flow for working with Subversion (skip down to “Step Five: Using this thing”) . When you want to update your working branch from subversion, try this:

$ git checkout master  # switches to local master branch
$ git svn rebase       # pulls recent changes from subversion
$ git checkout work    # switches to local work branch
$ git rebase master    # pulls recent changes to master into work

There’s a good description of why rebase is so great on James Bowes’s blog.

When you want to commit changes back into subversion:

$ git checkout work    # you were probably there already
... run your tests ...
$ git commit -a -m "Blah blah blah..."
$ git checkout master
$ git merge work
$ git svn rebase       # checks for fresh changes in subversion
... run tests again if rebase pulled any changes ...
$ git svn dcommit

Make sure that you use git svn rebase to pull upstream changes into your master branch before you check in.

What git-me-up does with svn:externals plugins

The Rails app that I cloned above has a plugin that is installed via svn:externals called site_theme. I usually use piston or braid to install Rails plugins, but when the plugin is stored on my own servers I’ve always preferred to pull in the source directly by setting the svn:externals property.

I initially tried to convert svn:externals into Git submodules, but it didn’t work well. No matter what I tried, I couldn’t persuade git-svn to ignore the submodule when committing back to Subversion. It turns out that it’s easy to achieve with a symbolic link.

git-me-up cloned the site-theme plugin into a Git repository, then made it available with a symbolic link:

$ ls
myproject/  plugins/
$ ls plugins/
site_theme/
$ ls -l myproject/vendor/plugins/ | grep site_theme
lrwxrwxrwx  1 graham development   32 Apr 24 18:41 site_theme -> ../../../plugins/site_theme/

To avoid checking the symlink into Subversion, git-me-up told Git to ignore it:

$ grep site_theme myproject/.git/info/exclude
/vendor/plugins/site_theme

The result is a trouble free alternative to submodules that allows you to integrate Git with svn:externals in your Subversion repo.