Notes from the Ruby Manor 2008 (part 1)

I'm lucky enough to be at RubyManor today; a Ruby conference organised by Ruby users, for Ruby users, costing the grand total of twelve of your British pounds. A bargain, as you'll see if you check the lineup of talks and speakers.

I started making notes, but it seems to have evolved into some kind of blog post. Apologies for typos, crap grammar and glaring errors.

Paul Battley -- I'm an Evil iPlayer hacker

Paul Battley is a hacker. BBC News said so, so it must be true. Paul started by running through the history of iPlayer's development. It launched in July 2007 in open beta, but required Windows XP and a custom client with built-in DRM. In December 2007 a flash streaming version was released, which worked on Windows, Mac and Linux. However you could only watch content for seven days.

In March 2008 an iPhone version of the iPlayer was released. The iPhone doesn't support flash, so the iPhone version streams MPEG4 over HTTP. When first released it worked by sniffing the user agent, and checking for the string "iphone"! Within hours people had released scripted solutions for downloading the videos, but one week later the BBC had plugged the hole.

Wasting no time, Paul discovered that the video is pulled with a different user agent to the page, and an extra HTTP header was required. All was well with the world, but in June the downloaded files stopped working. Comparing old files to the new ones, they discovered that the BBC had XOR'd some data in the middle of the MPEG files. That was easy to work around.

Later in June version 2 of the iPlayer came out with new URLs, but no new counter measures. In September the iPlayer started to serve radio, which can also be downloaded with Paul's scripts.

So the basic technique for downloading content from the iPlayer is to configure your client to pretend to be an iPhone.

Paul then talked about how to get a programme's ID from some JavaScript served up with the programme's web page. This was one of the hardest parts of the job, requiring him to write a JavaScript tokenzer.

The flash version of the iPlayer uses the RTMP protocol, to make it a bit more difficult for people to "steal" the content. The next thing that Paul decided to try and reverse engineer RTMP and write a Ruby library to process RTMP streams. He's not finished yet, but it looks as though he's got quite a long way through the job.

Lessons learnt:

  • Releasing Ruby software to users is hard.
  • Writing GUIs is tedious (but check out the iPlayer grabber).
  • Reverse engineering is fun and enlightening.
  • Obfuscation doesn't work.
  • You need a feedback channel (or people will make one).

It was a cracking presentation. More information is available here:

If you want a graphical app for the Mac, iPlayer grabber looks great.

Dan Webb -- 8 Minutes on Rack

Dan Webb's main message was that rack is worth everybody's time, not just that of people who write web frameworks. But what is it? It's not a framework. It's not a library. It's just a convention.

The convention is this. You have a call method on a ruby object that returns an array of three values; the HTTP status code, a hash of headers to send to the client, and the response body. Here's an example:

app = Proc.new do |env|
  [200, { "Content-Type" => "text/plain" }, "Hello World"]
end
Rack::Handler::Thin.run(app, :Port => 4000)

The response body can be any object that responds to the each method, such as an open file object.

file = File.new("myfile.xml")
[200, { "Content-Type" => "application/xml" }, file]

For binary files you can write your own class to stream anything you like. Here's an example of an each method that you could use to stream a binary file:

def each
  File.open(@file, "rb") do file
    while part = file.read(8192)
      yield part
    end
    File.delete(@file)
  end
end

Why should you care? Passenger, Mongrel, CGI, SCGI, FastCGI, Thin, Webrick, Fuzed (the list goes on).

Other things that Dan thinks are worth looking into:

  • The Rack Gem
  • Middleware (very simple and clever, well worth a look)
  • Rack and Passenger
  • rackup

Check http://rack.rubyforge.org and read the (slightly impenetrable) documentation. Dan will be writing some Rack stuff up on his blog, and his slides are available on slideshire.

David Black popped his hand up at the end and pointed out that strings in Ruby 1.9 don't have an each method. Ooops.

George Palmer -- Nanite

George Palmer gave an introduction to Nanite. I missed the very beginning because I was chatting in the canteen. Sorry.

There are currently three different broad brush approaches available to running background tasks in Ruby web frameworks:

  • Fork into background of rails process (e.g. spawn, run_later).
  • Record a job to a file or database, for handling by a background deamon.
  • Fork into some kind of queue (e.g. backgroundRB, beanstalk, starling, etc.).

Nanite is great for heavy lifting, and is designed for running jobs on other servers. It's based on RabbitMQ; a queuing system written in Erlang. As an aside, when installing Erlang on a Mac you need to avoid using the R12B5 of Erlang; this means that you don't want to use MacPorts, and are better off just downloading the source form the Erlang site.

AMQP stands for Advanced Messaging Queueing Protocol. It's an enterprise quality protocol, used by (for example) big financial corporations. It's also language neutral. RabbitMQ implements AMQP 0.8.

To install AMQP for Ruby you need the eventmachine gem, then grab the source from github and install the gem:

$ sudo gem install eventmachine
$ git clone git://github.com/tmm1/amqp.git
$ cd amqp && rake gem && sudo gem install amqp-<version>.gem

Nanite mappers control and track the work. You can run as many mappers as you like; they get updates from the mapper exchange. Mappers are run either inside your Rails/Merb app (on Thin) or via the command line.

Mappers assign work to agents. Agents announce their presence when they start up so that the mappers know that they're available. Actors know how to do the work. Here's an example actor:

class Manor < Nanite::Actor
  expose :name

  def name(vars)
    # Do something interesting
    :result => "RubyManor"
  end
end

Nanite:Dispatcher.register(Manor.name)

There were a lot more code examples that I didn't manage to catch, but George will be uploading the slides to his blog, and I recommend taking a look.

This is what I did catch:

  • Nanite has no concept of security, but relies on the security in RabbitMQ.
  • George recommends monitoring your agents with a program like monit or god.
  • Once a nanite request has been sent to an agent and processed by an actor you can't check it for status; use the database for that.
  • You can send files to your agents (e.g. an image that needs resizing).
  • Nanite allocates work by looking at the currently available agents and sending the job to (by default) the least loaded server. If you prefer you can choose to send a request to all agents, or a randomly selected agent.
  • You can also define your own metrics (using Ruby) for an agent's "load".

Finally, George ran a quick demo. Members of the audience fired up a nanite agent (an example agent that is distributed with the nanite source code) that announced itself to a server running on his laptop, and he used the server to list the gems on everybody's laptops. A neat demo.

For more information, try these links:

The Nanite docs aren't currently that great, but the code is very readable.

Rob McKinnon -- Rugalytics

Rob wondered if there was a Ruby API for Google Analytics. jnunemaker had written statwhore that gets you some XML, and wrapped it with helper methods such as pageviews and visits.

So that was cool, but Rob wanted more. He wanted to emerge class definitions at runtime from the data. So he forked statwhore, then hacked on it so much that he renamed it to Rugalytics. Then he converted the XML to CSV to make it easier to work with.

Rob is now using Rugalytics to put a list of popular pages on his site, pulling the data direct from Google Analytics. Fantastic stuff.

To get an Analytics report you pass the report name when calling analytics. You can find the available reports by calling the brand new profile.report_names method.

Google have a tendency to tweak the format of the CSV files, and this has been known to break Rugalytics. There are also some language issues -- if you've got your Google account setup to US english then you end up with a method (dynamically created from the CSV) called pageviews. If you're Google account is configured for UK english you used to get a method called page_views (but Rob has now hardcoded it to pageviews). In Italian you get pagine_visualizzate. Nice! The README file now advises that you choose US/UK english.

Rob also has a local web server (which will be included in the gem) -- similar to a Rack example that Dan talked about earlier -- that serves JavaScript to Rob's Greasemonkey enabled browser. Greasemonkey then annotates the pages on his site (for him, and only for him) showing which keywords people have used to find the page that he's currently looking at. Pretty impressive.

See Rob's account on github to get the code. Rugalytics is also available as a gem.

$ sudo gem install rugalytics

Murray Steele and Martin Saddler -- GUI Manor Born

Apparrently Ben Griffiths suggested all talks should be named around the phrase "To The Manor Born". As Murray put it, "at least we tried".

So Murray and Martin are talking about how to build GUI apps with Ruby. There are plenty of GUI toolkits available to Ruby, such as wxRuby, fxRuby, etc. However for their example app (a Twitter client) Martin used Monkeybars and Murray used Shoes.

Monkeybars

Monkeybars is built using jRuby and produces a swing front end. What you really need (to follow along with Martin's demo) is JRuby, Netbeans and Ruby gems. To start a new Monkeybars project you run a command that makes you a blank project (much as Rails does):

$ monkeybars myapp
$ cd myapp
$ rawr install  # choose option 2
$ jruby -S rake generate ALL='src/login'

At this point Martin demonstrated (in the style of Blue Peter -- "here's one I made earlier") how to build his Twitter client using Netbeans. There's a GUI builder that allows you to drag and drop GUI controls into the user interface. You can then bind the widgets to the code using Netbeans' normal GUI builder tools.

Monkeybars uses a form of the MVC design pattern. The view binds the widgets defined in the GUI builder to Ruby code that you can access in the controller. The controller and view are pretty much standard stuff.

You can easily create callback methods in your controller for widgets in your UI by taking the name of the widget (e.g. login_button) and creating a method called login_button_action_performed. Seems prety straightforward.

You can use the rawr program to package your app.

The downsides? Not everything works in JRuby at the moment (for example Martin couldn't use the twitter4r gem). Build paths can get mucked up. Assumes knowledge of Java, there's a learning curve, etc. But these problems are being worked on.

The good bits? GUI Builder. Interface to mature Java libs. It's all Ruby. Cross platform distribution. No platform dependency on Ruby.

See tweetobix.com to follow through with Martin's example.

Shoes

Shoes is a GUI framework written by whytheluckystiff. It was designed as a replacement for an HTML rendering engine for use when teaching people to code, so it's pretty familiar to people who know a fair bit of HTML.

To install it you don't install a gem. Shoes is it's own version of Ruby. Download it and install it.

The code is pretty simple. Murray's app is called Talon. You start with a block, which makes a window. Then you can do stuff to the window. Here's the main window (with no widgets in it):

Shoes.app :title => "Talon" do
  background "#fff"
  background "talon.jpg",
             :bottom => 0,
             :right => -20
  flow :width => "100%" do
    background "#df9", :curve => 12
    title "Talon",
          :stroke => "#691FFF",
          :align => "center"
  end
end

The styling is a lot like CSS. Backgrounds can be layered.

To layout widgets you don't worry about layout managers; you use stacks and flows. Stacks allow you to lay things out vertically (a bit like a vbox in GTK). Flows are similar, but run horizontally (a GTK hbox).

To load a gem you must reference it in the setup block. This will ensure that when somebody runs your app that all required gems will be installed, rather than throwing a backtrace.

Shoes.setup do
  gem "twitter"
end

require "twitter"

Murray showed a lot of example code for laying out and animating widgets. It seems very clear and succinct.

You can ask Shoes to package your app with one click, and it builds a package for Windows, Mac, etc.

For more info you can access the manual:

$ shoes --manual

You should also check out Nobody Knows Shoes, which -- Murray assures us -- is fantastic.

The code for Murray's example (Talon) is available on github.

Cracking stuff.

The Afternoon Session

At this point we went off to the pub for lunch, but when we got back I took notes in the afternoon session too. See Notes from the Ruby Manor (part 2).

Related articles

If you enjoyed this post, you might also enjoy: