Effectif Development - Home

Optimising PNG images for the web

2008-11-17T23:36:00Z

Stoyan Stefanov has recently written an interesting series of articles on optimising image files for the web. In his latest article he shows how to use pngcrush to safely shrink your files, one file at a time:

$ pngcrush -rem alla -brute -reduce src.png dest.png

This evening I found myself wanting to repeat that command for all the PNG files in my current directory, and then (having reviewed the new images) overwrite the originals. Rather than do it by hand I did what all pragmatic programmers do these days, and wrote a script. Here it is in action:

$ pngcrushall 
Crushing test4012.png ... ok
Crushing test9200a.png ... ok
Crushing test9200b.png ... ok
Overwrite original files? (y/N) y
Overwriting test4012.png ... ok
Overwriting test9200a.png ... ok
Overwriting test9200b.png ... ok

You can get it from github – download pngcrushall.

Obviously you’ll need to install pngcrush too. On a Mac this is pretty easy with MacPorts:

$ sudo port install pngcrush

Integration Testing for Multiple Web Applications – Introducing Integral

2008-11-11T23:20:00Z

I’ve recently been developing a Rails app that relies on a database that is maintained by another team as part of a different application. As you can imagine, unexpected changes to the database schema can bring our Rails app to it’s knees if they’re deployed to the live server before our application is updated.

So far we’ve been lucky, but the issue has been at the back of our minds for a while. So this week I sat down and cooked up a solution, and we’ve called it Integral.

Integral consists of some Thor tasks that:

Run tests on your test server.
Keep track of which versions of your applications work well together.
Can be integrated into your deployment recipes to ensure that deploying an app should never break your live server again…

All the Integral code is on github, and I’ve written some fairly extensive documentation on the Wiki. Please try it out, give us some feedback, fork it, burn it, give us some feedback, etc. etc.

Installing merb with thor

2008-10-05T08:36:00Z

Update (22 October): Things move very fast around here – the merb.thor file has been overhauled and now all you have to do to get the latest edge is download a single .thor file and run one command.

Matt Aimonetti wrote a neat post on installing merb edge with thor. Thor seems to have become the defacto technique for installing merb edge these days, and I’m going to give it a go in preference to installing merb with sake, as I had been doing.

Since Matt wrote his article the merb.thor file underwent a serious overhaul during the sprint at MerbCamp – what you need to do these days is quite a bit simpler:

curl -L http://merbivore.com/merb.thor > merb.thor
sudo thor merb:stack:install --edge

The source code for all the Merb gems will be checked out into a directory called ./src.

To get the full low down on how to use Thor with Merb, head over to the Thor Howto on the excellent new Merb wiki.

Building do_mysql on a 64-bit Mac

2008-10-02T22:24:00Z

Whilst installing merb with edgy I’ve had run into some trouble building the do_mysql gem on my Macbook Pro. It’s a 64-bit machine, so I installed the 64-bit version of MySQL, and that was my first mistake…

The do_mysql gem installed fine. I didn’t realise there was a problem until I tried to run some migrations in a blank database:

$ merb -i
 ~ Loaded DEVELOPMENT Environment...
 ~ loading gem 'dm-validations' ...
 ~ loading gem 'merb_datamapper' ...
 ~ loading gem 'merb-haml' ...
 ~ loading gem 'dm-core' ...
 ~ Merb::Orms::DataMapper::Connect block.
 ~ Connecting to database...
 ~ Checking if we need to use DataMapper sessions
 ~ Merb::Orms::DataMapper::Connect complete
 ~ Parent pid: 9932
 ~ Compiling routes...
>> Post.auto_migrate!
dyld: lazy symbol binding failed: Symbol not found: _mysql_init
  Referenced from: /Library/Ruby/Gems/1.8/gems/do_mysql-0.9.6/lib/do_mysql_ext.bundle
  Expected in: dynamic lookup

dyld: Symbol not found: _mysql_init
  Referenced from: /Library/Ruby/Gems/1.8/gems/do_mysql-0.9.6/lib/do_mysql_ext.bundle
  Expected in: dynamic lookup

Oh dear. I did some Googling, and found a few articles from people having similar problems (e.g. Ben Novakovic, who had problems with the mysql gem).

It turns out that the gems cannot be compiled properly against a 64-bit version of MySQL, as they don’t take proper notice of the output of mysql_config. Chris Cruft investigated the problem and ended up working around it by using the 32-bit version.

Setting the ARCHFLAGS environment variable (as suggested in Chris’s article) to the output of mysql_config --cflags didn’t work for me either, so I’ve gone for the same solution as Chris and installed the i386 build of MySQL. The do_mysql gem now works fine.

If you check the comments on Chris’s post you’ll see that some people have managed to get the mysql gem to build properly against 64-bit MySQL. I’ve opted not to bother trying to shovel lots of command line switches at the problem because I’m installing merb with edgy, and am getting by fine with the 32-bit version for now…

Debugging JavaScript in WebKit

2008-10-02T19:59:00Z

If you do much web development you’re probably a big fan of the Firebug plugin for Firefox. It really is a rare gem in the web developer’s arsenal. When I’m not developing I prefer to use Safari – it seems faster and leaner, so I often find that I’ve got both browsers open at once.

A couple of days ago something rather fantastic happened – Safari got a makeover.

More specifically, the WebKit Web Inspector got a makeover, and now seems to be a viable competitor to Firebug. Here’s a quick screenshot of the debugger in action:

I’m also a big fan of the JavaScript code completion in the console. Here’s an example, with the code completion reminding me of the name of Prototype’s _getElementsByXPath() function (which I rarely manage to type correctly in Firebug, but find very useful for debugging my XPath statements):

You can see from the position of the cursor that I’ve typed document._get, and that Web Inspector is suggesting “ElementsByXPath”. Pressing tab cycles between matching names, and the right cursor key completes the function, moving the cursor to the end of the line. It’s the little things…

You can open the console at any time by pressing Esc (or clicking the second icon from the left at the bottom of the Web Inspector window).

There are loads more features. Check out the WebKit blog post that covers the redesign, or get yourself over to the WebKit nightly build page and download a copy (it doesn’t affect your existing installation of Safari, but can function as a drop in replacement, so there’s no risk and you can flip between them at will).

Configure the Expires header for Rails under nginx

2008-09-13T06:22:00Z

The images, CSS and JavaScript files served up by your Rails application can usually be cached by the web browser, rather than being downloaded (or checked) every time the browser loads a page on your site. For a properly configured site the net effect is faster page load times and a vastly improved user experience. Rails gives you some help here, but it doesn’t happen automatically – you need to configure your web server to set the HTTP Expires header. This article explains how to configure the expires header for a Rails application running behind the nginx HTTP proxy server.

Nginx is popular amongst Rails developers for fronting a cluster of mongrel web servers. I’ll not say much more about it, on the assumption that this article probably won’t be of much interest to you if you’re not already using it.

So how do we speed your Rails site up?

When a web browser downloads a web page for the first time it also retrieves copies of the images, CSS and JavaScript files that are required to display it. On subsequent requests it will typically make a new connection to the web server to verify whether each of these files has been updated. When you know that the files won’t have changed you can gain a significant performance increase by telling the browser when it first downloads the file that it doesn’t need to check it again. More information on the Expires header (and other great performance enhancing tips) can be found amongst Yahoo’s Best Practices for Speeding Up Your Web Site.

Files that don’t change between requests are known in the Rails community as “static assets”. When running in production mode your Rails application will append all URLs for static assets with the file’s timestamp (in seconds since the 1 January 1970). Here’s an example:

http://effectif.com/stylesheets/blog.css?1221178271

If we were to edit the file and redeploy the application the timestamp would be updated, and the browser would realise that it needed to download a new copy of the file.

The Rails documentation for AssetTagHelper covers how to configure a far future expires header under Apache (scroll down to the section entitled “Using asset timestamps”).

The equivalent nginx configuration looks like this:

if ($request_uri ~* ".(ico|css|js|gif|jpe?g|png)\?[0-9]+$") {
    expires max;
    break;
}

Note that we’re matching suitable file types, but only setting the expires header for files that have a timestamp appended.

Here’s the configuration again, in the context of a location directive:

location / {
    root /var/apps/myapp/current/public;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect false;

    if ($request_uri ~* ".(ico|css|js|gif|jpe?g|png)\?[0-9]+$") {
        expires max;
        break;
    }
    if (-f $request_filename) {
        break;
    }
    if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
    }
    if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
    }

    proxy_pass http://mymongrels;
}

Testing your configuration

It’s all well and good sticking the config in nginx and restarting it, but it’s nice to know that the stuff works, right? If you’re not already using it, get a copy of Firefox and install Firebug. Then install Yahoo’s YSlow plugin for Firebug.

I recommend that you read Yahoo’s introduction to YSlow to learn how to use it. You can check whether the expires header is set for all the assets required by your web page using the Components view, a screenshot of which I’ve cheekily purloined here:

Learning Erlang with the Pragmatic Programmer Screencasts

2008-06-18T22:15:00Z

I’ve been meaning to get around to playing with Erlang for a while now. Tonight I set aside 30 minutes to watch the first of the Pragmatic Programmer’s Erlang Screencasts by Kevin Smith: Erlang By Example.

Each screencast is very reasonably priced at $5 a piece. I bought the first one, in which you learn how to build a simple online chat system (see the introductory video for an overview of what’s covered in the entire series). I downloaded it immediately, excited to think that in only 30 minutes time I’d know lots more about Erlang than I did when I woke up this morning.

Installing Erlang

While the video was downloading I kicked off the installation of Erlang on my Mac with MacPorts…

$ sudo port install -d erlang

and grabbed a copy of the Erlang TextMate bundle:

$ cd ~/Library/Application\ Support/TextMate/Bundles
$ svn co https://macromates.com/svn/Bundles/trunk/Bundles/Erlang.tmbundle
$ osascript -e 'tell app "TextMate" to reload bundles'

Following Along

If you’ve never done any Erlang at all (and to all intents and purposes, I hadn’t) then you’ll see quite a bit of new syntax during this screencast. It won’t all be explained to you straight away, but don’t worry about that as it doesn’t detract from what’s going on.

Kevin edits his code in Emacs and compiles it with some magic incantation using an Erlang interpreter that is running in an Emacs buffer. It seems to be assumed that when you start watching the screencast you already know how to compile a code module. Worry not; if you’re trying to run the examples in a terminal, you should be able to get by with something like this:

$ erl
Erlang (BEAM) emulator version 5.6.2 [source] [async-threads:0] [kernel-poll:false]

Eshell V5.6.2  (abort with ^G)
1> c('message_router').
{ok,message_router}
2>

“c” means “compile”. Note the “.” character, which is used as a terminator in Erlang (it’s strange that this should seem noteworthy, given that this is exactly how we use it in English!). Incidentally, if you’re an Emacs fan then Kevin has written up his Emacs configuration on his blog.

Kevin does a good job of giving you a the flavour of concurrent programming in Erlang. Concentrating on the syntax would have distracted from the overall message, without contributing much to your understanding (and if you really must know the syntax, there’s plenty of Erlang documentation that will fill in the gaps).

Once the simple chat system was put together the code was refactored. While recompiling my edits and fixing my typos I picked up some useful tips on using the interactive interpreter.

So in summary, I enjoyed it. I’m a long way from being able to claim that I know Erlang, but I know hell of a lot more about it than I did half an hour ago, and you can’t say fairer than that! For $5 a throw, the screencasts are a bargain.

Useful Links

The Erlang By Example Screencasts.
Getting Started with Erlang (also available as a PDF).
Kevin Smith’s blog.

database.yml should be checked in

2008-05-24T11:05:00Z

It’s generally acknowledged to be good practice to keep your production database.yml file away from your version control system. Why? So that people snooping around your version control system don’t find your database password. There are plenty of examples out there that tell you how to achieve this with Capistrano.

When I say that database.yml should be checked in, I’m talking the version that you use during development.

Teams often resist sharing database config files so they can keep their individual database passwords private and specify operating system specific settings (such as the location of MySQL’s socket file).

Can you share configuration between different developers? A couple of approaches spring to mind.

Don’t store database.yml in version control, but create database.yml.example instead. Force each developer to edit it themselves when they check out the code.
Persuade your team mates to use the same password and check that in.

Neither is satisfactory. When I check my code out I expect it to just work, and I’m sure as hell not going to give anybody who can access the source a free run at my laptop’s database.

Read on for my solution.

So how do you check in database.yml without storing your password in version control? I’m using MySQL at the moment. As we’ve already established, I’m a lazy git, so I hate having to specify a password when I connect to MySQL. I’ve set my local database password in the ~/.my.cnf file:

$ cat ~/.my.cnf
[mysql]
user = root
password = gumo4Gum

It’s a good idea to set the permissions on that file so that only you can read it:

$ chmod 600 ~/.my.cnf

Now I can just connect without any trouble. Neat.

$ mysql myapp_development
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.0.45-Debian_1ubuntu3.3-log Debian etch distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

Once you’ve gone to the trouble to make it easy to login to MySQL, why not let Rails connect using the same settings when it starts up?

This is how I’ve been doing it (“Oh my God! He’s used Python!”):

<%
def get_mysql_password
  script =<<EOF
import ConfigParser
parser = ConfigParser.SafeConfigParser()
parser.read("#{ENV['HOME']}/.my.cnf")
print parser.get("mysql", "password")
EOF
  %x(python -c '#{script}')
end
%>

mysql: &mysql
  adapter: mysql
  username: root
  password: <%= get_mysql_password %>
  host: localhost
  socket: <%= [
    '/var/lib/mysql/mysql.sock',
    '/var/run/mysqld/mysqld.sock',
    '/tmp/mysqld.sock',
    '/tmp/mysql.sock',
  ].detect { |socket| File.exist?(socket) } %>

development:
  database: myapp_development
  <<: *mysql

test: &test
  database: myapp_test
  <<: *mysql

Before I explain the Python, note the sneaky socket trick that allows Rails to try lots of different locations for MySQL’s Unix socket—it allows you to develop the same app on a Mac, or various flavours of Linux/BSD, without editing database.yml.

So then. This Python code. I suspect there’s a rather good way of doing it in Ruby, but when I first started developing with Rails I’d just spent five years doing Python full time, and I knew that the ConfigParser module was designed to read files like ~/.my.cnf. So I used it, and it stuck.

Send me improvements!

How would you do it?

Can you write a neat snippet of Ruby that achieves the same thing as the Python code?

If anybody knows a way to make this trick work with Merb I’d love to hear it. Last time I tried, Merb wasn’t processing the database config file with erb…

Easy git-svn for Rails (or "git-me-up")

2008-04-24T23:13:00Z

I’ve been trying to persuade git-svn to work properly with Rails plugins that have been installed via svn:externals. Whilst working out how to do it I stumbled across several great articles, but I couldn’t get any of the solutions presented to work perfectly. Samuel Tesla’s article is especially informative, but for a long time I couldn’t stop git-svn from trying to commit Git metadata back into my Subversion repository…

After a bit of head scratching I got it sorted, and wrapped the process up in a little utility called git-me-up.

What does git-me-up do?

In a nutshell, it:

Clones the latest version of the source in your Subversion repository into a new local Git repository. It doesn’t import any history for performance reasons.
Recreates empty directories that git-svn ignored (but that your app might need).
Converts any svn:ignore Subversion properties into entries in the .git/info/exclude file.
Checks for Rails plugins installed via svn:externals and clones them into their own git repositories. Each external plugin is then symlinked into vendor/plugins.
Creates a local working branch.

In other words, it takes all the hard work out of creating a local Git repository that points at your Subversion repo.

Installation

You need git-svn installed first. On a Mac, install MacPorts and then type:

$ sudo port install -u git-core +svn +bash_completion

When the package has finished building run these commands to benefit from bash completion:

$ cp /opt/local/etc/bash_completion.d/git ~/.git-bash-completion.sh
$ echo '[ -f ~/.git-bash-completion.sh ] && . ~/.git-bash-completion.sh' \
    >> ~/.bashrc
$ . ~/.bashrc

On Ubuntu Linux:

$ sudo apt-get install git-svn

Installing git-me-up

The source is hosted on GitHub. You can either download the tar ball or pull the source with git:

$ git clone git://github.com/gma/git-me-up.git

The Makefile will install it into /usr/local/bin:

$ cd git-me-up
$ sudo make install

Using git-me-up

This is pretty easy. Let’s start with an empty directory so you can see what gets created:

$ pwd
/home/graham/test
$ git-me-up https://myhost.com/path/to/myproject/trunk myproject
git-me-up: finding latest revision of https://myhost.com/path/to/myproject/trunk
git-me-up: creating git repository in /home/graham/test/myproject
git-me-up: making empty directory: app/views/comments
git-me-up: making empty directory: app/views/pages
git-me-up: making empty directory: components
git-me-up: making empty directory: log
git-me-up: making empty directory: test/mocks/development
git-me-up: making empty directory: test/mocks/test
git-me-up: making empty directory: tmp/cache
git-me-up: making empty directory: tmp/pids
git-me-up: making empty directory: tmp/sessions
git-me-up: making empty directory: tmp/sockets
git-me-up: setting ignored files from subversion (this can take a while)
git-me-up: checking for plugins installed with svn:externals
git-me-up: finding latest revision of https://myhost.com/path/to/plugin/trunk
git-me-up: creating git repository in /home/graham/test/myproject/../plugins/site_theme
git-me-up: setting ignored files from subversion (this can take a while)
git-me-up: symlinking site_theme into myproject/vendor/plugins
git-me-up: creating 'work' branch on /home/graham/test/myproject
Switched to a new branch "work"
$ ls
myproject/  plugins/

If you’re running on a Mac you may see this error message part way through the output:

No such file or directory: PROPFIND request failed on '/app': Could not open the requested SVN filesystem at /opt/local/bin/git-svn line 1872

It appears as though the git svn show-ignore command (using the current version of git from MacPorts–version 1.5.4.4) doesn’t work on my Mac. If you see this too don’t worry about it; it will only mean that Git doesn’t know which files you’ve chosen to ignore in your Subversion repo.

Now we’re ready to use our new Git repo. Note that we’ve got a work branch by default:

$ cd myproject
$ git branch
  master
* work

If you want to call the branch something else, set the BRANCH environment variable before you run git-me-up.

Samuel recommends a good work flow for working with Subversion (skip down to “Step Five: Using this thing”) . When you want to update your working branch from subversion, try this:

$ git checkout master  # switches to local master branch
$ git svn rebase       # pulls recent changes from subversion
$ git checkout work    # switches to local work branch
$ git rebase master    # pulls recent changes to master into work

There’s a good description of why rebase is so great on James Bowes’s blog.

When you want to commit changes back into subversion:

$ git checkout work    # you were probably there already
... run your tests ...
$ git commit -a -m "Blah blah blah..."
$ git checkout master
$ git merge work
$ git svn rebase       # checks for fresh changes in subversion
... run tests again if rebase pulled any changes ...
$ git svn dcommit

Make sure that you use git svn rebase to pull upstream changes into your master branch before you check in.

What git-me-up does with svn:externals plugins

The Rails app that I cloned above has a plugin that is installed via svn:externals called site_theme. I usually use piston or braid to install Rails plugins, but when the plugin is stored on my own servers I’ve always preferred to pull in the source directly by setting the svn:externals property.

I initially tried to convert svn:externals into Git submodules, but it didn’t work well. No matter what I tried, I couldn’t persuade git-svn to ignore the submodule when committing back to Subversion. It turns out that it’s easy to achieve with a symbolic link.

git-me-up cloned the site-theme plugin into a Git repository, then made it available with a symbolic link:

$ ls
myproject/  plugins/
$ ls plugins/
site_theme/
$ ls -l myproject/vendor/plugins/ | grep site_theme
lrwxrwxrwx  1 graham development   32 Apr 24 18:41 site_theme -> ../../../plugins/site_theme/

To avoid checking the symlink into Subversion, git-me-up told Git to ignore it:

$ grep site_theme myproject/.git/info/exclude
/vendor/plugins/site_theme

The result is a trouble free alternative to submodules that allows you to integrate Git with svn:externals in your Subversion repo.