Auditing Rails Projects

Posted by Venerable High Pope Swanage I, Cogent Animal of Our Lady of Discord 20 December 2007 at 10:36AM

Or how I learned to stop worrying and love SVN, Cerberus, and RCov

I recently was given carte blanche to enact some authoritarian controls on our source code at work, and ran with it. My general feeling is not to limit what people can do with our repository, just to audit that they have done it, and when it is problematic, make their work widely known rapidly. Thus, my intent was not to lockout checkins which would cause rake to fail, just to report on the checkins when they happened.

We had at a prior point in time incorporate the Cerberus system with a cron job on our source control server, every 10 minutes it would checkout the revision and then bitch about whoever had the last checkin if the build was broken to a mailing list all developers were signed up to. This was okay except:

  • If two people comitted in that 10 minute period, the last got the blame.
  • If you made huge structural changes to the project, it would cause the build to fail.

So our cerberus system got switched off and we quickly stopped having a reliable rake build. People broke things and didn't know they broke them, and we were slowly marching down the path of madness.

I decided that I wanted us to have a different system, one that:

  1. Would run the build process on every checkin.

  2. Could cope with large structural changes to the source tree.
  3. Would not violate our prior decision not to have config files checked into the source tree.
  4. Would generate code coverage reports.

It was more work than I anticipated!

I started off by cloning our repository to a sandbox that I could muddy up with many pointless commits and broken hooks without disturbing other people's work. After that I set off to tackling the default config problem. We do not have a database.yml in our source trees for our rails projects, instead having a database.yml.example that is instructive of how your config should look. Additionally we use a config.yml to specify startup options for the server, but again, it is not in the default tree. Instead we use a config.yml.example that developers rename when they check out the tree.

To solve this first issue I created a config/defaults directory, which would have sensible defaults pointing at the database instances running on the build machine. In addition I made a rake task to copy the files from the config/defaults directory to the config directory, and placed this .rake file in the lib/tasks directory of our rails project.

Now, when cerberus checked out our code from the repository, I could specify that it should run the "copy default config files" target before the "test application" target and there'd be sensible defaults in place.

Next I set up cerberus for this repository. The cerberus setup was mostly straightforward according to cerberus' documentation. The only interesting work I did on it comes later.

In my testing I set up a SVN hook to run cerberus on checkin, by adding the line:

sudo -u cerberus cerberus buildall

to hooks/post_commit

This had the unfortunate consequence of making it so that the comitting developer was force fed all of the output of running cerberus. It also meant that the commit process had to wait around for the whole cerberus build to complete! This was going to slow down people (and arguably, commits shouldn't be rushed), but sucked either way. I tried just redirecting stdout to /dev/null but svn holds onto the error output pipe of child processes spawned by hooks as well. Ultimately to fix the problem, I ended up having to change the post-commit to read as follows:

sudo -u cerberus cerberus buildall >>dev/null 2>&1 &

This worked great! Next I wanted RCov to fire on checkin and successful build. Cerberus allows you to define hooks in the config.yml for specific projects and the global config.yml that fire at certain lifecycles in the cerberus execution cycle. They run just like they're invoked from a shell, with the working directory being the root of the local sandbox of whatever you have checked out from subversion. Between adding the hook, and making the config file copy fire, my ~/.cerberus/config/<project>.yml ended up with the following additions:

    task: config:default_files test
    on_event: successful
    action: 'rake test:test:rcov'

Testing with a few checkins, cerberus was building properly and code coverage reports were getting fired! Success!

Moving it over to the real repository snagged a few problems on the way though. They were all solvable.

First, our repository is hosted "officially" over https. I had done my testing over svn+ssh, since that does not require bumping the http daemon, and the test sandbox was going to go away, so there was no point in polluting our apache config with some temporary information. Of course, this meant I wasn't testing the path of coming in from https. Since I was authenticated to the system as as user with sudo permissions, everything worked great. When coming in as wwwrun, pam auth was puking all over my attempts to sudo. This was not going to be acceptable, further wwwrun lacked some of the environment information stored for my cerberus user.

First I needed to make a constrained way for wwwrun to trigger the cerberus build. This was solved with a quick script in my cerberus user's home directory. This would execute the cerberus buildall command and only the cerberus buildall command with the correct home directory (We are running on an X-Serve).

Finally we granted wwwrun the permission to run this script and ONLY this script without authentication via the /etc/sudoers file. I fired up visudo and added the following line:

wwwrun ALL = (cerberus) NOPASSWD: /Users/cerberus/cerb-cerb.rb

The post-commit hook needed tweaking, becoming:

sudo -u cerberus /Users/cerberus/cerb-cerb.rb >>dev/null 2>&1 &

Also, I wanted the code coverage reports viewable by all developers. This was done by tweaking the hook in my cerberus config file to the following:

    on_event: successful
    action: 'rake test:test:rcov && cp coverage/test/* /srv/www/htdocs/rcov/<project>/'

Now we copied the code coverage reports to a publicly accessible directory.

Finally the problem of a re-arranged source tree reared its head again. This time I was going to solve it by starting from a blank slate on every build. The means to do this was, by my judgement, best served by having the sandbox deleted on every successful build. On broken builds it would be possible to go into the sandbox and see what was breaking if it was a problem with the build system and not with the commit. Thus we had one final addition to the cerberus hook:

    on_event: successful
    action: 'rake test:test:rcov && cp coverage/test/* /srv/www/htdocs/rcov/<project>/ && rm -rf /Users/cerberus/.cerberus/work/<project>/sources'

This was all we needed, and now we had builds and code coverage running on every checkin.