Why Rails and Their Committers Are Awesome

I love keyword arguments when they were introduced in Ruby 2.0, and use them every chance I get. As I was hacking around with ActiveJob today, and thought it would be cool to pass keyword arguments into the ActiveJob#perform_later method.

I thought it should work. It didn’t.

You get an  ArgumentError: wrong number of arguments (1 for 0) error. The reason is the hash is converted to string keys upon serialization.

So I was thrilled when I did a bit of searching to see if anyone else had the same issue. Nope. Looked at the ActiveJob::Arguments module. Passing keyword arguments are not supported. I actually get to report a real feature request to Rails!

So here is my ActiveJob should support passing of keyword arguments to perform method.

This this is where the magic began… within a few minutes Sean Griffin (@sgrif) picked up on the issue report. I went to lunch, planning to fork ActiveJob and start hacking away when I got back. But Sean beat me to it. Sean added the ability to pass keyword arguments to ActiveJob jobs, and even backported it to Rails 4.2.1.

Within an hour, it was all done. Check out the full commit.

Want a reason to love Rails and their committers? This is one, of many.

Thank you Sean.

Posted in Rails, Ruby. Tagged with , .

Self-joined HABTM with Cache Updates

You have a self-joined HABTM table. It’s probably called User too. All is well, until you realize that when relationships change, your site does not display those changes. The reason is that the related objects (users) are not being updated when the relation is created/removed or one of the objects are updated/destroyed. And the cache keys that rails generate for the parent object are still referencing old data.

Here is how to fix it.

Self-Joined Users

Pretty simple really. There is a table in the database called  user_relationships with two fields:  adult_id ,  child_id . You can create it with this migration:

Now every User can have  .adults and  .children. And those adults and children can have adults and children. And so on. And so on.

User Views

You are using rails fragment caching aren’t you? If not, stop right now and read up on how to implement caching in your rails app because even if you don’t need it now, you most like will need it later. Your users will love if for it!

Watch more: Greg Pollack has an excellent video on Dalli & Cache Digests. He does a much better job explaining caching, cache keys, and fragment caching that I.

Rails uses the concept of a cache_key for each object. Cache keys are simply the model name, record id, and the datetime of the  updated_at attribute for the ActiveRecord object. Check it out:

Where cache keys come into play are in views. When a view generates it’s html, the cache  method will ask your caching system (most likely memcached) if a key exists. If it does, then it pulls that fragment from the cache without the expense of generating the html. If they key doesn’t exists, then it renders the html and then stores the result in the cache. Here is a very simple example.

When rendered, this will display the user’s name, and the names of all their children.

Changing Relationship Doesn’t Change the Rendered View

All is going well, until you start getting reports of problems. A user deletes one of her children, but when she views her show page (see above view), she still see the deleted child’s name. The reason is that when you make a change to a HABTM relationship, it only removes the relationship in the uses_relationships  table. It doesn’t update the  updated_at datetime for the parent record. Thus, the cache key for the parent record does not change, and the old (incorrect) information is read from cache.

How To Fix

I wan’t able to find an definitive example of how to fix this completely and correctly for all cases. So here is my implementation.

What you need to do is “touch” all related objects in a relationship anytime that relationship changes. And those changes could be:

  • Parent (adult) object is updated.
  • Any children objects are updated.
  • Relationships are created.
  • Relationships are deleted.

To fix the first two cases, we just need a couple of callbacks.

Astute coders will notice that we used update_all instead of  touch here. While we could have used touch, there is the possibility that you could create a circular update infinite loop. The touch  method will cause all touched objects to also execute any touches that are defined on their relationships.

In our particular case, we are not concerned about touching any additional relationships here because were are only updated the updated_at  attribute to trigger a cache invalidation. We aren’t actually changing the related objects data like a name or birthday.

Now, we need to fix the last two cases when relationships are created or deleted.

Note that we added  after_add: :touch_updated_at, after_remove: :touch_updated_at  to the HABTM definition. These are callbacks that are triggered upon relationship additions or deletions. They both call the touch_updated_at  method that like above, will update the updated_at  attribute. But in this case, the HABTM will pass a reference to the modified user object.

Here we use the update_column  method because we do not want to trigger any callbacks, validations, or cascading touches.


Now, anytime a user adds a child, deletes a child, or any data on all related children is changed, the updated_at  attribute is set to the current datetime. This will change the calculated cache key for the user objects, and voilà, the view will regenerate the html and users will see the modifications.

Posted in Uncategorized.

Rails HABTM with Unique Scope and Select Columns

This one left me scratching my head for a while. Not quite sure I would call it a bug, but this one definitely has a GOTCHA.

Let say you have a Users model and Units model. Each can have_and_belong_to_many  of each other. But, you want to make sure there are no multiple User->Unit relationships, or Unit->User relationships. So you use the association scope -> { uniq } .

It won’t work. Here is why.


So we have these models:

And the relationship table:

You can read more about ActiveRecord associations and queries at Rails Guides.


When you query a Unit for Users, you get only DISTINCT users, and vice-versa.

But All is Not Well

Let’s check out what happens if we use a .select(:first_name)  with the Unit.users query.

Whoa there… our count of users went from 130 (the correct number) to just 108. Why? Take a look at the SQL with the .select(:first_name)  query. Specifically the  SELECT DISTINCT COUNT(DISTINCT "users"."first_name") . The query that ActiveRecord is generating is doing a DISTINCT (alias for uniq) on the "users"."first_name" . This query will only find users with DISTINCT first names. What if you have 3 people with the first name of “John”. Well, it will only find and return 1 of those John’s.

This is not what I expected. POLA any one?


Rails Guides has a good section on Selecting Specific Fields that goes into detail about using .select()  with .unique . But it doesn’t mention that using the -> { uniq }  scope on an ActiveRecord association will have the same affect. Well, it does.

So, it got me to thinking… could I add -> { distinct(:id) }  to the HABTM declaration? Yes you can. But it doesn’t work with .select() . For some reason when you use .select()  ActiveRecord will override any column declarations you used with the distinct() .

Proof eh?

I find it interesting that the first query, without specifying a .select()  generates the correct DISTINCT query with the "users"."id"  column. But as you can see on the second query, it is overriding the specified scope with the .select()  columns.

This is my logic: if I use .select()  that doesn’t necessarily mean I want DISTINCT columns, regardless if weather I specify a .distinct()  or not. In the above case, .select()  always includes the :id column wether or not you are also using .distinct() . So why would the .select()  override my .distinct(:id)  declaration?


First solution is to not use .select()  with a query. Duh.

But I really like .select() . It’s a great way to reduce memory size when instantiating hundreds or thousands of User ActiveRecord objects. And in my real-world use case, the User table has a couple of text columns that contain large amounts of html. So I like to use .select()  where ever I can.

So I guess there really isn’t a solution that I know of, except not to use .select()  with ActiveRecord queries on HABTM tables that use the -> { uniq }  scope.


Best Solution!

You knew it was coming, didn’t cha. I wouldn’t leave you hanging.

The best solution is to not use the -> { uniq }  scope with HABTM relations, but enforce the uniqueness condition at the database level with unique compound indexes.

In your migration for the relationship table just add unique: true  to the index declaration.


Posted in Rails.

Model Doesn’t Save After Adding before_save Callback

File this one under the Duh folder… I added a new before_save callback yesterday, and suddenly nothing in the model would save. Here is the offending code:

Spot the problem? I didn’t. At least for a while.

In a callback, if you return  false the save is cancelled. I knew that. So, when setting the sms_number_verified  attribute to false , it also returns false  from the callback, and therefore the model does not save. I did not expect that. Seems evident now that you see it.

Here is the fix:

Just return true from the callback.

Would have been nice if rails notified me in the logs with something like “save cancelled, before_save callback returned false on line XXX”.

Posted in Rails.

SQLite3::ConstraintException: table.created_at may not be NULL

So, I’m working along in Rails today, added a new migration, when all of a sudden tests that were passing only a few minutes ago suddenly start failing. The error I in rspec is something along the lines of:

In test.log:

WTF? This was working only a few minutes ago.

If you google this, you will read endless others who experience the same error… when using HABTM relations. Well, EventSignup has no HABTM!

This has happened to me a couple of times, and each time it leaves me puzzled for far too many minutes, and there is no real solution.

Okay, well there is only one solution I have found to work: delete the SQLite database file and recreate. Yeah, that sux. Oddly enough, just a quick:

And all is good again.

If anyone has an idea why this happens, I would love to hear why.

Posted in Rails. Tagged with , .

Rails Nested Attributes with Scoped Uniqueness Validation of Association

This is going to be a long one, so grab your favorite beverage…


I have a fairly standard belongs_to  and has_many model association. And I want to save nested attributes along with the parent when creating a new parent record. But, I don’t want the user to be able to submit duplicate values for the child records.

Sounds easy. That’s what I thought when I started.

My Solution


We have an Organization and a SubUnit models, pretty standard stuff here.


The controller is simple (standard actions removed). Keep scrolling down.


Now it’s getting interesting.

Your form should look like this.


Polluting Links with Javascript

Like everyone else who has started down the road to nested attribute bliss, you have watched Ryan Bates’ Railscast on Complex Forms. It’s superlative.

But there’s one thing I didn’t like… the heavy handed approach of using link_to_function. Especially when including a fair amount of escaped html into an onclick script. I’m not saying it’s wrong, just that I believe html should stay where it belongs… in the DOM. Heck, they even created a gem nested_form.

I think we can make it simpler, painless, elementary.

That’s where the following little ditty comes in handy. We are creating the exact same field structure that is used for entering a SubUnit, but hiding the html so that we can use JavaScript to copy-and-append to the visible entry area.

Line #1: hide the div.

Line #2: use fields_for to generate the proper field id and name attributes so that it is submitted as a child object to our Organization. Notice the SubUnit.new? That prevents this block from iterating over all the @organization.sub_units that may be built, either as a new object (controller line #5), or when re-rendered due to a validation error with @organization.

Line #2: don’t forget the child_index: ‘new_sub_unit’ option! Without it, rails will attempt to index the fields id and name attributes with numbers, which in this case would be 0 (zero), and mess up the prior sub_unit fields. IOW, you will end up with two params[:organization][:sub_units_attributes][0] elements.

If you don’t understand all the details, take a few minutes and read the ActiveRecord::NestedAttributes::ClassMethods.

Now for some JS magic:

Pretty simple, bind to the click events of the add-sub-unit-button and remove-sub-unit-button to functions.

add_sub_unit function copies the new-sub-unit-fields div, replacing the text of ‘new-sub-unit’ in the id and name attributes of the input/label elements with an integer.

remove_sub_unit function hides the input field group, and set the ‘_destroy’ hidden field to “1” (‘true’ also works) so that it will not be created when the form is submitted.

Inspiration for this JavaScript goes to Ryan Bates.

Status Check

At this point we have a parent model, Organizations, with nested attributes for children SubUnits. When creating a new Organization, you can add and remove SubUnits. The entire relationship is submitted with a single form post.

Eliminate Duplicates in the DOM

Everyone else does it in model validations, manipulation of the params hash, or some other convoluted way. But I decided to eliminate duplicates before the submission of the parent form. Pretty easy, actually.

When the Create Organization button is clicked, the above JavaScript loops through all the SubUnit form field elements, marking duplicate entries hidden ‘_destroy’ field with ‘true’, thus preventing them from being saved. Could you just remove the elements? Sure.

Inspiration for the JavaScript remove duplicates in an array to Roman Bataev in this post.

Screen Shot 2012-12-21 at 4.38.21 PM

If you watch your logs, you’ll see that all the SubUnits attributes are submitted, but only those marked with the hidden field ‘_destroy’ == ‘false’ will be created.

The astute reader will also notice that the SubUnit model is attempting to validate the uniqueness of the name attribute, scoping it to a non-existant Organization. I admit it would be best to skip this validation, but I will leave that exercise to you, humble reader.

Sample App

What? You wanted a demo app on the GitHubs? You shall be rewarded, post haste.


The Good

  • Submission of a parent Organization with children SubUnits utilizing nested attributes.
  • Duplicate SubUnits are removed.
  • Darn simple, easy to reproduce. Pretty darn unobtrusive, too.
  • No funky helpers to jack some html into a link element.
  • Is it better than Ryan Bates’ solution? No. Just simpler.

The Bad

  • Only works on creating a new Organization. Does not work for editing an Organization with the ability to add additional SubUnits or destroy existing SubUnits. Future post?
  • My JavaScript is not abstracted to make it easily reusable. Fork it, and submit a patch if you so desire.
  • The SubUnit validation should be skipped when creating through an Organization.
  • No fallback if JavaScript is disabled.

Additional Reading


Posted in Rails. Tagged with , .

Small Revelation – FactoryGirl, build_stubbed, associations, and let

Not really a problem, but something I didn’t really look out for because all my tests were passing… and hey, if tests are passing, I sleep well.

TL;DR Dumped let{}, removed FactoryGirl associations, back to before(:all) and instance variable/FG.build pattern = super speedy tests. 
TL;DR.1 And read the docs.


I need to test a fairly complex permission matrix at the model level. To properly test it, I need to build out several complete company structures, with a multitude of users at various authorization levels. And 200+ actual tests.

Being a fan of rspec’s let{}, I decide to use it copiously. 40 lines worth.

In these tests, I don’t really need to have persisted records, so using FactoryGirl.build_stubbed fits the bill here.

let(:company) { FactoryGirl.build_stubbed(:company)}
… an so on, 39 times. 

What I Found

I thought one particular spec was running a little slow. So I converted the FactoryGirl.build to the shiny new FactoryGirl.build_stubbed. No difference in time. So I decided to watch the test.log. Well, I quickly saw what was happening… it was persisting thousands of records.

But wait. I was using FactoryGirl.build_stubbed. Where were all these records coming from?

Problem #1

In most of my FactoryGirl factories, I was using ‘association :area‘, ‘association :store‘, and so on. Didn’t think much about these until yesterday.

Turns out that FactoryGirl will build/create and save those associated factories, even if you are using FactoryGirl.build or FactoryGirl.build_stubbed. Learned something new there. Honestly, I didn’t expect this behavior, but I understand why. Shoulda read the docs.

Now, the easy way around this is to pass into a FactoryGirl.create/build/build_stubbed a nil for the association, if it is not needed. Ala:

FactoryGirl.build_stubbed(:store, company: fg_company, area: nil)

Now it won’t build out the associated area. Alas, I had forgotten just one of these nil associations. And at the worst possible relationship level, the bottom. So every time one factory was built, it create the entire supporting structure above. Thus, every hit to an let(:invoice) builds an entire company structure from one single FactoryGirl.build_stubbed(:invoice) call.

But it get’s worse.

Problem #2

I love the let{}. But to be honest, I never read the docs on it. Well, I did read them yesterday. Relavent line being (emphasis added):

The value will be cached across multiple calls in the same example but not across examples.

Uh-oh. Let{} is like a before(:each). Which is what most specs need. But I don’t, not for this spec. I’m never modifying the AR records, just testing some methods within models, which don’t modify AR instances.

Resulting Big Problem

Ok, not really a problem. But certainly very, very inefficient.

By forgetting to nil an association in a FactoryGirld.build_stubbed, and with let{} recreating an entire company structure, to the database, for every 200+ test. Well, you get the picture. It’s slooooow. 22 seconds worth of slow.


You know I wouldn’t drag you along this far without a solution.

  1. Just remove all the ‘association :model‘ statements from all FactoryGirl definitions. I know they are handy, but I want CONTROL over my factories. And just one small mistake can make a spec run many X-times longer.
  2. Remove the let{} and replace with the good ol’ instance variable/build pattern.
  3. Move all the instance variables into a before(:all).
before(:all) do
  @company = FactoryGirl.build_stubbed(:company)
  @store = FactoryGirl.build_stubbed(:store, company: company)
  … and so on, 38 times.

Note for step #1. It caused me to refactor some other specs as well. This turned out to be a good thing, as I was able to speed up several other specs, and add some clarity to those specs that required building out a company structure.


2.5 seconds. Not too shabby.

After the refactoring for all tests, I dropped ~30 more seconds off the entire suite.


Hope this helps someone else out there improve the speed of their specs too.

Posted in Rails, Ruby. Tagged with , , .

Monitor Your Elk-M1, Log to MySQL, and Daemonize it!

I received a request a couple of days ago from someone who saw my posts on the Elk–Control perl library that I took over responsibility for. I had mentioned that I logged every Elk event into a MySQL database.

I am happy to report that I have this script running as a daemon, logging every ElkM1 event into my database. And it runs for months without issue.

I will keep these instructions brief. If you have detailed questions, please ask in the comments and I’ll help the best I can.


You can download all the scirpts, just click here.


  1. If you don’t know a least a little perl and some MySQL basics, you may want to look elsewhere.
  2. The script and accompanying files are designed to work in concert as a daemonized process.
  3. Everything is logged into ‘elk_monitor.log’ in the same directory (or directory of your choosing) as the script. If you have a lot of passive infra-red motion detectors, you will want to monitor the log size.

Basic File Structure

file structure


This is the main script, it runs in a loop checking for new messages every second. You will need edit the %zonelist, %outputs, %areas near the top. Also, you will need to set some paths, look near the top of the file.

Required: ElkM1::Control, Log4perl


This is the shell script that controls elk_monitor.pl as a daemon. Setting up daemons can be complex as each system is different, so google around for some help.


A MySQL script to setup the tables and fields that the elk_monitor.pl script uses. You will need to add a database named ‘elkm1′. After that you can execute this script to create everything else for you.


The database setting and basic sql commands are abstracted into this library. This is where you will need to set MySQL’s host, name, password, etc.


I use a custom ruby script to monitor the daemon. This library just touches a file every minute to let us know the elk_monitor.pl script is still running. Why? Well, I have had the script ‘freeze’ without exiting and most monitoring applications do not detect this. Obviously, this part is optional.


If you get all the settings right, she should run all day/week/month for you.

I and not a perl expert. Far from it. So I’m sure everything could use some tidying. If you do, please let me know and I’ll incorporate your changes.

Posted in Home Automation, perl. Tagged with , , .

Sinatra-Tailer: a small app for viewing server log files

UPDATE (Jan 9, 2015): I have not updated this project in year. While it may work, I doubt it. If you would like to take over the project, leave me a comment and we’ll get in touch.

I was reading Jason Seifer’s: Offline Gem Server Rdocs, which is an apache/passenger served Sinatra app that allows you to view the Rdocs of installed gems without using gem server. Nice. So I installed it on our sandbox server for all to enjoy.

But it got me thinking, there is another think I like to keep an eye on on our servers… log files. Oh, and I was looking for good excuse to play around with Sinatra. So, “with a one, and-a-two, and-a-three…” we have Sinatra-Tailer.

You can read all about it on the github page, but in short it simply performs a tail and displays the last X lines of the log file.

[sourcecode language=’bash’]
tail -n /path/to/my/log/file.log


  1. refreshes the list every X seconds, set by the user
  2. only one config.yml file to edit
  3. supports file globs, so you can grab a whole list of file with one line
  4. specify the number of lines to show
  5. completely unobtrusive js, because I’m cool?


Sinatra framework

      (tested on, earlier may work)

I whipped this up pretty quickly so I’m sure there are a few bugs. There is some testing for a few unit tests, but nothing functional.

One word of warning, if you want to put this on a production server my recommendation is to put it on a separate port (like 9876) and for heaven’s sake, at a minimum use http basic authentication. From the sinatra readme:


Posted in Ruby. Tagged with , , , .

Apache Virtual Hosts, Rails, Rack, Passenger on Local Net

This one has been bugging me for a couple of years and I just didn’t want to put in the time and testing to nail it down. But tonight was my night.

The Problem

Here in my development palace I have a couple of laptops that I use for development and an Ubuntu (hardy) server as a Rails/Rack sandbox that I run with Passenger. While I have always been able to serve up either a Rails or Sinatra project, I typically have 3 or 4 going at the same time. But I had to access them with urls like: and

So, what’s the problem, eh?

Well, for starters, you can’t do anything concerning subdomains. That’s a show stopper for many apps.

Second, it really messes with your environment variables like [“PATH_INFO”]. So if you do any routing and you expect the base url to be ‘/’, it won’t be. The root url will be ‘/projectx’. Bummer.

The Solution

To remedy the issues with non-tld virtual hosts (also called sub URIs?) we need to add the domains to the laptops and server, then add the virtualhosts on the server for apache. This means that I can now goto http://project1.ubuntu.remote url to get to my rails/rack app.

For reference, I am using the following:

On Each Laptop

These instructions are for Mac OS X, but should be identical for your favorite flavor of *nix. We are going to add some friendly names to our ubuntu server using it’s ip address ( In this example we will be able to access the root web documents using http://ubuntu.remote, and one of our Sinatra projects using http://project1.ubuntu.remote.

Important! Don’t use a name ending in ‘.local’. While it did work for me, I read about a few instances where some systems will not resolve domain names ending in .local outside the local machine. Also, don’t change anything else in your /etc/hosts file except to add the lines at the end.

[source language=’python’]
mate /etc/hosts

# add the following line to the END of your /etc/hosts file and save
# note: there is a character between the ip address and each host name ubuntu.remote project1.ubuntu.remote

On The Server: Hosts File

Nothing will work until we tell the server machine that it can accept incoming traffic from our new friendly names. Similar to on the mac, we need to edit our /etc/hosts file but we will add a slightly different line. In this case we are telling the server that incoming traffic from ‘ubuntu.remote’ and ‘gems.ubuntu.remote’ are just names for the localhost (

[source lang=’shell’]sudo pico /etc/hosts

# add the following line to the end of our /etc/hosts file and save
# you can added after your existing line if you wish ubuntu.remote project1.ubuntu.remote

On The Server: VirtualHosts

As a final step we will need to add or change the virtual host definitions for your rails/rack project. Depending on your Apache setup or Linux distro, the default enabled site may be slightly different and you may have to adapt your virtual host file to your default setup.

First, in my /etc/apache2/apache.conf file, I have the following line near the end of the file. This was to eliminate a warning every time apache was started/restarted. It may affect your virtual host default, but shouldn’t, so add it if you get errors.

[source lang=’shell’]
#added near end of /etc/apache2/apache.conf
ServerName localhost

Next we need to check our default virtual host setup to see how the NameVirtualHost is defined. We will duplicate that in each new virtual host that we add. The parts we are looking is the NameVirtualHost and VirtualHost near the top. They could be ‘NameVirtualHost *’ and ‘<VirtualHost *>’ or ‘NameVirtualHost *:80′ and ‘<VirtualHost *:80>’. See the pattern? We need to use the same definition in our new virtual files.

Let’s check our default virtual host setup:

[source lang=’shell’]
cd /etc/apache2/sites-enabled
ls # your are looking for the name of your default site file

total 0

lrwxrwxrwx 1 root root 36 2009-02-27 18:49 000-default -> /etc/apache2/sites-available/default

cat /etc/apache2/sites-enabled/000-default

NameVirtualHost *

  ServerAdmin webmaster@localhost
# snipped several lines #


On my system the NameVirtualHost is ‘*’, so that is what we need to use for the new virtual host. In this case, I am going to add a virtual host file for a rack based project. It’s a Sinatra app, but that doesn’t matter as this will work for any rack based project, even a rails/rack project (since rails 2.2.2?).

Also, I like using the Apache2 system to enable and disable sites. It’s clean, easy to use, and if any apache admin comes along they will know exactly what is going on.

It’s also work mentioning that you should disable or delete any existing virtual host files that you were using previously.

[source lang=’shell’]cd /etc/apache2/sites-available
sudo pico gems

# enter this as your virtual host setup

ServerName project1.ubuntu.remote

DocumentRoot “/path/to/your/application/project/public”

RackEnv production

Order allow,deny
Allow from all

# save the file and reload apache

sudo /etc/init.d/apache2 reload

The Fun Part

Assuming you saw no errors when you reloaded apache, you should be able to go to your new urls see both the root files and your rack application.




Bonus Points!

For extra credit, let’s say you have admin access to your router, and you are one of those gotta-have-control-of-everything dudes (or dudettes) so you installed a 3rd party firmware. You’re in luck! You can probably add some custom entries in your DNS and you don’t have to edit the /etc/hosts files on your laptop.

I installed Tomato firmware on my Linksys wireless router. Tomato uses Dnsmasq with allows you to set internal host names.

Maybe I’ll cover that in another post.

Posted in apache, Rails, Ruby. Tagged with , , , , .