.

June 12, 2006

Rails Testing Strategies: Valid and Invalid Data

Consider this test…

  1. def test_create
  2.   @item = Item.new
  3.   assert_valid @item
  4. end

Can you tell what the expected result of this test is?

It turns out that you can’t tell from reading this test if it will pass or fail without knowing if ‘Item.new’ returns a valid or invalid object. If ‘Item’ has a ‘validates_presence_of’ statement in there, this new item will be invalid and the test will fail. In addition, this test is ‘fragile’, subtle changes to the model definition can make the result of this test change. This isn’t such a big deal if all you are doing is testing to see if an object is valid or not, but if you are writing a more involved test that depends on this assumption then you might be in trouble.

Currently the only way to ensure that your test data is valid is to add an assertion to each test making the appropriate test. This isn’t very DRY.

One trick that I use to avoid repeating myself a lot in tests is to define a mock object like this..(putting this in a mock object keeps these methods out of production mode).

  1. # mocks/test/item.rb
  2. require ‘models/item’
  3.  
  4. class Item
  5.   #this assumes that an empty object is invalid
  6.   #modify this as needed
  7.   def self.new_invalid
  8.     Item.new
  9.   end
  10.  
  11.   def self.new_valid
  12.     item = self.new_invalid
  13.     item.name = "new valid item"
  14.     # add any additional statements
  15.     # required to make the item valid
  16.   end
  17. end

Then in my tests I add…

  1. def test_valid
  2.   assert_valid Item.new_valid
  3. end
  4.  
  5. def test_invalid
  6.   assert !Item.new_invalid.valid?, "Item not invalid"
  7. end
  8.  
  9. def test_save_invalid
  10.   @invalid_item = Item.new_invalid
  11.   assert !@invalid_item.save, "Saved invalid item"
  12. end

This way I am guaranteed to get a valid or invalid item when I need one. in addition, if the definition of valid or invalid changes sufficiently the ‘test_valid’ and ‘test_invalid’ tests will fail, warning me that my assumptions in the rest of the tests are incorrect.

One problem with this approach is if your tests care about HOW the object is invalid. In that case, you might need additional methods like ‘new_with_invalid_name’.

* Filed by Kevin Olbrich at 1:36 pm under Ruby on Rails, testing
* 3 Comments

Rails Testing Strategies: Positive and Negative Controls

Testing your rails application can be a great way to make your application robust. Proper testing aids in debugging, and can alert you when some minor change you made breaks something horribly. Rails has some nice testing features ‘baked-in’, but the best advice I have seen so far regarding which actual tests to run is … ‘it depends’.

Needless to say, that isn’t very practical advice.

Over the course of developing my own rails application, I have developed a list of ‘tests’ that I like to run on my models and controllers. This article will discuss some of the strategic design of tests, but won’t cover how to implement them (well, not much anyway).

Positive Controls

Tests should be designed with both ‘positive’ and ‘negative’ controls in mind. A positive control is a test or assertion that succeeds when you expect it to, and likewise a negative control fails when you expect it to. For example, if you create and save an object using predefined good data, an appropriate positive control could be…

  1. def test_create
  2.   num_items = Item.count
  3.   @item = Item.new
  4.   assert @item.save, "Item not saved"
  5.   assert_valid @item  # a positive control
  6.   assert_equal num_items + 1, Item.count  #another positive control
  7. end

In this case, we just check to make sure that the object is valid and that the number of them in the database has increased by one. These types of tests don’t really tell you much. The only time these types of tests fail is when something is seriously wrong with your database server, your ruby install, or your rails install. The only other time this type of test will fail is if you change the definition of the ‘Item’ model so that creating one using the ‘new’ method results in an invalid object (more on that later). The problem with positive controls is that when they pass, all they tell you is that nothing in the test caused an assertion to fail. This could mean that everything is hunky-dory, or it could mean that your assertions are not working as expected.

Passing the test doesn’t mean much if the test itself can’t detect a failure. Our test would pass just fine, even if the object model should really have a ‘validates_presence_of :name’ specified. It’s also easy to see that if we write a test like this…

  1. def test_name
  2.   @object = Object.new(:name=>’test name’)
  3.   @object.save
  4.   assert_valid @object
  5.   assert_equal @object.name, ‘test name’
  6. end

This test will still pass, even if the validation is not specified. Still, it’s nice to know that things work as expected.

Negative Controls

Negative controls are really much more handy. A Negative control is a test that is designed to detect when things fail.
If we write another test for our ‘item’

  1. class Item < ActiveRecord::Base
  2.   validates_presence_of :name
  3. end
  4.  
  5. def test_valid
  6.   @item = Item.new
  7.   @item.name = "test item"
  8.   assert_valid @item
  9. end
  10.  
  11. def test_invalid
  12.   @item = Item.new
  13.   assert !@item.valid?, "Object is not invalid"
  14. end

If ‘test_invalid’ passes, then you can be confident that your ‘validates_presence_of’ is working. If it had been commented out during a bug fix and never replaced, this test would fail, while ‘test_valid’ would still happily report a success.

The same approach should be used when testing model methods. At least one test for each type of expected output, and one for each possible failure mode. Then make sure that the negative controls are in place so that they will pass when the model is fed corrupt data.

The bottom line… tests should be written to pass when you pass valid data, but they should also be written to pass when you pass missing, corrupt, or extra data. The failure modes are the most useful during development since they will point out places where bugs and security holes may creep into your application.

* Filed by Kevin Olbrich at 1:36 pm under Ruby on Rails, testing
* 5 Comments