Friday, September 5, 2014

Ruby unit testing is weak as a safety net

Ruby unit testing feels very nice and natural to write with RSpec. Since I started working with Ruby more often that is one of the things I liked the most. However I still love Java (or for this particular example anything with strong typing) and is still my main language, and in many cases is superior to Ruby.

One of those is the value of Unit Tests over the long run and as safety net and aid in refactoring.

Let's see this example in Ruby first:

class Collaborator
  def do_stuff

  end
end

class Unit
  def initialize(collaborator)
    @collaborator = collaborator
  end

  def call_collaborator
    @collaborator.do_stuff
  end
end

describe Unit do
  let(:collaborator) { Collaborator.new }
  subject { Unit.new(collaborator) }

  it 'should call collaborator' do
    expect(collaborator).to receive(:do_stuff)
    subject.call_collaborator
  end
end

That seems like a very simple Unit test. Is making sure that my class under test makes a necesary method call to a collaborator.

I run the test and it passes as expected:

 $ bundle exec rspec
.

Finished in 0.00051 seconds
1 example, 0 failures

Now, sometime later I want to change my collaborator. The collaborator would have its own unit test that I would need to change first of course. However I won't look all over my code base where this method is used (already very difficult to do in Ruby). Then I make my collaborator look like this:

class Collaborator
  def do_the_important_stuff

  end
end

If I rerun my test for Unit this is what I get:

.

Finished in 0.00051 seconds
1 example, 0 failures

Yes, the test still pass even if the method that is supposed to be called in the collaborator doesn't exist anymore!. This would for sure break in production when this branch of code is reached. This can (and need) to be mitigated with more integration tests and not trust so much on the unit tests, but still it leaves the original useless unit test in place.

With Java it is a completely different story and the Unit tests can be trusted a lot more. Here is the same example:

interface Collaborator {
  void doStuff();
}

 public class Unit {
   private Collaborator collaborator;

   public Unit(Collaborator collaborator) {
    this.collaborator = collaborator;
  }

  public void callCollaborator() {
    this.collaborator.doStuff();
  }
}

public class TestEr {

  private Collaborator collaborator = Mockito.mock(Collaborator.class);
  private Unit unit = new Unit(collaborator);

  @Test
  public void shouldCallCollaborator() {
    unit.callCollaborator();
    Mockito.verify(collaborator).doStuff();
  }
}

This test passes as well. But now if go about and change the Collaborator to be

interface Collaborator {
  void doSomeStuff();
}

We will automatically get a compilation error in all places that are using this method. Including our unit test. This will drag us direcly to the error to fix the dependency properly before deploying anything. In this case the unit test has proven to be really valuable as a safety net to capture a problem introduced by a seemingly simple refactor.

Of course the example is super simple, but you can imagine in a big scale application the Ruby problem is not so unrealistic.

8 comments:

Saša Ranisavljević said...

describe Collaborator do
subject { Collaborator.new }

it {should respond_to(:do_stuff)}
end

As Ruby is dynamic language, this is way to test for object API (fields/methods). Yeah, this is thing that you do not need to do in languages with static typing, but every language type has it's pros and cons.

Saying that Java is superior than Ruby is strong statement :)

Cheers,
Sasa

Carlo Scarioni said...

Hi Sasa, thanks for writing.

I don't think that in general Java is stronger, but I do think it definitely have stronger points (in this particular case static typing vs dynamic). As Ruby also has stronger points.

Even in your example, that is the only test that would fail. The original test that used the collaborator as a client would still be passing. Meaning, you go and fix this new test you added, and then you are back in the original problem.

Saša Ranisavljević said...

Hey Carlo,

sure, you can go and change failing example, but for me failing of this example means: class api is changed, find all usages of old method and change them. In my head this is equal to IntelliJ/Eclipse red underline notifications that method/field in object/class is missing. Sure, it is more convenient when IDE is doing this for you, but as I said, this is the way to "protect" in dynamic languages.

Cheers,
Sale

Carlo Scarioni said...

Hey Sale, Sasa?. you sign differently in your both comments :)

Yeah, that is the point. it is definitely more difficult to be confident and make sure that the tests will simply "catch these bugs while refactoring" the way they do in statically typed. And is not about the IDE, it is about not letting you deploy it broken. The application will fail to compile and you won't be able to deploy it until you fix the API clients everywhere. You are forced to do it by the compiler.
And as you mention the "way to protect" does feel hacky in the Ruby example. Expecting a class to respond to a method, kind of defeats what duck typing is about, and then having to remember to manually check all your project because you are changing a method in a class definitely feels weaker than "change and let the compiler tell you".
But as I mentioned before, I do love Ruby and programming in it, but in bigger projects these things start to make changes more difficult to achieve.

Cheers,
Carlo

Saša Ranisavljević said...

Hey, Sale is nickname for Sasa in Serbia :)

Yes, I agree that this is kinda "hacky" in dynamic languages, especially when you come from static background :)

Cheers,
Sale

AkitaOnRails said...

Wow, seriously? This title and this kind of content is what's defined as 'link bait'.

If this is not the case, this example is checking a type's interface. This shouldn't be tested to begin with. And of course, Java being statically typed, it's utterly redundant as the compiler tests that already.

Unit tests are never about testing interfaces (unless, of course, you're making a lint test for some web service API, and even then).

This is so wrong, it's difficult even to start a discussion on the subject.

Sorry for being harsh, but when I see provocative titles like this, I do expect a lot more meat.

Carlo Scarioni said...

Hey Akita, didn't mean to make the title provocative at all. I get nothing for page clicks. Just to show an actual issue I encounter in bigger Ruby projects. So apologies if it seems intentionally provocative.

The example is deliberately simple, but for a real life problem. But you obviously don't want to discuss it which is fine of course. Thanks for your input anyway.

Cheers

Saša Ranisavljević said...

Hey Akita, I have question regarding this:

"Unit tests are never about testing interfaces (unless, of course, you're making a lint test for some web service API, and even then)."

So, how do you take care of NoMethodError exceptions (given that you changed class api)? By doing integration tests or what? Could you point to some resources?

Cheers and thanks,
Sasa