Re: [boost] What Should we do About Boost.Test?

Subject: Re: [boost] What Should we do About Boost.Test?
From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2012-10-04 04:16:08

> Mathieu Champlon <m.champlon <at> free.fr> writes:

Hi Mathieu,

This is a big subject. And a riddles with a lot of confusion IMO.

> I'm a bit puzzled by the kitchen_robot example (and not just because it 
> grill chicken without any chicken ! :p).

OK. Let me first reply to the comments specific to my example and later on we 
tackle the problem in general.
 
> MockMicrowave::get_max_power looks hard-coded to return a value of 1000, 
> so to me this looks like a stub rather than a mock object, or am I 
> missing something ?

1000 is indeed hardcoded in this particular mock example, but note that it is 
never tested against, so we are not testing the state. In a sense it is part of 
mock behavior that this particular value is returned. We could have written mock 
differently, where this value is initialized in constructor (or multiple 
possible values). In many scenarios such mocks with hardcoded values will 
properly represent reality and satisty our needs from interaction testing 
standpoint.

> How would you test the robot calls set_power_level properly in respect 
> to different max power values ? 

We can actually implement the check in this method and throw exception if it 
fails. I believe the framework may not log exceptions properly yet. It is 
something to be improved.

> What if the max power could change at 
> any time and you would like to 'program' the mock object to test the 
> robot (first time return 1000, second time return 2000, etc..) ? And 
> what now about a malfunctioning oven which would throw exceptions ?
> Would you write a new MockMicrowave implementation for every test case ?

1. I can indeed write a separate subclass with varying behavior in some methods. 
If this mock is end up being used in 10 different test scenarios it is much 
better than repeating mock setup in each test case with all this configuration

2. We can write mock class in a such a way that it can be configured through 
some mock specific interface to tailor to specific behavior you like. This is 
most flexible and will allow you reuse the same class and the same time 
implement arbitrary change in behavior no framework will ever be able provide 
for you.

3. Finally you possibly can implement some support in mock library to implement 
support to specifying some subset of possible behaviors through some template 
magic and compiler specific hacks. I am somewhat doubtful one can provide a nice 
generic interface for this. More over I an mot convince it is worth the effort.

Most importantly though either of these approaches does NOT imply specifying 
your expectations in a test case. It is all about specifying mock behavior. So 
either way it can be used within a bound of my approach.

> Then there seem to be some kind of trace logging involved which if I 
> understand correctly can be seen as describing the expectations a 
> posteriori.

Yes. Indeed.

>  From my understanding this logs on the first run then reload the 
> expectations and use them as a base to validate new runs.

Yes. Basic record/reply approach.

> I see a number of problems with this (despite the serialization 
> requirement on arguments to be checked), the major one being to not 
> allow TDD.

Why is this?

> Also another common usage of mock objects in test is to document how the 
> object under test reacts to the outside world. Moving the expectations 
> outside the test makes this difficult (although not impossible, I 

Actually IMO it is the other way around. Having these expectations in a plain 
text log file and not in a source code does not decrease a test value from any 
prospective and:
  * the actual source code does not interfere with log, so you can read it 
easier
  * there is no way "unaware" person can peek into your test case and understand 
what exactly interaction expectations are. Looking at log file it is much 
easier. Look at any example of these test cases and tell me this is not the 
case.

> suppose the test cases and expectations log files could be 
> post-processed to produce sequence diagrams or something similar).

Yes. Indeed. And given fixed format of these you can generate proper 
documentation out of them or visa verse you can generate these "pattern" files 
on your requirement phase by people who are not developers at all. It is common 
to describe interaction expectation in requirements (not so much with specific 
behavior of your class, which you test using state expectations).  
 
> Actually we (my team and company) have attempted this approach in the 
> past. It works nicely on small use cases, but quickly tends to get in 
> the way of refactoring. When each code change fails a dozen test cases 
> which then have to be manually checked only to discover that a small 
> variation to the algorithm still produced a perfectly valid expected 
> output but with a slightly different resolution path, it tends to be 
> very counter-productive.

This only indicates that you maybe testing wrong thing. With interaction based 
testing, while it is useful in some scenarios, it is also very easy to fall into 
a trap of testing implementation. This is the nature of the whole process. 
Either the algorithms above are better of being tested by checking a state of 
produced values or it is possible that your tests where valid, but how can you 
be sure algorithms changes are valid? At the very least your test cases notified 
you about these changes. All you need to do now (at least in my approach) is to 
go through new logs and "accept" them as new pattern files. This is also very 
common and valid approach to interaction based testing.

> Therefore we started to add ways to relax the expectations in order to 
> minimize the false positive test failures : the number of times and the 
> order in which expectations happen, that some arguments sometimes are to 
> be verified and sometimes not, etc..

And more complicated behavior variations you accept the less valuable your test 
becomes. Somehow you do not write your test cases like this: ok, this function 
sometimes returns 5, sometimes 6 and some rare cases 10. You can do this, but 
this is bad test. The same applies to interaction based testing. You are much 
better of with specific interaction being expected given the same input.

> In the end spelling out the expectations started to look more like a 
> solution and less like a problem.

IMO spelling out non trivial expectation of interactions indicates tool misuse. 
Some behavior configuration of the mocks is fine. Non deterministic behavior 
expectations are not.

> Do you have any experience on this matter ? Did you manage to overcome 
> this issue ?

Well I can go as far as admitting that there is a potential for improvement 
where you can implement some relaxed rules for matching pattern files, but 
misuse of these will make your test case useless.

As I said above, I still believe record/replay approach is more preferable to 
any other one and I do not see a big issues with it as is.

Regards,
Gennadiy