Re: [Boost-users] Library Interface Design

From: Jeff Garland (jeff_at_[hidden])
Date: 2006-09-14 13:57:36

Scott Meyers wrote:
> Jeff Garland wrote:
>> Testing in total isolation is a myth.  To be a little softer -- it's really 
>> going to depend on the type of class you are testing whether or not it can be 
>> rationally tested in isolation.  If you haven't lately, you should re-read 
>> Lakos's treatment of this subject in Large Scale C++ Software Design. This 
>> book is 10 years old, but he breaks down testability in a way I've not seen 
>> anyone else do since doing testing became all the rage.  Most of the 'test 
>> first' stuff I've seem ignores the inherent untestability of some software.
> 
> That's been my impression.  One of the things I've been trying to figure 
> out wrt the whole testing hoopla is how well it translates to large 
> projects and how it has to be adjusted when things move beyond toy 
> examples.  And yes, I probably should go back and reread Lakos.

Well, the testing hoopla 'applies' to the extent that in my experience big 
systems that *don't* have significant testing discipline never see the light 
of day.  That is, they fail under an avalanche of integration and basic 
execution problems before ever being fielded.  As an aside, I always get a 
good laugh out of all the agonizing by various folks over how this and that 
testing technique that they've *recently discovered* on a 15 person project 
applies to large systems.  Big systems have been using these approaches for 
years...or they failed. Now, that's not to say that the level of rigor advised 
by many of the test-first proponents really happens on big projects either. 
Is it economical to spend time writing code to check a 'getter'/'setter' 
interface that will just obviously work?  The answer is no.  In fact, the 
testing you can avoid, just like the coding you can avoid, is really a big 
part of successful big system development.

 From my experience the best-practice of testing depends on what the code is 
used for and what else depends on it.  If it's a widely used library (say 
date-time to pick one :) you want it to be very well unit tested because 
thousands of LOC will depend on it.  Every time you modify it you have to 
retest a large amount of code. It also turns out to be easy to unit test 
because it doesn't depend on much. On the other hand, take the case of a user 
interface which has no other code that depends on it -- my advice is to skip 
most of the unit and automated tests.  For one thing, it's very hard to write 
useful test code.  For another, a human can see in 1 second what a machine can 
never see (ugly layout, poor interaction usability, etc).  Since testing at 
the 'top level' of the architecture depends on basically all the other 
software in the system it tends to change rapidly -- people can quickly adjust 
to the fact that the widgets moved around on the screen, test programs tend to 
be fragile to these sort of changes. And finally,  since no other code depends 
on this code it isn't worth the time -- you can chance it at will. Bottom line 
is that not all code is created equal w.r.t to the need or ease of testing.

Of course the landscape isn't static either -- some good things have happened. 
One thing that's really changed is that the test first/XP/Agile folks have 
managed to convince developers that they actually need to execute their code 
before they deliver -- a good thing.  This often wasn't common practice 10 
years ago. Also, developers have more and more pre-tested code to pull off the 
shelf -- better libraries and less low level code to write and test.

Even with all that, I still say testing isn't enough because I know that even 
the stuff that's *easy* to test will have gaps.  There are literally thousands 
of Boost date-time tests (2319 'asserts' to be exact) that run in the 
regression every day, but I don't believe for a minute that the library is 
bug-free or can't be the source of bugs in other code.

As an example of the latter, initially the date class had no default 
constructor and it is built to guarantee that you can't construct an invalid 
date.  It's also an immutable type, so you can't set parts of a date to make 
an invalid one (you can assign, but you have to go thru checks to do that).  I 
wanted these properties so that I could pass dates around in interfaces and 
wouldn't have to 'check' the precondition that a date is valid when I go to 
use it.  All good, except that dates also allowed 'not_a_date_time', 
+infinity, and -infinity as a valid values.  So if you call date::year() on 
something that's set to not_a_date_time the results are undefined.

Now it's trivial to write some 'incorrect' code and a bunch of tests that 
will always work:

void f(const date& d)
{
    int year = d.year(); //oops....fails in some cases
}

should really always be:

   if (d.is_special()) {
      //do something here
   }
   else {....
      int year = d.year()

So going back to the default constructor, I eventually added one that 
constructs to not_a_date_time after many users requested it.  Mostly for use 
in collections that need this. A very logical choice for default, but my worry 
all along was that people would make the mistake above.  That is, now instead 
of being forced to think about putting some sort of correct date value or 
using not_a_date_time explicitly:

    date d(not_a_date_time);

they can just say

    date d;

Aside from the obvious loss of readability, I worried that with just these few 
lines of code the correctness of a larger program can be undermined by failing 
to check the special states.

So far, I'm not aware of anyone having an issue with this in a large program, 
but I'd be shocked if someone didn't create a bug this way eventually.  It's 
trivial to write and test code that always uses 'valid dates', ship it, and 
everything will work fine.  Then one day someone else will unknowingly make a 
call using a default constructed date and 'boom' a function that's been 
working fine and is fully 'tested' will blow up with unexpected results.

So, is it the right set of design decisions?  I don't know, but there's 
clearly a tension between correctness, 'ease of use', and overall 
applicability.  My take on the EventLogger example is that it's the wrong set 
of choices. There's very little valid use of the object without the stream. 
The stream is a low-level stable library that all programmers should know 
anyway. It's wide open to creating runtime errors that are not localized, and 
it's low level library that I would expect to use all over in a program. So 
I'd want the number of error modes to be as small as possible, because I'm 
certain they won't be writing code to test all the cases....

Jeff