Re: [boost] Boost 1.36.0 release notice

From: Robert Ramey (ramey_at_[hidden])
Date: 2008-08-18 03:12:56

Vladimir Prus wrote:

>> If regression testing were setup for
>> boost tools, they would be demonstrated to be functioning as
>> expected before they were used in the actual release process.
>> Had this been procedure been in place, the issue raised
>> above would not have occurred.

> Last time I've checked, boost.serialization had some automatic tests.
> Does it mean that no issue has ever occurred with
> boost.serialization, and no issue
> will ever occur in future? I don't think so.

Of course not, but I'm sure that the extensive testing of the
boost serialization has avoided untold numbers of problems
for users.  And, it has made the package much easier to develop.
I believe that it would do the same for boost tools.
It's well known that testing can't prove the absense
of faults, only their existence.  That doesn't make testing
pointless.

> Quoting Beman:
>
>        Case in point, the doc build tool chain broke for this
>        release, forcing a really messy workaround. Joel is trying to
>        fix it now, but it is slow going since the test cases that
>        fail for others works for him.
>
> So, it appears like the problem is not that there's no *testing
> infrastructure* is missing, it's that for some behaviours, there are
> no tests or those tests are not sufficiently reliable.

Taking the case of boost book.  As far as I know there is
no regression testing to verify that it works when something
it depends upon changes.

>Therefore, it does not seem like
> generic advice of "we need tests" is going to help much -- there
> needs to be actual work on specific tests.

My suggestion is a lot more than "we need tests".  My suggestion
is that boost tools should be subjected to the same procedures
that boost libaries are.  Using boost book as an example, I would
like to see boost/tools/boostbook/test/Jamfile ... and see that
boost book shows up in the test matrix just like any library does.
I would like to see boost book "packaged" with documentation,
examples and tests so that users would feel confident to use it for their 
own
projects.  If the "boost process" is good for libraries, I think
it would be good for tools too.  I would love to use boost book
/ quickbook for my own projects, but I don't feel its polished
enough to depend upon.

The only argument I can see against this is that it seems to
be alot of extra work.  I'm sympathetic to this.  I used to think that.
But, the boost requirement for a separate test suite and
the development of the serialization library  made me
realize I was wrong.  I believe that extending the application of
of the "boost process" to the tools would save effort in the long run.

> And even if you add a new test for each issue encountered during
> release process, it does not automatically mean that the resulting
> release packages need not be examined. For example, here at work we
> do have very extensive automatic tests, but still, every user-visible
> package goes through manual QA -- and yes, it does checks that
> documentation exists.

No one has suggested that testing proves the absense of bugs. Only
that unit testing is a cost effective proposition.

Robert Ramey