Re: [boost] Boost 1.36.0 release notice

From: Vladimir Prus (vladimir_at_[hidden])
Date: 2008-08-18 03:30:32

Robert Ramey wrote:

>> Quoting Beman:
>>
>>        Case in point, the doc build tool chain broke for this
>>        release, forcing a really messy workaround. Joel is trying to
>>        fix it now, but it is slow going since the test cases that
>>        fail for others works for him.
>>
>> So, it appears like the problem is not that there's no *testing
>> infrastructure* is missing, it's that for some behaviours, there are
>> no tests or those tests are not sufficiently reliable.
> 
> Taking the case of boost book.  As far as I know there is
> no regression testing to verify that it works when something
> it depends upon changes.

I don't know, either. But large part of the doc toolchain is outside
of Boost -- e.g. Docbook and FOP. Those are huge, and it's not
likely somebody will be able to create tests for those. So, manual
inspection of results remain the only solution.

> 
>>Therefore, it does not seem like
>> generic advice of "we need tests" is going to help much -- there
>> needs to be actual work on specific tests.
> 
> My suggestion is a lot more than "we need tests".  My suggestion
> is that boost tools should be subjected to the same procedures
> that boost libaries are.  Using boost book as an example, I would
> like to see boost/tools/boostbook/test/Jamfile ... and see that
> boost book shows up in the test matrix just like any library does.

I think that presence or lack of boost/tools/boostbook/test/Jamfile
is an implementation detail. 

> I would like to see boost book "packaged" with documentation,
> examples and tests so that users would feel confident to use it for their
> own
> projects.  If the "boost process" is good for libraries, I think
> it would be good for tools too.  I would love to use boost book
> / quickbook for my own projects, but I don't feel its polished
> enough to depend upon.

I'm positively sure I saw documentation for boost.book. Quickbook is
also documented. There's tools/quickbook/test/Jamfile.v2, and some
test files there. For Boost.Build, there's both documentation and extensive
test suite.

So, it appears that your proposal boils down to:
1. Boost.Book should have tests
2. Test for tools should be included in the main test matrix

Have I left out something?

> The only argument I can see against this is that it seems to
> be alot of extra work.  I'm sympathetic to this.  I used to think that.
> But, the boost requirement for a separate test suite and
> the development of the serialization library  made me
> realize I was wrong.  I believe that extending the application of
> of the "boost process" to the tools would save effort in the long run.
> 
>> And even if you add a new test for each issue encountered during
>> release process, it does not automatically mean that the resulting
>> release packages need not be examined. For example, here at work we
>> do have very extensive automatic tests, but still, every user-visible
>> package goes through manual QA -- and yes, it does checks that
>> documentation exists.
> 
> No one has suggested that testing proves the absense of bugs. 

I've understood your prior statement ("the issue raised above would not have occurred.") 
in exactly this way. I apologize if I've misunderstood what you was saying.

- Volodya