From: David Abrahams (dave_at_[hidden])
Date: 2006-05-28 08:48:47
"Robert Ramey" <ramey_at_[hidden]> writes:
> David Abrahams wrote:
>> "Robert Ramey" <ramey_at_[hidden]> writes:
>>> As for not running the cross product of the various
>>> combinations - the problem is that failures now occur
>>> in just a couple of select situations that can't be
>>> predicted in advance. BCB failed on text archives
>>> only in some cases. STLPort 5.0 failed when
>>> using mapped collections. etc. Paring down the
>>> test suite would likely have missed these failures.
>>> The selected tests would all pass - thereby giving the
>>> false impression that the library was bug free.
>> Oh, there's a lot you can do to reduce it without losing any
>> interesting data. For example, there is no reason to believe you're
>> going to uncover serialization library bugs by testing against both
>> static and dynamic runtimes,
> I am testing for both static and dynamic runtimes.
I think you meant "not testing," which I see is true. You're testing
against static and dynamic versions of your library.
> When creating the DLL versions and also auto-linking it was
What was different?
> It is A LOT more than just recompiling.
> Now that things are working this is not necessary for debugging the
> serialization library itself.
>> or especially by testing both singlethreaded and multithreaded
>> builds, since you're not doing any threading in your tests.
> I don't believe I'm doing this. Currently, tests are built with
> the default threading model. That is, the Jamfile doesn't specify
> anything in this regard.
OK, maybe in my shock I overestimated. Sorry.
>>If you discover a test failure that appears only in one of these
>>cases it is undoubtedly due to a compiler or standard library bug,
>>not anything having to do with your code.
> That was my original point. If I'm not making any changes, which
> I haven't been, almost no failures have anything to do with the
> serialization code.
But we want to find out when changes to other Boost libraries break
it. And we want to find out when other people break it.
>> Also, you could cut the testing time by 2/3 instantly if you'd just
>> test all the archive types in a single executable, rather than
>> building a separate executable for each one.
> Hmmm - that was what I did originally but it was objected to
> in the first review.
On what grounds?
> Also, more compilers failed with due to resource exhaustion.
On some weaker compilers you might have to break a few tests into
multiple translation units.
> The current system has the benefit of permitting
> any user to test his own archive with all the tests. The documentation
> encourages users to do this with their own archives and I know they
> are doing it. I know this because when ever a user has a problem
> with his archive - I just tell him to run the whole test suite on it.
> The current system also lets me run just one archive by
> adding a command line argument - extremely useful.
I noticed. It's not only useful, but necessary, to cut down on the
number of targets the Jamfile generates.
All of that should be possible even with the tests built into a single
> I actually didn't have any idea how compilation and execution
> time varies by making the separation. I had just presumed
> that compilation was more or less proportional to the number
> of archives - given that a lot of time is spent on serialization
> code instantiation which would be proportional to the number
> of archives. Other code is already compiled in the library.
> Truth is since I run while I'm not watching I don't know
> how the time would vary. How do you know this?
I don't know for sure, but it's easy to see that every time you start
a new compilation command there is overhead that can be avoided.
Every time you include the same file in a new TU there's overhead that
can be avoided. One eventually develops a pretty good instinct for
these things after thinking about them a while.
>>And that would cover the
>> case where multiple archives are in use, which I don't think you're
>> covering today.
> see test_mult_archive_types
> In anycase, I have no problem with diminishing the the tests
> by just commenting out any combination of tests. Also the
> dynamic library versions can easily be suppressed by tweaking
> the Jamfile. As I said before, given that I'm making very few
> changes and I test those here, the boost test results are not
> used by me - or anyone else as far as I know.
The importance, to Boost as a whole, of having continuous testing of
all libraries should not be underestimated.
-- Dave Abrahams Boost Consulting www.boost-consulting.com
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk