Re: [boost] Re: [1.33.0] Let's start preparations...

From: Martin Wille (mw8329_at_[hidden])
Date: 2005-03-07 17:20:05

David Abrahams wrote:
> "Victor A. Wagner Jr." <vawjr_at_[hidden]> writes:
> 
> 
>>At Sunday 2005-03-06 18:52, you wrote:
>>
>>>Let's start revving up to release Boost 1.33.0. Personally, I'd like
>>>to get it out the door by mid-April at the latest, and I'm offering
>>>to manage this release.
>>
>>thank you for your offer, but if you don't get the damned regression 
> 
> 
> Please keep your language civil.
> 
> 
>>testing working FIRST (it's been non-responsive
> 
> 
> Can you please be more specific about what has been non-responsive?  I
> doubt anyone can fix anything without more information.

Whatever tone might be appropriate or not ...

Several testers have raised issues and plead for better communication 
several (probably many) times. Most of the time, we seem to get ignored, 
unfortunately. I don't want to accuse anyone of voluntarily neglecting 
our concerns. However, I think we apparently suffer from a "testing is 
not too well understood" problem at several levels.

The tool chain employed for testing is very complex (due to the 
diversity of compilers and operation systems involved) and too fragile.
Complexity leads to lack of understanding (among the testers and among 
the library developers) and to false assumptions and to lack of 
communication. It additionally causes long delays between changing code 
and running the tests and between running the tests and the result being 
rendered. This in turn makes isolating bugs in the libraries more 
difficult. Fragility leads to the testing procedure breaking often and 
to breaking without getting noticed for some time and to breaking 
without anyone being able to recognize immediately exactly what part 
broke. This is a very unpleasant situation for anyone involved and it 
causes a significant level of frustration at least among those who run 
the tests (e.g. to see the own test results not being rendered for 
severals days or to see the test system being abused as a change 
announcement system isn't exactly motivating).

Please, understand that a lot of resources (human and computers) are 
wasted due to these problems. This waste is most apparent those who run 
the tests. However, most of the time, issues raised by the testers 
seemed to get ignored. Maybe, that was just because we didn't yell loud 
enough or we didn't know whom to address or how to fix the problems.

Personally, I don't have any problem with the words Victor chose. Other 
people might have. If you're one of them, then please understand that 
we're feeling there's something going very wrong with the testing 
procedure and we're afraid it will go on that way and we'll lose a lot 
of the quality (and the reputation) Boost has.

The people involved in creating the test procedure have put very much 
effort in it and the resulting system does its job nicely when it 
happens to work correctly. However, apparently, the overall complexity 
of the testing procedure has grown above our management capabilities.
This is one reason why release preparations take so long.

Maybe, we should take a step back and collect all the issues we have and 
all knowledge about what is causing these issues.

I'll make a start, I hope others will contribute to the list.
Issues and causes unordered (please, excuse any duplicates):

- testing takes a huge amount of resources (HD, CPU, RAM, people 
operating the test systems, people operating the result rendering 
systems, people coding the test post processing tools, people finding 
the bugs in the testing system)
- the testing procedure is complex
- the testing procedure is fragile
- the code-change to result-rendering process takes too long
- bugs in the testing procedure take too long to get fixed
- changes to code that will affect the testing procedure aren't 
communicated well
- incremental testing doesn't work flawlessly
- deleting tests requires manual purging of old results in an 
incremental testing environment.
- the number of target systems for testing is rather low; this results 
in questionable portability.
- lousy performance of Sourceforge
- resource limitations at Sourceforge (e.g. the number of files there)
- between releases the testing system isn't as well maintained as during 
the release preparations.
- test results aren't easily reproducible. They depend much on the 
components on the respective testing systems (e.g. glibc version, system 
compiler version, python version, kernel version and even on the 
processor used on Linux)
- library maintainers don't have access to the testing systems; this 
results in longer test-fix cycles.
- changes which will cause heavy load at the testing sites never get 
announced in advance. This is a problem when testing resources have to 
be shared with the normal workload (like in my case).
- changes that requires old test results to get purged usually don't get 
announced.
- becoming a new contributor for testing resources is too difficult.
- we're supporting compilers that compile languages significantly 
different from C++.
- there's no common concept of which compilers to support and which not.
- post-release displaying of test results apparently takes too much 
effort. Otherwise, it would have been done.
- tests are run for compilers for which they are known to fail. 100% 
waste of resources here.
- known-to-fail tests are rerun although the dependencies didn't change.
- some tests are insanely big.
- some library maintainers feel the need to run their own tests 
regularly. Ideally, this shouldn't be necessary.
- test post processing has to work on output from different compilers. 
Naturally, that output is formatted differently.
- test post processing makes use of very recent XSLT features.
- several times the post processing broke due to problems with the XSLT 
processor.
- XSLT processing takes long (merging all the components that are input 
to the result rendering takes ~1 hour just for the tests I run)
- the number of tests is growing
- there's no way of testing experimental changes to core libraries 
without causing reruns of most tests (imagine someone would want to test 
an experimental version of some part of MPL).
- switching between CVS branches during release preparations takes 
additional resources and requires manual intervention.

I'm sure testers and library developers are able to add a lot more  to 
the list.

Regards,
m