Re: [boost] Regression testing goes horribly wrong! (+ patch)

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2004-07-10 03:01:22

Guillaume Melquiond writes:
> Hi,

Hi Guillaume,

>
> If you take a look at the regression logs [1], you will see a few
> problems.
>
> First, a lot of test types [2] are missing. The "bind" or
> "date-time" libraries don't have any value in the test type
> column. Other libraries like the "array" library do have values in
> this column, however.

This was fixed in the CVS about two weeks ago (see
http://tinyurl.com/36v6h#dont_redirect, for instance); simple
rebuilding "process_jam_log" executable will take care of it (I would
have posted a notice about this fix if I didn't forget that unlike us
not everybody rebuilds "process_jam_log" on every run).

>
> Second, the name of some "nested" libraries are wrong. For example,
> the "algorithm/string" is shown as "algorithm", the
> "numeric/conversion" and "numeric/interval" libraries are both mixed
> under "numeric". But the "numeric/ublas" is correctly named.

This is still an open issue.

>
> Third, expected failures are not correctly shown in the advanced
> regression logs [3] since some library names are wrong.

Consequently, this one is still open as well.

>
> Fourth, some libraries are compiled in the wrong
> directories. Instead of being compiled in
> "bin/boost/libs/whatever/test/something.test", they are directly
> compiled in "bin/boost/status/something.test" [4]. If you run the
> regression tests, you will see what I mean.

I wouldn't say that they are "wrong", because that implies that this
is a newly introduced behavior (read "bug"), and it's not the
case. 

Basically, whether the test results go into "bin/boost/status" or
"bin/boost/libs/" is determined by the age of the library: the new
libraries that provide their own Jamfile for testing and use bjam's
"subinclude" feature to bring it into "boost/status/jamfile" are put
in "bin/boost/libs/<library-name>" location, and the libraries that
are listing their test cases directly in the "status/jamfile" are put
in "bin/boost/status".

Granted, the latter scheme is deficient and makes it unnecessarily
hard to post-process the test results. How about we simply factor out
these "inline" test-suites into the corresponding Jamfiles, and be
done with the issue?

>
> It's quite possible that the problems for the first three points are
> related to the fourth.

The library name issue is not; it's simply a matter of tweaking
"process_jam_log" to extract it correctly.

> Indeed, it seems only libraries wrongly compiled in
> "bin/boost/status/something.test" get to be correctly identified.
> It's the case for the "array" and "numeric/ublas" libraries for
> example.
>
> I don't understand how regression testing takes place into
> Boost. From my humble point of view, it is quite a mess.

The particular part that you've stumbled upon most definitely
is, and it's about time to get it cleaned.

> I didn't understand why some libraries were compiled in the wrong
> place; I suppose it has to do with bjam and their Jamfiles. I did
> however find a way to correct the first two bugs (and probably the
> third). The test_log.xml are indeed wrongly generated by
> process_jam_log.
>
> So here is a patch to process_jam_log.cpp so that it correctly
> generates the test_log.xml files. Unfortunately I don't understand
> at all some parts of this program, so I only corrected what seemed
> obviously wrong to me.

Unfortunately, the patch is no good. In particular, it breaks the case
when there are multiple identically named tests in different
libraries. If you were trying to solve the composite library name
issue ("algorithm" vs. "algorithm/string"), I suggest we first do the
Jamfile refactoring I talked about above; it would allow us to
simplify the code significantly, and then we can tackle this one.

> But maybe the bugs were somewhere else and I only designed a lame
> workaround? Could the author of these lines shed some light on what
> their purpose was supposed to be?

The purpose is to correctly match the test results with a
corresponding test case from the right library, in presence of the
discussed variability in the output directory structure. 

>
> In the meantime, maybe the regression testers could clear their
>"bin" directory (so that all the test_log.xml are created from
>scratch) and apply the patch? 

Due to the above, I don't think it's a good idea.

> Since Boost 1.31 is supposed to ship in less than 20 days, it would
> be great if the regression logs were correctly generated.  

But this one is definitely worth fighting for. I'm going to look into
the library name issue tomorrow.

-- 
Aleksey Gurtovoy
MetaCommunications Engineering