Re: [boost] Re: Regression testing goes horribly wrong! (+ patch)

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2004-07-11 21:22:43

Guillaume Melquiond writes:
Le dim 11/07/2004 à 04:46, Aleksey Gurtovoy a écrit :
>
> [...]
>
> > >> This was fixed in the CVS about two weeks ago (see
> > >> http://tinyurl.com/36v6h#dont_redirect, for instance); simple
> > >> rebuilding "process_jam_log" executable will take care of it (I would
> > >> have posted a notice about this fix if I didn't forget that unlike us
> > >> not everybody rebuilds "process_jam_log" on every run).
> > >
> > > That's strange. I do use the CVS version and I did rebuild the
> > > process_jam_log.
> >
> > Did you do a clean run?
>
> Yes. It's what my sentence was supposed to meant.

OK.

> > Given that we remove all binaries and sources and rebuild everything
> > from scratch on every run, I'm inclined to think that our fix works
> > and that the problems you see are caused by a stale environment.
>
> Seems like your fix never get commited if it does fix for you.

It's not the same fix, but you are actually right, please see below.

> > If you still can reproduce the problem on a clean state, you can
> > always step through the process_jam_log code in debugger and see how these
> > empty test types appear there.
>
> I did. The reason is quite simple. And it's what I have been explaining
> in my previous mails: test_info is not created with the same name that
> the name used to access it later. The test type is set at creation time,
> and the test_log.xml is used at a later time. So they are no more
> related since the test type cannot be retrieved during the second
> access.

OK, understood. It all works for us because we apply an [uncommitted] patch
to "testing.jam" that modifies bjam output to prefix a test case name with a
library name (when appropriate).

> [...]
>
> > > But process_jam_log can't actually deal with them,
> >
> > It definitely can, please see below.
>
> Please see below why it definitely cannot.
>
> > > because it does
> > > around line 520:
> > >
> > > test2info.insert( std::make_pair( test_name, info ) );
> > >
> > > test2info is a map and test_name is the short name of test.
> >
> > Nope, it's not. It's the name extracted from one of the top lines of
> > the bjam log, and the lines corresponding to the "new" test cases do
> > contain the library name:
> >
> >     boost-test(RUN) "utility/ref_test" : "libs\utility\ref_test.cpp"
> >     boost-test(RUN) "utility/addressof_test" : "libs\utility\addressof_test.cpp"
> >     ...
>
> boost-test(RUN) "ref_test" : "libs/utility/ref_test.cpp"
> boost-test(RUN) "addressof_test" : "libs/utility/addressof_test.cpp"

Ouch! We've fixed this one so long ago that I totally forgot that it's still
uncommitted. Sorry for the confusion it caused! It's in the CVS now. Rebuilding
everything should bring us to the desired state, at last.

>
> Maybe you did patch some Jamfiles or bjam?

"boost\tools\build\v1\testing.jam"

> It would explain why your
> tests work. And why the tests of all other regression testers fail in a
> similar way.

It definitely does.

> [...]
>
> > > So maybe we should replace it with something like:
> > >
> > > test2info.insert( std::make_pair( library_name + '/' + test_name, info ) );
> > >
> > > What do you think about it?
> >
> > That's basically what is already there.
>
> No it isn't unfortunately, since the library_name is not in the
> regress.log file when using the CVS version. My second patch is *really*
> required so that process_jam_log correctly works.

Part of your patch that I already committed + our fix to "testing.jam", yes.
Your full second patch, no. The reason we append the library name in "testing.jam"
instead of doing it in "process_jam_log" is that at the point of parsing
the old bjam log's

    boost-test(RUN) "test1" : "libs\numeric\ublas\test1\test1.cpp" "libs\numeric\ublas\test1\test11.cpp"

line you have no clue whether the test's *output directory* will be

    bin\boost\libs\numeric\ublas\test1.test

or

    bin\boost\status\test1.test

Therefore you cannot, in all cases, reliably construct the corresponding
'test2info' key so that it's retrievable later. "testing.jam" has this
information.

> So, unless you find and commit your patch that puts library_name in
> regress.log, I will commit my second patch.

Please understand that the only reason for my unwillingness to commit it at
once is that I didn't feel that we got to the root of the issue (then).

Thank you for your patience,
--
Aleksey Gurtovoy
MetaCommunications Engineering