[boost] Regression testing goes horribly wrong! (+ patch)

From: Guillaume Melquiond (guillaume.melquiond_at_[hidden])
Date: 2004-07-09 13:20:39

Hi,

If you take a look at the regression logs [1], you will see a few
problems.

First, a lot of test types [2] are missing. The "bind" or "date-time"
libraries don't have any value in the test type column. Other libraries
like the "array" library do have values in this column, however.

Second, the name of some "nested" libraries are wrong. For example, the
"algorithm/string" is shown as "algorithm", the "numeric/conversion" and
"numeric/interval" libraries are both mixed under "numeric". But the
"numeric/ublas" is correctly named.

Third, expected failures are not correctly shown in the advanced
regression logs [3] since some library names are wrong.

Fourth, some libraries are compiled in the wrong directories. Instead of
being compiled in "bin/boost/libs/whatever/test/something.test", they
are directly compiled in "bin/boost/status/something.test" [4]. If you
run the regression tests, you will see what I mean.

It's quite possible that the problems for the first three points are
related to the fourth. Indeed, it seems only libraries wrongly compiled
in "bin/boost/status/something.test" get to be correctly identified.
It's the case for the "array" and "numeric/ublas" libraries for example.

I don't understand how regression testing takes place into Boost. From
my humble point of view, it is quite a mess. I didn't understand why
some libraries were compiled in the wrong place; I suppose it has to do
with bjam and their Jamfiles. I did however find a way to correct the
first two bugs (and probably the third). The test_log.xml are indeed
wrongly generated by process_jam_log.

So here is a patch to process_jam_log.cpp so that it correctly generates
the test_log.xml files. Unfortunately I don't understand at all some
parts of this program, so I only corrected what seemed obviously wrong
to me. But maybe the bugs were somewhere else and I only designed a lame
workaround? Could the author of these lines shed some light on what
their purpose was supposed to be?

In the meantime, maybe the regression testers could clear their "bin"
directory (so that all the test_log.xml are created from scratch) and
apply the patch? Since Boost 1.31 is supposed to ship in less than 20
days, it would be great if the regression logs were correctly generated.
Thanks.

Best regards,

Guillaume

[1] For example, a small regression log:
http://boost.sourceforge.net/regression-logs/cs-AIX.html
All recent regression logs are affected.

[2] "run", "compile", "compile_fail", etc

[3] Some of the yellow cells should actually be gray in this log:
http://boost.sourceforge.net/regression-logs/cs-win32_metacomm/developer/summary.html

[4] For example, look at the compile line used for:
http://boost.sourceforge.net/regression-logs/cs-AIX-links.html#lambda-bll_and_function-vacpp

Index: process_jam_log.cpp
===================================================================
RCS file: /cvsroot/boost/boost/tools/regression/process_jam_log.cpp,v
retrieving revision 1.25
diff -u -r1.25 process_jam_log.cpp
--- process_jam_log.cpp 28 Jun 2004 03:04:21 -0000      1.25
+++ process_jam_log.cpp 9 Jul 2004 17:38:59 -0000
@@ -241,11 +241,11 @@
       }

       test_info info;
-      test2info_map::iterator itr( test2info.find( library_name + "/" + test_name ) );
+      test2info_map::iterator itr( test2info.find( test_name ) );
       if ( itr != test2info.end() )
         info = itr->second;

-      if ( library_name.empty() )
+      if ( !info.file_path.empty() )
         library_name = test_path_to_library_name( info.file_path );

       if ( info.type.empty() )
@@ -511,8 +511,6 @@
           line.find( "\"", pos+3 )-pos-3 );
         convert_path_separators( info.file_path );
         if ( info.file_path.find( "libs/libs/" ) == 0 ) info.file_path.erase( 0, 5 );
-        if ( test_name.find( "/" ) == string::npos )
-            test_name = "/" + test_name;
         test2info.insert( std::make_pair( test_name, info ) );
   //      std::cout << test_name << ", " << info.type << ", " << info.file_path << "\n";
       }