Re: [boost] [all][testing] Regression summary upgrades

Subject: Re: [boost] [all][testing] Regression summary upgrades
From: Edward Diener (eldiener_at_[hidden])
Date: 2015-05-11 12:54:05

On 5/11/2015 12:09 PM, John Maddock wrote:
>
>
> On 11/05/2015 16:46, Adam Wulkiewicz wrote:
>> Hi,
>>
>> Recently the regression summary was upgraded. The cells corresponding
>> to the failing tests can have names comp, link, run or fail.
>> This way we can see what's the nature of the failure without going
>> into details.
> This is great: many thanks!
>
>> See, e.g. the tests for Math:
>> http://www.boost.org/development/tests/develop/developer/math.html
>>
>> The last category of failures (generic "fail") is for the others or
>> unknown.
>> Currently the "Lib" errors are falling into this category. There are
>> some in Math, e.g.:
>> http://www.boost.org/development/tests/develop/developer/output/CrystaX-NET-apilevel-19-armeabi-v7a-hard-boost-bin-v2-libs-math-test-powm1_sqrtp1m1_test-test-clang-linux-3-5-release-link-static-target-os-android-threading-multi.html
>>
> That's a weird one - if you follow the links it actually says that
> linking the lib succeeded - which leaves me wondering what actually went
> wrong?
>
>> So we could mark them explicitly as "lib" failures and then the
>> other/unknown failures could be marked as "unkn" or left as it is now
>> - "fail".
>> This has sense if there can be other "types" of failures, e.g. caused
>> by some error in the testing scripts, Build or Regression, for which
>> the reason is somehow unknown. Though I haven't seen them.
>>
>> I propose to go further with this and to mark the compilation failures
>> for which we know the reason of the failure in a special way:
>> - "file" - file too big (reported on Windows, e.g.
>> http://www.boost.org/development/tests/develop/developer/output/MinGW-w64-4-8%20jc-bell-boost-bin-v2-libs-intrusive-test-unordered_multiset_test-test-gcc-mingw-4-8-2-debug.html)
>>
>> - "time" - time limit exceeded (e.g.
>> http://www.boost.org/development/tests/develop/developer/output/teeks99-03b-Ubuntu12-04-64-boost-bin-v2-libs-math-test-test_carlson-test-gcc-4-5-debug-link-static.html)
>>
>> I saw them in many libraries and I think it would be very useful for
>> libraries having many tests like Math or Geometry.
>>
> +1, time limit exceptions are a big issue for the Math and
> Multiprecision libs... and yes, I've already spent a lot of time
> refactoring to make them smaller, but they still persist.  I suspect
> many of these are caused by virtual machines with insufficient RAM
> allocated, that then end up thrashing the HD: many of the tests that
> time out compile really pretty quickly here (even on obsolete hardware).

I see some time-out errors in VMD testing. The code makes heavy use of 
the preprocessor but no single test should last more than 5 minutes ( 
300 seconds ).

I also would love to see time limit exceeded marked differently, and 
maybe some way of having regression testers to increase the time limit 
when those failures occur.

I think it is great work to have the regression test matrix marked more 
specifically.