Re: [boost] [all][testing] Regression summary upgrades

Subject: Re: [boost] [all][testing] Regression summary upgrades
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2015-05-12 06:25:48

Edward Diener wrote:
> On 5/11/2015 12:09 PM, John Maddock wrote:
>> On 11/05/2015 16:46, Adam Wulkiewicz wrote:
>>> I propose to go further with this and to mark the compilation failures
>>> for which we know the reason of the failure in a special way:
>>> - "file" - file too big (reported on Windows, e.g.
>>> http://www.boost.org/development/tests/develop/developer/output/MinGW-w64-4-8%20jc-bell-boost-bin-v2-libs-intrusive-test-unordered_multiset_test-test-gcc-mingw-4-8-2-debug.html) 
>>>
>>>
>>> - "time" - time limit exceeded (e.g.
>>> http://www.boost.org/development/tests/develop/developer/output/teeks99-03b-Ubuntu12-04-64-boost-bin-v2-libs-math-test-test_carlson-test-gcc-4-5-debug-link-static.html) 
>>>
>>>
>>> I saw them in many libraries and I think it would be very useful for
>>> libraries having many tests like Math or Geometry.
>>>
>> +1, time limit exceptions are a big issue for the Math and
>> Multiprecision libs... and yes, I've already spent a lot of time
>> refactoring to make them smaller, but they still persist.  I suspect
>> many of these are caused by virtual machines with insufficient RAM
>> allocated, that then end up thrashing the HD: many of the tests that
>> time out compile really pretty quickly here (even on obsolete hardware).
>
> I see some time-out errors in VMD testing. The code makes heavy use of 
> the preprocessor but no single test should last more than 5 minutes ( 
> 300 seconds ).
>
> I also would love to see time limit exceeded marked differently, and 
> maybe some way of having regression testers to increase the time limit 
> when those failures occur.
>
> I think it is great work to have the regression test matrix marked 
> more specifically.

I created a PR with the change: 
https://github.com/boostorg/regression/pull/13
Feel free to write your suggestions.

Regards,
Adam