|
|
Boost Testing : |
Subject: Re: [Boost-testing] Comparing different runs of regression tests
From: Ben Pope (benpope81_at_[hidden])
Date: 2013-07-22 23:53:42
On 03/07/13 02:59, Marshall Clow wrote:
>
> On Jul 2, 2013, at 7:14 AM, Ben Pope wrote:
>
>> I'm just about to define BOOST_THREAD_VERSION=4 for my test runners and it would be interesting to see if there is a difference in test results.
>
> Ok. I have two tools. Once compares two runs (all toolsets)
> Example output:
Well, my experiment didn't go very well, as defining
BOOST_THREAD_VERSION=4 and BOOST_THREAD_USES_CHRONO=1 broke around 129
tests, but strangely all of those failing tests were in clang and not in
gcc - actually, that's not strange; I think I only defined them on the
clang configurations!
ben:~/development/boost/test/results$ ./boostLog.py BP\ x86_64\
C++11-2013-07-22.xml BP\ x86_64\ C++11-2013-07-23.xml
Source: trunk (revision 85098)
Date: 2013-07-21T20:57:46Z
102 tests have empty toolset names
332 tests have empty test names
Test count by toolset (18795 total tests)
102 (21 failed)
6235 (143 failed) gcc-4.8.1~c++11
6229 (237 failed) clang-linux-3.3~c++11_libc++
6229 (239 failed) clang-linux-3.2~c++11_libc++
Source: trunk (revision 85117)
Date: 2013-07-22T20:59:17Z
102 tests have empty toolset names
334 tests have empty test names
Test count by toolset (18797 total tests)
102 (1 failed)
6235 (144 failed) gcc-4.8.1~c++11
6230 (170 failed) clang-linux-3.3~c++11_libc++
6230 (171 failed) clang-linux-3.2~c++11_libc++
New failing tests (1)
thread:ex_lambda_future_lib:gcc-4.8.1~c++11
New Passing tests (129)
Well, I found a minor problem with boostLog.py; the new passing tests
are not listed.
Obviously I can reverse the arguments and get the new passing tests
listed as new failing ones:
ben:~/development/boost/test/results$ ./boostLog.py BP\ x86_64\
C++11-2013-07-23.xml BP\ x86_64\ C++11-2013-07-22.xml
Source: trunk (revision 85117)
Date: 2013-07-22T20:59:17Z
102 tests have empty toolset names
334 tests have empty test names
Test count by toolset (18797 total tests)
102 (1 failed)
6235 (144 failed) gcc-4.8.1~c++11
6230 (170 failed) clang-linux-3.3~c++11_libc++
6230 (171 failed) clang-linux-3.2~c++11_libc++
Source: trunk (revision 85098)
Date: 2013-07-21T20:57:46Z
102 tests have empty toolset names
332 tests have empty test names
Test count by toolset (18795 total tests)
102 (21 failed)
6235 (143 failed) gcc-4.8.1~c++11
6229 (237 failed) clang-linux-3.3~c++11_libc++
6229 (239 failed) clang-linux-3.2~c++11_libc++
New failing tests (129)
atomic:atomicity:clang-linux-3.2~c++11_libc++
atomic:atomicity:clang-linux-3.3~c++11_libc++
atomic:ordering:clang-linux-3.2~c++11_libc++
atomic:ordering:clang-linux-3.3~c++11_libc++
circular_buffer:bounded_buffer_comparison:clang-linux-3.2~c++11_libc++
circular_buffer:bounded_buffer_comparison:clang-linux-3.3~c++11_libc++
date_time:testmicrosec_time_clock:clang-linux-3.2~c++11_libc++
interprocess:condition_any_test:clang-linux-3.2~c++11_libc++
interprocess:condition_any_test:clang-linux-3.3~c++11_libc++
interprocess:condition_test:clang-linux-3.2~c++11_libc++
interprocess:condition_test:clang-linux-3.3~c++11_libc++
interprocess:file_lock_test:clang-linux-3.2~c++11_libc++
interprocess:file_lock_test:clang-linux-3.3~c++11_libc++
interprocess:message_queue_test:clang-linux-3.2~c++11_libc++
interprocess:message_queue_test:clang-linux-3.3~c++11_libc++
interprocess:mutex_test:clang-linux-3.2~c++11_libc++
interprocess:mutex_test:clang-linux-3.3~c++11_libc++
interprocess:mutex_timeout_test:clang-linux-3.2~c++11_libc++
interprocess:mutex_timeout_test:clang-linux-3.3~c++11_libc++
interprocess:named_condition_any_test:clang-linux-3.2~c++11_libc++
interprocess:named_condition_any_test:clang-linux-3.3~c++11_libc++
interprocess:named_condition_test:clang-linux-3.2~c++11_libc++
interprocess:named_condition_test:clang-linux-3.3~c++11_libc++
interprocess:named_mutex_test:clang-linux-3.2~c++11_libc++
interprocess:named_mutex_test:clang-linux-3.3~c++11_libc++
interprocess:named_recursive_mutex_test:clang-linux-3.2~c++11_libc++
interprocess:named_recursive_mutex_test:clang-linux-3.3~c++11_libc++
interprocess:named_semaphore_test:clang-linux-3.2~c++11_libc++
interprocess:named_semaphore_test:clang-linux-3.3~c++11_libc++
interprocess:named_sharable_mutex_test:clang-linux-3.2~c++11_libc++
interprocess:named_sharable_mutex_test:clang-linux-3.3~c++11_libc++
interprocess:named_upgradable_mutex_test:clang-linux-3.2~c++11_libc++
interprocess:named_upgradable_mutex_test:clang-linux-3.3~c++11_libc++
interprocess:recursive_mutex_test:clang-linux-3.2~c++11_libc++
interprocess:recursive_mutex_test:clang-linux-3.3~c++11_libc++
interprocess:robust_emulation_test:clang-linux-3.2~c++11_libc++
interprocess:robust_emulation_test:clang-linux-3.3~c++11_libc++
interprocess:robust_recursive_emulation_test:clang-linux-3.2~c++11_libc++
interprocess:robust_recursive_emulation_test:clang-linux-3.3~c++11_libc++
interprocess:semaphore_test:clang-linux-3.2~c++11_libc++
interprocess:semaphore_test:clang-linux-3.3~c++11_libc++
interprocess:sharable_mutex_test:clang-linux-3.2~c++11_libc++
interprocess:sharable_mutex_test:clang-linux-3.3~c++11_libc++
interprocess:upgradable_mutex_test:clang-linux-3.2~c++11_libc++
interprocess:upgradable_mutex_test:clang-linux-3.3~c++11_libc++
locale:test_boundary:clang-linux-3.2~c++11_libc++
locale:test_boundary:clang-linux-3.3~c++11_libc++
locale:test_codepage:clang-linux-3.2~c++11_libc++
locale:test_codepage:clang-linux-3.3~c++11_libc++
locale:test_codepage_converter:clang-linux-3.2~c++11_libc++
locale:test_codepage_converter:clang-linux-3.3~c++11_libc++
locale:test_collate:clang-linux-3.2~c++11_libc++
locale:test_collate:clang-linux-3.3~c++11_libc++
locale:test_config:clang-linux-3.2~c++11_libc++
locale:test_config:clang-linux-3.3~c++11_libc++
locale:test_convert:clang-linux-3.2~c++11_libc++
locale:test_convert:clang-linux-3.3~c++11_libc++
locale:test_date_time:clang-linux-3.2~c++11_libc++
locale:test_date_time:clang-linux-3.3~c++11_libc++
locale:test_generator:clang-linux-3.2~c++11_libc++
locale:test_generator:clang-linux-3.3~c++11_libc++
locale:test_icu_vs_os_timezone:clang-linux-3.2~c++11_libc++
locale:test_icu_vs_os_timezone:clang-linux-3.3~c++11_libc++
locale:test_ios_prop:clang-linux-3.2~c++11_libc++
locale:test_ios_prop:clang-linux-3.3~c++11_libc++
locale:test_message:clang-linux-3.2~c++11_libc++
locale:test_message:clang-linux-3.3~c++11_libc++
locale:test_posix_collate:clang-linux-3.2~c++11_libc++
locale:test_posix_collate:clang-linux-3.3~c++11_libc++
locale:test_posix_convert:clang-linux-3.2~c++11_libc++
locale:test_posix_convert:clang-linux-3.3~c++11_libc++
locale:test_posix_formatting:clang-linux-3.2~c++11_libc++
locale:test_posix_formatting:clang-linux-3.3~c++11_libc++
locale:test_std_collate:clang-linux-3.2~c++11_libc++
locale:test_std_collate:clang-linux-3.3~c++11_libc++
locale:test_std_convert:clang-linux-3.2~c++11_libc++
locale:test_std_convert:clang-linux-3.3~c++11_libc++
locale:test_utf:clang-linux-3.2~c++11_libc++
locale:test_utf:clang-linux-3.3~c++11_libc++
locale:test_winapi_collate:clang-linux-3.2~c++11_libc++
locale:test_winapi_collate:clang-linux-3.3~c++11_libc++
locale:test_winapi_convert:clang-linux-3.2~c++11_libc++
locale:test_winapi_convert:clang-linux-3.3~c++11_libc++
locale:test_winapi_formatting:clang-linux-3.2~c++11_libc++
locale:test_winapi_formatting:clang-linux-3.3~c++11_libc++
lockfree:freelist_test:clang-linux-3.2~c++11_libc++
lockfree:freelist_test:clang-linux-3.3~c++11_libc++
lockfree:queue_bounded_stress_test:clang-linux-3.2~c++11_libc++
lockfree:queue_bounded_stress_test:clang-linux-3.3~c++11_libc++
lockfree:queue_fixedsize_stress_test:clang-linux-3.2~c++11_libc++
lockfree:queue_fixedsize_stress_test:clang-linux-3.3~c++11_libc++
lockfree:queue_interprocess_test:clang-linux-3.2~c++11_libc++
lockfree:queue_interprocess_test:clang-linux-3.3~c++11_libc++
lockfree:queue_test:clang-linux-3.2~c++11_libc++
lockfree:queue_test:clang-linux-3.3~c++11_libc++
lockfree:queue_unbounded_stress_test:clang-linux-3.2~c++11_libc++
lockfree:queue_unbounded_stress_test:clang-linux-3.3~c++11_libc++
lockfree:spsc_queue_stress_test:clang-linux-3.2~c++11_libc++
lockfree:spsc_queue_stress_test:clang-linux-3.3~c++11_libc++
lockfree:spsc_queue_test:clang-linux-3.2~c++11_libc++
lockfree:spsc_queue_test:clang-linux-3.3~c++11_libc++
lockfree:stack_bounded_stress_test:clang-linux-3.2~c++11_libc++
lockfree:stack_bounded_stress_test:clang-linux-3.3~c++11_libc++
lockfree:stack_fixedsize_stress_test:clang-linux-3.2~c++11_libc++
lockfree:stack_fixedsize_stress_test:clang-linux-3.3~c++11_libc++
lockfree:stack_interprocess_test:clang-linux-3.2~c++11_libc++
lockfree:stack_interprocess_test:clang-linux-3.3~c++11_libc++
lockfree:stack_test:clang-linux-3.2~c++11_libc++
lockfree:stack_test:clang-linux-3.3~c++11_libc++
lockfree:stack_unbounded_stress_test:clang-linux-3.2~c++11_libc++
lockfree:stack_unbounded_stress_test:clang-linux-3.3~c++11_libc++
signals2:mutex_test:clang-linux-3.2~c++11_libc++
signals2:mutex_test:clang-linux-3.3~c++11_libc++
spirit/classic:bug_000008:clang-linux-3.2~c++11_libc++
spirit/classic:bug_000008:clang-linux-3.3~c++11_libc++
spirit/classic:grammar_mt_tests:clang-linux-3.2~c++11_libc++
spirit/classic:grammar_mt_tests:clang-linux-3.3~c++11_libc++
spirit/classic:owi_mt_tests:clang-linux-3.2~c++11_libc++
spirit/classic:owi_mt_tests:clang-linux-3.3~c++11_libc++
spirit/classic:scoped_lock_tests:clang-linux-3.2~c++11_libc++
spirit/classic:scoped_lock_tests:clang-linux-3.3~c++11_libc++
spirit/classic:scoped_lock_tests_debug:clang-linux-3.2~c++11_libc++
spirit/classic:scoped_lock_tests_debug:clang-linux-3.3~c++11_libc++
test:sync_access_test:clang-linux-3.2~c++11_libc++
test:sync_access_test:clang-linux-3.3~c++11_libc++
wave:test_re2c_lexer:clang-linux-3.2~c++11_libc++
wave:test_re2c_lexer:clang-linux-3.3~c++11_libc++
wave:test_slex_lexer:clang-linux-3.2~c++11_libc++
wave:test_slex_lexer:clang-linux-3.3~c++11_libc++
New Passing tests (1)
I took the liberty of sorting the result; my first feature request :P
That's a fair few tests that will break when thread moves to V4 by
default. A very significant proportion seem to be:
no member named 'scoped_lock' in 'boost::mutex'
Well, that's all for today, the scripts are useful, thanks!
Ben