Help with understanding regression test results
Hi, Recently I observed lots of test cases showing as failed in regression tests on branch 'develop' in Boost.Optional: http://www.boost.org/development/tests/develop/developer/optional.html When I try to inspect the results they are either empty or refer to the places in the code that do not exist anymore in branch 'develop', as though the test environments were using partial results of GIT commits. But the test dates are after the last commit, so it looks like something is broken with the testing framework. Anyone can help me with figuring out what to do? Regards, &rzej
On Monday 23 June 2014 09:46:56 Andrzej Krzemienski wrote:
Hi, Recently I observed lots of test cases showing as failed in regression tests on branch 'develop' in Boost.Optional: http://www.boost.org/development/tests/develop/developer/optional.html
When I try to inspect the results they are either empty or refer to the places in the code that do not exist anymore in branch 'develop', as though the test environments were using partial results of GIT commits. But the test dates are after the last commit, so it looks like something is broken with the testing framework. Anyone can help me with figuring out what to do?
Which tests do you mean?
2014-06-23 11:40 GMT+02:00 Andrey Semashev <andrey.semashev@gmail.com>:
On Monday 23 June 2014 09:46:56 Andrzej Krzemienski wrote:
Hi, Recently I observed lots of test cases showing as failed in regression tests on branch 'develop' in Boost.Optional: http://www.boost.org/development/tests/develop/developer/optional.html
When I try to inspect the results they are either empty or refer to the places in the code that do not exist anymore in branch 'develop', as though the test environments were using partial results of GIT commits. But the test dates are after the last commit, so it looks like something is broken with the testing framework. Anyone can help me with figuring out what to do?
Which tests do you mean?
This one, for instance: http://www.boost.org/development/tests/develop/developer/output/teeks99-03e-... The error message quotes a piece of implementation: value_or ( U const& v ) const { return this->is_initialized() ? get() : (v); } But there exists no such code in the current implementation in branch 'develop'. I am using no ternary operators (I used to in the previous commits). The test's date is that of today. My last commit was on Friday. Regards, &rzej
On Monday 23 June 2014 12:32:31 Andrzej Krzemienski wrote:
2014-06-23 11:40 GMT+02:00 Andrey Semashev <andrey.semashev@gmail.com>:
On Monday 23 June 2014 09:46:56 Andrzej Krzemienski wrote:
Hi, Recently I observed lots of test cases showing as failed in regression tests on branch 'develop' in Boost.Optional: http://www.boost.org/development/tests/develop/developer/optional.html
When I try to inspect the results they are either empty or refer to the places in the code that do not exist anymore in branch 'develop', as
though
the test environments were using partial results of GIT commits. But the test dates are after the last commit, so it looks like something is
broken
with the testing framework. Anyone can help me with figuring out what to do?
Which tests do you mean?
This one, for instance: http://www.boost.org/development/tests/develop/developer/output/teeks99-03e-> Ubuntu12-04-64-boost-bin-v2-libs-optional-test-optional_test_value_access-te st-gcc-4-8-debug.html
The error message quotes a piece of implementation:
value_or ( U const& v ) const { return this->is_initialized() ? get() : (v); }
But there exists no such code in the current implementation in branch 'develop'. I am using no ternary operators (I used to in the previous commits). The test's date is that of today. My last commit was on Friday.
Yes, looks like a problem. Maybe the checkout failed at some point for the tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If the update scripts ignore such errors then that's what may have happened.
2014-06-23 13:53 GMT+02:00 Andrey Semashev <andrey.semashev@gmail.com>:
On Monday 23 June 2014 12:32:31 Andrzej Krzemienski wrote:
2014-06-23 11:40 GMT+02:00 Andrey Semashev <andrey.semashev@gmail.com>:
On Monday 23 June 2014 09:46:56 Andrzej Krzemienski wrote:
Hi, Recently I observed lots of test cases showing as failed in regression tests on branch 'develop' in Boost.Optional:
http://www.boost.org/development/tests/develop/developer/optional.html
When I try to inspect the results they are either empty or refer to
the
places in the code that do not exist anymore in branch 'develop', as
though
the test environments were using partial results of GIT commits. But the test dates are after the last commit, so it looks like something is
broken
with the testing framework. Anyone can help me with figuring out what to do?
Which tests do you mean?
This one, for instance:
http://www.boost.org/development/tests/develop/developer/output/teeks99-03e-> Ubuntu12-04-64-boost-bin-v2-libs-optional-test-optional_test_value_access-te
st-gcc-4-8-debug.html
The error message quotes a piece of implementation:
value_or ( U const& v ) const { return this->is_initialized() ? get() : (v); }
But there exists no such code in the current implementation in branch 'develop'. I am using no ternary operators (I used to in the previous commits). The test's date is that of today. My last commit was on Friday.
Yes, looks like a problem. Maybe the checkout failed at some point for the tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If the update scripts ignore such errors then that's what may have happened.
What's the procedure in such cases? Can the owners of the machines/environments /scripts be contacted? regards, &rzej
On Monday 23 June 2014 13:57:50 Andrzej Krzemienski wrote:
2014-06-23 13:53 GMT+02:00 Andrey Semashev <andrey.semashev@gmail.com>:
Yes, looks like a problem. Maybe the checkout failed at some point for the tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If the update scripts ignore such errors then that's what may have happened.
What's the procedure in such cases? Can the owners of the machines/environments /scripts be contacted?
Yes, if the contact information is available in the tester description (the link in the test results table header). E.g. for your tester the information is: http://www.boost.org/development/tests/develop/teeks99-03e-Ubuntu12-04-64.ht... Tom is usually reading this list, I hope he's reading this.
On Mon, Jun 23, 2014 at 7:06 AM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
On Monday 23 June 2014 13:57:50 Andrzej Krzemienski wrote:
2014-06-23 13:53 GMT+02:00 Andrey Semashev <andrey.semashev@gmail.com>:
Yes, looks like a problem. Maybe the checkout failed at some point for
the
tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If the update scripts ignore such errors then that's what may have happened.
What's the procedure in such cases? Can the owners of the machines/environments /scripts be contacted?
Yes, if the contact information is available in the tester description (the link in the test results table header). E.g. for your tester the information is:
http://www.boost.org/development/tests/develop/teeks99-03e-Ubuntu12-04-64.ht...
Tom is usually reading this list, I hope he's reading this.
I'm not sure what is going on...that commit "c12eaea160fa97a0b6c503eb8f7fdba938f3126f" was only a few hours old, and was the most recent at the time that the run you linked started (4AM). I didn't think it was possible to get a partial checkout, but who knows. Because of space issues on the VMs that are running, I don't keep anything between runs, so I can't actually go and look at the repo that was used for the test in question. The next run that should be complete is teeks99-03g. It looks like it has the correct code at line 1073 (am I reading that correctly?). Once that one comes up, we'll see if everything has caught up. Also, I verified that teeks99-04a just started up and there were no checkout errors that I could see during the start up, so that one might be worth checking in a couple hours as well. Tom
2014-06-23 15:36 GMT+02:00 Tom Kent <lists@teeks99.com>:
On Mon, Jun 23, 2014 at 7:06 AM, Andrey Semashev < andrey.semashev@gmail.com> wrote:
On Monday 23 June 2014 13:57:50 Andrzej Krzemienski wrote:
2014-06-23 13:53 GMT+02:00 Andrey Semashev <andrey.semashev@gmail.com :
Yes, looks like a problem. Maybe the checkout failed at some point
for the
tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If the update scripts ignore such errors then that's what may have happened.
What's the procedure in such cases? Can the owners of the machines/environments /scripts be contacted?
Yes, if the contact information is available in the tester description (the link in the test results table header). E.g. for your tester the information is:
http://www.boost.org/development/tests/develop/teeks99-03e-Ubuntu12-04-64.ht...
Tom is usually reading this list, I hope he's reading this.
I'm not sure what is going on...that commit "c12eaea160fa97a0b6c503eb8f7fdba938f3126f" was only a few hours old, and was the most recent at the time that the run you linked started (4AM).
I didn't think it was possible to get a partial checkout, but who knows. Because of space issues on the VMs that are running, I don't keep anything between runs, so I can't actually go and look at the repo that was used for the test in question.
The next run that should be complete is teeks99-03g. It looks like it has the correct code at line 1073 (am I reading that correctly?). Once that one comes up, we'll see if everything has caught up.
Currently on branch 'develop' when you look at GitHub: https://github.com/boostorg/optional/blob/develop/include/boost/optional/opt... line 1073 is in a middle of an if-statement. Tehere is no ternary operator?: involved when calling function value_or. Latest commit (4cbb67e5059b8771768dd6598670fe723ca50475) was done three days ago.
Also, I verified that teeks99-04a just started up and there were no checkout errors that I could see during the start up, so that one might be worth checking in a couple hours as well.
Ok, I'll do that. Thanks. &rzej
2014-06-23 15:36 GMT+02:00 Tom Kent <lists@teeks99.com>:
On Mon, Jun 23, 2014 at 7:06 AM, Andrey Semashev < andrey.semashev@gmail.com> wrote:
On Monday 23 June 2014 13:57:50 Andrzej Krzemienski wrote:
2014-06-23 13:53 GMT+02:00 Andrey Semashev <andrey.semashev@gmail.com :
Yes, looks like a problem. Maybe the checkout failed at some point
for the
tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If the update scripts ignore such errors then that's what may have happened.
What's the procedure in such cases? Can the owners of the machines/environments /scripts be contacted?
Yes, if the contact information is available in the tester description (the link in the test results table header). E.g. for your tester the information is:
http://www.boost.org/development/tests/develop/teeks99-03e-Ubuntu12-04-64.ht...
Tom is usually reading this list, I hope he's reading this.
I'm not sure what is going on...that commit "c12eaea160fa97a0b6c503eb8f7fdba938f3126f" was only a few hours old, and was the most recent at the time that the run you linked started (4AM).
I didn't think it was possible to get a partial checkout, but who knows. Because of space issues on the VMs that are running, I don't keep anything between runs, so I can't actually go and look at the repo that was used for the test in question.
The next run that should be complete is teeks99-03g. It looks like it has the correct code at line 1073 (am I reading that correctly?). Once that one comes up, we'll see if everything has caught up.
Also, I verified that teeks99-04a just started up and there were no checkout errors that I could see during the start up, so that one might be worth checking in a couple hours as well.
7 days after, teeks99-03g and teeks99-04a still fail and error messages refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional. Regards, &rzej
7 days after, teeks99-03g and teeks99-04a still fail and error messages refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.
Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well. John.
2014-06-30 12:29 GMT+02:00 John Maddock <boost.regex@virgin.net>:
7 days after, teeks99-03g and teeks99-04a still fail and error messages
refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.
Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well.
I am not sure if this is related, but this is the response from Niklas, who is hosting the regression on QNX: The symbolic links in $BOOST_ROOT/boost created by "bjam headers" before
the build starts point to the wrong files:
$ ls -l /extra/boost_regression/x86/boost_root/boost/optional* lrwxrwxrwx 1 boost users 42 Jun 30 07:10 /extra/boost_regression/x86/boost_root/boost/optional.hpp -> .../libs/convert/include/boost/optional.hpp
/extra/boost_regression/x86/boost_root/boost/optional: total 4 lrwxrwxrwx 1 boost users 65 Jun 30 07:10 bad_optional_access.hpp -> .../../libs/convert/include/boost/optional/bad_optional_access.hpp lrwxrwxrwx 1 boost users 54 Jun 30 07:10 optional.hpp -> .../../libs/convert/include/boost/optional/optional.hpp lrwxrwxrwx 1 boost users 58 Jun 30 07:10 optional_fwd.hpp -> .../../libs/convert/include/boost/optional/optional_fwd.hpp lrwxrwxrwx 1 boost users 57 Jun 30 07:10 optional_io.hpp -> .../../libs/convert/include/boost/optional/optional_io.hpp
Can anyone help me understand where it is controlled which links are pointed to which files when I run 'b2 headers'? Regards, &rzej
On Mon, Jun 30, 2014 at 6:20 AM, Andrzej Krzemienski <akrzemi1@gmail.com> wrote:
2014-06-30 12:29 GMT+02:00 John Maddock <boost.regex@virgin.net>:
7 days after, teeks99-03g and teeks99-04a still fail and error messages
refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.
Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well.
I am not sure if this is related, but this is the response from Niklas, who is hosting the regression on QNX:
The symbolic links in $BOOST_ROOT/boost created by "bjam headers" before
the build starts point to the wrong files:
$ ls -l /extra/boost_regression/x86/boost_root/boost/optional* lrwxrwxrwx 1 boost users 42 Jun 30 07:10 /extra/boost_regression/x86/boost_root/boost/optional.hpp -> .../libs/convert/include/boost/optional.hpp
/extra/boost_regression/x86/boost_root/boost/optional: total 4 lrwxrwxrwx 1 boost users 65 Jun 30 07:10 bad_optional_access.hpp -> .../../libs/convert/include/boost/optional/bad_optional_access.hpp lrwxrwxrwx 1 boost users 54 Jun 30 07:10 optional.hpp -> .../../libs/convert/include/boost/optional/optional.hpp lrwxrwxrwx 1 boost users 58 Jun 30 07:10 optional_fwd.hpp -> .../../libs/convert/include/boost/optional/optional_fwd.hpp lrwxrwxrwx 1 boost users 57 Jun 30 07:10 optional_io.hpp -> .../../libs/convert/include/boost/optional/optional_io.hpp
Can anyone help me understand where it is controlled which links are pointed to which files when I run 'b2 headers'?
I show the same thing on mine...all the symlinks under boost point to a copy of the same files that reside in the convert repository: https://github.com/boostorg/convert/tree/develop/include/boost/optional Tom
I show the same thing on mine...all the symlinks under boost point to a copy of the same files that reside in the convert repository: https://github.com/boostorg/convert/tree/develop/include/boost/optional
This is bad, looks like there are two copies of the boost/optional/ tree in Git: * The correct one under optional. * What's described as a "local copy" under convert. But you just can't do that, ever. The convert lib needs to remove it's copies from both develop and master otherwise the tree is completely broken. John.
This is bad, looks like there are two copies of the boost/optional/ tree in Git:
* The correct one under optional. * What's described as a "local copy" under convert.
But you just can't do that, ever. The convert lib needs to remove it's copies from both develop and master otherwise the tree is completely broken.
There are also a bunch of duplicated headers such as boost/none.hpp which have to go too. John.
On 07/01/2014 04:13 AM, John Maddock wrote:
I show the same thing on mine...all the symlinks under boost point to a copy of the same files that reside in the convert repository: https://github.com/boostorg/convert/tree/develop/include/boost/optional
This is bad, looks like there are two copies of the boost/optional/ tree in Git:
* The correct one under optional. * What's described as a "local copy" under convert.
But you just can't do that, ever. The convert lib needs to remove it's copies from both develop and master otherwise the tree is completely broken.
The duplicate "optional" code has been removed from "convert". Apologies. V.
But you just can't do that, ever. The convert lib needs to remove it's copies from both develop and master otherwise the tree is completely broken.
The duplicate "optional" code has been removed from "convert". Apologies.
Thanks! John.
On Mon, Jun 30, 2014 at 5:29 AM, John Maddock <boost.regex@virgin.net> wrote:
7 days after, teeks99-03g and teeks99-04a still fail and error messages
refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.
Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well.
Which files are out of date? All the failures I saw referenced a pch.obj...does that depend on changes to bjam?
7 days after, teeks99-03g and teeks99-04a still fail and error messages
refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.
Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well.
Which files are out of date? All the failures I saw referenced a pch.obj...does that depend on changes to bjam?
It depends on this change to the test Jamfile which disables pch support for now: https://github.com/boostorg/math/commit/b6081dc422b6067c6df28ead64336c1f376c... John.
The runner teeks99-01a-win2008-64on64 isn't using '--remove-test-targets', so it shouldn't be affected by that change? On Mon, Jun 30, 2014 at 11:28 AM, John Maddock <boost.regex@virgin.net> wrote:
7 days after, teeks99-03g and teeks99-04a still fail and error messages
refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.
Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well.
Which files are out of date? All the failures I saw referenced a pch.obj...does that depend on changes to bjam?
It depends on this change to the test Jamfile which disables pch support for now: https://github.com/boostorg/math/commit/ b6081dc422b6067c6df28ead64336c1f376ccb53
John.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/ mailman/listinfo.cgi/boost
The runner teeks99-01a-win2008-64on64 isn't using '--remove-test-targets', so it shouldn't be affected by that change?
Well that's very strange then, the next question then would be how come those object files are missing: http://www.boost.org/development/tests/develop/developer/output/teeks99-01a-... Thanks for looking into this, John.
participants (5)
-
Andrey Semashev -
Andrzej Krzemienski -
John Maddock -
Tom Kent -
Vladimir Batov