RE: [boost] [Boost.Test] New testing procedure?
According to http://tinyurl.com/3f32c tests have been failing (IIUC for several days if not longer) because of protected/private access violations in the test library. See the end of http://tinyurl.com/35ole for details. IMO this situation is unacceptable, and I'm looking for practical solutions. Problems with the Boost.Test affect our ability to get meaningful feedback on the state of other libraries.
If Gennadiy can't test on more compilers before checking in, and respond faster to problems introduced in its source, and if library authors genuinely find Boost.Test useful, maybe we need to move to a different model wherein the test library's own tests are run on a CVS branch of the code (?) so that Gennadiy can see and deal with his problems before they are merged into the main trunk and break everything else?
I do test on as much compilers I have accessible. In this particular case It was working on all compilers but Borland. I posted request for workaround while ago. Still no response. I had to test other configuration and I committed it. Actually I was surprised that so many compilers had an issue with completely innocent using declaration. Anyway I think it should be fixed as of last night. I found some hack that should work on complaining compilers. I will see the results of regression test today. As for creating separate brunch for Boost.Test development, I do not really mind. But I believe it will create an extra headache for regression testers (and me). Essentially we will need to have two copies of development tree. And run Boost.Test unit test in a separate tree. I will need to keep moving files back and forth 2 branches. Also I wonder how it will interact with release procedure. You may noticed that I am trying to introduce modification in Boost.Test in "packages", meaning I do not d code modifications all the time. I am not sure that several days of "no regression test on some compilers", worth all that trouble. Especially since all the developers could always rollback Boost.Test modification locally for development testing. Regards, Gennadiy.
"Rozental, Gennadiy" <gennadiy.rozental@thomson.com> writes:
According to http://tinyurl.com/3f32c tests have been failing (IIUC for several days if not longer) because of protected/private access violations in the test library. See the end of http://tinyurl.com/35ole for details. IMO this situation is unacceptable, and I'm looking for practical solutions. Problems with the Boost.Test affect our ability to get meaningful feedback on the state of other libraries.
If Gennadiy can't test on more compilers before checking in, and respond faster to problems introduced in its source, and if library authors genuinely find Boost.Test useful, maybe we need to move to a different model wherein the test library's own tests are run on a CVS branch of the code (?) so that Gennadiy can see and deal with his problems before they are merged into the main trunk and break everything else?
I do test on as much compilers I have accessible.
I'm sure you do. That hasn't stopped problems in Boost.Test from becoming a problem for the rest of Boost (several times at a critical moment for a release), so as I said I'm looking for practical solutions.
In this particular case It was working on all compilers but Borland.
??! The links I posted were for problems uncovered by Comeau. Did you look at http://tinyurl.com/35ole? Several other compilers also detect the issue.
I posted request for workaround while ago. Still no response.
I could be wrong, but AFAICT there's a bug in the library. It shouldn't be up to others to find workarounds for that.
I had to test other configuration and I committed it. Actually I was surprised that so many compilers had an issue with completely innocent using declaration. Anyway I think it should be fixed as of last night.
Not according to the links I posted *this morning*.
I found some hack that should work on complaining compilers. I will see the results of regression test today. As for creating separate brunch for Boost.Test development, I do not really mind. But I believe it will create an extra headache for regression testers (and me).
Not if it's automated.
Essentially we will need to have two copies of development tree.
I doubt it. How many libraries does Boost.Test depend on? Are you sure we can't just do this with branched copies of boost/test and libs/test?
And run Boost.Test unit test in a separate tree. I will need to keep moving files back and forth 2 branches.
Better to take the effort to be responsible for not breaking things than to develop quickly and dangerously when your library is a correctness bottleneck for so many others.
Also I wonder how it will interact with release procedure. You may noticed that I am trying to introduce modification in Boost.Test in "packages", meaning I do not d code modifications all the time. I am not sure that several days of "no regression test on some compilers", worth all that trouble.
I think you underestimate the problems it causes, and probably also how long the particular problem in question has existed in the codebase (I think it's been over a week). Many people have been talking about trying to shorten the time it takes to get test results for any particular library. To go from 12-24 hours to "several days" is probably the wrong direction.
Especially since all the developers could always rollback Boost.Test modification locally for development testing.
It makes a lot more sense to me that the author of one library should do a small amount of extra work to stay out of the way of everyone else than to make all the other library authors adapt to Boost.Test's problems. More importantly, that's not even possible for the test results library authors want to get from compilers they don't own. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
In this particular case It was working on all compilers but Borland.
??! The links I posted were for problems uncovered by Comeau. Did you look at http://tinyurl.com/35ole? Several other compilers also detect the issue.
I do not have cameau not inter compilers. Among compilers I have access to it failed on command line Borland only.
I posted request for workaround while ago. Still no response.
I could be wrong, but AFAICT there's a bug in the library. It shouldn't be up to others to find workarounds for that.
Hardly. I am quite confident library code is correct.
I had to test other configuration and I committed it. Actually I was surprised that so many compilers had an issue with completely innocent using declaration. Anyway I think it should be fixed as of last night.
Not according to the links I posted *this morning*.
There may be some differences between metacom and mine "last night". Check now. I do not know what is the version of intel compiler and couldn't switch to hack workaround for it. BTW I remember that for non-metacom formats it is possible to see the runtime result of config test. It is very convenient. Could we make metacom format to include it?
I found some hack that should work on complaining compilers. I will see the results of regression test today. As for creating separate brunch for Boost.Test development, I do not really mind. But I believe it will create an extra headache for regression testers (and me).
Not if it's automated.
What you intend to automate?
Essentially we will need to have two copies of development tree.
I doubt it. How many libraries does Boost.Test depend on? Are you sure we can't just do this with branched copies of boost/test and libs/test?
I am quite sure, that it would be much easier to have separate development tree than to support subset needed for Boost.Test. It's much bigger that you imagine. And it's growing.
And run Boost.Test unit test in a separate tree. I will need to keep moving files back and forth 2 branches.
Better to take the effort to be responsible for not breaking things than to develop quickly and dangerously when your library is a correctness bottleneck for so many others.
Do you have any grounds to say that I develop quickly of dangerously? I was working on this modifications for more than a two month before committing it.
Also I wonder how it will interact with release procedure. You may noticed that I am trying to introduce modification in Boost.Test in "packages", meaning I do not do code modifications all the time. I am not sure that several days of "no regression test on some compilers", worth all that trouble.
I think you underestimate the problems it causes, and probably also how long the particular problem in question has existed in the codebase (I think it's been over a week). Many people have been talking about trying to shorten the time it takes to get test results for any particular library. To go from 12-24 hours to "several days" is probably the wrong direction.
Once I start "modification period" I am fixing library issues ASAP. It takes longer to figure out how to find workaround for compiler deficiency, especially since I do not have an access to it. Gennadiy. P.S. Could guys who support metacom development tree update it with -d. There is directory missing under libs/test/test
"Gennadiy Rozental" <gennadiy.rozental@thomson.com> writes:
In this particular case It was working on all compilers but Borland.
??! The links I posted were for problems uncovered by Comeau. Did you look at http://tinyurl.com/35ole? Several other compilers also detect the issue.
I do not have cameau not inter compilers. Among compilers I have access to it failed on command line Borland only.
Oh, I'm sorry. I didn't realize you were referring only to compilers you had tested on.
I posted request for workaround while ago. Still no response.
I could be wrong, but AFAICT there's a bug in the library. It shouldn't be up to others to find workarounds for that.
Hardly. I am quite confident library code is correct.
OK, having looked, I am inclined to agree. At least the standard strongly implies that derived classes can increase access to base class protected members by use of a using declaration.
I had to test other configuration and I committed it. Actually I was surprised that so many compilers had an issue with completely innocent using declaration. Anyway I think it should be fixed as of last night.
Not according to the links I posted *this morning*.
The dates on the test were from the 21st, but...
There may be some differences between metacom and mine "last night". Check now.
...Comeau seems to be cured now.
I do not know what is the version of intel compiler and couldn't switch to hack workaround for it.
I think the simplest answer for compilers that can't handle your using declaration correctly is just to make the member public. Misuses will still show up with other compilers.
BTW I remember that for non-metacom formats it is possible to see the runtime result of config test. It is very convenient. Could we make metacom format to include it?
I think you'd better post a specific question about that to make sure those guys see it.
I found some hack that should work on complaining compilers. I will see the results of regression test today. As for creating separate brunch for Boost.Test development, I do not really mind. But I believe it will create an extra headache for regression testers (and me).
Not if it's automated.
What you intend to automate?
Checking out the right branched material and starting the tests.
Essentially we will need to have two copies of development tree.
I doubt it. How many libraries does Boost.Test depend on? Are you sure we can't just do this with branched copies of boost/test and libs/test?
I am quite sure, that it would be much easier to have separate development tree than to support subset needed for Boost.Test. It's much bigger that you imagine. And it's growing.
Even so, you would be developing a branch of your code against the current CVS state of the rest of Boost, right? I still don't see why a 2nd copy of all of Boost should be needed.
And run Boost.Test unit test in a separate tree. I will need to keep moving files back and forth 2 branches.
Better to take the effort to be responsible for not breaking things than to develop quickly and dangerously when your library is a correctness bottleneck for so many others.
Do you have any grounds to say that I develop quickly of dangerously?
I'm sorry, I didn't mean to imply that. I meant that when so many other libraries' tests are dependent on yours, any substantial change you make that can't be widely tested beforehand is dangerous. It's better to move slowly if that's what it takes.
I was working on this modifications for more than a two month before committing it.
At least the timing is OK; we can afford a little breakage now, before we get into a new release cycle.
Also I wonder how it will interact with release procedure. You may noticed that I am trying to introduce modification in Boost.Test in "packages", meaning I do not do code modifications all the time. I am not sure that several days of "no regression test on some compilers", worth all that trouble.
I think you underestimate the problems it causes, and probably also how long the particular problem in question has existed in the codebase (I think it's been over a week). Many people have been talking about trying to shorten the time it takes to get test results for any particular library. To go from 12-24 hours to "several days" is probably the wrong direction.
Once I start "modification period" I am fixing library issues ASAP. It takes longer to figure out how to find workaround for compiler deficiency, especially since I do not have an access to it.
Sure thing.
Gennadiy.
P.S. Could guys who support metacom development tree update it with -d. There is directory missing under libs/test/test
Again, a separate posting or direct email might be advisable. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
Gennadiy Rozental writes:
There may be some differences between metacom and mine "last night". Check now. I do not know what is the version of intel compiler
It's encoded in the toolset name -- intel-win32-7.1-vc6. ^^^
and couldn't switch to hack workaround for it.
BTW I remember that for non-metacom formats it is possible to see the runtime result of config test. It is very convenient. Could we make metacom format to include it?
Put on the TODO list.
I found some hack that should work on complaining compilers. I will see the results of regression test today. As for creating separate brunch for Boost.Test development, I do not really mind. But I believe it will create an extra headache for regression testers (and me).
Not if it's automated.
What you intend to automate?
Essentially we will need to have two copies of development tree.
I doubt it. How many libraries does Boost.Test depend on? Are you sure we can't just do this with branched copies of boost/test and libs/test?
I am quite sure, that it would be much easier to have separate development tree than to support subset needed for Boost.Test. It's much bigger that you imagine. And it's growing.
For the purpose of testing, it doesn't matter whether you have the whole tree on the branch, or just a subset of it. As long as it's only the test and config libraries that are being tested on that branch, we can afford it. [...]
P.S. Could guys who support metacom development tree update it with -d. There is directory missing under libs/test/test
We remove all the sources and do a full checkout on every cycle, so whatever problems you might have, they are not caused by a missing directory, if the latter is in the CVS, of course. -- Aleksey Gurtovoy MetaCommunications Engineering
There may be some differences between metacom and mine "last night". Check now. I do not know what is the version of intel compiler
It's encoded in the toolset name -- intel-win32-7.1-vc6. ^^^
It not very useful for BOOST_WORKAROUND statement
P.S. Could guys who support metacom development tree update it with -d. There is directory missing under libs/test/test
We remove all the sources and do a full checkout on every cycle, so whatever problems you might have, they are not caused by a missing directory, if the latter is in the CVS, of course.
I donno. It present in cvs and errors indicate that it missing. My local tests passing. Gennadiy.
Gennadiy Rozental writes:
There may be some differences between metacom and mine "last night". Check now. I do not know what is the version of intel compiler
It's encoded in the toolset name -- intel-win32-7.1-vc6. ^^^
It not very useful for BOOST_WORKAROUND statement
P.S. Could guys who support metacom development tree update it with -d. There is directory missing under libs/test/test
We remove all the sources and do a full checkout on every cycle, so whatever problems you might have, they are not caused by a missing directory, if the latter is in the CVS, of course.
I donno. It present in cvs and errors indicate that it missing.
A link? -- Aleksey Gurtovoy MetaCommunications Engineering
P.S. Could guys who support metacom development tree update it with -d. There is directory missing under libs/test/test
We remove all the sources and do a full checkout on every cycle, so whatever problems you might have, they are not caused by a missing directory, if the latter is in the CVS, of course.
I donno. It present in cvs and errors indicate that it missing.
A link?
See here http://tinyurl.com/287uk and here http://tinyurl.com/3ckkc Gennadiy.
Gennadiy Rozental writes:
P.S. Could guys who support metacom development tree update it with -d. There is directory missing under libs/test/test
We remove all the sources and do a full checkout on every cycle, so whatever problems you might have, they are not caused by a missing directory, if the latter is in the CVS, of course.
I donno. It present in cvs and errors indicate that it missing.
A link?
See here
and here
There is nothing about a missing directory in there. The failures say ../libs/test/test/output_test_stream_test.cpp(55): error in "test_constructor": output.match_pattern() failed. Couldn't open pattern file for writing There might be a number of reasons for it -- one, for instance, is that the tests incorrectly assume that the current directory is set to "../libs/test/test". -- Aleksey Gurtovoy MetaCommunications Engineering
There is nothing about a missing directory in there. The failures say
../libs/test/test/output_test_stream_test.cpp(55): error in "test_constructor": output.match_pattern() failed. Couldn't open pattern file for writing
There might be a number of reasons for it -- one, for instance, is that the tests incorrectly assume that the current directory is set to "../libs/test/test".
Isn't it true? How would it work otherwise. My other tests rely on pattern files that used to be present in ../libs/test/test (now under ../libs/test/test/test_files) and it was working Gennadiy.
Aleksey Gurtovoy <agurtovoy@meta-comm.com> writes:
Gennadiy Rozental writes:
P.S. Could guys who support metacom development tree update it with -d. There is directory missing under libs/test/test
We remove all the sources and do a full checkout on every cycle, so whatever problems you might have, they are not caused by a missing directory, if the latter is in the CVS, of course.
I donno. It present in cvs and errors indicate that it missing.
A link?
See here
and here
There is nothing about a missing directory in there. The failures say
FYI, if there are no files in that directory of the repository, cvs will generally not create the directory on update/checkout. Not sure what's happening here, but you might just have to add a dummy file to ensure it gets created. That said, Boost tests should be able to be run from a read-only CVS tree, so if you're creating files you probably ought to do it elsewhere (like /tmp/...). -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
participants (4)
-
Aleksey Gurtovoy -
David Abrahams -
Gennadiy Rozental -
Rozental, Gennadiy