Boost.Atomic, pushing forward and a little request for help
Hi folks, from the comments to Boost.Lockfree I understand that you consider a non-reviewed Boost.Atomic as a "blocker"? I apologize that I have not pushed forward for review (considering that "it just works for me", and considering the length of the review queue...), but if you consider it blocking please just let me know what I would need to do to clear the path. What I will do shortly is to provide and uptodate package containing several fixes, several contributed by others (they are in the repo, but not in any release archive) as well as some missing bits to more closely match latest C++0X draft. The big sore point is still support for the various flavours of VC -- I know that there are (especially older) versions for which it is not working, but I am lacking test systems, so if you are affected please let me know and I will do my best to resolve the issues. Cheers, Helge
On 8/1/2011 4:56 PM, Helge Bahmann wrote:
Hi folks,
from the comments to Boost.Lockfree I understand that you consider a non-reviewed Boost.Atomic as a "blocker"? I apologize that I have not pushed forward for review (considering that "it just works for me", and considering the length of the review queue...), but if you consider it blocking please just let me know what I would need to do to clear the path.
What I will do shortly is to provide and uptodate package containing several fixes, several contributed by others (they are in the repo, but not in any release archive) as well as some missing bits to more closely match latest C++0X draft.
The big sore point is still support for the various flavours of VC -- I know that there are (especially older) versions for which it is not working, but I am lacking test systems, so if you are affected please let me know and I will do my best to resolve the issues.
My own point of view when doing new development for VC++ and Boost is that it is simply not worth worrying about versions that are older than VC++ 8 ( VS2005 ). Others may disagree, but I have run into enough compiler bugs in the last three releases to not have to worry about any problems which might occur in releases before those. So my opinion is not to worry unduly about any releases prior to VC++ 8 for whatever you do in your development of a new library unless you are pretty sure that a fix for VC++ 7 or VC++ 7.1 is pretty trivial ( I don't even consider VC++ 6 worth discussing ). This reply is also an encouragement for you to update Boost.Atomic, let others test it with compilers which you may not have, and get it working as best you can.
On 1:59 PM, Edward Diener wrote:
On 8/1/2011 4:56 PM, Helge Bahmann wrote:
Hi folks,
....
The big sore point is still support for the various flavours of VC -- I know that there are (especially older) versions for which it is not working, but I am lacking test systems, so if you are affected please let me know and I will do my best to resolve the issues.
My own point of view when doing new development for VC++ and Boost is that it is simply not worth worrying about versions that are older than VC++ 8 ( VS2005 ). Others may disagree, but I have run into enough compiler bugs in the last three releases to not have to worry about any problems which might occur in releases before those. So my opinion is not to worry unduly about any releases prior to VC++ 8 for whatever you do in your development of a new library unless you are pretty sure that a fix for VC++ 7 or VC++ 7.1 is pretty trivial ( I don't even consider VC++ 6 worth discussing ).
If it's just a matter of doing an SVN update, running bjam, and e-mailing you the results, I can help a little (VS2005/MSVC8). Shoot me an e-mail off list (I don't read this list often enough).
On Aug 1, 2011, at 4:56 PM, Helge Bahmann wrote:
I apologize that I have not pushed forward for review (considering that "it just works for me", and considering the length of the review queue...), but if you consider it blocking please just let me know what I would need to do to clear the path.
It isn't really a queue and from the schedule it looks like there might be time as early as October if you can find a review manager. (Myself not qualified sorry.) Cheers, Gordon
Helge Bahmann wrote:
Hi folks,
from the comments to Boost.Lockfree I understand that you consider a non-reviewed Boost.Atomic as a "blocker"? I apologize that I have not pushed forward for review (considering that "it just works for me", and considering the length of the review queue...)
It's not really a queue, and I believe Atomic should jump in at the front.
, but if you consider it blocking please just let me know what I would need to do to clear the path.
Get a review manager.
What I will do shortly is to provide and uptodate package containing several fixes, several contributed by others (they are in the repo, but not in any release archive) as well as some missing bits to more closely match latest C++0X draft.
The big sore point is still support for the various flavours of VC -- I know that there are (especially older) versions for which it is not working, but I am lacking test systems, so if you are affected please let me know and I will do my best to resolve the issues.
Sorry, I can't help with VC. The thing that I believe is missing is tests. Do you have any thoughts about that? Regards, Phil.
, but if you consider it blocking please just let me know what I would need to do to clear the path.
Get a review manager.
would one of the review managers be interested in managing the review? or what are the requirements for a possible review manager? i guess, i would not really qualify as i am biased? tim --------------------------------------------------------------- diese nachricht wurde ueber klingt.org gesendet this message was sent through klingt.org's webmail.
Tim Blechmann wrote:
, but if you consider it blocking please just let me know what I would need to do to clear the path.
Get a review manager.
would one of the review managers be interested in managing the review? or what are the requirements for a possible review manager? i guess, i would not really qualify as i am biased?
You mean review wizards? Everyone wants atomic to be accepted, so we are all biased. What the review is going to do is determine if it is ready for release, and if not, what needs to be done before it is released. No matter how much you want the library to be accepted, you still won't accept it with glaring flaws that would hurt your own library that depends on it. The review will help bring these flaws to the surface so that we can fix them. Even after the library is released the community will provide feedback and patches to get support for more platforms added in. It is important to get the library out there so that we can leverage the wider user community to improve it, we just need to do that in a responsible way so that it at least works well on several of the dominant platforms so that at least the majority of users will benefit rather than be harmed by attempting to use it. As review manager that responsibility would be yours. Regards, Luke
On 8/2/2011 12:28 PM, Phil Endecott wrote:
The thing that I believe is missing is tests. Do you have any thoughts about that?
I agree some basic thread safety tests are good to at least suggest that things are working as expected. It's impossible to guarantee it. The effectiveness of such non-deterministic tests depends on the hardware concurrency of the testing host (and also what other processes it's running). My basic thread safety testing technique is: void testThreadSafetyWorker(boost::barrier* testBarrier) { testBarrier->wait(); // wait until all threads have started try { // run tests using shared atomics } catch (exception& e) { cerr << "Exception in worker thread: " << e.what() << endl; } catch (...) { cerr << "Unhandled exception in worker thread." << endl; } } void testThreadSafety(const int& testThreadCount) { boost::barrier testBarrier(testThreadCount); boost::thread_group testThreadGroup; for (int i=0; i < testThreadCount; ++i) testThreadGroup.add_thread(new boost::thread(&testThreadSafetyWorker, &testBarrier)); testThreadGroup.join_all(); } I try it for 2, 4, 6, 8 threads. Hope this helps, -Matt
On Tuesday 02 August 2011 19:28:08 Phil Endecott wrote:
Helge Bahmann wrote:
Hi folks,
from the comments to Boost.Lockfree I understand that you consider a non-reviewed Boost.Atomic as a "blocker"? I apologize that I have not pushed forward for review (considering that "it just works for me", and considering the length of the review queue...)
It's not really a queue, and I believe Atomic should jump in at the front.
, but if you consider it blocking please just let me know what I would need to do to clear the path.
Get a review manager.
What I will do shortly is to provide and uptodate package containing several fixes, several contributed by others (they are in the repo, but not in any release archive) as well as some missing bits to more closely match latest C++0X draft.
The big sore point is still support for the various flavours of VC -- I know that there are (especially older) versions for which it is not working, but I am lacking test systems, so if you are affected please let me know and I will do my best to resolve the issues.
Sorry, I can't help with VC.
The thing that I believe is missing is tests. Do you have any thoughts about that?
Well, I have a few simple tests for atomicity only, they are hand-built and I am working to push them into the boost build system. I don't know how useful the tests are though -- while on a true SMP platform (ppc and x86 being available to me) they manage to fail quite realibly if I intentionally break something (like removing "lock" prefixes, or replacing stwcx. with an unconditional store), but it takes ages (if it works at all) to produce test failures on non-SMP (all threads to the same CPU). I have not yet looked into writing something for memory ordering tests, but arguably that's both even more important and more difficult to reproduce failures. The other aspect that I want to provide is compile-time test for API completeness. I suspect there are a few small things missing still. Best regards Helge
Helge Bahmann wrote:
On Tuesday 02 August 2011 19:28:08 Phil Endecott wrote:
The thing that I believe is missing is tests. Do you have any thoughts about that?
Well, I have a few simple tests for atomicity only, they are hand-built and I am working to push them into the boost build system.
I don't know how useful the tests are though -- while on a true SMP platform (ppc and x86 being available to me) they manage to fail quite realibly if I intentionally break something (like removing "lock" prefixes, or replacing stwcx. with an unconditional store), but it takes ages (if it works at all) to produce test failures on non-SMP (all threads to the same CPU).
I recall someone once increasing the kernel's scheduling frequency to make this work better. On Linux, I think this means changing HZ and recompiling the kernel. Perhaps there is some trick that could be used to force the kernel to reschedule more frequently without going to those lengths. Or maybe it is easier on some other platform. Even so, scheduling only happens between instructions, not between memory accesses. When a single instruction does more than one memory access (i.e. a typical CISC instruction), process scheduling will never occur between the two memory accesses. I think it is probably safe to test only on multi-core systems, and to assume that if code works on them then it will also work on single-core systems.
I have not yet looked into writing something for memory ordering tests, but arguably that's both even more important and more difficult to reproduce failures.
I agree. I have no idea where to start on that. Hopefully there is someone else reading with a clue....
The other aspect that I want to provide is compile-time test for API completeness. I suspect there are a few small things missing still.
Yes. Also, it would be very easy to make a copy&paste error and find that something is doing an atomic add instead of an atomic and. Or that it was only changing the least-significant byte, or something. So some basic "does everything compile (and assemble)?" and "do I get the same answer as with non-atomics?" tests would be good. Also, I think it would be useful to verify at compile time that the expected atomic code is being used, and not the fallback (this is something that has bitten me). In fact, it might be worth putting in a #warning for the fallback implementation that the user has to explicitly disable. Regards, Phil.
On Wed, Aug 3, 2011 at 11:52 AM, Phil Endecott <spam_from_boost_dev@chezphil.org> wrote:
Helge Bahmann wrote:
On Tuesday 02 August 2011 19:28:08 Phil Endecott wrote:
The thing that I believe is missing is tests. Do you have any thoughts about that?
Well, I have a few simple tests for atomicity only, they are hand-built and I am working to push them into the boost build system.
I don't know how useful the tests are though -- while on a true SMP platform (ppc and x86 being available to me) they manage to fail quite realibly if I intentionally break something (like removing "lock" prefixes, or replacing stwcx. with an unconditional store), but it takes ages (if it works at all) to produce test failures on non-SMP (all threads to the same CPU).
I recall someone once increasing the kernel's scheduling frequency to make this work better. On Linux, I think this means changing HZ and recompiling the kernel. Perhaps there is some trick that could be used to force the kernel to reschedule more frequently without going to those lengths. Or maybe it is easier on some other platform.
Even so, scheduling only happens between instructions, not between memory accesses. When a single instruction does more than one memory access (i.e. a typical CISC instruction), process scheduling will never occur between the two memory accesses.
Maybe we should look at using Relacy (google for it) for tests. Tony
Gottlob Frege wrote:
On Wed, Aug 3, 2011 at 11:52 AM, Phil Endecott <spam_from_boost_dev@chezphil.org> wrote:
Helge Bahmann wrote:
On Tuesday 02 August 2011 19:28:08 Phil Endecott wrote:
The thing that I believe is missing is tests. ?Do you have any thoughts about that?
Well, I have a few simple tests for atomicity only, they are hand-built and I am working to push them into the boost build system.
I don't know how useful the tests are though -- while on a true SMP platform (ppc and x86 being available to me) they manage to fail quite realibly if I intentionally break something (like removing "lock" prefixes, or replacing stwcx. with an unconditional store), but it takes ages (if it works at all) to produce test failures on non-SMP (all threads to the same CPU).
I recall someone once increasing the kernel's scheduling frequency to make this work better. ?On Linux, I think this means changing HZ and recompiling the kernel. ?Perhaps there is some trick that could be used to force the kernel to reschedule more frequently without going to those lengths. ?Or maybe it is easier on some other platform.
Even so, scheduling only happens between instructions, not between memory accesses. ?When a single instruction does more than one memory access (i.e. a typical CISC instruction), process scheduling will never occur between the two memory accesses.
Maybe we should look at using Relacy (google for it) for tests.
http://www.1024cores.net/home/relacy-race-detector Interesting. Have you used it? Is it really useful for testing the atomic implementation, rather than testing application code that is built on top of it? I once tried to use Helgrind, which is a Valgrind race-detection tool. I didn't get very far. Regards, Phil.
On Sat, Aug 6, 2011 at 10:35 AM, Phil Endecott <spam_from_boost_dev@chezphil.org> wrote:
Gottlob Frege wrote:
Maybe we should look at using Relacy (google for it) for tests.
http://www.1024cores.net/home/relacy-race-detector
Interesting. Have you used it? Is it really useful for testing the atomic implementation, rather than testing application code that is built on top of it?
I once tried to use Helgrind, which is a Valgrind race-detection tool. I didn't get very far.
I haven't actually used it, but I've heard good things about it. You might be write about implementation vs application. But I think it is at least worth a look. Tony
On Saturday 06 August 2011 07:40:02 Gottlob Frege wrote:
On Wed, Aug 3, 2011 at 11:52 AM, Phil Endecott
<spam_from_boost_dev@chezphil.org> wrote:
Helge Bahmann wrote:
On Tuesday 02 August 2011 19:28:08 Phil Endecott wrote:
The thing that I believe is missing is tests. Do you have any thoughts about that?
Well, I have a few simple tests for atomicity only, they are hand-built and I am working to push them into the boost build system.
I don't know how useful the tests are though -- while on a true SMP platform (ppc and x86 being available to me) they manage to fail quite realibly if I intentionally break something (like removing "lock" prefixes, or replacing stwcx. with an unconditional store), but it takes ages (if it works at all) to produce test failures on non-SMP (all threads to the same CPU).
I recall someone once increasing the kernel's scheduling frequency to make this work better. On Linux, I think this means changing HZ and recompiling the kernel. Perhaps there is some trick that could be used to force the kernel to reschedule more frequently without going to those lengths. Or maybe it is easier on some other platform.
Even so, scheduling only happens between instructions, not between memory accesses. When a single instruction does more than one memory access (i.e. a typical CISC instruction), process scheduling will never occur between the two memory accesses.
Maybe we should look at using Relacy (google for it) for tests.
It is only able to aid in exercising things that use the atomic interface, not the implementation itself. Regards Helge
On Aug 7, 2011 1:41 PM, "Helge Bahmann" <hcb@chaoticmind.net> wrote:
On Saturday 06 August 2011 07:40:02 Gottlob Frege wrote:
On Wed, Aug 3, 2011 at 11:52 AM, Phil Endecott
<spam_from_boost_dev@chezphil.org> wrote:
Helge Bahmann wrote:
On Tuesday 02 August 2011 19:28:08 Phil Endecott wrote:
The thing that I believe is missing is tests. Do you have any
thoughts
about that?
Well, I have a few simple tests for atomicity only, they are hand-built and I am working to push them into the boost build system.
I don't know how useful the tests are though -- while on a true SMP platform (ppc and x86 being available to me) they manage to fail quite realibly if I intentionally break something (like removing "lock" prefixes, or replacing stwcx. with an unconditional store), but it takes ages (if it works at all) to produce test failures on non-SMP (all threads to the same CPU).
I recall someone once increasing the kernel's scheduling frequency to make this work better. On Linux, I think this means changing HZ and recompiling the kernel. Perhaps there is some trick that could be used to force the kernel to reschedule more frequently without going to those lengths. Or maybe it is easier on some other platform.
Even so, scheduling only happens between instructions, not between memory accesses. When a single instruction does more than one memory access (i.e. a typical CISC instruction), process scheduling will never occur between the two memory accesses.
Maybe we should look at using Relacy (google for it) for tests.
It is only able to aid in exercising things that use the atomic interface, not the implementation itself.
How will c++1x implementors test it?
participants (10)
-
Edward Diener -
Gordon Woodhull -
Gottlob Frege -
Helge Bahmann -
Jim Bell -
Matt Chambers -
Matthew Chambers
-
Phil Endecott -
Simonson, Lucanus J -
Tim Blechmann