- Boost - lists.boost.org

[RFC][Patch][Boost.context] Allowing alternative context switch API
by microcai 09 Dec '14

09 Dec '14

Hi Oliver Kowalke, and others last weak, I report a bug that boost.context causes LoadLibrary to fail. And Oliver Kowalke response that might be because boost.context didn't swap all TIB parts. Allowing alternative context switch API other than reverse-engineered boost.context might solve the problem. So I speed some time to write the patch. users that wish to use Win32 API shoul define BOOST_COROUTINE_USE_FIBER then users should be responsibe to make sure all threads that might call coroutine must be converted to fiber first. since IsThreadAFIber() is not available on some windows platform. if the user can make 100% sure that the platform he/she wish to support has IsThreadAFIber(), then define BOOST_COROUTINE_USE_IS_THREAD_A_FIBER and boost.coroutine will take care of converting to fiber and converting back to thread. Best regards. --- >From 11668b6641c877a4ea154d51ad2171483eced921 Mon Sep 17 00:00:00 2001 From: microcai <microcaicai(a)gmail.com> Date: Sat, 6 Dec 2014 15:50:46 +0800 Subject: [PATCH] allow Win32 Fiber as alternative context switch API windows has some undocumented TIB parts, as boost.context can't keep 100% compatibility with all windows versions, and coroutine is essential to many building blocks. It has to be stable, to allowing alternative context switch API can be more use full. --- .../boost/coroutine/detail/coroutine_context.hpp | 11 ++++ .../coroutine/detail/pull_coroutine_object.hpp | 9 ++++ .../coroutine/detail/push_coroutine_object.hpp | 9 ++++ include/boost/coroutine/stack_allocator.hpp | 5 ++ .../coroutine/windows/null_stack_allocator.hpp | 53 +++++++++++++++++++ src/detail/coroutine_context.cpp | 60 ++++++++++++++++++++-- 6 files changed, 142 insertions(+), 5 deletions(-) create mode 100644 include/boost/coroutine/windows/null_stack_allocator.hpp diff --git a/include/boost/coroutine/detail/coroutine_context.hpp b/include/boost/coroutine/detail/coroutine_context.hpp index 6c4ebb6..f6a53e2 100644 --- a/include/boost/coroutine/detail/coroutine_context.hpp +++ b/include/boost/coroutine/detail/coroutine_context.hpp @@ -16,6 +16,9 @@ #include <boost/coroutine/detail/config.hpp> #include <boost/coroutine/stack_context.hpp> +#ifdef BOOST_COROUTINE_USE_FIBER +# include <windows.h> +#endif #ifdef BOOST_HAS_ABI_HEADERS # include BOOST_ABI_PREFIX #endif @@ -31,6 +34,12 @@ class BOOST_COROUTINES_DECL coroutine_context private: stack_context stack_ctx_; context::fcontext_t ctx_; +#ifdef BOOST_COROUTINE_USE_FIBER + void (*fn_)(intptr_t); + intptr_t param_; + LPVOID fiber_; + static VOID WINAPI fb_start_proc(LPVOID lpFiberParameter); +#endif // BOOST_COROUTINE_USE_FIBER public: typedef void( * ctx_fn)( intptr_t); @@ -51,6 +60,8 @@ public: stack_context & stack_ctx() { return stack_ctx_; } + + void destory(); }; }}} diff --git a/include/boost/coroutine/detail/pull_coroutine_object.hpp b/include/boost/coroutine/detail/pull_coroutine_object.hpp index cfe34bd..d15d94f 100644 --- a/include/boost/coroutine/detail/pull_coroutine_object.hpp +++ b/include/boost/coroutine/detail/pull_coroutine_object.hpp @@ -65,6 +65,9 @@ private: stack_context stack_ctx( obj->stack_ctx_); StackAllocator stack_alloc( obj->stack_alloc_); obj->unwind_stack(); +#ifdef BOOST_COROUTINE_USE_FIBER + obj->callee.destory(); +#endif obj->~obj_t(); stack_alloc.deallocate( stack_ctx); } @@ -153,6 +156,9 @@ private: stack_context stack_ctx( obj->stack_ctx_); StackAllocator stack_alloc( obj->stack_alloc_); obj->unwind_stack(); +#ifdef BOOST_COROUTINE_USE_FIBER + obj->callee.destory(); +#endif obj->~obj_t(); stack_alloc.deallocate( stack_ctx); } @@ -241,6 +247,9 @@ private: stack_context stack_ctx( obj->stack_ctx_); StackAllocator stack_alloc( obj->stack_alloc_); obj->unwind_stack(); +#ifdef BOOST_COROUTINE_USE_FIBER + obj->callee.destory(); +#endif obj->~obj_t(); stack_alloc.deallocate( stack_ctx); } diff --git a/include/boost/coroutine/detail/push_coroutine_object.hpp b/include/boost/coroutine/detail/push_coroutine_object.hpp index 1a39c07..b41a3b2 100644 --- a/include/boost/coroutine/detail/push_coroutine_object.hpp +++ b/include/boost/coroutine/detail/push_coroutine_object.hpp @@ -77,6 +77,9 @@ private: stack_context stack_ctx( obj->stack_ctx_); StackAllocator stack_alloc( obj->stack_alloc_); obj->unwind_stack(); +#ifdef BOOST_COROUTINE_USE_FIBER + obj->callee.destory(); +#endif obj->~obj_t(); stack_alloc.deallocate( stack_ctx); } @@ -165,6 +168,9 @@ private: stack_context stack_ctx( obj->stack_ctx_); StackAllocator stack_alloc( obj->stack_alloc_); obj->unwind_stack(); +#ifdef BOOST_COROUTINE_USE_FIBER + obj->callee.destory(); +#endif obj->~obj_t(); stack_alloc.deallocate( stack_ctx); } @@ -253,6 +259,9 @@ private: stack_context stack_ctx( obj->stack_ctx_); StackAllocator stack_alloc( obj->stack_alloc_); obj->unwind_stack(); +#ifdef BOOST_COROUTINE_USE_FIBER + obj->callee.destory(); +#endif obj->~obj_t(); stack_alloc.deallocate( stack_ctx); } diff --git a/include/boost/coroutine/stack_allocator.hpp b/include/boost/coroutine/stack_allocator.hpp index 662533e..811f352 100644 --- a/include/boost/coroutine/stack_allocator.hpp +++ b/include/boost/coroutine/stack_allocator.hpp @@ -14,6 +14,9 @@ #include <boost/context/detail/config.hpp> #include <boost/coroutine/segmented_stack_allocator.hpp> #include <boost/coroutine/standard_stack_allocator.hpp> +#if defined(BOOST_WINDOWS) +# include <boost/coroutine/windows/null_stack_allocator.hpp> +#endif #ifdef BOOST_HAS_ABI_HEADERS # include BOOST_ABI_PREFIX @@ -24,6 +27,8 @@ namespace coroutines { #if defined(BOOST_USE_SEGMENTED_STACKS) typedef segmented_stack_allocator stack_allocator; +#elif defined(BOOST_COROUTINE_USE_FIBER) +typedef null_stack_allocator stack_allocator; #else typedef standard_stack_allocator stack_allocator; #endif diff --git a/include/boost/coroutine/windows/null_stack_allocator.hpp b/include/boost/coroutine/windows/null_stack_allocator.hpp new file mode 100644 index 0000000..25f9b34 --- /dev/null +++ b/include/boost/coroutine/windows/null_stack_allocator.hpp @@ -0,0 +1,53 @@ + +// Copyright Oliver Kowalke 2009. +// Distributed under the Boost Software License, Version 1.0. +// (See accompanying file LICENSE_1_0.txt or copy at +// http://www.boost.org/LICENSE_1_0.txt) + +#ifndef BOOST_COROUTINES_NULL_STACK_ALLOCATOR_H +#define BOOST_COROUTINES_NULL_STACK_ALLOCATOR_H + +extern "C" { +#include <windows.h> +} + +#include <cmath> +#include <cstddef> +#include <new> + +#include <boost/config.hpp> + +#include <boost/coroutine/detail/config.hpp> +#include <boost/coroutine/stack_traits.hpp> + +#ifdef BOOST_HAS_ABI_HEADERS +# include BOOST_ABI_PREFIX +#endif + +namespace boost { +namespace coroutines { + +struct stack_context; + +template< typename traitsT > +struct basic_null_stack_allocator +{ + typedef traitsT traits_type; + + void allocate(stack_context & ctx, std::size_t size) + {} + + void deallocate(stack_context & ctx) + {} +}; + +typedef basic_null_stack_allocator< stack_traits > null_stack_allocator; + +} +} + +#ifdef BOOST_HAS_ABI_HEADERS +# include BOOST_ABI_SUFFIX +#endif + +#endif // BOOST_COROUTINES_NULL_STACK_ALLOCATOR_H diff --git a/src/detail/coroutine_context.cpp b/src/detail/coroutine_context.cpp index f63479e..e021281 100644 --- a/src/detail/coroutine_context.cpp +++ b/src/detail/coroutine_context.cpp @@ -28,6 +28,14 @@ void __splitstack_setcontext( void * [BOOST_COROUTINES_SEGMENTS]); namespace boost { namespace coroutines { namespace detail { +#ifdef BOOST_COROUTINE_USE_FIBER +VOID WINAPI coroutine_context::fb_start_proc(LPVOID lpFiberParameter) +{ + coroutine_context * ctx = (coroutine_context*)lpFiberParameter; + ctx->fn_(ctx->param_); + DeleteFiber(ctx->fiber_); +} +#endif coroutine_context::coroutine_context() : stack_ctx_(), @@ -35,18 +43,35 @@ coroutine_context::coroutine_context() : { #if defined(BOOST_USE_SEGMENTED_STACKS) __splitstack_getcontext( stack_ctx_.segments_ctx); +#elif defined(BOOST_COROUTINE_USE_FIBER) + fiber_ = NULL; + param_ = 0; + fn_ = 0; #endif } coroutine_context::coroutine_context( ctx_fn fn, stack_context const& stack_ctx) : - stack_ctx_( stack_ctx), - ctx_( context::make_fcontext( stack_ctx_.sp, stack_ctx_.size, fn) ) -{} + stack_ctx_( stack_ctx) +#ifndef BOOST_COROUTINE_USE_FIBER + , ctx_(context::make_fcontext( stack_ctx_.sp, stack_ctx_.size, fn) ) +#endif // !BOOST_COROUTINE_USE_FIBER +{ +#ifdef BOOST_COROUTINE_USE_FIBER + fn_ = fn; + fiber_ = CreateFiber(0, fb_start_proc, this); +#endif // BOOST_COROUTINE_USE_FIBER +} coroutine_context::coroutine_context( coroutine_context const& other) : stack_ctx_( other.stack_ctx_), ctx_( other.ctx_) -{} +{ +#ifdef BOOST_COROUTINE_USE_FIBER + real_fn = other.real_fn; + fiber_ = other.fiber_; + param_ = other.param_; +#endif // BOOST_COROUTINE_USE_FIBER +} coroutine_context & coroutine_context::operator=( coroutine_context const& other) @@ -55,7 +80,11 @@ coroutine_context::operator=( coroutine_context const& other) stack_ctx_ = other.stack_ctx_; ctx_ = other.ctx_; - +#ifdef BOOST_COROUTINE_USE_FIBER + real_fn = other.real_fn; + fiber_ = other.fiber_; + param_ = other.param_; +#endif // BOOST_COROUTINE_USE_FIBER return * this; } @@ -71,11 +100,32 @@ coroutine_context::jump( coroutine_context & other, intptr_t param, bool preserv __splitstack_setcontext( stack_ctx_.segments_ctx); return ret; +#elif BOOST_COROUTINE_USE_FIBER + other.param_ = param; +#ifdef BOOST_COROUTINE_USE_IS_THREAD_A_FIBER + if (!IsThreadAFiber()) + { + this->fiber_ = ConvertThreadToFiber(this); + SwitchToFiber(other.fiber_); + ConvertFiberToThread(); + this->fiber_ = NULL; + } + else +#endif // BOOST_COROUTINE_USE_IS_THREAD_A_FIBER + SwitchToFiber(other.fiber_); + return this->param_; #else return context::jump_fcontext( & ctx_, other.ctx_, param, preserve_fpu); #endif } +void coroutine_context::destory() +{ +#ifdef BOOST_COROUTINE_USE_FIBER + DeleteFiber(fiber_); +#endif +} + }}} #if defined(_MSC_VER) -- 2.2.0

1 1

What is the e-mail of the Boost webmaster ?
by Edward Diener 08 Dec '14

08 Dec '14

What is the e-mail of the Boost webmaster, as in "If you want to do some XXX on the Boost website, contact the Boost webmaster" ? The term "Boost webmaster" is used on the development portion of the website with no clue who that might be.

1 0

[serialization] Fwd: [Boost-testing] Develop Tests Failing Again
by Rene Rivera 08 Dec '14

08 Dec '14

---------- Forwarded message ---------- From: Tom Kent <teeks99(a)yahoo.com> Date: Mon, Dec 8, 2014 at 2:09 PM Subject: [Boost-testing] Develop Tests Failing Again To: Running Boost regression tests <boost-testing(a)lists.boost.org> It looks like my windows and linux tests have been failing since last saturday, around 4pm pacific time. The end of results/bjam.log is as follows: boost-test(RUN) "serialization/test_helper_ support_xml_warchive" : "libs/serialization/test/test_helper_support.cpp" error: at D:/local/teeks99-08/a/boost_bb/src/kernel\modules.jam:107 error: Unable to find file or target named error: 'test_interrupts.cpp' error: referred to from project at error: '../libs/serialization/test' Tom _______________________________________________ Boost-Testing mailing list Boost-Testing(a)lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-testing -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

1 0

[graph] Lowest common ancestor (LCA), etc.
by Jeremy Murphy 08 Dec '14

08 Dec '14

Hi, I'm working on some graph algorithms that I thought might be useful additions to Boost.Graph, such as the classic LCA and more recent lowest single common ancestor (LSCA) [1]. Is there interest in having these added? I saw there was some discussion of LCA between Cromwell Enage and others back in 2004 but I'm unaware of the outcome (in terms of proposed code, etc). Cheers. Jeremy [1] https://github.com/jeremy-murphy/graph-algorithms

1 0

How to contact test runner "trippels_gcc-5.0_c++11"
by Antony Polukhin 08 Dec '14

08 Dec '14

Hi, There's a GCC bug https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64218 that would possibly require some comments from test runner. Somebody knows how to contact test runner trippels_gcc-5.0? -- Best regards, Antony Polukhin

1 0

[mpl] [build] [testing] Merge pull requests
by Andrey Semashev 08 Dec '14

08 Dec '14

Hi, There were a lot of discussions about MPL recently, and I created a few pull requests to address issues: https://github.com/boostorg/boost/pull/39 Invoke 'headers' target before building testing tools. Note that several testers are currently failing to run because of this problem. Since this affects the testing process, I'd appreciate if someone familiar with it could take a look. https://github.com/boostorg/boost/pull/38 Add MPL.Core tests to the test report. These tests are currently not run by testers. https://github.com/boostorg/mpl/pull/12 Move a misplaced header from MPL to MPL.Core to resolve circular dependency between the two. I believe the first PR is especially critical. Could someone with permissions merge these? Thanks.

13 52

[quickbook] Where to put my pictures?
by Andrzej Krzemienski 05 Dec '14

05 Dec '14

Hi Everyone, How do I put images into QuickBook documentation? The manual says I should put: [$images/img.png] But in which location am I supposed to put the picture? Currently my documentation works if I put folder "images" with my PNG image inside folder "html". But the latter folder ("html") is generated by b2: it would not be there if I haven't run b2. Assuming I only want to keep QBK sources, but no generated files, where do I put my images, so that b2 works? Regards, &rzej

1 0

Re: [boost] [optional] operator<(optional<T>, T) -- is it wrong?
by Felix Uhl 05 Dec '14

05 Dec '14

David Stone wrote: >I have been bitten in the past by operator<. I was converting code that >used unsigned with a 'magic value' to represent nullness to >boost::optional<std::uint32_t>. However, the maximum value it used was >std::numeric_limits<std::uint32_t>:max(). A simple find and replace for >this magic constant (which was referred to everywhere as DWORD_NULL) with >boost::none and fixing compile errors led to some erroneous code. We had >code that assumed an 'uninitialized' std::uint32_t was greater than any >valid value. >The concept of "value not present" is not inherently less-than or >greater-than any value, but because of operator<, code compiled that >shouldn't have. I suppose this is really an argument against any operator< >overload, not just the mixed-type comparison. I don't expect this to be a >major problem for new code, but there is a lot of code out there that uses >a magic value, and that is what optional is supposed to replace. I was irritated by it at first, too, and if you think about it the comparison brakes down to a philosophical problem: If I have a pair of pants and you don’t, are my pants smaller than yours? Hard to say, but to me the most logical solution is “no”. My pants aren’t smaller than yours, and my pants aren’t larger than yours. Also, your and my logic would apply to both the operator<(optional<T>, T) and operator<(optional<T>, optional<T>), but I think we’re only talking about the latter. --- Felix Uhl

14 67

[spinlock] Spin on volatile read, NUMA fairness?
by Benedek Thaler 05 Dec '14

05 Dec '14

Hi All, Niall, 1) I was reading this Intel paper [0], and this section grabbed my attention: "One common mistake made by developers developing their own spin-wait loops is attempting to spin on an atomic instruction instead of spinning on a volatile read. Spinning on a dirty read instead of attempting to acquire a lock consumes less time and resources. This allows an application to only attempt to acquire a lock only when it is free." As I can tell by looking at the source code, spinlock spins on atomic consume. I wonder if a volatile read would produce better performance characteristic? 2) AFAIK spinlocking is not necessarily fair on a NUMA architecture. Is there something already implemented or planned in Boost.Spinlock to ensure fairness? I'm thinking of something like this: [1] Thanks, Benedek [0]: https://software.intel.com/en-us/articles/implementing-scalable-atomic-lock… [1]: http://www.google.com/patents/US7334102

6 13

[review] [sort] Sort library review manager results
by Edward Diener 05 Dec '14

05 Dec '14

This is the results from the recent review of the Sort library of Steven Ross. First I would like to thank all those who made comments during the review, whether or not they officially gave a final Yes or No vote to whether the Sort library should be accepted as a Boost library. This list includes: Niall Douglas, Julian Gonggrijp, Phil Endecott, Vladimir Prus, Mathias Gaunard, Jeremy Murphy, Peter Dimov, Robert Ramey, Adam Walling, Anthony Polukhin, Phil Endecott, Paul Bristow, Thijs (M.A.) van den Berg Dupuis Etienne, and Frank Gennari If I have missed anyone I do apologize. Secondly I would like to thank Steven Ross for patiently answering all of the review comments to the best of his ability. My tally of Yes and No votes for acceptance are: Yes votes (5) : Niall Douglas ( conditional ), Julian Gonggrijp ( conditional ), Frank Gennari, Phil Endecott, Paul Bristow. No votes (3) : Vladimir Prus, Adam Walling ( conditional ), Anthony Polukhin I believe the condtional Yes vote from Julian Gonggrijp was completely met in the discussions of issues with Steven Ross and the conditional Yes vote from Niall Douglass was almost completely met in the discussion with Steven Ross about the issues mentioned, and enough to maintain his Yes vote. I want to also mention that the conditional No vote by Adam Walling has an implied Yes vote to it if the Sort library were one among other sort implementations. Since my final decision is not entirely based on the Yes and No votes I want to adumbrate some of the major issues brought up by the review without necessarily focussing on every one of the people who brought them up initially, as well as my own reactions to them as Review Manager. 1) The first major issue was whether a library whose basic merit lies in its algorithm and its speed/space constraints needs better theoretical backing. A number of reviewers discussed this after it was brought up. I tend to agree with reviewers that while the best theoretical basis is always desirable, it is not necessary for a Boost library whose empirical evidence can be and has been measured by its implementor and can be measured by any user. Furthermore Steven Ross has provided an extensive discussion in his documentation, as well as his original paper, about the theoretical merits of his technique. While much of this discussion is probably beyond the understanding of any but sorting experts and afficionados enough of it adequately explains the basic ideas behind SpreadSort for those who understand and have knowledge of basic sorting ideas and popular sorting techniques. 2) An important issue is the need for the Sorting library to provide extensive timing charts/tables comparing SpreadSort to at least std::sort ( and possibly other popular mainstream sorts ) with both different numbers of sort keys and different initial unsorted distributions. Steven Ross has been providing these charts/tables in the 'develop' branch of the library. 3) Connected to the previous issue is that a number of reviewers asked for better documentation on the conditions in which SpreadSort will show a marked improvement in speed compared to std::sort, as well as any condition(s) where SpreadSort performs worse than std::sort. While Steven Ross has explained that generally SpreadSort will fall back to using std::sort for some cases ( low N ), I too feel that the documentation needs to be more extensive for the average user in attempting to delineate the conditions where SpreadSort works best. 4) A really important issue discussed by a number of reviewers involves the creation of a synthetic radix, where most users of sorting algorithm are used to comparison based sorting functors. While the documentation provides information on the various individual sorts, as well as examples, I also feel that a discussion in the documentation of how to use SpreadSort when sorting structs/classes with multiple keys of different types would be invaluable. In other words, what and if are the common techniques for using SpreadSort with complicated keys which, when using comparison-based sorting, can be represented fairly easily be a functor. 5) Niall Douglass brought up the issue of exception guarantees when using SpreadSort, and I concur. The documentation should be stronger specifying what happens during the sorting algorithms if an exception occurs and/or does the sorting algorithm itself ever throw any exceptions and, if so, under what conditions. Steven Ross mentioned that he is adding exception guarantee information to his docs. 6) Many reviewers did not like the idea that the library should be called Sort when it is basically implementing a single sorting algorithm called SpreadSort. Alternate suggestions are: a) The library should be called SpreadSort. b) The library can be called Sort only if other sorting algorithms are added to it. c) The library can be called Sort with the proviso that it serves as a general library for sorting algorithms potentially added to it in the future. d) The library should be part of the Boost Algorithm library. Steven Ross mentioned other potential sorting implementations as part of the Sort library, under that general name, but all providing a good deal of extra work. These included a copy-based stable sorting algorithm, an LSD radix sorting, and an OpenMP based parallel implementation. Other reviewers mentioned some of the same types of sorting implementations and a few reviewers remarked that they also have created sorting implementations of their own. While I will give my own view at the end, I would like to mention here that I strongly believe that each sorting implementation needs to be reviewed separately for acceptance into Boost and never automatically added to a general library, whether it be called Sort or Algorithm or whatever. 7) Random issues brought up in comments I will mention, which have importance but which I do not believe are absolutely intrinsic yet still might be considered. a) C++11 techniques in general and inline namespaces in particular. b) A version that can work with less than random access. c) Optimizations for small sets of data ( Steven Ross has already addressed this to an extent in the 'developer' branch of the library ). d) Possible documentation of using SpreadSort as a subsort for parallel sorting techniques and possible discusion in the docs about Spreadort and parallel sorts. e) General O() information in docs. f) SpreadSort examples hoisted directly into docs using Quickbook technique and updating the Spreadort docs to use Quickbook. Review Result: Based on the voting for acceptance or not, based on the discussions during the review period, and based on Steven Ross's responses, I have decided that Steven Ross's library should be accepted into Boost. My main reason for my decision, aside from the votes and comments and specific issues discussed, is twofold: 1) There is a general consensus that Steven Ross has an excellent understanding of sorting techniques and algorithms and will support and refine his library for the benefit of programmers. 2) The empirical tests show that SpreadSort offers an improvement over std::sort, significantly in the area of large data sets, and this could be a major benefit to programmers. I do not know if the review manager has a say in this but based on the remarks of the reviewers I would like to see the library as accepted be named 'SpreadSort' rather than just 'Sort'. I do think that Boost can have a library called 'Sort' but I agree with the general consensus that a 'Sort' library needs more than one type of sorting algorithm. I would like to see other people, who mentioned in the reviews/comments that they have their own sorting implementation, also submit their own implementations to Boost and, if this happens and they are accepted, I can see combining them with 'SpreadSort' into a general Boost sorting library called 'Sort' in the future.

10 26