Request for help in porting colony to boost from experienced container boost dev
Hi all- I require help in adapting colony to the boost framework, as colony is light on dependencies, as per the boost recommendations, whereas the boost containers are extremely dependency-heavy - which is somewhat confusing. I need somebody who has some experience with writing or editing the boost containers, to help me sort this from that, wrong practice from right practice in this area. It is, perhaps, something I could spend half a year trying to understand, but unfortunately I do not have that luxury. Whoever helps me will of course get credit for their involvement, with my gratitude. If yourself or anyone you know can help, please get them or yourself in touch. If anyone is interested, my initial port of colony to boost is at the following github repository: https://github.com/mattreecebentley/boost_colony It is an incomplete port but functional. I posted to the boost-users list about colony previously, if anyone is unsure about what it is please see the website (plflib.org) for benchmarks, overview reasoning and code- Thanks, Matt
On 8/24/2015 5:15 PM, Soul Studios wrote:
Hi all- I require help in adapting colony to the boost framework, as colony is light on dependencies, as per the boost recommendations, whereas the boost containers are extremely dependency-heavy - which is somewhat confusing. I need somebody who has some experience with writing or editing the boost containers, to help me sort this from that, wrong practice from right practice in this area. It is, perhaps, something I could spend half a year trying to understand, but unfortunately I do not have that luxury. Whoever helps me will of course get credit for their involvement, with my gratitude. If yourself or anyone you know can help, please get them or yourself in touch. If anyone is interested, my initial port of colony to boost is at the following github repository: https://github.com/mattreecebentley/boost_colony
It is an incomplete port but functional. I posted to the boost-users list about colony previously, if anyone is unsure about what it is please see the website (plflib.org) for benchmarks, overview reasoning and code-
Do you mean "boost containers" as in the Boost container library ? If so, posting questions starting with '[container...]' in the subject line may help you.
On Mon, Aug 24, 2015, [Matt] wrote:
I require help in adapting colony to the boost framework, as colony is light on dependencies, as per the boost recommendations, whereas the boost containers are extremely dependency-heavy - which is somewhat confusing. I need somebody who has some experience with writing or editing the boost containers, to help me sort this from that, wrong practice from right practice in this area. It is, perhaps, something I could spend half a year trying to understand, but unfortunately I do not have that luxury.
Hi Matt, Have you contacted the maintainer of Boost.Container about contributing this? (I assumed from the e-mail and the code in GitHub that your intention is to contribute it to that library, instead of submit a new library for review). Glen
Hi Matt,
Have you contacted the maintainer of Boost.Container about contributing this? (I assumed from the e-mail and the code in GitHub that your intention is to contribute it to that library, instead of submit a new library for review).
Glen
I'm not sure who that is, or how you would go about finding that out - a google search reveals nothing.
On Tue, Aug 25, 2015, [Matt] wrote:
Have you contacted the maintainer of Boost.Container about contributing this? (I assumed from the e-mail and the code in GitHub that your intention is to contribute it to that library, instead of submit a new library for review).
I'm not sure who that is, or how you would go about finding that out - a google search reveals nothing.
That would Ion. See https://github.com/boostorg/container/blob/master/meta/libraries.json for contact details. Glen
On Mon, Aug 24, 2015, [Matt] wrote:
It is an incomplete port but functional. I posted to the boost-users list about colony previously, if anyone is unsure about what it is please see the website (plflib.org) for benchmarks, overview reasoning and code-
Matt, Regarding your use of allocators in your implementation: a. Support the C++ allocator model 1. Either use a.allocate(size) or allocator_traits<A>::allocate(a, size, hint) if you want to supply a hint, but don't bother calling a.allocate(size, 0) - you alienate anyone who writes a C++11 allocator that may not provide an allocate(size, hint) member. 2. Use allocator_traits<A>::construct(a, ...) and allocator_traits<A>::destroy(a) instead of a.construct(...) and a.destroy(...) 3. Use allocator_traits<A> to obtain the type members (pointer, rebind, etc.) (You could use boost::allocator_traits if you want to support pre-C++11) b. Use a compressed_pair<> or the EBCO to reduce the storage for when stateless allocators are supplied (i.e. instead of the 1 + 1 bytes that would be used for the empty objects). c. Support element_type's whose constructor might throw: Do you catch, accordingly destroy any constructed elements, deallocate any allocated elements, before rethrowing? Best, Glen
On Mon, Aug 24, 2015, Glen Fernandes wrote:
Regarding your use of allocators in your implementation: a. Support the C++ allocator model 1. Either use a.allocate(size) or allocator_traits<A>::allocate(a, size, hint) if you want to supply a hint, but don't bother calling a.allocate(size, 0) - you alienate anyone who writes a C++11 allocator that may not provide an allocate(size, hint) member. 2. Use allocator_traits<A>::construct(a, ...) and allocator_traits<A>::destroy(a) instead of a.construct(...) and a.destroy(...) 3. Use allocator_traits<A> to obtain the type members (pointer, rebind, etc.) (You could use boost::allocator_traits if you want to support pre-C++11) b. Use a compressed_pair<> or the EBCO to reduce the storage for when stateless allocators are supplied (i.e. instead of the 1 + 1 bytes that would be used for the empty objects). c. Support element_type's whose constructor might throw: Do you catch, accordingly destroy any constructed elements, deallocate any allocated elements, before rethrowing?
d. Storage for 'elements' and 'erased' could probably be allocated with a single allocation (instead of the two allocations presently). Glen
Matt,
Regarding your use of allocators in your implementation: a. Support the C++ allocator model 1. Either use a.allocate(size) or allocator_traits<A>::allocate(a, size, hint) if you want to supply a hint, but don't bother calling a.allocate(size, 0) - you alienate anyone who writes a C++11 allocator that may not provide an allocate(size, hint) member.
Right - this might be something that has to change for the boost submission, the original is designed to work with both C++0x and C++11, and the hints made a substantial difference to speed with std::allocator.
2. Use allocator_traits<A>::construct(a, ...) and allocator_traits<A>::destroy(a) instead of a.construct(...) and a.destroy(...)
Why, and is there a C++0x-friendly way of doing the same thing?
3. Use allocator_traits<A> to obtain the type members (pointer, rebind, etc.) (You could use boost::allocator_traits if you want to support pre-C++11)
What is the advantage of doing this?
b. Use a compressed_pair<> or the EBCO to reduce the storage for when stateless allocators are supplied (i.e. instead of the 1 + 1 bytes that would be used for the empty objects).
Empty base optimization is not something I understand at this point. I've read a bit about it but I am not sure in which particular area of the colony it would make a difference in?
c. Support element_type's whose constructor might throw: Do you catch, accordingly destroy any constructed elements, deallocate any allocated elements, before rethrowing?
I guess that makes sense in the limited scenario where memory is full. My general feeling is more in line with Mike Acton's in that if something throws, you've got a problem anyway - peaceably unrolling isn't really helping anyone. But I grant that in the context of boost, and in some scenarios, this is vital, and will look at adding it. Thank you, Matt
On Tue, Aug 25, 2015, [Matt] wrote:
Regarding your use of allocators in your implementation: a. Support the C++ allocator model 1. Either use a.allocate(size) or allocator_traits<A>::allocate(a, size, hint) if you want to supply a hint, but don't bother calling a.allocate(size, 0) - you alienate anyone who writes a C++11 allocator that may not provide an allocate(size, hint) member.
Right - this might be something that has to change for the boost submission, the original is designed to work with both C++0x and C++11, and the hints made a substantial difference to speed with std::allocator.
Hints are fine. But use them with std::allocator_traits::allocate or boost::allocator_traits::allocate.
2. Use allocator_traits<A>::construct(a, ...) and allocator_traits<A>::destroy(a) instead of a.construct(...) and a.destroy(...)
Why, and is there a C++0x-friendly way of doing the same thing?
std::allocator_traits is the C++11 and above way of doing things.
3. Use allocator_traits<A> to obtain the type members (pointer, rebind, etc.) (You could use boost::allocator_traits if you want to support pre-C++11)
What is the advantage of doing this?
The motivation for using std::allocator_traits in all of these 3 points (a.1-a.3) is to support the C++11 allocator model. The C++ standard library does this. Most Boost libraries also do this.
b. Use a compressed_pair<> or the EBCO to reduce the storage for when stateless allocators are supplied (i.e. instead of the 1 + 1 bytes that would be used for the empty objects).
Empty base optimization is not something I understand at this point. I've read a bit about it but I am not sure in which particular area of the colony it would make a difference in?
When someone supplies a stateless allocator, like std::allocator<T>, the sizeof(allocator) is still 1 and thus the size of your container object is increased by 1 when it doesn't need to be. The EBCO (or compressed_pair) can be used here to make sure that empty allocators do not add the extra +1 byte to your container object storage. This is also a commonly used technique in C++ standard library implementations, as well as in many Boost libraries that support custom allocators and custom deleters.
c. Support element_type's whose constructor might throw: Do you catch, accordingly destroy any constructed elements, deallocate any allocated elements, before rethrowing?
I guess that makes sense in the limited scenario where memory is full.
No. We're not talking about allocate() throwing (which could happen because of memory being full, or just some constraints a particular generic custom allocator may have that are violated). We're talking about construct() throwing. Both C++ standard library containers, like std:;vector<T>, and containers in Boost libraries, must acknowledge the fact that constructing a T object (i.e. ::new(p) T(...) or std::allocator_traits<A_>::construct(a_, p, ...) may throw - and if you're constructing N objects in a loop, and the J-th construction throws, you need to destroy the (J-1)-th down to 0-th constructed elements before you deallocate the storage.
My general feeling is more in line with Mike Acton's in that if something throws, you've got a problem anyway - peaceably unrolling isn't really helping anyone.
You're actually misunderstanding Mike completely, and out of context. Writing generic code around T (value type) or A (allocator type) requires that you handle things like this. As I mentioned, the C++ standard library containers do this. So do Boost library containers. If you do things like both allocate and construct, and construction throws, and the user of your library has no way to deallocate what you've allocated, it completely alienates them using your container in any of their generic code where they support T types whose constructors can throw, etc. Best, Glen
That would Ion. See https://github.com/boostorg/container/blob/master/meta/libraries.json for contact details.
Cool, thanks!
When someone supplies a stateless allocator, like std::allocator<T>, the sizeof(allocator) is still 1 and thus the size of your container object is increased by 1 when it doesn't need to be. The EBCO (or compressed_pair) can be used here to make sure that empty allocators do not add the extra +1 byte to your container object storage. This is also a commonly used technique in C++ standard library implementations, as well as in many Boost libraries that support custom allocators and custom deleters.
If you could help me with some basic code here, I'd appreciate it- I understand what you're saying, but am unsure whether you're meaning to wrap the allocator instance, or something else entirely.
My general feeling is more in line with Mike Acton's in that if something throws, you've got a problem anyway - peaceably unrolling isn't really helping anyone.
You're actually misunderstanding Mike completely, and out of context.
I'm pretty sure I'm not - I watched the entirety of his Data Driven Design speech at cppcon - several times - and if you watch that he's very specific.
Writing generic code around T (value type) or A (allocator type) requires that you handle things like this. As I mentioned, the C++ standard library containers do this. So do Boost library containers. If you do things like both allocate and construct, and construction throws, and the user of your library has no way to deallocate what you've allocated, it completely alienates them using your container in any of their generic code where they support T types whose constructors can throw, etc.
That's fine, it's just not something I immediately felt was important, but I can see that it is for boost- thanks again, Matt
On Fri, Aug 28, 2015, [Matt] wrote:
I'm pretty sure I'm not - I watched the entirety of his Data Driven Design speech at cppcon - several times - and if you watch that he's very specific.
When you write application code, you're free to do what you want (and avoid caring about exception handling entirely). When you write library code, you're not. You may have people consuming your library code who have exception handling needs that are not your own. Look at the two possible implementations of f() below. Mike isn't telling you to put the former in public/library code over the latter: namespace lib { template<class T> T* f() { void* p = aligned_alloc(alignof(T), sizeof(T)); if (!p) { return nullptr; } return ::new(p) T(); } } ... versus ... namespace lib { template<class T> T* f() { void* p = aligned_alloc(alignof(T), sizeof(T)); if (!p) { return nullptr; } try { return ::new(p) T(); } catch (...) { aligned_free(p); return nullptr; } } }
That's fine, it's just not something I immediately felt was important, but I can see that it is for boost-
For Boost. For the C++ standard library. For any good library. As above, there's a difference between writing application code and [good] library code.
If you could help me with some basic code here, I'd appreciate it- I understand what you're saying, but am unsure whether you're meaning to wrap the allocator instance, or something else entirely.
Look up boost::compressed_pair or the EBO, give it an attempt, and we can review it. I hope that you understand the issue, though. Consider what you have already: template</* ... */> class colony { public: // ... private: // ... element_allocator_type a; group_allocator_type b; // ... }; If the allocator provided is a stateless allocator who has no members (e.g. std::allocator<>), sizeof(a) is still 1, and sizeof(b) is still 1, so sizeof(colony) is increased by 2 bytes when it doesn't have to be. Best, Glen
When you write application code, you're free to do what you want (and avoid caring about exception handling entirely). When you write library code, you're not. You may have people consuming your library code who have exception handling needs that are not your own. Look at the two possible implementations of f() below. Mike isn't telling you to put the former in public/library code over the latter:
As I said, that's fine - Colony was originally conceived for games work, so some of the attitudinal approach comes from there.
If you could help me with some basic code here, I'd appreciate it- I understand what you're saying, but am unsure whether you're meaning to wrap the allocator instance, or something else entirely.
Look up boost::compressed_pair or the EBO, give it an attempt, and we can review it. I hope that you understand the issue, though. Consider
I do understand the issue and want to address it, so to be clear, you are talking about wrapping the following and other rebinds:
element_allocator_type a;
? BTW I didn't actually get an answer to the following:
2. Use allocator_traits<A>::construct(a, ...) and
allocator_traits<A>::destroy(a) instead of a.construct(...) and a.destroy(...)
Why, and is there a C++0x-friendly way of doing the same thing?
std::allocator_traits is the C++11 and above way of doing things.
I get that using allocator_traits seems to be the preferred C++11 way of doing things but I don't know why? Also can I use boost::allocator_traits to do the same thing in C++0x-friendly code?
On Fri, Aug 28, 2015, [Matt] wrote:
I do understand the issue and want to address it, so to be clear, you are talking about wrapping the following and other rebinds:
Yes.
I get that using allocator_traits seems to be the preferred C++11 way of doing things but I don't know why?
It's because a conforming C++11 custom allocator does not need to provide certain members. it is designed to be used through std::allocator_traits which will infer those members from what the allocator does provide.
Also can I use boost::allocator_traits to do the same thing in C++0x-friendly code?
Yes. :-) boost::allocator_traits is a perfectly reasonable way to have the same thing in C++03 friendly code. Best, Glen
On Fri, Aug 28, 2015, [Matt] wrote:
I do understand the issue and want to address it, so to be clear, you are talking about wrapping the following and other rebinds:
Yes.
I get that using allocator_traits seems to be the preferred C++11 way of doing things but I don't know why?
It's because a conforming C++11 custom allocator does not need to provide certain members. it is designed to be used through std::allocator_traits which will infer those members from what the allocator does provide.
Also can I use boost::allocator_traits to do the same thing in C++0x-friendly code?
Yes. :-) boost::allocator_traits is a perfectly reasonable way to have the same thing in C++03 friendly code.
Gotcha - thank you so much for your response, it is much appreciated :) M@
participants (3)
-
Edward Diener -
Glen Fernandes -
Soul Studios