[boost] Re: Boost Shmem-Shared memory library first archiveavailable

From: ION_G_M (ION_G_M_at_[hidden])
Date: 2004-11-23 06:03:20

Thank you Pavel for your comments, I would like to comment on some 
things

>1. The HTML documentation does't allow to change font in browser.
>2. End of document can be marked so it doesn't
>   look like cut in half during transmission.
>3. docs 3.2:

 Sorry for errors and bad html. It was generated from MS Word so I will 
try to rewrite it using plan html. I'll change that shortly.

>4. Example in 3.2: the "alignement" parameter in
>   segment.create() isn't found in code.

I can't find the error you mention, I've just downloaded the zip file 
and 3.2 does not have any example. In 4.2 segment.create is missing 
a ",".

>5. Example in 4.2:
>   segment.named_new<MyType>
>      ("MyType instance",  /*name of the object*/
>      10                  /*number of elements*/,
>      false  /*if previously present, return error*/,
>      0      /*ctor first argument*/,
>      0      /*ctor second argument*/);

a.It returns false. Exceptions are used only to indicate memory errors 
throwing bad_alloc. You are right there is info missing here. I will 
add more documentation in examples.

b.If you find this approach more useful, I have no problem. I really 
don't like the boolean parameter, but I wanted to have a "find or 
create" functionality. If you like a find_or_named_new<>() additional 
function to indicate that approach I find it more clear than with a 
boolean parameters.

c.The syntax you propose is better, no doubt. I don't have experience 
with it so to implement this I suppose named_new<> should return an 
proxy object with overloaded operator()() functions. Is that right? If 
you want to help me I'm open. If boosters prefer throwing exceptions 
instead or returning false no problem here. A problem I see is that my 
interface allows creating an array like new[]. Do you consider this 
necessary? You prefer a different function? Maybe the proxy object 
should have an operator[] that can be used to specify array allocation? 

>   std::pair< MyType *, std::size_t> res =
>      segment.find_named_new< MyType > ("MyType instance");
>Why do I need the "size"? Doesn't a type have
>always the same size regardless? 

Size contains the number of elements in case you allocate an array. with

   segment.named_new<MyType>
      ("MyType instance",  /*name of the object*/
      10                  /*number of elements*/, ...

you allocate an array of 10 elements. so it will return 10.

>6. Could you use namespace shmem_detail or so
>   instead of "detal" to avoid possible clashes?
No problem. I've seen detail namespace in some projects so I thought it 
was not a problem. detail namespace is inside boost::shmem namespace so 
it wouldn't be necessary. You find it necessary even if detail 
namespace is really boost::shmem::detail?

>7. exceptions.hpp:
>a) file name should be shmem_exceptions.hpp or so

No problem

>b) does it make sense to have common base
>   for both exceptions there?

My exception handling experience is null, so I based all in thread 
exception examples, where lock_error inherits from thread_exception. No 
problem changing this is you find it necessary.

>8. offset_ptr.hpp: full_offset_ptr class
>a) using char as de-facto bool class has usually
>   no practical advantage and may be actually slowe.

No problem. You are right, with all platforms using 4 byte alignment 
bool and char use the same space and it can be slower I suppose due to 
mask operations. 

>b) The flag could be eliminated completely.
>   If m_offset == 0 it is NULL pointer and
>   no data allocated in shmem will starts
>   on the beginning. This would also eliminate
>   need for min_offset_ptr.

The offset indicates the distance between the pointee and the this 
pointer of offset_ptr, so m_offset == 0 indicates a pointer pointing to 
itself and this is quite common in STL containers when empty, since 
next pointer in the end node points to end node, resulting in a 
m_offset = 0. Obviously this is different from a null pointer. If I 
change the meaning of m_offset to offset from the beginning of the 
segment I need the base address stored somewhere (and the base address 
is different in each process), so that I can convert from offset_ptr<A> 
to A* using get() or the constructor. 

Using my approach I can convert between A* and offset_ptr<A> without 
any additional data. Obviously, your approach is more performance 
friendly since m_offset does not need to be changed with each 
assignment and constructor. If boost people find the additional member 
as a waste comparing to A* <-> offset_ptr<A> conversions I would change 
it but I find my approach quite useful, since I don't want to deal with 
base addresses stored somewhere for each process when building a 
offset_ptr.

>c) swap() could be added and other operators.

Ok. I will add it. Which other operator you miss?

>9. Maybe the protection of mutext from shared ptr
>   lib could be worked around

I don't understand your point. Could you give me an example please?

>10. The example in 4.3 uses very dirty C-like
>    approach with casts. Cannot it be rewritten
>    in C++ way with overloaded new?
If you refer to (list_node*)segment.allocate(...) line, segment in this 
case is a low-level void* returning function. If you prefer an 
allocate<Type> approach, I don't have any problem, but remember that 
this segment object is of type shmem_alloc  which allocs raw memory, 
something like when writing your own void* operator new(size_t size) 
for a class. If you have I workaround in mind, please let me know.

>11. Some source files use Unix line ends,
>    some DOS line ends. Just bit strange.
It's because I've tested and changed things in both windows and linux. 
I'll try to convert all to a common line feed.

>12. The simple algorithm to find fitting memory
>    block may not be adequate for high-performance
>    apps (who are most likely to use shared
>    memory).

You are right. The default algorithm is space-friendly, which I thought 
it was more important than performance for fixed size segments. You can 
write your own algorithm and use it since shmem_alloc is a typedef of 
basic_shmem_alloc<default_algo>. If you prefer another algo like 
segregated lists, I can try to write it, so that the user can choose 
the allocation algorithm. I've written the pooled allocator due to 
default algorithm slowness.

>13. Can be be possible to identify named objects
>    with something else than C string? Say
>    wchar_t* or numeral or other templated type?

Well, I've chosen a c string as an universal name, but other types 
could be used. Do you think that the key type should be templatized? I 
think that an integer key can speed up a lot searches but I have to 
think about which classes should be templatized. When storing other 
type of strings (for example std::string, I would need to build an 
allocator for strings in shared memory but also a std::string since 
it's probable that current STL won't work with Shemem STL allocator). 
The key meaning would be different since right now, I copy the string 
to shared memory but with configurable key type things are more 
complicated.

>14. What I would like to see:
>a) avoiding shmem specific containers/mutexes/etc
>   as much as possible.

I think you can't avoid mutexes if you want to guarantee atomic memory 
allocation, since I have no skills to write a lock-free memory 
allocator. Regarding to containers, it was no my intention to write 
them, but I needed some of them to store name-buffer mappings and a 
node container to test the pooled allocator in several systems. For 
now, I have only succeed using shmem STL allocators in a modified 
Dinkum STL. STLport and libstc++ use suppose allocator::pointer to be a 
raw pointer, so I can't use STL containers. I've chosen internal 
containers to be public because I find them very useful, but if this is 
not accepted they can be used only for internal uses and removed from 
documentation.

Regarding mutexes and etc... can you be more explicit? I propose 
mutexes to be general. If boost want to implement process-shared 
mutexes in other way, say in another library, I would use them, but for 
the moment, this is all that I have.

>b) ability to "equalize" shared memory features

I would need some help in this because my operating system knowledge is 
very limited. Mimic-ing UNIX way in windows can be very difficult, I 
think, unless you use a mapped file. I would need some serious help 
here.

>c) support for inheritance in shmem using object
>   factories, e.g. like one in Classloader
>   http://s11n.net/papers/classloading_cpp.html

I'll check it. Thanks for the url.

>d) support for "transactions": I would like to

My knowledge in transaction world is null so I can't help you with 
that. I suppose that a shared memory condition variable should be very 
interesting to notify events to other processes, but I'm afraid this is 
a work for more skilled programmers than me (people from 
boost::threads, perhaps?). As far as I know in windows is difficult to 
implement a shared memory condition variable, pthreads-win32 does no 
support it and I don't know how cygwin solves this.

Thank you again for all your comments. I would like to make some 
changes you've suggested, so I will wait your response regarding open 
issues and comments from other boosters before doing any change. 
Regards,

Ion