Re: [Boost-users] Interprocess : vectors of vectors

Subject: Re: [Boost-users] Interprocess : vectors of vectors
From: Anthony Foiani (tkil_at_[hidden])
Date: 2013-05-14 12:18:23

Oodini --

Oodini <svdbg____at_[hidden]> writes:
> I'd like to share a vector of vectors (jagged 2D array).  The data
> come from a deserialization.  I know the size of the "external"
> vector. 
>
> The sizes of the "internal" vectors are discovered during
> deserialization.
>
> managed_shared_memory segment(create_only,
>                                "MySharedMemory",  //segment name
>                                65536);
>
> how to set the size (3rd argument), when considering my use case ?

If you have plenty of memory (especially virtual memory), then just
make it huge, and don't worry about using it all.

Alternately, get some guarantee from the previous step about the
largest possible sizes.  You probably need some maximum anyway;
otherwise, one day someone will try to pump gigabytes of data into
your routine, and things will go pear-shaped.

If you can make multiple passes over the serialized data, then you can
calculate exactly how many elements in each vector, and size your
segment based on that, but as you ask...

> Even with a simple vector, and if the number and the types of the
> elements are know? i woldn't know what size to set for truncation.
> On VS 2010, a sizeof(vector<int>) is 40; I suppose I have to add it
> sizeof(int)*nbElements to get the full size.
>
> 1. I don't know if this assumption is valid

Probably not perfectly valid, but it should get you to the right
order of magnitude.

> 2. If so, I don't know if it's still valid when using
>    Boost.Interprocess (I suppose it isn't)

It might be; the main difference with most of the interprocess
containers is that they work with offset pointers, not with raw
pointers.  Offset pointers should be the same size as regular
pointers, although there might be a bit of extra bookkeeping in the
vector "header" itself.

> 3. I don't know how to generalize that to a vector of vectors in
>    order what should be the truncation size.

It's easier to talk about this if we have some concrete types:

    namespace bi = boost::interprocess;

    typedef bi::managed_shared_memory
      managed_shared_memory_t;

    typedef bi::allocator<
        int,
        managed_shared_memory_t::segment_manager
    > shared_int_allocator_t;

    typedef bi::vector<
        int,
        shared_int_allocator_t
    > inner_vec_t;

    typedef bi::allocator<
        inner_vec_t,
        managed_shared_memory_t::segment_manager
    > shared_inner_vec_allocator_t;
 
    typedef bi::vector<
        inner_vec_t,
        shared_inner_vec_allocator_t
    > outer_vec_t;

Further, assume we can do two passes over the data, and end up with a
vector of the length of each serialized sub-vector:

    std::vector< size_t > subs;

We can compute the amount of user-visible space like so:

    // first, the size of the outermost containing struct
    size_t total_memory = sizeof( outer_vec_t );

    // this is too tight of a constraint, since the outer_vec_t will
    // really allocate more than the requested number of inner_vec_t
    // structs, but as a minimum:
    total_memory += subs.size() * sizeof( inner_vec_t );

    // and then we have to add in the memory for each inner vector:
    for ( const size_t inner_len : subs )
        total_memory += inner_len * sizeof( int );

(There are probably some int vs. size_t type conversion issues in that
code; season with casts or larger types to taste.)

However, realize that this is a bare minimum: the actual amount of
memory needed is guaranteed to be larger for at least the following
reasons:

a. vectors tend to allocate extra space (think "reserve()" and
   "capacity()" vs. "resize()" and "size()").

b. chunks of memory are often larger than actually requested (to
   improve performance, or to stash housekeeping information with
   each chunk).

c. the "managed" part of "managed memory segment" includes storage for
   object name to location+size mapping, which reduces the total
   amount of memory available in the segment.

So whatever value you get for "total_memory", remember that you need
to increase it by some amount.  If you have the memory to spare, go
for double, and you should be fine; if you're tight, try maybe 20%
more, and pay extra attention to your exception paths.

> Thanks for help.

Hope this helps.

I've written a few other replies on related interprocess issues:

   http://preview.tinyurl.com/a877vlf
   http://preview.tinyurl.com/c5l6mtw

> PS : I also read
> http://www.boost.org/doc/libs/1_52_0/doc/html/interprocess/managed_memory_segments.html,
> but I havn't been enlightened about these points.

In a sense, that should be at a lower level than the questions you're
working on.  The segments just provide raw memory; it's up to the
containers to use it.

Hope this helps,
Tony