Re: [Boost-users] Boost::interprocess::string

Subject: Re: [Boost-users] Boost::interprocess::string
From: Anthony Foiani (tkil_at_[hidden])
Date: 2012-01-05 22:32:47

Josmon --

I'd hoped to be able to give you a more complete example, but I ran
out of time.

I have a working program at the bottom of this message; here's a bit
of text I was trying to keep in sync with the code.

------------------------------------------------------------------------

I think that the important observation is that a string which works in
a snared memory container does not have the same C++ type as a
"std::string".  The interfaces are similar (probably identical), but
they have different template arguments; specifically, strings used in
shared memory must have an allocator that pulls from that shared
memory.

I learned about this when I was trying to use strings as keys in a
shared map.  I'll try to provide a quick example below, but there are
relevant (if somewhat terse) examples in the Boost.Interprocess
documentation:

  http://www.boost.org/doc/libs/1_48_0/doc/html/interprocess/allocators_containers.html#interprocess.allocators_containers.containers_explained.where_allocate

  (or here, if that wraps: http://preview.tinyurl.com/7cmb76n )

Another link:

  http://en.highscore.de/cpp/boost/interprocesscommunication.html#interprocesscommunication_managed_shared_memory
  (or: http://preview.tinyurl.com/6tpg9dm )

Josmon Paul <josmon.paul_at_[hidden]> writes:

> I have a class whose private members are string type of data. Each
> time i will create a new instance of my class and to each instance i
> will set some string type of data to it . After this i have to push
> this object it to the vector.

So we have a simple local memory object:

  typedef std::string local_string_t;

  struct local_info_t
  {
      local_info_t( const local_string_t & str ) : m_str( str ) {}
      local_string_t m_str;
  };

And a corresponding vector of them:

  typedef std::vector< local_info_t > local_info_vector_t;

Which you fill by doing something like:

  local_info_vector_t local_vec;

  local_vec.push_back( local_info_t( "foo" ) );
  local_vec.push_back( local_info_t( "bar" ) );
  local_vec.push_back( local_info_t( "baz" ) );

Good so far?

> Now i have a function called write_data which will take this vector
> list as input and will iterate through each vector element and push
> to shared memory vector. 

Ok, so now we have our shared structures (with some bits left
undeclared until later):

  struct shared_info_t
  {
    // shared_info_t constructor, version 1
    shared_info_t( const local_info_t & loc ) : m_str( loc.m_str ) {}
    shared_string_t m_str; // mystery type 1
  };

  shared_info_vector_t shared_vec; // mystery type 2

So what are are the actual C++ types of "shared_string_t" and
"shared_info_vector_t"?

Both std::vectors and std::strings need to allocate "bookkeeping"
space (capacity, length) as well as actual data-holding space.  In the
case of shared memory, we need to make sure that the bookkeeping space
is in the shared memory segment as well as the contents.

This is where Ion's previous comments about allocators is relevant.
(I think they were to you, but I'm not sure -- they were within the
last week or so.)  

I'm not an expert, so I'll just try to transcribe what I did to get
something similar working.  (I needed a shared map, not a shared
vector, but the overall outline is very similar.)

First, we need something that will manage the shared memory itself:

  namespace bi = boost::interprocess;

  typedef bi::managed_shared_memory
    managed_shared_memory_t;

Next, we need an allocator that can give us memory for holding
characters:

  typedef bi::allocator<
      char,
      shared_manager_t::segment_manager
  > shared_char_allocator_t;

This is enough for us to define strings which can be safely stored in
shared memory:

  typedef bi::basic_string<
      char,
      std::char_traits< char >,
      shared_char_allocator_t
  > shared_string_t;

The interprocess vectors need to know how to allocate strings in
shared memory, so we need another allocator (one for shared_strings,
instead of just characters in shared memory):

  typedef bi::allocator<
      shared_info_t,
      shared_manager_t::segment_manager
  > shared_info_allocator_t;

  typedef bi::vector<
      shared_info_t,
      shared_info_allocator_t
  > shared_info_vector_t;

Whew.

Finally, it takes a bit of magic to make sure that the shared bits are
actually placed in shared storage.  (This is all well-described at the
documentation link given above.)  Within a segment manager, individual
objects are referred to by name, and they must be either found or
constructed.

So the call site looks like so:

  // create the shared memory area
  managed_shared_memory_t managed_shared_memory(
      create_only,         // throw if already exists
      "my_shared_memory",  // unique name for shared memory block
      10000                // size in bytes
  );

  // get a pointer to the manager (just to save typing)
  managed_shared_memory_t::segment_manager * shared_manager_p =
    managed_shared_memory.get_segment_manager();

  // now create allocators
  shared_char_allocator_t shared_char_allocator( shared_manager_p );
  shared_info_allocator_t shared_info_allocator( shared_manager_p );

  // get a pointer to the shared vector
  shared_info_vector_t * shared_vec_p =
    shared_manager_p->find_or_construct< shared_info_vector_t >
      ( "my_shared_vector" )       // name of vector in shared memory
        ( shared_info_allocator ); // vector constructor args

  // finally, copy from local to shared storage
  BOOST_FOREACH( const local_info_t & i, local_vec )
    shared_vec->push_back( shared_info_t( i ) );

> Now from this shared memory i have to iterate these objects and i
> have to get the string data present in these instance. Right now i
> am declaring this string type of data as boost::interprocess::string
> and will set this data to the instance of the class.

To get those values in another process, you first need to get a
pointer to the shared vector.  If you're forking the same process that
created it, then that's easy; if this is an unrelated process, you'll
need to do something like this:

  // find the previously-created shared memory area
  managed_shared_memory_t managed_shared_memory(
      open_only,           // throw if it doesn't already exist
      "my_shared_memory"   // same name as used in creation
  );

  // find the already-created shared vector by name
  shared_info_vector_t * shared_vec_p =
    shared_manager.find< shared_info_vector_t >
      ( "my_shared_vector" );

  // and now iterate over the shared vector
  BOOST_FOREACH( const shared_info_t & i, *shared_vec_p )
    cout << i.m_str << endl;

> But it will create segmentation fault when i am accessing this
> string type of data from the object when the size of string is more
> than 10. So is there any method to increase the buffer size of
> boost::interprocess::string .?

You have the correct cause, but increasing the internal buffer isn't a
good answer (and, honestly, I don't know if it's possible at all).

The segmentation fault is occurring because the string living in
shared memory tries to allocate an outside-of-object buffer.  Without
the correct allocator, that buffer will be in memory local to the
writing process; however, the pointer value is written to shared
memory.  When another process tries to use that pointer, it's (at
best) pointing to non-mapped memory and giving you yours seg-fault.
(At worst, it's pointing at valuable information that might get
overwritten or cause nearly-endless loops, etc.)

The more correct answer is to make sure that all the allocations are
done from shared memory.

------------------------------------------------------------------------

And the program follows my signature.  I'm sure there are things that
could be fixed, but this was intended to be a quick demo...

Best regards,
Anthony Foiani

------------------------------------------------------------------------

// linux compile line:
//   g++ -o shared-vec shared-vec.cpp -lpthread -lrt

#include <iostream>
#include <string>
#include <vector>

#include <boost/foreach.hpp>

#include <boost/interprocess/containers/string.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/managed_shared_memory.hpp>

// =====================================================================
// local types

typedef std::string local_string_t;

struct local_info_t
{
    local_info_t( const local_string_t & str ) : m_str( str ) {}
    local_string_t m_str;
};

typedef std::vector< local_info_t > local_info_vector_t;

// =====================================================================
// shared types

namespace bi = boost::interprocess;

typedef bi::managed_shared_memory
  managed_shared_memory_t;

typedef bi::allocator<
    char,
    managed_shared_memory_t::segment_manager
> shared_char_allocator_t;

typedef bi::basic_string<
    char,
    std::char_traits< char >,
    shared_char_allocator_t
> shared_string_t;

inline local_string_t
local_from_shared_string( const shared_string_t & ss )
{
    return local_string_t( ss.begin(), ss.end() );
}

struct shared_info_t // version 2
{
    shared_info_t( const local_info_t & loc,
                   shared_char_allocator_t & char_alloc )
        : m_str( loc.m_str.begin(), loc.m_str.end(), char_alloc ) {}
    shared_string_t m_str;

    shared_info_t & operator=( const shared_info_t & rhs )
    {
        m_str = rhs.m_str;
    }
};

typedef bi::allocator<
    shared_info_t,
    managed_shared_memory_t::segment_manager
> shared_info_allocator_t;

typedef bi::vector<
    shared_info_t,
    shared_info_allocator_t
> shared_info_vector_t;

// =====================================================================

int main( int argc, char * argv [] )
{
    const std::string cmd( argc > 1 ? argv[1] : "show" );

    // get some shared memory
    managed_shared_memory_t managed_shared_memory(
        bi::open_or_create, "my_shared_memory", 10000 /* bytes */
    );

    // get a pointer to the manager (just to save typing)
    managed_shared_memory_t::segment_manager * shared_manager_p =
      managed_shared_memory.get_segment_manager();

    // construct our allocators
    shared_char_allocator_t shared_char_allocator( shared_manager_p );
    shared_info_allocator_t shared_info_allocator( shared_manager_p );

    // get a pointer to the shared vector
    shared_info_vector_t * shared_vec_p =
      shared_manager_p->find_or_construct< shared_info_vector_t >
        ( "my_shared_vector" )       // name of vector in shared memory
          ( shared_info_allocator ); // vector constructor args

    if ( cmd == "add" )
    {
        // create a vector of info objects in process-local storage
        local_info_vector_t local_vec;

        if ( argc > 2 )
        {
            for ( int i = 2; i < argc; ++i )
                local_vec.push_back( local_info_t( argv[i] ) );
        }
        else
        {
            local_vec.push_back( local_info_t( "foo" ) );
            local_vec.push_back( local_info_t( "bar" ) );
            local_vec.push_back( local_info_t( "baz" ) );
        }

        // copy from local storage to shared storage
        BOOST_FOREACH( const local_info_t & i, local_vec )
          shared_vec_p->push_back( shared_info_t( i, shared_char_allocator ) );
    }
    else if ( cmd == "show" )
    {
        // use data from shared storage directly
        for ( int i = 0; i < shared_vec_p->size(); ++i )
            std::cout << i << ". " << shared_vec_p->at( i ).m_str << std::endl;
    }
    else if ( cmd == "slurp" )
    {
        // copy from shared to local
        local_info_vector_t local_vec;
        BOOST_FOREACH( const shared_info_t & i, *shared_vec_p )
          local_vec.push_back( local_info_t( local_from_shared_string( i.m_str ) ) );

        // and display it out of local memory
        for ( int i = 0; i < local_vec.size(); ++i )
            std::cout << i << ". " << local_vec.at( i ).m_str << std::endl;
    }
    else if ( cmd == "clear" )
    {
        shared_vec_p->clear();
    }
    else if ( cmd == "destroy" )
    {
        shared_manager_p->destroy_ptr( shared_vec_p );
        bi::shared_memory_object::remove( "my_shared_memory" );
    }
    else
    {
        using std::clog;
        using std::endl;

        clog << "usage: " << argv[0] << " COMMAND"                 << endl
             << "commands:"                                        << endl
             << "  add     - insert values into shared vector"     << endl
             << "  show    - list values in shared vector"         << endl
             << "  slurp   - copy from shared to local, then show" << endl
             << "  clear   - empty shared vector"                  << endl
             << "  destroy - remove shared memory"                 << endl;

        if ( cmd != "help" && cmd != "?" )
        {
            clog << argv[0] << ": unknown command: '" << cmd << "'" << endl;
            return 1;
        }
        else
        {
            return 0;
        }
    }

    return 0;
}