Re: [boost] [Review] Shmem

From: Ion Gaztañaga (igaztanaga_at_[hidden])
Date: 2006-02-12 08:59:11

Hi Andy,

> ***Design***
> 
> Shmem is limited in scope. Its a low level building block for implementing local
> inter process Object transfer.  AFAICS shmem doesnt concern itself much with the
> protocols of IPC. It wont scale to distributed communications. Is there scope
> for proprietary IPC mechanisms , (such as COM, DDE, and CORBA), to be built on
> top of it ?

You are right. Shmem is focused on classic inter-process mechanisms like
shared memory/memory mapped files and it supposes ABI-compatible C++
clients. However, it can be used to build complex binary-serialized
messages through network. But I haven't put any effort in distributed
approach, mainly because I would be reinventing the wheel. I think that 
for distributed approach you could use boost::serialization over TCP/IP. 
  Shmem is focused on raw, maximum efficiency transport between processes.

> The restrictions placed on the types of objects that can be placed in shared
> memory are understandable, but heavy and presumably cant be checked?. 

Well, I could put some is_polymorphic<> checks. However, I don't know
how to detect member pointers and references.

> Now there
> Must be a special definition of an shareable-object outside C++ class eg like a
> COM interface. In fact some evidence that this is occurring with classes such as
> basic_named_shared_object. However should these objects not be designed for
> distributed environment as well as local one? If shmem doesnt provide protocols
> for building objects that can be passed over a network and in a scaleable way
> then objects using its mechanisms direct will be of limited use, so larger
> picture should be taken into account Now.

> 
> *** Offset Pointers ***
> 
> Couldnt pointer_offsets be a class?. This would mean overloadable functions,
> type safety, rather than ptrdiff_t which I'm guessing is an int or something?

That seems a good idea. You prefer some kind of handle or identifier 
that can identify an object in shared memory that can be passed between 
processes. I think it's a good idea to encapsulate it since in a 
multi-segment architecture, the unique identifier of an object in a 
multi-segment group could be a more complex object. I would need to change

get_offset_from_address()/get_address_from_object

functions of named_shared_object to a

get_handle_from_address/get_address_from_handle approach. So you can 
pass the handle of an object through an IPC mechanism instead of raw 
ptrdiff-s.

> ----------
> 
> *Documentation*
> 
> It seems a lot of effort has been put into documentation. Its noted that Ion is
> not a native English speaker, but it reads OK in spite of that.  It probably
> needs to be better organised though. Concepts and Definitions section should be
> split into Concepts section and Definitions section. Traditionally definitions
> are put near to start of docs so user knows where they are before defined words
> are used. Definitions should be expanded A Lot ... to include many more entities
> such as mutex, semaphore, shared-memory, named-shared-memory,offset-pointer,
> process  etc etc etc.  

I will try to improve that.

> ----------
> 
> **Construction ***
> 
> On construction issue , only because its been brought to light recently.

Ready to implement RAII as you know.

> ***Should it be a boost library**
> 
> You betcha. Yes indeed It should definitely be a boost library. C++ needs this
> type of library badly. It is all very low level however and I would like to see
> a higher level abstraction for designing objects that can be passed about over a
> network too though and shmem objects should then try to conform to that, but
> that shmems role would be maybe as a building block for that. Needs to be
> discussed. If it is in Docs.. Sorry I missed it!

The idea of Shmem is to start building a better shared message queue, 
and a pipe-like stream. After that, maybe a interprocess datagram/stream 
mechanism similar to sockets. A Solaris doors like mechanism would be 
fine too. And apart from this, I want to do some research with a 
multi-segment Shmem architecture so that we can allocate new shared 
memory segments automatically when a segment is full of data. That would 
require also a special pointer type to be able to point from one segment 
to other when every segment can be mapped in a different address in each 
  process. This can be a good base for multi mapped-file data-bases. A 
lot of work to do, as you can see.

Ion