Re: [boost] [interprocess] shared memory lifetime

From: Kim Barrett (kab_at_[hidden])
Date: 2006-03-10 17:30:55

At 9:25 PM +0100 3/10/06, Ion Gaztañaga wrote:
>You can use MSDN (www.msdn.com)

Thanks for the pointer.

>  > A POSIX shared memory object exists from the time it is created until
>>  it is unlinked and all references are gone (or the system is
>>  rebooted). (It becomes inaccessible to further shm_open calls if
>>  unlinked, but remains open to processes that had already opened it.)
>
>Right. The difference is that in windows, if there's still a process
>connected to the shared memory, another process can open the same
>segment. If a UNIX process calls shm_unlink, them the processes that
>opened the shared memory can still work on it, but if another process
>can't use that memory, because the name is removed.

The other difference is that on Windows, if there are no processes
connected to the shared memory, then it goes away. For both POSIX and
SysV shared memory, once created, a shared memory exists until unlinked
(or system reset). So a process can create and put some stuff in shared
memory, and then exit normally, and sometime later another process can
come along and still find that same shared memory data. That doesn't
work on Windows (or with the current reference counting implementation
of the library).

>So if a process calls shm_unlink another process can create new shared
>memory segment with the same name, but never connect to that old shared
>memory. This is different from a file unlink behavior, I think.

No, this is actually the same a a file unlink behavior. Intentionally
so, so that an implementation can basically use files (possibly in a
special file system) to implement shared memory. If several processes
all have opened a normal file, and that file is then deleted from the
file system, the processes still retain their handles on the file and
can perform file-io operations on it. Internally the file handles are
attached to the inode, which doesn't get reclaimed until there are no
references (including from the filesystem).

>  > What the present library implementation is trying to do with this
>>  reference count mechanism is to emulate the Windows behavior on POSIX
>>  systems.
>
>Right. I find windows behaviour easier to implement with posix (when
>there is no crash) than emulating posix with windows.

Yes, I can't think of a better way to implement the POSIX behavior on
Windows than your suggestions of either memory mapped files or a server
program to act as a proxy for the POSIX create/unlink operations. The
proxy server is particularly not pretty.

>I'm sure there is a solution.

I wish :(

There almost could be, by using a server program on POSIX systems.
Rough sketch: open connection to server, tell server about opens and
closes. When connection is closed, server fixes reference counts for
any shared memory opens not yet closed. select() can be used to get
notification of connection closes even for crashes. I haven't thought
this through carefully, so there may be race conditions in there.

>One option is to use a memory mapped file to emulate posix-like behavior
>in windows. But it surely will be slower (the OS will dump data to file)
>and the unlink is not trivial.

That does seem like it would be bad if true. Are there any file
manipulation options that might be used to control that? I don't know,
but I'll ask around and do some reading and see what I can find.

[Later] As long as you don't flush the file it would seem like a memory
mapped file should be no different from a shared memory in performance,
the only difference is where data gets written if it needs to be swapped
out. The big downside to using an actual file would be that you need a
place in the file system to put it. Then you get into the whole name of
the shared memory portability problem again.

>Windows natively does not support posix nor System V. I think that with
>Unix Services for Windows (previously Interix)you have system V
>interface. I think Unix Services for Unix is a free download now, but I
>think you need to use that environment to build a Unix-like application
>(using gcc compiler). I don't know if those system V functions are
>available with a normal windows environment.

Yeah, I did some more research, and that matches what I found. That path
looks like a good way to ensure that Windows programmers don't use the
library. Oh well...

But wait. We don't need to use SysV shared memory to get reference counting
behavior on Windows. So if reference counting behavior is indeed what the
library will provide, Windows native API's provide it, and (most or perhaps
all) Unix ports could use the SysV API, though as noted in my earlier
message, that may have some issues too, regarding kernel limits. For example,
I figured out where to find the information for OSX and (by default) you only
get 4M maximum size, 32 total, 8 per process.

[Note that my own preference is toward not reference counting, but the
potential impact on Windows is certainly an issue.]