Re: [boost] Random Access to Files (RAF) - any interest ?

From: Slava, Alex (raf.devel_at_[hidden])
Date: 2006-05-23 14:04:12

On 5/23/06, Tomas Puverle <Tomas.Puverle_at_[hidden]> wrote:
> > It is already done.
> > We have 2 classes one  for read-only access and another for read-write.
>
> Great!
>
> > Changing the file "from outside" is a big problem. What if it will be
> shortened?
> > Also it seems varying in diffrent OSes.
>
> I understand the problem but there isn't much that you can do about a file
> being truncated under you.  That is a higher level synchronisation problem
> that your library won't solve.  Now, assuming you can ignore the file getting
> shorter, can you provide something for the file getting longer?
>

Conceptually it should be the same thing. If nobody notifies the
library about change in file size, it just doesn't know.
Theoretically, to handle such a situation checking can be done whether
a hit into the area that was originally beyond the file bounds is
actually an error. If not, depending on the implementation, a
remapping can be done, or "mapping window" will be moved
appropriately, otherwise an exception is raised, or whatever. Such
behaviour would make all access operations costly on all
implementations, and render RAF useless for the purpose it was
originally designed. On the other hand, if the hit outside the file
bounds was made, it is either an application error, or an evidence
that the application *knows* of the changes in file size. Than the
appropriate measures can be taken explicitly by the application
itself. At least this is how we see it at the moment.

> > > Finally, it would be nice to be able to specify some of the flags that
> memory
> > > mapped files can take when they are being opened, such as MAP_PRIVATE,
> > > MAP_ANON, MAP_FIXED etc.
> > >
> > 1. Windows doesn't have such flags
>
> But it has an equivalent set of flags/functionality.  You can specify a base
> address to MapViewOfFileEx.  This is similar to MAP_FIXED.  MAP_PRIVATE can be
> set up by calling MapViewOfFileEx with FILE_MAP_COPY (And the underlying
> mapping object has to be create with PAGE_WRITECOPY protection).  MAP_ANON can
> be achieved by setting the file handle to map to INVALID_HANDLE_VALUE when
> creating the file mapping.
>

Right. Thanks. Overlooked in the MSDN.

> > 2. RAF may be implemented without MM for systems that don't provide it.
>
> I understand.  These flags could be emulated or it could be a caveat of the
> implementation.  But like you said yourself, there are a lot of systems that
> provide mmapping these days.
>

There is one more thing. There is a thouhgt to provide at least both
MMF and non-MMF implementations on all systems because of concurrent
locking problems with memory mapping. Then these flags either have to
be emulated, or they would add confusion, because of being of use for
one of implementations. On the other hand this functionality seems
important. This requires thorough analysis...

> > 3. Working with large files will map part of the file and will perform
> > mapping/unmapping not only at the begining, so MAP_FIXED is not good
> > here.
>
> But shouldn't that be up to the user to decide?
>

If we provide other flags, then this one perhaps also should be provided...

> > > Also as a word of caution: please don't assume any particular page size in
> > > your library.  (You may need to know the page size if you decide to provide
[snip]
> The reason why I mention it is that I think you will need to play with it.
> For example, if I request a range of file starting at byte 104 to byte 30035,
> you will need to request a mapping for an integral number of pages and then
> set your begin() and end() pointers inside the mapped region.  To perform this
> mapping, you will need to know the page size.  I expect the same case will
> happen if you try to map a large file into a small address space, since you
> will need to perform a map on a subrange of the whole file.
>

Thanks. Very true.

> Tom
>
>
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>

-- 
Svyatoslav Trukhanov,
Oleksii Ursulenko