Re: [boost] Random Access to Files (RAF) - any interest ?

From: Tomas Puverle (Tomas.Puverle_at_[hidden])
Date: 2006-05-23 09:58:59

> It is already done.
> We have 2 classes one  for read-only access and another for read-write.

Great!

> Changing the file "from outside" is a big problem. What if it will be 
shortened?
> Also it seems varying in diffrent OSes.

I understand the problem but there isn't much that you can do about a file 
being truncated under you.  That is a higher level synchronisation problem 
that your library won't solve.  Now, assuming you can ignore the file getting 
shorter, can you provide something for the file getting longer?

> > Finally, it would be nice to be able to specify some of the flags that 
memory
> > mapped files can take when they are being opened, such as MAP_PRIVATE,
> > MAP_ANON, MAP_FIXED etc.
> >
> 1. Windows doesn't have such flags

But it has an equivalent set of flags/functionality.  You can specify a base 
address to MapViewOfFileEx.  This is similar to MAP_FIXED.  MAP_PRIVATE can be 
set up by calling MapViewOfFileEx with FILE_MAP_COPY (And the underlying 
mapping object has to be create with PAGE_WRITECOPY protection).  MAP_ANON can 
be achieved by setting the file handle to map to INVALID_HANDLE_VALUE when 
creating the file mapping.

> 2. RAF may be implemented without MM for systems that don't provide it.

I understand.  These flags could be emulated or it could be a caveat of the 
implementation.  But like you said yourself, there are a lot of systems that 
provide mmapping these days.

> 3. Working with large files will map part of the file and will perform
> mapping/unmapping not only at the begining, so MAP_FIXED is not good
> here.

But shouldn't that be up to the user to decide?

> > Also as a word of caution: please don't assume any particular page size in
> > your library.  (You may need to know the page size if you decide to provide
> > subrange mappings).  For very large files we use page sizes larger than the
> > usual 4/8KB because using anything smaller destroys our TLB caches.  But I 
am
> > sure you're not doing anything like that anyway.
> >
> Thank you for caution. It seems that small page size is a reason of
> performance degrading when working with large files. Also page size
> related issues is OS dependant. For now I think we will not play with
> this at all. In the future maybe this will be considered as a better
> performance implementation.

The reason why I mention it is that I think you will need to play with it.  
For example, if I request a range of file starting at byte 104 to byte 30035, 
you will need to request a mapping for an integral number of pages and then 
set your begin() and end() pointers inside the mapped region.  To perform this 
mapping, you will need to know the page size.  I expect the same case will 
happen if you try to map a large file into a small address space, since you 
will need to perform a map on a subrange of the whole file.

Tom