Re: [boost] Random Access to Files (RAF) - any interest ?

From: Slava, Alex (raf.devel_at_[hidden])
Date: 2006-05-23 02:07:00

On 5/22/06, Tomas Puverle <Tomas.Puverle_at_[hidden]> wrote:
> > Dear Boost Community,
> >
> > Is there any interest in submission of a Random Access to Files (RAF)
> > Library to Boost?
> > Please see http://www.trukhanov.kiev.ua/RAF/ for documentation and an
> > initial implementation.
>
> I'm affraid I've not had the time to look at your implementation but I had a
> quick scan of the docs.
>
> An approach I've taken when implementing a memory mapped file class in the
> pass was to try to improve on the compile-time guarantees one can provide.
> When you open a file as read only, you can still read it but write access
> causes a SEGV on Unix.  This is undesirable, especially if at compile time you
> know that you will only access the MMF for reads.  What I am suggesting is
> that you take advantage of the type system and distinguish between readonly
> (const) MMFs (which will consequently return const_iterators).
>
It is already done.
We have 2 classes one  for read-only access and another for read-write.
Read-only class provides only const version of operator[], at() and iterators
so the above error will be detected at compile time.
 >
> Also, aside from the RW criteria, it would be useful to be able to provide a
> range for the mapping.  This brings up the possibility of what happens should
> the file be smaller than the range that was initially requested, and also an
> interesting question as to what would I do if I am trying to read a file that
> is growing as I read it (similar to tail -f).
>
We are thinking about range mapping and it will be implemented soon.
Changing the file "from outside" is a big problem. What if it will be shortened?
Also it seems varying in diffrent OSes.

>
> Finally, it would be nice to be able to specify some of the flags that memory
> mapped files can take when they are being opened, such as MAP_PRIVATE,
> MAP_ANON, MAP_FIXED etc.
>
1. Windows doesn't have such flags
2. RAF may be implemented without MM for systems that don't provide it.
3. Working with large files will map part of the file and will perform
mapping/unmapping not only at the begining, so MAP_FIXED is not good
here.

>
> Also as a word of caution: please don't assume any particular page size in
> your library.  (You may need to know the page size if you decide to provide
> subrange mappings).  For very large files we use page sizes larger than the
> usual 4/8KB because using anything smaller destroys our TLB caches.  But I am
> sure you're not doing anything like that anyway.
>
Thank you for caution. It seems that small page size is a reason of
performance degrading when working with large files. Also page size
related issues is OS dependant. For now I think we will not play with
this at all. In the future maybe this will be considered as a better
performance implementation.

> Thanks!
>
> Tom

-- 
Svyatoslav Trukhanov,
Oleksii Ursulenko