Re: [boost] Random Access to Files (RAF) - any interest ?

From: Tomas Puverle (Tomas.Puverle_at_[hidden])
Date: 2006-05-23 09:37:16

Slava, Alex <raf.devel <at> gmail.com> writes:

> 
> On 5/22/06, Tomas Puverle <Tomas.Puverle <at> morganstanley.com> wrote:
> ...
> > > * Large file support - A practical approach to this would be to limit the
> > > support to 64-bit systems, which are starting to be very common and in 
which
> > > you don't have mmap issues. Would this be a good idea ?
> >
> > There are filesystems out there that support 128 bits for file sizes.  So I
> > think there needs to be a more general solution.
> >
> We think that large file implementation is required and it will be provided.
> "Large" will be different for different OSes. Now for Windows systems
> "Large" will start after 2GB, 

Perhaps it would be worthwhile to also allow for /3G configured systems? Since 
Windows dropped its support for MIPS, it's been possible to extend the user 
space to 3GB under x86 (provided the system is booted with the flag)

> Linux by default allows just 1GB for a
> process user space, so the bound in less then 1GB, 

Again, Linux can now support 3GB and even 4GB of userspace in so called 
hugemem kernels.

> SunOS in 64-bit
> mode allows (according to manual, not tested) to mmap file of size up
> to 2^64 bytes.

Yes.  I've tested some very large files on some Solaris boxes with 200GB of 
RAM.  Not quite 2^64 but it seems that so far, so good.

> If OS provides necessary API to 128-bit file system the corresonding
> implementation of RAF may be developed.

Well, you can't map more of the file than the allowable address space of the 
machine at once :)  Hence my earlier reference to the fact that this will be 
the same problem as trying to map 64-bit files under 32-bit OSs.

Also, AFAIK these 128-bit filesystems preserve posix semantics, so they 
support read(), write() etc transparently.  Since you can't put more than 64-
bits of data into memory on a 64-bit system, the interface of most syscalls 
doesn't need to be changed.

Tom