[boost] Re: A library: out-of-core containers and algorithms

From: Roman Dementiev (dementiev_at_[hidden])
Date: 2005-02-08 06:01:30

Jonathan Turkanis wrote:
> Roman Dementiev wrote:
> 
>>Thorsten Ottosen wrote:
>>
>>>"Roman Dementiev" <dementiev_at_[hidden]> wrote in message
> 
> 
>>>>yes. One must rewrite only the lower layer of Stxxl using native
>>>>file access methods and native multithreading.
>>>
>>>is that because you need mem.mapped files? Otherwise, why don't you
>>>use boost filsystem and boost threads?
>>
>>I do not necessarily need memory mapped files.
>>
>>The Boost filesystem library is more about manipulating files and
>>directories. But what I need is the file access itself:
>>create/open/read/write/close file. I would have been using
>>std::fstream as a portable file access method, but it
>>lacks support of files larger than 2 GB. It is really big
>>disadvantage in my case.
> 
> 
> The boost iostreams library, which is in CVS now and will be part of the next
> boost release, contains a class file_descriptor for accessing files using OS or
> runtime library file descriptors.
> 
> http://www.kangaroologic.com/iostreams/libs/iostreams/doc/?path=6.1.3
> 
> Currently, random access if performed using offsets of type std::streamoff,
> which is often implemented as a 32-bit unsigned long. However, I recently
> decided to relax this restriction and allow offsets to be specified using larger
> intergral type. On windows, seeking will be implemented using _lseek64 where
> available. On Posix, lseek will be used; it uses off_t, which is (I think) often
> 64 bits.

Under Linux if preprocessor macros _LARGEFILE_SOURCE, _LARGEFILE64_SOURCE,
_FILE_OFFSET_BITS=64 are defined then off_t is 64 bit, otherwise it is 
32 bit.

> 
> Access via iostreams will still have to use std::streamoff, since that is out of
> my control; but you should be able to use an instance of file_descriptor
> directly.
> 
> The new code should be available soon.

I think boost::iostreams::file_descriptor is what I need. Of course with 
64 bit seek.
Another reason why I do not like std::fstream is its bad performance. It 
introduces
a superfluous copying into it's internal buffer when doing I/O on user 
buffer.

Another issue is that the performance of my library will benefit from 
unbuffered file
system access. Many operating system support it (O_DIRECT open option in 
Unix systems,
FILE_FLAG_NO_BUFFERING|FILE_FLAG_WRITE_THROUGH option for "CreateFile"
Windows API call).

Roman