[boost] Re: A library: out-of-core containers and algorithms

From: Jonathan Turkanis (technews_at_[hidden])
Date: 2005-02-08 14:33:43

Roman Dementiev wrote:
> Jonathan Turkanis wrote:
>> Roman Dementiev wrote:

>>> The Boost filesystem library is more about manipulating files and
>>> directories. But what I need is the file access itself:
>>> create/open/read/write/close file. I would have been using
>>> std::fstream as a portable file access method, but it
>>> lacks support of files larger than 2 GB. It is really big
>>> disadvantage in my case.

>> The boost iostreams library, which is in CVS now and will be part of
>> the next boost release, contains a class file_descriptor for
>> accessing files using OS or runtime library file descriptors.

> Under Linux if preprocessor macros _LARGEFILE_SOURCE,
> _LARGEFILE64_SOURCE, _FILE_OFFSET_BITS=64 are defined then off_t is
> 64 bit, otherwise it is 32 bit.

>> Access via iostreams will still have to use std::streamoff, since
>> that is out of my control; but you should be able to use an instance
>> of file_descriptor directly.
>>
>> The new code should be available soon.
>
> I think boost::iostreams::file_descriptor is what I need. Of course
> with 64 bit seek.

Okay, I'll make sure to add this.

> Another reason why I do not like std::fstream is its
> bad performance. It introduces a superfluous copying into it's internal
> buffer when doing I/O on user buffer.

You can also use a FILE* and specify _IONBF with setvbuf.

> Another issue is that the performance of my library will benefit from
> unbuffered file system access. Many operating system support it
> (O_DIRECT open option in Unix systems,
FILE_FLAG_NO_BUFFERING|FILE_FLAG_WRITE_THROUGH option for "CreateFile"
> Windows API call).

Could you point me to documentation for O_DIRECT? It looks like
FILE_FLAG_NO_BUFFERING requires some care to use properly, so I think it would
have to go into a specialized component instead of being added to
file_descriptor.

> Roman

Jonathan