[boost] Re: A library: out-of-core containers and algorithms

From: Roman Dementiev (dementiev_at_[hidden])
Date: 2005-02-09 05:12:21

Jonathan Turkanis wrote:
> Roman Dementiev wrote:
> 
>>Jonathan Turkanis wrote:
>>
>>>Roman Dementiev wrote:
> 
> 
>>>>The Boost filesystem library is more about manipulating files and
>>>>directories. But what I need is the file access itself:
>>>>create/open/read/write/close file. I would have been using
>>>>std::fstream as a portable file access method, but it
>>>>lacks support of files larger than 2 GB. It is really big
>>>>disadvantage in my case.
> 
> 
>>>The boost iostreams library, which is in CVS now and will be part of
>>>the next boost release, contains a class file_descriptor for
>>>accessing files using OS or runtime library file descriptors.
> 
> 
>>Under Linux if preprocessor macros _LARGEFILE_SOURCE,
>>_LARGEFILE64_SOURCE, _FILE_OFFSET_BITS=64 are defined then off_t is
>>64 bit, otherwise it is 32 bit.
> 
> 
>>>Access via iostreams will still have to use std::streamoff, since
>>>that is out of my control; but you should be able to use an instance
>>>of file_descriptor directly.
>>>
>>>The new code should be available soon.
>>
>>I think boost::iostreams::file_descriptor is what I need. Of course
>>with 64 bit seek.
> 
> 
> Okay, I'll make sure to add this.
> 
> 
>>Another reason why I do not like std::fstream is its
>>bad performance. It introduces a superfluous copying into it's internal
>>buffer when doing I/O on user buffer.
> 
> 
> You can also use a FILE* and specify _IONBF with setvbuf.
> 
> 
>>Another issue is that the performance of my library will benefit from
>>unbuffered file system access. Many operating system support it
>>(O_DIRECT open option in Unix systems,
> 
> FILE_FLAG_NO_BUFFERING|FILE_FLAG_WRITE_THROUGH option for "CreateFile"
> 
>>Windows API call).
> 
> 
> Could you point me to documentation for O_DIRECT? 
man 2 open
http://www.die.net/doc/linux/man/man2/open.2.html
http://www.mcsr.olemiss.edu/cgi-bin/man-cgi?open+2

> It looks like
> FILE_FLAG_NO_BUFFERING requires some care to use properly, so I think it would
> have to go into a specialized component instead of being added to
> file_descriptor.
Do you mean the offset requirements described in
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp 
?

In particular Stxxl file access patterns meet these requirements.

Roman