Re: [boost] Re: A library: out-of-core containers and algorithms (a job for boost::iostreams)

From: christopher diggins (cdiggins_at_[hidden])
Date: 2005-02-07 13:56:00

Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org
----- Original Message ----- 
From: "Roman Dementiev" <dementiev_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Monday, February 07, 2005 10:04 AM
Subject: [boost] Re: A library: out-of-core containers and algorithms

> Thorsten Ottosen wrote:
>> "Roman Dementiev" <dementiev_at_[hidden]> wrote in message 
>> news:cu7m3l$lfo$1_at_sea.gmane.org...
>> | Thorsten Ottosen wrote:
>> | > "Roman Dementiev" <dementiev_at_[hidden]> wrote in message
>> | > news:cu62pe$a1i$1_at_sea.gmane.org...
>> | > | Hello,
>> | > |
>> | > | I am developing a library called Stxxl.

>> | > | Would such a library be interesting for Boost users? Does it fit 
>> here?
>> | > | Is it worth to boostify it?

Yes.

>> | > if you intend to submit it, would it then be possible to make it
>> | > work on more platforms, unix (mac), windows?

I'd suggest that it should work on *all* platforms which support files, not 
just a handful.

>> | yes. One must rewrite only the lower layer of Stxxl using native
>> | file access methods and native multithreading.

[snip]

> But what I need is the file access itself: create/open/read/write/close 
> file. I would
> have been using std::fstream as a portable file access method, but it 
> lacks
> support of files larger than 2 GB. It is really big disadvantage in my 
> case.

On most (many/all?) windows platforms you can not have files of more than 
2GB, so I'd suggest you write the code to work with this restriction.

> The boost threads library does fit my requirements. I would use
> when it comes to boostifying Stxxl.

I would strongly support the introduction of a new library such as proposed 
by Roman, however on the sole condition that it is portable.

The current proposal to require native file access methods I think is too 
limiting. I would propose instead writing a new version of fstream which 
operates on vectors of files. This can be done relatively easily by using 
the boost::iostreams library by Jonathan Turkanis, I have posted some very 
preliminary prototype code (i.e. untested, naive, non-portable) at 
http://www.cdiggins.com/big_file.hpp just to give a glimpse of one possible 
approach. The code is modeled after the boost::iostreams::seekable concept 
( 
http://home.comcast.net/~jturkanis/iostreams/libs/iostreams/doc/concepts/seekable_device.html )

If Jonathan heeds our call, he could probably finish what I started in less 
than an hour. ;-)

Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org