Re: [boost] Re: Re: A library: out-of-core containers and algorithms (ajob for boost::iostreams)

From: christopher diggins (cdiggins_at_[hidden])
Date: 2005-02-07 23:58:23

----- Original Message ----- 
From: "Jonathan Turkanis" <technews_at_[hidden]>
>> ----- Original Message -----
>> From: "Roman Dementiev" <dementiev_at_[hidden]>

>> I would strongly support the introduction of a new library such as
>> proposed by Roman, however on the sole condition that it is portable.
>
> I don't see why it couldn't be made to work on a wide variety of systems.
> However, the initial goal should just be to support the most widely used
> systems. Porting to other systems can be done later as needed, preferably 
> with
> the people who need it helping with the porting.

Sounds to me like a great deal of work for something that isn't that 
complicated to begin with. The vector fstream solution I proposed could be 
up and running in a couple of days and would be extermely portable.

>> The current proposal to require native file access methods I think is
>> too limiting. I would propose instead writing a new version of fstream
>> which operates on vectors of files. This can be done relatively easily by
>> using the boost::iostreams library by Jonathan Turkanis, I have posted 
>> some
>> very preliminary prototype code (i.e. untested, naive, non-portable) at
>> http://www.cdiggins.com/big_file.hpp just to give a glimpse of one
>> possible approach. The code is modeled after the 
>> boost::iostreams::seekable
>> concept (
> http://home.comcast.net/~jturkanis/iostreams/libs/iostreams/doc/concepts/seekable_device.html
> )
>
> This is an intersting idea; I'd like to be convinced that it's necessary 
> before
> implementing it.

Because it's far easier to implement and is more portable than the other 
method of writing a whole bunch of ports, and testing all of the ports. Also 
I think it would be an excellent example for your iostreams library. I think 
Scott had something to say about performance of huge files. The file vector 
solution is probably significantly more efficient.

> I think there will be some tricky issues similar to the ones encountered 
> when
> implementing temp files.

The library I assume, already has dealt with the problem of temp files, so 
the solution should likely be the same.

> Specifically, you can't assume that the names file.1,
> file.2, ... will always be available; when you need a new file, you have 
> to look
> for a name which is not used and create the file atomically. Also, the 
> naming
> convention should be customizable.

Yes.

>> If Jonathan heeds our call, he could probably finish what I started
>> in less than an hour. ;-)
>
> :-)

Done yet?

CD