[boost] Re: Re: Re: A library: out-of-core containers and algorithms(ajob for boost::iostreams)

From: Jonathan Turkanis (technews_at_[hidden])
Date: 2005-02-08 00:26:53

christopher diggins wrote:
> ----- Original Message -----
> From: "Jonathan Turkanis" <technews_at_[hidden]>
>>> ----- Original Message -----
>>> From: "Roman Dementiev" <dementiev_at_[hidden]>
>
>>> I would strongly support the introduction of a new library such as
>>> proposed by Roman, however on the sole condition that it is
>>> portable.
>>
>> I don't see why it couldn't be made to work on a wide variety of
>> systems. However, the initial goal should just be to support the
>> most widely used systems. Porting to other systems can be done later
>> as needed, preferably with the people who need it helping with the porting.
>
> Sounds to me like a great deal of work for something that isn't that
> complicated to begin with.

It's really not very hard to support several platforms using native APIs. See,
e.g., file_descriptor.cpp:

http://www.kangaroologic.com/iostreams/libs/iostreams/src/file_descriptor.cpp

> The vector fstream solution I proposed
> could be up and running in a couple of days and would be extermely portable.

I'm not yet sure what the portability issues are. If it's just the ability to
use large offsets, a couple of small changes to file_descriptor would be
sufficient.

>> This is an intersting idea; I'd like to be convinced that it's
>> necessary before implementing it.
>
> Because it's far easier to implement and is more portable than the
> other method of writing a whole bunch of ports, and testing all of
> the ports.

Often you just need two or three versions of a small bit of code; the boost
regression testing system makes testing on many platforms easy.

> Also I think it would be an excellent example for your
> iostreams library. I think Scott had something to say about
> performance of huge files. The file vector solution is probably
> significantly more efficient.

What interests me about this solution is that it uses a collection of files.
Whether the files are accessed using std::filebuf's or file_descriptors may turn
out to be not so important. Or it may turn out that using one or the other is
actually better for some reason. So there are several different issues here.

>> I think there will be some tricky issues similar to the ones
>> encountered when implementing temp files.
>
> The library I assume, already has dealt with the problem of temp
> files, so the solution should likely be the same.

No, unfortunately I haven't done it yet.

>> Specifically, you can't assume that the names file.1,
>> file.2, ... will always be available; when you need a new file, you
>> have to look
>> for a name which is not used and create the file atomically. Also,
>> the naming
>> convention should be customizable.
>
> Yes.
>
>>> If Jonathan heeds our call, he could probably finish what I started
>>> in less than an hour. ;-)
>>
>> :-)
>
> Done yet?

Give me five more minutes.

> CD

Jonathan