[boost] Re: Re: A library: out-of-core containers and algorithms (ajob for boost::iostreams)

From: Jonathan Turkanis (technews_at_[hidden])
Date: 2005-02-07 17:05:48

christopher diggins wrote:
> Christopher Diggins
> Object Oriented Template Library (OOTL)
> http://www.ootl.org
> ----- Original Message -----
> From: "Roman Dementiev" <dementiev_at_[hidden]>
> To: <boost_at_[hidden]>
> Sent: Monday, February 07, 2005 10:04 AM
> Subject: [boost] Re: A library: out-of-core containers and algorithms

>> Thorsten Ottosen wrote:
>>> "Roman Dementiev" <dementiev_at_[hidden]> wrote in message
>>> news:cu7m3l$lfo$1_at_sea.gmane.org...
>>>> Thorsten Ottosen wrote:
>>>>> "Roman Dementiev" <dementiev_at_[hidden]> wrote in message
>>>>> news:cu62pe$a1i$1_at_sea.gmane.org...
>>>>>> Hello,
>>>>>>
>>>>>> I am developing a library called Stxxl.
>
>
>>>>>> Would such a library be interesting for Boost users? Does it fit
>>> here?
>>>>>> Is it worth to boostify it?
>
> Yes.
>
>>>>> if you intend to submit it, would it then be possible to make it
>>>>> work on more platforms, unix (mac), windows?
>
> I'd suggest that it should work on *all* platforms which support
> files, not
> just a handful.

> I would strongly support the introduction of a new library such as
> proposed by Roman, however on the sole condition that it is portable.

I don't see why it couldn't be made to work on a wide variety of systems.
However, the initial goal should just be to support the most widely used
systems. Porting to other systems can be done later as needed, preferably with
the people who need it helping with the porting.

> The current proposal to require native file access methods I think is
> too limiting. I would propose instead writing a new version of fstream
> which operates on vectors of files. This can be done relatively easily by
> using the boost::iostreams library by Jonathan Turkanis, I have posted some
> very preliminary prototype code (i.e. untested, naive, non-portable) at
> http://www.cdiggins.com/big_file.hpp just to give a glimpse of one
> possible approach. The code is modeled after the boost::iostreams::seekable
> concept (
http://home.comcast.net/~jturkanis/iostreams/libs/iostreams/doc/concepts/seekable_device.html
)

This is an intersting idea; I'd like to be convinced that it's necessary before
implementing it.

I think there will be some tricky issues similar to the ones encountered when
implementing temp files. Specifically, you can't assume that the names file.1,
file.2, ... will always be available; when you need a new file, you have to look
for a name which is not used and create the file atomically. Also, the naming
convention should be customizable.

> If Jonathan heeds our call, he could probably finish what I started
> in less than an hour. ;-)

:-)

> Christopher Diggins
> Object Oriented Template Library (OOTL)
> http://www.ootl.org

Jonathan