Boost logo

Boost :

From: dylan_at_[hidden]
Date: 2001-11-20 01:38:24


Has anyone tried implementing something to support huge containers
that couldn't possibly fit into memory (even using swap space). I'm
not talking about sparse arrays, I mean where you actually need 8Gig
of space to store information that you are working on actively, and
need to be able to access it transparently without caring whether it
is stored on disk or in memory (so that, for instance, you can
dynamically adjust how much is stored in memory and how much on disk).
8 gig is just about enough to hold the matrices necessary to do
certain forms of boundary element and finite element analysis on
complex input models, so this is not a purely theoretical need!

I tried to think about what would be involved in implementating, say,
std::vector so that you could specify that a certain amount would be
cached in memory, while the rest was manually read and written to
disk. One implication is that operations that reduce the size of the
vector (including destruction) would be potentially extremely slow as
all the elements being destructed would have to be "swapped" in first.
An option to disable this would probably be necessary (user beware of

Ideally of course it should be possible by implementing just an
custom allocator. Assuming that a vector's iterator ultimately uses
allocator::pointer then if you write an allocator where pointer is
actually a 'smart pointer' that swaps as needed, this would probably
be 90% of the work.

Well actually "ideally" the OS would handle all this for you, but the
reality is that very few OS's are set up to optimally use all
available disk space for swapping. Even most unices don't seem to
handle trying to configure more than 4 gig of swap space too well,
and NT is doing well if it handles 4 meg...
Anyway size_t is usually a 32-bit int on 32-bit systems which severly
limits how much memory new[] can handle - I'm making the assumption
that size_type will be 64-bit.



Boost list run by bdawes at, gregod at, cpdaniel at, john at