huge container types?

From: dylan_at_[hidden]
Date: 2001-11-20 01:38:24

Hi,

Has anyone tried implementing something to support huge containers 
that couldn't possibly fit into memory (even using swap space). I'm 
not talking about sparse arrays, I mean where you actually need 8Gig 
of space to store information that you are working on actively, and 
need to be able to access it transparently without caring whether it 
is stored on disk or in memory (so that, for instance, you can 
dynamically adjust how much is stored in memory and how much on disk).
8 gig is just about enough to hold the matrices necessary to do 
certain forms of boundary element and finite element analysis on 
complex input models, so this is not a purely theoretical need!

I tried to think about what would be involved in implementating, say, 
std::vector so that you could specify that a certain amount would be 
cached in memory, while the rest was manually read and written to 
disk.  One implication is that operations that reduce the size of the 
vector (including destruction) would be potentially extremely slow as 
all the elements being destructed would have to be "swapped" in first.
An option to disable this would probably be necessary (user beware of 
course).

Ideally of course it should be possible by implementing just an 
custom allocator.  Assuming that a vector's iterator ultimately uses 
allocator::pointer then if you write an allocator where pointer is 
actually a 'smart pointer' that swaps as needed, this would probably 
be 90% of the work.

Well actually "ideally" the OS would handle all this for you, but the 
reality is that very few OS's are set up to optimally use all 
available disk space for swapping.  Even most unices don't seem to 
handle trying to configure more than 4 gig of swap space too well, 
and NT is doing well if it handles 4 meg...
Anyway size_t is usually a 32-bit int on 32-bit systems which severly 
limits how much memory new[] can handle - I'm making the assumption 
that size_type will be 64-bit.

Thoughts/comments...

Dylan