Re: [BGL] Using BGL for huge graphs

From: Erik Arner (yg-boost-users_at_[hidden])
Date: 2003-02-03 05:11:30

Chris Russell wrote:
> Erik, how huge is huge? Heap requirements in excess of 2^64 bytes huge? If
> so, that's one _massive_ graph and some supercomputer programmer will have
> to help. 

Well, 2^64 bytes is several orders of magnitude larger than I had in 
mind. I guess my graph would be something like 1000 GB tops. It's still 
a pretty massive graph though...

> If not, then why not leverage the full virtual memory space
> allocated to your process and let the OS worry about paging the data in/out
> of physical memory?

This is definitely an alternative, but my worry is that this solution 
might be much slower than it needs to be. When swapping back and forth, 
the OS doesn't really know what's going on, whereas I might have the 
knowledge of the order in which the different parts of the graph are 
processed etc.

Then again, maybe this argument is bogus and the OS actually is better 
at deciding what parts should be kept in memory for the moment.

> Seems like if you can fit into 2^32 (Xeon workstation)
> or 2^64 then an Itanium or Alpha (4-processor Alpha servers running DEC UNIX
> are real cheap on eBay these days) with a fiber channel drive array (striped
> config for maximum throughput) might do the trick? If this is even close to
> reasonable suggestion then you might check out the non-linear video editing
> guys for ideas on drive arrays (e.g. http://www.avid.com,
> http://www.media100.com).

Thanks for the tip, it'll be worth looking into.

/Erik