Boost logo

Boost Users :

Subject: Re: [Boost-users] [interprocess] Reading huge files
From: Sensei (senseiwa_at_[hidden])
Date: 2013-10-14 05:45:15

On 10/11/13 7:14 PM, Brian Budge wrote:
> If you're on a 64-bit system, you can simply mmap the entire file.
> There is no need to break the file into regions just because it's huge
> :) The OS will page the data in as required. On 32-bit, you do need
> to manage regions because you would otherwise exceed your address
> space. This might be kinda crappy as you'd ideally want to split your
> regions at EOL boundaries, and you need to parse your file before you
> know where these are. In practice, you'd be stuck worrying about
> straddling EOL, but hey, that's the price you pay if you want to run
> 32-bit code.

But on 32-bit systems I need to say "hey this program will go fubar as
you load big files, use it at your own peril!" :)

>> Another side-question, if you don't mind. I'm not sure that what I'm doing
>> is efficient, especially the need to copy from the region to a string. If
>> you have suggestions, I'm more than happy to hear these.
> I would use boost's new string_ref instead of string. The obvious
> solution would be to use boost.tokenizer to break up the giant string
> into string_ref lines; however, I'm unsure that this is supported yet.
> An EOL tokenizer should be only a few lines of code though, and you
> could fairly trivially tokenize your string into string_refs.

Awesome classes, I will try them!


Boost-users list run by williamkempf at, kalb at, bjorn.karlsson at, gregod at, wekempf at