Re: [boost] Designing a multi-threaded file parser

Subject: Re: [boost] Designing a multi-threaded file parser
From: Aaron Boxer (boxerab_at_[hidden])
Date: 2016-04-22 10:31:37

Hi Niall,

On Fri, Apr 22, 2016 at 8:18 AM, Niall Douglas <s_sourceforge_at_[hidden]>
wrote:

> On 21 Apr 2016 at 20:47, Aaron Boxer wrote:
>
> > The read thread will spend a lot of time waiting for file IO to
> > complete. Would it be faster to us the ASIO classes to do the reading?
>
> No, it won't.
>
> Memory mapped file i/o is almost guaranteed to be the correct
> technique to use here. Simply map the entire file into memory, and
> fire threads at processing it. Let the kernel figure out how best to
> do the i/o.
>

Thanks. I do have a memory mapped file interface in my library for the
compressed file,
so I can certainly try that.

My impression is that memory mapping is best when reading a file more than
once, because
the first read gets cached in virtual memory system, so subsequent reads
don't have to go to disk.
Also, it eliminates system calls, using simple buffer access instead

Since memory mapping acts as a cache, it can create memory pressure on the
virtual memory system,
as pages need to be recycled for the next usage. And this can slow things
down, particularly when reading
files whose total size meets are exceeds current physical memory.

In my case, I am reading the file only once, so I think the normal file IO
methods will be better.
Don't know until I benchmark.

Thanks again,
Aaron

>
> Niall
>
> --
> ned Productions Limited Consulting
> http://www.nedproductions.biz/
> http://ie.linkedin.com/in/nialldouglas/
>
>
>
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>