Re: [boost] ACID transactions/flushing/MySQL InnoDB

Subject: Re: [boost] ACID transactions/flushing/MySQL InnoDB
From: Stefan Strasser (strasser_at_[hidden])
Date: 2009-12-10 18:57:43

Am Thursday 10 December 2009 22:38:54 schrieb Dean Michael Berris:
> On Fri, Dec 11, 2009 at 7:02 AM, Stefan Strasser <strasser_at_[hidden]> 
wrote:
> > Am Thursday 10 December 2009 21:47:53 schrieb Dean Michael Berris:
> >> When you say logged on commit, are you writing to a file that's
> >> already created with enough space, or are you writing to a file which
> >> is "grown" every time you add data to it?
> >
> > I had tried that. and I've tried writing a sector instead of 1 byte. and
> > I've tried removing O_CREAT.
> > but you have to actually do ALL THREE, and one more: the sector writes
> > need to be sector-aligned.
> > so, writing 512 bytes, aligned to 512 bytes, without O_CREAT, when the
> > file already exists, brings the desired results
> > 2 seconds with much less disk usage.
> > that's some set of conditions.
> >
> > thanks for helping with this.
>
> Nice. :) You're welcome.

it's actually 1.2 seconds. I fergot to remove a call to this_thread::sleep(), 
from some earlier tests ;-)
and there's a fifth condition: no lseek() calls in between. a call to lseek() 
not only has its normal overhead, but brings the whole thing back to the 5-7 
seconds range.
so I'll have to waste some log space to ensure sector writes without seeking.

> > that's almost equal to my approach. I do write to the data files, but
> > only sync them when a large transaction is committed or the log is rolled
> > to a new one.
> > I should probably think about sector-aligning those data writes, too,
> > given the new insights.
>
> Sounds like a good approach. If you're thinking of multi-threading and
> having an active object do the flush management, that should be
> something worth looking into to move the latency from persistence away
> from "worker" threads to a single serializing writer thread.

there's already a TODO in the code to seperate the sync preparations (flushing 
internal buffer) from the actual system call, so the mutex can be unlocked 
and other threads can continue to write to the log.
but it is some effort in my current design, and I'm not sure if it even helps. 
according to this report, fsync() currently blocks other writes:
http://bugzilla.kernel.org/show_bug.cgi?id=9911
not sure about windows.