Subject: Re: [boost] Stacking iterators vs. dataflow
From: Sebastian Redl (sebastian.redl_at_[hidden])
Date: 2008-09-03 15:23:16
Phil Endecott wrote:
> Here's a practical example:
> cat email_with_attached_picture | decode_base64 | decode_jpeg |
> resize_image > /dev/framebuffer
> How can I convert that shell pipeline into C++? Naive approach:
> vector<byte> a = read_file("/path/to/email");
> vector<byte> b = decode_base64(a);
> vector<byte> c = decode_jpeg(b);
> vector<byte> d = resize_image(c);
> The problem with that is that I don't start to decode anything until
> I've read in the whole of the input. The system would be perceptibly
> faster if the decoding could start as soon as the first data were
> So I can use some sort of iterator adaptor stack or dataflow graph to
> process the data piece at a time. But it's important that I process
> it in pieces of the right size. Base64 encoding converts 6 input
> bytes into 4 output bytes, but it would be a bad idea to read the data
> from the file 6 bytes at a time; we should probably ask for BUFSZ
> bytes. libjpeg works in terms of lines, and you can ask it (at
> runtime, after it has read the file header) how many lines it suggests
> processing at a time (it's probably the height of the DCT blocks in
> the image). Obviously that corresponds to a variable number of bytes
> in the input.
> I would love to see how readers would approach this problem using the
> various existing and proposed libraries.
Since we're talking about systems that combine various approaches to
data processing, I should point out my IOChain library once more.
It does something very similar to what you describe here.
IOStreams also does something like this with its filters.
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk