Boost logo

Boost :

Subject: Re: [boost] gil::io "non-review" (was: [gil] Can not open test.jpg)
From: Mateusz Loskot (mateusz_at_[hidden])
Date: 2010-03-22 20:43:40

Phil Endecott wrote:
> Christian Henning wrote:
>>>> - Scaled Decoding: Todo - Never figured out how to do that. Couldn't
>>>> find anything on the net.
>>>> - Partial Image Decoding - Same as Scaled Decoding
>>> Well, I think both of those things are described in the libjpeg
>>> documentation. ?If you can't find it and are interested in implementing
>>> it, please let me know.
>> I would be interested in a pointer to details of Partial Image
>> Decoding. Depending on the amount of effort I'll either add it now or
>> after the review. My time frame is kinda small right now.
> I think the situation was that you were decoding and discarding the
> stuff after the area of interest, which was obviously wasteful and
> fixable. The more difficult issue was whether it is possible to reduce
> the work done prior to the area of interest. I don't have a good
> enough understanding of libjpeg to know the answer to that. Presumably
> there are similar issues with the other formats.

AFAIU if not fast ROI access (or tiled) is supported by a format, then by
scanline access is necessary. I'm not libjpeg expert, but it's scanline

For example, here is IReadBlock function of JPEG driver from GDAL library:

It uses LoadScanline function:

IReadBlock performs optimal reading for particular format and raster size.
In JPEG, block is a single scanline: N-COLUMNS x 1-ROW

In TIFF, it is possible to perform tile-based reading, then I/O is based
on blocks of tiles but not scanlines.
It is noted in performance optimisation guidelines for warping
(resampling, etc.) here

I give references to GDAL library because it has been my main raster
I/O and processing tool for long time and I have some experience with it.
I think it could serve as a good reference on how to perform I/O
efficiently against large datasets in various raster formats.

Personally, I'm strongly interested in developing some of GDAL strategies
for Boost.GIL, as extensions, for instance some algorithms, resampling,
I/O strategies support (scanline, tiles, etc.)

>>>> - Rotation: Same state as Scaled Decoding.
>>> I agree that this is not well described anywhere, but the program
>>> jpegtran
>>> can do it.
>> I consider this a "nice to have". This extension is about IO but again
>> if it's easy to integrate I'll do it.
> Unfortunately it's not always possible to respect partitioning like
> "this bit is IO" when that breaks the performance. But again, I still
> don't know how to do this rotation. I have a feeling that there really
> aren't many people around who actually understand libjpeg properly...

I may be missing what is this idea of rotation here, but if
it's rotate while decoding, I doubt it's possible.

>>>> - In-memory jpeg: This is done. Please see unit tests.
>>> The last time that I looked at your code - about a year ago - my
>>> problem was
>>> to take a very large TIFF and to chop it up into 256x256 PNG tiles. ?
>>> ?I
>>> wanted to do this without having to keep all of the image data in
>>> memory
>>> simultaneously. ?This seemed to be beyond what you could offer. ?I'm
>>> unsure
>>> whether or not it's fundamentally possible with gil or not; I had hoped
>>> that
>>> its idea of "views" would make it possible if the views could lazily
>>> obtain
>>> the data from the source image. ?Even if gil can't do this, it would be
>>> good
>>> if your libtiff/png/jpeg wrappers were usable for general C++-ing of
>>> those libraries.
>> You can read subimages from a tiff image.

Do you mean subimages of multi-image tiff?
There is also an idea of overviews and tiles. I'm not sure if all
TIFF container variations are supported in the GIL I/O.
I hope to find out while I'm reviewing it.

>> Once that's done you have a gil image which you make into a png
> So for my problem of chopping up the TIFF into lots of tiles, I think I
> have two choices:
> (1) Decode the whole TIFF into one gil image. For each tile, create a
> gil view and create and save a PNG.
> (2) For each tile, partially decode the TIFF into a gil image, and
> create and save a PNG.
> The first choice uses lots of RAM. The second choice uses lots of time
> because the partial decoding has to decode and discard all of the
> content that precedes the area of interest. Is that true?

You may try to read by tiles or whole strips of scanlines and cut them to

Best regards,

Mateusz Loskot

Boost list run by bdawes at, gregod at, cpdaniel at, john at