Subject: Re: [boost] gil::io "non-review" (was: [gil] Can not open test.jpg)
From: Mateusz Loskot (mateusz_at_[hidden])
Date: 2010-03-22 20:43:40
Phil Endecott wrote:
> Christian Henning wrote:
>>>> - Scaled Decoding: Todo - Never figured out how to do that. Couldn't
>>>> find anything on the net.
>>>> - Partial Image Decoding - Same as Scaled Decoding
>>> Well, I think both of those things are described in the libjpeg
>>> documentation. ?If you can't find it and are interested in implementing
>>> it, please let me know.
>> I would be interested in a pointer to details of Partial Image
>> Decoding. Depending on the amount of effort I'll either add it now or
>> after the review. My time frame is kinda small right now.
> I think the situation was that you were decoding and discarding the
> stuff after the area of interest, which was obviously wasteful and
> fixable. The more difficult issue was whether it is possible to reduce
> the work done prior to the area of interest. I don't have a good
> enough understanding of libjpeg to know the answer to that. Presumably
> there are similar issues with the other formats.
AFAIU if not fast ROI access (or tiled) is supported by a format, then by
scanline access is necessary. I'm not libjpeg expert, but it's scanline
For example, here is IReadBlock function of JPEG driver from GDAL library:
It uses LoadScanline function:
IReadBlock performs optimal reading for particular format and raster size.
In JPEG, block is a single scanline: N-COLUMNS x 1-ROW
In TIFF, it is possible to perform tile-based reading, then I/O is based
on blocks of tiles but not scanlines.
It is noted in performance optimisation guidelines for warping
(resampling, etc.) here http://www.gdal.org/warptut.html
I give references to GDAL library because it has been my main raster
I/O and processing tool for long time and I have some experience with it.
I think it could serve as a good reference on how to perform I/O
efficiently against large datasets in various raster formats.
Personally, I'm strongly interested in developing some of GDAL strategies
for Boost.GIL, as extensions, for instance some algorithms, resampling,
I/O strategies support (scanline, tiles, etc.)
>>>> - Rotation: Same state as Scaled Decoding.
>>> I agree that this is not well described anywhere, but the program
>>> can do it.
>> I consider this a "nice to have". This extension is about IO but again
>> if it's easy to integrate I'll do it.
> Unfortunately it's not always possible to respect partitioning like
> "this bit is IO" when that breaks the performance. But again, I still
> don't know how to do this rotation. I have a feeling that there really
> aren't many people around who actually understand libjpeg properly...
I may be missing what is this idea of rotation here, but if
it's rotate while decoding, I doubt it's possible.
>>>> - In-memory jpeg: This is done. Please see unit tests.
>>> The last time that I looked at your code - about a year ago - my
>>> problem was
>>> to take a very large TIFF and to chop it up into 256x256 PNG tiles. ?
>>> wanted to do this without having to keep all of the image data in
>>> simultaneously. ?This seemed to be beyond what you could offer. ?I'm
>>> whether or not it's fundamentally possible with gil or not; I had hoped
>>> its idea of "views" would make it possible if the views could lazily
>>> the data from the source image. ?Even if gil can't do this, it would be
>>> if your libtiff/png/jpeg wrappers were usable for general C++-ing of
>>> those libraries.
>> You can read subimages from a tiff image.
Do you mean subimages of multi-image tiff?
There is also an idea of overviews and tiles. I'm not sure if all
TIFF container variations are supported in the GIL I/O.
I hope to find out while I'm reviewing it.
>> Once that's done you have a gil image which you make into a png
> So for my problem of chopping up the TIFF into lots of tiles, I think I
> have two choices:
> (1) Decode the whole TIFF into one gil image. For each tile, create a
> gil view and create and save a PNG.
> (2) For each tile, partially decode the TIFF into a gil image, and
> create and save a PNG.
> The first choice uses lots of RAM. The second choice uses lots of time
> because the partial decoding has to decode and discard all of the
> content that precedes the area of interest. Is that true?
You may try to read by tiles or whole strips of scanlines and cut them to
-- Mateusz Loskot http://mateusz.loskot.net
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk