Boost logo

Boost :

Subject: Re: [boost] [gil] New IO release
From: Domagoj Saric (dsaritz_at_[hidden])
Date: 2010-10-24 16:07:08

"Christian Henning" <chhenning_at_[hidden]> wrote in message

> The reason I choose this unified interface is first of all historical.
> The current gil::io uses free functions as well, which are by the way
> not global but inside a namespace. I also provide header files which
> supports the old styles to avoid breaking existing code. The second
> reason is the ease of use. I believe using these free functions is
> easy for users to understand since they all have their own distinct
> behavior and meaning.

These are basically the same arguments you said the first time ('way back
then') and just like then I still fail to see any objective strength in

- 'historical' reasons make little or no importance if you are breaking
interface compatibility anyway
- the 'ease of use' argument seems purely subjective as is based solely on
your 'belief' and, more importantly, it contains no comparative substance in
itself even in a purely assertive form...IOW you do not try to claim to the
effect of "my interface is easier to use than yours" but simply "my
interface is easy to use" which, in itself, makes little importance as my
interface can also be "easy to use"...

To add some substance to the above, 'IO2' design/interface provides >both<
'quick'/'easy' interfaces like:
libXXX_image::read( <source>, <target> );
  where I fail to see any significant difference 'ease of use'-wise compared
  to the 'IO_new' interface:
  read_image( <source>, <target>, xxx_tag() );

as well as 'powerfull'/'advanced' interfaces like:
libXXX_image::reader_for<...>::type my_reader( <source> );
<do something with my_reader, even by accessing the underlying
my_reader.copy_to( <target>, <policies> );

Not to mention the possibilities that the 'full'/'generic' io2
interface/design leaves for possible future improvements and extensions
without interface breaking changes...

Finally, as far as I remember, in all discussions so far, of the few user
arguments/'wishes' that touched the issue of the interface of GIL.IO all
wished for an interface change exactly in the direction where IO2 went...

ps. placing a function inside a namespace does not make it non-global...

> Don't wanna get into a "streaming" fight here. Point is noted. Did you
> know there are several fast streams implementation available on the
> net? They might change some arguments.

std::streams are bad by design, so any non-bloated* IOStreams implementation
you find is certainly not std:: compliant making it irrelevant for this

* under which I mean 'not making you pay for what you do not use'...

>>>> Unlike io and io_new it uses objects that represent on-disk images
>>>> ("formatted images" in io2-speak). This has several advantages:
>>> Could you go into more details? What are "formatted images"?
>> Well, images in different 'formats' like JPEG, PNG, GIF...obviously not a
>> very clever name that I just came up with to somehow distinguish them
>> from
>> 'raw'/in-memory images (like gil::image<>) are more than welcome to
>> come up with a more intuitive name ;)
> How about in-memory vs. of-file? ;-)

As Mateusz noted, location of an image (in-memory, on-disk) is independent
of the 'format' with/in which the image is stored at that location...
The first two candidates that spring to mind for a name for the 'format'
that would designate what we understand under gil::image<> are 'raw' and
'uncompressed'. However both of them are 'unhappy' choices:
- the name 'raw' is also used for some image 'formats' that have layouts
that differ between each other as well as being far off from a 'simply
serialized gil::image<>'...And require separate 'codecs' for reading and
writing (e.g. digital camera 'raw' formats)
- the name/concept 'uncompressed' is also obviously flawed if we think of
'formatted' but 'uncompressed' images (e.g. BMP)...

If we throw in the concept of GIL's virtual images things get even more
complicated....Maybe we can/have to firstly distinguish between "virtual
images" (not backed by 'physical storage' but by 'algorithms') and "physical
images" (everything else)...And then somehow distinguish between what we
intuitively mean under gil::image<> (as "GIL's native raw format") and
"everything else" (JPEGs, PNGs, ...)...

Finally the 'concept' of format is also ambiguous, as it is sometimes used
to refer to 'formats' like JPEG and PNG and sometimes to 'formats' like RGB8
and CMYK16 (which maybe can be disambiguated by distinguishing between
"image formats" and "pixel formats")...

>> (AFAICT) You can reuse the low level file object but not the backend
>> library
>> object, which is what I was talking about and what is prerequisite for,
>> for
>> example, efficient 'moving ROI'/sequential-in-blocks image reading...
> Are you talking about FILE* only or does that include std::ifstream as
> well?

Um, to rewind things a little to make sure we are talking about the same
thing. I was talking about reusing the backend library objects/instances
(e.g. TIFF, jpeg_decompress_struct, ...), which IO2 supports and IO_new
AFAICT does not.
Simply reusing the low level source object (like a FILE) will make little
difference if, for every ROI you read, you have to recreate the backend
library object which in turn implies rereading the whole image up to the
requested ROI instead of just continuing where you left off...

>>>> libtiff_image my_tiff( "my_tiff.tiff" );
>>>> ::TIFFSetDirectory( &my_tiff.lib_object(), <a directory number> );
>>>> my_tiff.copy_to(....);
>>> That's a neat feature that I like to have too.
>> But I don't see how you can have that with free functions...
> Maybe I should give out access to my reader< ..., xxx_tag, ... > objects.

Again, I don't see how can you do that with free functions except by
allowing users to pass in low-level-object-modifier-functors to your global
interface functions...which IMO would pretty much defeat your original
argument of a 'simple and easy to use interface'...

>> No jamfile yet (Boost build/bjam are definitely not one of my favourite
>> subjects or skills for that matter :)
>> OTOH, I'm not quite sure (with my limited bjam knowledge) why/how would
>> different backends influence the library's jamfile (atleast as long as it
>> is
>> header only)..?
> Check out Mateusz's blog:

I checked it out and still could not see 'bjam-specific' material in IO2
that otherwise would not exist/be required for IO or IO_new...note that the
two additional backends I added require no 3rd party libraries (that need to
be built from source) but use existing native/OS functionality...

There is however the 'linking policy' issue that I currently solve purely
through source code/C++ and that may, at least in part, be solvable through
a build system (which is what I asked in the related thread)...

>>> I would love to know how you seek through a image using a 3rd party
>>> lib, for instance with libjpeg. Right now I'm just reading and
>>> discarding unwanted regions. Not the most ideal solution to say the
>>> least.
>> With LibJPEG you cannot really/literally skip unwanted data but you can
>> make
>> the library read the unwanted data using faster/less precise methods
>> which
>> is what I currently do (and admittedly use a bit of LibJPEG's internal
>> implementation detail knowledge for that)...
> Cool, I would love to know more. Can you point where in your code you do
> that?

In libjpeg_image.hpp search for "void skip_rows("...

> Yep, working on a fix. Tiff is very flexible and reading files without
> conversion works fine. But when reading with conversion I get into a
> problem with template member overloading. Basically I have a switch
> statement that would overload the read_data member with the on-file
> pixel format as template parameter. Now this pixel parameter can be
> anything for gray1 to rgb23_55_12, etc. Which makes the switch
> statement approach not feasible. I'm inclined to only allow
> read_and_convert_image/view for the most common channel sizes.

I use Steven Watanabe's switch_ for this. And yes, when it comes to LibTIFF
(and WIC) and the most generic case of runtime pixel format detection and
conversion, this approach can cause generation of an uber huge amount of
code of which the user may actually want to support/allow only a small
For this reason I planned to allow the user to specify compile-time filters
for allowed pixel formats so that the code for other pixel formats does not
get generated even if the backend supports them...

>>> That sounds great. I have to look at how you deal with certain
>>> functionalities, like reading ROI in a jpeg image.
>> Actually, I forgot to mention, with the LibX (JPEG, PNG, TIFF) backends I
>> do
>> not provide 'full'/2D/rectangular ROI capability but only 'vertical' ROIs
>> (i.e. you can specify from which to which row/scanline to read but whole
>> rows/scanlines are always read...) because the backends do not support
>> reading partial rows/scanlines so adding full/proper ROIs for those
>> backends
>> would require emulation which in turn would cause code bloat/complication
>> and (possible) redundant data copying...For now it seems to me that the
>> user
>> should handle 'full' ROI support in such cases (if required)...
> I try to allow all sizes of ROI. It can be anything from a single
> pixel to a scanline or a vertical line to a rectangle. This was bit**
> to implement for tiff's tiled images. ;-)
> The basic algorithm is to read scanline by scanline and decided if
> there are portions that are needed or not. Same for tiles.

Right, but this inefficient:
- it is slow because of all the branching and extra data copying
- it causes code bloat because the extra logic is template-replicated
slightly modified for all the possible source-target-backend combinations
and, when implemented like in IO_new, these overheads are unavoidable
(meaning that even users that do not need this ROI functionality will still
have to pay for it) making such a design decision 'really bad' because the
very idea of ROIs is almost entirely efficiency oriented/motivated...

For these reasons I still hold that for backends with inflexible/nonexisting
ROI related interfaces, like LibJPEG and LibPNG, full 2D ROI capability
should be provided by the user if required or a different interface should
be added that provides this capability so that you can also have it and not
pay for if you do not need it...
I had/have an idea how I could actually have it both ways (w/o an extra
interface) in IO2 through automatic compile-time detection of the "user's
wishes" but simply had no time to implement that also...

>> Place a breakpoint in the std::string is mostly due to
>> the
>> change you made to the io_error() helper function (to take a std::string
>> const & instead of char const *)...
> You're absolutely right. Are you recommending a string table?

No, simply revert your change and use char const * where they are
enough...unlike some think, a char const * is a perfectly valid C++

Also consider making friends with the disassembly window in release

> Are you suggesting your lib is not header-only?

Um, no...why did you get that impression?

However, IO2 has sufficient non-templated functionality to make it
non-header only...

> You're more than welcome to add stuff into the toolbox extension. Let me
> know.

For now my main priority is to get an efficient, flexible and powerful new
GIL.IO...the toolbox can/will have to wait ;)

"What Huxley teaches is that in the age of advanced technology, spiritual
devastation is more likely to come from an enemy with a smiling face than
from one whose countenance exudes suspicion and hate."
Neil Postman 

Boost list run by bdawes at, gregod at, cpdaniel at, john at