[IO] Hosting test images

From: Mateusz Loskot (mateusz_at_[hidden])
Date: 2019-07-23 19:16:10

Hi,

FYI, although this is related to my recent thread about "Managing input images
for tests" [1], which is more about test images for core tests; oferring
"PNM as GIL built-in dependency-less human-friendly easy-debuggable formats"
always supported by core IO and useful for testing, issues reporting, etc. :-)
This thread here is more specific to test images for test/extension/io stuff.

I'm slowly preparing for I/O tests overhaul with these major goals:

1. Improve structure of test/extension/io tests, physical and logical
   (i.e. targets, in both, Boost.Build and CMake)
2. Improve distinction between basic, light and quick VS thorough, fat and
   slow tests; make clear what is tested on CI and what is tested locally.
3. Get rid of tweaking tests content with compile-time macros:
   BOOST_GIL_IO_TEST_ALLOW_READING_IMAGES
   BOOST_GIL_IO_TEST_ALLOW_READING_IMAGES
   BOOST_GIL_IO_USE_PNG_TEST_SUITE_IMAGES
   Shortly, most of this terrible complexity displayed by
   the "Compiler Symbols" table [2] must go!
4. Maintain collection of test images for I/O tests.

If you know about any other issues/annoyances related to this, please share.

First we need to decide how to approach hosting of test images.
So, let's consider some feasible options and their pros and cons:

1. Keep test images required for tests inside GIL repo
    + simplifies overall maintenance
    + makes it easier for contributors to run completely tests coverage
    - affects size of repository and Boost distribution
    - requires maintenance of .gitattributes
    - may require use of Git LFS

    Examples: OpenCV, https://github.com/OSGeo/gdal/

2. Keep test images in external repository or archive
    + keeps repository and Boost distribution size smaller
    - extra maintenance to keep tests and data in sync
    - extra scripts to download and deploy images for tests on CI and locally

    Examples: https://github.com/JuliaImages (see below)

3. Identify public sets of standard test images vailable online (e.g. [3], [4])
    + keeps repository and Boost distribution size smaller
    - extra scripts to download and deploy images for tests on CI and locally

    Examples: none found

Keep in mind, collection of test images may grow, in files number and disk size.
For example, addressing a GitHub issue may bring in new test image(s) specific to
testing a bug fix. For example, see my PR with tests for PNG tRNS [5] which
addressed issues reported by FreeOrion project.

Have I missed any other options, any pros/cons? Please, complete my list.

------
Note on JuliaImages, which is an intersting case, which I think is based on
the option 2. The developers of the image processing packages for Julia language
maintain dedicated repository with test images [6]. The repository is used as Julia
package loadable at run-time offering API to find and load image by name.

If we choose the option 2., likely, we will want to achieve something similar, but
implemented as a hybrid semi-automatic solution:

1. User or CI runs script to deploy test images in known location

   libs/gil/test/get-test-images.{sh|ps1} --target-dir libs/gil/test/images/

2. C++ test cases load images from known location using dedicated fixture

   rgb8_image_t i = fixture::load_test_image('lena/lena_gray_16bit.png');

In details, the known default location can be overriden by user for both, the
script and the C++ test cases using dedicated command line argument (also
passed via CMake, if configured, etc.). Things can be tweaked for convenience,
but that is implementation detail, not important at this stage.
------

Question: what is our preferred way to host test images?

Although it is a bit more work, I am inclined towards the option 2. with
JuliaImages-like approach outlined above.

[1] https://lists.boost.org/boost-gil/2019/06/0252.php
[2] https://github.com/boostorg/gil/blob/69adf0e8a544d1575ae5947973b1325c07c1d907/doc/io.rst
[3] https://en.wikipedia.org/wiki/Standard_test_image
[4] https://testimages.org
[5] https://github.com/boostorg/gil/pull/125
[6] https://github.com/JuliaImages/TestImages.jl

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net
Fingerprint=C081 EA1B 4AFB 7C19 38BA  9C88 928D 7C2A BB2A C1F2