Re: [iostreams] Interest in multithreaded lzma filter?

From: James E. King III (jking_at_[hidden])
Date: 2019-05-23 00:04:55

On Wed, May 22, 2019 at 5:40 PM Jeff Bonyun <jeff_at_[hidden]> wrote:
>
> > ...  How will we detect whether
> > liblzma is capable?  If it is not capable but specified, will we allow
> > the compression anyway?
>
> I've been investigating this. liblzma defines a version macro,
> LZMA_VERSION. It looks like liblzma obtained its multithreaded function
> in 5.1.1 (in 2011). So the idea would be to branch on this macro, and
> fall back to the currently-used single-threaded function for an older
> liblzma version. In that case, requesting any number of threads will
> carry out the operation, but will only use a single thread. This makes
> it tolerant of different versions, but does mark up the code with a few
> #if branches. I'm assuming boost goes for tolerance in this case.
>
> > Is there also an option to have it use the number of cores the
> > operating system is advertising?
>
> I'm glad you asked! The xz utility, that liblzma is packaged with, does
> this. But it isn't inherent to liblzma itself, which considers "0"
> threads to be an error. Copying the approach of xz, I'm intending to use
> lzma_cputhreads, which is exposed by liblzma. This was added in 5.2.0
> (2014); prior to that it used an internal function that I don't think we
> should/can use. This would necessitate yet another #if branch. Assuming
> this is acceptable, using "0" threads would indicate that this all-cores
> method is desired. This matches the xz utility's approach. For liblzma
> versions before 5.2.0, a value of "0" would be translated into "1"
> instead. This isn't ideal, as the user clearly wanted more than 1
> thread, but I'm not sure we should guess how many the user wanted.
>
> It would still fail in one situation: if your liblzma was built with
> `--disable-threads`. In this case, the documentation says that
> lzma_cputhreads is omitted. I would imagine it would result in a
> link-time error when building Boost.iostreams. I do not have a solution
> to this problem, because I don't know how to detect this before the link
> error. The best I can offer is a `#define BOOST_*` to disable it
> manually if the user runs into this problem.
>
> An alternative is using Boost.Thread, which would solve the problem
> gracefully. But I was going to avoid pulling in the extra library, and
> not do this. If you think the Official Boost Way would be to use
> Boost.Thread, I can change directions.
>

Fewer dependencies is better, so if lzma exposes something you can use that.

> >
> >> If there is appetite for it, I can make a pull request on github.
> > That's the best way to get started.  Be sure to include tests that exercise it.
>
> I'll work on it! The bones are done. It's the tests, particular
> formatting for boost, and testing-with-different-liblzma versions that
> needs to be done.
>
> Jeff

Thanks,

Jim