Re: [iostreams] Interest in multithreaded lzma filter?

From: Jeff Bonyun (jeff_at_[hidden])
Date: 2019-05-22 21:40:37

> ...  How will we detect whether
> liblzma is capable?  If it is not capable but specified, will we allow
> the compression anyway?

I've been investigating this. liblzma defines a version macro, 
LZMA_VERSION. It looks like liblzma obtained its multithreaded function 
in 5.1.1 (in 2011). So the idea would be to branch on this macro, and 
fall back to the currently-used single-threaded function for an older 
liblzma version. In that case, requesting any number of threads will 
carry out the operation, but will only use a single thread. This makes 
it tolerant of different versions, but does mark up the code with a few 
#if branches. I'm assuming boost goes for tolerance in this case.

> Is there also an option to have it use the number of cores the
> operating system is advertising?

I'm glad you asked! The xz utility, that liblzma is packaged with, does 
this. But it isn't inherent to liblzma itself, which considers "0" 
threads to be an error. Copying the approach of xz, I'm intending to use 
lzma_cputhreads, which is exposed by liblzma. This was added in 5.2.0 
(2014); prior to that it used an internal function that I don't think we 
should/can use. This would necessitate yet another #if branch. Assuming 
this is acceptable, using "0" threads would indicate that this all-cores 
method is desired. This matches the xz utility's approach. For liblzma 
versions before 5.2.0, a value of "0" would be translated into "1" 
instead. This isn't ideal, as the user clearly wanted more than 1 
thread, but I'm not sure we should guess how many the user wanted.

It would still fail in one situation: if your liblzma was built with 
`--disable-threads`. In this case, the documentation says that 
lzma_cputhreads is omitted. I would imagine it would result in a 
link-time error when building Boost.iostreams. I do not have a solution 
to this problem, because I don't know how to detect this before the link 
error. The best I can offer is a `#define BOOST_*` to disable it 
manually if the user runs into this problem.

An alternative is using Boost.Thread, which would solve the problem 
gracefully. But I was going to avoid pulling in the extra library, and 
not do this. If you think the Official Boost Way would be to use 
Boost.Thread, I can change directions.

>
>> If there is appetite for it, I can make a pull request on github.
> That's the best way to get started.  Be sure to include tests that exercise it.

I'll work on it! The bones are done. It's the tests, particular 
formatting for boost, and testing-with-different-liblzma versions that 
needs to be done.

Jeff