Re: [boost] [utf8_codecvt_facet]

Subject: Re: [boost] [utf8_codecvt_facet]
From: Marshall Clow (mclow.lists_at_[hidden])
Date: 2012-12-11 09:32:55

On Dec 10, 2012, at 9:27 PM, Robert Ramey <ramey_at_[hidden]> wrote:

> change Revision: 81616
> 
> Author: marshall
> Date: 11:12:42 AM, Wednesday, November 28, 2012
> Message:
> Fixed incorrect 'const' param for utf8_codecvt_facet::do_length
> ----
> Modified : /trunk/boost/detail/utf8_codecvt_facet.hpp
> 
> This change causes the a failure in the serialization library test
> suite with "test_utf8_codecvt" on all platforms which use
> the gcc standard library.

Ok, I'm seeing that too - the test doesn't "fail" :-(, it just hangs.

> I'm not sure what motivated the change or what it addressed
> but I'm pretty sure it's causing this (and a couple other)
> test failures.  Note that all the failures are related to usage
> of wchar_t - perhaps that helps.

Back on June 28th, I posted a message to the list titled "[Survey] utf8_codecvt_facet::do_length parameter" which described the problem:

> Marshall Clow <mclow.lists_at_[hidden]>
> To: boost_at_[hidden] List <boost_at_[hidden]>
> [Survey] utf8_codecvt_facet::do_length parameter
> 
> There's some confusion in boost (and in various standard libraries) as to the declaration of 
> 	std::codecvt<>::do_length
> 
> The '03 and '11 standards say:
>> 	template <class internT, class externT, class stateT>
>> 	class codecvt : public locale::facet, public codecvt_base {
>> 	// blah blah blah
>> 		virtual int do_length(stateT&, const externT* from, const externT* end, size_t max) const;
>> 	};
> 
> But some standard libraries (notably older Dinkumware libraries, and Visual C++ before Visual Studio11) say:
>> 	virtual int do_length(const stateT&, const externT* from, const externT* end, size_t max) const;
> (the first parameter is a const reference)
> 
> There's some preprocessor stuff to deal with this in boost/detail/utf8_codecvt_facet.hpp, but it's wrong:
> 
>> #if !defined(__MSL_CPP__) && !defined(__LIBCOMO__) \
>>    && !defined(_LIBCPP_VERSION)
>>    #define BOOST_CODECVT_DO_LENGTH_CONST const
>> #else
>>    #define BOOST_CODECVT_DO_LENGTH_CONST
>> #endif
>> 
>>    virtual int do_length(
>>        BOOST_CODECVT_DO_LENGTH_CONST std::mbstate_t &,
>>        const char * from,
>>        const char * from_end, 
>>        std::size_t max_limit
>> #if BOOST_WORKAROUND(__IBMCPP__, BOOST_TESTED_AT(600))
>>        ) const throw();
>> #else
>>        ) const;
>> #endif
> 
> I'm happy to fix this, but I'd like to be sure which systems actually have const parameters.
> 
> I've checked gcc and libc++, and they match the standard.
> STL assures me that all versions of Visual Studio before VS11 do not match, but VS11 does.
> The #defines above suggest to me that LibCOMO has this problem as well.
> Does anyone have access to such an install?
> 
> And if anyone's standard library (that I haven't listed) does define the first parameter to std::codecvt<>::do_length as a const reference, please let me know.

FWIW, I got zero responses.

Then Microsoft shipped Visual Studio 12, which also defined the parameter as "not const".
I looked at LibComo, and it too defines the parameter as non-const.

So, after discussing this with Beman (since it's his library), I changed the code to look like this:

//	Only do CONST on older Dinkumware libraries.
#if defined(_CPPLIB_VER) && (_CPPLIB_VER < 540)
    #define BOOST_CODECVT_DO_LENGTH_CONST const
#else
    #define BOOST_CODECVT_DO_LENGTH_CONST
#endif

If that's causing problems for you, I apologize, and I'll be happy to work with you to fix it - though I don't understand _how_ it's causing problems.
I tried changing the macro to always be "const", and the test still hung for me (on both clang and gcc)

Looking into this a bit more, it appears that Boost.IOStreams has it's own #define for this. Sigh.

-- Marshall

Marshall Clow     Idio Software   <mailto:mclow.lists_at_[hidden]>

A.D. 1517: Martin Luther nails his 95 Theses to the church door and is promptly moderated down to (-1, Flamebait).
        -- Yu Suzuki