[boost] [Iostreams] Support std::codecvt<char> and std::codecvt_base::noconv in boost::iostreams::code_converter

From: Ferdinand Prantl (prantlf_at_[hidden])
Date: 2008-07-09 11:16:27

Hello,

I found two problems when using boost::iostreams::code_converter with char
as source (internal) type for the conversion. Firstly, I was not able to use
facets parameterized with char extracted from std::locale and secondly, my
test run in an endless loop if the facet returned std::codecvt_base::noconv
from do_in() or do_out().

I would like to use the boost::iostreams::code_converter in a class
parametrized either for char or wchar_t using the default std::codecvt.
This code works for wchar_t:

    namespace io = boost::iostreams;
    class test {
        io::stream<io::code_converter<xxxodevice,
            io::default_codecvt> > output;
        ...
    };

However I cannot make in templated like this without inheriting from
std::codecvt and pasing the inherited type to io::code_converter instead
of std::codecvt:

    template <class C> class test {
        io::stream<io::code_converter<xxxodevice,
            std::codecvt<C, char, std::mbstate_t> > output;
        ...
    };

The compiler complains about the usage of std::codecvt:

    error C2248: 'std::codecvt<_Elem,_Byte,_Statype>::~codecvt'
        : cannot access protected member declared in class
            'std::codecvt<_Elem,_Byte,_Statype>'

io::detail::codecvt_holder<Codecvt> in codecvt_holder.hpp stores a copy of
Codecvt but std::codecvt defines only protected destructor:

    template<typename Codecvt> struct codecvt_holder {
        ...
        Codecvt codecvt_;
    };

Standard std::codecvt specialized for wchar_t is solved by its explicit
specialization storing the pointer:

    struct default_codecvt {
        typedef wchar_t         intern_type, from_type;
        typedef char            extern_type, to_type;
        typedef std::mbstate_t  state_type;
    };

    template<> struct codecvt_holder<default_codecvt> {
        ...
        const codecvt_type* codecvt_;
    };

Obviously it supports the most usual usage with an instance extracted from
std::locale which owns it:

    typedef std::codecvt<wchar_t, char, std::mbstate_t> codecvt_type;
    ... = &std::use_facet<codecvt_type>(locale);

What about facets not converting from sequence of wchat_ts but chars? If I
imbue such facet to a locale I am not able to extract it and use it in
io::code_converter:

    template <class C> class test {
        io::stream<io::code_converter<xxxodevice,
            std::codecvt<C, char, std::mbstate_t> > output;

        void imbue(std::locale const & locale) {
            output.imbue(locale);
        }
    };

I helped it by converting io::detail::default_codecvt and the specialization
of io::detail::codecvt_holder for it in codecvt_holder.hpp to this:

    template <class C> struct basic_default_codecvt {
        typedef C intern_type, from_type;
        ...
    };

    template <class C> struct codecvt_holder<basic_default_codecvt<C> > {
        typedef std::codecvt<C, char, std::mbstate_t> codecvt_type;
        ...
    };

    typedef basic_default_codecvt<wchar_t> default_codecvt;

Now I was able to compile my slightly modified code:

    template <class C> class test {
        io::stream<io::code_converter<xxxodevice,
            io::detail::basic_default_codecvt<C> > output;

        void imbue(std::locale const & locale) {
            output.imbue(locale);
        }
    };

After being able to compile it a simple output code run endlessly. I found
out that the facet in the default locale did not convert but it did not
return true from always_noconv() from some reason. io::code_converter can
cope with it - it copies the input buffer to the output one but it does not
increase the pointer to the next character in the output buffer nor does it
decrease the total bytes processed in io::code_converter::write.

I am not sure about the behavior of the coverter if the facet returns
std::codecvt_base::noconv. Currently it copies the buffer if internal type
is the same as external one (both char) otherwise it does nothing (?).
Should it not copy or assert if they are diferent?

At least the writing should succeed if there was no conversion done and the
characters were copied. I tried to modify the code accordingly and it
succeeded then.

I am attaching a patch to code_converter.hpp and codecvt_holder.hpp from
boost 1.35.0. I could provide a test for it too.

Thank you,
Ferda