[boost] [iostreams][1.33.1] Re: possible code_converter bug?

From: Jonathan Turkanis (technews_at_[hidden])
Date: 2005-10-12 12:35:28

[Doug, I think I'd like to fix this problem. The noconv case is not
covered by the regression test; if it were, this problem would have
been noticed before]

Niklas Wiberg wrote:

> Hi,
> Sorry for mailing you privately, but this feels a bit too detailed to
> bother the list readers with...

I don't mind your writing me privately, but we discuss lots of detailed
things on the list, and someone else might have some good insight. With
your permission, I'll post this to the list.

> I'm using a "null codecvt" as described at
>
http://www.codeproject.com/vcpp/stl/upgradingstlappstounicode.asp?df=100&forumid=16224&exp=0&select=596642
>
> together with code_converter.
>
> In code_converter<Device, Codevt, Alloc>::write(), there is a piece of
> code for handling noconv results:
> case codecvt_base::noconv:
>             {
>                 // This can be shortened to two memcpy's.
>                 const char* c = (const char*) (s + total);
>                 for ( std::size_t index = 0;
>                       index < sizeof(intern_type);
>                       index += sizeof(extern_type),
>                       ++buf.ptr() )
>                 {
>                     memcpy(buf.ptr(), c + index, sizeof(extern_type));
>                     if (buf.eptr() == buf.end())
>                         buf.flush(dev());
>                 }
>                 ++total;
>             }
>
> eptr() is never increased. Can that be correct?
> If I debug and look at the call to buffer::flush() (which is called when
> my stream is flushed, not by this code), I can see that
> streamsize amt = static_cast<std::streamsize>(eptr_ - ptr_);
> results in amt being negative since ptr_ is larger than eptr_.
> Perhaps that's why I don't get anything in my file?

This code probably doesn't do what I wanted it to do, but it turns
out it doesn't matter, since there's a more fundamental problem. When
I wrote this, I was working from the 1998 standard; in the 2003 edition,
the role of noconv was clarified, as follows:

   result do_out(stateT& state,
    const internT* from, const internT* from_end, const internT*& from_next,
    externT* to, externT* to_limit, externT*& to_next) const;

   result do_in(stateT& state,
    const externT* from, const externT* from_end, const externT*& from_next,
    internT* to, internT* to_limit, internT*& to_next) const;

   .... If returns noconv, internT and externT are the same type and
   the converted sequence is identical to the input sequence
   [from, from_next).

As a result, the NullCodecvt in the article you cite should never be
returning noconv, since wchar_t != char.

The next version of code_converter will handle noconv correctly, and may
also be more correct in other respects. Unfortunately, it will have to
wait for 1.34.

> Niklas Wiberg

-- 
Jonathan Turkanis
www.kangaroologic.com