Re: [boost] [General] Always treat std::strings as UTF-8

Subject: Re: [boost] [General] Always treat std::strings as UTF-8
From: Patrick Horgan (phorgan1_at_[hidden])
Date: 2011-01-18 16:18:58

On 01/18/2011 08:23 AM, Chad Nelson wrote:
> On Tue, 18 Jan 2011 16:04:29 +0000
> Alexander Lamaison<awl03_at_[hidden]>  wrote:
>
>> On Tue, 18 Jan 2011 10:54:57 -0500, Chad Nelson wrote:
>>
>>> Why delegate them to another library? Those classes already have
>>> efficient, flexible, and correct iterator-based template code for the
>>> conversions between the UTF-* types. I'd rather just farm out the
>>> stuff that those types are weak at, like converting to and from
>>> system-specific locales.
>> If they can do that, that's great!  The conversion code was so short
>> that I assumed it wasn't a full, complete conversion algorithm.
> They're complete, and accurate. The algorithms aren't overly complex,
> they just translate between different forms of the exact same data,
> after all.
If you can assume that the encoding is correct already that's true.  
Most the code to convert from utf-8 to utf-32 or utf-16, for example, is 
to check that you don't have overly long encodings that cause security 
issues or other violations of the well-formedness table in the unicode 
spec.  Otherwise, especially if you carry things around in utf-8 by 
preference, and do your checking in that encoding, you open yourself up 
to problems. (http://capec.mitre.org/data/definitions/80.html).  If you 
don't ever accept utf-8 encoded things from users, of course, you don't 
have to worry about this, but I would write the conversion defensively.

I should say that I haven't read your code yet and you might very well 
do this correctly.  The code conversion facet used by a lot of boost 
code doesn't.  It was written to an older version of the spec for utf-8 
and allows 5 and 6 character encodings.  It does have these security 
concerns.  I offered awhile back to replace it, but assume that with the 
locale stuff coming up for review it would be better to go with that.  I 
did write a replacement for utf8_codecvt_facet.cpp 
utf8_codecvt_facet.hpp that could be dropped in for the use of 
serialization and passes the tests in that part of boost.

Patrick