[boost] Re: Boost Unicode support ideas

From: Miro Jurisic (macdev_at_[hidden])
Date: 2004-04-13 14:32:04

In article <87ptabsq2h.fsf_at_[hidden]>, Jeremy Maitin-Shepard <jbms_at_[hidden]> 
wrote:

> What it comes down to is that basic_string is designed with fixed-width
> character representations in mind.
> 
> I would be more in favor of creating a separate type to represent
> Unicode strings.

I agree with this completely. 

> > 4) define low level access to the core Unicode data properties (in
> > unidata.txt).
> 
> Reuse of the ICU library would probably be very helpful in this.
> 
> > 5) Begin to add locale support - a big job, probably a few facets at a
> > time.
> 
> The issue is that, despite what you say, most or all of the standard
> library facets are not suitable for use with Unicode strings.  For
> instance, the character classification and toupper-like operations need
> not be tied to a locale.  Furthermore, many of the operations such as
> toupper on a single character are not well defined, and rather must be
> defined as a string to string mapping.  Finally, the single-character
> type must be a 32-bit integer, while the code unit type will probably
> not be (since UTF-32 as the internal representation would be
> inefficient).

You are forgetting that abstract Unicode characters are defined as sequences of 
code points (even if those code points are 32-bit) and string manipulation has 
to take this into account (there are numerous combinations of characters and 
combining marks that must be treated as single units for purpose of searching, 
collation, etc.) A single encoded character type may be 32 bits, but encoded 
characters are often not the level on which the clients need to manipulate 
strings.

I am happy to see that there is someone here who knows more about locales in C++ 
than I do; I haven't had the time to research that as thoroughly as I would like 
to.

meeroh

-- 
If this message helped you, consider buying an item
from my wish list: <http://web.meeroh.org/wishlist>