Re: [boost] [General] Always treat std::strings as UTF-8

Subject: Re: [boost] [General] Always treat std::strings as UTF-8
From: Artyom (artyomtnk_at_[hidden])
Date: 2011-01-17 13:09:13

> I've done some research,  and it looks like it would require little
> effort to create an os::string_t  type that uses the current locale, and
> assume all raw std::strings that  contain eight-bit values are coded in
> that instead.
> 
> Design-wise,  ascii_t would need to change slightly after this, to throw
> on anything that  can't fit into a *seven*-bit value, rather than
> eight-bit. I'll add the  default-character option to both types as well,
> and maybe make other  improvements as I have time.
> 

Unfortunately this is not the correct approach as well.

For example why do you think it is safe to pass ASCII subset of utf-8
to current non-utf-8 locale?

For example Shift-JIS that is in use on Windows/ANSI API has different
subset in 0-127 range - it is not ASCII!

Also if you want to use std::codecvt facet...
Don't relay on them unless you know where they come from!

1. By default they are noop - in the default C locale

2. Under most compilers they are not implemented properly.

   OS \ Compiler      MSVC       GCC      SunOS/stlport  SunOS/standard
   -------------------------------------------------------------------
   Windows             ok         none        -              -
   Linux               -           ok         ?              ?
   Mac OS X            -          none        -              -
   FreeBSD             -          none        -              -
   Solaris             -          none      buggy!      ok-but-non-standard

Bottom lines don't relate on "current locale" :-)

> 
> Artyom, since you seem to have more experience with  this stuff than I,
> what do you think? Would those alterations take care of  your objections?
>

The rule of thumb is following:

- When you hadle with strings as text storage just use std::string

- When you do a system call

  a) on Posix - pass it as is
  b) on Windows - Convert to Wide API from UTF-8
  
- When handling text as text (i.e. formatting, collation etc.)
  use good library.

I would strongly recommend to read the answer of Pavel Radzivilovsky
on Stackoverflow:

  
http://stackoverflow.com/questions/1049947/should-utf-16-be-considered-harmful/1855375#1855375

And he is hard-core-windows-programmer, designer, architext and developer
and still he had chosen UTF-8!

The problem that the issue is so completated that making
it absolutly general and on the other hand right is only 
one - decide what you are working with and stick with it.

In CppCMS project I work with (and I developed Boost.Locale
because of it) I stick by default with UTF-8 and use plain
std::string - works like a charm.

Invening "special unicode strings or storage" does not
improve anybody's understanding of Unicode neither improve
its handing.

Best,
  Artyom