Re: [boost] Review Request: Boost.Locale

Subject: Re: [boost] Review Request: Boost.Locale
From: Robert Ramey (ramey_at_[hidden])
Date: 2010-05-24 04:47:41

Artyom wrote:
>> One thing in particular that I was interested in is(are)
>> codecvt
>> facets.
>> I didn't any thing on this.  Why is
>> that?
>
> Take a look on http://www.cplusplus.com/reference/std/locale/codecvt/
>
> They allow you to imbue special charset to fstream and automatically
> translate wide characters to normal encoding like UTF-8 or ISO-8859-8.
>
>
>> Is this a separate
>> subject or is that you believe they're not useful.
>
> Theoretically they are very useful.
>
> For example:
>
>    std::wofstream fs;
>    fs.imbue(std::locale("he_IL.UTF-8"));
>    fs.open("file.txt");
>    fs << L"????"!
>
> Would print UTF-8 output.

I'm familiar with the codecvt facet as several of them are used
with the serialization library.

>
> But...
>
> - Many compilers/standard libraries do not implement locales at all.
>  (GCC under Windows and Solaris, STL Port library)
> - Support of locales and encoding is strictly limited to OS
>  configuration. So on some host the above example would work on other
>  it would throw invalid locale error.
> - Some compilers/OSes do not support UTF-8 encodings (MSVC) so
>  you can't create UTF-8 locale at all.
> - Locales name are platform depended. For example under Windows
>  you need Hebrew_Israel.1255 locale and under Linux he_IL.ISO-8859-8
>  (and BTW 1255!=iso-8859-8)

I think this is a separate issue than codecvt facet. I've found them to work
with all C++ implemenations that boost uses.

> So Boost.Locale reimplements standard codecvt facet to make this work
> on any platform.

I didn't seen anything in the documentation about that.
>
> However there is still a limitation when working with 2 byte
> characters (ie char16_t or wchar_t under windows) as Boost.Locale
> would work correctly only with UCS-2

> But this is actually C++ standard's limitation.

The reason I ask is that I often see things requested on the list
which I think could be better implemented as codecvt facets.
Also, it seemed to me that a large part of the iostreams library
could have been implemented more efficiently wiht codecvt
facets.  Admitidly, it's somewhat unobvious how to make the
best use of this - (needs another library and documentation of course)
but I'm surprised that don't seem to be mentioned at all
in the documentation.

Robert Ramey