Re: [boost] [locale] Review results for Boost.Locale library

Subject: Re: [boost] [locale] Review results for Boost.Locale library
From: Jeremy Maitin-Shepard (jeremy_at_[hidden])
Date: 2011-04-27 15:42:56

On 04/27/2011 12:07 AM, Artyom wrote:
>
> How the catalog works. It searches the key in the hash table,
> as the last stage it compares the strings bytewise.
>
> It is fast and efficient.
>
> In order to support both L"", "", u"" and U"" I need to
> create a 4 variants of same string to make sure it works
> fast (waste of memory) or I need to convert the
> string from UTF-16/32 to UTF-8 that is run-time
> memory allocation and conversion.
>
> So no, I'm not going to do this, especially that
> it is nor portable enough.

Why not simply provide a compile-time or run-time option to allow the 
user to specify the following:

- encoding of narrow keys to be given as char * arguments, or specify 
that none is to be supported (in which case narrow keys cannot be used 
at all), the default being UTF-8;

- whether wchar_t * arguments are supported (the encoding will be 
assumed to be UTF-16 or UTF-32 depending on sizeof(wchar_t)) [by 
default, not supported]

- whether char16_t arguments are supported [by default, not supported]

- whether char32_t arguments are supported [by default, not supported]

The library would simply convert the UTF-8 encoded keys in the message 
catalogs to each of the supported key argument encodings.  In most 
cases, there would only be a single supported encoding.  Because the 
narrow version could be disabled, with Japanese text and UTF-16 wchar_t, 
this would actually _save_ space since UTF-16 is more efficient than 
UTF-8 for encoding Japanese text.

More to the point, you as a library author can offer this functionality 
(since it shouldn't be too much of an implementation burden) even if you 
as a user of your library wouldn't want to use it (because you are happy 
to provide English string literals).

I agree that it is very unfortunate that wchar_t can mean either UTF-16 
or UTF-32 depending on the platform, but in practice the same source 
code containing L"" string literals can be used on both Windows and 
Linux to reliably specify Unicode string literals (provided that care is 
taken to ensure the compiler knows the source code encoding).  The fact 
that UTF-32 (which Linux tends to use for wchar_t) is space-inefficient 
does in some ways make render Linux a second-class citizen if a solution 
based on wide string literals is used for portability, but using UTF-8 
on MSVC is basically just impossible, rather than merely less efficient, 
so there doesn't seem to be another option.  (Assuming you are unwilling 
to rely on the Windows "ANSI" narrow encodings.)

>
>
>>> One possibility is to provide per-domain basis a key  in po file
>>> "X-Boost-Locale-Source-Encoding" so user would be able to  specify in
>>> special record (which exists in all message catalogs)  something
>>> like:
>>>
>>>          "X-Boost-Locale-Source-Encoding: windows-936"
>>> or
>>>          "X-Boost-Locale-Source-Encoding: UTF-8"
>>>
>>> Then when  the catalog would be loaded its keys would be converted
>>> to the  X-Boost-Locale-Source-Encoding.
>>
>> This isn't a property of the message  catalog, but rather a property of
>> the program itself, and therefore should  be specified in the program,
>> and not in the message catalog, it would  seem.  Something like the
>> preprocessor define I mentioned would be a  way to do this.
>>
>
> Two problem with define that I want
>
>   translate("foo") to work automatically and not being a define.
>
> So I either need to provide an encoding in catalog itself
> or when I provide domain name (the reason it is done
> per domain name as one part of the project may use UTF-8 and
> other cp936 and other may use US-ASCII at all)
>
> So I can either specify it when I load a catalog or in
> catalog itself.

Okay.