Re: [boost] Fw: [locale] Formal review of Boost.Locale library

Subject: Re: [boost] Fw: [locale] Formal review of Boost.Locale library
From: Artyom (artyomtnk_at_[hidden])
Date: 2011-04-14 16:11:01

> > 1. Wide characters are not second class citizens in Boost.Locale
> >   everything is fully supported for Wide API.
> transalte only accept char * or std::string.
>

Ok, let me explain.

Yes, boost::locale::translate receives "char *" as input
as it uses ASCII text as key.

You write the code originally in English using ASCII
and they translated to Wide Strings or to narrow
according to the context.

So basically your keys or original text should be ASCII
(and gettext would warn you if it does not)

However translations are anything you wish.

If you want to embed Japanese to the text you
are welcome to do this but in such case you
do not need translate.

Only when the message is not found in dictionary
it is converted to wide strings by simple
casting each ASCII character to wide one.

This is how gettext (and most other translations
systems) work.

There is nothing wrong with this.

>
> As I said, using only UTF-8 is impossible in Windows.
> Because MSVC doesn't support UTF-8 literal yet.
> Input(parameter of translate) must support UTF-16 as well.

Actually it does... Take a look on libs/locale/examples/w*

You need to have UTF-8 BOM and it would work perfectly
(I know it is stupid to add BOM but that is how MSVC understands
Unicode)

> >
> >> Further  reading of the document reveals that auther of this library believes
> >> "should  use wide strings everywhere" is a  myth.
> >> http://cppcms.sourceforge.net/boost_locale/html/recommendations_and_myths.html
> >>
> >> I  strongly object about that is a myth.
> >>
> >
> > I'm sorry but in context of cross platform programming
> > wide characters quite useless.
> So this library doesn't work well on Windows.
> Good cross platform.
>

It is not the problem of the library but the problem
of badly defined wchar_t (or windows API)

And wide characters are fully supported.

> >
> > So I can't recommend using wide characters as they may be
> > UTF-16 or UTF-32, however for Windows only development
> > wide API is fine and fully supported.
> You say "fully supported" without taking  wide characters as an input?
>

It takes ASCII as input not wide characters should not be used
for translations at all.

See above.

> >> Why it doesn't  support char set conversion between UTFs?
> >
> > It does
>
> hmm I can't find it from document.
>

http://cppcms.sourceforge.net/boost_locale/html/namespaceboost_1_1locale_1_1conv.html

> >> I suspect auther of this library doesn't have any  serious experience
> >> in Windows programming and languages like  Japanese.
> >> What kind of joke it doesn't support Win32 API backend in  MSVC?
> >
> > I'm sorry? Win32API backend supported by MSVC, GCC/Mingw and even Cygwin!
>
> It looks like document is bad.
> http://cppcms.sourceforge.net/boost_locale/html/using_localization_backends.html
>
>

Two ponts:

> You need GCC-4.x to use it.

Meaning you can't use GCC-3.4 but 4.x series as the
3.x does not support wide characters. Sorry if it mislead you.

MSVC is first class citizen and actually much better
supported then GCC.

> Only UTF-8 encoding is supported.

Ok you misread the documentation.

This part that describes locale generation

   http://cppcms.sourceforge.net/boost_locale/html/locale_gen.html

The encoding defines encoding of narrow
strings not wide ones.
The wide strings encoding is UTF-16 or UTF-32
according to sizeof(wchar_t)

If you want to use locale ja_JP.Shift-JIS then you
should use either ICU or std backends.

Meaning if you want to treat narrow strings
as Shift-JIS encoded strings

> Ryou Ezoe

I'd strongly recommend you to read the documentation
and take a look on examples carefully before
you make such statements.

Artyom