Re: [Boost-users] Inconsistent unicode encoding between boost andwxonmac osx

Subject: Re: [Boost-users] Inconsistent unicode encoding between boost andwxonmac osx
From: Sachin Garg (schngrg_at_[hidden])
Date: 2010-02-13 19:40:24

On Sun, Feb 14, 2010 at 5:11 AM, Peter Dimov <pdimov_at_[hidden]> wrote:
> Sachin Garg wrote:
>
>> Peter, I am guessing that the problem is not the OSX locale which
>> could be UTF-8 or UTF-16 or X (I am calling this 'source' encoding).
>
> The encoding of OS X's path names is always UTF-8. It does not depend on the
> OS locale.
>
>> Problem *seems* to be that boost converts that X into UTF16 (or UTF32)
>> for storing it in wpath and that is using a normalization which is
>> incompatible with normalization used by wx for its UTF16 (or 32).
>
> I don't think that this has anything to do with normalization. By looking at
> the source, Filesystem, by default, uses the global locale to convert betwen
> narrow and wide paths, and the default locale in your case seems to perform
> no conversion. That is, it just takes every 8 bit char and converts it to a
> 32 bit wchar_t. This, of course, doesn't work when the source is UTF-8,
> regardless of its normalization.

It looks like you are correct, thanks :-)

I added this to my code and it fixes the problem:

  std::locale global_loc = std::locale();
  boost::filesystem::detail::utf8_codecvt_facet utf8_facet(1);
  std::locale loc(global_loc, &utf8_facet);
  boost::filesystem::wpath_traits::imbue(loc);

Taken from:
  http://archives.free.net.ph/message/20071110.132150.af9dc620.en.html

And it seems this is an issue since atleast 2007, I can't tell why
more people haven't got hurt by this. This solution works on my
computer but I don't know if it will work if OS uses any random
different locale etc...

Thanks again,
SG