Re: [boost] [review] Review of Nowide (Unicode) starts today

Subject: Re: [boost] [review] Review of Nowide (Unicode) starts today
From: Peter Dimov (lists_at_[hidden])
Date: 2017-06-16 12:40:31

Artyom Beilis wrote:
> Actually it is what exactly you should do. Once you turn on **nowide 
> integration** with boost filesystem
>
> path::string does not perform any conversions on POSIX platforms.
>
> And you can safely concatenate two different strings and valid file will 
> be created. Even if dir is in ISO 8859-1 and file in UTF-8. The file will 
> be valid even if not representable in any encoding.

"POSIX" is not correct here. File names under Mac OS X are UTF-8. Always 
UTF-8, not merely by convention, and no mater what locale is set. (They're 
stored in UTF-16 in the filesystem, just like in NTFS. Unlike NTFS, it's 
_valid_ UTF-16.)

For encoding-neutral Unix-es such as Linux and Solaris, the above is 
correct.

In any case, a library that attempts to reencode from the current locale to 
UTF-8 in the narrow case would be useless. (If the locale isn't UTF-8, 
nothing will work anyway, the library trying to handle this case would just 
make things worse.) This is well established empirically by now.