RE: [boost] Unicode and filesystem

From: Beman Dawes (bdawes_at_[hidden])
Date: 2003-09-15 20:30:24

At 10:17 AM 9/15/2003, Adrian Michel wrote:

 >> Standard C++ does not require any std::basic_fstream ctor which takes a
 >> wchar_t const *, and such a ctor is neither available in VC++ 6, 7, or
 >> 7.1.
 >
 >This is where a template version of boost::filesystem::path would have
 >helped :) Don't get me wrong - like many other developers, I appreciate 
all
 >the work you guys are putting into this amazing boost library.
 >
 >I am curious though, is this a non-issue on platforms other than Windows?

No, file and directory names using various character sets and 
representations are a major issue for many platforms. Actually "major 
issue" doesn't begin to describe how slippery the problem is.

I've put a lot of effort into the problem, and have also talked to others 
who have dealt with the problem. I've gotten to the point where I have a 
detailed understanding of many of the sub-issues, and have solutions for 
many of the problems. In particular, the problem of conversion between 
internal and external representations seems more tractable than it did 
eight months ago. There is simply no single one "right way"; there are two 
levels of sensible defaults, and then the user needs to be able to supply 
her own conversions to handle corner cases.

I haven't completed work on how this would apply specifically to the 
Filesystem library, but I'm guessing there would be path and wpath 
predefined classes, plus a basic_path template for the roll-your-own user.

Now that's all fine if the external representation is only ever char. It 
isn't even too bad if the external representation is some other single type 
(like with Windows CE). It gets messy with Windows NTFS though, because 
there are two external representations available - 8-bit characters and 
16-bit characters. Perhaps wpath with map to 16-bit external representation 
on Windows NTFS, and 8-bit external reps on POSIX and other systems which 
use 8-bit external names. User code would be portable. So far, so good.

But what happens with basic_path? Is the external rep implementation 
defined? A template parameter limited to certain implementation defined 
choices? That's hardly portable.

Anyhow, the problem is being worked on, but it is messy beyond belief.

--Beman