Re: [boost] Making Boost.Filesystem work with GENERAL filenames with g++ in Windows (a solution)

Subject: Re: [boost] Making Boost.Filesystem work with GENERAL filenames with g++ in Windows (a solution)
From: Yakov Galka (ybungalobill_at_[hidden])
Date: 2011-10-26 08:17:42

On Wed, Oct 26, 2011 at 12:59, Alf P. Steinbach <
alf.p.steinbach+usenet_at_[hidden]> wrote:

> On 26.10.2011 12:24, Yakov Galka wrote:
>
>> On Wed, Oct 26, 2011 at 11:56, Alf P. Steinbach<
>> alf.p.steinbach+usenet_at_gmail.**com <alf.p.steinbach%2Busenet_at_[hidden]>>
>>  wrote:
>>
>>  On 26.10.2011 11:47, Artyom Beilis wrote:
>>>
>>>>
>>>>>  That is what very good about stream buffers... You can implement
>>>> anything
>>>> you need.
>>>>
>>>>
>>> If the authors of Boost.Filesystem are happy to reimplement stream
>>> buffers
>>> from scratch, all the way, then that would indeed be good news.
>>>
>>> But do you *really* think that is realistic?
>>>
>>>
>>>  I think when someone gives you a working implementation, it's rather
>> dumb to
>> ask if it's realistic or not.
>>
>
> Let me try to clarify what I meant, then.
>
> Artyom's suggestion, as I understand it, is to forsake each compiler's
> streambuf implementation and use a Boost-provided general one that extends
> the standard's interface. With that Boost implementation building on some
> non-standard extension such as _wfopen, or implementing buffers from
> scratch, all the way. Which in the general case will be necessary, and thus,
> no point in also using extensions.
>
> Now is that realistic?

Artyom's suggestion, if I understand (if not then assume it's my
suggestion), is to use the native streambuf whenever possible, but use
Artyom's implementation of streambuf on windows when compiling with
non-Dinkumware standard library. It's definitely realistic.

>   Using short file names is no go for two reasons:
>>>
>>>>
>>>> 1. It works only when file exists (can't create new file)
>>>> 2. It is quite deprecated
>>>>
>>>>
>>> Both points are incorrect.
>>>
>>> Regarding (1), did you notice that I wrote "full source code
>>> provided"...?
>>>
>>>
>> Yes, you provide a hackish workaround, that is vulnerable to a
>> race-condition btw.
>>
>
> No, AFAIK there is no extra realistic vulnerability.
>
> There is a vulnerability, yes, but it's there anyway.
>
> And there is a non-realistic (extremely low probability) vulnerability that
> process A attempts to create a non-existing file F, while process B tries to
> delete F. In this case creating F might fail /also with the fix/. But
> arguing that the fix is "hackish" because it fails to fix such an esoteric
> case, is IMO not serious.

It's extremely low until someone finds a way to increase it... It's unfair
for you as a library developer to introduce a security hole or a bug to the
user's code if you're aware of it. For me the sequence CreateFile() ->
CloseHandle() -> Process B -> fopen/GetShortPathName is suspicious. Process
B can do anything, including renaming files, deleting, creating or calling
SetFileShortName(). If you can't prove that it's secure, then it's not. I
admit that I intentionally exaggerate this point, but such 'small' things
make systems robust.

Still you ignored Artyom's no. (2). You know that, they exist only for
backward compatibility and can be disabled on newer versions of windows. I
would say they're unofficially deprecated.

 Even a more serious problem is that it doesn't work with
>> paths longer than MAX_PATH.
>>
>
> The vague impression you have in that direction is in a sense valid, that
> there is something fishy there, because with the fix one may conceivably
> have a problem that a path that is longer than MAX_PATH works, when by
> rights it should not (I have not tested this though).
>
> However,
>
>  1 It is trivial to restrict narrow character path lengths to
>    MAX_PATH so that that problem does not occur.
>

I prefer a solution that handles them rather than restricts my software.

>  2 It is only where the existing implementation fails to access
>    a too long path, that the fix may fail to erroneously fix it.
>

What?

>  3 It is not a good idea to use Windows paths longer than MAX_PATH
>    since Windows Explorer and many other tools do not support them.
>    Unless you want your application to create unusable files.
>

If someone doesn't support them, it's not an excuse for limiting the
software that needs to. On the contrary, add the support to the low-level
libraries so that the software that builds on those libraries will
transparently benefit from the upgrade.

> In short your "serious problem" has it backwards as to what the problems
> are, and the fix does not introduce any restriction.
>
>
>
>  I think when something demonstrably works, it's rather dumb to assert that
>>> it can't work, sorry.
>>>
>>>
>>>
>>>  But considering that Boost.Filesystem is slated for later inclusion in
>>> the
>>>
>>>> C++
>>>>> standard library (or at least into TR2), I think it would be nice if it
>>>>> is able
>>>>> to give access to all accessible files in Windows, also with g++,
>>>>>
>>>>>
>>>> This is a problem that can be fixed easily in Boost.Filesystem.
>>>>
>>>>
>>> Yes, I think so, for Windows.
>>>
>>> However, more important, the existence of an actual fix (which I linked
>>> to)
>>> shows that the present interface does not prevent a fix for Windows.
>>>
>>> For some other OS-es it may not necessarily be easy to fix, however, and
>>> thus, for inclusion of Boost.Filesystem in the standard library I think
>>> that
>>> should be thoroughly investigated.
>>>
>>
>>
>> Personally I think that boost::filesystem::paths are a sad joke, it's a
>> pity
>> they're heading to the standard. Although the OS-part is definitely good,
>> the way path class is design isn't suitable for paths outside the unix
>> world. Even if you fix the Unicode problems, you still cannot use long
>> paths
>> on windows (longer than MAX_PATH), although they are supported by the OS.
>> Moreover, judging by the last fixes to the library, it looks like Beman
>> wants to shift the burden of this on the user of the library, instead of
>> implementing something that works transparently.
>>
>
> Hm, I haven't used the path class.
>
> However the MAX_PATH issue is not really an issue in practice.
>
> The *only* uses I have seen of paths longer than MAX_PATH, have been silly
> script-kiddies trying to create problems for people running FTP servers. And
> that's because the ordinary tools can't handle them, thus, difficult to
> remove the script-kiddie's nested folders. Conversely, as a serious software
> developer one should stay well away from such paths.

This offends me. I'm facing the MAX_PATH limitation at my work. If you
develop nothing more than desktop apps that interact directly with the user,
then please don't infer from this that others don't need it either. MAX_PATH
is a problem in a large scale systems where you have enormous amounts of
data. In particular this is why we don't switch to boost::filesystem (since
it neither workarounds the problem nor works well when given a long path
with \\?\).

-- 
Yakov