Re: [boost] [interprocess] OS version dependencies

Subject: Re: [boost] [interprocess] OS version dependencies
From: Domagoj Saric (dsaritz_at_[hidden])
Date: 2010-10-23 16:35:11

"Marsh Ray" <marsh_at_[hidden]> wrote in message 
news:4CBDEF97.6000101_at_extendedsubset.com...
> On 10/19/2010 10:28 AM, Domagoj Saric wrote:
>> I'm aware of that, however considering that: - Interprocess already
>> uses the native NT API
>
> The documented parts or undocumented parts?

Search for "ntdll.dll" in details/win32_api.hpp ;)

>> - interdependencies between Microsoft's/Windows' own components (and
>> the sheer number of those) make it reasonable to assume breaking
>> changes to the native NT API can in practice ever occur only between
>> major OS releases
>
> Or service packs, or whenever.
>
> http://msdn.microsoft.com/en-us/library/bb432200.aspx :
>> The functions and structures in Winternl.h are internal to the
>> operating system and subject to change from one release of Windows to
>> the next, and possibly even between service packs for each release.
>> To maintain the compatibility of your application, you should use the
>> equivalent public functions instead. Further information is available
>> in the header file, Winternl.h, and the documentation for each
>> function.
>
> I.e., "fair notice is given: ignore it at your peril".

Right, I know this, that is why I wrote >in practice<...
And notice that the wording is between releases and possibly between service 
packs...without the 'whenever' part...

>> - in practice breaking changes to system APIs have to be expected
>> between major OS releases even for Win32 and other 'old&public' API
>> as was demonstrated for example by the changes for interactive
>> services between XP and Vista
>
> That was a rare architectural change to Windows (explicit kernel support 
> for multiple terminal services sessions) that broke the previous 
> single-user assumption. App developers had been told that change was 
> coming since post-NT4 days, IIRC. It doesn't discredit the general claim 
> of compatibility across Win32.

However you look at it, it still did happen...and it is not the only such 
change...it happens even on much more higher levels...for example a single 
hotfix broke backward compatibility for the cryptographic API in Vista...
Also please note my point correctly, it is not to 'discredit' the Win32 API 
but the (in practice) unjustified over-emphasis on the discrepancy between 
the stability of Win32 API and the native NT API (and the arguments drawn 
from that 'unjustified over-emphasis')...

> To the contrary, look at all the processes on your Windows system. How 
> many are created as acutal native NT processes? Maybe two or three 
> (winlogon, csrss), and last I checked the first thing they did was to load 
> kernel32 like everybody else. Step through MS's own internally developed 
> code and you'll see that even they use the documented Win32 APIs as much 
> as possible.

Loading the kernel32.dll first is no proof of not using the native NT 
API...e.g. Boost.Interprocess will also cause kernel32.dll to load first 
(because of static linking) while at the same time using the native NT API 
(through dynamic runtime loading of ntdll.dll, which is by then already 
loaded by kernel32.dll anyway...)...

Similarly for 'as much as possible', while I have nothing against the 
principle per se, it obviously does not imply 'always'/'only'/'solely'...

>> I'd consider this 'danger' practically
>> irrelevant (more like 'academic/by the book paranoia')
>
> My impression from reading the list is that a lot of Boost developers 
> appreciate the "by the book" observance of documented APIs. It's a design 
> principle, not just a paranoid fear based on a perceived probability of 
> breakage. Interfaces are defined not just for the purposes of assigning 
> "blame", but so that we can discuss program correctness rather than just 
> making empirical statements about observed behavior.

Right, however, the whole point of my unholy POV on 'almost dogmatic 
by-the-bookness' boils down to the fact that "in theory there is no 
difference between practice and theory, in practice there is"...
IOW in practice the choice is not black and white, there is only the more 
treaded path and the less treaded path(s)...If there are no objections to 
the more treaded path the choice is obvious but if even the more treaded 
path gives you thorns in your feet the possibility of exploring the less 
treaded path(s) may become a viable cost-benefit analysis outcome (which is 
where I think we basically agree)...

But, this is again 'academic' side tracking :)
To return to the subject, IMNHO, from the practical/empirical POV, the 
issue/goal/ideal of 'correctness' must also include some measure of 
efficiency...
Somehow, when using today's software on today's 
trilions-of-bytes-of-memory-and-trilions-of-operations-per-second in 
Joe-Sixpack-hardware that still performs far below one's expectations, this 
seems self-evident...

The fact that a browser plugin is able to 'correctly' bring up a functional 
dialog means little (to me) if it, for that single dialog, also brings in a 
15 MB Qt-like monster of an 'auxiliary' library...and when you multiply that 
by the number of plugins one can have......

> Next time you read industry reports like "IT departments are delaying 
> deployment of the update (containing 43 critical security fixes) for a few 
> months until they better assess app compatibility" you might take that as 
> evidence that app developers as a whole are not excessively "paranoid".

I'm not sure if I understand this point...If it is something along the lines 
that 'you cannot call A large if there exists B that is larger' such 
arguments are a priori false...
IOW I am aware that when it comes to paranoia (as well as irrational/vanity 
based policies) few things compare to the corporate world :D but this does 
not necessarily excuse app developers ;)

> Sometimes, we have to do some fairly "heroic" and undocumented things to 
> implement the functionality of our product. DLL injection, API hooking, 
> the type of stuff antivirus programs need to do. Right now I'm working on 
> the Vista implementation of a feature that works three different ways on 
> three different versions of Windows. So we are willing to go there when 
> necessary and, most importantly, when there is no other option. But we do 
> it with our eyes open, knowing the maintenance effort it may bring.

And the practical question is, is there a significant discrepancy between 
the number of these inter-version specifics/incompatibilities caused by 
public/Win32 API incompatiblities and those caused by native NT API changes?

> Again, if it's necessary to implement useful functionality, that's fine. 
> Yeah I agree that the Nt* APIs aren't going to change all that often.

And this is the core point, where we seem to agree fully ;)

> BUT PLEASE STATE THIS DEPENDENCY BOLDLY IN THE LIBRARY DOCUMENTATION

Of course, I would also hold this requirement as self-evident...Considering 
that the relative number of API functions used would probably not be large 
I'd go for both implementations (if the Win32 API allows it) that one can 
choose from with a configuration macro...

Finally, after all this hard talk, if we consider that the undocumented part 
of the native NT API (of which, for the use in question, only a few 
functions would have to be used, probably NTQuerySection and 
NTExtendSection) is actually documented by the SysInternals guys that now 
work for Microsoft I again must conclude that the whole issue much more 
'theoretical' than 'practical'...

ps. As you have correctly deduced by splitting this into a different thread, 
the issue/idea of using the native NT API is only a side issue of the 
original ideas which were to make the Interprocess shared/mapped memory 
interface more cross-platform like and the implementation of shared/mapped 
memory for both Interprocess and IOStreams more efficient/less bloated...

-- 
"What Huxley teaches is that in the age of advanced technology, spiritual
devastation is more likely to come from an enemy with a smiling face than
from one whose countenance exudes suspicion and hate."
Neil Postman