Re: [boost] ASIO into the standard

Subject: Re: [boost] ASIO into the standard
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-07-03 06:25:35

On 2 Jul 2014 at 17:59, Beman Dawes wrote:

> > You must remember that from the perspective of ISO, the only
> > engineering standards which exist are other ISO standards. That in
> > this case equals POSIX. This is why I said that Windows support
> > cannot be standardised and would have to be omitted.
> > pletely agree on this sentiment.
> 
> You have been misinformed.

I believe I used loose language. But I am not misinformed. For your 
reference, I used to serve as the ISO SC22 mirror convenor for 
Ireland, though you have many more years of ISO experience than I do.

> ISO rules are somewhat different for dealing with documents that are not
> ISO standards, but mostly that is just a matter of careful drafting. For
> example, see the  ISO/IEC Directives, Part 2, Rules for the structure and
> drafting of International Standards, section 6.6.3 Use of trade names and
> trademarks.
> 
> The C++ standard and its TRs and TSes support Windows, as well as a lot of
> other operating systems, and that is one of the criteria for evaluating
> proposals to the committee. Any proposal that cannot support a common
> operating system would likely be dead on arrival.

I'll come back to this at the end.

> Look at the Filesystem TS for an example. It mentions Windows and several
> other operating systems by name, and has a compliance section that explains
> how that TS copes with differences between platforms.

As ISO would call it, it's all about "normative references" which can 
only be to other ISO documents (source: 
http://www.iec.ch/standardsdev/resources/draftingpublications/writing_
editing/directives/normative_references.htm). As POSIX is an ISO 
standard, it can be normatively referenced. Everything else goes into 
a Bibliography.

What this turns into is that you can have an informative section 
which explains similarities and differences between the standard and 
implementations and other non-ISO standards, but which forms no part 
of the standard itself. You can also claim that there is no 
substantive difference between the standard and some named 
implementation(s), but again this is informative. While you *could* 
standardise a non-normatively referenced item aka some proprietary 
technology, you would receive a lot of heat from ISO and the 
potential for JTC1 to reject it at a plenary session which would be 
deeply embarrassing. The reason why is because there is a formal 
process for proprietary technologies to become standardised which 
involves patent agreements etc. There are also other ISO working 
groups to consider, so if a WG21 Networking TS standardised the 
Windows IOCP model I could rightly see the Austin Working Group going 
bananas over the potential consequences on POSIX, never mind the ADA 
WG, or even the Ruby WG.

Let's apply this to the Filesystem TS. Windows provides most of a 
semantic equivalence to POSIX in filesystem, so file handles, reading 
and writing, file paths referencing a universal file system namespace 
(apart from a drive letter, differences in allowed characters and the 
slash being opposite they're very semantically close, much closer 
than say file paths on DEC VMS), ability to use unicode (albeit in 
different binary formats), they all map onto the same thing, and 
these are what Filesystem standardises. This is because this is the 
common subset of Windows *to* POSIX, not *with* POSIX.

Let's briefly cover what Filesystem does not currently standardise. 
Memory mapping files on Windows is similar to POSIX, though with a 
nasty gotcha difference [1]. Symbolic links are also similar, though 
also with a nasty gotcha difference [2]. The Win32 API doesn't expose 
an atomic stat() call, even though the NT kernel does, so metadata 
can be racy which is another nasty gotcha. Off the top of my head, 
the only substantial divergence of Windows from POSIX is security and 
access permissions [3], and of course file locking where the Windows 
implementation is not stupidly broken.

So, in the Filesystem TS, you can standardise an API which if 
applications use will produce identical effects on all systems (apart 
from access perms on Windows [see below]), and you can claim this 
legitimately as it is true (sans bugs of course) for the subset of 
POSIX-equivalent Filesystem functionality you have standardised. As 
your informative, you can point out minor differences about platform 
specifics and deviations from POSIX, and all is good.

But let us be very clear here, the "gold standard" aimed at is POSIX. 
If Windows provides a feature not provided by POSIX - and in 
Filesystem there are many e.g. async file i/o - you cannot 
standardise it in an ISO TS without significant political 
consequences. If you look at access perms, note the direction of 
travel: the Filesystem TS can standardise a POSIX feature even though 
it isn't available on Windows and that's permitted. The opposite is 
not permitted.

> The C++ standard and its TRs and TSes support Windows, as well as a lot
> of other operating systems, and that is one of the criteria for
> evaluating proposals to the committee. Any proposal that cannot support
> a common operating system would likely be dead on arrival. 

And here we arrive onto the rub of the matter.

Windows originally borrowed the 4.4 BSD network stack, and as a 
result has a closely similar socket API to POSIX - if used 
*synchronously*. Unfortunately, all i/o (with a few exceptions) on 
Windows is *always async* with the synchronous functions acting as 
wrappers of the underlying async implementation. Windows in fact 
pretends to be a micro-kernel and provides a 98% [4] async i/o API as 
if it were micro-kernel.

Thus, on Windows, ASIO uses async sockets whilst on POSIX it uses 
non-blocking sockets. This produces a ton of semantic difference 
which is why there is such a plethora of POSIX-specific and 
Windows-specific class types in the ASIO reference page 
(http://www.boost.org/doc/libs/1_55_0/doc/html/boost_asio/reference.ht
ml, bottom of page).

Now, a Networking TS standardising ASIO can simply say 
"implementation defined" whenever these async vs. non-blocking 
semantic differences turn up, and I suspect that is exactly what will 
happen because all the alternatives are harder. This implies that 
Windows-specific support aka the whole true async socket semantic 
model aka cool stuff like IOCP (i/o completion ports) cannot be 
standardised. What will result in the Networking TS will still be 
useful for some C++ network applications, but I suspect that real C++ 
network applications are going to remain on ASIO because it lets you 
code around the semantic differences.

Does this explain my original claim? I am not claiming that a 
standardised ASIO in a Networking TS would not function on Windows. I 
*am* claiming that so much Windows-specific aka async-specific 
functionality would have to be left unspecified that no serious C++ 
networking user would bother using the functionality in the TS.

So what would I propose instead? What I personally think is needed 
here is that WG21 provide a canonical Networking TS implementation 
library rather than writing a specification and leaving it up to STL 
implementators. That canonical implementation would be a 
substantially rejigged and slimmed down all C++14 subset of ASIO [5].

This of course is very new territory - I am not aware of WG21 ever 
providing a canonical library implementation before. But WG21 has 
already been breaking new ground with letting library writers go 
direct to std::experimental and skip getting a new STL type into 
Boost first and later submit for standardisation after some 
experience is gained. I note the fact that activity in Boost dropped 
substantially around the same time WG21 allowed this, and that "all 
the action" in new library features appears to be happening around 
WG21 processes instead of via Boost.

Personally I think this is the best way of standardising ASIO into a 
Networking TS. But I appreciate that such an approach is fraught with 
dangers, not least who will pay the engineers who write and test such 
a canonical library? What if there is substantial disagreement in the 
direction of refactor? What if real users just don't care, and keep 
using original ASIO? Still, it's better than every STL having a 
separate toy implementation of ASIO that is useless.

[1]: munmap() will unmap all maps falling into the range supplied. 
UnmapViewOfFile() only unmaps the precise map given. This can be a 
nasty source of bugs.

[2]: Derereferencing a link takes different semantics to POSIX 
unfortunately. It works similarly enough most won't notice, until 
they do.

[3]: xattr ACL support on Linux and BSD looks similar enough to 
Windows ACLs that a common subset of functionality should be 
possible. Of course, POSIX famously demurred on standardising these 
in the 1990s, but ACLs are now pretty much standard in implementation 
across Linux, BSD and Windows. They are ripe for being formally 
standardised into POSIX.

[4]: A glaring counterexample is that CreateFile() which is used to 
open file handles cannot be executed asynchronously. Neither can 
CloseHandle() nor FlushFileBuffers(). There are others, but these 
three are the worst offenders, especially as CreateFile() is 
incredibly slow (~30k ops/sec).

[5]: ASIO could do with explicit support for micro-kernel operating 
systems, by 
which I mean you can completely dispense with threads on such 
systems. 
Something which annoyed me with the QNX port of ASIO is that it used 
the POSIX 
multiplexing API which was woefully inefficient when all syscalls in 
a micro-kernel OS can be multiplexed at source, and therefore the 
ASIO io_service could be very tightly integrated into the core QNX 
message passing system. Such an adaptation I think would not require 
much change to ASIO.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/