Boost logo

Boost :

Subject: Re: [boost] ASIO into the standard
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-07-03 06:25:35

On 2 Jul 2014 at 17:59, Beman Dawes wrote:

> > You must remember that from the perspective of ISO, the only
> > engineering standards which exist are other ISO standards. That in
> > this case equals POSIX. This is why I said that Windows support
> > cannot be standardised and would have to be omitted.
> > pletely agree on this sentiment.
> You have been misinformed.

I believe I used loose language. But I am not misinformed. For your
reference, I used to serve as the ISO SC22 mirror convenor for
Ireland, though you have many more years of ISO experience than I do.

> ISO rules are somewhat different for dealing with documents that are not
> ISO standards, but mostly that is just a matter of careful drafting. For
> example, see the ISO/IEC Directives, Part 2, Rules for the structure and
> drafting of International Standards, section 6.6.3 Use of trade names and
> trademarks.
> The C++ standard and its TRs and TSes support Windows, as well as a lot of
> other operating systems, and that is one of the criteria for evaluating
> proposals to the committee. Any proposal that cannot support a common
> operating system would likely be dead on arrival.

I'll come back to this at the end.

> Look at the Filesystem TS for an example. It mentions Windows and several
> other operating systems by name, and has a compliance section that explains
> how that TS copes with differences between platforms.

As ISO would call it, it's all about "normative references" which can
only be to other ISO documents (source:
editing/directives/normative_references.htm). As POSIX is an ISO
standard, it can be normatively referenced. Everything else goes into
a Bibliography.

What this turns into is that you can have an informative section
which explains similarities and differences between the standard and
implementations and other non-ISO standards, but which forms no part
of the standard itself. You can also claim that there is no
substantive difference between the standard and some named
implementation(s), but again this is informative. While you *could*
standardise a non-normatively referenced item aka some proprietary
technology, you would receive a lot of heat from ISO and the
potential for JTC1 to reject it at a plenary session which would be
deeply embarrassing. The reason why is because there is a formal
process for proprietary technologies to become standardised which
involves patent agreements etc. There are also other ISO working
groups to consider, so if a WG21 Networking TS standardised the
Windows IOCP model I could rightly see the Austin Working Group going
bananas over the potential consequences on POSIX, never mind the ADA
WG, or even the Ruby WG.

Let's apply this to the Filesystem TS. Windows provides most of a
semantic equivalence to POSIX in filesystem, so file handles, reading
and writing, file paths referencing a universal file system namespace
(apart from a drive letter, differences in allowed characters and the
slash being opposite they're very semantically close, much closer
than say file paths on DEC VMS), ability to use unicode (albeit in
different binary formats), they all map onto the same thing, and
these are what Filesystem standardises. This is because this is the
common subset of Windows *to* POSIX, not *with* POSIX.

Let's briefly cover what Filesystem does not currently standardise.
Memory mapping files on Windows is similar to POSIX, though with a
nasty gotcha difference [1]. Symbolic links are also similar, though
also with a nasty gotcha difference [2]. The Win32 API doesn't expose
an atomic stat() call, even though the NT kernel does, so metadata
can be racy which is another nasty gotcha. Off the top of my head,
the only substantial divergence of Windows from POSIX is security and
access permissions [3], and of course file locking where the Windows
implementation is not stupidly broken.

So, in the Filesystem TS, you can standardise an API which if
applications use will produce identical effects on all systems (apart
from access perms on Windows [see below]), and you can claim this
legitimately as it is true (sans bugs of course) for the subset of
POSIX-equivalent Filesystem functionality you have standardised. As
your informative, you can point out minor differences about platform
specifics and deviations from POSIX, and all is good.

But let us be very clear here, the "gold standard" aimed at is POSIX.
If Windows provides a feature not provided by POSIX - and in
Filesystem there are many e.g. async file i/o - you cannot
standardise it in an ISO TS without significant political
consequences. If you look at access perms, note the direction of
travel: the Filesystem TS can standardise a POSIX feature even though
it isn't available on Windows and that's permitted. The opposite is
not permitted.

> The C++ standard and its TRs and TSes support Windows, as well as a lot
> of other operating systems, and that is one of the criteria for
> evaluating proposals to the committee. Any proposal that cannot support
> a common operating system would likely be dead on arrival.

And here we arrive onto the rub of the matter.

Windows originally borrowed the 4.4 BSD network stack, and as a
result has a closely similar socket API to POSIX - if used
*synchronously*. Unfortunately, all i/o (with a few exceptions) on
Windows is *always async* with the synchronous functions acting as
wrappers of the underlying async implementation. Windows in fact
pretends to be a micro-kernel and provides a 98% [4] async i/o API as
if it were micro-kernel.

Thus, on Windows, ASIO uses async sockets whilst on POSIX it uses
non-blocking sockets. This produces a ton of semantic difference
which is why there is such a plethora of POSIX-specific and
Windows-specific class types in the ASIO reference page
ml, bottom of page).

Now, a Networking TS standardising ASIO can simply say
"implementation defined" whenever these async vs. non-blocking
semantic differences turn up, and I suspect that is exactly what will
happen because all the alternatives are harder. This implies that
Windows-specific support aka the whole true async socket semantic
model aka cool stuff like IOCP (i/o completion ports) cannot be
standardised. What will result in the Networking TS will still be
useful for some C++ network applications, but I suspect that real C++
network applications are going to remain on ASIO because it lets you
code around the semantic differences.

Does this explain my original claim? I am not claiming that a
standardised ASIO in a Networking TS would not function on Windows. I
*am* claiming that so much Windows-specific aka async-specific
functionality would have to be left unspecified that no serious C++
networking user would bother using the functionality in the TS.

So what would I propose instead? What I personally think is needed
here is that WG21 provide a canonical Networking TS implementation
library rather than writing a specification and leaving it up to STL
implementators. That canonical implementation would be a
substantially rejigged and slimmed down all C++14 subset of ASIO [5].

This of course is very new territory - I am not aware of WG21 ever
providing a canonical library implementation before. But WG21 has
already been breaking new ground with letting library writers go
direct to std::experimental and skip getting a new STL type into
Boost first and later submit for standardisation after some
experience is gained. I note the fact that activity in Boost dropped
substantially around the same time WG21 allowed this, and that "all
the action" in new library features appears to be happening around
WG21 processes instead of via Boost.

Personally I think this is the best way of standardising ASIO into a
Networking TS. But I appreciate that such an approach is fraught with
dangers, not least who will pay the engineers who write and test such
a canonical library? What if there is substantial disagreement in the
direction of refactor? What if real users just don't care, and keep
using original ASIO? Still, it's better than every STL having a
separate toy implementation of ASIO that is useless.

[1]: munmap() will unmap all maps falling into the range supplied.
UnmapViewOfFile() only unmaps the precise map given. This can be a
nasty source of bugs.

[2]: Derereferencing a link takes different semantics to POSIX
unfortunately. It works similarly enough most won't notice, until
they do.

[3]: xattr ACL support on Linux and BSD looks similar enough to
Windows ACLs that a common subset of functionality should be
possible. Of course, POSIX famously demurred on standardising these
in the 1990s, but ACLs are now pretty much standard in implementation
across Linux, BSD and Windows. They are ripe for being formally
standardised into POSIX.

[4]: A glaring counterexample is that CreateFile() which is used to
open file handles cannot be executed asynchronously. Neither can
CloseHandle() nor FlushFileBuffers(). There are others, but these
three are the worst offenders, especially as CreateFile() is
incredibly slow (~30k ops/sec).

[5]: ASIO could do with explicit support for micro-kernel operating
systems, by
which I mean you can completely dispense with threads on such
Something which annoyed me with the QNX port of ASIO is that it used
multiplexing API which was woefully inefficient when all syscalls in
a micro-kernel OS can be multiplexed at source, and therefore the
ASIO io_service could be very tightly integrated into the core QNX
message passing system. Such an adaptation I think would not require
much change to ASIO.


ned Productions Limited Consulting

Boost list run by bdawes at, gregod at, cpdaniel at, john at