[boost] Re: [network] An RFC - updated

From: Don G (dongryphon_at_[hidden])
Date: 2005-04-23 18:30:16

Hi Michel,

Don G wrote:
>> There is the question of what network types does
>> an application need. It is not likely they need
>> all of them. So, the app must pre-register the
>> types it wants and only those address forms will
>> work.
>
> I would prefer for a fixed set of networks (mainly
> ip/ipv6 to be available from start maybe lazily
> created on first reference). And registering a
> network creation function to the registry would
> let the registry use weak references to networks
> and all objects spawned from a network holds strong
> references in that way the network is destroyed
> when the last object spawned from it goes away. The
> network creation function could be passed along
> with an address so an address could be used across
> dll boundaries where each dll has its own registry.

I have to admit that I avoid DLL/.so's whenever possible, so this
approach is not one I had considered. I don't see any problem with
the application registering network objects inside main or near
there<g>, that way only the networks one needs to support are linked.

> Wouldn't you need to store the settings and then
> start the applications from the stored settings
> and isn't the map needed then?

At most, the network settings we store are some port overrides. So we
just read them as we create the appropriate network object. Following
that, we have "mTcp", "mSerial", etc. and which is needed is known by
context. This is just my experience. A map would come into play with
the address-tells-all approach. More on this in my reply to Peter. :)

>> While I still agree with the jist of your argument, I
>> am not so sure everyone would appreciate their apps
>> growing by sizeof(http_lib) + sizeof(ftp_lib) +
>> sizeof(uucp_lib) + sizeof(gopher_lib) + ... to
>> support this approach. :)
>
> They could be dynamically loaded.

True, but they can also be created by the application as well. The
application developer (as opposed to the library developer) generally
knows what kinds of pieces are needed and can register them in code.
Or use some more elaborate mechanism such as dynamic loading, but
that just pushes the "link" out to distribution building time. The
app developer must still know.

> I think the syntax of the address should be kept
> simple to use, explain and parse and having several
> ways of expressing the same thing in general isn't
> a good idea. And in that case we also would need 
> canonicalization to be able to compare the addresses.
> Ie ipv4:tcp:http://www.boost.org shold be equal to 
> ipv4:http://www.boost.org:80. But i guess they both
> could be canonicalized to something like
> tcp:/66.35.250.210:80.

I agree that one form is desirable, and the simpler the better. Peter
talks more about this, so I will respond over there.

>> I agree that net::poll() would fit with a common
>> desire to have single threaded network programs,
>> but might complicate things where the program is
>> a GUI. Integrating with the GUI loop is a study
>> in compromise (at least for Windows).
>
> I see no reason whatso ever to have a windows gui
> program using the library to be single threaded. It
> would be easy enough to spawn a single io thread
> polling and handling the notifications and posting
> results to be processed by the gui. I have only
> gotten into trouble trying to integrate this kind
> of polling in the eventloop of a gui application
> and always moved to an simple thread model posting
> notifications to the gui thread.

The desire to eliminate threads from code is not restricted to
servers<g>. There is a long history of using the windows message
queue (via PostMessage) for handling networking that pre-dates
threads just as select and signal-driven I/O pre-date threads for
Unix. [FWIW: the "right" way is PostMessage-to-hidden-window, not
splicing up the GetMessage/DispatchMessgae loop, and no polling is
required]

I see this as a perfectly reasonable approach, especially when the
needs are modest. Even a complex GUI app (at least for X, Windows and
Mac) is typically single-threaded and asynchronous. Network
notification can be fairly simply added to such a program w/o forcing
multi-threaded coding. Again, this is the "general async" stuff that
is not in the network interfaces I have proposed.

>> I like the idea of net::poll() for some uses, but
>> it doesn't fit what I often need (GUI integration).
>> It does fit with select/epoll especially on platforms
>> where the number of objects that can go in an fd_set
>> is > 64.
>
> There still is some arbitrary limit albeit higher
> isn't there. The FD_SETSIZE can be set to a large
> value before including winsock (althoug some lsp
> makes presumptions about the size).

I hadn't noticed that tid bit in select's docs before, but you are
correct: it can be set higher, and it probably could cause issues
with some drivers. On a side note: the product I work on a the job
has run into issues related to specific network drivers when using
DuplicateHandle on a socket as prescribed in MSDN, so this kind of
issue is real. :(

>> I agree, but others don't. Some folks don't want
>> any background threads; just a single thread doing
>> one uber select/epoll call (again, for platforms
>> that can handle it). The only way I can see to
>> accommodate that is a different concrete network
>> object. The ones I would write initially would
>> probably not fit this desire, but again, the
>> interface and behavior contract probably can.
>
> I don't really get why that couldn't that be
> accomodated whitout another network class?

I am presupposing that one network impl will make heavy use of worker
threads, and deliver callbacks as I described elsewhere. The same
impl cannot also be fully single threaded. :) Of course, what I have
been saying as well is that one can use the general async facility to
get callbacks queued to a specific (single) thread, thereby hiding
the worker threads. The issue is that some don't find this detail
insignificant: they want to eliminate the worker threads altogether.

>> I would be happy to entertain ideas on how to
>> provide both kinds of buffer management,
>> especially if there is a way to eliminate as
>> much cost as possible (in terms of "if checks"
>> and code linkage) when automatic is never used.
>
> I really don't see how some if checks and linking
> against std::vector<char> could bear any costs in
> terms of performance of a network library.

I'm just curious to see if someone has some simple ideas on automatic
buffering that doesn't require massive overhead when unused. It
wasn't that all if-checks had to be eliminated, or that some extra
code might get linked w/o being used. What I want to avoid is large
chunks coming in via iostream (for example) that in many uses will be
dead code.

Best,
Don

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com