Proposal for protocol negotiation

I guess my view is quite diffferent from what I have read so far (Henrik and Robert proposals, which look quite similar in spirit), which makes me scared probably because I will again push things too far. Anyway, after more thoughts I decided to still go and write this, just to make sure that the opinion reflected here has been seen at least once. (as usual I would probably not recommend everything below).

Goals

My vision for HTTP-NG is that it should provide two distinct functionailities:

Multiplexing
Multiplexing has two roles: first, cope with existing protocol deficiencies (eg in HTTP) and second amortize the cost of a connection across multiple short-lived protocol sessions.
Protocol negotiation
Allows to know what protocols I can speak with some other parties, to take best advantage of the features we both support (eg multicast, real time protocols, etc).

With these two features, HTTPNG becomes a framework of protocols. Smart transition path (one can advertize new protocols incrementally), specialized protocols were needed (stop the POST-tunneling tragedy), more adaptative protocols were bandwidth constraints requires it and experiementations (proxy-to-proxy protocols etc).

The most important of these two features - at least from my point of view - is the second one. The first one may not last as long since it's main purpose is to overcome limitations in protocols that are meant to evolve in the right direction (eg HTTP can be fixed by itself, without requiring MUX, even if MUX ease the transition, at least from an implementor perspective).

Orthognal to these goals, we have a set of constraints that has to be taken into account, all of them have been noted by the two previous proposals except for the first and second ones (I have ordered them according to my own level of preferences):

Let's go through each of these.

Firewalls and proxies

HTTP has shown that these are to be taken into account as soon as possible when designing new protocols.

A firewall administrator should be able to configure easily what protocols are allowed to come in get out from his firewall on any mux session. This has some impact on the way mux specifies the protocol to be used on a given session. It also has impacts on the way protocols are negotiated, since the firewall has to know enough about protocol ids to be able to make a policy (eg HTTP should be bound to some well-known protocol identifier).

Note: From my point of view, this argue for fixed protocol ids when available, this will also help dealing with the simple cases.

Run protocols outside the scope of the mux physical link

I want to be able to use PNP to negotiate protocols on some other links: this allows to advertize/negotiate and use various transport layers without limiting us to the single connection that runs MUX.

A typical situation: say my browser knows about some real time protocol (UDP based) and the server also knows about that protocol, it might be much better for them to party using this protocol rather then some other (less adequat) common denominator (say some TCP based protocol).

This also has some implications of what negotiation means: negotiation is now not only about knowing what protocol I can use between some client/server, it also is about locating servers. Basically the result of one procol negotiation should result in two things:

Simple case should be simple

We all agree on that one: speaking http shouldn;t require any negotiation, but rely on shared knowledge (ie well known protocol ids, be it 80, as is specified in the current spec).

Authentication and confidentiality

I am not a security expert !

At the mux layer, we should make sure that either the appropriate hooks exist, or that they can be plugged into easily to support these two things (without hard coding any alorithm, of course). Something like RSTP ID and ID-REP request/reply.

A proposal

Given the above goals, my proposition is much more complex then the one that have been presented before. Th rest of this section is again out of my head, with no filtering.

They are three operations that PNP should support:

Protocol advertizing
Some way to advertize supported protocols
Protocol locating
A way to locate a server for a given protocol
Protocol resolution
A way to turn a protocol description into a protocol identifier suitable for use at the mux layer, for example.

Protocol advertizing

Some mean to advertize the protocols I speak, along with the various stacks that can be used to access them. My preference (at this very early stage) would be to go for some X-atom like protocol.

The basic protocol should provide support for interning atoms inside a given server, have some well-known predefined atom shared among all implementations.

A protocol stack is described by an ordered collection of atoms. Examples:

The tcp:ssl:gzip:http protocol stack.
Assuming tcp, ssl, gzip and http are well defined atom, whose identifiers are resp. 93, 65, 79, 89 the identifier of the above protocol stack would be { 93, 65, 79, 89 }.
The tcp:abaird-secure-layer:abaird protocol stack
As abaird-secure-layer and abaird are not well known atoms, for a client to compute the above protocol identifier would require one round trip (to intern abaird and abaird-secure-layer). Atom identifier allocation can be either client or server side. In the first case, the exchange would look like:
CLIENT -> INTERN abaird-secure-layer, abaird
SERVER <- 19, 67
In the second case (no round trip required):
CLIENT -> INTERN abaird-secure-layer, abaird, 19, 67
Anyway, in that case the protocol identifier would then be:
 { 93, 19, 67 }
Note that this is not a "well known protocol identifier", which is fine since that protocol is probably experimental, and no proxy or firewall need to know about it specifically.

This atom or token based language allows for protocol stack description, one could follow the same kind of route for protocol parameters...

Protocol resolution

Protocol resolution is the operation that given a protocol identifier (as "specified" above) results in a protocol identifier. A protocol identifier is probably a small number (0xffff) designating one protocol. As said above, they are two kind of protocols identifiers:

Well known protocol identifiers
The protocol identifier that are to be recognized by firewall administrators, say HTTP, MUX, etc. For these identifiers, one can (should ?) reuse assigned tcp/udp port numbers.
Extension protocol identifiers
These are created specifically by servers supporting them (such as the IABAIRDI protocol above). Their life time is small (eg asking for the protocol identifier of tcp:abaird-secure-layer:abaird twice at different time might not yield the same result).

Protocol locating

Once the protocol has been negotiateed and resolved into an identifier, the next step is to start speaking it with someone. Protocol locatin is about getting the address of a server to which the client can speak that protocol. Typically this operation is of the kind:

protocol identifier -> URL

If I sum up all these, the "protocol negotiation protocol" has the following requests/replies:

Op input output
InternAtoms StringArray ShortArray
ResolveProtocol ShortArray Short
LocateProtocol Short URL

Locating a protocol is optional (not need to locate a protocol if it is to run on top of mux itself). For a client to speak HTTP requires no PNP interactions at all, for speaking any other well known tcp protocol, since protocol identifier are mapped to tcp port numbers for all of those.

Given the complexity (and optionality) of PNP, I would suggest that it be a separate protocol from mux itself, and that they live together by sharing the notion of protocol identifiers: PNP is used to get some protocol identifier, then that protocol identifier can be used straight on top of mux if needed.