Caching and Content Negotiation

[Neither of the note-takers fully understands the issues here, and I was out of the room for a lot of this discussion doing other tasks, so this is a bit sketchy. I would appreciate some editing of this section by anyone who thinks they understand better what happened!]

For the purposes of caching, the two key points seem to be:

(1) How does the cache decide if can return a response from its cache or if it must contact the origin server?

(2) How does the origin server know if the cache already holds the response that it should return (and hence we could avoid actually transferring the body of the response)?

Question (1) is the topic that has seen the most discussion so far. We seem to agree that there are at least these three cases to handle:

(a) there is exactly one variant of the resource. Therefore, if there is a cache entry and it is fresh, the cache can use it without checking with the server.
(b) there is a fixed and small set of variants, and the cache knows the algorithm that the server uses to choose the variant to send in the response. Therefore, the cache can decide whether it has the right response already, or if it must obtain it from the server.
(c) there is a fixed and small set of variants, but the cache does not know the algorithm to pick the right variant. This means that unless the client's request headers "suitably match" [see below for what this means] the effective cache key for an existing cache entry, the cache must contact the origin server.

KC's notes include a fourth case, "large # alternatives" but I can't remember or figure out if this is significantly different from cases (b) or (c). That is, I would guess that operationally this is essentially the same as case (c) once "large" gets large enough.

We have two proposed mechanisms for dealing with this that appear to be more or less complementary. There is the "Vary:" response header, which the server uses to inform the cache that the response depends on zero or more of the request header fields (besides the URL, of course). For example, "Vary: {const}" means that the response does not depend on anything, and so the cache knows it is in case (a) [only one variant].

Even for case (c), I believe it is safe to assume that if all of the request fields match those stored with a cached value, then the cache can return that value to the client (if the Expires: value permits this). The Vary: header allows us to loosen this restriction somewhat: the "suitable match" between the new request headers and the previous headers is determined by the Vary: value, so if (for example) the response does not depend on the User-agent: header, the cache can ignore that in deciding what to do.

If we are in case (b), the cache knows (from Vary:) what headers matter (for example, Accept-language:) but it still needs to know the universe of available variants (languages, in this example) before it can decide whether it already has the right response. So this means that the server needs a way to communicate this set (and perhaps other discrimination values) to the cache, which is apparently what the URI: response header is for.

Back several paragraphs, I mentioned a second key point:

(2) How does the origin server know if the cache already holds the response that it should return (and hence we could avoid actually transferring the body of the response)?

We talked about a number of possible solutions for this. For example, a cache could annotate its request to the origin server with all of the information it has about the variants it already has in its cache, and let the origin server decide. But that seems to require huge amounts of request headers on all requests, which somewhat defeats the purpose of occasionally not sending a response body.

Someone also proposed tagging variants with unique URIs. However, this might not work for some kinds of content negotiation.

After some discussion, Jeff proposed a "variant-ID" mechanism that would provide a compact (and optional) way of communicating between cache and server what variants the cache already held. Someone else suggested that this also needed to include cache validators for each of the variants, so that the necessary response *would* be returned if the cached copy was no longer valid.

It was agreed that this scheme had the advantage (over URI-tagging) that there was little or no chance that the tags could leak out into other contexts (i.e., nobody would try to use them instead of a proper URL).

I described this in more detail today in

http://www.roads.lut.ac.uk/lists/http-caching/0254.html

ACTION ITEM: Paul Leach willing to write up a proposed spec with Jeff's help. Larry Masinter willing(?) to integrate this with other content-negotiation stuff.

http working group issues