URIs were originally used primarily to identify documents on the Web, or with the use of fragment identifiers, portions of those documents. As Web content has evolved to include Javascript and similar applications that have extensive client-side logic, a need has arisen to use URIs to identify states of such applications, to provide for bookmarking and linking those states, etc. This finding sets out some of the challenges of using URIs to identify application states, and recommends some best practices. A more formal introduction to the Finding and its scope can be found in its abstract.
The W3C TAG would like to thank Ashok Malhotra, who did much of the analysis and editing for this work, and also former TAG member T.V. Raman, who first brought this issue to the TAG's attention, and who wrote earlier drafts on which this finding is based.
]]>There’s been quite a bit of discussion recently about the use of hash-bang URIs following their adoption by Gawker, and the ensuing downtime of that site.
Gawker have redesigned their sites, including lifehacker and various others, such that all URIs look like http://{domain}#!{path-to-content} — the #! is the hash-bang. The home page on the domain serves up a static HTML page that pulls in Javascript that interprets the path-to-content and requests that content through AJAX, which it then slots into the page. The sites all suffered an outage when, for whatever reason, the Javascript couldn’t load: without working Javascript you couldn’t actually view any of the content on the site.
This provoked a massive cry of #FAIL (or perhaps that should be #!FAIL) and a lot of puns along the lines of making a hash of a website and it going bang. For analysis and opinions on both sides, see:
While all this has been going on, the TAG at the W3C have been drafting a document on Repurposing the Hash Sign for the New Web (originally named Usage Patterns For Client-Side URI parameters in April 2009) which takes a rather wider view than just the hash-bang issue, and on which they are seeking comments.
All matters of design involve weighing different choices against some criteria that you decide on implicitly or explicitly: there is no single right way of doing things on the web. Here, I explore the choices that are available to web developers around hash URIs and discuss how to mitigate the negative aspects of adopting the hash-bang pattern.
The semantics of hash URIs have changed over time. Look back at RFC 1738: Uniform Resource Locators (URL) from December 1994 and fragments are hardly mentioned; when they are, they are termed “fragment/anchor identifiers”, reflecting their original use which was to jump to an anchor within an HTML page (indicated by an <a> element with a name attribute; those were the days).
Skip to RFC 2396: Uniform Resource Identifiers (URI): Generic Syntax from August 1998 and fragment identifiers have their own section, where it says:
When a URI reference is used to perform a retrieval action on the identified resource, the optional fragment identifier, separated from the URI by a crosshatch (“#”) character, consists of additional reference information to be interpreted by the user agent after the retrieval action has been successfully completed. As such, it is not part of a URI, but is often used in conjunction with a URI.
At this point, the fragment identifier:
Forward to RFC 3986: Uniform Resource Identifier (URI): Generic Syntax from January 2005 and fragment identifiers are defined as part of the URI itself:
The fragment identifier component of a URI allows indirect identification of a secondary resource by reference to a primary resource and additional identifying information. The identified secondary resource may be some portion or subset of the primary resource, some view on representations of the primary resource, or some other resource defined or described by those representations.
This breaks away from the tight coupling between a fragment identifier and a representation retrieved from the web and purposefully allows the use of hash URIs to define abstract or real-world things, addressing TAG Issue 37: Definition of abstract components with namespace names and frag ids and supporting the use of hash URIs in the semantic web.
Around the same time, we have the growth of AJAX, where a single page interface is used to access a wide set of content which is dynamically retrieved using Javascript. The AJAX experience could be frustrating for end users, because the back button no longer worked (to let them go back to previous states of their interface) and they couldn’t bookmark or share state. And so applications started to use hash URIs to track AJAX state (that article is from June 2005, if you’re following the timeline).
And so we get to hash-bangs. These were proposed by Google in October 2009 as a mechanism to distinguish between cases where hash URIs are being used as anchor identifiers, to describe views, or to identify real-world things, and those cases where they are being used to capture important AJAX state. What Google proposed is for pages where the content of the page is determined by a fragment identifier and some Javascript to also be accessible by combining the base URI with a query parameter (_escaped_fragment_={fragment}). To distinguish this use of hash URIs from the more mundane kinds, Google proposed starting the fragment identifier #! (hash-bang). Hash-bang URIs are therefore associated with the practice of transcluding content into a wrapper page.
To summarise, hash URIs are now being used in three distinct ways:
Hash-bang URIs are a particular form of the third of these. By using them, the website indicates that the page uses client-side transclusion to give the true content of the page. If it follows Google’s proposal, the website also commits to making that content available through an equivalent base URI with a _escaped_fragment_ parameter.
Let’s have a look at how hash-bang URIs are used in a couple of sites.
First, we’ll look at lifehacker, which is one of Gawker’s sites whose switch to hash-bangs triggered the recent spate of comments. What happens if I link to the article http://lifehacker.com/#!5770791/top-10-tips-and-tricks-for-making-your-work-life-better?
The exact response to this request seems to depend on some cookies (it didn’t work the first time I accessed it in Firefox, having pasted the link from another browser). If it works as expected, in a browser that supports Javascript, the browser gets the page at the base URI http://lifehacker.com/, which includes (amongst a lot of other things) a script that POSTs to http://lifehacker.com/index.php?_actn_=ajax_post a request with the data:
op=ajax_post
refId=5770791
formToken=d26bd943151005152e6e0991764e6c09
The response to this POST is a 53kB JSON document that contains a bit of metadata about the post and then its escaped HTML content. This gets inserted into the page by the script, to display the post. As this isn’t a GETtable resource, I’ve attached this file to this post so you can see what it looks like.
(Honestly, I could hardly bring myself to describe this: a POST to get some data? a .php URL? query parameter set to ajax_post? massive amounts of escaped HTML in a JSON response? Geesh. Anyway, focus… hash-bang URIs…)
A browser that doesn’t support Javascript simply gets the base URI and is none the wiser about the actual content that was linked to.
What about the _escaped_fragment_ equivalent URI, http://lifehacker.com/?_escaped_fragment_=5770791/top-10-tips-and-tricks-for-making-your-work-life-better? If you request this, you get back an 200 OK response which is an HTML page with the content embedded in it. It looks just the same as the original page with the embedded content.
What if you make up some rubbish URI, which in normal circumstances you would expect to give a 404 Not Found response? Naturally, a request to the base URI of http://lifehacker.com/ is always going to give a 200 OK response, although if you try http://lifehacker.com/#!1234/made-up-page you get page furniture with no content in the page. A request to http://lifehacker.com/?_escaped_fragment_=1234/made-up-page results in a 301 Moved Peramently to the hash-bang URI http://lifehacker.com/#!1234 rather than the 404 Not Found that we’d want.
Now let’s look at Twitter. What happens if I link to the tweet http://twitter.com/#!/JeniT/status/35634274132561921? Although it’s not indicated in the Vary header, Twitter determines what to do about any requests to this hashless URI based on whether I’m logged in or not (based on a cookie).
If I am logged on, I get the new home page. This home page GETs (through various iframes and Javascript obfuscation) several small JSON files through Twitter’s API:
http://api.twitter.com/1/statuses/show.json?include_entities=true&contributor_details=true&id=35634274132561921: the details of the tweethttp://api.twitter.com/1/statuses/35634274132561921/retweeted_by.json?count=15: details about retweetshttp://api.twitter.com/1/users/lookup.json?user_id=&screen_name=unhosted: details about the twitter user @unhosted, who was mentioned in the tweetThis JSON gets converted into HTML and embedded within the page using Javascript. All the links within the page are to hash-bang URIs and there is no way of identifying the hashless URI (unless you know the very simple pattern that you can simply remove it to get a static page).
If I’m not logged on but am using a browser that understands Javascript, the browser GETs http://twitter.com/; the script in the returned page picks out the fragment identifier and redirects (using Javascript) to http://twitter.com/JeniT/status/35634274132561921.
If, on the other hand, I’m using curl or a browser without Javascript activated, I just get the home page and have no idea that the original hash-bang URI was supposed to give me anything different.
The response to the hashless URI http://twitter.com/JeniT/status/35634274132561921 also varies based on whether I’m logged in or not. If I am, the response is a 302 Found to the hash-bang URI http://twitter.com/#!/JeniT/status/35634274132561921. If I’m not, for example using curl, Twitter just returns a normal HTML page that contains information about the tweet that I’ve just requested.
Finally, if I request the _escaped_fragment_ version of the hash-bang URI http://twitter.com/?_escaped_fragment_=/JeniT/status/35634274132561921 the result is a 301 Moved Permanently redirection to the hashless URI http://twitter.com/JeniT/status/35634274132561921 which can be retrieved as above.
Requesting a status that doesn’t exist such as http://twitter.com/#!/JeniT/status/1 in the browser results in a page that at least tells you the content doesn’t exist. Requesting the equivalent _escaped_fragment_ URI redirects to the hashless URI http://twitter.com/JeniT/status/1. Requesting this results in a 404 Not Found result as you would expect.
Why are these sites using hash-bang URIs? Well, hash URIs in general have four features which make them useful to client-side applications: they provide addresses for application states; they give caching (and therefore performance) boosts; they enable web applications to draw data from separate servers; and they may have SEO benefits.
Interacting with the web is all about moving from one state to another, through clicking on links, submitting forms, and otherwise taking action on a page.
Backend databases on web servers, cookies, and other forms of local storage provide methods of capturing application state, but on the web we’ve found that having addresses for states is essential for a whole bunch of things that we find useful:
On the web, the only addressing method that meets these goals is the URI. Addresses that involve more than a URI, such as “search http://example.com/ with the keyword X and click on the third link” or “access http://example.org/ with cookie X set to Y” or “access http://example.net with the HTTP header X set to Y” simply don’t work. You can’t bookmark them or link to them or put them on the side of a bus.
Application state is complex and multi-faceted. As a web developer, you have to work out which parts of the application state need to be addressable through URIs, which can be stored on the client and which on a server. They can be classified into four rough categories; states that are associated with:
States that have different content almost certainly need to have different URIs so that it’s possible to link to that content (the web being nothing without links). At the other extreme, it’s very unlikely that the state of a drop-down menu would need to be captured at all. In between is a large grey area, where a web developer might decide not to capture state at all, to capture it in the client, in the server, or to make it addressable by giving it a URI.
If a web developer chooses to make a state addressable through a URI, they again have choices to make about which part of the URI to use: should different states have different domains? different paths? different query parameters? different fragment identifiers? Hash URIs make states addressable that developers might otherwise leave unaddressable.
To give some examples, on legislation.gov.uk we have decided to:
/ukpga/1985/67/section/6/ukpga/1985/67/section/6?view=timeline&timeline=true/ukpga/1985/67/section/6#section-6-2/ukpga/1985/67/section/6#text%3Dschool%20busThe last of these states would probably have gone un-addressed if we couldn’t use a hash URI for it. The only changes that it makes to the normal page are currently to the links to other legislation content, so that you can go (back) to a highlighted table of contents (though we hope to expand it to provide in-section highlighting). Given that we rely heavily on caching to provide the performance that we want and that there’s an infinite variety of free-text search terms, it’s simply not worth the performance cost of having a separate base URI for those views.
Fragment identifiers are currently the only part of a URI that can be changed without causing a browser to refresh the page (though see the note below). Moving to a different base URI — changing its domain, path or query — means making a new request on the server. Having a new request for a small change in state makes for greater load on the server and a worse user experience due both to the latency inherent in making new requests and the large amount of repeated material that has to be sent across the wire.
Note: HTML5 introduces
pushState()andchangeState()methods in its history API that enable a script to add new URIs to the browser’s history without the browser actually navigating to that page. This is new functionality, at time of writing only supported in Chrome, Safari and Firefox (and not completely in any of them) and unlikely to be included in IE9. When this functionality is more widely adopted, it will be possible to change state to a new base URI without causing a page load.
When a change of state involves simply viewing a different part of existing content, or viewing it in a different way, a hash URI is often a reasonable solution. It supports addressability without requiring an extra request.
Things become fuzzier when the same base URI is used to support different content, where transclusion is used. In these cases, the page that you get when you request the base URI itself gets content from the server as one or more separate AJAX requests based on the fragment identifier. Whether this ends up giving better performance depends on a variety of factors, such as:
Hash URIs can also be very useful in distributed web applications, where the code that is used to provide an interface pulls in data from a separate, unconnected source. Simple examples are mashups that use data provided by different sources, requested using AJAX, and combine that data to create a new visualisation.
But more advanced applications are beginning to emerge, particularly as a reaction to silo sites such as Google and Facebook, which lock us in to their applications by controlling our data. From the unhosted manifesto:
To be unhosted, a website’s code will need to be very ajaxy first, so that all the servers do is store and serve json data. No server-side processing. This is because we need to switch from transport-layer encryption to client-side payload encryption (we no longer necessarily trust the server we’re talking to). From within the app’s source code, that should run entirely in JavaScript and HTML5, json-objects can be stored, retrieved, sent, and received. The user will have the same experience (we even managed to avoid needing a plugin), but the website is unhosted in the sense that the servers you talk to only see encrypted data and don’t even know which application you are running.
The aim of unhosted is to separate application code from user data. This divides servers (at least functionally) into those that store and make available user data, and those that host applications and any supporting code, images and so on. The important feature of these sites is that user data never passes through the web application’s server. This frees users to move to different applications without losing their data.
This doesn’t necessarily stop the application server from doing any processing, including URI-based processing; it is only that the processing cannot be based on user data — the content of the site. Since this content is going to be accessed through AJAX anyway, there’s little motivation for unhosted applications to use anything other than local storage and hash URIs to encode state.
A final reason for using hash URIs that I’ve seen cited is that it increases the page rank for the base URI, because as far as a search engine is concerned, more links will point to the same base URI (even if in fact they are pointing to a different hash URI). Of course this doesn’t apply to hash-bang URIs, since the point of them is precisely to enable search engines to distinguish between (and access content from) URIs whose base URI is the same.
So hash-bangs can give a performance improvement (and hence a usability improvement), and enable us to build new kinds of web applications. So what are the arguments against using them?
The main disadvantages of using hash URIs generally to support AJAX state arise due to them having to be interpreted by Javascript. This immediately causes problems for:
The most recent statistic I could find, about access to the Yahoo home page indicates that up to 2% of access is from users without Javascript (they excluded search engines). According to a recent survey, about the same percentage of screen reader users have Javascript turned off.
This is a low percentage, but if you have large numbers of visitors it adds up. The site that I care most about, legislation.gov.uk, has over 60,000 human visitors a day, which means that about 1,200 of them will be visiting without Javascript. If our content were completely inaccessible to them we’d be inconveniencing a large number of users.
Depending on hash-bang URIs to serve content is also brittle, as Gawker found. If the Javascript that interprets the fragment identifier is temporarily inaccessible or unable to run in a particular browser, any portions of a page that rely on Javascript also become inaccessible.
There are other, less obvious, impacts which occur when you use a hash-bang URI.
The URI held in the HTTP Referer header “MUST NOT include a fragment”. As Mike Davies noted, this prevents such URIs from showing up in server logs, and stops people from working out which of your pages are linking to theirs. (Of course, this might be a good thing in some circumstances; there might be aspects of the state of a page that you’d rather a referenced server not know about.)
You should also consider the impact on the future proofing of your site. When a server knows the entirety of a URI, it can use HTTP mechanisms to indicate when pages have moved, gone, or never existed. With hash URIs, if you change the URIs you use on your site, the Javascript that interprets the fragment identifier needs to be able to recognise and support any redirections, missing, or never existing pages. The HTTP status code for the wrapper page will always be 200 OK, but be meaningless.
Even if your site structure doesn’t change, if you use hash-bang URIs as your primary set of URIs, you’re likely to find it harder to make a change back to using hashless URIs in the future. Again, you will be reliant in perpetuity on Javascript routing to decipher the hash-bang URI and redirect it to a hashless URI.
A final factor is that fragment identifiers can become overcrowded with state information. In a purely hash-URI-based site, what if you wanted to jump to a particular place within particular content, shown with a particular view? The hash URI has to encode all three of these pieces of information. Once you start using hash-bang URIs, there is no way to indicate within the URI (for search engines, for example) that a particular piece of the URI can be ignored when checking for equivalence. With normal hash URIs, there is an assumption that the fragment identifier can basically be ignored; with hash-bang URIs that is no longer true.
Having looked at the advantages and disadvantages, I would echo what seems to be the general sentiment around traditional server-based websites that use hash-bang URIs: pages that give different content should have different base URIs, not just different fragment identifiers. In particular, if you’re serving large amounts of document-oriented content through hash-bang URIs, consider swapping things around and having hashless URIs for the content that then transclude in the large headers, footers and side bars that form the static part of your site.
However, if you are running a server-based, data-driven web application and your primary goal is a smooth user experience, it’s understandable why you might want to offer hash URIs for your pages to the 98% of people who can benefit from it, even for transcluded content. In these cases I’d argue that you should practice progressive enhancement:
_escaped_fragment_ query parameter, the result of which should be a redirection to the appropriate hashless URIThis is roughly what Twitter has done, except that it doesn’t make it easy to get the hashless URI from a page or from links within the page. Of course the mapping in Twitter’s case is the straight-forward removal of the #! from the URI, but as a human it’s frustrating to have to do this by hand.
The above measures ensure that your site will remain as accessible as possible to all users and provides a clear migration path as the HTML5 history API gains acceptance. The slight disadvantage is that encouraging people to use hashless URIs for links means that you you can no longer depend quite so much on caching as the first page that people access in a session might be any page (whereas with a pure hash-bang scheme everyone goes to the same initial page).
Distributed, client-based websites can take the same measures — the application’s server can send back the same HTML page regardless of the URI used to access it; Javascript can pull information from a URI’s path as easily as it can from a fragment identifier. The biggest difficulty is supporting the static page through the _escaped_fragment_ convention without passing user data through the application server. I suspect we might find a third class of service arise: trusted third-party proxies using headless browsers to construct static versions of pages without storing either data or application logic. Time will tell.
There are some deeper issues here regarding web architecture. In the traditional web, there is a one-to-one correspondence between the representation of a resource that you get in response to a request from a server, and the content that you see on the page (or a search engine retrieves). With a traditional hash URI for a fragment, the HTTP headers you retrieve for the page are applicable to the hash URI as well. In a web application that uses transclusion, this is not the case.
Note: It’s also impossible to get metadata about hash URIs used for real-world or abstract things using HTTP; in these cases, the metadata about the thing can only be retrieved through interpreting the data within the page (eg an RDF document). Whereas with the
303 See Otherpattern for publishing linked data, it’s possible to use a404 Not Foundresponse to indicate a thing that does not exist, there is no equivalent with hash URIs. Perhaps this is what lies at the root of my feeling of unease about them.
With hash-bang URIs, there are in fact three (or more) URIs in play: the hash-bang URI (which identifies a wrapper page with particular content transcluded within it), a base URI (which identifies the wrapper HTML page) and one or more content URIs (against which AJAX requests are made to retrieve the relevant content). Requests to the base URI and the content URIs provide us with HTTP status codes and headers that describe those particular representations. The only way of discovering similar metadata about the hash-bang URI itself is through the _escaped_fragment_ query parameter convention which maps the hash-bang URI into a hashless URI that can be requested.
Does this matter? Do hash-bang URIs “break the web”? Well, to me, “breaking the web” is about breaking the implicit socio-technical contract that we enter into when we publish websites. At the social level, sites break the web when they withdraw support for URIs that are widely referenced elsewhere, hide content behind register- or pay-walls, or discriminate against those who suffer from disabilities or low bandwidth. At the technical level, it’s when sites lie in HTTP. It’s when they serve up pages with the title “Not Found” with the HTTP status code 200 OK. It’s when they serve non-well-formed HTML as application/xhtml+xml.
These things matter because we base our own behaviour on the contract being kept. If we cannot trust major websites to continue to support the URIs that they have coined, how can we link to them? If we cannot trust websites to provide accurate metadata about the content that they serve, how can we write applications that cache or display or otherwise use that information? On their own, pages that use Javascript-based transclusion break both the social side (in that they limit access to those with Javascript) and the technical side (in that they cannot properly use HTTP) of the contract.
But contracts do get rewritten over time. The web is constantly evolving and we have to revise the contract as new behaviours and new technologies gain adoption. The _escaped_fragment_ convention gives a life line: a method of programmatically discovering how to access the version of a page without Javascript, and to discover metadata about it through HTTP. It is not a pretty pattern (I would much prefer that the server returned a header containing a URI template that described how to create the hashless equivalent of a hash-bang URI, and to have some rules about the parsing of a hash-bang fragment identifier so that it could include other fragments identifiers) but it has the benefit of adoption.
In short, hash-bang URIs are an important pattern that will be around for several years because they offer many benefits compared to their alternatives, and because HTML5’s history API is still a little way off general support. Rather than banging the drum against hash-bang URIs, we need to try to make them work as well as they can by:
_escaped_fragment_ pattern when they do not have Javascript availableWe also need to keep a close eye on emerging patterns in distributed web applications to ensure that these efforts are supported in the standards on which the web is built.
]]>Most semantic web application developers are probably familiar with three ways to nose-follow from a URI:
In case 3, X refers to what I'll call a "web page" (a more technical term is used in the TAG's httpRange-14 resolution). One of the new RFCs extends case 3 to situations where the RDF can't be embedded in the content, either because the content-type doesn't provide a place to put it (e.g. text/plain) or because for administrative reasons the content can't be modified to include it (e.g. a web archive that has to deliver the original bytes faithfully). The others cover this case as well as offering improved performance in case 2.
Before getting into the new nose-following protocols, I'll amplify case 3 above by listing a few applications of RDF in which a web page occurs as a subject. I'll rather imprecisely call such RDF "metadata".
All sorts of other statements can be made about a web page, such as a type (wiki page, blog post, etc.), SKOS concepts, links to comments and reviews, duration of a recording, how to edit, who controls it administratively, etc. Anything you might want to say about a web page can be said in RDF.
Embedded metadata is easy to deploy and to access, and should be used when possible. But while embedded metadata has the advantages of traveling around with the content, a protocol that allows the server responsible for the URI to provide metadata over a separate "channel" has two advantages over embedded metadata: First, the metadata doesn't have to be put into the content; and second, it doesn't have to be parsed out of the content. And it's not either/or: There is no reason not to provide metadata through both channels when possible.
The 'Web Linking' proposed standard defines the HTTP Link: header, which provides a way to communicate links rooted at the requested resource. These links can either encode interesting information directly in the HTTP response, or provide a link to a document that packages metadata relevant to the resource.
In the former case, one might have:
Link: <http://xmlns.com/foaf/0.1/Document>;
rel="http://www.w3.org/1999/02/22-rdf-syntax-ns#type"
meaning that the request URI refers to something of type foaf:Document. In the latter case one might have:
Link: <http://example.com/about/foo.rdf>;
rel="describedby"; type=application/rdf+xml
meaning that metadata can be found in <http://example.com/about/foo.rdf>, and hinting that the latter resource might have a 'representation' with media type application/rdf+xml.
The motivation for the "well-known URIs" RFC is to collect all "well-known URIs" (analogous to "robots.txt") in a single place, a root-level ".well-known" directory, and create a registry of them to avoid collisions. The most pressing need comes from protocols such as webfinger and OpenID; see Eran Hammer-Lahav's blog post for the whole story.
For linked data, .well-known provides an opportunity for providing metadata for web pages, as well improving the efficiency of obtaining RDF associated with other "slash URIs", what is currently done using 303 responses.
Ever since the TAG's httpRange-14 decision in 2005, there have been concerns that it takes two round trips to collect RDF associated with a slash URI. While some might question why those complaining aren't using hash URIs, in any case the "well-known URIs" mechanism gives a way to reduce the number of round trips in many cases, eliminating many GET/303 exchanges.
The trick is to obtain, for each host, a generic rule that will transform the URI at that host that you want RDF for into the URI of a document that carries that RDF. This generic rule is stored in a file residing in the .well-known space at a path that is fixed across all hosts. That is: to find RDF for http://example.com/foo, follow these steps:
The form of the about-URI is chosen by the particular host, e.g. "http://example.com/foo,about" or "http://about.example.com/foo" or whatever works best.
Why is this fewer round trips than using 303? Because you can fetch and cache the generic rule once per site. The first use of the rule still costs an extra round trip, but subsequent URIs for a given site can be nose-followed without any extra web accesses.
A worked example can be found here.
As with any new protocol, figuring out exactly how to apply the new proposed standards will require coordination and consensus-building. For example, the choice of the "describedby" link relation and "host-meta" well-known URI need to be confirmed for linked data, and agreement reached on whether multiple Link: headers is in good taste or poor taste. (Link: and .well-known put interesting content in a peculiarly obscure place and it might be a good idea to limit their use.) Consideration should be given to Larry Masinter's suggestion to use multiple relations reflecting different attitudes the server might have regarding the various metadata sources: For example the server may choose to announce that it wants the Link: metadata to override any embedded metadata, or vice versa. Agreement should be reached on the use of Link: and host-meta with redirects (302 and so on) - personally I think it would be a great thing as you could then use a value-added forwarding service to provide metadata that the target host doesn't or can't provide.
This is not a particularly heavy coordination burden; the design odds-and-ends and implementations are all simple. The impetus might come from inside W3C (e.g. via SWIG) or bottom-up. All we really need to get this going are a bit of community discussion, a server, and a cooperating client, and if the protocols actually fill a need, they will take off.
For past TAG work on this topic, please see TAG issue 62 and the "Uniform Access to Metadata" memo.
]]>The new job starts in just another week or two; I'll update the contact information and such on my home page before I'm done here. While my new position is likely to keep me particularly busy for a few months, I hope to surface in Mad Mode from time to time; it's a blog where I'm consolidating writing on free software, semantic web research, and other things I'm mad-passionate about.
Thanks to all of you who contribute to the work at W3C; I'm proud of a lot of things that we built together. And thanks to all my mentors and collaborators who taught me, helped me, and challenged me.
The Web is an incredibly important part of so many parts of life these days, and W3C plays an important role in ensuring that it will work for everyone over the long haul. Although it's hard to leave an organization with a mission I support, I am excited to get into bioinformatics, and I look forward to what W3C and the Web community come up with next as well.
]]>I began with W3C membership. Through meetings, phone calls, technical conferences, and informal sessions I've met upwards of one hundred members and have had profound conversations with many of them.
I also made a point of meeting with organizations that are part of the ecosystem within which W3C works. This includes other standards organizations, government ministers, students, researchers in Web science, and thought leaders in the industry.
I also reached out to organizations that “should” be in W3C. Often this includes presenting our activities and roadmaps. I've reached over one thousand people in this way.
And it was important to do this on a global basis. During these two and a half months I travelled to eight countries, but have spoken to participants from many other locations.
The primary purpose of all of these meetings was to listen. W3C has been an effective organization, but any organization can do better. What are the stakeholders of W3C asking from us?
W3C has established principles including Web for All and Web on Everything. We've established a technical vision as well. There is broad agreement to these principles and technical vision.
People are asking us to be more tangible and specific in how we achieve this.
There are many ways of summarizing the requests, but four recurring themes best capture the idea. W3C needs to:
Having identified clear imperatives, we are building teams that will look at each of these topics. Typically a team involves W3C staff, participating members, and outside experts. I expect to update you from time to time as this work gets underway.
As we try to improve the global accessible Web; the Web of Users, new standards work, and strong delivery of a core mission, there is a legitimate danger that we will find more work to do without the resource to do it. So we will also make sure that this clearer exposition of our mission is aligned with the resources required to complete that mission!
]]>If this is such a useful feature, then why does the browser address bar show the temporary URI instead of the permanent one? After all, the permanent one is the one you want to copy and paste to email, to bookmark, to place in HTML documents, and so on. The HTTP specification says to hang on to the permanent link ("since the redirection MAY be altered on occasion, the client SHOULD continue to use the Request-URI for future requests."). Tim Berners-Lee says the same thing in User Agent watch points (1998): "It is important that when a user agent follows a "Found" [302] link that the user does not refer to the second (less persistent) URI. Whether copying down the URI from a window at the top of a document, or making a link to the document, or bookmarking it, the reference should (except in very special cases) be to the original URI." Karl Dubost amplifies this in his 2001-2003 W3C Note Common User Agent Problems: "Do not treat HTTP temporary redirects as permanent redirects.... Wrong: User agents usually show the user (in the user interface) the URI that is the result of a temporary (302 or 307) redirect, as they would do for a permanent (301) redirect."
So why do browsers ignore the RFC and these repeated admonitions? Possibly due to lack of awareness of the issue, but more likely because the status quo is seen as protecting the user. If the original URI (the permalink) were shown we might have the following scenario:
an attacker discovers a way to establish a 3xx redirect from http://w3.org/resources/looksgood to http://phishingsite.org/pretendtobew3 - either because w3.org is being careless, or because of a conscious decision to deed part of its URI space to other parties
user sees address bar = http://w3.org/resources/looksgood with content X, and concludes that the X is attributable to the resource http://w3.org/resources/looksgood
user treats the http://w3.org/ prefix as an informal credential and treats the http://w3.org/resources/looksgood content as coming from W3C (without any normative justification; they just do) when in fact it's a phishing site pretending to be W3C
user enters their W3C password into phishing form, etc.
Were the user to observe address bar = http://phishingsite.org/pretendtobew3 with the same content, she might suspect an attack and decline to enter a password.
An attacker might make use of an explicit redirection service on a site similar to that provided by purl.org, or it might exploit a redirect script that takes a URL as part of the query string, e.g. http://w3.org/redirect?uri=http://phishingsite.org/pretendtobew3 .
This line of reasoning is documented in the Wikipedia article URL redirection and its references and in Mozilla bug 68423.
There are two possible objections. One is that the server in these cases is in error - it shouldn't have allowed the redirects if it didn't really mean for the content source to speak on behalf of the original resource (similar to an iframe or img element). The other is that the user is in error - s/he shouldn't be making authorization decisions based on the displayed URI; other evidence such as a certificate should be demanded. Unfortunately, while correct in theory, neither of these considerations is very compelling.
If browser projects are unwilling to change address bar behavior - and it seems unlikely that they will - is there any other remedy?
Perhaps some creative UI design might help. Displaying the permalink in addition to the tempolink might be nice, so that it could be selected (somehow) for bookmarking, but that might be confusing and take too much screen real estate. One possible partial solution would be an enhancement to the bookmark creation dialog. In Firefox on selecting "Bookmark This Page" one sees a little panel with text fields "name" and "tags" and pull-down "folder". What if, in the case of a redirection, there were an additional control that gave the option of bookmarking the permalink URI in place of the substitute URI? With further thought I bet someone could devise a solution that would work for URI copy/paste as well.
(Thanks to Dan Connolly, other TAG members, and David Wood for their help with this note.)
]]>The ideas behind the proposal presented here are neither particularly new nor particularly mine. I've made the effort to write this down so anyone wishing to refer to ideas in this space can say "Something along the lines of [this posting]" rather than "Something, you know, like, uhm, what we talked about, prefix binding, media-type-based defaulting, that stuff".
Criticism of XML namespaces as an appropriate mechanism for enabling distributed extensibility for the Web typically targets two issues:
Of these, the first is arguably the more significant, because the number of authors exceeds the number of developers by a large margin. Accordingly, this proposal attempts to address the first problem, by providing a defaulting mechanism for namespace prefix bindings which covers the 99% case.
dpd for use in the (X)HTML
header;xml-dpd and/or an
attribute xml:dpd for use at the top of XML
documents;XML namespaces provide two essentially distinct mechanisms for 'owning' names, that is, preventing what would otherwise be a name collision by associating names in some way with some additional distinguishing characteristic:
In XML namespaces as they stand today, the association with a URI is done via a namespace declaration which takes the form of an attribute, and whose impact is scoped to the subtree rooted at the owner element of that attribute.
Liam Quin has proposed an additional, out-of-band and defaultable, approach to the association for unprefixed names, using patterns to identify the subtrees where particular URIs apply. I've borrowed some of his ideas about how to connect documents to prefix binding definitions.
The approach presented here is similar-but-different, in that its primary goal is to enable out-of-band and defaultable associations of namespaces to names with prefixes, with whole-document scope. The advantages of focussing on prefixed names in this way are:
aria- and data-;Provision is also made for optionally specifying a binding for the default namespace at the document element, primarily for the media type registry case, where it makes sense to associate a primary namespace with a media type.
If this proposal were adopted, and a dpd document for use in HTML 4.01 or XHTML1:
<dpd ns="http://www.w3.org/1999/xhtml"> <pd p="xf" ns="http://www.w3.org/2002/xforms"/> <pd p="svg" ns="http://www.w3.org/2000/svg"/> <pd p="ml" ns="http://www.w3.org/1998/Math/MathML"/> </dpd>
was registered against the text/html media type, the following would result in a DOM with html and body elements in the XHTML namespace and an input element in the XForms namespace:
<html> <body> <xf:input ref="xyzzy">...</xf:input> </body> </html>
The general principle of platform design is that platforms consist of a set of standard interfaces. Standard interfaces allow substitution of components across the interface boundary, while independence of interfaces allow evolution of the interfaces themselves. In a PC, for example, the disk bus interface allows many different disk vendors to offer disk products independent of the model of display or keyboard, but the orthogonality of interfaces allow evolution of the interfaces themselves. If the display interface were linked to the disk interface too tightly, it wouldn't be possible to evolve ISA to SATA without updating VGA.
In the web platform, the three important interfaces are transport, format and reference, and the current definitions of those interfaces are HTTP, HTML and URI. The interfaces are standard, allowing many different implementations: HTTP standard lets you use HTTP servers from many vendors, the HTML standard lets you use many different HTML authoring tools or template systems, and the URI specification allows identification of many different components.
While HTTP is the current "common denominator" protocol that all web agents are expected to speak, the web should continue to work if web content is delivered by other protocols -- FTP, shared file systems, email, instant messaging, and so forth. HTTP as it has evolved has severe difficulties, and designing a Web that only works with HTTP as it is currently implemented and deployed would unfortunate. We should work harder to reduce the dependencies and isolate them.
HTML is the 'lingua franca', the common language that all agents are currently expected to be able to produce, process, read and interpret (or at least a well-defined subset of it). Having a common language is important for interoperability, but the web should also work for other formats -- extensions to HTML including scripting, DOM APIs, but also other formats and application environments such as XHTML, Java, PDF, Flash, Silverlight, XForms, 3D objects, SVG, other XML languages and so forth. Certainly HTML has it has evolved is overly complex for the purposes to which it is designed.
The URI is the fundamental element of reference, but the URI itself is evolving to deal with internationalization, reference to session state, IRIs, LEIRIs, HREFs and so forth. Many applications use URIs and IRIs, not just the formats described above but other protocols and locations, including databases, directories, messaging, archiving, peer-to-peer sharing and so forth.
The is just one of many communication applications on the global Internet; for web browsing to integrate will with the rest of the distributed networking, web components should be independent of the application, and work well with messaging, instant messaging, news feeds, etc etc.
A sign of a breakdown of this architectural principle would be for a specification of a format (say HTML) to attempt to redefine, for its purposes, the protocol (say HTTP) or the method of reference (URI). The specifications should be independent, or at least, dependencies isolated, minimized, reduced. If those other elements of the web architecture are incorrect, need to evolve to meet current practice or have flaws in their definitions, they need to evolve independently, so that orthogonality of the specifications and reusability of the components are the promoted.
There may well be reasons to link some features of HTML to the fact that it is delivered over an interactive protocol, but linking HTML directly to HTTP in a way that features would work only for HTTP and not for any other protocol with similar features – that would be unfortunate. It might not matter in the short-term (that’s all we have right now) but it is harmful to the long-term evolution of the web.
(Should go without saying, but just in case: this is a personal post, not reviewed by the TAG)
]]>In some posts on the www-tag mailing list, I was trying to point out the risks in defining languages such that the "meaning" of the language depends on operational behavior. In some ways, of course, this is a fallacy: in general, what an utterance "means" in some operational way depends on what the speaker intends and how the listener will interpret the utterance.
However, as an organization, W3C can, and should, define languages in which the meaning is defined in the document, in terms of abstractions rather than in terms of operational behavior. The result is more robust standards, those that have wider applicability, that can be used for more purposes, and that create a more vibrant and extensible web.
Google's documentation shows support for both microformats and RDFa. It follows the hReview microformat syntax with small vocabulary changes (name vs fn). Support for RDFa syntax, in theory, means support for vocabularies that anyone makes; but in practice, Google is starting with a clean slate: data-vocabulary.org. That's a place to start, though it doesn't provide synergy with anyone who has uses FOAF or Dublin Core or the like to share their data.
The policy questions are perhaps the most difficult. Structured data is a pointy instrument; if anyone can say anything about anything, surely the system will be gamed and defrauded. Google's rollout is one step at a time, starting with some trusted sites and an application process to get your site added. The O'Reilly interview with Guha and Hansson is an interesting look at where they hope to go after this first step; if you're curious about how this fits in to HTML standards, see Sam Ruby's microdata.
While issues remain--there are syntactic i's to dot and t's to cross and even larger policy issues to work out--between Google's rollout and Yahoo's searchmonkey and the UK Central Office of Information rollout, it seems that the industry is ready to take on the challenges of using structured data in search engines.
Unfortunately, hAudio changed out from under me between when I started and when I finished. So this time, a simple search found the music ontology and I tried it with RDFa, which lets you use any RDF vocabulary in HTML*. I'm mostly pleased with the results:
- from A Song's Best Friend_ The Very Best Of John Denver [Disc 1]
by John Denver
Poems, Prayers And Promises- from WOW Worship (orange)
by Compilations
Did you Feel the Mountains Tremble- from Family Music Party
by Trout Fishing In America
Back When I Could Fly
The album names come before the track names because I didn't read enough of the the RDFa primer when I was coding; RDFa includes @rev as well as @rel for reversing subject/object order. See an advogato episode on m3uin.py for details about the code.
The Music Ontology was developed by a handful of people who staked out a claim in URI space (http://musicontology.org/...) and happily took comments from as big a review community as they could manage, but they had no obligation to get a really global consensus. The microformats process is intended to reach a global consensus so that staking out a claim in URI space is superfluous; it works well given certain initial conditions about how common the problem is and availability of pre-web designs to draw from. Perhaps playlists (and media syndication, as hAudio seems to be expanding in scope to hMedia) will eventually reach these conditions, but the music ontology already meets my needs, since I'm the sort who doesn't mind declaring my data vocabulary with URIs.
My view of Web architecture is shaped by episodes such as this one. While giga-scale deployment is always impressive and definitely something we should design for, small scale deployment is just as important. The Web spread, initially, not because of global phenomena such as Wikipedia and Facebook but because you didn't need your manager's permission to try it out; you didn't even need a domain name; you could just run it on your LAN or even on just one machine with no server at all.
In an Oct 2008 tech plenary session on web architecture, Henri Sivonen said:
I see the Web as the public Web that people can access. The resources you can navigate publicly. I define Web as the information space accessible to the public via a browser.
If a mobile operator operates behind walls, this is not part of the Web.
I can't say that I agree with that perspective. I'm no great fan of walled gardens either, but freedom means freedom to do things we don't like as well as freedom to do things we do like. And architecture and policy should have a sort of church-and-state separation between them.
Plus, data interchange happens not just at planetary scale, but also within mobile devices, across devices, and across communities and enterprises of all shapes and sizes.
I've gone a little outside the scope of current standards; RDFa has only been specified for use in modular XHTML, with the application/xhtml+xml media type, so far.
See also:
]]>Just after the mobile buzz at Web Directions North and the TAG declared victory on how to build The Self-Describing Web with URI-based Extensibility , I get some details on how Palm is building on the open web platform:
A widget is declared within your HTML as an empty div with an x-mojo-element attribute.
<div x-mojo-element="ToggleButton" id="my-toggle"></div>
Oh great; x- tokens... aren't those passe by now?
The suggestion in the HTML 5 draft is data-* attributes. The ARIA draft suggests @role. The Palm design looks like new information for issue-41, Decentralized-extensibility, in the HTML WG.
Anybody know how frozen the Palm design is? Or if they looked at ARIA, data-* or URI-based namespaces?
]]>Sorry. You must have JavaScript enabled to view this page. Click the BACK button below or enable JavaScript in your browser preferences and click TRY AGAIN.Let's turn that around, shall we? Sorry, if you're a network provider and you want my business, read up on unobtrusive javascript (aka the rule of least power), go BACK to work on your web site design and TRY AGAIN.
The first time someone told me W3C should be working on specs help the browser prevent sensitive data from leaking out of enterprises, I didn't get it. "Use the browser as part of the trusted computing base? Are you kidding?" was my response. I didn't see the bigger picture. Crockford explains in an April 2008 item:
... there are multiple interests involved in a web application. We have here the interests of the user, of the site, and of the advertiser. If we have a mashup, there can be many more interests.
Most of my study of security protocols concentrated on whether a request from party A should be granted by party B. You know, Alice and Bob. Using BAN logic to analyze the Kerberos protocols was very interesting.
I also enjoyed studying capability security and the E system, which is a fascinating model of secure multi-party communication (not to mention lockless concurrency), though it seems an impossibly high bar to reach, given the worse-is-better tendency in software deployment, and it seemed to me that capabilities are a poor match for the way linking and access control work in the Web:
The Web provides several mechanisms to control access to resources; these mechanisms do not rely on hiding or suppressing URIs for those resources.
On the other hand, after wrestling with the patchwork of javascript security policies in browsers in the past few weeks, the capability approach in adsafe looks simple and elegant by comparison. Is there any chance we can move the state-of-the-art that far? And what do we do in the mean time? Crockford's Jan 2008 post is quite critical of W3C's current work:
This same sort of wrong-end-of-the-network thinking can be seen today in the HTML 5 working group's crazy XHR access control language.
Access Control for Cross-Site Requests is a mouthful, and "Access Control" is too generic, which leads to "W3C Access Control". Didn't we already go through this with "W3C XML Schema"? Generic names are awkward. I think I'll call it WACL... yeah... rhymes with spackle... let's see if it sticks. Anyway...
Crockford's comment cites his proposal and argues...
JSONRequest does not allow the server to abdicate its responsibility of deciding if the data should be delivered to the browser. Therefore, no policy language is needed. JSONRequest requires explicit authorization. Cookies and other tokens of ambient authority are neither sent nor delivered.
I'm not sure I understand that. I'm glad to learn there's more to the difference between XMLHttpRequest and JSONRequest than just <pointy-brackets> vs {curly-braces}, but I'd like to understand better how "ambient authority" relates to the interests of users, sites, advertisers, and the like.
In response, the FAQ in the WACL spec says:
JSONRequest has been considered by the Web Applications Working Group and the group has concluded that it does not meet the documented requirements. E.g., requests originating from the JSONRequest API cannot include credentials and JSONRequest is format specific.
Including credentials seems more like a solution than a requirement; can someone help me understand how it relates to the multiple interests involved in a web application?
]]>To try to help put these numbers into perspective, this blog post is currently #1 on slashdot, #7 on reddit, the top page of del.icio.us/popular , etc; yet www.w3.org is still serving more than 650 times as many DTDs as this blog post, according to a 10-min sample of the logs I just checked.
Evidently there's software out there that makes a lot of use of the DTDs at W3C and they fetch a new copy over the Web for each use. As far as this software is concerned, these DTDs are just data files, much like the timezone database your operating system uses to convert between UTC and local times. The tz database is updated with respect to changes by various jurisdictions from time to time and the latest version is published on the Web, but your operating system doesn't go fetch it over the Web for each use. It uses a cached copy. A copy was included when your operating system was installed and your machine checks for updates once a week or so when it contacts the operating system vendor for security updates and such. So why doesn't XML software do likewise?
It's pretty easy to put together an application out of components in such a way that you don't even realize that it's fetching DTDs all the time. For example, if you use xsltproc like this...
$ xsltproc agendaData.xsl weekly-agenda.html >,out.xml
... you might not even notice that it's fetching the DTD and several related files. But with a tiny HTTP proxy, we can see the traffic. In one window, start the proxy:
$ python TinyHTTPProxy.py Any clients will be served... Serving HTTP on 0.0.0.0 port 8000 ...
And in another, run the same XSLT transformation with a proxy:
$ http_proxy=http://127.0.0.1:8000 xsltproc agendaData.xsl weekly-agenda.html
Now we can see what's going on:
connect to www.w3.org:80 localhost - - [05/Sep/2008 15:35:00] "GET http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd HTTP/1.0" - - connect to www.w3.org:80 localhost - - [05/Sep/2008 15:35:01] "GET http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent HTTP/1.0" - - bye bye connect to www.w3.org:80 localhost - - [05/Sep/2008 15:35:01] "GET http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent HTTP/1.0" - - bye connect to www.w3.org:80 localhost - - [05/Sep/2008 15:35:01] "GET http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent HTTP/1.0" - - bye
This is the default behaviour of xsltproc, but it's not the only choice:
To set up this sort of cache, first grab copies of what you need:
$ mkdir xhtml1
$ cd xhtml1/
$ wget http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
--15:29:04-- http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
=> `xhtml1-transitional.dtd'
Resolving www.w3.org... 128.30.52.53, 128.30.52.52, 128.30.52.51, ...
Connecting to www.w3.org|128.30.52.53|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32,111 (31K) [application/xml-dtd]
100%[====================================>] 32,111 170.79K/s
15:29:04 (170.65 KB/s) - `xhtml1-transitional.dtd' saved [32111/32111]
$ wget http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent
...
$ wget http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent
...
$ wget http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent
...
$ ls
xhtml1-transitional.dtd xhtml-lat1.ent xhtml-special.ent xhtml-symbol.ent
And then in a file such as xhtml-cache.xml, put a little catalog:
<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
<rewriteURI
uriStartString="http://www.w3.org/TR/xhtml1/DTD/"
rewritePrefix="./" />
</catalog>
Then point xsltproc to the catalog file and try it again:
$ export XML_CATALOG_FILES=~/xhtml1/xhtml-cache.xml $ http_proxy=http://127.0.0.1:8000 xsltproc agendaData.xsl weekly-agenda.html
This time, the proxy won't show any traffic. The data was all accessed from local copies.
While XSLT processors such as xsltproc and Xalan have no technical dependency on the XHTML DTDs, I suspect they're used with XHTML enough that shipping copies of the DTDs along with the XSLT processing software is a win all around. Or perhaps the traffic comes from the use of XSLT processors embedded in applications, and the DTDs should be shipped with those applications. Or perhaps shipping the DTDs with underlying operating systems makes more sense. I'd have to study the traffic patterns more to be sure.
p.s. I'd rather not deal with DTDs at all; newer schema technologies make them obsolete as far as I'm concerned. But
I don't particularly care for the rel="profile" design, but one should choose ones battles and I'm not inclined to choose this one. I'm content for the market to choose.
]]>I encourage you to build a copy of Firefox without content sniffing and try surfing the web. I tried this for a while, and I remember there being a lot of broken sites ...
That reminded me of an idea I heard in TAG discussions of MIME types and error recovery: a browser mode for "This is my content, show me problems rather than fixing them for me silently."
Though Adam offered a patch, building firefox is not something I have mastered yet, so I'm interested to learn about run-time configuration options in IE (notes Julian) and Opera (notes Michael). Eric Lawrence's reply points out:
Please do keep in mind, however, that most folks (even the ultra-web engaged on these lists) see but a small fraction of the web, especially considering private address space/intranets, etc.
A report from one developer suggests there's light at the end of the tunnel, at least for sniffing associated with feeds:
I did, partly as an experiment, stop sniffing text/plain in the latest release of SimplePie (which, inevitably, isn't the nicest of things to do, seeming there are tens of thousands of users). Next to nothing broke. I know for a fact this couldn't have been done a year or two ago: things have certainly moved on in terms of the MIME types feeds are served with ...
If you get a chance to try life without MIME type sniffing, please let us know how it goes.
]]>This analysis is intended to be neutral with respect to ideology, history and constituency. For a useful overview of how we got here, see WAI-ARIA Implementation Concerns (member-only link) by Michael Cooper.
The W3C's WAI PF Working Group recently published the first public working draft of the Accessible Rich Internet Applications (WAI-ARIA) specification, which "describes mappings of user interface controls and navigation to accessibility APIs".
The ARIA spec. defines roles, states and properties to manage the interface between rich web documents and assistive technologies. The primary expression of roles, states and properties in markup languages is via attributes. Since ARIA is meant to augment web applications across a range of languages and user agents, ARIA has to specify how its vocabulary of attributes and values can be integrated into both existing and future languages.
In preparing this analysis, I have reviewed the available concrete evidence bearing on the matter, and have carried out a considerable amount of work to replicate and, in some cases, correct or extend, testing which has been done in the past. The details are available in a report entitled Some test results concerning ARIA attribute syntax.
ARIA is useful only if it is widely supported. It therefore needs to integrate cleanly into existing and future languages as easily as possible. Before looking at possible answers to the spelling question, we need to consider exactly what supporting ARIA means.
We can distinguish two levels of support for ARIA on the part of user agents, which I'll call 'passive' and 'active' support. By passive support, I mean that documents with ARIA-conformant markup are not rejected by the agent, and the markup is available in the same way any other markup is, e.g. via a DOM API or for matching by CSS selectors. By 'active' support I mean the user agents actually implement their part of ARIA semantics, that is, reflecting changes to ARIA-defined states and properties via accessibility APIs.
Although already deployed implementations cannot offer active support, an optimal answer to the spelling question would maximise passive support from existing languages, as well as encouraging active support from subsequent implementations.
There are in principle three possible approachs to the spelling question:
The land-grab approach is pretty clearly unacceptable, because of clashes with existing vocabularies and the likelihood of clashes with future ones, and will not be considered further.
The current
ARIA WD specifies a combination of the colon and
dash approachs, with the colon being specified for use
with XML-based
languages, with the necessary additional requirement that 'aria' is bound to
the ARIA namespace in the usual way, i.e.
xmlns:aria="http://www.w3.org/2005/07/aaa", and the
dash approach being specified for use with non-XML languages. We'll
call this the mixed approach hereafter.
My understanding is that as of the date of this note, the WAI PF working group have indicated that their intention is that the next draft of the ARIA specs will move to the dash appropach.
Choosing an approach is made complicated by the landscape of language and infrastructure standards it has to fit in to, and by the fact that these are moving targets. We therefor have to distinguish between what is currently in place, what we have reason to expect in the near future, and what we can foresee in the longer term. Furthermore, for existing languages we have two categories: XML-based languages, with more or less explict provision for extensibility in general, typically namespace-based, and non-XML languages, which for the purposes of this analysis we will take to be HTML 4.01 and nothing else.
As noted above, the best we can expect from deployed user agents is passive support. The table below sets out the extent of passive support which is available for the colon and dash approaches for each of three host languages, which exemplify the major relevant categories: HTML 4.01 (for the non-XML languages), XHTML (an XML language, but not always treated as such, so we actually get two columns for it below) and SVG (only an XML language).
Notes:
It should be noted that some of the entries above disagree with assertions made in the past about browser behaviour. At least some of those assertions were based on flawed test materials---see the discussion of experiments 1 and 2 in my testing report for details on the information summarised above.
A number of browser implementors have responded positively to the ARIA initiative and have included experimental active support for ARIA in pre-release versions of their products. Most of the test materials and implementation information I can find suggests that only the dash approach, and only HTML or XHTML, are currently being implemented.
With regard to improving passive support, it seems very possible that
IE8 will support attribute selectors of the form [aaa\:checked],
which would remove the qualification recorded in the table above by footnote 4.
The situation with respect to HTML5 is complicated. As it currently stands, the HTML5 draft specification supports namespaces internally, and all HTML elements are parsed into the DOM nodes in the HTML namespace, regardless of whether they are parsed "as HTML" or "as XML". But when parsing documents "as HTML", no other namespaces are recognised. Unless this changes before HTML5 is completed, the HTML/"XHTML (as if HTML)" columns above will apply to HTML5-conformant user agents in at least some cases.
On the basis of the above survey, there follows below an attempt at a
cost-benefit analysis with respect to the colon and
dash approaches, as well as the mixed
approach as currently specced in the ARIA working draft and a fourth approach, as proposed by me in
a
message to www-tag, which I'll call the xcolon approach.
The xcolon approach attempts to address some of the problems
revealed in the passive support table by defining a
pair of getter/setter Javascript functions for access to ARIA information in the
DOM, and giving a design pattern for duplicated CSS selectors (one using
[aria\:xxx] and the other [aria|xxx]).
aria prefix
and may not (i.e. in some browsers) correctly set the namespaceURI
property even when targetting an XML DOM.[aria\:...] selectorsFor wholly commendable reasons, development of the ARIA spec. and pilot implementation work have proceeded in parallel. Most if not all existing implementations support only the dash approach. What is the likely cost for those implementations of any decision to adopt any other approach? My conclusion, having examined one implementation in some detail, is that the cost is likely to be very modest.
Michael Cooper, WAI PF staff contact, captured the reason for this very well, albeit unintentionally:
"The ARIA roles and properties are conceptually simple enough, but they are designed to provide a bridge between HTML and desktop accessibility APIs, a bridge which is exploited by the operating system, user agent, and assistive technology all working together. There's a complex set of interdependencies there and the feasibility and details of many of the ARIA features could only be worked out by testing in deployed systems, and therefore doing early implementation."
The complexity referred to above is fundamentally one of architecture, both static and dynamic. Not surprisingly, therefore, syntactic concerns account for a tiny fraction of the code needed to implement ARIA as it stands. Furthermore, and again not surprisingly, as it's what sound software engineering practice requires, the details of the concrete syntax are isolated, and the vast bulk of the code I looked at refers to it only indirectly. The consequence of all this is that the changes necessary to manage any change away from the dash approach will be very straightforward. For more details, see the discussion of experiment 3 in my testing report.
Many existing XML languages make explicit, generic, provision for extensibility by including in their formal schemas and/or spec. prose allowance for any namespace-qualified elements and attributes from namespaces other than those which make up the language itself. Tools such as NVDL and, to a lesser extent, W3C XML Schema and RelaxNG, make it possible to combine the schemas for multiple XML languages to give a complete characterisation of mixed-language documents.
One particularly important example of this approach is SVG. ARIA integration into SVG is clean and straightforward under the colon or mixed approaches, but will require amending the spec. under the dash approach.
In trying to weigh the tradeoffs which must of necessity be considered when confronted by the information given above, the matter of timescale is particular hard to address. Any assertion about how things will look five, or even two, years hence can always be countered with a contrary assertion. None-the-less, the centrality of the HTML languages for the Web, and the fundamental importance of accessibility for all of us, suggest that we must take the long-term impact of this decision seriously, and be prepared to discount some short-term discomfort in return for long-term stability and simplicity.
The general scope of the proposed Video on the Web activity is to provide cohesion in the video related activities of W3C, as well helping other W3C Groups in their effort to provide video functionalities. In addition, this activity will focus at implementing the next steps from the W3C workshop on Video on the Web. The proposal is to create 3 new Working Groups around Video on the Web. Please, have a look at the following documents:
We welcome general feedback, general expressions of interest (or lack of!) and comments on the discussion list public-video-comments@w3.org.
If you should have questions or need further information, please feel free to contact me as well. I will be presenting the activity proposal during the Web Conference next week, on Thursday afternoon.
]]>The modularity of HTML itself has been discussed recently, for example by Ian Hickson, co-Editor of HTML5:
Note that it really isn't that easy. For example, the HTML parsing rules are deeply integrated with the handling of <script> elements, due to document.write(), and also are deeply integrated with the definition of innerHTML. Scripting, in turn, is deeply related to the concept of scripting contexts, which depends directly on the definition of the Window object and browsing contexts, which, in turn, are deeply linked with session history and the History object (which depends on the Location object) and with arbitrary browsing context navigation (which is related to hyperlinks and image maps) and its related algorithms (namely content sniffing and encoding detection, which, to complete the circle, is part of the HTML parsing algorithm). - Brainstorming test cases, issues and goals, etc., Ian Hickson
and in reply by Laurens Holst:
I don't know the spec well enough to answer that question, but I'd say modularization (if I may call it so) would make it both easier to grasp as individual chunks, for both the reviewing process and the implementing process. brainstorming: test cases, issues, goals, etc.. - Laurens Holst
The <canvas> element introduces a complex 2D drawing API different in nature from the other interfaces, which concentrate on setting and retrieving values in the markup itself; the client-side database storage section of the specification is another such interface. While the <canvas> element has a place in the specification, the drawing API should be defined in a separate document. Hixie expressed a similar sentiment (and see the group's issues about scope):
The actual 2D graphics context APIs probably should be split out on the long term, like many other parts of the spec. On the short term, if anyone actually is willing to edit this as a separate spec, there are much higher priority items that need splitting out and editing...
It would also be nice if the <canvas> element and the SVG elements which embed in HTML did so in just the same way, in terms of the context (style, etc.) which is passed (or not passed) across the interface, in terms of the things an implementer has to learn about, and things which users have to learn about. So that <canvas> and SVG can be perhaps extended to include say 3D virtual reality later, and so that all of these can be plugged into other languages just as they are plugged into HTML.
There are lots of reasons for modularity. The basic one is that one module can evolve or be replaced without affecting the others. If the interfaces are clean, and there are no side effects, then a developer can redesign a module without having to deeply understand the neighboring modules.
It is the independence of the technology which is important. This doesn't, of course, have to directly align with the boundaries of documents, but equally obviously it makes sense to have the different technologies in different documents so that they can be reviewed, edited, and implemented by different people.
The web architecture should not be seen as a finished product, not as the final application. We must design for new applications to be built on top of it. There will be more modules to come, which we cannot imagine now. The Internet transport layer folks might regard the Web as an application of the Net, as it is, but always the Web design should be to make a continuing series of platforms each based on the last. This works well when each layer provides a simple interface to the next. The IP is simple, and so TCP can be powerfully built on top of it. The TCP layer has a simple byte stream interface, and so powerful; protocols like HTTP can be built on top of it. The HTTP layer provides, basically, a simple mapping of URIs to representations: data and the metadata you need to interpret it. That mapping, which is the core of Web architecture, provides a simple interface on top of which a variety of systems -- hypertext, data, scripting and so on -- can be built.
So we should always be looking to make a clean system with an interface ready to be used by a system which hasn't yet been invented. We should expect there to be many developers to come who will want to use the platform without looking under the hood. Clean interfaces give you invariants, which developers use as foundations of the next layer. Messy interfaces introduce complexity which we may later regret.
Let us try, as we make new technology, or plan a path for old technology, always to keep things as clean as we can.
]]>Good practice: Version information
A data format specification SHOULD provide for version information.
So, it's always a good idea when you design a language or data format to provide a way for instance documents to include something like a version attribute, probably near the beginning of the document, to indicate what version of the language is being used.
Or is it always a good idea?
What does a version identifier convey?
In fact, do we even agree on what it means to put something like a language version marker on a document? Let's imagine a simple XML language designed for setting down recipes. In the first version of the language, the markup looks like this:
<recipe name="Tuna Salad" recipeLanguageVersion="1.0">
<ingredients>
<ingredient name="Tuna Fish" amount="1 can"/>
<ingredient name="Mayonnaise" amount="3 tablespoons"/>
<ingredient name="Capers" amount="a few"/>
</ingredients>
<steps>
<step>Open can</step>
<step>Drain liquid from can. Put fish in bowl.</step>
<step>Add mayonnaise. Stir well.</step>
<step>Add capers. Stir gently.</step>
</steps>
</recipe>
The allowed markup in version 1.0 of the recipe is just what's shown above: an outer <recipe> containing <ingredients> and <steps>, etc. Eventually it's decided that it would be useful to provide optional pictures for ingredients or steps. So in version 2 of the language we can do things like:
<recipe name="Tuna Salad" recipeLanguageVersion="2.0">
<ingredients>
<ingredient name="Tuna Fish" amount="1 can"
picture="./CanPicture.jpg"/>
<ingredient name="Mayonnaise" amount="3 tablespoons"/>
<ingredient name="Capers" amount="a few"/>
</ingredients>
<steps>
<step>Open can</step>
<step picture="./DrainCanPicture.jpg">
Drain liquid from can. Put fish in bowl.</step>
<step>Add mayonnaise. Stir well.</step>
<step>Add capers. Stir gently.</step>
</steps>
</recipe>
Question: let's imagine that version 2 of the language, the one that supports the optional pictures, has been out for awhile, but I still want to write a simple recipe with no pictures:
<recipe name="ice cubes" recipeLanguageVersion="??">
<ingredients>
<ingredient name="water" amount="1.5 cups"
</ingredients>
<steps>
<step>Put water into ice cube tray.</step>
<step>Freeze.</step>
</steps>
</recipe>
What's the best value to put in the version attribute? I know that version 2.0 is the latest version of the recipe language. In fact, that's the only version of the specification I have next to me, so maybe I should use that?
There's a problem, though. That version="2.0" marker might not work with software that's written to version 1.0, and in fact, my document would otherwise be a fine 1.0 recipe document.
So, maybe I should label it 1.0? Unfortunately, that's a bit hard for me. I don't want to have to go through the specifications for every version of the recipe language that's ever existed just to find the oldest that works. I really don't want to do that if the language has been revised a lot! Also, these sample recipes are small, but if I were using software to write very long documents, then that software would either have to keep track of the latest features used, or else search the entire document before writing it to a file, in order to get that version identifier at the front.
Indeed, just these complexities have proven troublesome for the deployment of languages like XML 1.1. XML 1.1 is similar to XML 1.0, but it enables the use of some new Unicode characters (just as recipe language V2 allows for use of new image tags.) The XML 1.1 Recommendation suggests that:
XML Programs which generate XML SHOULD generate XML 1.0, unless one of the specific features of XML 1.1 is required.
In fact, it has often proven difficult to write software that generates documents labeled as XML 1.1 only when necessary: it's much easier for XML 1.1-compatible software to label all output as <xml version="1.1">, resulting in documents that are unusable with widely deployed XML 1.0 software.
Perhaps for reasons like this, adoption of XML 1.1 has been slow.
Returning to the recipe example, maybe the version attribute should take a list of versions, and I should put in both 1.0 and 2.0? That could be helpful to consuming software, but it still means that I (or my software) must be familiar with all the previous versions of the specification.
So, we need to ask, is the version identifier used to convey:
The best answer is probably different depending on the language, how often it's revised, whether revisions tend to maintain backwards compatibility, etc.
Is having some sort of version identifier always a good idea?
That Good Practice Note quoted above says "provide a version indicator", but we've just shown that we're not always quite clear on what that would do anyway. Is it still good advice to suggest that surely each language should provide for something in the instance? If so, should its use be required or optional?
As shown above, it's common for the same instance document to be legal in many versions of a language. As long as such documents are likely to have the same or sufficiently compatible meanings per the different versions, then it may be better to omit any indication of version in the instance, and leave it to the receiving software to decide whether the document can be processed. After all, with the second recipe above, the receiver will soon enough discover that it can or can't process picture attributes, and if not, it either will or won't know that they can be safely ignored. Version attributes can be helpful in giving early warning of incompatibilities, or as a crosscheck for catching errors, but they're usually not essential to correct operation.
One important exception is in the case where the language is likely to change in incompatible ways. If the same document means different things in different versions of a language, then it's very important to indicate which version the author had in mind when creating the document. Putting that version indicator into the document itself is one good way to do it. So maybe the right advice is:
Proposal for Future Architecture Document: Version information
If a language or data format will change in incompatible ways, then indicate the language version used for each instance.
Are namespaces a good way to identify language versions?
If version identifiers aren't always a good bet, what about namespaces? Many modern languages allow the creation of globally unique names, identifiers, tags, etc. In XML this is done through use of Namespaces. In RDF, it's done by using URIs as identifiers, etc..
Sometimes it's appropriate to use new identifiers for each version of a language, and mechanisms like namespaces can make that easier:
<r:step xmlns:r="http://example.org/recipeLanguage1">
vs.
<r2:step picture="./food.jpg"
xmlns:r2="http://example.org/recipeLanguage2">
In this example, the element with expanded name {http://example.org/recipeLanguage2, step} allows a picture attribute, but {http://example.org/recipeLanguage1, step} does not.
A full discussions of the pros and cons of using namespaces this way is beyond the scope of this note. One important advantage of using namespaces is that they can be easily applied not just to the root element for the language as a whole, but to mixtures of compound document markup, in which each sublanguage evolves with its own namespaces. Also, because namespace names are URIs, you can use the Web itself to get information about them.
Namespaces do have drawbacks. Imagine if there were 50 different namespaces for a language just because 50 separate bugs had been fixed in different errata. Would you republish all the markup in 50 namespaces? Would each document have lots of namespaces, with each element named with the last namespace in which it had been revised? Namespaces can be very useful for designating language versions, but there's no one idiom that's right for all languages. We note that most widely deployed tag-based languages for the Web (HTML, XML Schema, XSLT) have chosen either to use the same namespace(s) across multiple versions, or in the case of some flavors of HTML, not to use namespaces at all.
Conclusions
So, the TAG is having second thoughts about the suggestion that all data formats SHOULD provide for version identification. Sometimes it's a good thing to do, but sometimes not. Perhaps the right advice will be what's proposed in the revised Good Practice Note above. In any case, the TAG has been working for several years on a finding that will explore in detail many issues relating to versioning, and version attributes are likely to be among the topics covered. In the meantime, we thought we'd take the opportunity to signal that we're not so sure that the advice in the Architecture Document is as good as we thought.
By the way, TAG member David Orchard has covered some of the same topics as well as many others relating to versioning in his personal blog. Links to a few of his postings follow my signature below. Dave is also the principle author of the TAG's draft finding on versioning. Working drafts covering Terminology, Strategies, and Versioning of XML Languages are available for review. New drafts come out every few months, and we're hoping to have something more or less complete, well, real soon now.
Noah Mendelsohn
Note: unless otherwise indicated, opinions expressed in the TAG's blog are those of the individual authors, and do not necessarily represent consensus of the TAG as a whole.
Links to Dave Orchard's Blog Postings on Versioning
On Saturday, I took an action in the HTML Working Group discussion of ARIA to write it up. Little did I know that by the time I got back to the office, Norm would have written up Implicit Namespaces for me.
Thanks, Norm!