W3C

Repurposing the Hash Sign for the New Web

W3C Working Draft: Putative TAG Finding 31 December 2010

This version:
http://www.w3.org/2001/tag/2010/12/HashInURI-20101231
Latest version:
http://www.w3.org/2001/tag/2010/12/HashInURI-20101231
Previous versions:
http://www.w3.org/2001/tag/2010/11/HashInURI

http://www.w3.org/TR/2009/WD-hash-in-uri-20090415/
Editors:
T.V. Raman, Google raman@google.com
Ashok Malhotra, Oracle ashok.malhotra@oracle.com

Abstract

The Hash sign (#) in a URI was originally used to introduce a static "fragment identifier", but recently it is being used in many more complex ways as it is set by and interpreted by JavaScript in Web applications. Fragment identifiers are used to provide several different kinds of parameters to the client-side application, such as the actual URI of a video to be played to a video player, or the position and zoom to a map. Unlike search parameters preceded by "?", the characters in the URI bar after the hash can be changed without causing the page to be reloaded. Applications and toolkits using fragment identifiers in this way often go to some effort to keep a history and make sure the back button works as expected. Accessibility and search can, however, be compromised because without running JavaScript, the URI has no meaning. Such uses of the "fragment identifier" have interesting and different properties, and differs from the way it is currently described in specs.

This document explores the issues that arise from these new uses of fragment identifiers, and attempts to define best practices.

Status of this Document

This document has been developed for discussion by the W3C Technical Architecture Group and is being published as a Public Working Draft in order to get additional input from the Web community. This version, dated 31 December, 2010 is a follow-up to the previous version dated November 30, 2010.

Publication of this draft finding does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time.

Please send comments on this document to the publicly archived TAG mailing list www-tag@w3.org (archive ).

Table of Contents

1 Introduction
2 Use Case Scenarios
    2.1 Addressing Into Multimedia Streams
        2.1.1 Things to Note
        2.1.2 Extrapolating From This Pattern
        2.1.3 Architectural Questions
    2.2 Interaction State and Browser History
    2.3 AJAX Libraries And State Management
    2.4 Web Command Lines
    2.5 The Naked Hash-Ref
3 Recommended Best Practices
4 These patterns violate RFC 3986
5 Affected Communities To Liaise With
    5.1 WHATWG:PushState()
6 Conclusions
7 References


1 Introduction

[RFC 3986] defines the character string following the ? sign in a URI as the "query component". The character string follwing the # sign is known as the "fragment identifier" and used to address specific locations in a document. Nearly 20 years later, the Web has built a strong set of conventions around how URI parameters are used. As transactional applications began moving on to the Web in the late 1990's, query parameters formed a core building block for how application state was communicated between client and server. In this phase of Web evolution, clients were still comparatively simple, and client-side URI parameters did not move beyond the use of fragment identifiers. But with Web 2.0 applications increasingly moving traditional client-side applications to the Web and highly interactive applications being built using Web parts (HTML, CSS and JavaScript component resources that are themselves Web addressible, see [tvr-cacm2009]), fragment identifiers are being used in a variety of ways quite different from their originally intended use. The Web is beginning to discover and codify design patterns based on fragment identifiers for many of these use cases.

This document explores the issues that arise in this context, and attempts to define best practices that help:

In this document we use the term "URI" to indicate both URLs and URIs.

2 Use Case Scenarios

This section discusses several usage scenarios that make innovative use of fragment identifiers as URI parameters on the Web.

2.1 Addressing Into Multimedia Streams

When publishing multimedia streams, there is often a need to address into specific points in the multimedia stream, e.g., by using a time-index. The simplest means of doing this is to pass in the start-time as a query parameter in the URI to the server, e.g. http://www.example.com/media.stream?start=03:06:09 and have the server start streaming the content starting at 3 hours, 6 minutes and 9 seconds into the content. This has the additional side-benefit of creating distinct URIs for each point in the media stream and such URIs can be used to bookmark locations of interest.

It is also possible to leverage client-side parameters encoded as part of the URI (using a # ), where this pseudo fragment identifier is used by client-side scripts as an argument to be passed to an appropriate locator function. Consider the following example taken from cnn.com:

 <a href="http://www.cnn.com/video/#/video/tech/2008/02/19/vo.aus.sea.spider.ap">
          Giant sea spider filmed deep underwater
 </a>

CNN uses links like the above for all the topical video segments that are published on its site. The URI in this case has the following components:

Component Value
Protocol http
Host www.cnn.com
Path video
Client Param #/video/tech/2008/02/19/vo.aus.sea.spider.ap

2.1.1 Things to Note

The browser is expected to do a GET of the URI leading up to the # sign, and the processing application, in this case, the JavaScript embedded in the HTML Response processes the portion of the URI following the # . Note that in the general case, the JavaScript function that eventually processes the client parameter may not have been present in the original HTTP Response. It may come from a JavaScript library that was loaded as the result of a subsequent HTTP GET request or as a result of a script in the text/html response.

The fragment identifier in this pattern is intentionally refrerred to as a client parameter. Treating it as a regular fragment identifier in this usage would result in one incorrectly inferring that the URI for the video resource being addressed is http://www.cnn.com/video. This would result in all the video links on the CNN site getting the same URI. Thus, the entire URI in this case is http://www.cnn.com/video/#/video/tech/2008/02/19/vo.aus.sea.spider.ap. A consumer of this URI who goes looking for an id within the Response that matches the #-suffix of this URI will fail. The reported Content-Type for the resource is text/html. However, the behavior of the #-suffix in this case is not defined by the HTML specification.

As used, the #-suffix is a first-class client parameter in that it gets consumed by a script that is served as part of the HTML document returned by the server upon receiving a GET request. This embedded script examines the URI available to it as script variable content.location, strips off the # and uses the rest of the prefix as an argument to function that generates the actual URI. Having constructed this content URI, the script then proceeds to instruct the browser to play the media at the newly constructed location.

The content URI is constructed purely for processing by the JavaScript code. Thus, there is no conflict if the content URI contains a fragment identifier that is the same as the fragment identifier in the original URI. Errors in processing the content URI would be handled by the JavaScript code.

2.1.2 Extrapolating From This Pattern

The CNN example cited above is not unique with respect to its use of # within the URI for encoding parameters to the receiving application. It shows that in a world of dynamic documents, the traditional fragment identifier need no longer be an idref value that addresses an existing node in the serialized HTML making up the HTTP Response. In addition to possibly being a static idref, the fragment identifier in the URI, the pattern demonstrated here generalizes to the following:

  • An idref to a dynamically generated node.
  • A parameter to be consumed by the application that is delivered as the HTTP Response to the original GET request.

2.1.3 Architectural Questions

This section discusses some of the architectural questions raised by this design pattern:

  • What happens if the receiving client does not implement JavaScript, or has had scripting turned off? The behavior of a user-agent that does not execute the embedded JavaScript is different, given this URI, from one that does. Notice further that the HTTP Response headers do not give the client any indication that this is likely to be so. Until now, URIs have been equally useful to browsers and non-browser consumers. This pattern demonstrates a case where the URI inferred by browsers as opposed to non-browsers is different. A non-browser that receives a URI as in the above, and sees a Content-Type of text/html might assume (incorrectly) that the URI for this video resource is http://www.cnn.com/video.html.
  • A related fragment id meaning arises when one considers content-negotiation. For instance:
    1. get application/rdf+xml "http://example.com/exp/#something"
    2. get text/html "http://example.com/exp/#something"

See also 4 These patterns violate RFC 3986.

2.2 Interaction State and Browser History

A variety of methods are available in Web Architecture to save application state. Cookies store information on the client-side that is sent along with the GET request. Similarly, data can be stored on the server-side -- in a database, for example, identified by a cookie -- and can be used to change the details of the GET request. There are also specifications under development (See [Web Storage]) that extend the cookie mechanism in several directions. These specifications allow large amounts of data to be stored on the client and can also be used to encode application state.

These mechanisms, however, encode private applications states. In some cases, an application may want to allow selected states to be made public and shareable. For this we require a URI, appropriately decorated with client-side and server-side parameters. The challenge in designing a mechanism to encode state is to preserve the familiar user experience especially to make the back button do the right thing. For live examples of this design pattern, see GMail and Google Maps both of which take extreme care to ensure that the user's expectations of Web interaction are preserved. These applications use iframe proxies to achieve the desired effect.

A very early interactive Web application was the [Xerox Parc Map Viewer]. When you bring up the application it shows you, by default, a map of the world. If you select a spot on the map it changes to show you a map centered on the selected spot. Users can interact with the map in various ways: pan, zoom, select degree of detail, etc. Each interaction is encoded as a parameter in a URL which is sent back to the server who generates a new map and refreshes the page. The format of the URL is:

http://hostname/map/option=value/.../option=value

This application was created before JavaScript or AJAX Toolkits. Essentially, it creates maps as documents, one for each URL and retains the Web paradigm of displaying a document for each URL.

Google Maps, though many times more functional than the simple Map Viewer attempts to retain the same paradigm. Maps are displayed as documents and each interaction with a map or request for more information, such as nearby hotels, generates and displays a different map/document.

If you work with Google Maps you will notice that even after you have customized the map to show you what you want, the address bar has not changed - it still says http://maps.google.com/. If you want a link to the displayed map, you click the "Link" button on the right and it gives you a URI to the map displayed. For example:

http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=212+Hessian+Hills+Rd,+Croton-on-Hudson,+NY+10520&sll=37.0625,-95.677068&sspn=32.527387,51.679688&ie=UTF8&hq=&hnear=212+Hessian+Hills+Rd,+Croton-on-Hudson,+Westchester,+New+York+10520&z=16

Notice the structure of this URI: it includes the address as well as other parameters. The URI that Google Maps creates for the displayed map has a long query parameter but no fragment identifier. In fact, what happens is that some JacaScript is downloaded for each new display and this JavaScript fetches the appropriate information from the server and performs some client-side manipulation to display the requested map. The Web paradigm is preseved: maps are displayed as documents, the back button works, each map has a URI that can be transmitted in email or an instant message and used to regenerate the map.

GMail, on the other hand, uses fragment identifiers to encode the state. For example, https://mail.google.com/mail/?shva=1#inbox/12c7e6abbc328af4 identifies the inbox and a specific piece of mail in the inbox. If a piece of mail is not selected, the fragment identifier merely identifies the inbox: https://mail.google.com/mail/?shva=1#inbox

These mechanisms behind this are discussed below in more detail.

2.3 AJAX Libraries And State Management

AJAX applications use features of Dynamic HTML (DHTML) to create highly reactive user experiences. Updates to the Web user interface in response to user actions no longer require a full page reload. Consequently, the user can perform a sequence of interaction steps while remaining on the same page at least as seen from the browser's perspective of content.location. This makes for a good user experience, except for the following:

  • Recording key points in the interaction flow, e.g., for bookmarking.
  • Providing intuitive behavior for the browser's history mechanism.
  • Snapshotting interaction state to enable returning to a partially completed task at a later time.

Today, many of the details of AJAX programming have been abstracted away by higher level toolkits such as [Dojo AJAX Toolkit] and [google-gwt]. Management of interaction state and browser history is one of the key affordances implemented in these libraries. History mechanisms in AJAX libraries like GWT and Dojo share a lot in common, and the approach can be traced back to Really Simple History (RSH). In addition, the mechanism described here has also been adopted by a recent update to GMail.

The basic premise is to keep track of the application's internal state in the URI fragment identifier. This works because updating the fragment doesn't typically cause the page to be reloaded. This approach has several benefits:

  • It is about the only way to control the browser's history reliably.
  • It provides good feedback to the user.
  • It is bookmarkable i.e., the user can create a bookmark to the current state and save it, email it, or whatever.

One of the techniques that is used to provide this functionality is to open a number of frames within a browser window. In such an architecture, parent and child frames are allowed to change each others' location URI as long as the frames display information from the same domain or have agreed to collaborate by some other means. Otherwise, changing a frame's location URI opens up a cross-site scripting hole. If the frames can collaborate, then one of the frames, say the parent, passes data to the child via a fragment identifier by reseting the child's location URI. Thus, given a parent frame P and a child frame C , where the location URIs U_P and U_C may come from different domains, the parent frame might pass data to the child by resetting its location URI to U_C#data; the child picks up this data by polling for changes in its location URI. This technique is used in Comet Programming. As an example, the [Dojo AJAX Toolkit] uses an IFrame proxy to enable cross-domain XML HTTP Requests. this is a useful technique when writing cross-site mashups. As an example, see XKCD and AxsJAX a cross-site mashup that mashes together XKCD comics with their associated transcripts to create a speech-friendly XKCD experience.

2.4 Web Command Lines

When applications are built from Web parts, there is often a need to configure them when the application is launched. Traditional applications would call these default start-up or command-line options. We see the equivalent emerging for configuring desktop gadgets and widgets where command-line options are passed in via URI parameters in this context, the URI is the Web command-line. For one sample implementation and its associated usage, see Using URIs To Pass Parameters To The Web. Dave Raggett's HTMLSlidy uses URIs of the form ...#(nn) to address into a deck of slides.

2.5 The Naked Hash-Ref

In some situations a single # sign is used as the value of the href attribute on HTML anchors. This can be thought of as a relative URI with a null fragment identifier. Web sites wishing to override the default-target behavior of anchors use this when attaching a JavaScript event-handler to anchor elements for mouse-clicks. The only justification for using a naked # as the value of the href attribute appears to be to avoid anything showing up on the browser status bar as the user activates the link. Note that this idiom also creates significant hurdles for non-mouse users of the Web.

3 Recommended Best Practices

Applications that choose to make some states public and shareable should use fragment identifiers to identify such staes.

4 These patterns violate RFC 3986

Not surprisingly, some of these new uses of fragment identifiers violate [RFC 3986]. Here is what [RFC 3986] says: "The fragment identifier component of a URI allows indirect identification of a secondary resource by reference to a primary resource and additional identifying information. The identified secondary resource may be some portion or subset of the primary resource, some view on representations of the primary resource, or some other resource defined or described by those representations. ... The semantics of a fragment identifier are defined by the set of representations that might result from a retrieval action on the primary resource. The fragment's format and resolution is therefore dependent on the media type [RFC2046] of a potentially retrieved representation ..."

Highly intercative applications built using JavaScript and Ajax libraries use fragment identifiers in ways that were not envisaged when [RFC 3986] was written. Should we condemn such applications as non-conformant? Or do we need to revise [RFC 3986] to take into account new usage patterns and emerging functionality? And should there be different standards for browsers that support/run JavaScript and those that don't? All these are good questions that need to be addressed.

5 Affected Communities To Liaise With

We will need to liaise effectively with standards groups that are active in defining the formats and protocols that come together in turning an HTTP Response into an interactive user interface for a Web application. This section will be used to track these dependencies, and may be removed upon final publication of this document.

The WhatWG that presently defines the behavior of conforming HTML5 Web browsers in conjunction with the W3C HTMLWG. There is also ongoing HTTP work in the IETF.

5.1 WHATWG:PushState()

There is ongoing discussion regarding encoding client-side state in the WHATWG by using pushState() which allows for changing the whole URI using ECMAScript so that the URI exposed to copy-and-paste can still make sense in contexts without scripting. It also addresses the back button concern.See pushState().

6 Conclusions

As the Web has evolved from showing things to doing things the fragment identifier has been repurposed to indicate a broad range of functionality. This document discusses some of these uses. We would be grateful for comments and feedback showing other uses of the fragment identifier.

7 References

www-tag archive
Mail thread on WWW-TAG from 2007 that initiated some of these discussions. (See http://lists.w3.org/Archives/Public/www-tag/2007Jul/0148.html.).)
JSON with Padding
JSON with Padding (See http://ajaxian.com/archives/jsonp-json-with-padding..)
Wikipedia Comet
Comet Programming from Wikipedia (See http://en.wikipedia.org/wiki/Comet_(programming)..)
sidewinder-hash
Using URLs To Pass Parameters To The Web , Mark Birbeck. (See http://internet-apps.blogspot.com/2007/11/using-urls-to-pass-parameters-to-web.html.)
google-gwt
Google Web Toolkit -- Java software development framework that makes writing AJAX applications like Google Maps and GMail easy for developers taking care of browser and platform details. (See http://code.google.com/webtoolkit/.)
tvr-cacm2009
Toward 2^W Beyond Web-2.0, Communications Of The ACM, ACM, New York., T.V. Raman. (See http://portal.acm.org/citation.cfm?id=1461945.)
Dojo AJAX Toolkit
The Javascript Toolkit by the Dojo Foundation. (See http://dojotoolkit.org/.)
Web Storage
Web Storage (See http://dev.w3.org/html5/webstorage/.)
RFC 3986
Uniform Resource Identifier (URI): Generic Syntax T. Berners-Lee, R. Fielding, L. Masinter. (See http://www.ietf.org/rfc/rfc3986.txt.)
Xerox Parc Map Viewer
Uniform Resource Identifier (URI): Generic Syntax An Interactive Map Viewer (See http://www2.parc.com/istl/projects/www94/mapviewer.html.)