Publishing and Linking on the Web

2.1 Background

Many content publishers seek to control the use of their content on the Web. In some cases, they employ means that do not take into account the Web's true architecture, and do not use the technical mechanisms available to them. A few illustrative examples are provided below.

Licenses that describe how material may be copied and altered by others tend not to distinguish between a proxy compressing a web page to make it load faster and someone editing and republishing the page on their own website. To illustrate, the Creative Commons Attribution-NoDerivs defines the terms (emphasis added):

Adaptation

means a work based upon the Work, or upon the Work and other pre-existing works, such as a translation, adaptation, derivative work, arrangement of music or other alterations of a literary or artistic work, or phonogram or performance and includes cinematographic adaptations or any other form in which the Work may be recast, transformed, or adapted including in any form recognizably derived from the original, except that a work that constitutes a Collection will not be considered an Adaptation for the purpose of this License. For the avoidance of doubt, where the Work is a musical work, performance or phonogram, the synchronization of the Work in timed-relation with a moving image ("synching") will be considered an Adaptation for the purpose of this License.

Distribute

means to make available to the public the original and copies of the Work through sale or other transfer of ownership.

Reproduce

means to make copies of the Work by any means including without limitation by sound or visual recordings and the right of fixation and reproducing fixations of the Work, including storage of a protected performance or phonogram in digital form or other electronic medium.

Consider, now, the following questions:

• Does using the automatic Google Translate service (which can be invoked automatically within Chrome) count as an "adaptation" of a Work, given that it's a translation, or is it only non-automated translations that would count?
• When a document is made available on a website, the original owner still retains their copy, it is not transferred, so does this count as "distribution"?
• If someone accesses a document released under this license on the web, it has to be copied onto their machine in order to be displayed; does this mean anyone accessing the document "reproduces" it?

Terms and Conditions statements on websites also list acceptable and unacceptable behavior on a site, with any browsing on the site implicitly indicating acceptance of the terms. These generally do not take into account the behavior of proxies. For instance, one standard set of Terms and Conditions includes:

You may view, download for caching purposes only, and print pages from the website for your own personal use, subject to the restrictions set out below and elsewhere in these terms of use.

You must not:

(a) republish material from this website (including republication on another website);

(b) sell, rent or sub-license material from the website;

(c) show any material from the website in public;

(d) reproduce, duplicate, copy or otherwise exploit material on our website for a commercial purpose;

(e) edit or otherwise modify any material on the website; or

(f) redistribute material from this website except for content specifically and expressly made available for redistribution (such as our newsletter)

It is not possible to view material on the web without it being downloaded onto your computer, so forbidding downloading except for caching purposes essentially means that people cannot view the page. In addition, many proxies automatically transform the documents that pass through them, for example to compress them so that they take up less bandwidth for mobile consumption or to introduce advertisments into pages that are accessed free of charge. Should this be prohibited?

Limits placed on the use of a website often include limitations on automatic indexing of the website, without exceptions for search engines that make the website discoverable or archives that ensure its longevity. For example, the set of terms and conditions quoted above goes on to say:

You must not conduct any systematic or automated data collection activities (including without limitation scraping, data mining, data extraction and data harvesting) on or in relation to our website without our express written consent.

Search engines rely on systematic data collection from websites in order to provide users with accurate search results, and archives do so in order to retain websites for posterity. So, these terms and conditions, if adhered to strictly, put the website out of the reach of search engines and hence makes it undiscoverable; surely this is not in the best interest of the website. Another problem is that automated agents (webcrawlers, spiders, robots) that gather information from the web are unable to read these terms and conditions; the only things they understand are the technical signals that a website provides about what is permitted. See more on this below.

As another example, the terms and conditions for gsig.com include:

Use of Materials: Upon your agreement to the Terms, GSI grants you the right to view the site and to download materials from this site for your personal, non-commercial use. You are not authorized to use the materials for any other purpose. If you do download or otherwise reproduce the materials from this Site, you must reproduce all of GSI?s proprietary markings, such as copyright and trademark notices, in the same form and manner as the original.

...

You may not use any "deep-link", "page-scrape"?, "robot", "spider" or any other automatic device, program, algorithm or methodology or any similar or equivalent manual process to access, acquire, copy or monitor any portion of the Site or any of its content, or in any way reproduce or circumvent the navigational structure or presentation of the Site.

It would be simpler and more effective for the site to use technical means for controlling what webcrawlers, spiders or robots access on the site, namely a robots.txt file which is a set of machine processable instructions instructing automated web agents what thay can and cannot do. They could also exempt automated web agents from the Terms and Conditions as discussed below.

Many sites have a linking policy that limits what links can be made to the site from other sites. These conditions are not backed up through relatively simple technical mechanisms that would prevent such links from being made. For example, the website at quotec.co.uk has a linking policy that includes:

Links pointing to this website should not be misleading.

Appropriate link text should be always be used.

From time to time we may update the URL structure of our website, and unless we agree in writing otherwise, all links should point to http://www.quotec.co.uk.

You must not use our logo to link to this website (or otherwise) without our express written permission.

You must not link to this website using any inline linking technique.

You must not frame the content of this website or use any similar technology in relation to the content of this website.

Technically it is straightforward to prevent linking to pages that the website does not want others to link to: you simply do not give these pages URLs or make the URLs undiscoverable. This is likely to be more effective than asking people to read and adhere to the Terms and Conditions. Several techniques for controlling linking and inclusion are discussed in section 6. Techniques.

Legislation that governs the possession and distribution of unlawful material (such as child pornography, information that is under copyright or material that is legally suppressed through a gag order) needs to exempt certain types of services, such as caching or hosting, as it would be impractical for such services to police all the material that passes through their servers. This does not, however, always happen and intermediaries are often held accountable for material that did not originate with them and have no control over. An example of good legislation that does exempt intermediaries the UK is the Coroners and Justice Act 2009 Schedule 13; from the Explanatory Notes (emphasis added):

Paragraphs 3 to 5 of [Schedule 13] provide exemptions for internet service providers from the offence of possession of prohibited images of children in limited circumstances, such as where they are acting as mere conduits for such material or are storing it as caches or hosts.

3.1 Actors and Agents

Archiving Server

A service that stores copies of web content, e.g. to provide an ongoing historical record.

Browser

A user agent that accesses web content and displays the resulting web pages for users.

Controller

A company, organization or individual who manages web content on a web server and determines what it serves.

Crawler

A user agent which accesses web content automatically.

Search Engine

A service that indexes web content and provides an interface to search this index, typically including links to the indexed content and often copies of it as well.

Service Provider

A company, organization or individual who owns and operates a web server.

User

A real person. In this document, somebody who is using a browser.

User Agent

Software that accesses web content.

Web Server

Software that makes web content available on the web.

Origin Server

The web server from which particular web content originates.

Proxy

A kind of web server that relays web content

Caching Proxy

A proxy that keeps a copy of the web content that it relays.

Reuser

In this document, a web server that that reuses web content from elsewhere on the web, adding value to it by combining it with other information or transforming it, e.g., a ?server-side mashup.?

3.2 Artifacts

Address

A string used to name web content, with which a user agent can access it. Web standards call this a URL, URI or IRI.

Link

An address, visibile within a web page, with which a user can interact.

Web Content

Content available on the web. Web content includes data in a variety of formats: text, HTML, images, video, audio, style sheets and scripts, or anything hosted by a web server that may be accessed by a user agent. When the user agent is a browser, top-level access usually results in a user viewing a web page.

Web Page

What the user sees and/or hears when a browser accesses a web content.

3.3 Actions

Access

To retrieve a web content using its address, and display or otherwise process it. For some web content, processing may involve accessing additional web content which is embedded or included, recursively. . .

Archive

To permanently store a copy of web content that is hosted elsewhere.

Cache

To (temporarily) store a copy of web content which is hosted by another server in order to avoid retrieving it again. The cached copy is used instead of passing through the request.

Copy

In the context of this document, the duplication of bits from one place to another place. Since copy is an imprecise term, we have tried to use more specific terms such as cache, relay, alias and include.

Relay

To provide access to web content hosted by another server without keeping a copy.

Embed

To include web content (such as text, an image or video) within a Web Page in a way that the embedded web content appears as an integral part of the parent web page when it is presented to a user.

Host

In this document, to provide original access (not as a proxy or relay) to web content.

Include

Within this document, to access additional web content (such as an image or a script), while processing a web page.

Index

To extract information from web content (hosted elsewhere) to facilitate searching and/or retrieval of that content.

Transform

To change content such as by reformatting, resizing, translating, or rewriting content.

Upload

To put a file on a server such that it is given an address on the web

4.1 Hosting

The basic form of publication on the web is hosting. A server hosts a file if it stores the file on disk or generates the file from data that it stores, and that file did not (to the server's knowledge) originate elsewhere on the web.

The presence of data on a server does not necessarily mean that the organisation that owns and maintains the server has an awareness of the presence of the data or its content. Many websites are hosted on shared hardware that is owned by a service provider that stores and serves data at the direction of controlling individuals and organisations which determine the data they provide on the site. Because of this, multiple servers may host the same file at different locations.

There are many different types of service provider. Some exercise practically no control over the software and data that they host, merely providing a base platform on which code can run. Others may focus on particular types of content, such as images (e.g. Flickr), videos (e.g. YouTube) or messages (e.g. Twitter). Also, there may be many service providers involved in the publication of a particular file on the web: some providing hardware, others providing different kinds of publishing support.

Some service providers automatically perform transformations on material that they host, as a service, such as converting to alternative formats, clipping or resizing, or marking up text. When they sign up to a service, controllers explicitly or implicitly enter into an agreement with the service provider that grants them a license to perform transformations on the material which they upload.

Service providers that host particular types of material often employ automatic filters to prevent the publication of unlawful material, but it is impossible for a service provider to detect and filter out everything that might be unlawful.

To add to the complexity of this area, it is possible for each of the following to be in different jurisdictions:

• the individual or organisation who controls the data
• the service provider(s)
• the physical servers that host the data

and be controlled by different laws and conventions.

4.2 Caching and Relaying

Some servers provide access to files that are hosted elsewhere on the web: on an origin server that holds the original version of the file. These files might be stored on the server and provided again at a later time (a caching proxy in this document), or might simply pass through the server in response to a request (a relaying server in this document).

It is often impossible to tell whether a server is providing a stored response or whether it has made a new request to the origin server and is serving the results of that request. Servers commonly store the results of some requests and not others, acting as a caching proxy some of the time and as a relaying server the rest.

In both cases, the file the caching or relaying server provides might be different from the original web content that was accessed from the origin server. For example:

• Links within HTML content might be rewritten so that they point to content that is also served by the caching proxy.
• Javascript and CSS files might be combined and compressed to provide speedier access.
• Banners might be added within an HTML page to highlight that the page is a copy of an original from somewhere else.
• Content might be compressed or converted to different formats.
• Wholly new content might be created that bring together information from multiple different sources.

Caching and relaying servers are extremely useful on the web. There are four main types of caching and relaying servers discussed here: proxies, archives, search engines and reusers. The distinctions between them are summarised in the table below.

4.3 Including

A web page written in HTML may include other web content, such as images, video, scripts, stylesheets, data and other HTML. The HTML in a web page refers to this external web content using markup. For example, an <img> element uses the src attribute to refer to an image which should be shown within the page. Material that is included within a web page may appear to be a hosted copy to the user of a website, but in fact may come from somewhere else, entirely outside the control of the owner of the web page.

HTML supports several different mechanisms for including external web content in a web page but they all work in essentially the same way. When a user navigates to a web page, the browser automatically fetches all the included web content into its local cache and executes them or displays them within the page.

Inclusion is different from hosting, copying or disseminating a file because the information is never stored on, nor passes through, the server that hosts the web page doing the including. As such, although the included web content is an essential component of the page to make it appear and function as a whole, the server of the web page does not have control over content which may change without its knowledge.

4.4 Linking

Linking is a fundamental facility on the web. In fact, it has been argued that linking is what makes the Web the Web. HTML pages can include links using the <a> element to other pages on the web with the href attribute holding the URI for the linked page. Some of the links may be to be pages from the same origin; while others will be cross-origin links to pages on third-party's sites that hold related information.

A user can usually tell where a link is going to take them prior to selecting it by "mousing over" it or after the link is selected through the status bar in the browser, although some links are overridden by onclick event handling that takes them to a different location. Some websites, such as Wikipedia, use icons to indicate whether a link is a cross-origin link or whether it will take a user to a page on the same server. The use of interstitial pages or dialog boxes which warn the user they are about to leave the site in question can obscure the eventual destination of the link.

6.1 Controlling Access

Publishers can control access to pages in several ways. In addition to not giving out URIs to these pages, they can control access via the Referer HTTP header which indicates the last page that was referenced. If it was not a page on your own site, then you can redirect to your site's home page, for example. It is also possible to do this check in JavaScript, which can then be used to bring up a dialog to check whether the contractual terms have been read, to confirm that the user is over 18, or to ask for a password.

You can also use a cookie, for example, to start a session only when a page is accessed through a given gateway page and reject or provide an alternative path for requests that don't have the cookie set.

The User-Agent HTTP header which indicates the identity of the user agent making the request is particularly useful in preventing access from crawlers and search engines. A robots.txt file on the website can be also used for the same purpose.

The domain name or IP address of the client making the connection can also be used to prevent specific reusers from accessing material.

6.2 Controlling Inclusion

As well as the techniques above, which can be used to control any access to pages, it's also possible to provide additional control over the inclusion of content in a third-party's web pages.

To prevent an HTML page from being embedded within a frame, publishers can include a script that checks whether the document is the top document in the window.

The Cross-Origin Resource Sharing Working Draft [CORS] defines a set of HTTP headers that can be used to give the publisher of the third-party resource greater control over access to their resources. These are usually used to open up cross-origin access to resources that publishers want to be reused, such as JSON or XML data exposed by APIs, by indicating to the browser that the resource can be fetched by a cross-origin script.

Publishers should ensure actions are not taken on behalf of their users in response to an HTTP GET on a URI, as otherwise sites are open to security breaches through inclusions, as described in section 4.3 Including. It is also good practice to check the Referer header in these cases to prevent actions being taken as the result of the submission of forms within other website's web pages, unless that functionality is desired.

6.3 Controlling Caching

There are a number of HTTP headers [HTTP11] that enable content providers to indicate whether a proxy should cache a given page and for how long it should keep the copy. These are described in detail within Section 13: Caching in HTTP. For example, a server can use the HTTP header Cache-Control: no-store to indicate that a particular response should not be cached by a proxy server.

Publishers of websites can also indicate which pages should not be fetched or indexed by any search engine or archive through robots.txt [ROBOTS] and the robots <meta> element [META]. They can indicate other characteristics of web pages, such as how frequently they might change and their importance on the website, through sitemaps [SITEMAPS]. More sophisticated publishers may use the Automated Content Access Protocol (ACAP) extensions [ACAP] to attempt to indicate access policies.

Publishers can also use the rel="canonical" link relationship to indicate a canonical URI for a page which should be used by search engines and other reusers to reference a given page.

6.4 Controlling Processing

The Cache-Control: no-transform HTTP header indicates that a proxy server must not change the original content, nor the headers:

• Content-Encoding
• Content-Range
• Content-Type

For example, an proxy server must not convert a TIFF served with Cache-Control: no-transform into a JPG, nor should it rewrite links within an HTML page.

6.5 Licensing

Websites can include a license that describes how the information served by the website can be reused by others.

Just as with HTTP headers, robots.txt and sitemaps, there can be no technical guarantees that crawlers will honor license information within a site. However, to give well behaved crawlers a chance of identifying the license under which a page is published, websites should:

• Indicate the license of a web page, an image or a video using a mechanism whereby the license can be detected automatically such as
  • a <link> or <a> element with rel="license"
  • microformats [MICROFORMATS], RDFa [RDFA-CORE] or microdata [MICRODATA] with a common property such as xhv:license to indicate the license of included content./li>
• Ensure that the conditions of the license are machine-understandable by
  • referencing a well-known license (such as a Creative Commons license) through its URI, or
  • providing a machine-readable description of the license, for example using the Creative Commons Rights Expression Language.

6.6 How to Write Good Websites

This section describes techniques that you should use when operating a website that incorporates material from other sources, whether by caching, transforming or simply linking.