Abstract

This document defines a mechanism enabling web sites to declare a reporting policy that can be used by the user agent to report encountered network errors that prevented it from successfully loading the requested resource, the format of the error reports, and their transmission mechanism.

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document was published by the Web Performance Working Group as a Working Draft. This document is intended to become a W3C Recommendation. If you wish to make comments regarding this document, please send them to public-web-perf@w3.org (subscribe, archives). All comments are welcome.

A lot of significant changes were done to the document. Readers are advised to read the document entirely again.

Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

This document is governed by the 1 August 2014 W3C Process Document.

Table of Contents

1. Introduction

Accurately measuring performance characteristics of web applications is an important aspect in helping site developers understand how to improve their web applications. The worst case scenario is the failure to load the application, or a particular resource, due to a network error, and to address such failures the developer requires assistance from the user agent to identify when, where, and why such failures are occurring.

Today, application developers do not have real-time web application availability data from their end users. For example, if the user fails to load the page due to a network error, such as a failed DNS lookup, a connection timeout, a reset connection, or other reasons, the site developer is unable to detect and address this issue. Existing methods, such as synthetic monitoring provide a partial solution by placing monitoring nodes in predetermined geographic locations, but require additional infrastructure investments, and cannot provide truly global and near real-time availability data for real end users.

Network Error Logging (NEL) addresses this need by defining a mechanism enabling web applications to declare a reporting policy that can be used by the user agent to report network errors for a given origin. To take advantage of NEL, a web application opts into using NEL by supplying a NEL HTTP response header field that describes the reporting policy. Then, if the NEL policy is available for a given origin, and an end user fails to successfully fetch a resource from that origin, the user agent logs the network error report and attempts to deliver it to the report URL defined by the policy - the delivery is done on a best effort basis and may be delayed due to connectivity issues or other reasons.

2. Network Error Logging

2.1 Key Concepts and Terminology

origin
Defined by the Origin specification. [RFC6454]
JSON object
JSON stringification
Defined in the JSON specification. [RFC4627]
URL
Defined by [URL].

2.2 Policy Delivery and Processing

The server delivers the NEL policy to the user agent via an HTTP response header field (NEL header field). The policy MUST be delivered over a secure transport. If the policy is delivered over a secure transport with no underlying secure transport errors or warnings, and its format conforms to the specified grammar, the user agent MUST either:

Otherwise, if the policy is received over an insecure transport, or if the policy does not conform to the specified grammar, then the user agent MUST ignore it.

Note

The application does not need to be HTTPS-only to take advantage of NEL. To enable NEL reporting for an HTTP site the application can fetch a single "NEL registration resource" (e.g fetch an empty pixel) from the same domain name but over a secure transport - e.g. http://example.com can fetch https://example.com/registerNEL. Once registered, the NEL policy applies to both HTTPS and HTTP fetches for that NEL host - i.e. an NEL registration for https://example.com also applies to http://example.com fetches.

2.2.1 NEL Header Field

The NEL header field is used to communicate the NEL policy to the user agent. The ABNF (Augmented Backus-Naur Form) syntax for the NEL header field is as follows:

NEL = "NEL" ":" [ directive ]  *( ";" [ directive ] )

directive                 = directive-name [ "=" directive-value ]
directive-name            = token
directive-value           = token | quoted-string

See [RFC7230] section 3.2.6 for definitions of token and quoted-string. This specification defines three NEL directives: report-uri, max-age, and includeSubDomains. The overall requirements for directives are:

  • The order of appearance of directives is not significant.
  • All directives MUST appear only once in an NEL header field. Directives are either optional or required, as stipulated in their definitions.
  • Directive names are case-insensitive.
  • User agents MUST ignore any NEL header field containing directives, or other header field value data, that does not conform to the syntax defined in this specification.
  • If an NEL header field contains directive(s) not recognized by the user agent, the user agent MUST ignore the unrecognized directives, and if the NEL header field otherwise satisfies the above requirements, the user agent MUST process the recognized directives.

Additional directives extending the semantic functionality of the NEL header field can be defined in other specifications, with a registry (having an IANA policy definition of IETF Review [RFC5226]) defined for them at such time.

The user agent MUST ignore the NEL header specified via a meta element to mitigate hijacking of error reporting via scripting attacks. The NEL policy MUST be delivered via the NEL header field.

Note

The restriction on meta element is consistent with [CSP] specification, which restricts reporting registration to HTTP header fields only for same reasons.

2.2.1.1 The report-uri Directive

The report-uri directive specifies a URL to which the user agent sends reports about network errors. The report-uri directive is a REQUIRED field to register an NEL policy, and OPTIONAL if the intent is to remove a previous registration - see max-age. The ABNF grammar for the name and value of the directive is:

directive-name    = "report-uri"
directive-value   = 1*uri-reference

The set of report URLs is the value of the report-uri directive, which contains one or more uri-reference's, each resolved relative to the URL of the fetched resource that specified the policy - see [RFC3986] section 4 and appendix C for definition of uri-reference and delimiting recommendations.

Each report URL in the provided set of report URLs MUST use a secure transport to receive the NEL reports. If any of the provided report URL's does not use a secure transport, the user agent MUST ignore the provided policy. The process of sending network error reports to the specified URL's in this directive's value is defined in this documents 2.4 Reporting section.

Note

To improve delivery of NEL reports the application should set report-uri to an alternative origin whose infrastructure is not coupled with the origin from which the resource is being fetched — otherwise network errors cannot be reported until the problem is solved, if ever — and provide multiple report-uri's to provide fallback alternatives if the preceding report-uri is unreachable.

2.2.1.2 The max-age Directive

The REQUIRED max-age directive specifies the number of seconds, after the reception of the NEL header field, during which the user agent regards the host (from whom the policy was received) as a known NEL host. The ABNF grammar for the name and value of the directive is:

directive-name    = "max-age"
directive-value   = delta-seconds

See [RFC7234] section 1.2.1 for definition of delta-seconds. A max-age value of zero (i.e. "max-age=0") signals the user agent to cease regarding the host as a known NEL host, including the includeSubDomains directive if provided.

2.2.1.3 The includeSubDomains Directive

The OPTIONAL includeSubDomains directive is a valueless directive that, if present, signals the user agent that the NEL policy applies to this NEL host as well as any subdomains of the host's domain name.

2.3 Policy Storage and Maintenance

An HTTP host declares itself an NEL host by issuing an NEL policy, which is communicated via the NEL header field over secure transport (e.g., TLS). Upon error-free receipt and processing of this header by a conformant user agent, the user agent regards the host as a known NEL host.

The user agent MUST store and index NEL policies based strictly upon the domain names of the issuing NEL hosts. An NEL policy may contain an optional includeSubDomains directive specifying that this policy also applies to any hosts whose domain names are subdomains of the known NEL host's domain name.

The user agent MUST maintain the NEL policy of any given NEL host separately from any NEL policies issued by any other NEL hosts whose domain names are superdomains or subdomains of the given NEL host's domain name. Only the given NEL host can update or cause deletion of its NEL policy. This is accomplished by sending a NEL header field to the user agent with new values for the policy report URL, time duration, and subdomain applicability. Thus, the user agent MUST store the "freshest" NEL policy information on behalf of an NEL host, and specifying a zero time duration MUST cause the user agent to delete the NEL policy (including any asserted includeSubDomains directive) for that NEL host.

2.4 Reporting

A network error is any condition where a connection or a protocol error is encountered by the user agent, thus preventing it from successfully completing the request-response exchange. This may include, but is not limited to DNS, TCP, TLS, and HTTP connection and protocol errors. For example, a network error is triggered when the user agent:

The user agent MAY classify and report server error responses (5xx status code, [RFC7231]) as network errors. For example, a network error report may be triggered when a fetch fails due to proxy or gateway errors, service downtime, and other types of server errors.

The failure to fetch a resource when the user agent is known to be offline (when navigator.onLine returns false) MUST NOT be considered to be a network error.

Note

Note that the above definition of "network error" is different from definition in [Fetch]. The definition of network error in this specification is a subset of [Fetch] definition - i.e. all of the above conditions would trigger a "network error" in [Fetch] processing, but conditions such as blocked requests due to mixed content, CORS failures, etc., would not.

When a network error occurs for a URL that belongs to a known NEL host the user agent MUST log the error and attempt to deliver it to the report URL defined by the NEL policy of the associated NEL host:

To generate a network error object, the user agent MUST use an algorithm equivalent to the following:

  1. Prepare a JSON object neterror with the following keys and values:
    uri
    The URL that encountered the network error, with any fragment component removed.
    referrer
    The referrer attribute of the resource, or the empty string if the resource has no referrer.
    server-ip
    The IP address of the host to which the user agent sent the request, if available. Otherwise, an empty string.
    • A host identified by an IPv4 address is represented in dotted-decimal notation (a sequence of four decimal numbers in the range 0 to 255, separated by "."). [RFC1123]
    • A host identified by an IPv6 address is represented as an ordered list of eight 16-bit pieces (a sequence of x:x:x:x:x:x:x:x, where the 'x's are one to four hexadecimal digits of the eight 16-bit pieces of the address). [RFC4291]
    protocol
    The network protocol used to fetch the resource as identified by the ALPN Protocol ID, if available. Otherwise, an empty string.
    status-code
    The status-code of the HTTP response, if available. Otherwise, the number 0.
    elapsed-time
    The elapsed number of milliseconds between the start of the resource fetch and when it was aborted by the user agent.
    age
    The elapsed number of milliseconds between the time when the user agent aborted the request and when the error report is being delivered.
    type
    The description of the error type, which may be one the following strings:
    dns.unreachable
    DNS server is unreachable
    dns.name_not_resolved
    DNS server responded but is unable to resolve the address
    dns.failed
    Request to the DNS server failed due to reasons not covered by previous errors
    tcp.timed_out
    TCP connection to the server timed out
    tcp.closed
    The TCP connection was closed by the server
    tcp.reset
    The TCP connection was reset
    tcp.refused
    The TCP connection was refused by the server
    tcp.aborted
    The TCP connection was aborted
    tcp.address_invalid
    The IP address is invalid
    tcp.address_unreachable
    The IP address is unreachable
    tcp.failed
    The TCP connection failed due to reasons not covered by previous errors
    tls.version_or_cipher_mismatch
    The TLS connection was aborted due to version or cipher mismatch
    tls.bad_client_auth_cert
    The TLS connection was aborted due to invalid client certificate
    tls.cert.name_invalid
    The TLS connection was aborted due to invalid name
    tls.cert.date_invalid
    The TLS connection was aborted due to invalid certificate date
    tls.cert.authority_invalid
    The TLS connection was aborted due to invalid issuing authority
    tls.cert.invalid
    The TLS connection was aborted due to invalid certificate
    tls.cert.revoked
    The TLS connection was aborted due to revoked server certificate
    tls.cert.pinned_key_not_in_cert_chain
    The TLS connection was aborted due to a key pinning error
    tls.protocol.error
    The TLS connection was aborted due to a TLS protocol error
    tls.failed
    The TLS connection failed due to reasons not covered by previous errors
    http.protocol.error
    The connection was aborted due to an HTTP protocol error
    http.response.invalid
    Response is empty, has a content-length mismatch, has improper encoding, and/or other conditions that prevent user agent from processing the response
    http.response.redirect_loop
    The request was aborted due to a detected redirect loop
    http.failed
    The connection failed due to errors in HTTP protocol not covered by previous errors
    abandoned
    User aborted the resource fetch before it is complete
    unknown
    error type is unknown

    The user agent MAY extend the above error type list with custom values - e.g. new error types to accommodate new protocols, or more detailed error descriptions of existing ones. When doing so, the user agent SHOULD follow the dot-delimited pattern ([group].[optional-subgroup].[error-name]) to facilitate simple and consistent processing of the error reports - e.g. the collector may provide aggregation by category and/or one or multiple subgroups.

  2. Return neterror.

To send network error reports, the user agent MUST use an algorithm equivalent to the following:

  1. Prepare a JSON object report object with the following keys and values:
    nel-report
    An array object containing one or more network error objects.
  2. Let report body be the JSON stringification of the report object.
  3. For each report URL in the set of report URLs:
    1. Queue a task to fetch report URL using HTTP method POST, with a Content-Type header field of application/nel-report, and an entity body consisting of report body. If the origin of report URL is not the same as the origin of the NEL host for which the report is generated, the block cookies flag MUST also be set. The user agent MUST NOT follow redirects when fetching this resource and ignore the fetched response if one is provided. The task source for these tasks is the "Network Error Logging" task source.
    2. If the fetch is successful (2xx HTTP response code), the user agent MUST abort the remaining steps.
    3. If the fetch failed with a 410 Gone HTTP response code ([RFC7231]), the user agent MUST update the NEL policy by removing the current report URL from the set of report URLs.
    4. If the fetch failed after multiple delivery attempts the user agent MAY update the NEL policy by removing the current report URL from the set of report URLs.
      Note

      The user agent is allowed to garbage collect report-uri's that are no longer functional, but it should account for common failure cases such as captive portals, which may temporarily prevent it from delivering the error reports. The exact implementation logic is deferred to the user-agent, which may use own mechanisms and heuristics to detect such cases.

    5. If the NEL policy was updated and the new NEL policy contains an empty set of report URLs the user agent MUST remove the NEL policy and abort the remaining steps.
    6. If the user agent has reached the end of the set of report URLs, the user agent MUST sleep before returning to the beginning of the set and reattempting delivery - e.g. use exponential backoff. Otherwise, the user agent MUST attempt immediate delivery of the error report to the next report URL in the set of report URLs.

2.5 Examples

2.5.1 Sample Policy Definitions

Example 1
> GET / HTTP/1.1
> Host: example.com

< HTTP/1.1 200 OK
< ...
< NEL: report-uri="/report"; max-age=2592000

The above NEL policy provided in the server response specifies that the user agent should register a new NEL policy, or update an existing one if one already exists, for the example.com NEL host: the user agent should report network errors to /report URL resolved relative to the origin of the NEL host, and that the policy applies for 2592000 seconds (30 days).

Note that above registration will only succeed if the response is delivered over a secure transport - see 2.2 Policy Delivery and Processing. To register an NEL policy:

  • An HTTPS application can respond with the NEL policy in its response.
  • An HTTP application can provide a single "NEL registration resource" on the same domain but over a secure transport.
Example 2
> GET / HTTP/1.1
> Host: example.com

< HTTP/1.1 200 OK
< ...
< NEL: report-uri="https://other-origin.com/report"; max-age=2592000

The above NEL policy provided in the server response is similar to the previous example but tells the user agent to report network errors to https://other-origin.com/report. The use of an alternative origin that is not coupled with the origin that is being accessed is strongly encouraged to enable real-time reporting and improved report delivery - see 2.2.1.1 The report-uri Directive.

Example 3
> GET / HTTP/1.1
> Host: example.com

< HTTP/1.1 200 OK
< ...
< NEL: report-uri="/report" "https://other-origin.com/report"; max-age=2592000; includeSubDomains

The above NEL policy provided in the server response specifies that the user agent should report network errors to /report URL resolved relative to the origin of the NEL host, or https://other-origin.com/report if /errors is unreachable. Further, the policy is extended to all of the subdomains of the issuing NEL host - see 2.2.1.3 The includeSubDomains Directive.

Example 4
> GET / HTTP/1.1
> Host: example.com

< HTTP/1.1 200 OK
< ...
< NEL: max-age=0

The above NEL policy provided in the server response contains max-age set to zero, which indicates that the user agent must delete the current registered NEL policy associated with the example.com NEL Host and all of its subdomains:

2.5.2 Sample Network Error Reports

This section contains an example network error report the user agent might send when a network error is encountered for a known NEL host.

{
  "nel-report": [
      {
        "uri": "https://www.example.com/",
        "referrer": "http://example.com/",
        "server-ip": "123.122.121.120",
        "protocol": "h2-15",
        "status-code": 200,
        "elapsed-time": 823,
        "age": 0,
        "type": "http.protocol.error"
      }
    ]
  }

The above report indicates that the user agent attempted to navigate from "example.com" to "www.example.com" (known NEL host), which successfully resolved to the "123.122.121.120" IP address. However, while the user agent received a "200" response from the server via the "h2-15" protocol, it encountered a protocol error in the exchange and was forced to abandon the navigation. 823 milliseconds elapsed between the start of navigation and when the user agent aborted the navigation. Finally, the user agent sent this report immediately after the network error was encountered - i.e. the report age is 0.

{
  "nel-report": [
      {
        "uri": "https://widget.com/thing.js",
        "referrer": "https://www.example.com/",
        "server-ip": "234.233.232.231",
        "protocol": "",
        "status-code": 0,
        "elapsed-time": 143,
        "age": 0,
        "type": "http.dns.name_not_resolved"
      }
    ]
  }

The above report indicates that the user agent attempted to fetch "https://widget.com/thing.js", which belongs to a previously registered NEL host, from "www.example.com" origin. However, the user agent was unable to resolve the DNS name and the request was aborted by the user agent after 143 milliseconds. Because "widget.com" is a known NEL host, a network error report was logged and sent to the report URL specified by the NEL policy of that host immediately after the network error was encountered - i.e. the report age is 0.

2.6 Use cases

2.6.1 Reporting of Navigation Failures

A navigation request initiated by the user (e.g. via a click on a link, direct input via the location bar, script-initiated due to user interaction, etc.) may fail due any number of connectivity reasons: DNS failure, TCP error, TLS protocol violation, and so on. These errors may be caused by network misconfiguration, transient routing issues, server downtime, malware or other attacks against the user, etc.

In such cases the destination host is often left unaware of the failed navigation since, by definition, it cannot see the request reach its infrastructure and it is unable to investigate the problem. To address this, the host can register an NEL policy with the user agent, which specifies where reports of such failures should be delivered such that they can be investigated.

2.6.2 Reporting of First-party Subresource Fetch Failures

A typical application requires dozens of resources, the fetching of which is typically initiated via HTML, CSS, or JavaScript. The application requesting such resources can observe failures of most such fetches (e.g. via onerror callbacks), but it does not have access to the detailed network error report of why the failure has occurred - e.g. DNS failure, TCP error, TLS protocol violation, etc.

To address this, the application can register relevant NEL policies with the user agent for the first-party hosts from which the subresources are being fetched. Then, if such a policy is present and a network error is encountered for a resource associated with a registered NEL host, the user agent will report the detailed network error report and enable the application developers to investigate the error.

2.6.3 Reporting of Third-party Subresource Fetch Failures

In the case where a resource is embedded by a third party, the provider of the resource is often unable to instrument and observe the failure. For example, if example.com embeds a widget.com/thing.js resource on its site, and the user visiting example.com fails to fetch such resource due to a network error, the widget.com host is both unaware of the failure and unable to detect it.

To address this, widget.com can register an NEL policy for its host. Then, if such policy is present and a network error is encountered while fetching a resource — regardless of whether it is being requested from a first-party or third-party origin — from the registered NEL host, the user agent will report the network error and enable the provider to investigate the error.

2.7 IANA Considerations

... TODO ...

A. Acknowledgments

This document reuses text from the [CSP] and [RFC6797] specification, as permitted by the licenses of those specifications. Additionally, sincere thanks to Thomas Tuttle, Chris Bentzel, Todd Reifsteck, and Aaron Heady for their helpful comments and contributions to this work.

B. References

B.1 Normative references

[RFC3986]
T. Berners-Lee; R. Fielding; L. Masinter. Uniform Resource Identifier (URI): Generic Syntax. January 2005. Internet Standard. URL: https://tools.ietf.org/html/rfc3986
[RFC5226]
T. Narten; H. Alvestrand. Guidelines for Writing an IANA Considerations Section in RFCs. May 2008. Best Current Practice. URL: https://tools.ietf.org/html/rfc5226
[RFC7230]
R. Fielding, Ed.; J. Reschke, Ed.. Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing. June 2014. Proposed Standard. URL: https://tools.ietf.org/html/rfc7230
[RFC7234]
R. Fielding, Ed.; M. Nottingham, Ed.; J. Reschke, Ed.. Hypertext Transfer Protocol (HTTP/1.1): Caching. June 2014. Proposed Standard. URL: https://tools.ietf.org/html/rfc7234

B.2 Informative references

[CSP]
Brandon Sterne; Adam Barth. Content Security Policy 1.0. 19 February 2015. W3C Note. URL: http://www.w3.org/TR/CSP1/
[Fetch]
Anne van Kesteren. Fetch Standard. Living Standard. URL: https://fetch.spec.whatwg.org/
[RFC1123]
R. Braden, Ed.. Requirements for Internet Hosts - Application and Support. October 1989. Internet Standard. URL: https://tools.ietf.org/html/rfc1123
[RFC4291]
R. Hinden; S. Deering. IP Version 6 Addressing Architecture. February 2006. Draft Standard. URL: https://tools.ietf.org/html/rfc4291
[RFC4627]
D. Crockford. The application/json Media Type for JavaScript Object Notation (JSON). July 2006. Informational. URL: https://tools.ietf.org/html/rfc4627
[RFC6454]
A. Barth. The Web Origin Concept. December 2011. Proposed Standard. URL: https://tools.ietf.org/html/rfc6454
[RFC6797]
J. Hodges; C. Jackson; A. Barth. HTTP Strict Transport Security (HSTS). November 2012. Proposed Standard. URL: https://tools.ietf.org/html/rfc6797
[RFC7231]
R. Fielding, Ed.; J. Reschke, Ed.. Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content. June 2014. Proposed Standard. URL: https://tools.ietf.org/html/rfc7231
[URL]
Anne van Kesteren; Sam Ruby. URL. 9 December 2014. W3C Working Draft. URL: http://www.w3.org/TR/url-1/