*** Bug 22027 has been marked as a duplicate of this bug. ***

This raises the question which characters are allowed. Is it specified somehow?

It seems that indirectly it is specified for the XHTML syntax, since it must follow XML 1.0 rules, and they define the allowed characters. In particular, U+0000 NULL is not allowed.

NULL is not allowed in HTML 4.01 either. I think browsers usually ignore NULL, but validators may not, and this has caused some confusion, especially since NULL usually appears due to some feature in some software rather than an author’s informed action.

If rules are set for character repertoire, they could also specify some general processing rules, e.g. requiring that some characters, though forbidden, must be ignored by user agents when in HTML mode. (In XHTML mode, XML 1.0 rules imply that e.g. NULL is a well-formedness error, with Draconian implications.)

(In reply to comment #2)
> This raises the question which characters are allowed. Is it specified
> somehow?

Yes. See "parse error" in e.g.

http://www.whatwg.org/specs/web-apps/current-work/multipage/parsing.html#preprocessing-the-input-stream
http://www.whatwg.org/specs/web-apps/current-work/multipage/tokenization.html#data-state
http://www.whatwg.org/specs/web-apps/current-work/multipage/tree-construction.html#parsing-main-inbody

But this is a bit off-topic for this bug.

There are two test cases:

<pre>NULL
next line

and:

<textarea>NULL
next line

where "NULL" is a literal NULL character (U+0000) expressed in the appropriate character encoding.

For <pre> the NULL will be tokenized in the data state, and passed up to tree construction as a character token, but then ignored by the "in body" insertion mode. Since the token is generated, but ignored, does it count as the "next token" or not? The browsers seem to think not, and they still strip the following newline. So the spec could be clarified to define "next token" in a way that reflects this.

For <textarea> the NULL will be tokenized in the rcdata state, which generates a character token containing the replacement character (U+FFFD) instead. This is clearly the "next token", so the following newline should *not* be stripped. Chrome acts as expected, but Firefox strips it anyway. This appears to be a bug in Firefox.

Given the lack of a definition for "next token", there may be other inconsistencies and ambiguous cases that we have not noticed yet.

The U+0000 token is the "next token" in these cases.

Checked in as WHATWG revision r7949.
Check-in comment: Clarify 'next token' in the HTML parser.
http://html5.org/tools/web-apps-tracker?from=7948&to=7949

So Firefox and Chrome are incorrect then, given that they both strip the newline even though it follows an (ignored) NUL character?

Yup. File bugs. :-)