I think there are two issues here.

First, how to handle IDREFs that don't refer to anything:  IDs are used in other places such as CSS rules and named links:
- p#foo { text-align: center; }
- <a href="www.somewhere.com/#bar">

Without doing a deep dive into the specs, neither CSS nor HTML say what happens if the ID doesn't exist.  But, the implication is that if there is no <p> element with ID="foo", then that style is not applied.  And, if there is no element with id="bar", then the link goes nowhere.

Similarly for the text alternative computation (TAC).  If an element/id doesn't exist, then the accumulated text remains as is -- nothing is added to it.

However, it's possible that an implementation of the TAC may want detect this situation.  In that case, the result might "null" instead of "the empty string".  That is, 
"Set the current node to the node referenced by the IDREF.  If there is no such node, return <null>".

That would distinguish between a bad IDREF vs. a valid IDREF that results in the return of an empty string.

Would that distinction be useful?

A second issue is illustrated by your first example:
<div role="button" aria-labelledby="i-dont-refer-to-anything">Contents</div>

The TAC produces an empty string for the name of the button, which is useless.  FF produces "Contents" as the name of the button.  I suspect that FF is (1) calculating the name via the bad ID, and when it comes up empty, (2) re-runs the TAC as if the markup was <div role="button">Contents</div>.

In other words, if the TAC fails to generate a name based on name-from-author, run it again using name-from-contents.

I'm adding Alex to the CC list for his insights.

(Note:  the step in question is:
http://rawgit.com/w3c/aria/master/accname-aam/accname-aam.html#step2B)

I think wrong IDRefs should be ignored, and processed as they weren't present, since it's clearly an author error. I'm not sure about empty aria-labelledby, I guess authors may want to put it on demand to say explicitly that an accessible doesn't have a name, similarly to what they do img@alt="".

This was most recently addressed at https://github.com/w3c/accname/issues/5 , so closing this here for now. Please address comments on GitHub against the AccName 1.2 milestone if anything is unclear.