If some shadow root's apply-author-styles is true, styles in a different tree scope, e.g. document, will be applied to elements in the shadow dom tree.

In this case, we need to compare document positions of styles in a document tree and styles in the shadow dom tree. (also need to consider styles in an enclosing shadow tree).

I'm not sure whether it is ok to use a document position of a shadow host to solve the above case or not.
If ok, I'm thinking of the following algorithm:

Suppose that two nodes, A and B. A is in some shadow dom tree and B is in another shadow dom tree. B's tree scope is an ancestor tree scope of A's.

To compare document position of A and B:
(1) going up from node A to the shadow host C in B's tree scope. I mean, A's host -> A's host's host... to reach B's tree scope.
(2) compare C with B (by using Node::compareDocumentPosition).

In general, 
(1) finding a common ancestor node of A and B. 

(2) comparing immediate children of the common ancestor, A' and B'. A' is an ancestor node of A and B' is an ancestor node of B.

(3) if A' and B' are in the same tree scope, just compare A' with B' in the original compareDocumentPosition's way.

(4) if A' is a shadow root and B' is not a shadow root, treat A' as a shadow host. Since B' is a child of the shadow host, B' is following A'. So B is following A.

(5) if both A' and B' are shadow roots, see which is a younger shadow root and follow the rule defined in shadow dom spec: 
"When a shadow host has multiple shadow roots, the sorting order of CSS rules, defined in shadow roow style sheets, in ascending order of precedence must match the order of the tree stack, ending with the youngest tree."

Best regards,
Takashi Sakamoto