This is a joint research with Travis Safford. 
window.open(url, name, [args]) makes it easy for websites accepting user supplied URLs to be vulnerable when attackers can cause a collision on the the “name” parameter. For example, consider a genuine site embedding an iframe with name “myFrame”, and allows user to supply the url of the iframe. 

e.g. www.example.com/myPage
<iframe src=”http://www.google.com/” id=”myFrameId” name=”myFrame”></iframe>
<script>
window.open(user_supplied_url, “myFrame”);
</script>

In normal cases, if the user_supplied_url is a regular http url, the frame would be redirected to the new URL. Otherwise if the user_supplied_url is a “javascript:” script, an exception would be thrown because of the Same Origin Policy (SOP).

However, if an attacker frames this page on his own site and names the frame “myFrame” as well, the URL will be opened in the parent frame instead of the frame within the genuine site. In fact, if the url is “javascript:”, it will execute on the genuine domain causing an XSS. 

e.g. www.attacker.com
<iframe src=”http://www.example.com/myPage” name=”myFrame”></iframe>

You can also test this out with Test3 at  “http://test.attacker-domain.com/sopresearch/windowopen.html”. This page assumes that “test.attacker-domain.com” is the malicious domain while “exam.attacker-domain.com” is the genuine domain. You can observe that JavaScript runs in the genuine domain when it’s framed by the malicious domain with the same “myFrame” name. 

The reason this happens is because whenever there are multiple browser contexts with the same “name”, the spec does not enforce the order of selecting the browser contexts. In fact, the spec says “the user agent should select one in some arbitrary consistent manner” which looks very vague as a spec. The full spec on searching for the browser context for the “name” is below. 

“If the given browsing context name is not _blank and there exists a browsing context whose name is the same as the given browsing context name, and the current browsing context is familiar with that browsing context, and the user agent determines that the two browsing contexts are related enough that it is ok if they reach each other, then that browsing context must be the chosen one. If there are multiple matching browsing contexts, the user agent should select one in some arbitrary consistent manner, such as the most recently opened, most recently focused, or more closely related.”

It is not intuitive for the parent frame to be chosen as the browsing context when there is a child iframe with the same name in the current page. Not only does this potentially present XSS vulnerabilities, but it makes pages render differently when iframed. We would recommend the spec enforce the searching order from the bottom up, from within the current page, and reach out to parent or outer pages if nothing is found within the page. This would limit the impact of sites becoming vulnerable when accepting a user supplied url to the URL parameter of window.open.