Since the containers are very different, it should be a separate entry in the registry with its own initDataType. We may also want to update the name of the existing one to reference BMFF.

Bob, are you referring to https://www.iso.org/obp/ui/#iso:std:iso-iec:23001:-9:ed-1:v1:en? (Note that it is not freely available.)

Is the Initialization Data format exactly the same? That is, does the file contain concatenated PSSH boxes identical to the BMFF case, including the BMFF box header and Full box header?

(In reply to David Dorwin from comment #2)
> Bob, are you referring to
> https://www.iso.org/obp/ui/#iso:std:iso-iec:23001:-9:ed-1:v1:en? (Note that
> it is not freely available.)
> 
> Is the Initialization Data format exactly the same? That is, does the file
> contain concatenated PSSH boxes identical to the BMFF case, including the
> BMFF box header and Full box header?

David, the CENC MPEG-2 TS carries concatenated PSSH boxes identical to
the BMFF case.  23001-9 section 5.2.1 says:

5.2.1    General

A CETS PSSH packet carries the complete payload of a `pssh` box, as
defined in ISO/IEC 23001-7. Each packet uses private syntax and carries a
`pssh` box along with an MD5 hash for integrity.

(In reply to Bob Lund from comment #3)
> David, the CENC MPEG-2 TS carries concatenated PSSH boxes identical to
> the BMFF case.  23001-9 section 5.2.1 says:
> 
> 5.2.1    General
> 
> A CETS PSSH packet carries the complete payload of a `pssh` box, as
> defined in ISO/IEC 23001-7. Each packet uses private syntax and carries a
> `pssh` box along with an MD5 hash for integrity.

Thanks. Questions about the above text:
* "payload of a... box" seems a bit ambiguous as to whether it is the entire box or something within it.
* The spec only supports one PSSH box per packet? Can there be multiple CETS PSSH packets to support multiple System IDs?
* Would the `pssh` box be stripped from the packet and provided in the "encrypted" event? (Answers to the above questions may add complexity here.)
* Why does the packet have a private syntax? How is the user agent supposed to know how to parse it?

(In reply to David Dorwin from comment #4)
> (In reply to Bob Lund from comment #3)
> > David, the CENC MPEG-2 TS carries concatenated PSSH boxes identical to
> > the BMFF case.  23001-9 section 5.2.1 says:
> > 
> > 5.2.1    General
> > 
> > A CETS PSSH packet carries the complete payload of a `pssh` box, as
> > defined in ISO/IEC 23001-7. Each packet uses private syntax and carries a
> > `pssh` box along with an MD5 hash for integrity.
> 
> Thanks. Questions about the above text:
> * "payload of a... box" seems a bit ambiguous as to whether it is the entire
> box or something within it.

It is the entire contents of the PSSH. The 'cets_pssh_packet' contains a 'pssh_box()' field defined in  23001-9 section 5.2.3 as "pssh_box: complete `pssh` box, as defined in ISO/IEC 23001-7".

> * The spec only supports one PSSH box per packet? Can there be multiple CETS
> PSSH packets to support multiple System IDs?

Yes, PSSH for each system is carried in a separate PID.

> * Would the `pssh` box be stripped from the packet and provided in the
> "encrypted" event? (Answers to the above questions may add complexity here.)

The UA would provide the contents of the 'cets_pssh_packet.pssh_box()' field, i.e. the exact same PSSH box contents , as in the CENC ISO BMFF case.

> * Why does the packet have a private syntax?

The packet syntax is not "private". It is defined in ISO/IEC 23001-9 which is available under exactly the same terms as CENC ISO BMFF ISO/IEV 23001-7. 

> How is the user agent supposed to know how to parse it?

By purchasing the spec. Here is the link: http://www.iso.org/iso/catalogue_detail.htm?csnumber=63440

(In reply to Bob Lund from comment #5)
> (In reply to David Dorwin from comment #4)
> > (In reply to Bob Lund from comment #3)
> > > David, the CENC MPEG-2 TS carries concatenated PSSH boxes identical to
> > > the BMFF case.  23001-9 section 5.2.1 says:
> > > 
> > > 5.2.1    General
> > > 
> > > A CETS PSSH packet carries the complete payload of a `pssh` box, as
> > > defined in ISO/IEC 23001-7. Each packet uses private syntax and carries a
> > > `pssh` box along with an MD5 hash for integrity.
> > 
> > Thanks. Questions about the above text:
> > * "payload of a... box" seems a bit ambiguous as to whether it is the entire
> > box or something within it.
> 
> It is the entire contents of the PSSH. The 'cets_pssh_packet' contains a
> 'pssh_box()' field defined in  23001-9 section 5.2.3 as "pssh_box: complete
> `pssh` box, as defined in ISO/IEC 23001-7".
> 
> > * The spec only supports one PSSH box per packet? Can there be multiple CETS
> > PSSH packets to support multiple System IDs?
> 
> Yes, PSSH for each system is carried in a separate PID.

How are the PIDs determined? If the PIDs are different for different System IDs, how does the user agent know which PIDs contain PSSH boxes?

> > * Would the `pssh` box be stripped from the packet and provided in the
> > "encrypted" event? (Answers to the above questions may add complexity here.)
> 
> The UA would provide the contents of the 'cets_pssh_packet.pssh_box()'
> field, i.e. the exact same PSSH box contents , as in the CENC ISO BMFF case.

In the ISO BMFF case, the PSSH boxes for all System IDs are concatenated in the container and can easily be extracted and used in a single "encrypted" event.

For MPEG-2 TS, the user agent would need to collect the PSSH boxes from each CETS PSSH packet, determine that there are no more CETS PSSH packets for this location, and concatenate the PSSH boxes into a single "encrypted" event. Is it possible to collect PSSH boxes and determine when there is a "complete set" as I have described?

> > * Why does the packet have a private syntax?
> 
> The packet syntax is not "private". It is defined in ISO/IEC 23001-9 which
> is available under exactly the same terms as CENC ISO BMFF ISO/IEV 23001-7. 

I was referring to "Each packet uses private syntax" from the quoted text in 5.2.1 (comment #3). Why does 23001-9 say the syntax is "private" if it defines it?

> > How is the user agent supposed to know how to parse it?
> 
> By purchasing the spec. Here is the link:
> http://www.iso.org/iso/catalogue_detail.htm?csnumber=63440

I was referring to a "private syntax", which I interpreted as not being defined in the spec. See above.

(In reply to David Dorwin from comment #6)
> (In reply to Bob Lund from comment #5)
> > (In reply to David Dorwin from comment #4)
> > > (In reply to Bob Lund from comment #3)
> > > > David, the CENC MPEG-2 TS carries concatenated PSSH boxes identical to
> > > > the BMFF case.  23001-9 section 5.2.1 says:
> > > > 
> > > > 5.2.1    General
> > > > 
> > > > A CETS PSSH packet carries the complete payload of a `pssh` box, as
> > > > defined in ISO/IEC 23001-7. Each packet uses private syntax and carries a
> > > > `pssh` box along with an MD5 hash for integrity.
> > > 
> > > Thanks. Questions about the above text:
> > > * "payload of a... box" seems a bit ambiguous as to whether it is the entire
> > > box or something within it.
> > 
> > It is the entire contents of the PSSH. The 'cets_pssh_packet' contains a
> > 'pssh_box()' field defined in  23001-9 section 5.2.3 as "pssh_box: complete
> > `pssh` box, as defined in ISO/IEC 23001-7".
> > 
> > > * The spec only supports one PSSH box per packet? Can there be multiple CETS
> > > PSSH packets to support multiple System IDs?
> > 
> > Yes, PSSH for each system is carried in a separate PID.
> 
> How are the PIDs determined? If the PIDs are different for different System
> IDs, how does the user agent know which PIDs contain PSSH boxes?
> 

The transport stream contains a 'CA_Descriptor' (section 5.3) where 'num_systems' defines the number of system ID's,'encryption_algorithm' which specifies the encryption algorithm, same as `tenc`.IsEncrypted. There are 'num_systems' entries in 'encryption_algorithm'. Each entry contains the fields: 'system_id': same as `pssh`.SystemID and 'pssh_pid': PID on which `pssh` box(es) can be found for this content protection system

> > > * Would the `pssh` box be stripped from the packet and provided in the
> > > "encrypted" event? (Answers to the above questions may add complexity here.)
> > 
> > The UA would provide the contents of the 'cets_pssh_packet.pssh_box()'
> > field, i.e. the exact same PSSH box contents , as in the CENC ISO BMFF case.
> 
> In the ISO BMFF case, the PSSH boxes for all System IDs are concatenated in
> the container and can easily be extracted and used in a single "encrypted"
> event.
>
> For MPEG-2 TS, the user agent would need to collect the PSSH boxes from each
> CETS PSSH packet, determine that there are no more CETS PSSH packets for
> this location, and concatenate the PSSH boxes into a single "encrypted"
> event. Is it possible to collect PSSH boxes and determine when there is a
> "complete set" as I have described?

Yes. 'CA_Descriptor' contains all System IDs and associated PIDs which identify where to look for PSSH to concatenate in the "encrypted" event.

> 
> > > * Why does the packet have a private syntax?
> > 
> > The packet syntax is not "private". It is defined in ISO/IEC 23001-9 which
> > is available under exactly the same terms as CENC ISO BMFF ISO/IEV 23001-7. 
> 
> I was referring to "Each packet uses private syntax" from the quoted text in
> 5.2.1 (comment #3). Why does 23001-9 say the syntax is "private" if it
> defines it?

Ah, I wasn't clear on your question. MPEG-2 TS defines 'private' to mean anything not defined by the MPEG-2 specs, but defined elsewhere. In this case the cets_pssh_packet() uses private syntax defined within 23001-9. The MPEG-2 system spec defines a couple of ways private data can be carried in a transport stream. It is not completely clear in 23001-9 what mechanism the author intended. I will find that out and suggest that 23001-9 be clarified. Nonetheless, the outcome of that doesn't change that the 'cets_pssh_packet()' would be processed by the UA according to the syntax in 5.2.1.

> 
> > > How is the user agent supposed to know how to parse it?
> > 
> > By purchasing the spec. Here is the link:
> > http://www.iso.org/iso/catalogue_detail.htm?csnumber=63440
> 
> I was referring to a "private syntax", which I interpreted as not being
> defined in the spec. See above.

Thanks, Bob.

My two takeaways are:
1) The Initialization Data format is identical as fars as the application is concerned ("encrypted" event and generateRequest()).
2) There are a lot of details related to detecting decryption and extracting the Initialization Data from the MPEG-2 TS that need to be documented.

The combination in comment #8 highlights another issue - the Stream Format and Initialization Data Format Registry unnecessarily binds container parsing with Initialization Data formats. In reality:
* The "cenc" Initialization Data Type can be present in multiple containers.
* The "keyids" Initialization Data Type is container-independent and has no stream format sections.
* The "webm" Initialization Data Type is still only used by WebM but is not specific to WebM. It could just as easily be "keyid".

Now that the Initialization Data format is no longer determined based on the container type, we should consider separating the container details from the Initialization Data types/formats. The container details could refer to the Initialization Data Type that they use.

Note that this means that simply checking whether the initDataType is supported is no longer sufficient to determine whether the user agent can play a stream - it only indicates whether the CDM can generate a request based on that type.

Specifically, any instances of
  isTypeSupported("com.example.somesystem", "cenc")
need to be replaced with
  isTypeSupported("com.example.somesystem", "cenc", "video/mp4")

This was already the case for "keyids", and providing the container and codecs should be a best practice anyway.

(In reply to Bob Lund from comment #7)
> (In reply to David Dorwin from comment #6)
> > (In reply to Bob Lund from comment #5)
> > > (In reply to David Dorwin from comment #4)
> > > > (In reply to Bob Lund from comment #3)
> > > > > David, the CENC MPEG-2 TS carries concatenated PSSH boxes identical to
> > > > > the BMFF case.  23001-9 section 5.2.1 says:
> > > > > 
> > > > > 5.2.1    General
> > > > > 
> > > > > A CETS PSSH packet carries the complete payload of a `pssh` box, as
> > > > > defined in ISO/IEC 23001-7. Each packet uses private syntax and carries a
> > > > > `pssh` box along with an MD5 hash for integrity.
> > > > 
> > > > Thanks. Questions about the above text:
> > > > * "payload of a... box" seems a bit ambiguous as to whether it is the entire
> > > > box or something within it.
> > > 
> > > It is the entire contents of the PSSH. The 'cets_pssh_packet' contains a
> > > 'pssh_box()' field defined in  23001-9 section 5.2.3 as "pssh_box: complete
> > > `pssh` box, as defined in ISO/IEC 23001-7".
> > > 
> > > > * The spec only supports one PSSH box per packet? Can there be multiple CETS
> > > > PSSH packets to support multiple System IDs?
> > > 
> > > Yes, PSSH for each system is carried in a separate PID.
> > 
> > How are the PIDs determined? If the PIDs are different for different System
> > IDs, how does the user agent know which PIDs contain PSSH boxes?
> > 
> 
> The transport stream contains a 'CA_Descriptor' (section 5.3) where
> 'num_systems' defines the number of system ID's,'encryption_algorithm' which
> specifies the encryption algorithm, same as `tenc`.IsEncrypted. There are
> 'num_systems' entries in 'encryption_algorithm'. Each entry contains the
> fields: 'system_id': same as `pssh`.SystemID and 'pssh_pid': PID on which
> `pssh` box(es) can be found for this content protection system
> 
> > > > * Would the `pssh` box be stripped from the packet and provided in the
> > > > "encrypted" event? (Answers to the above questions may add complexity here.)
> > > 
> > > The UA would provide the contents of the 'cets_pssh_packet.pssh_box()'
> > > field, i.e. the exact same PSSH box contents , as in the CENC ISO BMFF case.
> > 
> > In the ISO BMFF case, the PSSH boxes for all System IDs are concatenated in
> > the container and can easily be extracted and used in a single "encrypted"
> > event.
> >
> > For MPEG-2 TS, the user agent would need to collect the PSSH boxes from each
> > CETS PSSH packet, determine that there are no more CETS PSSH packets for
> > this location, and concatenate the PSSH boxes into a single "encrypted"
> > event. Is it possible to collect PSSH boxes and determine when there is a
> > "complete set" as I have described?
> 
> Yes. 'CA_Descriptor' contains all System IDs and associated PIDs which
> identify where to look for PSSH to concatenate in the "encrypted" event.
> 
> > 
> > > > * Why does the packet have a private syntax?
> > > 
> > > The packet syntax is not "private". It is defined in ISO/IEC 23001-9 which
> > > is available under exactly the same terms as CENC ISO BMFF ISO/IEV 23001-7. 
> > 
> > I was referring to "Each packet uses private syntax" from the quoted text in
> > 5.2.1 (comment #3). Why does 23001-9 say the syntax is "private" if it
> > defines it?
> 
> Ah, I wasn't clear on your question. MPEG-2 TS defines 'private' to mean
> anything not defined by the MPEG-2 specs, but defined elsewhere. In this
> case the cets_pssh_packet() uses private syntax defined within 23001-9. The
> MPEG-2 system spec defines a couple of ways private data can be carried in a
> transport stream. It is not completely clear in 23001-9 what mechanism the
> author intended. I will find that out and suggest that 23001-9 be clarified.


Here is clarification from the author of 23001-9 regarding how the 'cets_pssh_packet()' is carried in the MPEG-2 transport stream.

The carriage is using TS payloads (i.e., it is neither PES nor tables). The parsing process is:

1) Collect packets from the first packet with PUSI=1 till (not including) next packet with PUSI=1.

2) Concatenate the payload.

3) Result is 32 bits ('md5_flag' and 'reserved') followed by complete `pssh` box (including all fields inherited from Box), optionally followed by 128-bit MD5.


> Nonetheless, the outcome of that doesn't change that the
> 'cets_pssh_packet()' would be processed by the UA according to the syntax in
> 5.2.1.
> 
> > 
> > > > How is the user agent supposed to know how to parse it?
> > > 
> > > By purchasing the spec. Here is the link:
> > > http://www.iso.org/iso/catalogue_detail.htm?csnumber=63440
> > 
> > I was referring to a "private syntax", which I interpreted as not being
> > defined in the spec. See above.

As ddorwin mentions, MPEG2-TS is not royalty-free. IANAL, and certainly not an expert on patents, but how do we square this with the W3C Patent Policy?

Updating the summary to reflect the work this bug is tracking. The "cenc" Initialization Data Format should be extracted (bug 26811) and referenced by the new entry.

(In reply to John Luther from comment #11)
> As ddorwin mentions, MPEG2-TS is not royalty-free. IANAL, and certainly not
> an expert on patents, but how do we square this with the W3C Patent Policy?

From http://lists.w3.org/Archives/Public/public-html-media/2014Sep/0011.html:
"Status: Paul is discussing the Patent Policy question with the W3c Team"

In terms of W3C Patent Policy, the FAQ says the following:
[[
32. Can a W3C Recommendation normatively refer to technology developed outside W3C with licensing terms that differ from those of the W3C Patent Policy?

Yes. W3C Recommendations may include normative references to standards or technologies developed outside of W3C. However, the Working Group should keep in mind the importance of royalty-free implementations of Web standards. In the event it becomes clear that the licensing status of those externally-developed technologies could become a barrier to implementation of the technology according to the W3C Royalty-Free (RF) Licensing Requirements, W3C may choose not to publish the document or may launch a PAG.
]]
http://www.w3.org/2003/12/22-pp-faq#outside-normative-ref

So, the question is: is it a barrier to EME implementations according to W3C RF requirements?

Philippe

(In reply to Philippe Le Hegaret from comment #14)
> In terms of W3C Patent Policy, the FAQ says the following:
> [[
> 32. Can a W3C Recommendation normatively refer to technology developed
> outside W3C with licensing terms that differ from those of the W3C Patent
> Policy?
> 
> Yes. W3C Recommendations may include normative references to standards or
> technologies developed outside of W3C. However, the Working Group should
> keep in mind the importance of royalty-free implementations of Web
> standards. In the event it becomes clear that the licensing status of those
> externally-developed technologies could become a barrier to implementation
> of the technology according to the W3C Royalty-Free (RF) Licensing
> Requirements, W3C may choose not to publish the document or may launch a PAG.
> ]]
> http://www.w3.org/2003/12/22-pp-faq#outside-normative-ref
> 
> So, the question is: is it a barrier to EME implementations according to W3C
> RF requirements?

If an EME implementation is not *required* to support all formats in the registry to be spec compliant, then I suppose not. 

I don't believe supporting all formats is required, but can someone verify here in writing?


> 
> Philippe

(In reply to John Luther from comment #15)
> If an EME implementation is not *required* to support all formats in the
> registry to be spec compliant, then I suppose not. 
> 
> I don't believe supporting all formats is required, but can someone verify
> here in writing?

The whole point of creating the registry was to move these details out of the normative spec to avoid this problem. Each format is optional to implement. The registry says "*IF* you support this format then you should support it in this way". It doesn't say you MUST support a format.

Then I believe we're fine since the spec doesn't make it mandatory to support the format.

(In reply to Adrian Bateman [MSFT] from comment #16)
> (In reply to John Luther from comment #15)
> > If an EME implementation is not *required* to support all formats in the
> > registry to be spec compliant, then I suppose not. 
> > 
> > I don't believe supporting all formats is required, but can someone verify
> > here in writing?
> 
> The whole point of creating the registry was to move these details out of
> the normative spec to avoid this problem. Each format is optional to
> implement. The registry says "*IF* you support this format then you should
> support it in this way". It doesn't say you MUST support a format.

OK, thanks, Adrian.

As discussed in the telecon today, someone familiar with this spec will need to draft proposed (ReSpec) text based on the discussion above. It might make sense to wait for bug 26811 to be implemented. Or you can just assume that the "cenc" type is defined separately and can be referenced.

cenc-format-respec.html could be used as a base. Make sure you use the original ReSpec source and not the ReSpec output. See https://dvcs.w3.org/hg/html-media/ for hg clone instructions.

(In reply to David Dorwin from comment #19)
> As discussed in the telecon today, someone familiar with this spec will need
> to draft proposed (ReSpec) text based on the discussion above.

I can do this.

> It might make
> sense to wait for bug 26811 to be implemented.

When might this happen? I'd like see the new cenc type spec as well as the reformatted ISOBMFF container spec that references it.

> Or you can just assume that
> the "cenc" type is defined separately and can be referenced.
> 
> cenc-format-respec.html could be used as a base. Make sure you use the
> original ReSpec source and not the ReSpec output. See
> https://dvcs.w3.org/hg/html-media/ for hg clone instructions.

(In reply to David Dorwin from comment #19)
> As discussed in the telecon today, someone familiar with this spec will need
> to draft proposed (ReSpec) text based on the discussion above. It might make
> sense to wait for bug 26811 to be implemented. Or you can just assume that
> the "cenc" type is defined separately and can be referenced.
> 
> cenc-format-respec.html could be used as a base. Make sure you use the
> original ReSpec source and not the ReSpec output. See
> https://dvcs.w3.org/hg/html-media/ for hg clone instructions.

I've created an cenc mpeg2ts stream format spec [1]. This is referenced by the cenc init data spec [2] I created as a proposed resolution to bug 26811.

[1] http://rawgit.com/boblund/encrypted-media/bug26738/mpeg2ts-format-respec.html
[2] http://rawgit.com/boblund/encrypted-media/bug26811/cenc-init-format-respec.html

Migrated to https://github.com/w3c/encrypted-media/issues/106.