W3C

Disposition of comments for the Efficient Extensible Interchange Working Group

Single page view

In the table below, red is in the WG decision column indicates that the Working Group didn't agree with the comment, green indicates that a it agreed with it, and yellow reflects an in-between situation.

In the "Commentor reply" column, red indicates the commenter objected to the WG resolution, green indicates approval, and yellow means the commenter didn't respond to the request for feedback.

CommentorCommentWorking Group decisionCommentor reply
LC-2709 Rumen Kyusakov <rumen.kyusakov@ltu.se> (archived comment)
Subject: Limiting the evolutions of build-in grammars
Section: 2.1 Grammar Learning Disabling Mechanism
Paragraph: "If grammar learning is disabled in the case of a production insertion in a given
grammar, named G, the following rule happens ..."
Comments:
This is the first time the second mechanism (Limiting the evolutions of build-in grammars) is
mentioned in the spec. It is almost impossible to make the connection with the rest of the spec
without reading it few/several times. In my opinion this mechanism should be clearly described and
differentiated with the first one (using xsi:type).
A proper way of doing this is by mentioning it in the 2. Grammar Capping sections and then
describing it as a separate subsection. Giving it a proper name would also help a lot as "grammar
learning is disabled in the case of a production insertion" is definitely not a good one.
On 2013-02-13, the WG members (including Rumen Kyusakov) agreed to
clarify the specification to address the issue. See [4].

-------------------------------------------------------------------------------

On 2013-02-12, Rumen Kyusakov reflected further and responded in [3]:

Please consider adding the following paragraph to the "2. Grammar
Capping" section:

The arbitrary memory growth at runtime when processing EXI streams is
due to indexing QNames and string values for frequent use (string tables
growth) and adapting the processing strategy to unknown input in case of
schema-less encoding or deviations from the schema (build-in element
grammar generation and evolution). The EXI Profile provides mechanisms
to restricts the growth of the string tables and the grammar evolution
in order to predictably limit the memory growth at runtime. Limiting the
size of the string tables and the grammar evolution bounds the memory
growth but also increases the size of the event codes required for
encoding unknown input such as schema deviations hence leading to less
compact EXI streams.

Some mechanisms to lower the memory usage are already available in the
EXI 1.0 specification and should be considered in conjunction with the
EXI profile (see B Guidelines (Non-Normative)). For example the EXI
specification provides a mechanism to limit the memory growth due to
global value string table by using the valuePartitionCapacity. The local
value string table can only be disabled if the global value string table
is also disabled by setting the valuePartitionCapacity to 0. The Local
Value Capping mechanism described below allows for disabling the local
value table even if valuePartitionCapacity is greater than 0.

According to the EXI 1.0, an EXI processor creates a new build-in
element grammar for each unknown element it encounters. The EXI profile
limits the number of build-in element grammars created at runtime by
using schema informed complex ur-type grammars instead. Each build-in
element grammar that is allowed to be created by the EXI profile
processor can further evolve by insertion of new productions and hence
leads to memory growth during runtime. For that reason, the EXI profile
defines a mechanism to limit the number of dynamically inserted grammar
productions in the build-in element grammars if such are allowed during
processing.

The disabling of the local value table, the number of build-in element
grammars created at runtime and the number of dynamically inserted
grammar productions are all controlled by a set of Profile parameters
described in "4. Parameters representation" section.

-------------------------------------------------------------------------------

On 2012-11-12, Rumen Kyusakov responded in [2]:

I still find the text hard to read and confusing although the changes
you suggest are helping a lot.
I don't understand this part of your response:

> The second means, which is less efficient, should be used whenever the
> first mechanism is not available, namely:
>
> · For AT(xsi:type) attributes
>
> · For elements being processed using built-in grammars and
> within which memory cap is reached

Isn't it so that AT(xsi:type) means is not available only in case of
schema-less EXI streams? Maybe I'm missing something.

I'm also thinking isn't it better to explicitly specify when the second
means (using only second level productions) SHOULD be used - using
formal SHOULD and not the advisory form "should"?

One more comment: I somehow had the feeling that the second means is
also applicable to the Built-in Fragment Grammar
(http://www.w3.org/TR/2011/REC-exi-20110310/#builtinFragGrammars).

Reading your comments I realized that the EXI Profile does not say
anything about the limiting the evolution of Built-in Fragment Grammar.
The same principle can be applied there as well but if decided not to
maybe it is better to explicitly say that the learning mechanisms of the
Built-in Fragment Grammars are not disabled.

-------------------------------------------------------------------------------

On 2012-11-12, the EXI WG responded in [1]:

Here are the two proposed changes, all in section 2:

1. Add the following sentence at the end of the first paragraph:

Whenever using the xsi:type attribute switch is not feasible, EXI events
of a given element may be represented using only second level productions.

Given that this approach is generally less efficient, the xsi:type
attribute switch should be used wherever possible.

2. Change:

“Note that the EXI profile can only limit grammar for schema-informed
EXI streams but…”

To:

“Note that xsi:type attributes may be used to limit grammar learning
only for schema-informed EXI streams and…”


[1] http://lists.w3.org/Archives/Public/public-exi-comments/2012Nov/0002.html
[2] http://lists.w3.org/Archives/Public/public-exi-comments/2012Nov/0007.html
[3] https://lists.w3.org/Archives/Member/member-exi-wg/2013Feb/0013.html
[4] https://www.w3.org/2013/02/13-exi-minutes.html#item04
yes
LC-2711 Rumen Kyusakov <rumen.kyusakov@ltu.se> (archived comment)
Subject: The Profile parameters
Section: 4. Parameters representation
Paragraph: "In such a case, the actual profile parameters (or fine-grained capping strategies)
should be defined by an out-of-bound mechanism."
Comments:
The maximumNumberOfBuiltInElementGrammars and localValuePartitions parameters should not be
mandatory to be set/communicated-out-of-bound in my opinion. For the number of build in grammar
the encoder knows the number of the grammars so no need to be included in the header. On the
decoder side if the implementation is "smart" it won't create a new build-in element grammar
before examining the next event. If it is a xsi:type="SOME_EXISTING_TYPE", there is no need to
create a new build-in element grammar.
For the localValuePartitions again - no issues for the encoder if the parameter is not included in
the header. The decoder on the other hand can create local value tables only if it has enough
memory to do so. If it cannot support local value tables, it will only expect string values
represented literally in the stream. Receiving a string as a compact identifier will produce error
during decoding in any way if the decoder does not have enough memory.
The EXI WG agrees to the suggestion lessening the compliance requirements
of the explicit/implicit parameters communication from "should" to "may".

We are going to make this change understanding that while there are
cases that find the sharing of the parameters useful, there are some
others that do not need to be informed with the parameters values.
yes
LC-2706 Rumen Kyusakov <rumen.kyusakov@ltu.se> (archived comment)
Subject: Evaluations of EXI in application areas with memory restrictions
Section: 1. Introduction
Paragraph: "Certain evaluations of EXI in the context of such areas..."
Comments:
It is good to have access to these evaluations. From my own perspective and use cases this profile
will not bring much benefits. I am working with very small messages over low-bandwidth wireless
links (up to 250 kb/s). Of course, memory is an issue but the compression is even more important.
It would be good to have some statistics on what are the trade-offs (memory usage vs compression)
by using the Profile in different use-cases (size and structure of the messages).
As alluded in my previous emails, the next updated version of the profile will introduce a "Guidelines" section. In regard to restricted devices, this new section is going to give advice for sensible default options for both, the EXI profile and the EXI1.0 baseline spec. However, we do not plan to prohibit users experimenting with the various options to best fit their needs.
http://lists.w3.org/Archives/Public/public-exi-comments/2012Dec/0000.html
tocheck
LC-2707 Rumen Kyusakov <rumen.kyusakov@ltu.se> (archived comment)
Subject: Limiting the number of build-in grammars and Limiting the evolutions of build-in grammars
Section: 2. Grammar Capping
Paragraph: Whole section
Comments:
In the grammar capping section it is given a single mechanism to limit the number of build-in
grammars by using the xsi:type attribute (possibly with xsd:anyType). There is no mentioning of
the second very distinct mechanism described later on: limiting the evolutions of build-in
grammars by restricting the number of dynamic productions inserted. Mixing up between these two
mechanisms in the spec is the main source of confusion for me.
On 2013-02-13, the WG members (including Rumen Kyusakov) agreed to
clarify the specification to address the issue. See [4].

-------------------------------------------------------------------------------

On 2013-02-12, Rumen Kyusakov reflected further and responded in [3]:

Please consider adding the following paragraph to the "2. Grammar
Capping" section:

The arbitrary memory growth at runtime when processing EXI streams is
due to indexing QNames and string values for frequent use (string tables
growth) and adapting the processing strategy to unknown input in case of
schema-less encoding or deviations from the schema (build-in element
grammar generation and evolution). The EXI Profile provides mechanisms
to restricts the growth of the string tables and the grammar evolution
in order to predictably limit the memory growth at runtime. Limiting the
size of the string tables and the grammar evolution bounds the memory
growth but also increases the size of the event codes required for
encoding unknown input such as schema deviations hence leading to less
compact EXI streams.

Some mechanisms to lower the memory usage are already available in the
EXI 1.0 specification and should be considered in conjunction with the
EXI profile (see B Guidelines (Non-Normative)). For example the EXI
specification provides a mechanism to limit the memory growth due to
global value string table by using the valuePartitionCapacity. The local
value string table can only be disabled if the global value string table
is also disabled by setting the valuePartitionCapacity to 0. The Local
Value Capping mechanism described below allows for disabling the local
value table even if valuePartitionCapacity is greater than 0.

According to the EXI 1.0, an EXI processor creates a new build-in
element grammar for each unknown element it encounters. The EXI profile
limits the number of build-in element grammars created at runtime by
using schema informed complex ur-type grammars instead. Each build-in
element grammar that is allowed to be created by the EXI profile
processor can further evolve by insertion of new productions and hence
leads to memory growth during runtime. For that reason, the EXI profile
defines a mechanism to limit the number of dynamically inserted grammar
productions in the build-in element grammars if such are allowed during
processing.

The disabling of the local value table, the number of build-in element
grammars created at runtime and the number of dynamically inserted
grammar productions are all controlled by a set of Profile parameters
described in "4. Parameters representation" section.

-------------------------------------------------------------------------------

On 2012-11-12, Rumen Kyusakov responded in [2]:

I still find the text hard to read and confusing although the changes
you suggest are helping a lot.
I don't understand this part of your response:

> The second means, which is less efficient, should be used whenever the
> first mechanism is not available, namely:
>
> · For AT(xsi:type) attributes
>
> · For elements being processed using built-in grammars and
> within which memory cap is reached

Isn't it so that AT(xsi:type) means is not available only in case of
schema-less EXI streams? Maybe I'm missing something.

I'm also thinking isn't it better to explicitly specify when the second
means (using only second level productions) SHOULD be used - using
formal SHOULD and not the advisory form "should"?

One more comment: I somehow had the feeling that the second means is
also applicable to the Built-in Fragment Grammar
(http://www.w3.org/TR/2011/REC-exi-20110310/#builtinFragGrammars).

Reading your comments I realized that the EXI Profile does not say
anything about the limiting the evolution of Built-in Fragment Grammar.
The same principle can be applied there as well but if decided not to
maybe it is better to explicitly say that the learning mechanisms of the
Built-in Fragment Grammars are not disabled.

-------------------------------------------------------------------------------

On 2012-11-12, the EXI WG responded in [1]:

Here are the two proposed changes, all in section 2:

1. Add the following sentence at the end of the first paragraph:

Whenever using the xsi:type attribute switch is not feasible, EXI events
of a given element may be represented using only second level productions.

Given that this approach is generally less efficient, the xsi:type
attribute switch should be used wherever possible.

2. Change:

“Note that the EXI profile can only limit grammar for schema-informed
EXI streams but…”

To:

“Note that xsi:type attributes may be used to limit grammar learning
only for schema-informed EXI streams and…”


[1] http://lists.w3.org/Archives/Public/public-exi-comments/2012Nov/0002.html
[2] http://lists.w3.org/Archives/Public/public-exi-comments/2012Nov/0007.html
[3] https://lists.w3.org/Archives/Member/member-exi-wg/2013Feb/0013.html
[4] https://www.w3.org/2013/02/13-exi-minutes.html#item04
yes
LC-2708 Rumen Kyusakov <rumen.kyusakov@ltu.se> (archived comment)
Subject: Event code length for xsi:type attribute event
Section: 2.1 Grammar Learning Disabling Mechanism
Paragraph: "The xsi:type attribute event MUST always be represented by the AT(*) production whose
event code length is 2."
Comments:
I am sure there is a good reason to require this but I cannot deduct it. Maybe a hint in the spec
will make the job of the implementers easier. My guess is that this is connected to the separation
of the "original" xsi:type attributes and the xsi:type attributes added because of the Profile.

(Note also: "For a given element E, the disabling of grammar learning E ..."
should be "For a given element E, the disabling of grammar learning for E ...";
"..., the following rule happens:" it is better "..., the following rules apply:")
http://lists.w3.org/Archives/Public/public-exi-comments/2013Jan/0000.html tocheck
LC-2710 Rumen Kyusakov <rumen.kyusakov@ltu.se> (archived comment)
Subject: Confusing description
Section: 2.2 Grammar Learning Disabling Parameters
Paragraph: "Grammar learning is disabled...
...
Grammar learning is disabled in the case of a production ..."
Comments:
There should really be a better name for "Grammar learning is disabled in the case of a
production ..." A suggestion: "Disabling the evolution of build-in grammars"?
http://lists.w3.org/Archives/Public/public-exi-comments/2012Nov/0003.html tocheck
LC-2712 Rumen Kyusakov <rumen.kyusakov@ltu.se> (archived comment)
Subject: Unclear
Section: C Prefix Workarounds (Non-Normative)
Paragraph: All
Comments:
All the paragraph could be summarize by saying that the application should use only ONE prefix per
namespace. Then the two requirements are fulfilled.
> All the paragraph could be summarize by saying that the application should
> use only ONE prefix per namespace. Then the two requirements are fulfilled.

Your statement is actually not entirely equivalent to the statement in the
current specification.

Let’s take the following example:

<A>
<B xmlns:ns1=”uri1”/>
<B xmlns:ns1=”uri1”/>
</A>

The following document would match your statement but not the one in the specification.

In particular, the second prefix would be encoded as an index.

A decoder may not be actually able to give to the XML layer the knowledge for that second namespace declaration.
In addition, this requires two namespace declarations, which can be optimized in terms of compression as follow:

<A xmlns:ns1=”uri1”>
<B/>
<B/>
</A>

That is why we prefer to keep the more restricted statement in the specification.
yes
LC-2713 Rumen Kyusakov <rumen.kyusakov@ltu.se> (archived comment)
Subject: Striping of Profile xsi:type attributes
Section: E.2 Grammar Restriction Decoder Considerations
Paragraph: "An EXI profile decoder SHOULD strip any xsi:type attribute with the xsd:anyType value
from the infoset that corresponds to the grammar learning disabling mechanism."
Comments:
It is strange to have "SHOULD" in a (Non-Normative) section. In any case a non-Profile EXI decoder
will include the Profile xsi:type attributes.
> Subject: Striping of Profile xsi:type attributes
> Section: E.2 Grammar Restriction Decoder Considerations
> Paragraph: "An EXI profile decoder SHOULD strip any xsi:type attribute with
> the xsd:anyType value from the infoset that corresponds to the grammar
> learning disabling mechanism."
> Comments:
> It is strange to have "SHOULD" in a (Non-Normative) section. In any case
> a non-Profile EXI decoder will include the Profile xsi:type attributes.
>

Your comment is correct.
We therefore propose to replace uppercase SHOULD by lowercase may.
yes
LC-2714 Rumen Kyusakov <rumen.kyusakov@ltu.se> (archived comment)
Subject: Summary
Section: All spec
Paragraph: All
Comments:
The conformance with the Profile does not guarantee in any way a bounded memory for processing
(for example not setting valuePartitionCapacity to 0 is enough to make a Profile complying
implementation consuming huge amount of memory). Moreover a non-conforming to the Profile
implementation can still use the Limiting the number of build-in grammars mechanism without
declaring that in the EXI header.
Without being an expert in the field my personal experience is that the EXI spec and even more the
EXI Profile spec are very much requiring the users to be aware of some implementation issues in
order to use the format in an optimal way.
For example, I was involved to some small extend in the Smart Energy Profile 2.0 standard/draft
that considers using EXI. Main goal is compactness but also lean processing. The knowledge on what
is the optimal way to set all the parameters (schemaId, preserve, selfContained, valueMaxLength,
valuePartitionCapacity and now the Profile parameters) is simply not there.
The great flexibility provided by having so many things to tune is working against the wide spread
adoption. The users should not be burdened by knowing what all these parameters mean especially
the Profile ones that are purely connected to the implementation.
Again, without claiming to be an expert and have a wide range of use cases in mind, I think a more
restrictive Profile that actually provides some guarantees and not so much freedom of setting
implementation parameters will be much more useful. In any case the actual use of the EXI Profile
will require some "Application-specific Profile" that defines the fixed values for all these
parameters. So why not preset all these parameters and provide some guarantees with a good memory
usage vs compression trade-off? I know one size does not fit all, but then why defining an EXI
Profile in the first place?
In any case, if every single application sets these parameters in a different way then the
implementations won't be able to interoperate due to some memory issues or size constrains. What I
see as a need is a Profile that provides some guarantees for resource constrained devices. For
example, a conforming implementations should use only:
- maximumNumberOfBuiltInElementGrammars = 0
- valuePartitionCapacity = 0
- maximumNumberOfBuiltInProductions = 0
- localValuePartitions = 0
- selfContained = false
- preserve = all false
- fragment = false
- either strict schema mode OR "schemaId" set to the empty value schema-less mode

Note that by fixing the values of these parameters will not only make the RAM consumption
predictive but also lower the programming memory which is also an issue in some cases. The
implementations of the Profile will be much simpler as a codebase as compared to implementing the
generic version of the Profiles as it is now. In fact, the Profile as it is now will just extend
the code and make it more complex which I believe is the opposite of what is needed for embedded
applications.
As alluded in my previous emails, the next updated version of the profile will introduce a "Guidelines" section. In regard to restricted devices, this new section is going to give advice for sensible default options for both, the EXI profile and the EXI1.0 baseline spec. However, we do not plan to prohibit users experimenting with the various options to best fit their needs.
http://lists.w3.org/Archives/Public/public-exi-comments/2012Dec/0000.html
tocheck
LC-2715 Rumen Kyusakov <rumen.kyusakov@ltu.se> (archived comment)
Dear all,

When reading up on the EXI specification section "7.3.3 Partitions
Optimized for Frequent use of String Literals" last two paragraphs
regarding the processing of value content items, I have one more issue
to rise connected to the EXI Profile:
In section "3. Local Value Capping" the parameter localValuePartitions
is used to turn on or off the use of local value partitions. Note
however that this same effect can be achieved by the use of the EXI
parameter valuePartitionCapacity - it is not only used for the global
value indexing.
Setting valuePartitionCapacity to 0 effectively turns off both local and
global value string tables.
So the localValuePartitions parameter of the EXI profile has effect only
when valuePartitionCapacity is greater than 0.
However, the use of global value tables (valuePartitionCapacity > 0)
while turning off only the local value tables (localValuePartitions = 0)
is something that is highly unlikely to be useful in any scenario.

So why having yet another parameter for something that is already
available as option in the EXI specification?
As alluded in my previous emails, the next updated version of the profile will introduce a "Guidelines" section. In regard to restricted devices, this new section is going to give advice for sensible default options for both, the EXI profile and the EXI1.0 baseline spec. However, we do not plan to prohibit users experimenting with the various options to best fit their needs.
http://lists.w3.org/Archives/Public/public-exi-comments/2012Dec/0000.html
tocheck
LC-2705 Yusuke DOI <yusuke.doi@toshiba.co.jp>
The other problem for us is to have option grammar parser. I have a 1kB grammar of full option grammar (without capability to parse user-defined metadata) and even want to reduce it. But, these option values useful for embedded devices are placed in 'lesscommon' or user defined metadata (otherwise, we can omit 'lesscommon' grammar -- more than half of the option grammar). In SEP2, as you know, we have decided to have static no-choice options :) and SEP2 nodes can skip parsing the option document (something like bitcmp(expected, given) works), and I would do the same way in other embedded applications.
A new appendix section, "C.2 Implementation Strategies" will be added that sketches different strategies regarding an EXI header options parser:
* bit matching
* partial EXI Options grammar parser
* mix
* full
tocheck

Developed and maintained by Dominique Hazaël-Massieux (dom@w3.org).
$Id: index.html,v 1.1 2017/08/11 06:44:17 dom Exp $
Please send bug reports and request for enhancements to w3t-sys.org