Duplicate Entries

Description

The EXI Working Group is currently defining an EXI profile for processors that have constrained memory capacities. For that purpose, the EXI Working Group is contemplating relaxing the rules defined by the EXI 1.0 specification as an errata item. The envisioned change would extend the set of valid EXI 1.0 streams.

The envisioned change can be roughly summarized as follow:

• An EXI name table may include duplicate entries (i.e. twice the same URL, or the same local name in a partition)
• A qname is equal with another qname if both their URI indexes and their local name indexes match
• It is not recommended to duplicate entries

Duplicate entries should only be used in specific conditions such as closed environments or through dedicated frameworks like the EXI profile. The EXI profile would use duplicated entries to cap the memory usage of EXI processors.

Implementation Impacts

Some decoders may retrieve structures (partitions, grammars...) by using entry indexes while some other decoders may retrieve structures based on entry string values.

• As long as there are no duplicate entries, both types of decoder behave the same.
• When duplicate entries happen, they do not behave the same, leading to potential interoperability issues.

The following subsections enumerate identified potential issues.

Name tables

• Some decoders may create a new prefix and/or local-name partition for each URI entry.
• Some decoders may reuse the same prefix or local-name partition for duplicated entries of the same URI string.

Built-in grammars

• Some decoders may decide to create a new grammar for each new URI+local-name entry
• Some decoders will reuse the same grammar for duplicated entries of the same URI+local-name strings

Channel ordering

The errata item may or may not forbid duplicate entries for precompression and/or compression mode. This potential issue may become a non-issue.

• Some decoders may create a new channel for each new local name entry
• Some decoders may use the same channel for all values related to all duplicated entries of the same URI + local name strings

Global attributes

The errata item may or may not forbid duplicate entries of pre-populated entries. This potential issue may become a non-issue.

• Some decoders may detect global attributes based on predefined URI+local-name entries
• Some decoders may detect global attributes based on URI+local-name strings

Note: xsi:type and xsi:nil attributes are also in the scope of this issue.

Implementation Strategies

What needs to be checked within existing implementations

It needs to be identified whether EXI structures using qualified names (such as name entries, grammars, etc.) are compared based on

• textual representation of URI and localName --> may cause issues for duplicate entries
• comparison is done using EXIs Uri and LocalName IDs --> OK in any case

Note: textual comparisons are not sufficient given that two or more duplicate entries may dispose of the same string representation but a different Uri/LocalName ID combination.

How to make implementations aware of duplicate entries

A very simple and easy strategy to make a code aware of duplicate entries is as follows:

• Revise every textual qname comparison with the Uri/LocalName ID comparison.

Note: A more optimized implementation strategy (in terms of speed & retrieval of structures) is based on indexes and will be compliant with the proposed change also.

Implementation Status

This section describes whether existing implementations are compatible with the change or not.

Canon

• OK for all streams

EXIP

• OK for 5 streams
• KO for precompression stream (this mode may not be supported by the codec)

EXIficient

• Current release 0.8: OK for name tables only
• Upcoming release 0.9: OK for all streams

OpenEXI

• Of the 6 files provided for experiment, only the first two exi-stream_1.exi and exi-stream_2.exi are successfully decoded by OpenEXI.
• OpenEXI is observed to permit duplicate string entries in name-related partitions.
• The proposed extension to EXI 1.0 is not compatible with how OpenEXI implemented EXI 1.0 specification. The development of EXI 1.0 is complete, and is in maintenance phase.
• OpenEXI will need to undergo substantive code changes and testing in order to change its behaviour so as to be consistent with the contemplated changes.
• OpenEXI, which conforms to EXI 1.0 specification, has no plan to renovate the implementation to accomodate the additional requirements proposed retroactively at this point, given that the nature of the changes are trying to extend the format instead of fixing errors inherent to the published EXI 1.0 REC.