XML Binary Characterization Use Cases

3.1.1 Description

The constant progress of digital TV, the multiplication of channels, the competition and convergence with the Web, and the widespread deployment of a variety of set-top boxes (notably Personal Video Recorders, or PVRs for short) call for services on TV sets that extend beyond simply broadcasting audio/video content.

For instance, above a certain number of channels, broadcasters find themselves having to provide EPG (Electronic Program Guide) services to their users, without which they would be overwhelmed with the sheer amount of available content. These EPGs also allow PVRs to automatically pick up recording schedules for given programs, based on user-defined criteria that match against metadata broadcasted alongside the data.

Similarly, broadcasters are also trying to make their offer more attractive by integrating TV with Web technologies or the Web at large. This includes notably using Web Services from PVRs that benefit from a return channel, using Web UI technologies such as SVG, XHTML, or XForms to define their applications' interfaces, making TV services available to mobile devices, and so forth.

However, there are constraints that cause problems when trying to deploy such services, all of which rely on XML, to television sets:

• Bandwidth: TV bandwidth is extremely expensive, and how much data you use for services directly constrains the number of channels that you are able to send. In addition to the potential technical issues, there are strong economic motivations to reduce bandwidth usage as much as possible.

• Processing Power: Most set-top boxes are inexpensive, and the low-end ones have roughly half the power (if not less) of a low-end mobile device. Contrary to mobile devices, there are few limitations as to the processing power that can be embarked in a box the size of the average set-top box (STB), notably the problems relating to heat and battery life are of little or no concern. However, on the one hand large-scale deployment of STBs and similar devices into households requires them to be extremely inexpensive and therefore as limited as possible, and on the other hand convergence with mobile devices remains a prime motivator for the television industry and constraints applicable to mobile devices apply equally to broadcasted XML metadata.

• Unidirectional Network: This being broadcast, there is not typically a way for TVs to request data. Instead, it is being continuously streamed and re-streamed to them, a process which is called carouselling (the data itself being 'on a carousel'). Some set-top boxes do in fact have a return channel (notably the ones that support Web Services) but the vast majority doesn't. If the data were sent as an XML document, it would have to be fragmented so that STBs wouldn't have to wait for the end of the entire document to have been carouselled in order to start exploiting the data. XML has not shown to be easily fragmentable and currently the carouselling relies on the ability of the binary formats to be fragmented.

• Change Resilience. Upgrading several million STBs is often very impractical. Therefore, it must be possible to evolve the broadcast format without breaking older hardware. While XML is perfectly suited to this, the above issues make it unusable. It is thus required that the binary XML format replacing it be resilient to changes in the schema.

As a result, MPEG-7 BiM (a binary encoding of XML originally created to carry video metadata), has been integrated into a number of broadcasting standards, notably ARIB, DVB, and TV Anytime.

3.1.2 Domain & Stakeholders

This use case is relevant to the entirety of the television distribution industry, comprising content providers, broadcast infrastructure deployers, television and set-top box manufacturers, and of course the broadcasting companies themselves.

It also covers similar requirements that can be found in digital radio broadcasting, where one equally needs to broadcast EPG metadata to very limited devices, to integrate with mobile devices (for instance by sending SVG ads as part of the radio stream).

And finally, convergence with TV is considered to be a major next step in mobile services, and all participants on both sides of the fence are presently being extremely active in making television available anywhere, at any time, and on any device. Quite naturally, this leads to the need for common technology between the TV industry and the rest of the Web. As such, the major stakeholders in the domain have expressed interest in reusing a solution commonly agreed upon across several industries.

3.1.3 Justification

Television is a very large market that has a strong need for program metadata, and is increasingly converging with the Web at large (with a strong emphasis on mobile devices at first), notably using technologies such as XHTML, SVG, XForms, SMIL, and Web Services.

Deployed systems already use binary XML, currently standardized as part of ISO MPEG and industry fora such as ARIB, DVB, or TV-Anytime, but have expressed interest in using more broadly adopted technologies.

3.1.4 Analysis

XML is appropriate for these situations because:

1. existing specifications based on XML are being reused wholesale;

2. most major TV standards in the area are already XML-based, and the industry has no wish to go through another standards cycle;

3. XML is well-suited to describing structured information such as metadata;

4. XML has proven to be a good format to specify user interfaces in, using notably XHTML, SVG, or XForms. These are needed for TV applications, and they need to be broadcasted;

5. the industry wishes to publish its data, especially the Electronic Program Guides, to as many media as possible. XML enables it to publish directly to desktops, mobiles, and TVs using off-the-shelf or Open Source software.

3.1.5 Properties

XML serializations in this domain require Robustness in that a television set cannot go into error if a fragment it receives is malformed: it simply has to throw it away and wait to see if the next turn of the carousel will bring a well-formed version. Thus, it is essential that Robustness is supported at the fragment level so that the entirety of the stream is not corrupted by a single problem.

The low-end class of target devices has constraints that make Processing Speed and Small Footprint essential properties, however these systems are asymmetric and, in the typical case, unidirectional. Therefore, these constraints apply exclusively to the decoder.

Encoded data is not expected to be Self Contained, but rather the decoder can be required to have access to a schema definition of some form (typically, a kind of compiled schema is used). This is further underlined by the fact that the requirement for Compactness is so essential that support for Specialized Codecs becomes highly desirable.

Updating set-top boxes and related devices is inconvenient at best, when at all possible. However, services need to evolve over time, entailing evolution in the XML vocabulary. It is thus essential for the format to support Schema Instance Change Resilience for most changes.

Television systems worldwide use different technologies, and it is therefore essential for the format to support Transport Independence.

A Streamable format is essential for the decoder given the way in which fragments are carouselled (decoders can even start reading in mid-stream). Also, STBs will typically know which fragments on the carousel they have already processed in order not to have to process them again and will only process uncached ones. Thus, support for deltas and Accelerated Sequential Access are also essential properties.

Several of the properties listed above indirectly require a Fragmentable format, therefore this property is deemed essential as well.

3.1.6 Alternatives

DVB EIT schedules provide a small subset of the required functionality and do not integrate well with a more generic information ecosystem.

Without a W3C format of binary XML, application domains will likely adopt different formats--a phenomenon that already started in the Broadcast domain. This would further cluster the market. A convergence (which can be recognized right now) between the broadcast and the mobile communication services would be hindered by this clustering. In this concrete case, mobile devices would be required to implement codecs of both domains to enable value added services like interactive and location aware TV broadcast. This drives up the initial investment, which translate into a great obstacle for a converged service in the marketplace.

3.1.7 References

1. [DVB]

2. [TV Anytime]

3. [ARIB]

4. [MPEG-7]

3.2.1 Description

The upstream segment of the energy industry is concerned with exploration for and production of oil and gas. XML-based techniques have made very little penetration into the upstream technology part of the energy industry. The most basic reason for this is the nature of the data, which does not at this time lend itself to being represented usefully in XML.

There are basically two core types of data in this industry: well logs and seismic data. Well logs are moderately large datasets while seismic datasets are real large, typically in the order of gigabytes. Although the Petrotechnical Open Standards Consortium [POSC] has produced an XML schema for well logs [WellLogML], it has not been adopted by the industry. At the time of writing, nobody has even considered defining a schema for seismic data.

Both seismic and well log data include control data, easily represented in XML, as well as large arrays of floating point numbers, not easily represented efficiently in XML. Although in practice an XML representation is not used, such data may be represented as shown in the following fragment (with a whole document consisting of a large number of these fragments):

    <seisdata>
      <lineheader>...<lineheader> 
      <header>
        <linename>westcam 2811</linename>
        <ntrace>1207</ntrace>
        <nsamp>3001</nsamp>
        <tstart>0.0</tstart>
        <tinc>4.0</tinc>
        <shot>120</shot>
        <geophone>7</geophone>
      </header>
      <trace>0.0 0.0 468.34 3.245672E04 6.9762345E05 ... (3001 floats)</trace>
      <header> 
        ... 
      </header>
      <trace> ... </trace>
      ...
    </seisdata>

3.2.2 Domain

The scope within the Energy Industry as discussed above is very broad, encompassing a very large number of technical issues and usage scenarios involving, for example, integration of drilling information, processing of seismic and well data, integration of seismic and well data into interpretation systems, and so on.

3.2.3 Justification

There are a number of dominant technology vendors in this sector as well as a number of small companies that "work around the edges". The dominant technology vendors (which are none of the technology giants) provide proprietary solutions that do not interoperate easily with each other. Providing communications between these products within a company, or between companies, is a constant problem: this is the main motivator to develop Web service interfaces for these products. A second motivator for a standard is that it will open the door for smaller companies to provide useful add-on products. Large budgets in this sector are allocated to the purchase of software packages and display devices, but these budgets are small compared to the leverage of mass-market devices, so a longer term objective is to encourage a situation were more technologies with mass-market cost leverage can be used.

3.2.4 Analysis

Given that this scenario involves interoperability between companies using disparate systems, XML is a natural choice due to its ubiquity and tool availability.

The main shortcoming of XML for this application is the expense incurred while converting floating point data to and from a character representation, as well as the extra size of some of these representations. Thus, the main requirement for this use case is the ability to represent sequences of floating point numbers in a binary format (as close to the native representation as possible), in order to facilitate efficient binding into programmatic objects (primarily, floating point arrays). In the example shown above, the header information would still have a textual representation (useful for any infoset-based processing), but the trace of floating point numbers would appear as an opaque binary stream.

The format chosen to represent a floating point number must be platform independent, with tools supporting conversions to and from the appropriate native format. In practice, most operations involve moving data between machines with the same floating point formats, so the solution should not impose undue overhead on the most common situation in order to handle the less common ones.

3.2.5 Properties

The properties Compactness and Processing Speed are essential for this use case. Documents containing large arrays of floating points must be serialized in a compact format. It is expected that most of the compression is achieved by encoding floating point numbers in a binary format instead of characters. Further, parsing (extracting) out the floating point arrays and binding them to programmatic objects should be efficient.

The properties Robustness, Schema Instance Change Resilience and Data Model Versatility are desirable for this use case. If errors are encountered while transferring large documents in alternative serialization, it is often better to continue with the processing (as best as possible) instead of forcing a complete retransmission. The encoding of floating point arrays will most likely rely on schema information. Hence, resilience to compatible changes to the schema is desired. Support for multiple data models, especially those that are able to represent arrays of floating points will improve the adoption of the format.

3.2.6 Alternatives

• Data Compression: One expert in this area has said, "For us, binary compression is probably not that important because transmission speeds are constantly improving. The additional time needed to compress and decompress seismic data would probably slow things down. We also place a greater value in the message structures than the transmission mechanics". Or, in more picturesque words, again from an expert in the field when asked about compressing seismic data, "Been there, done that, doesn't work, not interested". Bear in mind that this epigram encapsulates decades of experience and highly sophisticated R&D.

• CORBA: There is, in fact, a CORBA-based integration platform currently deployed (although perhaps not widely) in this space. Without diving into technical details, it is clear that some companies would prefer an approach based on Web services.

• XML Protocol Attachments: It is possible to represent seismic data control information in XML and to put the floating point arrays in a binary attachment using XOP. This data architecture is certainly viable, assuming that the issues involving floating point numbers are addressed, as evidenced by the fact that many of the proprietary vendor data formats work this way. It is, however, less flexible than the header-trace architecture described above, which is probably one reason why the latter is used in industry-wide seismic data standards (e.g. SEGY). Nonetheless, Web services that return data using XOP are an attractive alternative for dealing with seismic data.

3.2.7 References

1. [POSC]

2. [WellLogML]

3. [SEGY]

3.3.1 Description

Extensible 3D (X3D) Graphics [Extensible 3D (X3D) Graphics] is an XML-enabled ISO Standard 3D file format to enable real-time communication of 3D data across all applications and networks. It is used for commercial applications, engineering and scientific visualization, medical imaging, training, modeling and simulation (M&S), multimedia, entertainment, education, and more. [X3D Markets] Computer-Aided Design (CAD) and architecture scenes are also supported, though because they have larger file sizes and are more complex they are not typically streamed for web-delivered viewing.

Web-delivered file sizes in this use case typically range from 1-1000 KB of data while CAD files may run to several hundred megabytes apiece. An optimized serialization of the X3D data may be performed in concert with application-specific compression (e.g. combining coplanar polygons, quantizing colors, etc.) Lossy geometric compression is sometimes acceptable. Due to interaction requirements, the latency time associated with deserialization, decompression and parsing must be minimal. Digital signature and encryption compatibilities are also important for protecting digital content assets.

3.3.2 Domain & Stakeholders

Support of Web-based interchange, rendering and interactivity for 3D graphics scenes. Stakeholders include tool builders, content authors, application developers and end users of 3D graphics models.

3.3.3 Justification

The X3D Compressed Binary Encoding Request For Proposals (RFP) from the X3D Consortium [X3D RFP] lists and justifies ten separate technical requirements. Many of these have parallels to a general optimized serialization format. The Web3D Consortium and X3D designers see great value in aligning with W3C standardization efforts in this area. This serves as significant evidence of the need for an industry standard.

3.3.4 Analysis

Taken together, the following technical requirements for the X3D Compressed Binary Encoding RFP (indicated by emphasized type) include many requirements for an optimized serialization format. Strictly speaking, such an efficient serialization is not necessarily "compressed", though it is very likely to be more compact. Other factors, such as speed of parsing or databinding performance, may override the desire for compact representation for some applications, such as CAD. The X3D Consortium's RFP has chosen to include the ability to perform application-specific size optimizations, both lossless and lossy, as part of the process of converting the document from XML to the optimized serialization. However, the optimized format can still be represented as XML; in other words, a lossy geometric compression to an efficient format can be translated back to a lossy XML representation (i.e., Transcodable to XML).

• X3D Compatibility: The compressed binary encoding shall be able to encode all of the abstract functionality described in X3D Abstract Specification. Since X3D is expressed in XML, any optimized serialization that encodes all XML features will also be capable of encoding an X3D document.

• Interoperability: The compressed binary encoding shall contain identical information to the other X3D encodings (XML and Classic VRML). It shall support an identical round-trip conversion between the X3D encodings. This corresponds to the Round Trippable property.

• Multiple, separable data types: The compressed binary encoding shall support multiple, separable media data types, including all node (element) and field (attribute) types in X3D. The RFC allows the possibility of performing domain-specific compressions or encodings of data in the XML document, for example of polygons and textures. The ability to make use of Specialized Codecs is essential to meeting this requirement.

• Processing Performance: The compressed binary encoding shall be easy and efficient to process in a runtime environment. Because the data for interactive applications is delivered across the web low latency is important. The ability to quickly process documents is important. In the case of CAD files, which may be several hundred megabytes in size, the ability to quickly process the file is very important. Often 3D files have long arrays of numeric data. Using XML format requires that a reader extract the string information and convert it to a binary representation. This is a computationally expensive process. Experimental data has shown that an optimized format can be 20 times as fast as XML. The ability to rapidly process arrays of floating point data is also important. Thus the Accelerated Sequential Access property is also relevant to this use case.

• Ease of Implementation: Binary compression algorithms shall be easy to implement. This corresponds to Implementation Cost.

• Streaming: Compressed binary encoding will operate in a variety of network-streaming environments. X3D documents are often streamed in web environments, and portions of the 3D scene are rendered as they arrive on the client computer. This corresponds to the Streamable property.

• Compression: Compressed binary encoding algorithms will together enable effective compression of diverse datatypes. 3D data often consists of large arrays of floating point data that can be compressed in various ways. The ability to employ Specialized Codecs, either lossless or lossy, would meet this requirement.

• Security: Compressed binary encoding will optionally enable security, content protection, privacy preferences and metadata such as encryption, conditional access, and watermarking. This corresponds to the Signable property.

• Bundling: Mechanisms for bundling multiple files (e.g. X3D scene, inlined sub-scenes, image files, audio file, etc.) into a single archive file will be considered. This corresponds to Embedding Support.

• Intellectual Property Rights (IPR): Technology submissions must meet the Web3D Consortium IPR policy. The W3C Patent Policy [W3C PP] is compatible with this.

3.3.5 Properties

A summary of the relevant properties is given below:

• Accelerated Sequential Access
• Compactness
• Embedding Support
• Encryptable
• Processing Speed
• Round Trippable
• Signable
• Specialized Codecs
• Streamable
• Transcodable to XML
• Implementation Cost

3.3.6 Alternatives

GZIP is the specified compression scheme for Virtual Reality Modeling Language (VRML 97) specification, the second-generation ISO predecessor to X3D. GZIP is not type-aware and does not compress large sets of floating-point numbers as well. GZIP allows staged decompression of 64KB blocks, which might be used to support streaming capabilities. GZIP outputs are strings and require a second pass for any parsing, thus degrading parsing and loading performance. A GZIPed file would not gain the parsing speed and databinding advantages of an optimized format.

Numerous piecemeal, incompatible proprietary solutions exist in the 3D graphics industry for Web-page plug-ins. None address the breadth of technical capabilities that might be enabled by a general purpose optimized serialization format.

An X3D-specific compression and serialization algorithm for XML is certainly feasible and demonstrated. Compatibility with a general recommendation for an optimized format is desirable in order to maximize interoperability with other XML technologies, and reduce implementation cost. Many of these issues are common to other use-case domains; broad mutual benefits become possible via a common recommendation.

3.3.7 References

1. [Extensible 3D (X3D) Graphics]

2. [X3D Markets]

3. [X3D RFP]

3.4.1 Description

As Web services become more and more ubiquitous, there is a greater demand to use this technology as a way to deliver content to small devices such PDAs, pagers and mobile phones. All these devices often share the following characteristics:

1. They have limited memory and limited processing power.

2. Battery life is at a premium.

3. They are connected to low-bandwidth, high-latency networks which in some cases are regulated by "pay-per-byte" policies.

XML-based messaging is at the heart of the current Web services technology. XML's self-describing nature has significant advantages, but they come at the price of bandwidth and performance. XML-based messages are larger and require more processing than other protocols, and are therefore not well suited for a domain having the characteristics outlined above. Increased bandwidth usage affects wireless networks due to bandwidth restrictions allotted for communication by each device. In addition, the larger the message the higher the probability of a retransmission as a result of an on-the-air collision.

The target platforms for this use case include a broad range of PDAs, handhelds and mobile handsets, including mass market devices that limit code size to 64K and heap size to 230K. The transport packet size may vary from network to network, but it is typically measured in bytes (e.g. 128 bytes).

3.4.2 Domain

Small devices connected to low-bandwidth, high-latency networks. Two examples in this domain are cellular phone networks and PDA networks employed by the military.

3.4.3 Justification

XML is the fundamental technology underlying a Web services infrastructure, and one of the main reasons why Web services are not being deployed on the mobile space. A number of alternative serializations have already been developed to deliver XML content to small devices, however, many of these are not interoperable. This lack of interoperability results in fragmentation and the need for specialized gateways to transcode proprietary formats.

3.4.4 Analysis

In order to satisfy the requirements of this use case, an alternative serialization must be faster to process and must produce smaller packets. Faster processing will result in lesser battery consumption while smaller packets will result in reduced latency as well as, assuming a pay-per-byte model, a more cost-effective service. In addition to small and fast, an alternative serialization should also be streamable, i.e. it should be possible for the client application to operate on any prefix of the serialized data.

Assuming that the same amount of information is encoded in an alternative serialization, a way to quantify efficiency is to consider the instruction to data ratio. In other words, the amount of effort that is needed to produce or consume a unit of data. Even though this is an implementation requirement, an alternative serialization must enable the creation of "thin" stacks with a low instruction to data ratio.

The reduction in latency that results by improving parsing speed may or may not be noticeable to the consumer depending on the transport latency of the network --transport latency is the dominant factor in many existing networks. Nevertheless, a more efficient parsing method will improve battery life on the device as well as throughput on the server.

3.4.5 Properties

The properties Processing Speed and Compactness are essential for this use case. Given the technological limitations on existing wireless networks for small devices, it is imperative to minimize the number of bytes sent on each transmission; a compact message will also increase the battery life of a device by reducing the usage time of the transmitter. Low parsing and serialization times of the underlying format result in an improved user experience by reducing response time and conserving battery life. In some circumstances, it is essential to employ a Streamable format to achieve these results.

Support for the Signable and Encryptable properties is also desirable, yet not essential since transport-level security (e.g., HTTPS) is often available and intermediaries are typically not deployed. A fundamental additional consideration for this use case is that the format must enable the implementation of Small Footprint software stacks.

3.4.6 Alternatives

Proprietary solutions result in the so-called gatewayed networks, where communication is always routed through a single point that translates to and from XML. This architecture not only creates a single point of failure within a network but also fragments the entire network by creating non-interoperable, domain-specific solutions.

Message size reductions are attainable via the use of standard data compression techniques. Even though in general decompression is less expensive than compression, it is still too costly for most small devices. Additionally, the extra burden of compressing packets has a negative impact on the overall system throughput.

In addition to the added cost, redundancy-based compression algorithms tend to perform very poorly on small messages, in many cases resulting in larger messages. Mobile clients often carry on dialogs with servers which consist of a large number of small messages. Examples of this include: data synchronization, stateful web services, multi-player games, querying and browsing data. In all of these use cases, the cumulative stream of messages that make up the dialog can grow very large even though all of the individual messages are rather small. Thus, there is still a need to reduce the amount of data exchanged, but doing this by compressing each message individually is not a viable solution.

3.4.7 References

1. [Comparing SOAP Performance]

2. [SOAP Performance in Wireless Networks]

3. [Fast Web Services]

4. [SOAP Performance]

3.5.1 Description

A large number of existing enterprise systems are built using distributed technologies such as RMI, DCOM and CORBA. As the industry moves from distributed object systems to Service Oriented Architectures (SOAs), the use of Web services technologies becomes more significant even within the confines of a single enterprise. Many of the concepts behind SOAs are applicable to divisions within an corporation, so it is only natural to extend the applicability of Web services to intranet systems.

A stumbling block that several re-architected systems are facing is that XML-based messages are larger and require more processing than those from existing protocols: data is represented inefficiently and binding requires more computation. It has been shown that an RMI service can perform up to an order of magnitude faster than an equivalent Web Service due to the processing required to parse and bind XML data into programmatic objects.

3.5.2 Domain

The domain is that of distributed systems, typically based on binary protocols, which for technical reasons (e.g., interoperability) or for economic reasons (e.g., reduction of software licenses) need to be re-architected as Web services. An important constraint for these type of re-deployments is to maintain (or improve) the system's performance, a task that has been found challenging given the additional processing requirements of XML-based protocols.

3.5.3 Justification

There are some important economic reasons that support the use of Web services as an alternative to existing technologies for building distributed systems. First, preliminary results show more powerful hardware is needed to re-deploy existing systems using Web services technologies given the additional processing requirements of an XML messaging system. Second, assuming the company in question already develops (or is planning to develop) Web services to communicate outside their firewall, their is the extra incentive in using the same set of tools and the same development team to build intranet applications. This reduces both software fees (e.g., by reusing application servers and development tools) as well as training costs associated with having separate development teams for each technology. Third, some companies that have successfully deployed CORBA-based systems, but are not planning on deploying Web services, may find an additional incentive to do so if a more efficient serialization is standardized.

3.5.4 Analysis

Intranet Web services differ from Internet Web services especially in the areas of deployment and security: deployments are easier to manage and security is typically defined by a single domain. The requirements for intranet systems are somewhat different from those for Internet systems, permitting the use of certain optimizations in the former which would be difficult or simply impossible to implement in the latter. Consequently, in many cases the degree of coupledness of the systems can be adjusted if this helps achieving the desired performance goals.

The main requirement for this use case is reducing XML processing time in order to achieve a level of performance comparable to the existing systems. Due to the availability of high-speed networks in these scenarios, reducing message sizes is of a lesser priority. It is worth pointing out that not all systems re-deployed using Web services will be unable to achieve their performance requirements. Therefore, this use case applies only to a subset of the aforementioned re-deployments.

3.5.5 Properties

The property Processing Speed is essential for this use case. In addition to parsing speed, data binding into programming language objects is an important aspect for this use cases since most Web services toolkits favor this form of processing. Compactness is also desirable, especially in those cases in which it enables faster processing.

Given the importance of confidentiality, authentication and integrity in a Web services scenario, the properties Signable and Encryptable are also deemed as essential. Self Contained and Schema Instance Change Resilience can be classified as essential if intermediaries are deployed as part of the system. If the are no intermediaries, or if intermediaries can be easily updated as schemas evolve, these two properties can be considered less essential or simply desirable. Other desirable properties are: Embedding Support.

3.5.6 Alternatives

In some cases, it may be possible to re-design the system's interfaces to make them more coarse grained in order to reduce the number of messages exchanged. Although this is technically feasible in most cases, the costs associated with this effort can be prohibiting.

3.5.7 References

1. [Fast Web Services]

2. [SOAP Performance]

3.6.1 Description

Although the data in an XML document is encoded as text, it is often the case that portions of that text are in fact embedded documents in and of themselves. This frequently occurs when XML documents contain images, recordings, or other multimedia elements which have their own file formats --JPEG, MP3, and so on. In order to embed these documents they are translated to a textual representation using base64-encoding or a similar scheme. It is worth noting that these embedded documents are often much larger than the encapsulating document.

For document-oriented applications, these embedded files are often part of the document, in the sense that they are intended to be rendered along with the text in the encapsulating XML document. Thus, the translation from text back to the original file format often occurs as the rest of the document is being parsed. The case in which the embedded document is in fact an XML document (e.g., an SVG graphic embedded within an XSL-FO document) can be regarded as a special case in which no translation is required.

For message-oriented applications, embedded documents are simply payload elements encapsulated within, e.g., a SOAP body element. In these cases, the payloads may be of arbitrary or even unknown formats. They are often translated to text during transmission and back to their original form upon reception, even if not otherwise immediately consumed, in order to reduce storage space. Because these embedded documents can be large, storage requirements are further reduced by streaming on input and output. In these cases the embedded document may also be XML but may contain either processing instructions or DTDs, both of which should not appear within a SOAP body element (WS-I Basic Profile, R1008 and R1009). Therefore, such files may be treated as if they were binary data and base64-encoded even if they are, in fact, valid XML files.

3.6.2 Domain

This use case considers the domains of both electronic documents and Web services.

3.6.3 Justification

XML was not designed to contain binary data, and other packaging mechanisms such as MIME, exist and are in many ways suitable to the purpose at hand. On the other hand, the ability to treat embedded documents as part of the primary documents, and therefore make them accessible to XML-based standards and tools like XPath without resorting to additional standards like MIME, is useful in practice. Thus, this use case is a good demonstration of why one might wish to extend XML with a binary encoding.

3.6.4 Analysis

This use case builds on applications of XML for documents and Web services that are already well established. Furthermore, those uses already involve the transmission of documents with binary data like integers and floats. The question is, for each given embeddable datum, should it be placed inside the document or should it be carried as an attachment?

The drawbacks to embedding are the penalties in time due to the translation into text, and in space, due to the larger size of the translated data. The benefits include access to other XML-based technologies like XPath, XQuery, etc. and the avoidance of an additional dependency on a packaging technology such as MIME.

3.6.5 Properties

TBD

3.6.6 Alternatives

SOAP with Attachments provides a MIME-based mechanism for packaging binary data with SOAP messages. It avoids the translation costs, but does not make the binary data part of the XML document itself. In that respect, it is not a streamable format.

XOP describes how a MIME-based package can be used to encapsulate the binary data without a translation overhead by keeping it (at least conceptually) as part of the encapsulating XML document. Because of its use of MIME, this approach suffers from many of the same shortcomings of the SOAP with attachments case.

3.6.7 References

1. [MTOM]

2. [XOP]

3. [SOAP with Attachments]

4. [WS-I Attachments Profile]

5. [MIME]

3.7.1 Description

Documents are the most basic form of recorded human communication, dating back thousands of years. Electronic documents are the transition of this invention to the online, computerized world. Books, forms, contracts, e-mails, spreadsheets, and Web pages are only some of the forms in which electronic documents are used. Unlike paper-based documents, electronic documents are not limited to static text and images. Electronic documents regularly contain both static content, dynamic content (e.g., animations, video), and interactive content (e.g., form fields). This wide range of content has a great affect on selecting an appropriate representation format and must be considered in evaluating this use case.

Documents are first created in some authoring environment. During the creation process the author may elect to include text, fonts, image, videos, or other resources which are to be rendered more than once when the document is displayed. For example, a company logo may appear in the header of each page of a document, but this should not require adding the logo to the document more than once.

In a special case of document creation, new documents are created by assembling a set of existing documents into a single aggregate document. For example, this may done to combine a basic product manual with additional documentation for optional product accessories into a customized manual for an individual purchaser. When documents are bound together in this way it may be important that the data in the original documents is not modified, so as to preserve signatures or other properties of the file, or it may be desirable to identify and eliminate duplicated resources, such as fonts.

After a document has been created it is usually read, in whole or in part. Documents are not necessarily read front to back; a particular reader may select a different order or read only part of a document. A reader may, for example, obtain the document by traversing a hyperlink which points to a specific location within the document. It is important that rendering a document for reading be fast, even when starting at an arbitrary location in this way, and even when documents are large (millions of pages). This implies that it must be possible to navigate to specific sections within a document quickly, as well as follow links to shared resources within the document, as mentioned above under document creation. Finally, if a document is being retrieved over a slow link, it may be useful to fetch portions of the document in the order in which they are being rendered and read (e.g., starting at page 700), as opposed to document order (i.e., starting at page 1).

Documents often contain information of a sensitive or proprietary nature and so can be secured using encryption technologies. Encrypting the document can serve either to keep the contents confidential, to--in conjunction with the rendering application--allow only certain operations ("rights") on the document, or both. Typically a description of any rights granted is embedded within the document itself when it is encrypted. It is often desirable that only portions of a document be encrypted so that intermediaries can access some portion of the data in the file.

Documents, and especially those used in business transactions, are often signed to indicate authenticity of, consent to, or agreement with the document. In electronic documents, this is implemented by digitally signing the document. The digital signature must itself be stored in the document. Multiple signatures may be applied to a document, each one signing those which came before it. Additional information is sometimes added to a document after it has been signed but without invalidating a signature--in the same way one can initial a correction to a paper document--but so that it is clear that any subsequent changes were not present when the pre-existing signatures were applied. In some cases signatures should apply to only part of a document, leaving other parts for later modification. Finally, it must be possible for a recipient to validate all of these signatures.

Documents are often long-lived and, during the course of their lives, used in different environments with varying constraints. For example, when a document is being published for general consumption, it might be most desirable to select an encoding such as XML which is widely understood. If, however, the same document is being transmitted between partners with known expectations a more compact format such as [XOP] might be preferred. Thus, a single document may sometimes be transformed between different encodings at different times and for different purposes. Such transformations should preserve the information in the document, but these operations cannot be expected to be compatible with encryption mechanisms used to secure documents.

Even when various encodings are available documents tend to push the available storage and bandwidth of the devices on which they are created, stored, transmitted, and read. In other words, as device capabilities increase, users respond by creating larger documents. Note that these documents rarely contain only text; they generally contain larger elements such as fonts and images and, increasingly, video and 3D models which these same enhanced devices make possible.

Electronic documents, like their paper counterparts, can be modified or re-purposed. In electronic documents, this typically occurs when pages, images, videos, and so forth are either copied out of a document to be used elsewhere or removed from a document to produce an altered version of that document. Again, these operations should be efficient: removing any one page from a one million page document should not take significantly longer than doing the same to a ten page document. Documents may also be modified by their recipients to include comments of various types--editors' marks, sticky notes, etc.--usually intended to communicate responses back to the author. These comments may be stored within the document itself; both adding them to and extracting them from the document should be efficient.

Finally, some documents are designed to be interactive beyond the limited interactions of rendering, signing, and annotating. These documents may contain form fields, GUI widgets such as buttons and listboxes, or other active elements, data islands bound to these widgets, and code, scripts, or declarative logic to validate input to these elements, enable or disable the elements, transmit the document, modify the document, interact with the rendering application, and so forth. It must be possible to describe and access all of these elements within the document itself.

3.7.2 Domain & Stakeholders

Electronic documents are used extensively throughout government, business, and personal domains as well as in the interchange between these entities.

3.7.3 Justification

XML is in its roots a syntax for marking documents, and so the electronic document use cases seem highly relevant. Interestingly, XML has a number of shortcomings (discussed below) with respect to many of the requirements derived from this use case. Arguably, these occur because XML (and SGML) were focused largely on textual documents, but such documents represent a decreasing fraction of all electronic documents. Thus, Binary XML as a natural extension of XML to handle new document types, and documents containing new content, seems particularly relevant.

3.7.4 Analysis

Documents are almost always exchanged between two or more people, and often between larger entities such as corporations or governments. It is, therefore, extremely desirable that an electronic document format should be easily consumable by all parties involved. XML, as a widely accepted, implemented, and used format, fits this need quite well.

Unfortunately there a number of requirements imposed by electronic documents which XML fails to address:

• Documents frequently contain embedded resources such as fonts, images, and video which are themselves encoded in binary formats. It must be possible to efficiently embed these resources in documents.

• It must be possible to navigate to and render a specified location in better than linear time with respect to the size of the document.

• The document encoding must be efficient with respect to space, that is, it must have low redundancy.

• The document encoding must be efficient with respect to space, that is, it must have low entropy.

• In order to make updates efficient, it must be possible to update a document in time proportional to the size of the update rather than the size of the document.

There are a number of requirements which XML does address, but which are enumerated here as well because they would also be requirements on any Binary XML encoding:

• The format should be widely accepted, available, and implemented.

• Re-usable resources may appear, or be referenced from, multiple locations within the document. In order to maintain reasonable document sizes, it must be possible for these resources to be used by reference, rather than by duplication.

• It must be possible to efficiently assemble even large documents.

• It must be possible to assemble signed documents in such a way that their signatures are preserved.

• It must be possible for a document to contain multiple signatures, full or selective, from one or more signers.

• It must be possible to read a secured (encrypted) document without suffering an unreasonable delay when first viewing the document, without unreasonably exposing the decrypted contents of the document, and while obeying rights associated with the document.

• It must be possible to efficiently extract data from the document (i.e., a document fragment) and without modification to the extracted data.

Finally, the introduction of multiple formats (i.e., XML and Binary XML), implies the following desirable requirement:

• The conversion of a document between different encodings must preserve all information in the document, including digital signatures.

3.7.5 Properties

TBD

3.7.6 Alternatives

The current de facto standard for interchange of electronic documents is Adobe's Portable Document Format, or PDF. PDF meets all of the requirements stated here except that it is not based on a widely accepted, implemented, and available format, and so while widely deployed for document viewing is not sufficiently easy to use for general-purpose interchange.

Earlier formats for electronic documents, such as TIFF, DVI, RTF, AFP, and Postscript do not, among other shortcomings, support the full range of required document features, such as dynamic and interactive content.

HTML/XHTML, SVG, XSL-FO, and other XML-based formats can, in combination, provide coverage for most of the requirement document features. However, they fail to meet certain file format requirements as described under Analysis, above.

There are other proprietary formats, such as Microsoft Word and SWF, which meet many of the functional requirements described here but, due to their proprietary nature, also lack sufficiently broad-spread acceptance, implementation, and availability.

3.7.7 References

1. [PDF]

3.8.1 Description

The Securities industry has cooperated to define a standard protocol and a common messaging language called FIX which allows real-time, vendor/platform neutral electronic exchange of securities transactions between financial institutions.

The original definition of FIX was as a tag-value pair format. Due to increased competition by the year 1999, and to better accommodate business models of emerging initiatives, an XML-based message format for application-layer messages called FIXML was devised. Even though FIXML was designed to have minimum impact on existing systems, in order to protect investments in traditional FIX systems and processes, it soon became evident that the new message size was as much as 6 times larger than its tag-value predecessor, a condition that precluded key participants in the industry to integrate FIXML into their systems. This problem, together with some positive findings made through experiments, spurred the discussion for size reduction of FIXML messages, which culminated in a new format called Transport Optimized FIXML (TO-FIXML) in FIXML version 4.4. TO-FIXML is essentially a collection of XML Schema definitions that uses name abbreviations as well as attributes instead of elements wherever possible to collectively reduce FIXML messages up to 4 times.

3.8.2 Domain

Securities industry engaging in capital markets such as derivatives, equity and fixed-income markets, where the FIX protocol is applicable and is moving towards SOA architectures based on FIXML. Major roles played in the industry include brokers, exchanges and clearing houses.

3.8.3 Justification

Even though TO-FIXML has been designed to minimize message sizes, some industry participants still consider it to be a sub-optimal solution and envisage the possibility of further optimization by studying binary-compatible XML formats.

3.8.4 Analysis

XML was the natural choice for the securities industry in light of its expandability and flexibility, which was required for the continuous and rapid evolution of the FIX protocol. There was also a demand for cross-industry interoperability given the broad adoption of XML by other financial industries.

XML Schema is the point of agreement for multiple parties to share a common transport format. However, the bloated size of the XML instances resulted in artificial changes to the schemas, with the sole purpose of reducing the number of bytes on the wire. The methods used for this purpose include the use of name abbreviations and the use of attributes in favor of elements wherever possible. Clearly, XML Schema is not the right place to tackle this problem given that the syntax verbosity is a property exclusive to the XML serialization. Stated differently, XML Schema is the point of agreement in terms of vocabulary and structure, not in terms of syntax.

Shown below are two sample FIXML order messages. The two messages carry the same information, yet their appearance is quite different. The first one is in FIXML version 4.3 format, while the second is in TO-FIXML format (FIXML version 4.4). As shown below, the one in TO-FIXML is much more compact than its equivalent FIXML 4.3 message. Some items such as 'Sender' and 'TransactTime' that were elements in FIXML 4.3 became attributes in TO-FIXML with abbreviated names 'SID' and 'TxnTm', respectively.

    <FIXML DTDVersion="2.0.0" FIXVersion="4.3">
      <FIXMLMessage>
        <Header>
          <Sender><CompID>CAT</CompID></Sender>
          <Target><CompID>DOG</CompID></Target>
          <SendingTime>2004-10-13T12:00:00</SendingTime>
        </Header>
        <Order>
          <ClOrdID>123456</ClOrdID>
          <Instrument>
            <Symbol>XYZ</Symbol>
          </Instrument>
          <Side Value="1"/>
          <TransactTime>2004-10-13T12:00:00</TransactTime>
          <OrderQtyData>
            <OrderQty>100</OrderQty>
          </OrderQtyData>
          <OrdType Value="2"/>
          <Price>85.00</Price>
        </Order>
      </FIXMLMessage>
    </FIXML>

    <FIXML v="4.4" r="20030618" s="20031218">
      <Order ID="123456" Side="1"
             TxnTm="2004-10-13T12:00:00" Typ="2" Px="85.00">
        <Hdr SID="CAT" TID="DOG"
             Snt="2004-10-13T12:00:00"/>
        <Instrmt Sym="XYZ"/>
        <OrdQty Qty="100"/>
      </Order>
    </FIXML>

3.8.5 Properties

The following format properties are essential:

• Human Readable/Editable/Deducible: Security exchanges and clearing houses are often required to archive documents for later retrieval, which requires the document format to be easily archivable.

• Processing Speed: The document format also has to allow for efficient data binding so that both sending and receiving parties can exchange documents in an efficient manner. This property is important for a system to perform as many transactions as possible, given that the run-time data-binding cost is known to have a considerable impact on the entire system's throughput.

3.8.6 Alternatives

Message size alone can be substantially reduced by standard compression methods. However, there is a study that shows compression of FIXML instances increases round trip time over 10 Mbps networks. Compression may be useful for considerably slower networks, which is not the typical case in FIXML. The same study also suggests that marshalling/unmarshalling costs do not seem to make tangible performance difference in those data sets typically seen in FIX scenarios.

3.8.7 References

1. [SOAP in Real-Time Trading Systems]

3.9.1 Description

The Service Enabler standard for mobile handsets benefits from extensive use of XML-based technologies for interoperability. For example, SMIL, SVG and XHTML are used as document formats for mobile content services such as:

• Multimedia Messaging Services (MMS): MMS in 3G consists of multiple XML documents, such as SMIL, SVG and XHTML. The handset is required to parse and render multi-namespaced XML documents.

• Map Services: Map data delivered to a handset is split into multiple chunks based on region and level of detail; handsets retrieve additional chunks in response to user zooms and scrolls. Additional data, such as restaurant information supplied by other content providers, can also be overlayed on top.

XML documents in these services are considerably large. For instance, the map data represented in SVG could be 100KB or more. Rich content MMS could also be very large. Even on today's high-end handsets with 120 MHz 32-bit RISC processors, parsing a raw 100 KB XML document takes approximately 10 seconds.

3.9.2 Domain

This use case applies to multimedia services for the mobile handsets.

3.9.3 Justification

XML is required for maximum interoperability. In fact, XML technology is already widely adopted in the mobile services space. As this area requires a solution for narrow band and limited footprint devices, the importance of this use case should be considered high.

3.9.4 Analysis

This use case requires the following capabilities of XML to be preserved:

• Interoperability

• Multiple namespace support

• In-memory, random access using a DOM

Interoperability is mandatory as the same documents must be shared among different handsets. Moreover, for map services, the layering of multiple source map data requires interoperability among the providers. Support for multiple namespaces is a must in order to deliver multi-format messages (e.g. HTML + SVG) to the devices. DOM access is required to support ECMA scripting as well as for efficient rendering of formats such as SVG.

The requirements not satisfied by current XML solutions that must be addressed are:

• Efficient transmission of XML documents by reducing their sizes

• Efficient access to a DOM, i.e. efficient DOM parsing

3.9.5 Properties

TBD

3.9.6 Alternatives

The WAP Forum defined a WAP Binary XML format as an alternative serialization for XML. However, this format has a number of shortcomings, the biggest of which is the lack of support for multi-namespace documents due to the use of a "single dimension" system of 6-bit tags.

3.9.7 References

1. [Mobile SVG]

   [Mobile XHTML]

   [SMIL]

   [G-XML]

   [MMS]

   [WAP WBXML]

3.10.1 Description

A large business communicates via XML with a number of remote businesses, some of which can be small business partners. These remote or small businesses often have access only to slow transmission lines and have limited hardware and technical expertise. The large business cannot expect the smaller partners to upgrade often or to use expensive technology. The primary illustrations of this use case come from the energy, banking, and retail industries.

In the energy industry, the major upstream (exploration and production) operations of oil companies are largely in developing countries and it is a common problem to have very slow and perhaps unreliable communications between the main office and remote sites. It's not that the oil companies don't know how to set up a satellite feed, it's that they are often required by the local governments to use the communication facilities provided by that government, and these communications can be technically low-end and expensive. So the common problem is one where there is plenty of processing power and bandwidth at both central and remote sites, but the communication between the two is slow.

Although many scenarios illustrating this problem have to do with upstream operations, this specific example will be from downstream (refining and marketing). It involves transmission of Point of Sale (POS) information back and forth between back office systems and remote sites. The data flowing to the remote sites includes "incremental price book" for dry goods and wet stock, currency exchange rates, promotion codes/rates/groups and so on. The data coming back includes raw sales transactions data, tank data, etc. One might have 1000 transactions per day per site with an average file size of 3 KB, for a total size of 3 Megs typically broken up into 12 documents (transmittal every 2 hours, referred to as "trickle feed"). Each document would then average 250 KB.

Currently the scope is for many thousands of sites connected to several regional back-office hubs. Connectivity ranges from VSAT to 32 kbps analog connections. The 32 kbps connections would only communicate once a day. This downstream situation includes a factor which is not common in upstream operations. Not only are there communication limitations, but in this case some of the remote sites also have limited processing capabilities because they are small businesses with limited resources.

In the banking industry, there is typically a main data center(s) and several connected branch offices, ATM machines, and business partners. The main data center may have the latest in technology, however, the connected branch offices, ATM machines and partners are often without access to high speed connections and powerful hardware. Communicating between the various entities can be accomplished with XML Web Services, however, the size and speed issues of XML are troublesome for those without access to high speed lines and/or powerful machines.

These same issues affect the retail industry as stores often are connected to the main data center over less than optimal links. In addition, the retail store needs to perform real-time purchase/return transactions that require round-trip communications with the main center.

3.10.2 Domain & Stakeholders

Retailing operations of large companies, particularly those where the actual retail outlets are SME's (Small to Medium Size Enterprises) and large companies with various small business partners and/or branch offices. The belief is that the experience gained in this situation is likely to be directly applicable to a number of other scenarios in the industry.

Note that the players in this use case have rather different situations and needs. The large company has significant sunk investment in complex back-office systems, lots of hardware and a team of IT professionals. The objective here is to integrate the solution into a complex, high-tech environment. The SME, partner, or branch office, may have very limited hardware and technical resources and is probably highly motivated toward a simple, low-cost solution, preferably one that plugs-and-plays off the shelf without extensive configuration or integration. This creates a tension between flexible and capable on one hand and simple and cheap on the other.

3.10.3 Justification

The need for compression is clear, but it is not so clear that a binary XML standard is required. One solution is to use a standard compression technique, like GZIP, on the XML and transmit it that way. This may not be an effective solution for all cases. One problem is that the SME, branch office, business partner, and/or ATM machine location might not have enough computing power to make the GZIP solution attractive. In addition, the data center, which may have a lot of computing power, may not want to absorb the increased CPU load that a GZIP solution requires. A binary solution that compressed the message and did not increase the CPU load could be beneficial in these cases.

3.10.4 Analysis

XML’s verboseness and size detract from its use in these scenarios. An alternate encoding would be required to be more compact than the original XML encoding with no loss of data. This would enable the branch office and other entities that operate over less than optimal transmission lines to take better advantage of XML and Web Services.

3.10.5 Properties

The following properties are essential:

• Compactness
• Encryptable
• Processing Speed
• Signable

The following properties are desirable:

• Efficient Update
• Fragmentable
• Random Access
• Schema Instance Change Resilience

3.10.6 Alternatives

This is a case where there is a need to compress entire data files, which are composed of a bunch of tags with relatively short data fields. That is, no huge arrays of floating point numbers causing special problems. Also note that the energy industry expects no particular problem with processing on either end, just the transmission, so the overhead of using compression algorithms is not a problem.

The energy industry currently plans to use native VSAT compression and probably one of many standard compression algorithms like ZIP for other transmission mechanisms. Initial tests with freeware ZIP compression software yielded a compression of 39:1, which is plenty. One problem that did occur, however, on small machines such as might be used by a small business, is that the compression algorithm may need to read the entire document into memory and work on it globally. On small machines this can cause paging and the resulting performance difficulties can be painful. Some sort of compression algorithm that works in a streaming mode or on "chunks" of data would obviously be preferable in these cases.

There are cases, however, where the SME is unwilling to use the CPU required by the compression. This has been encountered in cases where the small business has a computer, perhaps a mainframe, that is overburdened by other routine tasks. In this case binary serialization may be an attractive alternative assuming it can be done without extra CPU cycles.

The idea is that if one is going to have to parse the XML anyway (which may or may not be the case, depending on the business process), the CPU required to do that parsing is a "sunk cost". Once parsed, a binary serialization of the XML will probably be smaller than the usual text serialization because the tags are not repeated in text and some of the data fields (e.g. some numbers) may be smaller in their binary representation. For typical business documents one might expect a reduction on the order of a factor of two from binary serialization. This moderate reduction in file size may hit the "sweet spot" in cases where CPU is a big problem and file size a moderate concern. It seems likely, however, that this scenario will be less common than the case where reasonable computational capability is available in the small business and the slow transmission lines are the big problem. In these cases, as documented above, compression via standard techniques of the conventional text serialization of XML is probably the preferred solution.

Note that in both cases the needs of both small and large businesses can potentially be met. The large business gets the XML document it needs in order to integrate with its complex systems. The small business either uses inexpensive compression software or the parser outputs the binary serialization directly, so the complexity of the solution from their viewpoint is minimized.

3.10.7 References

1. [ACORD]

2. [HL7]

3. [RosettaNet]

3.11.1 Description

The Extensible Messaging and Presence Protocol (XMPP) [RFC 3920] [RFC 3921] [RFC 3922] [RFC 3923] formalize the Jabber instant messaging and presence protocol [Jabber] for the IETF. In addition, the Jabber Software Foundation develops extensions to Jabber, known as Jabber Enhancement Proposals [JEPs]. These protocols use XML as an underlying mechanism for instant messaging, group chat, and presence. During an instant messaging session the protocol opens an XML document and the participants exchange "stanzas" of XML document fragments during an open-ended conversation. The architecture is client-server; clients authenticate to a server. Messages sent to that user's ID on that server are forwarded to the client. There is a large and active installed base of XMPP/Jabber clients and servers.

XML is used in XMPP because it offers flexibility, ease of debugging, and extensibility. New XML semantics can be added to the protocol, and it is simple to examine the contents of messages and determine their meaning. In a very general sense, it can be said that XMPP acts as a sort of router for XML messages.

3.11.2 Domain & Stakeholders

Instant messaging and chat applications.

3.11.3 Analysis

XML requires more computational power to parse than equivalent binary protocols used in other instant messaging and chat protocols. When XML stanzas are received at the server they must be routed to the other servers or clients who have an interest in the message. Routing requires parsing enough of the XML stanzas to make a routing decision. Typically only part of the XML stanza at a well-known location needs to be examined by the server, and the message payload remains opaque to the server. Therefore, being able to quickly retrieve only the information needed is a significant benefit.

The traffic profile is that of a great number of small XML messages that must be parsed quickly, interspersed with a few larger messages with binary payloads. Single servers may support up to tens of thousands of users, so in high traffic environments parsing speed becomes a limiting factor in the number of users a server can support.

In order to satisfy this use case, a binary format must be faster to parse the data required to make a routing decision than the text XML format.

It is also helpful for XML stanzas being exchanged to be compact when the conversation is being carried out over bandwidth-limited links. A chat server being accessed across a wireless link using uncompressed XML will run out of bandwidth faster than using a compact binary format.

An XMPP server often sends out slightly modified messages to many people, for example when an XMPP participant's presence status changes. Notifications must be sent to other people on that person's buddy list. In this situation nearly identical messages that differ only in a few respects, such as address, should be sent. The ability to efficiently update an XML message with new data would be useful in this situation.

Any server or client that uses an XML binary standard must maintain the original XML format in order to preserve compatibility with the XMPP RFC standards. Thus, any binary format must be transcodable back to an equivalent textual XML format.

3.11.4 Justification

When the server receives an XML stanza it must parse the document and decide what clients or servers should receive the message. Also, across bandwidth-limited links, XML is more verbose than binary formats. Binary formats have the potential to improve these aspects of XMPP.

3.11.5 Properties

The following properties are essential:

• Fragmentable: XMPP's implementation works by opening an XML document and exchanging stanzas of XML fragments. When the XML document is closed the underlying socket connection is shut down. Supporting XMPP requires the ability to work with XML fragments.

• Processing Speed: XML must be processed efficiently.

• Round Trippable: Communication with other XMPP servers that do not support binary formats require that binary XML be translated to an equivalent text XML format.

• Streamable: Since XMPP relies on exchanging XML fragments a binary format must be able to encode less than a full document.

The following properties are desirable:

• Accelerated Sequential Access: Often the information required to route an XML fragment is co-located in the stanza. The ability to rapidly access data that are next to each other is useful.

• Compactness: It is useful to have messages that take up less space than text XML.

• Efficient Update: Ability to efficiently send out messages that differ only slightly from each other is useful.

• Embedding Support: Some XMPP messages may contain embedded multimedia data.

• Random Access:. Making efficient routing decisions depends on rapidly extracting only the information relevant to the routing decision.

3.11.6 Alternatives

The Jabber community has addressed compact representations of XML streams through an experimental add-on extension to XMPP that allows the streams to negotiate compression using zlib [Jabber Stream Compression]. The XMPP protocol may also use Transport Layer Security (TLS), which can optionally include zlib compression of encrypted traffic. Using zlib places a somewhat larger computational load on the server above and beyond the parsing duties it must perform. Again, with large numbers of subscribed users or heavy traffic the computational load can become a limiting factor. A compact binary format is likely to be intermediate in size between a zlib compressed XML stanza and the original XML, while retaining high processing speed. The JEP compression protocol can easily be extended to cover alternative formats, such as a binary XML format or a binary format that is also compressed.

3.11.7 References

1. [RFC 3920]

2. [RFC 3921]

3. [RFC 3922]

4. [RFC 3923]

5. [Jabber]

6. [JEPs]

7. [Jabber Stream Compression]

3.12.1 Description

Documents are stored in the persistent store. They are searched and updated. For its support, beside compression, it requires multiple techniques such as selective indexing, data access using groups of pre-compiled XPath expressions, etc. The size of documents varies depending on application domains and particular customer needs within those domains. The persistent store management system has to deal with many scenarios, from small documents to very, very large ones. The ratio of the space taken by actual data and structural information (tags, attribute names, and namespace definitions) also varies very significantly.

When documents are stored in the persistent store or retrieved from it different kinds of transformations and processing can be requested by different applications. Thus, the persistent store management system has to deal with various tools interfacing to it. The ways documents will be manipulated are never known in advance.

Since documents can be large or very large they have to be processed incrementally in many cases. The incremental processing can be of two types: stream like or lazy DOM like. The stream like processing requires documents to be transferred by pieces where each piece can be fully processed without information in the following pieces. The lazy DOM processing allows client applications to request only a part, subtree (full or partial), of the document.

Schema aware documents present multiple opportunities for various optimizations. These optimizations affect different aspects of the alternative XML serialization: compactness, efficiency of random access, efficiency of sequential processing, etc. The alternative XML serialization format must include means to apply these optimizations to the representation of XML documents. At the same time it should allow schema aware documents to be represented, on request, as non-schema aware documents.

Handling schemas in the persistent store has certain properties that are not necessarily present in other use cases. The most important one is the need to support schema evolutions. Schema evolutions include additions of new elements and attributes, modifications by extension of simple data types, etc. In this respect, the forward compatibility of the alternative XML serialization format is very important for persistently stored documents.

There are a couple of important and interesting aspects that need more discussion. One of them is versioning. There are two different kinds of versioning: versioning of schemas for schema aware documents and versioning of documents (in the course of different modifications made to the document) for all XML documents. Both kinds of versioning present technical and usability challenges.

The XQuery Data Model provides an abstract representation of one or more XML documents or document fragments. The data model is based on the notion of a sequence. A sequence is an ordered collection of zero or more items. An item may be a node (document, element, attribute, namespace, text, processing-instruction or comment) or an atomic value. The input and output of XQuery are defined in terms of the XQuery Data Model. Because of this, the efficient handling of XQuery Data Model instances, when storying, retrieving, or searching, is very important. While an XML level serialization of XQuery Data Model instances is defined, it is not sufficient for applying an alternative XML serialization to XQuery Data Model instances. Thus it is important that an alternative XML serialization is rich enough to handle not only the XML serialization but also the XQuery Data Model serialization.

3.12.2 Domain & Stakeholders

Stakeholders are all providers of persistent store management systems and all providers of applications using and relying on persistent store management systems.

3.12.3 Justification

Persistent store technologies are widely and globally used. XML documents have been stored (and queried, and manipulated) in persistent stores for some number of years. The use of XML capable persistent stores is rapidly growing. And nobody expects any decline in this process.

The efficiency of XML handling in the persistent store is becoming critically important. The experience of persistent store management system providers shows that the use of alternative XML serializations of documents significantly improves performance, both size wise and processing efficiency wise.

A standard for alternative XML serialization is important for several reasons. Persistent stores interoperate with multiple software components (application servers, various client applications, mobile devices, etc.) produced by multiple vendors. Experience shows that conversions between multiple proprietary formats can be quite expensive. In addition, proprietary formats tend to change rapidly. This increases development costs. Also, handling of patent related issues is not usually helpful in delivering products on time.

3.12.4 Analysis

While persistent store vendors usually handle several data representations, the necessity to handle XML documents is a given. It is not a question whether applications using persistent stores can use other data representations or not. It is the reality of persistent store customers using XML in more and more applications. The use of XML in the persistent store, as described above, leads to following requirements:

• Efficiently support the alternative XML serialization format for schema aware documents and be friendly to schema evolutions that maintain the validity of documents

• Efficiently support certain operations in the persistent store and in the memory. These operations should include querying, updating, indexing, access to fragments, and fragment extraction.

• Efficiently support XML schema datatypes. Transferring data in native binary format is typically more compact. For example, xsd:hexBinary data represented as binary bytes is half the size of the corresponding textual hexadecimal representation.

• Efficiently support incremental processing where both partial loading and partial transmission are included.

• The use of an alternative XML serialization should require minimal changes to existing application layers (only the underlying layers need to be changed). The impact to developers should be minimal.

• Efficiently support multiple ordering. It should be possible to process documents both sequentially and based on subtrees.

3.12.5 Properties

The requirements in the previous section clearly define what properties of XML representations, alternative or current (XML 1.x), are important for handling XML documents in the persistent store. These properties include:

• Accelerated Sequential Access

• Compactness

• Data Model Versatility

• Efficient Update

• Encryptable

• Fragmentable

• Integratable into XML Family

• Processing Speed

• Random Access

• Robustness

• Schema Instance Change Resilience

• Signable

• Streamable

• Transport Independence

• Version Identification

3.12.6 Alternatives

As it was mentioned in the justification section, persistent store vendors are required to handle XML to support multiple customer applications using XML. It would be absolutely inappropriate for persistent store vendors to offer their customers another format instead of XML.

3.12.7 References

None.