W3C

XML Binary Characterization Use Cases

W3C Working Draft 09 November 2004

This version:
http://www.w3.org/TR/2004/WD-xbc-use-cases-20041109/
Latest version:
http://www.w3.org/TR/xbc-use-cases
Previous version:
http://www.w3.org/TR/2004/WD-xbc-use-cases-20041104
Editors:
Mike Cokus, MITRE Corporation
Santiago Pericas-Geertsen, Sun Microsystems

Abstract

This document describes use cases for evaluating the potential benefits of an alternate serialization for XML. The use cases are documented here to understand the constraints involved in environments for which XML employment may be problematic because of one or more characteristics of XML 1.x. Desirable properties of XML and alternative serializations to address the use cases are derived and discussed in a separate publication of the XML Binary Characterization Working Group (XBC WG) [XBC Properties].

Status of this Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This is a Working Draft of the XML Binary Characterization Use Cases document. It has been produced by the XML Binary Characterization Working Group, which is part of the XML Activity.

This document is part of a series of documents following this work on Use Cases determination. Further work in the XML Binary Characterization Working Group focuses on characterizing the properties that are required by the use cases, and establishing objective, shared measurements to help judge whether XML 1.x and alternate binary encodings provide the required properties.

This document have been published recently with some obsolete content. This new publication is meant to reflect the up-to-date state of the document, it is recommended not to read the previous version.

This is a Working Draft and is expected to change. The XML Binary Characterization Working Group does not expect this document to become a Recommendation. Rather, after further development, review and refinement, it will be published and maintained as a Working Group Note.

Comments on this document should be sent to public-xml-binary-comments@w3.org (public archives). It is inappropriate to send discussion emails to this address.

Discussion of this document takes place on the public public-xml-binary@w3.org (public archives).

Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

Table of Contents

1 Introduction
2 Use Case Structure
3 Documented Use Cases
    3.1 Metadata in Broadcast Systems
        3.1.1 Description
        3.1.2 Domain & Stakeholders
        3.1.3 Justification
        3.1.4 Analysis
        3.1.5 Properties
        3.1.6 Alternatives
        3.1.7 References
    3.2 Floating Point Arrays in the Energy Industry
        3.2.1 Description
        3.2.2 Domain
        3.2.3 Justification
        3.2.4 Analysis
        3.2.5 Properties
        3.2.6 Alternatives
        3.2.7 References
    3.3 X3D Graphics Model Compression, Serialization and Transmission
        3.3.1 Description
        3.3.2 Domain & Stakeholders
        3.3.3 Justification
        3.3.4 Analysis
        3.3.5 Properties
        3.3.6 Alternatives
        3.3.7 References
    3.4 Web Services for Small Devices
        3.4.1 Description
        3.4.2 Domain
        3.4.3 Justification
        3.4.4 Analysis
        3.4.5 Properties
        3.4.6 Alternatives
        3.4.7 References
    3.5 Web Services within the Enterprise
        3.5.1 Description
        3.5.2 Domain
        3.5.3 Justification
        3.5.4 Analysis
        3.5.5 Properties
        3.5.6 Alternatives
        3.5.7 References
    3.6 Embedding External Data in XML Documents
        3.6.1 Description
        3.6.2 Domain
        3.6.3 Justification
        3.6.4 Analysis
        3.6.5 Properties
        3.6.6 Alternatives
        3.6.7 References
    3.7 Electronic Documents
        3.7.1 Description
        3.7.2 Domain & Stakeholders
        3.7.3 Justification
        3.7.4 Analysis
        3.7.5 Properties
        3.7.6 Alternatives
        3.7.7 References
    3.8 FIXML in the Securities Industry
        3.8.1 Description
        3.8.2 Domain
        3.8.3 Justification
        3.8.4 Analysis
        3.8.5 Properties
        3.8.6 Alternatives
        3.8.7 References
    3.9 Multimedia XML Documents for Mobile Handsets
        3.9.1 Description
        3.9.2 Domain
        3.9.3 Justification
        3.9.4 Analysis
        3.9.5 Properties
        3.9.6 Alternatives
        3.9.7 References
    3.10 Intra/Inter Business Communication
        3.10.1 Description
        3.10.2 Domain & Stakeholders
        3.10.3 Justification
        3.10.4 Analysis
        3.10.5 Properties
        3.10.6 Alternatives
        3.10.7 References
    3.11 XMPP Instant Messaging Compression
        3.11.1 Description
        3.11.2 Domain & Stakeholders
        3.11.3 Analysis
        3.11.4 Justification
        3.11.5 Properties
        3.11.6 Alternatives
        3.11.7 References
    3.12 XML Documents in Persistent Store
        3.12.1 Description
        3.12.2 Domain & Stakeholders
        3.12.3 Justification
        3.12.4 Analysis
        3.12.5 Properties
        3.12.6 Alternatives
        3.12.7 References
    3.13 Business and Knowledge Processing
        3.13.1 Description
        3.13.2 Domain & Stakeholders
        3.13.3 Justification
        3.13.4 Analysis
        3.13.5 Properties
        3.13.6 Alternatives
        3.13.7 References
    3.14 XML Content-based Routing and Publish Subscribe
        3.14.1 Description
        3.14.2 Domain & Stakeholders
        3.14.3 Justification
        3.14.4 Analysis
        3.14.5 Properties
        3.14.6 Alternatives
        3.14.7 References
    3.15 Web Services Routing
        3.15.1 Description
        3.15.2 Domain & Stakeholders
        3.15.3 Justification
        3.15.4 Analysis
        3.15.5 Properties
        3.15.6 Alternatives
        3.15.7 References
    3.16 Versioning Systems
        3.16.1 Description
        3.16.2 Domain & Stakeholders
        3.16.3 Justification
        3.16.4 Analysis
        3.16.5 Properties
        3.16.6 Alternatives
        3.16.7 References
4 Summary
5 References

Appendix

A Acknowledgments (Non-Normative)


1 Introduction

While XML has been enormously successful as a markup language for documents and data, the overhead associated with generating, parsing, transmitting, storing, or accessing XML-based data has hindered its employment in some environments. The question has been raised as to whether some optimized serialization of XML is appropriate to satisfy the constraints present in such environments. In order to address this question, a compatible means of classifying the requirements posed by specific use cases and the applicable characteristics of XML 1.x must be devised. This allows a characterization of the potential gap between what XML 1.x supports and use case requirements. In addition, it also provides a way to compare use case requirements to determine the degree to which an alternate serialization could be beneficial.

Use cases describing situations where some characteristics of XML may prevent its effective use are presented in this document. The XBC WG has made efforts through internal discussion and dialog with the XML community to define a comprehensive set of use cases to examine and compare potential solutions, including alternate serializations of XML as well as other available means. Comments are invited, especially if important use cases or solutions to address them have been ommitted.

2 Use Case Structure

In this section we elaborate on the template used to present the use cases. All the use cases collected by this WG are listed in 3 Documented Use Cases.

3 Documented Use Cases

The use cases identified by or submitted to the working group are documented below, in accordance with the meta data defined in 2 Use Case Structure.

3.1 Metadata in Broadcast Systems

Editorial note: SP15 October, 2004

Latest version from Robin.

3.1.1 Description

The constant progress of digital TV, the multiplication of channels, the competition and convergence with the Web, and the widespread deployment of a variety of set-top boxes (notably Personal Video Recorders, or PVRs for short) call for services on TV sets that extend beyond simply broadcasting audio/video content.

For instance, above a certain number of channels, broadcasters find themselves having to provide EPG (Electronic Program Guide) services to their users, without which they would be overwhelmed with the sheer amount of available content. These EPGs also allow PVRs to automatically pick up recording schedules for given programs, based on user-defined criteria that match against metadata broadcasted alongside the data.

Similarly, broadcasters are also trying to make their offer more attractive by integrating TV with Web technologies or the Web at large. This includes notably using Web Services from PVRs that benefit from a return channel, using Web UI technologies such as SVG, XHTML, or XForms to define their applications' interfaces, making TV services available to mobile devices, and so forth.

However, there are constraints that cause problems when trying to deploy such services, all of which rely on XML, to television sets:

  • Bandwidth: TV bandwidth is extremely expensive, and how much data you use for services directly constrains the number of channels that you are able to send. In addition to the potential technical issues, there are strong economic motivations to reduce bandwidth usage as much as possible.

  • Processing Power: Most set-top boxes are inexpensive, and the low-end ones have roughly half the power (if not less) of a low-end mobile device. Contrary to mobile devices, there are few limitations as to the processing power that can be embarked in a box the size of the average set-top box (STB), notably the problems relating to heat and battery life are of little or no concern. However, on the one hand large-scale deployment of STBs and similar devices into households requires them to be extremely inexpensive and therefore as limited as possible, and on the other hand convergence with mobile devices remains a prime motivator for the television industry and constraints applicable to mobile devices apply equally to broadcasted XML metadata.

  • Unidirectional Network: This being broadcast, there is not typically a way for TVs to request data. Instead, it is being continuously streamed and re-streamed to them, a process which is called carouselling (the data itself being 'on a carousel'). Some set-top boxes do in fact have a return channel (notably the ones that support Web Services) but the vast majority doesn't. If the data were sent as an XML document, it would have to be fragmented so that STBs wouldn't have to wait for the end of the entire document to have been carouselled in order to start exploiting the data. XML has not shown to be easily fragmentable and currently the carouselling relies on the ability of the binary formats to be fragmented.

  • Change Resilience. Upgrading several million STBs is often very impractical. Therefore, it must be possible to evolve the broadcast format without breaking older hardware. While XML is perfectly suited to this, the above issues make it unusable. It is thus required that the binary XML format replacing it be resilient to changes in the schema.

As a result, MPEG-7 BiM (a binary encoding of XML originally created to carry video metadata), has been integrated into a number of broadcasting standards, notably ARIB, DVB, and TV Anytime.

3.1.2 Domain & Stakeholders

This use case is relevant to the entirety of the television distribution industry, comprising content providers, broadcast infrastructure deployers, television and set-top box manufacturers, and of course the broadcasting companies themselves.

It also covers similar requirements that can be found in digital radio broadcasting, where one equally needs to broadcast EPG metadata to very limited devices, to integrate with mobile devices (for instance by sending SVG ads as part of the radio stream).

And finally, convergence with TV is considered to be a major next step in mobile services, and all participants on both sides of the fence are presently being extremely active in making television available anywhere, at any time, and on any device. Quite naturally, this leads to the need for common technology between the TV industry and the rest of the Web. As such, the major stakeholders in the domain have expressed interest in reusing a solution commonly agreed upon across several industries.

3.1.3 Justification

Television is a very large market that has a strong need for program metadata, and is increasingly converging with the Web at large (with a strong emphasis on mobile devices at first), notably using technologies such as XHTML, SVG, XForms, SMIL, and Web Services.

Deployed systems already use binary XML, currently standardized as part of ISO MPEG and industry fora such as ARIB, DVB, or TV-Anytime, but have expressed interest in using more broadly adopted technologies.

3.1.4 Analysis

XML is appropriate for these situations because:

  1. existing specifications based on XML are being reused wholesale;

  2. most major TV standards in the area are already XML-based, and the industry has no wish to go through another standards cycle;

  3. XML is well-suited to describing structured information such as metadata;

  4. XML has proven to be a good format to specify user interfaces in, using notably XHTML, SVG, or XForms. These are needed for TV applications, and they need to be broadcasted;

  5. the industry wishes to publish its data, especially the Electronic Program Guides, to as many media as possible. XML enables it to publish directly to desktops, mobiles, and TVs using off-the-shelf or Open Source software.

3.1.5 Properties

XML serializations in this domain require Robustness in that a television set cannot go into error if a fragment it receives is malformed: it simply has to throw it away and wait to see if the next turn of the carousel will bring a well-formed version. Thus, it is essential that Robustness is supported at the fragment level so that the entirety of the stream is not corrupted by a single problem.

The low-end class of target devices has constraints that make Processing Speed and Small Footprint essential properties, however these systems are asymmetric and, in the typical case, unidirectional. Therefore, these constraints apply exclusively to the decoder.

Encoded data is not expected to be Self Contained, but rather the decoder can be required to have access to a schema definition of some form (typically, a kind of compiled schema is used). This is further underlined by the fact that the requirement for Compactness is so essential that support for Specialized Codecs becomes highly desirable.

Updating set-top boxes and related devices is inconvenient at best, when at all possible. However, services need to evolve over time, entailing evolution in the XML vocabulary. It is thus essential for the format to support Schema Instance Change Resilience for most changes.

Television systems worldwide use different technologies, and it is therefore essential for the format to support Transport Independence.

A Streamable format is essential for the decoder given the way in which fragments are carouselled (decoders can even start reading in mid-stream). Also, STBs will typically know which fragments on the carousel they have already processed in order not to have to process them again and will only process uncached ones. Thus, support for deltas and Accelerated Sequential Access are also essential properties.

Several of the properties listed above indirectly require a Fragmentable format, therefore this property is deemed essential as well.

3.1.6 Alternatives

DVB EIT schedules provide a small subset of the required functionality and do not integrate well with a more generic information ecosystem.

Without a W3C format of binary XML, application domains will likely adopt different formats--a phenomenon that already started in the Broadcast domain. This would further cluster the market. A convergence (which can be recognized right now) between the broadcast and the mobile communication services would be hindered by this clustering. In this concrete case, mobile devices would be required to implement codecs of both domains to enable value added services like interactive and location aware TV broadcast. This drives up the initial investment, which translate into a great obstacle for a converged service in the marketplace.

3.1.7 References

  1. [DVB]

  2. [TV Anytime]

  3. [ARIB]

  4. [MPEG-7]

3.2 Floating Point Arrays in the Energy Industry

Editorial note: SP/MC21 October, 2004

Updated based on latest version from Ravi and comments from Roger.

3.2.1 Description

The upstream segment of the energy industry is concerned with exploration for and production of oil and gas. XML-based techniques have made very little penetration into the upstream technology part of the energy industry. The most basic reason for this is the nature of the data, which does not at this time lend itself to being represented usefully in XML.

There are basically two core types of data in this industry: well logs and seismic data. Well logs are moderately large datasets while seismic datasets are real large, typically in the order of gigabytes. Although the Petrotechnical Open Standards Consortium [POSC] has produced an XML schema for well logs [WellLogML], it has not been adopted by the industry. At the time of writing, nobody has even considered defining a schema for seismic data.

Both seismic and well log data include control data, easily represented in XML, as well as large arrays of floating point numbers, not easily represented efficiently in XML. Although in practice an XML representation is not used, such data may be represented as shown in the following fragment (with a whole document consisting of a large number of these fragments):

    <seisdata>
      <lineheader>...<lineheader> 
      <header>
        <linename>westcam 2811</linename>
        <ntrace>1207</ntrace>
        <nsamp>3001</nsamp>
        <tstart>0.0</tstart>
        <tinc>4.0</tinc>
        <shot>120</shot>
        <geophone>7</geophone>
      </header>
      <trace>0.0 0.0 468.34 3.245672E04 6.9762345E05 ... (3001 floats)</trace>
      <header> 
        ... 
      </header>
      <trace> ... </trace>
      ...
    </seisdata>

3.2.2 Domain

The scope within the Energy Industry as discussed above is very broad, encompassing a very large number of technical issues and usage scenarios involving, for example, integration of drilling information, processing of seismic and well data, integration of seismic and well data into interpretation systems, and so on.

3.2.3 Justification

There are a number of dominant technology vendors in this sector as well as a number of small companies that "work around the edges". The dominant technology vendors (which are none of the technology giants) provide proprietary solutions that do not interoperate easily with each other. Providing communications between these products within a company, or between companies, is a constant problem: this is the main motivator to develop Web service interfaces for these products. A second motivator for a standard is that it will open the door for smaller companies to provide useful add-on products. Large budgets in this sector are allocated to the purchase of software packages and display devices, but these budgets are small compared to the leverage of mass-market devices, so a longer term objective is to encourage a situation were more technologies with mass-market cost leverage can be used.

3.2.4 Analysis

Given that this scenario involves interoperability between companies using disparate systems, XML is a natural choice due to its ubiquity and tool availability.

The main shortcoming of XML for this application is the expense incurred while converting floating point data to and from a character representation, as well as the extra size of some of these representations. Thus, the main requirement for this use case is the ability to represent sequences of floating point numbers in a binary format (as close to the native representation as possible), in order to facilitate efficient binding into programmatic objects (primarily, floating point arrays). In the example shown above, the header information would still have a textual representation (useful for any infoset-based processing), but the trace of floating point numbers would appear as an opaque binary stream.

The format chosen to represent a floating point number must be platform independent, with tools supporting conversions to and from the appropriate native format. In practice, most operations involve moving data between machines with the same floating point formats, so the solution should not impose undue overhead on the most common situation in order to handle the less common ones.

3.2.5 Properties

The properties Compactness and Processing Speed are essential for this use case. Documents containing large arrays of floating points must be serialized in a compact format. It is expected that most of the compression is achieved by encoding floating point numbers in a binary format instead of characters. Further, parsing (extracting) out the floating point arrays and binding them to programmatic objects should be efficient.

The properties Robustness, Schema Instance Change Resilience and Data Model Versatility are desirable for this use case. If errors are encountered while transferring large documents in alternative serialization, it is often better to continue with the processing (as best as possible) instead of forcing a complete retransmission. The encoding of floating point arrays will most likely rely on schema information. Hence, resilience to compatible changes to the schema is desired. Support for multiple data models, especially those that are able to represent arrays of floating points will improve the adoption of the format.

3.2.6 Alternatives

  • Data Compression: One expert in this area has said, "For us, binary compression is probably not that important because transmission speeds are constantly improving. The additional time needed to compress and decompress seismic data would probably slow things down. We also place a greater value in the message structures than the transmission mechanics". Or, in more picturesque words, again from an expert in the field when asked about compressing seismic data, "Been there, done that, doesn't work, not interested". Bear in mind that this epigram encapsulates decades of experience and highly sophisticated R&D.

  • CORBA: There is, in fact, a CORBA-based integration platform currently deployed (although perhaps not widely) in this space. Without diving into technical details, it is clear that some companies would prefer an approach based on Web services.

  • XML Protocol Attachments: It is possible to represent seismic data control information in XML and to put the floating point arrays in a binary attachment using XOP. This data architecture is certainly viable, assuming that the issues involving floating point numbers are addressed, as evidenced by the fact that many of the proprietary vendor data formats work this way. It is, however, less flexible than the header-trace architecture described above, which is probably one reason why the latter is used in industry-wide seismic data standards (e.g. SEGY). Nonetheless, Web services that return data using XOP are an attractive alternative for dealing with seismic data.

3.2.7 References

  1. [POSC]

  2. [WellLogML]

  3. [SEGY]

3.3 X3D Graphics Model Compression, Serialization and Transmission

Editorial note: MC19 October, 2004

Updated from 18 October email. Should be ready for second draft.

3.3.1 Description

Extensible 3D (X3D) Graphics [Extensible 3D (X3D) Graphics] is an XML-enabled ISO Standard 3D file format to enable real-time communication of 3D data across all applications and networks. It is used for commercial applications, engineering and scientific visualization, medical imaging, training, modeling and simulation (M&S), multimedia, entertainment, education, and more. [X3D Markets] Computer-Aided Design (CAD) and architecture scenes are also supported, though because they have larger file sizes and are more complex they are not typically streamed for web-delivered viewing.

Web-delivered file sizes in this use case typically range from 1-1000 KB of data while CAD files may run to several hundred megabytes apiece. An optimized serialization of the X3D data may be performed in concert with application-specific compression (e.g. combining coplanar polygons, quantizing colors, etc.) Lossy geometric compression is sometimes acceptable. Due to interaction requirements, the latency time associated with deserialization, decompression and parsing must be minimal. Digital signature and encryption compatibilities are also important for protecting digital content assets.

3.3.2 Domain & Stakeholders

Support of Web-based interchange, rendering and interactivity for 3D graphics scenes. Stakeholders include tool builders, content authors, application developers and end users of 3D graphics models.

3.3.3 Justification

The X3D Compressed Binary Encoding Request For Proposals (RFP) from the X3D Consortium [X3D RFP] lists and justifies ten separate technical requirements. Many of these have parallels to a general optimized serialization format. The Web3D Consortium and X3D designers see great value in aligning with W3C standardization efforts in this area. This serves as significant evidence of the need for an industry standard.

3.3.4 Analysis

Taken together, the following technical requirements for the X3D Compressed Binary Encoding RFP (indicated by emphasized type) include many requirements for an optimized serialization format. Strictly speaking, such an efficient serialization is not necessarily "compressed", though it is very likely to be more compact. Other factors, such as speed of parsing or databinding performance, may override the desire for compact representation for some applications, such as CAD. The X3D Consortium's RFP has chosen to include the ability to perform application-specific size optimizations, both lossless and lossy, as part of the process of converting the document from XML to the optimized serialization. However, the optimized format can still be represented as XML; in other words, a lossy geometric compression to an efficient format can be translated back to a lossy XML representation (i.e., Transcodable to XML).

  • X3D Compatibility: The compressed binary encoding shall be able to encode all of the abstract functionality described in X3D Abstract Specification. Since X3D is expressed in XML, any optimized serialization that encodes all XML features will also be capable of encoding an X3D document.

  • Interoperability: The compressed binary encoding shall contain identical information to the other X3D encodings (XML and Classic VRML). It shall support an identical round-trip conversion between the X3D encodings. This corresponds to the Round Trippable property.

  • Multiple, separable data types: The compressed binary encoding shall support multiple, separable media data types, including all node (element) and field (attribute) types in X3D. The RFC allows the possibility of performing domain-specific compressions or encodings of data in the XML document, for example of polygons and textures. The ability to make use of Specialized Codecs is essential to meeting this requirement.

  • Processing Performance: The compressed binary encoding shall be easy and efficient to process in a runtime environment. Because the data for interactive applications is delivered across the web low latency is important. The ability to quickly process documents is important. In the case of CAD files, which may be several hundred megabytes in size, the ability to quickly process the file is very important. Often 3D files have long arrays of numeric data. Using XML format requires that a reader extract the string information and convert it to a binary representation. This is a computationally expensive process. Experimental data has shown that an optimized format can be 20 times as fast as XML. The ability to rapidly process arrays of floating point data is also important. Thus the Accelerated Sequential Access property is also relevant to this use case.

  • Ease of Implementation: Binary compression algorithms shall be easy to implement. This corresponds to Implementation Cost.

  • Streaming: Compressed binary encoding will operate in a variety of network-streaming environments. X3D documents are often streamed in web environments, and portions of the 3D scene are rendered as they arrive on the client computer. This corresponds to the Streamable property.

  • Compression: Compressed binary encoding algorithms will together enable effective compression of diverse datatypes. 3D data often consists of large arrays of floating point data that can be compressed in various ways. The ability to employ Specialized Codecs, either lossless or lossy, would meet this requirement.

  • Security: Compressed binary encoding will optionally enable security, content protection, privacy preferences and metadata such as encryption, conditional access, and watermarking. This corresponds to the Signable property.

  • Bundling: Mechanisms for bundling multiple files (e.g. X3D scene, inlined sub-scenes, image files, audio file, etc.) into a single archive file will be considered. This corresponds to Embedding Support.

  • Intellectual Property Rights (IPR): Technology submissions must meet the Web3D Consortium IPR policy. The W3C Patent Policy [W3C PP] is compatible with this.

3.3.6 Alternatives

GZIP is the specified compression scheme for Virtual Reality Modeling Language (VRML 97) specification, the second-generation ISO predecessor to X3D. GZIP is not type-aware and does not compress large sets of floating-point numbers as well. GZIP allows staged decompression of 64KB blocks, which might be used to support streaming capabilities. GZIP outputs are strings and require a second pass for any parsing, thus degrading parsing and loading performance. A GZIPed file would not gain the parsing speed and databinding advantages of an optimized format.

Numerous piecemeal, incompatible proprietary solutions exist in the 3D graphics industry for Web-page plug-ins. None address the breadth of technical capabilities that might be enabled by a general purpose optimized serialization format.

An X3D-specific compression and serialization algorithm for XML is certainly feasible and demonstrated. Compatibility with a general recommendation for an optimized format is desirable in order to maximize interoperability with other XML technologies, and reduce implementation cost. Many of these issues are common to other use-case domains; broad mutual benefits become possible via a common recommendation.

3.4 Web Services for Small Devices

Editorial note: SP13 October, 2004

Ready for second draft.

3.4.1 Description

As Web services become more and more ubiquitous, there is a greater demand to use this technology as a way to deliver content to small devices such PDAs, pagers and mobile phones. All these devices often share the following characteristics:

  1. They have limited memory and limited processing power.

  2. Battery life is at a premium.

  3. They are connected to low-bandwidth, high-latency networks which in some cases are regulated by "pay-per-byte" policies.

XML-based messaging is at the heart of the current Web services technology. XML's self-describing nature has significant advantages, but they come at the price of bandwidth and performance. XML-based messages are larger and require more processing than other protocols, and are therefore not well suited for a domain having the characteristics outlined above. Increased bandwidth usage affects wireless networks due to bandwidth restrictions allotted for communication by each device. In addition, the larger the message the higher the probability of a retransmission as a result of an on-the-air collision.

The target platforms for this use case include a broad range of PDAs, handhelds and mobile handsets, including mass market devices that limit code size to 64K and heap size to 230K. The transport packet size may vary from network to network, but it is typically measured in bytes (e.g. 128 bytes).

3.4.2 Domain

Small devices connected to low-bandwidth, high-latency networks. Two examples in this domain are cellular phone networks and PDA networks employed by the military.

3.4.3 Justification

XML is the fundamental technology underlying a Web services infrastructure, and one of the main reasons why Web services are not being deployed on the mobile space. A number of alternative serializations have already been developed to deliver XML content to small devices, however, many of these are not interoperable. This lack of interoperability results in fragmentation and the need for specialized gateways to transcode proprietary formats.

3.4.4 Analysis

In order to satisfy the requirements of this use case, an alternative serialization must be faster to process and must produce smaller packets. Faster processing will result in lesser battery consumption while smaller packets will result in reduced latency as well as, assuming a pay-per-byte model, a more cost-effective service. In addition to small and fast, an alternative serialization should also be streamable, i.e. it should be possible for the client application to operate on any prefix of the serialized data.

Assuming that the same amount of information is encoded in an alternative serialization, a way to quantify efficiency is to consider the instruction to data ratio. In other words, the amount of effort that is needed to produce or consume a unit of data. Even though this is an implementation requirement, an alternative serialization must enable the creation of "thin" stacks with a low instruction to data ratio.

The reduction in latency that results by improving parsing speed may or may not be noticeable to the consumer depending on the transport latency of the network --transport latency is the dominant factor in many existing networks. Nevertheless, a more efficient parsing method will improve battery life on the device as well as throughput on the server.

3.4.5 Properties

The properties Processing Speed and Compactness are essential for this use case. Given the technological limitations on existing wireless networks for small devices, it is imperative to minimize the number of bytes sent on each transmission; a compact message will also increase the battery life of a device by reducing the usage time of the transmitter. Low parsing and serialization times of the underlying format result in an improved user experience by reducing response time and conserving battery life. In some circumstances, it is essential to employ a Streamable format to achieve these results.

Support for the Signable and Encryptable properties is also desirable, yet not essential since transport-level security (e.g., HTTPS) is often available and intermediaries are typically not deployed. A fundamental additional consideration for this use case is that the format must enable the implementation of Small Footprint software stacks.

3.4.6 Alternatives

Proprietary solutions result in the so-called gatewayed networks, where communication is always routed through a single point that translates to and from XML. This architecture not only creates a single point of failure within a network but also fragments the entire network by creating non-interoperable, domain-specific solutions.

Message size reductions are attainable via the use of standard data compression techniques. Even though in general decompression is less expensive than compression, it is still too costly for most small devices. Additionally, the extra burden of compressing packets has a negative impact on the overall system throughput.

In addition to the added cost, redundancy-based compression algorithms tend to perform very poorly on small messages, in many cases resulting in larger messages. Mobile clients often carry on dialogs with servers which consist of a large number of small messages. Examples of this include: data synchronization, stateful web services, multi-player games, querying and browsing data. In all of these use cases, the cumulative stream of messages that make up the dialog can grow very large even though all of the individual messages are rather small. Thus, there is still a need to reduce the amount of data exchanged, but doing this by compressing each message individually is not a viable solution.

3.5 Web Services within the Enterprise

Editorial note: SP13 October, 2004

Ready for second draft.

3.5.1 Description

A large number of existing enterprise systems are built using distributed technologies such as RMI, DCOM and CORBA. As the industry moves from distributed object systems to Service Oriented Architectures (SOAs), the use of Web services technologies becomes more significant even within the confines of a single enterprise. Many of the concepts behind SOAs are applicable to divisions within an corporation, so it is only natural to extend the applicability of Web services to intranet systems.

A stumbling block that several re-architected systems are facing is that XML-based messages are larger and require more processing than those from existing protocols: data is represented inefficiently and binding requires more computation. It has been shown that an RMI service can perform up to an order of magnitude faster than an equivalent Web Service due to the processing required to parse and bind XML data into programmatic objects.

3.5.2 Domain

The domain is that of distributed systems, typically based on binary protocols, which for technical reasons (e.g., interoperability) or for economic reasons (e.g., reduction of software licenses) need to be re-architected as Web services. An important constraint for these type of re-deployments is to maintain (or improve) the system's performance, a task that has been found challenging given the additional processing requirements of XML-based protocols.

3.5.3 Justification

There are some important economic reasons that support the use of Web services as an alternative to existing technologies for building distributed systems. First, preliminary results show more powerful hardware is needed to re-deploy existing systems using Web services technologies given the additional processing requirements of an XML messaging system. Second, assuming the company in question already develops (or is planning to develop) Web services to communicate outside their firewall, their is the extra incentive in using the same set of tools and the same development team to build intranet applications. This reduces both software fees (e.g., by reusing application servers and development tools) as well as training costs associated with having separate development teams for each technology. Third, some companies that have successfully deployed CORBA-based systems, but are not planning on deploying Web services, may find an additional incentive to do so if a more efficient serialization is standardized.

3.5.4 Analysis

Intranet Web services differ from Internet Web services especially in the areas of deployment and security: deployments are easier to manage and security is typically defined by a single domain. The requirements for intranet systems are somewhat different from those for Internet systems, permitting the use of certain optimizations in the former which would be difficult or simply impossible to implement in the latter. Consequently, in many cases the degree of coupledness of the systems can be adjusted if this helps achieving the desired performance goals.

The main requirement for this use case is reducing XML processing time in order to achieve a level of performance comparable to the existing systems. Due to the availability of high-speed networks in these scenarios, reducing message sizes is of a lesser priority. It is worth pointing out that not all systems re-deployed using Web services will be unable to achieve their performance requirements. Therefore, this use case applies only to a subset of the aforementioned re-deployments.

3.5.5 Properties

The property Processing Speed is essential for this use case. In addition to parsing speed, data binding into programming language objects is an important aspect for this use cases since most Web services toolkits favor this form of processing. Compactness is also desirable, especially in those cases in which it enables faster processing.

Given the importance of confidentiality, authentication and integrity in a Web services scenario, the properties Signable and Encryptable are also deemed as essential. Self Contained and Schema Instance Change Resilience can be classified as essential if intermediaries are deployed as part of the system. If the are no intermediaries, or if intermediaries can be easily updated as schemas evolve, these two properties can be considered less essential or simply desirable. Other desirable properties are: Embedding Support.

3.5.6 Alternatives

In some cases, it may be possible to re-design the system's interfaces to make them more coarse grained in order to reduce the number of messages exchanged. Although this is technically feasible in most cases, the costs associated with this effort can be prohibiting.

3.6 Embedding External Data in XML Documents

3.6.1 Description

Although the data in an XML document is encoded as text, it is often the case that portions of that text are in fact embedded documents in and of themselves. This frequently occurs when XML documents contain images, recordings, or other multimedia elements which have their own file formats --JPEG, MP3, and so on. In order to embed these documents they are translated to a textual representation using base64-encoding or a similar scheme. It is worth noting that these embedded documents are often much larger than the encapsulating document.

For document-oriented applications, these embedded files are often part of the document, in the sense that they are intended to be rendered along with the text in the encapsulating XML document. Thus, the translation from text back to the original file format often occurs as the rest of the document is being parsed. The case in which the embedded document is in fact an XML document (e.g., an SVG graphic embedded within an XSL-FO document) can be regarded as a special case in which no translation is required.

For message-oriented applications, embedded documents are simply payload elements encapsulated within, e.g., a SOAP body element. In these cases, the payloads may be of arbitrary or even unknown formats. They are often translated to text during transmission and back to their original form upon reception, even if not otherwise immediately consumed, in order to reduce storage space. Because these embedded documents can be large, storage requirements are further reduced by streaming on input and output. In these cases the embedded document may also be XML but may contain either processing instructions or DTDs, both of which should not appear within a SOAP body element (WS-I Basic Profile, R1008 and R1009). Therefore, such files may be treated as if they were binary data and base64-encoded even if they are, in fact, valid XML files.

3.6.2 Domain

This use case considers the domains of both electronic documents and Web services.

3.6.3 Justification

XML was not designed to contain binary data, and other packaging mechanisms such as MIME, exist and are in many ways suitable to the purpose at hand. On the other hand, the ability to treat embedded documents as part of the primary documents, and therefore make them accessible to XML-based standards and tools like XPath without resorting to additional standards like MIME, is useful in practice. Thus, this use case is a good demonstration of why one might wish to extend XML with a binary encoding.

3.6.4 Analysis

This use case builds on applications of XML for documents and Web services that are already well established. Furthermore, those uses already involve the transmission of documents with binary data like integers and floats. The question is, for each given embeddable datum, should it be placed inside the document or should it be carried as an attachment?

The drawbacks to embedding are the penalties in time due to the translation into text, and in space, due to the larger size of the translated data. The benefits include access to other XML-based technologies like XPath, XQuery, etc. and the avoidance of an additional dependency on a packaging technology such as MIME.

To address this use case, a binary XML format must permit binary documents to be embedded within an XML document without requiring a translation to a text form; a binary XML format must also support the streaming of such XML documents, a desirable feature in Web service calls.

3.6.5 Properties

TBD

3.6.6 Alternatives

SOAP with Attachments provides a MIME-based mechanism for packaging binary data with SOAP messages. It avoids the translation costs, but does not make the binary data part of the XML document itself. In that respect, it is not a streamable format.

XOP describes how a MIME-based package can be used to encapsulate the binary data without a translation overhead by keeping it (at least conceptually) as part of the encapsulating XML document. Because of its use of MIME, this approach suffers from many of the same shortcomings of the SOAP with attachments case.

3.7 Electronic Documents

Editorial note: MC13 September, 2004
Update from 8 September email.

3.7.1 Description

Documents are the most basic form of recorded human communication, dating back thousands of years. Electronic documents are the transition of this invention to the online, computerized world. Books, forms, contracts, e-mails, spreadsheets, and Web pages are only some of the forms in which electronic documents are used. Unlike paper-based documents, electronic documents are not limited to static text and images. Electronic documents regularly contain both static content, dynamic content (e.g., animations, video), and interactive content (e.g., form fields). This wide range of content has a great affect on selecting an appropriate representation format and must be considered in evaluating this use case.

Documents are first created in some authoring environment. During the creation process the author may elect to include text, fonts, image, videos, or other resources which are to be rendered more than once when the document is displayed. For example, a company logo may appear in the header of each page of a document, but this should not require adding the logo to the document more than once.

In a special case of document creation, new documents are created by assembling a set of existing documents into a single aggregate document. For example, this may done to combine a basic product manual with additional documentation for optional product accessories into a customized manual for an individual purchaser. When documents are bound together in this way it may be important that the data in the original documents is not modified, so as to preserve signatures or other properties of the file, or it may be desirable to identify and eliminate duplicated resources, such as fonts.

After a document has been created it is usually read, in whole or in part. Documents are not necessarily read front to back; a particular reader may select a different order or read only part of a document. A reader may, for example, obtain the document by traversing a hyperlink which points to a specific location within the document. It is important that rendering a document for reading be fast, even when starting at an arbitrary location in this way, and even when documents are large (millions of pages). This implies that it must be possible to navigate to specific sections within a document quickly, as well as follow links to shared resources within the document, as mentioned above under document creation. Finally, if a document is being retrieved over a slow link, it may be useful to fetch portions of the document in the order in which they are being rendered and read (e.g., starting at page 700), as opposed to document order (i.e., starting at page 1).

Documents often contain information of a sensitive or proprietary nature and so can be secured using encryption technologies. Encrypting the document can serve either to keep the contents confidential, to--in conjunction with the rendering application--allow only certain operations ("rights") on the document, or both. Typically a description of any rights granted is embedded within the document itself when it is encrypted. It is often desirable that only portions of a document be encrypted so that intermediaries can access some portion of the data in the file.

Documents, and especially those used in business transactions, are often signed to indicate authenticity of, consent to, or agreement with the document. In electronic documents, this is implemented by digitally signing the document. The digital signature must itself be stored in the document. Multiple signatures may be applied to a document, each one signing those which came before it. Additional information is sometimes added to a document after it has been signed but without invalidating a signature--in the same way one can initial a correction to a paper document--but so that it is clear that any subsequent changes were not present when the pre-existing signatures were applied. In some cases signatures should apply to only part of a document, leaving other parts for later modification. Finally, it must be possible for a recipient to validate all of these signatures.

Documents are often long-lived and, during the course of their lives, used in different environments with varying constraints. For example, when a document is being published for general consumption, it might be most desirable to select an encoding such as XML which is widely understood. If, however, the same document is being transmitted between partners with known expectations a more compact format such as [XOP] might be preferred. Thus, a single document may sometimes be transformed between different encodings at different times and for different purposes. Such transformations should preserve the information in the document, but these operations cannot be expected to be compatible with encryption mechanisms used to secure documents.

Even when various encodings are available documents tend to push the available storage and bandwidth of the devices on which they are created, stored, transmitted, and read. In other words, as device capabilities increase, users respond by creating larger documents. Note that these documents rarely contain only text; they generally contain larger elements such as fonts and images and, increasingly, video and 3D models which these same enhanced devices make possible.

Electronic documents, like their paper counterparts, can be modified or re-purposed. In electronic documents, this typically occurs when pages, images, videos, and so forth are either copied out of a document to be used elsewhere or removed from a document to produce an altered version of that document. Again, these operations should be efficient: removing any one page from a one million page document should not take significantly longer than doing the same to a ten page document. Documents may also be modified by their recipients to include comments of various types--editors' marks, sticky notes, etc.--usually intended to communicate responses back to the author. These comments may be stored within the document itself; both adding them to and extracting them from the document should be efficient.

Finally, some documents are designed to be interactive beyond the limited interactions of rendering, signing, and annotating. These documents may contain form fields, GUI widgets such as buttons and listboxes, or other active elements, data islands bound to these widgets, and code, scripts, or declarative logic to validate input to these elements, enable or disable the elements, transmit the document, modify the document, interact with the rendering application, and so forth. It must be possible to describe and access all of these elements within the document itself.

3.7.2 Domain & Stakeholders

Electronic documents are used extensively throughout government, business, and personal domains as well as in the interchange between these entities.

3.7.3 Justification

XML is in its roots a syntax for marking documents, and so the electronic document use cases seem highly relevant. Interestingly, XML has a number of shortcomings (discussed below) with respect to many of the requirements derived from this use case. Arguably, these occur because XML (and SGML) were focused largely on textual documents, but such documents represent a decreasing fraction of all electronic documents. Thus, Binary XML as a natural extension of XML to handle new document types, and documents containing new content, seems particularly relevant.

3.7.4 Analysis

Documents are almost always exchanged between two or more people, and often between larger entities such as corporations or governments. It is, therefore, extremely desirable that an electronic document format should be easily consumable by all parties involved. XML, as a widely accepted, implemented, and used format, fits this need quite well.

Unfortunately there a number of requirements imposed by electronic documents which XML fails to address:

  • Documents frequently contain embedded resources such as fonts, images, and video which are themselves encoded in binary formats. It must be possible to efficiently embed these resources in documents.

  • It must be possible to navigate to and render a specified location in better than linear time with respect to the size of the document.

  • The document encoding must be efficient with respect to space, that is, it must have low redundancy.

  • The document encoding must be efficient with respect to space, that is, it must have low entropy.

  • In order to make updates efficient, it must be possible to update a document in time proportional to the size of the update rather than the size of the document.

There are a number of requirements which XML does address, but which are enumerated here as well because they would also be requirements on any Binary XML encoding:

  • The format should be widely accepted, available, and implemented.

  • Re-usable resources may appear, or be referenced from, multiple locations within the document. In order to maintain reasonable document sizes, it must be possible for these resources to be used by reference, rather than by duplication.

  • It must be possible to efficiently assemble even large documents.

  • It must be possible to assemble signed documents in such a way that their signatures are preserved.

  • It must be possible for a document to contain multiple signatures, full or selective, from one or more signers.

  • It must be possible to read a secured (encrypted) document without suffering an unreasonable delay when first viewing the document, without unreasonably exposing the decrypted contents of the document, and while obeying rights associated with the document.

  • It must be possible to efficiently extract data from the document (i.e., a document fragment) and without modification to the extracted data.

Finally, the introduction of multiple formats (i.e., XML and Binary XML), implies the following desirable requirement:

  • The conversion of a document between different encodings must preserve all information in the document, including digital signatures.

3.7.5 Properties

TBD

3.7.6 Alternatives

The current de facto standard for interchange of electronic documents is Adobe's Portable Document Format, or PDF. PDF meets all of the requirements stated here except that it is not based on a widely accepted, implemented, and available format, and so while widely deployed for document viewing is not sufficiently easy to use for general-purpose interchange.

Earlier formats for electronic documents, such as TIFF, DVI, RTF, AFP, and Postscript do not, among other shortcomings, support the full range of required document features, such as dynamic and interactive content.

HTML/XHTML, SVG, XSL-FO, and other XML-based formats can, in combination, provide coverage for most of the requirement document features. However, they fail to meet certain file format requirements as described under Analysis, above.

There are other proprietary formats, such as Microsoft Word and SWF, which meet many of the functional requirements described here but, due to their proprietary nature, also lack sufficiently broad-spread acceptance, implementation, and availability.

3.7.7 References

  1. [PDF]

3.8 FIXML in the Securities Industry

Editorial note: 15 October, 2004
Latest version from Takuki. Should be ready for second draft.

3.8.1 Description

The Securities industry has cooperated to define a standard protocol and a common messaging language called FIX which allows real-time, vendor/platform neutral electronic exchange of securities transactions between financial institutions.

The original definition of FIX was as a tag-value pair format. Due to increased competition by the year 1999, and to better accommodate business models of emerging initiatives, an XML-based message format for application-layer messages called FIXML was devised. Even though FIXML was designed to have minimum impact on existing systems, in order to protect investments in traditional FIX systems and processes, it soon became evident that the new message size was as much as 6 times larger than its tag-value predecessor, a condition that precluded key participants in the industry to integrate FIXML into their systems. This problem, together with some positive findings made through experiments, spurred the discussion for size reduction of FIXML messages, which culminated in a new format called Transport Optimized FIXML (TO-FIXML) in FIXML version 4.4. TO-FIXML is essentially a collection of XML Schema definitions that uses name abbreviations as well as attributes instead of elements wherever possible to collectively reduce FIXML messages up to 4 times.

3.8.2 Domain

Securities industry engaging in capital markets such as derivatives, equity and fixed-income markets, where the FIX protocol is applicable and is moving towards SOA architectures based on FIXML. Major roles played in the industry include brokers, exchanges and clearing houses.

3.8.3 Justification

Even though TO-FIXML has been designed to minimize message sizes, some industry participants still consider it to be a sub-optimal solution and envisage the possibility of further optimization by studying binary-compatible XML formats.

3.8.4 Analysis

XML was the natural choice for the securities industry in light of its expandability and flexibility, which was required for the continuous and rapid evolution of the FIX protocol. There was also a demand for cross-industry interoperability given the broad adoption of XML by other financial industries.

XML Schema is the point of agreement for multiple parties to share a common transport format. However, the bloated size of the XML instances resulted in artificial changes to the schemas, with the sole purpose of reducing the number of bytes on the wire. The methods used for this purpose include the use of name abbreviations and the use of attributes in favor of elements wherever possible. Clearly, XML Schema is not the right place to tackle this problem given that the syntax verbosity is a property exclusive to the XML serialization. Stated differently, XML Schema is the point of agreement in terms of vocabulary and structure, not in terms of syntax.

Shown below are two sample FIXML order messages. The two messages carry the same information, yet their appearance is quite different. The first one is in FIXML version 4.3 format, while the second is in TO-FIXML format (FIXML version 4.4). As shown below, the one in TO-FIXML is much more compact than its equivalent FIXML 4.3 message. Some items such as 'Sender' and 'TransactTime' that were elements in FIXML 4.3 became attributes in TO-FIXML with abbreviated names 'SID' and 'TxnTm', respectively.

    <FIXML DTDVersion="2.0.0" FIXVersion="4.3">
      <FIXMLMessage>
        <Header>
          <Sender><CompID>CAT</CompID></Sender>
          <Target><CompID>DOG</CompID></Target>
          <SendingTime>2004-10-13T12:00:00</SendingTime>
        </Header>
        <Order>
          <ClOrdID>123456</ClOrdID>
          <Instrument>
            <Symbol>XYZ</Symbol>
          </Instrument>
          <Side Value="1"/>
          <TransactTime>2004-10-13T12:00:00</TransactTime>
          <OrderQtyData>
            <OrderQty>100</OrderQty>
          </OrderQtyData>
          <OrdType Value="2"/>
          <Price>85.00</Price>
        </Order>
      </FIXMLMessage>
    </FIXML>

    <FIXML v="4.4" r="20030618" s="20031218">
      <Order ID="123456" Side="1"
             TxnTm="2004-10-13T12:00:00" Typ="2" Px="85.00">
        <Hdr SID="CAT" TID="DOG"
             Snt="2004-10-13T12:00:00"/>
        <Instrmt Sym="XYZ"/>
        <OrdQty Qty="100"/>
      </Order>
    </FIXML>

3.8.5 Properties

The following format properties are essential:

  • Human Readable/Editable/Deducible: Security exchanges and clearing houses are often required to archive documents for later retrieval, which requires the document format to be easily archivable.

  • Processing Speed: The document format also has to allow for efficient data binding so that both sending and receiving parties can exchange documents in an efficient manner. This property is important for a system to perform as many transactions as possible, given that the run-time data-binding cost is known to have a considerable impact on the entire system's throughput.

3.8.6 Alternatives

Message size alone can be substantially reduced by standard compression methods. However, there is a study that shows compression of FIXML instances increases round trip time over 10 Mbps networks. Compression may be useful for considerably slower networks, which is not the typical case in FIXML. The same study also suggests that marshalling/unmarshalling costs do not seem to make tangible performance difference in those data sets typically seen in FIX scenarios.

3.9 Multimedia XML Documents for Mobile Handsets

3.9.1 Description

The Service Enabler standard for mobile handsets benefits from extensive use of XML-based technologies for interoperability. For example, SMIL, SVG and XHTML are used as document formats for mobile content services such as:

  • Multimedia Messaging Services (MMS): MMS in 3G consists of multiple XML documents, such as SMIL, SVG and XHTML. The handset is required to parse and render multi-namespaced XML documents.

  • Map Services: Map data delivered to a handset is split into multiple chunks based on region and level of detail; handsets retrieve additional chunks in response to user zooms and scrolls. Additional data, such as restaurant information supplied by other content providers, can also be overlayed on top.

XML documents in these services are considerably large. For instance, the map data represented in SVG could be 100KB or more. Rich content MMS could also be very large. Even on today's high-end handsets with 120 MHz 32-bit RISC processors, parsing a raw 100 KB XML document takes approximately 10 seconds.

3.9.2 Domain

This use case applies to multimedia services for the mobile handsets.

3.9.3 Justification

XML is required for maximum interoperability. In fact, XML technology is already widely adopted in the mobile services space. As this area requires a solution for narrow band and limited footprint devices, the importance of this use case should be considered high.

3.9.4 Analysis

This use case requires the following capabilities of XML to be preserved:

  • Interoperability

  • Multiple namespace support

  • In-memory, random access using a DOM

Interoperability is mandatory as the same documents must be shared among different handsets. Moreover, for map services, the layering of multiple source map data requires interoperability among the providers. Support for multiple namespaces is a must in order to deliver multi-format messages (e.g. HTML + SVG) to the devices. DOM access is required to support ECMA scripting as well as for efficient rendering of formats such as SVG.

The requirements not satisfied by current XML solutions that must be addressed are:

  • Efficient transmission of XML documents by reducing their sizes

  • Efficient access to a DOM, i.e. efficient DOM parsing

3.9.5 Properties

TBD

3.9.6 Alternatives

The WAP Forum defined a WAP Binary XML format as an alternative serialization for XML. However, this format has a number of shortcomings, the biggest of which is the lack of support for multi-namespace documents due to the use of a "single dimension" system of 6-bit tags.

3.10 Intra/Inter Business Communication

Editorial note: MCOctober 19, 2004
Current as of October 13 email. Should be ready for second draft.

3.10.1 Description

A large business communicates via XML with a number of remote businesses, some of which can be small business partners. These remote or small businesses often have access only to slow transmission lines and have limited hardware and technical expertise. The large business cannot expect the smaller partners to upgrade often or to use expensive technology. The primary illustrations of this use case come from the energy, banking, and retail industries.

In the energy industry, the major upstream (exploration and production) operations of oil companies are largely in developing countries and it is a common problem to have very slow and perhaps unreliable communications between the main office and remote sites. It's not that the oil companies don't know how to set up a satellite feed, it's that they are often required by the local governments to use the communication facilities provided by that government, and these communications can be technically low-end and expensive. So the common problem is one where there is plenty of processing power and bandwidth at both central and remote sites, but the communication between the two is slow.

Although many scenarios illustrating this problem have to do with upstream operations, this specific example will be from downstream (refining and marketing). It involves transmission of Point of Sale (POS) information back and forth between back office systems and remote sites. The data flowing to the remote sites includes "incremental price book" for dry goods and wet stock, currency exchange rates, promotion codes/rates/groups and so on. The data coming back includes raw sales transactions data, tank data, etc. One might have 1000 transactions per day per site with an average file size of 3 KB, for a total size of 3 Megs typically broken up into 12 documents (transmittal every 2 hours, referred to as "trickle feed"). Each document would then average 250 KB.

Currently the scope is for many thousands of sites connected to several regional back-office hubs. Connectivity ranges from VSAT to 32 kbps analog connections. The 32 kbps connections would only communicate once a day. This downstream situation includes a factor which is not common in upstream operations. Not only are there communication limitations, but in this case some of the remote sites also have limited processing capabilities because they are small businesses with limited resources.

In the banking industry, there is typically a main data center(s) and several connected branch offices, ATM machines, and business partners. The main data center may have the latest in technology, however, the connected branch offices, ATM machines and partners are often without access to high speed connections and powerful hardware. Communicating between the various entities can be accomplished with XML Web Services, however, the size and speed issues of XML are troublesome for those without access to high speed lines and/or powerful machines.

These same issues affect the retail industry as stores often are connected to the main data center over less than optimal links. In addition, the retail store needs to perform real-time purchase/return transactions that require round-trip communications with the main center.

3.10.2 Domain & Stakeholders

Retailing operations of large companies, particularly those where the actual retail outlets are SME's (Small to Medium Size Enterprises) and large companies with various small business partners and/or branch offices. The belief is that the experience gained in this situation is likely to be directly applicable to a number of other scenarios in the industry.

Note that the players in this use case have rather different situations and needs. The large company has significant sunk investment in complex back-office systems, lots of hardware and a team of IT professionals. The objective here is to integrate the solution into a complex, high-tech environment. The SME, partner, or branch office, may have very limited hardware and technical resources and is probably highly motivated toward a simple, low-cost solution, preferably one that plugs-and-plays off the shelf without extensive configuration or integration. This creates a tension between flexible and capable on one hand and simple and cheap on the other.

3.10.3 Justification

The need for compression is clear, but it is not so clear that a binary XML standard is required. One solution is to use a standard compression technique, like GZIP, on the XML and transmit it that way. This may not be an effective solution for all cases. One problem is that the SME, branch office, business partner, and/or ATM machine location might not have enough computing power to make the GZIP solution attractive. In addition, the data center, which may have a lot of computing power, may not want to absorb the increased CPU load that a GZIP solution requires. A binary solution that compressed the message and did not increase the CPU load could be beneficial in these cases.

3.10.4 Analysis

XML’s verboseness and size detract from its use in these scenarios. An alternate encoding would be required to be more compact than the original XML encoding with no loss of data. This would enable the branch office and other entities that operate over less than optimal transmission lines to take better advantage of XML and Web Services.

3.10.5 Properties

The following properties are essential:

The following properties are desirable:

3.10.6 Alternatives

This is a case where there is a need to compress entire data files, which are composed of a bunch of tags with relatively short data fields. That is, no huge arrays of floating point numbers causing special problems. Also note that the energy industry expects no particular problem with processing on either end, just the transmission, so the overhead of using compression algorithms is not a problem.

The energy industry currently plans to use native VSAT compression and probably one of many standard compression algorithms like ZIP for other transmission mechanisms. Initial tests with freeware ZIP compression software yielded a compression of 39:1, which is plenty. One problem that did occur, however, on small machines such as might be used by a small business, is that the compression algorithm may need to read the entire document into memory and work on it globally. On small machines this can cause paging and the resulting performance difficulties can be painful. Some sort of compression algorithm that works in a streaming mode or on "chunks" of data would obviously be preferable in these cases.

There are cases, however, where the SME is unwilling to use the CPU required by the compression. This has been encountered in cases where the small business has a computer, perhaps a mainframe, that is overburdened by other routine tasks. In this case binary serialization may be an attractive alternative assuming it can be done without extra CPU cycles.

The idea is that if one is going to have to parse the XML anyway (which may or may not be the case, depending on the business process), the CPU required to do that parsing is a "sunk cost". Once parsed, a binary serialization of the XML will probably be smaller than the usual text serialization because the tags are not repeated in text and some of the data fields (e.g. some numbers) may be smaller in their binary representation. For typical business documents one might expect a reduction on the order of a factor of two from binary serialization. This moderate reduction in file size may hit the "sweet spot" in cases where CPU is a big problem and file size a moderate concern. It seems likely, however, that this scenario will be less common than the case where reasonable computational capability is available in the small business and the slow transmission lines are the big problem. In these cases, as documented above, compression via standard techniques of the conventional text serialization of XML is probably the preferred solution.

Note that in both cases the needs of both small and large businesses can potentially be met. The large business gets the XML document it needs in order to integrate with its complex systems. The small business either uses inexpensive compression software or the parser outputs the binary serialization directly, so the complexity of the solution from their viewpoint is minimized.

3.10.7 References

  1. [ACORD]

  2. [HL7]

  3. [RosettaNet]

3.11 XMPP Instant Messaging Compression

Editorial note: SPOctober 20, 2004
Latest version from DM. Need to identify desirable vs. essential properties.

3.11.1 Description

The Extensible Messaging and Presence Protocol (XMPP) [RFC 3920] [RFC 3921] [RFC 3922] [RFC 3923] formalize the Jabber instant messaging and presence protocol [Jabber] for the IETF. In addition, the Jabber Software Foundation develops extensions to Jabber, known as Jabber Enhancement Proposals [JEPs]. These protocols use XML as an underlying mechanism for instant messaging, group chat, and presence. During an instant messaging session the protocol opens an XML document and the participants exchange "stanzas" of XML document fragments during an open-ended conversation. The architecture is client-server; clients authenticate to a server. Messages sent to that user's ID on that server are forwarded to the client. There is a large and active installed base of XMPP/Jabber clients and servers.

XML is used in XMPP because it offers flexibility, ease of debugging, and extensibility. New XML semantics can be added to the protocol, and it is simple to examine the contents of messages and determine their meaning. In a very general sense, it can be said that XMPP acts as a sort of router for XML messages.

3.11.2 Domain & Stakeholders

Instant messaging and chat applications.

3.11.3 Analysis

XML requires more computational power to parse than equivalent binary protocols used in other instant messaging and chat protocols. When XML stanzas are received at the server they must be routed to the other servers or clients who have an interest in the message. Routing requires parsing enough of the XML stanzas to make a routing decision. Typically only part of the XML stanza at a well-known location needs to be examined by the server, and the message payload remains opaque to the server. Therefore, being able to quickly retrieve only the information needed is a significant benefit.

The traffic profile is that of a great number of small XML messages that must be parsed quickly, interspersed with a few larger messages with binary payloads. Single servers may support up to tens of thousands of users, so in high traffic environments parsing speed becomes a limiting factor in the number of users a server can support.

In order to satisfy this use case, a binary format must be faster to parse the data required to make a routing decision than the text XML format.

It is also helpful for XML stanzas being exchanged to be compact when the conversation is being carried out over bandwidth-limited links. A chat server being accessed across a wireless link using uncompressed XML will run out of bandwidth faster than using a compact binary format.

An XMPP server often sends out slightly modified messages to many people, for example when an XMPP participant's presence status changes. Notifications must be sent to other people on that person's buddy list. In this situation nearly identical messages that differ only in a few respects, such as address, should be sent. The ability to efficiently update an XML message with new data would be useful in this situation.

Any server or client that uses an XML binary standard must maintain the original XML format in order to preserve compatibility with the XMPP RFC standards. Thus, any binary format must be transcodable back to an equivalent textual XML format.

3.11.4 Justification

When the server receives an XML stanza it must parse the document and decide what clients or servers should receive the message. Also, across bandwidth-limited links, XML is more verbose than binary formats. Binary formats have the potential to improve these aspects of XMPP.

3.11.5 Properties

The following properties are essential:

  • Fragmentable: XMPP's implementation works by opening an XML document and exchanging stanzas of XML fragments. When the XML document is closed the underlying socket connection is shut down. Supporting XMPP requires the ability to work with XML fragments.

  • Processing Speed: XML must be processed efficiently.

  • Round Trippable: Communication with other XMPP servers that do not support binary formats require that binary XML be translated to an equivalent text XML format.

  • Streamable: Since XMPP relies on exchanging XML fragments a binary format must be able to encode less than a full document.

The following properties are desirable:

  • Accelerated Sequential Access: Often the information required to route an XML fragment is co-located in the stanza. The ability to rapidly access data that are next to each other is useful.

  • Compactness: It is useful to have messages that take up less space than text XML.

  • Efficient Update: Ability to efficiently send out messages that differ only slightly from each other is useful.

  • Embedding Support: Some XMPP messages may contain embedded multimedia data.

  • Random Access:. Making efficient routing decisions depends on rapidly extracting only the information relevant to the routing decision.

3.11.6 Alternatives

The Jabber community has addressed compact representations of XML streams through an experimental add-on extension to XMPP that allows the streams to negotiate compression using zlib [Jabber Stream Compression]. The XMPP protocol may also use Transport Layer Security (TLS), which can optionally include zlib compression of encrypted traffic. Using zlib places a somewhat larger computational load on the server above and beyond the parsing duties it must perform. Again, with large numbers of subscribed users or heavy traffic the computational load can become a limiting factor. A compact binary format is likely to be intermediate in size between a zlib compressed XML stanza and the original XML, while retaining high processing speed. The JEP compression protocol can easily be extended to cover alternative formats, such as a binary XML format or a binary format that is also compressed.

3.12 XML Documents in Persistent Store

Editorial note: MC20 October, 2004
Up to date as of 20 October email. Needs some references.

3.12.1 Description

Documents are stored in the persistent store. They are searched and updated. For its support, beside compression, it requires multiple techniques such as selective indexing, data access using groups of pre-compiled XPath expressions, etc. The size of documents varies depending on application domains and particular customer needs within those domains. The persistent store management system has to deal with many scenarios, from small documents to very, very large ones. The ratio of the space taken by actual data and structural information (tags, attribute names, and namespace definitions) also varies very significantly.

When documents are stored in the persistent store or retrieved from it different kinds of transformations and processing can be requested by different applications. Thus, the persistent store management system has to deal with various tools interfacing to it. The ways documents will be manipulated are never known in advance.

Since documents can be large or very large they have to be processed incrementally in many cases. The incremental processing can be of two types: stream like or lazy DOM like. The stream like processing requires documents to be transferred by pieces where each piece can be fully processed without information in the following pieces. The lazy DOM processing allows client applications to request only a part, subtree (full or partial), of the document.

Schema aware documents present multiple opportunities for various optimizations. These optimizations affect different aspects of the alternative XML serialization: compactness, efficiency of random access, efficiency of sequential processing, etc. The alternative XML serialization format must include means to apply these optimizations to the representation of XML documents. At the same time it should allow schema aware documents to be represented, on request, as non-schema aware documents.

Handling schemas in the persistent store has certain properties that are not necessarily present in other use cases. The most important one is the need to support schema evolutions. Schema evolutions include additions of new elements and attributes, modifications by extension of simple data types, etc. In this respect, the forward compatibility of the alternative XML serialization format is very important for persistently stored documents.

There are a couple of important and interesting aspects that need more discussion. One of them is versioning. There are two different kinds of versioning: versioning of schemas for schema aware documents and versioning of documents (in the course of different modifications made to the document) for all XML documents. Both kinds of versioning present technical and usability challenges.

The XQuery Data Model provides an abstract representation of one or more XML documents or document fragments. The data model is based on the notion of a sequence. A sequence is an ordered collection of zero or more items. An item may be a node (document, element, attribute, namespace, text, processing-instruction or comment) or an atomic value. The input and output of XQuery are defined in terms of the XQuery Data Model. Because of this, the efficient handling of XQuery Data Model instances, when storying, retrieving, or searching, is very important. While an XML level serialization of XQuery Data Model instances is defined, it is not sufficient for applying an alternative XML serialization to XQuery Data Model instances. Thus it is important that an alternative XML serialization is rich enough to handle not only the XML serialization but also the XQuery Data Model serialization.

3.12.2 Domain & Stakeholders

Stakeholders are all providers of persistent store management systems and all providers of applications using and relying on persistent store management systems.

3.12.3 Justification

Persistent store technologies are widely and globally used. XML documents have been stored (and queried, and manipulated) in persistent stores for some number of years. The use of XML capable persistent stores is rapidly growing. And nobody expects any decline in this process.

The efficiency of XML handling in the persistent store is becoming critically important. The experience of persistent store management system providers shows that the use of alternative XML serializations of documents significantly improves performance, both size wise and processing efficiency wise.

A standard for alternative XML serialization is important for several reasons. Persistent stores interoperate with multiple software components (application servers, various client applications, mobile devices, etc.) produced by multiple vendors. Experience shows that conversions between multiple proprietary formats can be quite expensive. In addition, proprietary formats tend to change rapidly. This increases development costs. Also, handling of patent related issues is not usually helpful in delivering products on time.

3.12.4 Analysis

While persistent store vendors usually handle several data representations, the necessity to handle XML documents is a given. It is not a question whether applications using persistent stores can use other data representations or not. It is the reality of persistent store customers using XML in more and more applications. The use of XML in the persistent store, as described above, leads to following requirements:

  • Efficiently support the alternative XML serialization format for schema aware documents and be friendly to schema evolutions that maintain the validity of documents

  • Efficiently support certain operations in the persistent store and in the memory. These operations should include querying, updating, indexing, access to fragments, and fragment extraction.

  • Efficiently support XML schema datatypes. Transferring data in native binary format is typically more compact. For example, xsd:hexBinary data represented as binary bytes is half the size of the corresponding textual hexadecimal representation.

  • Efficiently support incremental processing where both partial loading and partial transmission are included.

  • The use of an alternative XML serialization should require minimal changes to existing application layers (only the underlying layers need to be changed). The impact to developers should be minimal.

  • Efficiently support multiple ordering. It should be possible to process documents both sequentially and based on subtrees.

3.12.5 Properties

The requirements in the previous section clearly define what properties of XML representations, alternative or current (XML 1.x), are important for handling XML documents in the persistent store. These properties include:

3.12.6 Alternatives

As it was mentioned in the justification section, persistent store vendors are required to handle XML to support multiple customer applications using XML. It would be absolutely inappropriate for persistent store vendors to offer their customers another format instead of XML.

3.12.7 References

None.

3.13 Business and Knowledge Processing

Editorial note: 20 October, 2004
Up to date as of 15 October email. Should be ready for second draft.

3.13.1 Description

Large XML documents flow through a business process. During the flow of a document, various business processes perform different, disjoint tasks. In addition, each distinct business process may only require portions of the entire XML document to complete their task. For example, a purchase order document may contain various customer information, shipping information, payment and billing information, etc. A business process is then defined where this document is passed to various entities, some being serviced by outside vendors, to approve and fill the purchase order. Another example is a business workflow scenario where XML documents are shared between business partners that contain the relevant workflow information.

These business processes vary greatly in complexity. One successful strategy to implementation and maintenance of complex applications is the concentration of business rules and business logic in an engine or subsystem of some kind. This subsystem is referenced in one or more ways by other parts of an application while supporting the architectural principle of non-redundant implementation of changing and evolving business aspects. These applications include web-based transactional applications implemented as one or more tiers or component layers which can be distributed in more than one thread, process, or server system. The thread of execution of an application for a specific user is usually represented by a "context" which is maintained for the entire user "session". Distributed application clusters frequently employ methods for distribution of session context data for load balancing and failure recovery. This means that the data that represents a session needs to be serialized and copied to another server or to a database on every transaction.

Business rules may be simple validation descriptions and global variable settings or they may consist of complex business logic or declarative logic programming. Logic and other intelligent programming can consist of expert system production rules, constraint languages, ontology processing, and the integration of multiple dissimilar processing engines. The key considerations are the efficient representation, exchange, maintenance, and evolution of data and knowledge. Often, a knowledge base references or is created from a large template and activity in the application produces many distributed incremental changes to represent state and acquired knowledge. Many standards and research efforts are layered on the representation of entities, relationships, and attributes in RDF and related methods. Optimization for this body of work would be very beneficial to this use case.

The representation of knowledge is a difficult problem to solve generally. It is however important to strive in this direction as a major technique in complex applications is the combination of engines that use different methods but cooperatively work with common data. Representation of knowledge can differ qualitatively from simple data because of the maintenance of meta-data such as "unknown" status, probability, history, and representation of what rules have changed, could change, or are dependent on a particular 'fact'. The alternation of activation of different knowledge or other application modules requires that all or part of the current, updated, session state be made available to those modules. This means there would be frequent interchange of large amounts of changing data, optionally represented as many small changes to a large template, between separately developed modules that need to access the same data in a standard fashion.

A specific example of this use case is of a web insurance or loan application. These kinds of applications are typically hosted on distributed web application clusters. Each processing step, usually a web form submission, involves reception by a web server, processing by an application agent, transmission via queues or web services to an application engine, and processing through many tiers of applications, servers, and communication links.

Application data in these kinds of applications will involve complex relationships and involve complex updates. The organization of data is very likely to be different from the multiple relationships between data elements which leads to a need to efficiently express arbitrary data structures. This need, along with the need to be able to efficiently update data objects while also supporting delta layers, results in a requirement to support explicit pointers that are automatically managed in the format. These pointers, which can be considered sticky virtual pointers, need to be able to be created and maintained within a format instance and have a fixed reference convention. Pointers need to be dereferenced about as quickly as referencing data items via direct paths. The pointers also need to be able to be represented as element or attribute values both internal and external to a format instance.

3.13.2 Domain & Stakeholders

This Use Case pertains to small, medium, and large businesses that utilize XML to support intra and inter business process workflow. An appropriate domain for this use case is users of complex applications, especially web, n-tier, or component based architectures with distributed processing. This use case is particularly suited for knowledge processing systems that include multiple processing subsystems. Important emerging work related to this use case include the Service Oriented Architecture (SOA), Web Services (WS), and Business Process Execution Language (BPEL) related activities.

3.13.3 Justification

Business processes often utilize a workflow where each step in the process only needs and processes certain subsets of the entire document. This results in the different steps in the business process performing disjoint tasks on random parts of the overall document. These disjoint tasks, since they are only processing a subset of the document, do not require the entire schema to perform their task.

Even though the entire document is not required at each step in the business process, the entire document is passed each time and, with current methods, fully parsed and processed. In addition, each business process requires a distinct and disjoint subset of the entire document to perform its task.

Existing methods, including data binding methods of rule engines, methods of distributing and replicating context, and communication models fall far short of possible efficiency levels. Lack of rich data format standardization with any efficiency is holding back intelligent component integration in production systems.

3.13.4 Analysis

The document passed to each entity can be large, meaning that large amounts of potentially unused data are passed to each endpoint. GZIP may be an option, however you would pay for the zip and unzip at each endpoint, potentially negating its benefits. In addition, each endpoint may require random access into the document. If the document was compressed with GZIP, it would make this type of access impossible without first uncompressing it.

To avoid the potential zip and unzip problem and the bandwidth problem, a binary encoding that represented the data in a more compact fashion could be used. The encoding would also need to allow each endpoint to randomly access and randomly update a subset of the entire original document.

In addition, the document can be modified along the way. This requires that each endpoint have a way to quickly modify a part of the document, then send it to the next step in the workflow process. Modifying the document in place is desirable because creating a DOM, making the change, and writing it back out would be too costly. To support a much wider variety of processing and data structure needs, an ability to support direct, maintained, non-computed pointers is needed.

In some cases, the requirements include that the alternate form of the data be more compact than the original XML. In other cases, processing speed improvement is paramount and compactness is a nice-to-have feature. The more compact form must be lossless. This means the alternate form can be converted back into the original XML with no differences. In addition, the creation of the alternate form and conversion back to XML must be efficient such that the entire business process does not take more time than it did with XML. Furthermore, the alternate encoding must allow for efficient random access and random update into the document such that the entire document does not have to be processed only to access a small subset contained at some specified location.

3.13.5 Properties

The following properties are essential:

  • Efficient Update: Enables low overhead modification of a data instance for modify/communicate cycles which are highly prevalent in this kind of use case.

  • Encryptable: Enables efficient encryption of desired information between applications that may not trust each other or the communication link.

  • Fragmentable: Supports query results and other parameter representation in a direct, efficient way.

  • No Arbitrary Limits: Application characteristics can vary widely and must not be constrained by format limitations.

  • Processing Speed: Efficiency directly controls the feasibility of implementation methods and standards in many instances of this use case.

  • Random Access: Random access is one component of Incremental Overhead that allows efficient processing relative to work.

  • Round Trippable: Applications in this use case will require XML 1.x interaction in both directions at many points.

  • Schema Instance Change Resilience: Application tiers, components, and logical schema areas will evolve independently. Supporting this ability is required in many instances to avoid the requirement to simultaneously update many components.

  • Self Contained: Related to the need for Schema Instance change resilience, many application situations cannot require a schema at all, especially for redundancy removal purposes.

  • Signable: Enables efficient verification of desired information between applications that may not trust each other or the communication link.

  • Support for Open Content Models: Many types of applications in this use case frequently need to include data in an instance that have a format that can be XML, but for which the schema cannot be known by the application.

  • Transcodable to XML: Applications in this use case will require XML 1.x interaction in both directions at many points.

  • Transport Independence: Application development, communication, and API mechanisms change constantly and a format that is tied to a current generation mechanism is not usable.

  • Verifyable Integrity: Applications must be able to determine integrity reliably when it is desired and not already known. Any overhead to do this must be optional or delayable, such as in lazy evaluation, if required by the application context.

  • Version Identification: Applications in this use case can be long-lived and work with data from other applications and entities. It is essential that the format version be readily identified.

The following properties are desirable:

  • Accelerated Sequential Access: This property is an alternative to full random access and partially solves important efficiency problems.

  • Data Model Versatility: Applications related to certain fields or using certain methods may have special data models specified or created to more directly model the problem. While these can be mapped to standard data types and even to strings, the ability to represent extended data models is helpful.

  • Compactness: Compactness benefits bandwidth, storage, footprint, and, indirectly, speed of processing. All of these are important to applications in this use case, but less so than speed and end-to-end efficiency.

  • Extension Points: See Data Model Versatility note.

  • Forward Compatibility (additional consideration): Due to the differing evolution of applications in this use case and the long term representation of data, forward compatibility allows instances to be useful even after evolution.

  • Human Readable/Editable/Deducible: This is a minor concern for this use case, mainly concerned with debugging, inspection, and auditing.

  • Single Conformance Class (additional consideration): Multiple conformance classes would complicate applications far more than any simplification for certain implementations would save.

  • Small Footprint: Efficient use of heap, stack, and code space is important not only for small devices that may be part of this use case, but even powerful servers that are servicing many users and contexts.

  • Specialized Codecs: Specialized codecs allow optimization of pervasive data and structure types related to application domains covered by this use case.

  • Streamable: One key aspect of business application performance is latency of processing of transactions. This is directly controlled by how soon processing can begin after data has begun being received. Streamability allows processing to progress as data is received which could lower overall latency.

  • Ubiquitous Implementation (additional consideration): Applications in this use case will increasingly communicate with disparate devices, services, customers, and other external entities. Applications may operate in severely degraded mode when processing XML 1.x equivalent representations and would benefit from ubiquitous implementation.

3.13.6 Alternatives

Passing the XML document in its uncompressed or gzipped compressed form are two options. Both have drawbacks. Passing the uncompressed document can clog up the network whereas passing the gzipped document can alleviate the bandwidth concerns, but be too costly to compress and decompress at each intermediary. An efficient binary encoding could mitigate the bandwidth issues while at the same time avoiding the compress/decompress steps at each intermediary.

3.14 XML Content-based Routing and Publish Subscribe

Editorial note: SP20 October, 2004
Latest version from Michael. The justification seems to go too much into a discussion about properties. Maybe some of this discussion can be moved to the Properties section.

3.14.1 Description

The ability to perform full XML message content-inspection and decision-making based on rules applied to the content is a requirement for a wide array of message processing systems. Routing, firewall (which is a filtering router), auditing gateway (a router that records), a trusted gateway (a firewall that strongly implements security policy), and the rich, high-level communications model that underlies IM/Presence (a router with sessions, virtualized endpoints, identity management, login, high level point to point and pubsub communication all at once) are all related by a number of requirements and similar operations. One strong part of the essence of presence/IM is that it relies on pub/sub of presence information.

Some of these routers and routers+ simply inspect messages, possibly just a header or possibly deeper, before sending them to their new destination. Other situations, even in more or less pure routing situations like NAT/PAT, modify, repackage, or otherwise update messages. The concept of header may be problematic. With XML, it often simply means earlier XML ranges or subtrees rather than going deeper. If a reasonably opaque payload is used, the concept of header becomes more concrete, but that subset is not a relevant distinction for most XML messaging systems.

Layer 7 application-oriented routing devices and content publish subscribe systems are currently in use in the financial sector for performance-critical applications such as trade routing, where transactions may be routed from the front office to the back office, to brokers, and to exchanges. Routing or subscription criteria may be based on information in XML messages such as trade, client, task or ticket type, for example. Another application example is the dissemination of data services by mobile carriers based on granular subscriptions by a mobile user. In general, content-based routing and publish subscribe are extremely useful paradigms for all kinds of XML message dissemination and are relevant for Web Services, distributed computing, instant messaging, tactical military operations, government services, homeland security, and many types of online publishing. XML-based routing and publish subscribe as filtering technologies can be critical to reducing bandwidth utilization on LANs as well as WANs.

3.14.2 Domain & Stakeholders

This use case applies to many domains requiring deep content inspection of XML messages in a message processing environment including firewall, routing, publish/subscribe, security gateways, and instant messaging. These applications are relevant to almost all commercial and public entities concerned with enterprise-scale information and transactional infrastructures.

3.14.3 Justification

There exists no approach in software today which can meet all usage scenarios of this use case using XML 1.X. An alternate XML format could make such an implementation in software feasible by increasing the potential performance of the operations needed in a content-based routing or publish subscribe system. Reduction or elimination of parsing time and the ability to very rapidly find and test XML infoset items is necessary. For example, an alternative XML format with support for random access could enable extraction of data from the message with only a minimal lookup time and could potentially support the low latency needed by routing and the execution of the large numbers of rules needed for content-based publish subscribe. The Random Access property happens to map very strongly to this use case since XPath queries can be used for expressing the logical address of indexed content and the routing and subscriber rules. Another property of an alternate XML format which could assist in reducing parsing time is Accelerated Sequential Access. Compatibility with data binding could accelerate the matching of routing and subscriber rules. For example, such rules often have integer, floating point, and data type comparisons which would be more rapidly evaluated if the cost of conversion to these types in the programming language could be minimized.

XML content-based routing and publish subscribe systems may require additional processing such as message transformation. For example, it may be necessary to delete, insert, or change information used to route the message to ensure that the message does not loop back to the same processing node. WS-Eventing specifies the possibility of inserting a subscriber-specified SOAP header in a message before it is passed to the subscriber. Properties of an alternate XML format which would allow these types of message transforms to be performed very efficiently are and the ability to update using deltas. In some cases, it will be important to be able to support Self-Contained Subtrees so that some sub-segment of the message can be extracted and processed efficiently. One example might be filtering text or attachments through virus filters. Message security may also be a concern in routing and publish subscribe systems. An alternate XML format which supports properties for efficient security operations, such as signing and encryption, will also be useful in these type of scenarios.

3.14.4 Analysis

Routers at the edge of the LAN responsible for distributing traffic coming from the WAN may operate with a relatively limited set of content inspection rules and small routing tables, but must deal with considerable quantities of message traffic and require very short message processing latency. Publish Subscribe applications within the LAN will typically deal with less volume and may have more relaxed latency requirements compared to routers, but may need to deal with a large number of subscribers generating a very large set of rules to be evaluated against every published XML message.

Both XML routers and XML publish subscribe systems have in common that the greatest part of processing overhead comes from simply parsing each XML message. Typically, each message is converted into an in-memory representation (such as DOM) prior to evaluating routing or subscriber rules.

Evaluation of the rules may also require a great amount of processing power; most existing systems cannot, in fact, support complex rules because of the processing cost. XML Routers may be confined to processing routing information in message headers. XML Publish Subscribe applications may be unable to achieve dynamic subscription matching with full decoupling of publishers and subscribers. Adequate performance can only be achieved when subscribers are limited to matching a fixed set of keywords recognized by the publisher. The inability to perform full content inspection because of its cost in performance has limited architectures to static, inflexible, and narrowly-targeted systems which do not meet current needs.

XPath is sometimes used for expressing routing and subscribers rules. Its ability to locate any content item in an XML message and to apply additional node filtering criteria corresponds closely to the requirements for rule-based classification of XML content. As there are many tools for evaluating XPaths over DOM, we can conceptualize the problem of XML content-based routing and publish subscribe as a process of making decisions from the results of XPath evaluation.

3.14.5 Properties

Processing Speed is an essential property in for this use case. Other desirable properties are: Accelerated Sequential Access, Compactness, Efficient Update, Encryptable, , Random Access, Signable and Streamable.

3.14.6 Alternatives

Streaming XPath is a more efficient XPath implementation usually built on a SAX parser and limited to some subset of XPath expressions which can be evaluated in a single pass over an XML document. Streaming XPath may be an alternative for improving the performance of routing or publish-subscribe rules described with XPaths expressions. The performance gain may be sufficient for some usage scenarios but still will not approach the network speed required for inspection and direction of XML messages entering into a LAN.

Purpose-built hardware devices can perform XML content inspection and make firewall, routing, or publish decisions at throughput rates which may be sufficient for this use case. This alternative is proprietary and costly and is therefore of limited applicability. In addition, the problem is sufficiently challenging even for hardware-based solutions. An alternate XML format could have the effect of increasing the performance of hardware-based solutions, further extending the capacity of such solutions and also lowering their cost.

3.14.7 References

  1. [WS-Eventing]

3.15 Web Services Routing

Editorial note: SPOctober 20, 2004
Latest version from Peter. Minor edits. Should be ready for second draft.

3.15.1 Description

When SOAP messages flow in a network, rarely do they flow from the source system to the destination system directly. More often than not, they flow through various intermediaries between the two endpoints. The intermediaries route the message between systems for various reasons. These include routing the SOAP message to the appropriate endpoint in order to fulfill the request, routing the message based on system availability, and so on.

3.15.2 Domain & Stakeholders

This use case applies to medium to large business that utilize XML and/or Web Services in SOAP messaging applications.

3.15.3 Justification

In a system that is dynamically routing messages to support a real-time messaging application, the size of each message as well as the time it takes to determine the next intermediary is important for the overall system performance and response time. The smaller the message, the less time it takes to transmit it between intermediaries, thus freeing up the remaining bandwidth. In addition, an alternate encoding has the potential to make the process of determining the next intermediary more efficient.

3.15.4 Analysis

In some messaging systems, the endpoint for a message is not determined by the client, but rather in real time by intermediaries. An intermediary will route the message based on various factors. These factors include, but are not limited to, message type, message content, system availability, etc.

An intermediary looks only at the SOAP header of the message to determine the route the message will take. This is a more efficient process than reading the entire message, creating an in memory representation (DOM) and digging into the message to determine its route. The more efficient mechanism is to have the intermediary only look at some header information at the beginning of a message in order to make an intelligent decision about how to route it.

It is important to note that the message, when received by an intermediary for routing, could be digitally signed and/or encrypted. These issues need to be considered for the alternate encoding in the same ways they are considered for XML today.

An alternate encoding needs to support the ability for this type of message routing to take place in, at a minimum, without loss of performance. The intermediary needs to examine the alternate encoding’s header to determine the routing of the message. This, ideally, requires random access into the message header to determine the routing information.

3.15.5 Properties

The following properties are essential:

The following properties are desirable:

3.15.6 Alternatives

Various other technologies can be used to achieve routing. However, they would be more brittle and often more proprietary compared to XML. Being able to express routing information in the header of a message in XML ensures a level of interoperability and non-brittleness that XML is known for.

3.15.7 References

  1. [WS-Addressing]

  2. [WS-Routing-ref]

3.16 Versioning Systems

Editorial note: MC20 October, 2004
New use case - current as of 15 October email

3.16.1 Description

Data such as source code, documents, configuration, and publications change over time in ways that frequently need to be managed as explicit versions with support for highlighting changes made. Systems that track and report on changes rely mainly on automatic text differencing and opaque binary versioning. It is advantageous that changes in a format instance should tend to localize changes to support such systems.

Versioning systems include source code control, content management systems, and generalized enterprise configuration management systems. These systems track documents, data files, source code, binary objects that may or may not be fundamentally opaque, and database records.

One widely used example of automatic text differencing is CVS (the Concurrent Versions System). CVS handles both text files and binary files. Binary files are handled in a trivial, and therefore inefficient way by simply storing a complete copy of every version in a single large file. Text versioning has features that directly support the characteristics of source code such as: changes are primarily inserted, deleted, or changed lines; infrequent re-flow of text (all lines involved would be considered changed); spacing changes can be ignored if desired; magic comments can be updated to include change comments, document version, and history. Detection and characterization of differences are automatically computed by the versioning system. Traditionally, only ASCII text was handled explicitly. Many other source code control systems exist; most have similar characteristics. Some systems version binary data by noting what parts have changed which provides an opaque binary versioning. Users of these systems would benefit from a system where differences in binary data were represented in a way that the versioning system was aware of the actual boundaries of the changed data. A versioning system could analyze XML, or a new format related to XML, and characterize differences. Direct awareness of externally managed changes might be at least as useful.

Changes such as non-deterministic choice of alternative encodings of logical infoset areas that have not changed can cause inefficient and misleading differences. In other words, if the instance data for a format changes even when the infoset data has not, a versioning system that is expecting byte preservation may mistakenly detect differences.

3.16.2 Domain & Stakeholders

The domain of this use case is any application that records versions and differences in text, source code, or other data objects and makes those differences available later. The canonical example is a source control system such as CVS, but this case is very similar to content management and other applications. Stakeholders include any user or administrator of such an application that values storage and processing efficiency and the availability of meaningful differences between versions.

3.16.3 Justification

Versioning systems have been found to be very important to ongoing creative activities. Beyond simple source code, the management of differences between versions of a data object becomes rapidly less useful to users. A format that may be used for storage of important data in such systems must have characteristics that make difference detection and tracking straightforward.

3.16.4 Analysis

A new format will not be directly usable with a text differencing system if the format is not expressed as text-compatible data. Formats may have features or design tradeoffs that would enable a versioning system to be aware of changes and differences in a deep way rather than being limited to an opaque, and possibly misleading binary differencing method. An example might be a format that supports strongly localized changes. An illustration of this would be a format that supports deltas and supports deltas being appended cleanly to an existing file. In this case, a versioning system could efficiently track, separate, analyze, and report on changes with minimal effort.

3.16.5 Properties

The following properties are essential:

  • Localized Changes (to be defined in next draft of Properties document): A format exhibits localized changes if modifications that are near with respect to the data model level translate into modifications that are near with respect to the format level. This enhances the efficiency of difference detection and storage.

The following properties are desirable:

  • Support for Deltas (to be defined in next draft of Properties document): Provides one complete, compute-free method of identifying and recording differences.

  • Byte Preserving: This property allows shallow analysis to detect only real changes.

  • Compactness: Improves efficiency of storage.

  • Efficient Update: May support versioning system annotation, combination of differences, and merging.

  • Processing Speed: Helps speed up versioning activities.

  • Self Contained: Versioning objects requires that complete information must be available to the versioning system which may obviate schema-compiled format modes.

  • Verifyable Integrity: It would be useful for Versioning systems to verify integrity before storage.

3.16.6 Alternatives

The alternatives to a new format for this property are the continued use of opaque binary versioning and text-based XML differencing. Few or no source code control systems have direct support for XML differencing, although it is feasible. XML 1.x and many other formats lack a complete, efficient way to represent a difference which means that difference detection is a computational task.

3.16.7 References

  1. [CVS]

4 Summary

This section includes observations concerning overlaps and other patterns in use case requirements.

Editorial note: MC4 June, 2004
Perhaps we could have a table or something to indicate groups of related use cases, where the grouping is based on overlapping requirements?

5 References

XBC Properties
XML Binary Characterization Properties (See http://www.w3.org/TR/xbc-properties/.)
XML 1.0
Extensible Markup Language (XML) 1.0 (See http://www.w3.org/TR/REC-xml/.)
XML 1.1
Extensible Markup Language (XML) 1.1 (See http://www.w3.org/TR/xml11/.)
Extensible 3D (X3D) Graphics
Extensible 3D (X3D) Graphics (See http://www.web3D.org/x3d/.)
Fast Web Services
Fast Web Services (See http://java.sun.com/developer/technicalArticles/WebServices/fastWS/.)
SOAP Performance
Investigating the Limits of SOAP Performance for Scientific Computing (See http://www.extreme.indiana.edu/xgws/papers/soap-hpdc2002/soap-hpdc2002.pdf.)
MTOM
SOAP Message Transmission Optimization Mechanism (See http://www.w3.org/TR/2004/WD-soap12-mtom-20040209/.)
XOP
SOAP Message Transmission Optimization Mechanism (See http://www.w3.org/TR/2004/WD-xop10-20040209/.)
SOAP with Attachments
SOAP Messages with Attachments (See http://www.w3.org/TR/SOAP-attachments.)
WS-I Attachments Profile
WS-I: Attachments Profile Version 1.0 (See http://www.ws-i.org/Profiles/Basic/2003-08/AttachmentsProfile-1.0.pdf.)
DVB
Digital Video Broadcasting (See http://www.dvb.org/.)
TV Anytime
TV Anytime (See http://tv-anytime.org/.)
ARIB
Association of Radio Industries and Businesses (See http://www.arib.or.jp/.)
MPEG-7
MPEG-7 (See http://www.iso.org/iso/en/prods-services/popstds/mpeg.html.)
PDF
PDF Reference, 4th Ed. (See http://partners.adobe.com/asn/acrobat/sdk/public/docs/PDFReference15_v6.pdf .)
SOAP in Real-Time Trading Systems
Evaluating SOAP for High Performance Business Application: Real-Time Trading Systems. (See http://www2003.org/cdrom/papers/alternate/P872/p872-kohlhoff.html.)
MIME
The MIME Multipart/Related Content-type (See http://www.faqs.org/rfcs/rfc2387.html.)
Mobile SVG
Mobile SVG Profiles: SVG Tiny and SVG Basic (See http://www.w3.org/TR/SVGMobile/.)
Mobile XHTML
XHTML Mobile Profile (See http://www.openmobilealliance.org/tech/affiliates/wap/wap-277-xhtmlmp-20011029-a.pdf.)
SMIL
Synchornized Multimedia Integration Language (SMIL 2.0) (See http://www.w3.org/TR/smil20/.)
G-XML
G-XML (See http://gisclh.dpc.or.jp/gxml/contents-e/.)
MMS
Multimedia Messaging Services (See http://www.3gpp.org/ftp/Specs/html-info/26140.htm.)
WAP WBXML
Binary XML Content Format Specification (See http://www.openmobilealliance.org/tech/affiliates/wap/wap-192-wbxml-20010725-a.pdf.)
Comparing SOAP Performance
Comparing SOAP Performance for Various Encodings, Protocols, and Connections (See http://www.cs.helsinki.fi/u/jkangash/soap-performance.pdf.)
SOAP Performance in Wireless Networks
Web Services in Wireless Networks - What Happpened to the Performance? (See http://conferences.computer.org/icws/2003/.)
WellLogML
POSC WellLogML Project (See http://www.posc.org/ebiz/WellLogML/V1.0/index.html.)
POSC
Petrotechnical Open Standards Consortium (See http://www.posc.org/.)
ACORD
ACORD - Global Insurance Standards (See http://www.acord.org.)
HL7
Health Level 7 (See http://www.hl7.org/.)
RosettaNet
RosettaNet - ebusiness Standards for the Global Supply Chain (See http://www.rosettanet.org/RosettaNet/Rooms/DisplayPages/LayoutInitial/.)
X3D Markets
Application Markets for X3D-Based Open Standards (See http://www.web3d.org/applications/.)
X3D RFP
X3D Compressed Binary Encoding RFP (See http://www.web3d.org/x3d/binary/X3dBinaryRFP.html.)
W3C PP
W3C Patent Policy (See http://www.w3.org/Consortium/Patent-Policy/.)
RFC 3920
Extensible Messaging and Presence Protocol (XMPP): Core (See http://www.rfc-archive.org/getrfc.php?rfc=3920.)
RFC 3921
Extensible Messaging and Presence Protocol (XMPP): Instant Messaging and Presence (See http://www.rfc-archive.org/getrfc.php?rfc=3921.)
RFC 3922
Mapping the Extensible Messaging and Presence Protocol (XMPP) to Common Presence and Instant Messaging (CPIM) (See http://www.rfc-archive.org/getrfc.php?rfc=3922.)
RFC 3923
End-to-End Signing and Object Encryption for the Extensible Messaging and Presence Protocol (XMPP) (See http://www.rfc-archive.org/getrfc.php?rfc=3923.)
Jabber
Jabber (See http://www.jabber.com/.)
JEPs
Jabber: Jabber Enhancement Proposals (JEPs) (See http://www.jabber.org/jeps.)
Jabber Stream Compression
Jabber: Stream Compression JEP (See http://www.jabber.org/jeps/jep-0138.html.)
WS-Addressing
Web Services Addressing (See http://www.w3.org/Submission/ws-addressing/.)
WS-Routing-ref
Web Services Routing Protocol (See http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnglobspec/html/ws-routing.asp.)
WS-Eventing
Web Services Eventing (See http://ftpna2.bea.com/pub/downloads/WS-Eventing.pdf.)
OASIS WS BPEL
OASIS Web Services Business Process Execution Language TC (See http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wsbpel.)
CVS
Concurrent Versions System (See http://www.cvshome.org.)
SEGY
Society of Exploration Geophysicts (See http://www.seg.org/publications/tech-stand/.)

A Acknowledgments (Non-Normative)

The editors would like to thank the contributers...