Ink Markup Language (InkML)

1 Overview

As more electronic devices with pen interfaces have and continue to become available for entering and manipulating information, applications need to be more effective at using this method of input. Handwriting is a powerful and versatile input modality that is very familiar for most users since everyone learns to write in school. Hence, users will tend to use this as a mode of input and control when available.

A pen-based interface is enabled by a device that allows movements of the pen to be captured as digital ink. A number of methods may be used for ink capture, including those based on radio frequency, optical tracking, physical pressure, or other technologies. Digital ink can be passed on to recognition software that will convert the pen input into appropriate computer actions. Alternatively, the handwritten input can be organized into ink documents, notes or messages that can be stored for later retrieval or exchanged through telecommunications means. Such ink documents are appealing because they capture information as the user composed it, including text in any mix of languages and drawings such as equations and graphs.

Hardware and software vendors have typically stored and represented digital ink using proprietary or restrictive formats. The lack of a public and comprehensive digital ink format has severely limited the capture, transmission, processing, and presentation of digital ink across heterogeneous devices developed by multiple vendors. In response to this need, the Ink Markup Language (InkML) provides a simple and platform-neutral data format to promote the interchange of digital ink between software applications.

InkML supports a complete and accurate representation of digital ink. In addition to the pen position over time, InkML allows recording of information about device characteristics and detailed dynamic behavior to support applications such as handwriting recognition and authentication. For example, there is support to record additional information such as pen tilt and pen tip force (often referred to as "pressure") and information about the recording device such as accuracy and dynamic distortion. InkML also provides features to support rendering of digital ink captured optically to approximate the original appearance. For example, stroke width and color information can be recorded.

InkML provides means for extension. By virtue of being an XML-based language, users may easily add application-specific information to ink files to suit the needs of the application at hand.

1.1 Uses of InkML

With the establishment of a non-proprietary ink standard, a number of applications, old and new, are expanded where the pen can be used as a very convenient and natural form of input. Here are a few examples.

• Ink Messaging

  Two-way transmission of digital ink, possibly wireless, offers mobile-device users a compelling new way to communicate. Users can draw or write with a pen on the device's screen to compose a note in their own handwriting. Such an ink note can then be addressed and delivered to other mobile users, desktop users, or fax machines. The recipient views the message as the sender composed it, including text in any mix of languages and drawings.

• Ink and SMIL

  A photo taken with a digital camera can be annotated with a pen; the digital ink can be coordinated with a spoken commentary. The ink annotation could be used for indexing the photo (for example, one could assign different handwritten glyphs to different categories of pictures).

• Ink Archiving and Retrieval

  A software application may allow users to archive handwritten notes and retrieve them using either the time of creation of the handwritten notes or the tags associated with keywords. The tags are typically text strings created using a handwriting recognition system.

• Electronic Form-Filling

  In support of natural and robust data entry for electronic forms on a wide spectrum of keyboardless devices, a handwriting recognition engine developer may define an API that takes InkML as input.

• Pen Input and Multimodal Systems

  Robust and flexible user interfaces can be created that integrate the pen with other input modalities such as speech. Higher robustness is achievable because cross-modal redundancy can be used to compensate for imperfect recognition on each individual mode. Higher flexibility is possible because users can choose the most appropriate from among various modes for achieving a task or issuing commands. This choice might be based on user preferences, suitability for the task, or external conditions. For instance, when noise in the environment or privacy is a concern, the pen modality is preferred over voice.

1.2 Elements

The current InkML specification defines a set of primitive elements sufficient for all basic ink applications. All content of an InkML document is contained within a single <ink> element. The fundamental data element in an InkML file is the <trace>. A trace represents a sequence of contiguous ink points, where each point captures the values of particular quantities such as the X and Y coordinates of the pen's position. A sequence of traces accumulates to meaningful units, such as characters, words or diagrams. The

In its simplest form, an InkML file with its enclosed traces looks like this:

<ink>
   <trace>
     10 0, 9 14, 8 28, 7 42, 6 56, 6 70, 8 84, 8 98, 8 112, 9 126, 10 140,
     13 154, 14 168, 17 182, 18 188, 23 174, 30 160, 38 147, 49 135,
     58 124, 72 121, 77 135, 80 149, 82 163, 84 177, 87 191, 93 205
   </trace>
   <trace>
     130 155, 144 159, 158 160, 170 154, 179 143, 179 129, 166 125,
     152 128, 140 136, 131 149, 126 163, 124 177, 128 190, 137 200,
     150 208, 163 210, 178 208, 192 201, 205 192, 214 180
   </trace>
   <trace>
     227 50, 226 64, 225 78, 227 92, 228 106, 228 120, 229 134,
     230 148, 234 162, 235 176, 238 190, 241 204
   </trace>
   <trace>
     282 45, 281 59, 284 73, 285 87, 287 101, 288 115, 290 129,
     291 143, 294 157, 294 171, 294 185, 296 199, 300 213
   </trace>
   <trace>
     366 130, 359 143, 354 157, 349 171, 352 185, 359 197,
     371 204, 385 205, 398 202, 408 191, 413 177, 413 163,
     405 150, 392 143, 378 141, 365 150
   </trace>
</ink>

These traces consist simply of X and Y value pairs, and may look like this when rendered:

Figure 1 shows a trace of a sampled handwriting signal. The dots mark the sampling positions which were interpolated by the blue line. Green points represent pen-downs whereas red dots indicate pen-ups.

Information about the device used to collect the ink (e.g., the sampling rate and resolution) may be specified with the <inkSource> element.

Ink traces can have certain attributes such as color and width, writer identification, pen modes (eraser vs writing), and so on. These and other attributes are captured using the <brush> element. Traces that share the same characteristics, such as being written with the same brush, can be grouped together with the <traceGroup> element.

In all appropriate cases, the InkML specification defines default values for elements that are not specified, and rules that establish the scope of a given attribute.

Finally, the InkML specification is limited in scope: It is currently oriented to fixed Cartesian coordinate systems, it does not support sophisticated compression of trace data, and it does not support non-ink events (although the later could be handled via annotations).

1.3 Exchange Modes

Most ink-related applications fall into two broad categories: "Streaming" and "Archival". Archival ink applications capture and store digital ink for later processing, such as document storage/retrieval applications and batch forms processing . In these applications, an entire <ink> element is written prior to processing. For ease of implementation, it is recommended that, in archival mode, referenced elements be defined inside a declaration block using the <definitions> element.

Streaming ink applications, on the other hand, transmit digital ink as it is captured, such as in the electronic whiteboard example mentioned above. In order to support a streaming style of ink markup generation, the InkML language supports the notion of a "current" state (e.g., the current brush) and allows for incremental changes to this state.

<ink xmlns="http://www.w3.org/2003/InkML" 
     documentID="uuid:6B29FC40-CA47-1067-B31D-00DD010662DA"/>
...
</ink>

3 Traces and Trace Formatting

Traces are the basic element used to record the trajectory of a pen as a user writes digital ink. More specifically, these recordings describe sequences of connected points. On most devices, these sequences of points will be bounded by pen contact change events (pen-up and pen-down), although some devices may simply record proximity and force data without providing an interpretation of pen-up or pen-down state.

The simplest form of encoding specifies the X and Y coordinates of each sample point. For compactness, it may be desirable to specify absolute coordinates only for the first point in the trace and use delta-x and delta-y values to encode subsequent points. Some devices record acceleration rather than absolute or relative position; some provide additional data that may be encoded in the trace, including Z coordinates or tip force, or the state of side switches or buttons.

These variations in the information available from different ink sources, or needed by different applications, are supported in InkML through the <traceFormat> and <trace> elements. The <traceFormat> element specifies the encoding format for each sample of a recorded trace, while <trace> elements are used to represent the actual trace data. If no <traceFormat> is specified, a default encoding format of X and Y coordinates is assumed.

Traces generated by different devices, or used in differing applications, may contain different types of information. InkML defines channels to describe the data that may be encoded in a trace.

A channel can be characterized as either regular, meaning that its value is recorded for every sample point of the trace, or intermittent, meaning that its value may change infrequently and thus will not necessarily be recorded for every sample point. X and Y coordinates are examples of likely regular channels, while the state of a pen button is likely to be an intermittent channel.

3.1 Trace Formats

3.1.1 <traceFormat> element

Attributes:

Contents:

The <traceFormat> element describes the format used to encode points within <trace> elements. In particular, it defines the sequence of channel values that occurs within <trace> elements. The order of declaration of channels in the <traceFormat> element determines the order of appearance of their values within <trace> elements.

Regular channels appear first in the <trace>, followed by any intermittent channels. Correspondingly, the <traceFormat> element contains an ordered sequence of <channel>s, giving the regular channels (if any), followed by an optional <intermittentChannels> section. The order of the coordinates in each point of a trace is determined by the order of the <channel> elements in the trace format, including those from the intermittent channels part.

3.1.2 <intermittentChannels> element

Attributes:

Contents:

The <intermittentChannels> lists those channels whose value may optionally be recorded for each sample point. The order of the enclosed channel declarations gives the order of the intermittent channel data samples within traces having this format.

3.1.3 <channel> element

Attributes:

Contents:

Channels are described using the <channel> element, with various attributes.

The required name attribute specifies the interpretation of the channel in the trace data. The following channel names, with their specified meanings, are reserved:

The type attribute defines the encoding type for the channel (either boolean, decimal, or integer). If type is not specified, it defaults to decimal.

A default value can be specified for the channel using the default attribute; the use of default values within a trace is described in the next section. If no default is specified, it is assumed to be zero for integer and decimal-valued channels, and false for boolean channels.

Typically, a channel in the <traceFormat> will map directly to a corresponding channel provided by the digitizing device, and its values as recorded in the trace data will be the original channel values recorded by the device. However, for some applications, it may be useful to store normalized channel values instead, or even to remap the channels provided by the digitizing device to different channels in the trace data. This correspondence between the trace data and the device channels is recorded using a <mapping> element (described in the Mappings section) within the <channel> element. If no mapping is specified for a channel, it is assumed to be unknown.

3.1.4 Orientation Channels

The channels OTx, OTy, OA, OE and OR are defined for recording of pen orientation data. Implementers may choose to use either pen azimuth OA and pen elevation OE, or alternatively tilt angles OTx and OTy. The latter are the angles of projections of the pen axis onto the XZ and YZ planes, measured from the vertical. It is often useful to record the sine of this angle, rather than the angle itself, as this is usually more useful in calculations involving angles. The <mapping> element can be employed to specify an applied sine transformation.

The third degree of freedom in orientation is generally defined as the rotation of the pen about its axis. This is potentially useful (in combination with tilt) in application such as illustration or calligraphy, and signature verification.

Figure 2a displays the pen orientation using Azimuth and Elevation. The origin of the Azimuth is at the Y-axis. Azimuth increases anticlockwise up to 360 degrees. The origin of Elevation is located within the XY-plane. Elevation increases up to 90 degrees, at which point the pen is perpendicular to the XY-plane.

Figure 2b explains the definition of the Tilt-X and the Tilt-Y angles. For both the origin is along the Z-axis. Tilt-X increases up to +90 degrees for inclinations along the positive X-axis and decreases up to -90 degrees for inclinations along the negative X-axis. Respectively, Tilt-Y is defined for pen inclinations along the Y-axis.

Figure 3a displays the pen orientation decomposition as functions of Azimuth/Elevation or alternatively as function of Tilt-X/Tilt-Y. Thereby, Elevations of the pen which are mapped to the XZ- and to the YZ- plane lead to Tilt-X and Tilt-Y.

Figure 3b shows the Rotation of the pen along its longitudinal axis.

3.1.6 Width Channel

3.1.7 Time Channel

The time channel allows for detailed recording of the timing information for each sample point within a trace. This can be useful if the digitizing device has a non-uniform sampling rate, for example, or in cases where duplicate point data is removed for the sake of compactness.

The time channel can be specified as either a regular or intermittent channel. When specified as a regular channel, the single quote prefix can be used to record incremental time between successive points. Otherwise, the value of the time channel for a given sample point is defined to be the timestamp of that point in the units and frame of reference specified by its corresponding <inkSource> description (more precisely, by the <traceFormat> element for the channel).

As with the other predefined channels, the meaning of the integer or decimal values recorded by the time channel in a given trace is defined by the <inkSource> information associated with the trace's traceFormat. In the case of the time channel, its <channel> element contains both a units and respectTo attribute.

The units attribute gives the units of the recorded time values, and the relativeTo attribute describes the frame of reference for those recorded values. The value of the relativeTo attribute is a reference to a time stamp. If it is not given, the time channel values are relative to the beginning timestamps of the individual traces in which they appear.

   <channel name="T" 
               type="integer"
               units="ms"
               respectTo="#ts1"/>

If no <inkSource> information is provided, or if no value is specified for the respectTo attribute, the ink processor cannot make any assumption about the relative timing of points within different traces. Likewise, if no units are specified, no assumption can be made about the units of the time channel data.

3.1.8 User Defined Channels

In addition, user-defined channels are allowed, although their interpretation is not required by conforming ink markup processors.

3.1.9 Specifying Trace Formats

The following example defines a <traceFormat> which reports decimal-valued X and Y coordinates for each point, and intermittent boolean values for the states of two buttons B1 and B2, which have default values of F ("false"):

<traceFormat xml:id="xyb1b2">
   
   <channel name="X" type="decimal">
      <mapping type="identity"/>
   </channel>
   <channel name="Y" type="decimal">
      <mapping type="identity"/>
   </channel>
   
   <intermittentChannels>
      <channel name="B1" type="boolean" default="F">
         <mapping type="identity"/>
      </channel>
      <channel name="B2" type="boolean" default="F">
         <mapping type="identity"/>
      </channel>
   </intermittentChannels>
</traceFormat>

The appearance of a <traceFormat> element in an ink markup file both defines the format and installs it as the current format for subsequent traces (except within a <definitions> block). The id attribute of a <traceFormat> allows the format to be reused by multiple contexts (see the Context section). If no <traceFormat> is specified, the following default format is assumed:

<traceFormat xml:id="DefaultTraceFormat">
   
   <channel name="X" type="decimal"/>
   <channel name="Y" type="decimal"/>
   
</traceFormat>

Thus, in the simplest case, an InkML file may contain nothing but <trace> elements.

3.2 Traces

Contents:

trace ::=
     point ("," point)*  ","?

point ::=
     value+

value ::=
     qualifier? "-"? decimal | hex | "T" | "F" | "*" | "?"

decimal ::=
     digit+ ("." digit*)? | "." digit+

hex ::=
     "#" (digit | "A" | "B" | "C" | "D" | "E" | "F")+

qualifier ::=
     "!" | "'" | """

digit ::=
     "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"

wsp ::=
     #x20 | #x9 | #xD | #xA

The <trace> element is used to record the data captured by the digitizer. It contains a sequence of points encoded according to the specification given by the <traceFormat> element.

The type attribute of a <trace> indicates the pen contact state (either "penUp" or "penDown") during its recording. A value of "indeterminate" is used if the contact-state is neither pen-up nor pen-down, and may be either unknown or variable within the trace. For example, a signature may be captured as a single indeterminate trace containing both the actual writing and the trajectory of the pen between strokes.

Regular channels may be reported as explicit values, differences, or second differences: Prefix symbols are used to indicate the interpretation of a value: a preceding exclamation point (!) indicates an explicit value, a single quote (') indicates a single difference, and a double quote prefix (") indicates a second difference. If there is no prefix, then the channel value is interpreted as explicit, difference, or second difference based on the last prefix for the channel. If there is no last prefix, the value is interpreted as explicit.

A second difference encoding must be preceded by a single difference representation; which, in turn, must be preceded with an explicit encoding.

All traces must begin with an explicit value, not with a first or second difference. This is true of continuation traces as well. This allows the location and velocity state information to be discarded at the end of each trace, simplifying parser design.

Intermittent channels are always encoded explicitly, i.e. the qualifiers ' and " are not allowed.

Both regular and intermittent channels may be encoded with the wildcard character "*". This wildcard character means either that the value of the channel remains at the previous channel value (if explicit), or that the channel continues integrating with the previous velocity or acceleration values, as appropriate.

<trace> 11 12 9, 21 22 ? T, 31 32, 41 42 5, 51 52 ? F</trace>

Booleans are encoded as "T" or "F".

For each point in the trace, regular channel values are reported first in the order given by the <traceFormat>. If any intermittent values are reported for the point, they are given next, in the order they are specified in the <traceFormat>. Unreported intermittent channels are interpreted as though they were given by the wildcard "*".

Here is an example of a trace of 11 points, using the following traceFormat:

<traceFormat>
   
   <channel name="X" type="decimal"/>
   <channel name="Y" type="decimal"/>
   
   <intermittentChannels>
      <channel name="B1" type="boolean" default="F"/>
      <channel name="B2" type="boolean" default="F"/>
   </intermittentChannels>
</traceFormat>

<trace id = "id4525abc">
   1125 18432,'23'43,"7"-8,3-5,7 -3,6 2,6 8,3 6 T,2 4*T,3 6,3-6 F F
</trace>

The trace is interpreted as follows:

An ink markup generator might also include additional whitespace formatting for clarity. The following trace specification is identical in meaning to the more compact version shown above:

<trace id = "id4525abc">
  1125 18432,
  '23 '43,
  "7 "-8,
  3 -5,
  7 -3,
  6 2,
  6 8,
  3 6 T,
  2 4 * T,
  3 6,
  3 -6 F F
</trace>

3.3.1 <traceGroup> element

Attributes:

Contents:

The <traceGroup> element is used to group successive traces which share common characteristics, such as the same <traceFormat>. The brush and context sections describe other contextual values that can be specified for a <traceGroup>. In the following example the two traces enclosed in the <traceGroup> share the same brush (see the Brushes section for a description of brushes).

<traceGroup brushRef="#penA">
   <trace>...</trace>
   <trace>...</trace>
</traceGroup>

The <traceGroup> element may be used for various purposes, such as the containment of traces according to their properties at the time of capture. The element may be nested, and it may be used as a generic grouping mechanism, e.g. for the semantic labelling of traces.

Trace groups are the primary mechanism for assigning <context> to traces in archival ink markup. For additional details about this usage, see the Archival Applications section.

<ink>
   <trace xml:id="L1">911 912, 921 922, 931 932</trace>

   <traceGroup xml:id="L2">
       <trace>111 112, 121 122</trace>
       <traceGroup xml:id="L2-Larry">
           <trace>221 212, 221 222</trace>
           <trace>311 312, 321 322</trace>
       </traceGroup>
       <trace>411 412, 421 422</trace>
       <traceGroup>
           <traceGroup>
               <trace xml:id="L2-Moe">521 512, 521 522</trace>
               <trace>611 612, 621 622</trace>
           </traceGroup>
       </traceGroup>
       <trace>711 712, 721 722</trace>
   </traceGroup>

   <traceView xml:id="L3">
       <traceView traceDataRef="#L1" from="2"/>
       <traceView traceDataRef="#L2" from="2" to="4:1:1"/>
   </traceView>

   <traceView xml:id="L4" traceDataRef="#L3" from="1:2" to="2:1:2:1"/>
</ink>

   <traceGroup>
       <trace>921 922, 931 932</trace>

       <traceGroup>
           <traceGroup>
               <trace>221 212, 221 222</trace>
               <trace>311 312, 321 322</trace>
           </traceGroup>
           <trace>411 412, 421 422</trace>
           <traceGroup>
               <traceGroup>
                   <trace>521 512, 521 522</trace>
               </traceGroup>
           </traceGroup>
       </traceGroup>

   </traceGroup>

   <traceGroup>
       <trace>931 932</trace>
       <traceGroup>
           <traceGroup>
               <trace>221 212, 221 222</trace>
               <trace>311 312</trace>
           </traceGroup>
       </traceGroup>
   </traceGroup>

4 Contexts

A number of details comprise the context in which ink is written and recorded. Examples include the size of the surface the traces were recorded on, the pen tip used or the accuracy of the pressure measurements. This contextual information needs to be captured by InkML in order to fully characterize the recorded ink data. This section defines markup that provides a way to describe this information, including the <context> element which provides a means to associate a defined context with trace data.

4.1 The <context> element

This section describes the <context> element and its attributes. The context element both provides access to a useful shared context (canvas) and serves as a convenient agglomeration of contextual attributes. It is used by the <traceGroup> and <traceView> elements to define the complete shared context of a group of traces or may be referred to as part of a context change in streaming mode. In either mode, individual attributes may be overridden at time of use. Additionally, individual traces may refer to a previously defined context (again optionally overriding its attributes) to describe a context change that persists only for the duration of that trace.

Although the use of the <context> element and attributes is strongly encouraged, default interpretations are provided so that they are not required in an ink markup file if all trace data is recorded in the same virtual coordinate system, and its relationship to digitizer coordinates is either not needed or unknown.

Attributes:

Contents:

The <context> element consolidates all salient characteristics of one or more ink traces. It may be specified by declaring all non-default attributes, or by referring to a previously defined context and overriding specific attributes. The element is found either in the definitions element or as a child of the ink element in Streaming InkML

4.2 Ink Sources

4.2.1 <inkSource> element

Attributes:

Contents:

Examples:

<inkSource xml:id = "mytablet" 
   manufacturer = "Example.com" 
   model = "ExampleTab 2000 USB" 
   specificationRef="http://www.example.com/products/exampletab/2000usb.html">

   <traceFormat href="#XYF"/>

   <sampleRate uniform="True" value="200"/>

   <activeArea size="A6" height="100" width="130" units="mm"/>

   <srcProperty name="weight" value="100" units="g"/>
        
   <channelProperties>
      <resolution channel="X" value="5000" units="1/in"/>
      <resolution channel="Y" value="5000" units="1/in"/>
      <channelProperty
         channel="Y"
         name="peakRate"
         value="50"
         units="cm/s">
      <resolution channel="F" value="1024" units="dev"/>
   </channelProperties>
</inkSource>

The <inkSource> element will allow specification of:

The <inkSource> block will often be specified by reference to a separate xml document, either local or at some remote URI. Ideally, <inkSource> blocks for common devices will become publicly available.

<sampleRate uniform="True" value="200"/>
  

<latency value="50"/>
  

<activeArea size="A6" height="100" width="130" units="mm"/>;
  

<srcProperty name="weight" value="100" units="g"/>  
  

<channelProperties>
    <channelProperty channel="X" name="resolution" value="5000" units="1/in"/>
    <channelProperty channel="Y" name="resolution" value="5000" units="1/in"/>
    <channelProperty channel="Y" name="peakRate"   value="50"   units="cm/s">
    <channelProperty channel="F" name="resolution" value="1024" units="dev"/>
</channelProperties> 
  

 
<channelProperty channel="F" name="threshold" value="0.1" units="N"/>
<channelProperty channel="X" name="quantization" value="0.01" units="mm"/>
  

4.3 Brushes

Along with trace data, it is often necessary to record certain attributes of the pen during ink capture. For example, in a notetaking application, it is important to be able to distinguish between traces captured while writing as opposed to those which represent erasures. Because these attributes will often be application specific, this specification does not attempt to enumerate the brush attributes which can be associated with a trace. It also does not provide a language for describing brush attributes, since it is possible to imagine attributes which are described using complex functions parameterized by time, pen-tip force, or other factors. Instead, the specification allows for capturing the fact that a given trace was recorded in a particular brush context, leaving the details of precisely defining specific attributes of that context (such as width and color) to a higher-level, application specific layer.

Depending on the application, brush attributes may change frequently. Accordingly, there should be a concise mechanism to assign the attributes for an individual trace. On the other hand, it is likely that many traces will be recorded using the same sets of attributes; therefore, it should not be necessary to explicitly state the attributes of every trace (again, for reasons of conciseness). Furthermore, it should be possible to define entities which encompass these attribute sets and refer to them rather than listing the entire set each time. Since many attribute sets will be similar to one another, it should also be possible to inherit attributes from a prior set while overriding some of the attributes in the set.

4.3.1 <brush> element

Attributes:

Contents:

In the ink markup, brush attributes are described by the <brush> element. This element allows for the definition of reusable sets of brush attributes which may be associated with traces. For reference purposes, a brush specifies an identifier which can be used to refer to the brush. A brush can inherit the attributes of another <brush> element by including a brushRef attribute which contains the id of the referenced brush. As noted above, the definitions of specific brush attributes such as color and width are left to the application.

Brush attributes are associated with traces using the brushRef attribute. When it appears as an attribute of an individual <trace>, the brushRef specifies the brush attributes for that trace. When it appears as an attribute of a <traceGroup> element, the brushRef specifies the common brush attributes for all traces enclosed in the <traceGroup>. Within the <traceGroup>, an individual trace may still override the traceGroup's brush attributes using a brushRef attribute.

Brush attributes can also be associated with a context by including the brushRef attribute on a <context> element. Any traces which reference the context using a contextRef attribute are assigned the brush attributes defined by the context. If a trace includes both brushRef and contextRef attributes, the brushRef overrides any brush attributes given by the contextRef.

In streaming ink markup, brushes are assigned to a trace according to the current brush, which can be set using the <context> and <brush> elements. See section Streaming Applications for a detailed description of streaming mode.

4.4 Timestamps

Timestamping of traces is supported by the <timestamp> element and the timestampRef, timeOffset and duration attributes of the <trace> element. For ease of processing, all timestamps are expressed in milliseconds. Finer-grained timestamps are obtained using fractional values.

4.4.1 <timestamp> element

Attributes:

Contents:

The <timestamp> element establishes a reference timestamp which can then be used for relative timestamping of traces.

The four examples below illustrate the establishment of various reference timestamps. The first <timestamp> element, ts001, refers to January 2, 2004 at 7:00am, UTC. The second establishes timestamp ts002 which refers to January 2, 2004 at 7:10am, UTC (10 minutes after the refernce timestamp ts001), and the third time stamp, ts003, gives the same time using the timeString attribute. The fourth creates ts004 with time January 2, 2004 at 7:10:04.32, UTC (4.32 seconds after the timestamp of trace ts002).

<timestamp xml:id="ts001" time="1073026800000"/>
<timestamp xml:id="ts002" timeOffset="600000" timestampRef="#ts001"/>
<timestamp xml:id="ts003" timeString="2004-01-02T07:10:00Z"/>
<timestamp xml:id="ts004" timeOffset="4320" timestampRef="#ts002"/>

5 Canvases

5.1 <canvas> element

The <canvas> element provides the virtual coordinate system, which uniquely identifies a shared virtual space for cooperation of ink applications. Together with the trace-to-canvas coordinate transform (discussed below), it provides a common frame of reference for ink collected in multiple sessions on different devices.

Attributes:

Contents:

<canvas id="A4PaperCanvas">
  <traceFormat>
      <channel name="X" type="decimal" min="0" max="210" units="mm"/>
      <channel name="Y" type="decimal" min="0" max="297" units="mm"/>
  </traceFormat>
</canvas>

<canvasTransform>
   <mapping type="unknown"/>
   <mapping mappingRef="#map001"/>
</canvasTransform>

<canvas xml:id="DefaultCanvas">
  <traceFormat>
    <channel name="X" 
          type="decimal" default="0" orientation="+ve" units="em"/>
    <channel name="Y"
          type="decimal" default="0" orientation="+ve" units="em"/>
  </traceFormat>
</canvas>

6 Generics

This section describes components of the ink markup which are applicable to multiple aspects of the ink markup.

6.1 Mappings

The <mapping> element provides a uniform syntax for the various uses of mappings in the ink markup. The element has an id attribute, which allows a particular mapping to be applied in multiple places. When a previously defined mapping is reused, the mappingRef attribute is used to refer to the <mapping> element, which might be defined in a <definitions> block. Mappings appear in the following different places in InkML:

1. In a <channel> element of a <traceFormat>, the <mapping> element is used to describe the transformation from the values actually produced by the device to the values recorded in the trace data.
2. In a <crossCoupling> element, a mapping can be used to specify the cross-coupling effect of one or multiple channels on another channel.
3. Used by a <canvasTransform>, a mapping may be used to specify the forward or inverse transformations between an ink source and a canvas coordinate system.

InkML supports several types of mappings: unknown, identity, lookup table, affine, formula (specified using a subset of MathML) and cross product. The mapping type is indicated by the type attribute of a <mapping> element. Note: If no mapping appears for a <channel>, it defaults to "unknown", which is safer than assuming that 'X' is identical to the device's 'X' since some filtering or modifications could have been applied. Furthermore, one can specify whether the results of a mapping expression are absolutely or relatively applied to the current data value. This is done by means of the apply attribute. For lookup table mappings in particular, one can determine how to interpret intermediate mapping values. This is specified using the interpolation attribute.

6.1.1 <mapping> element

Attributes

Contents

If the type attribute has value identity then the element is empty.

Identity mappings are specified using an empty mapping element:

<mapping xml:id="m01" type="identity" />



<channel name="X" type="decimal" units="point" default="0"> 
<mapping  type="identity"/> 
</channel> 

They are used, for example, to define a <traceFormat> channel that reports the exact data that is recorded by a corresponding device channel, with no filtering or transformation.

<mapping type="mathml" m:xmlns="http://www.w3.org/1998/Math/MathML">
    <bind source="R" variable="r"/>
    <bind source="OTh" variable="theta"/>
    <math xmlns="http://www.w3.org/1998/Math/MathML">
       <list>
           <apply>
              <times>
              <ci>r</ci>
              <apply> <cos/> <ci>theta</ci> </apply>
           </apply>
           <apply>
              <times>
              <ci>r</ci>
              <apply> <sin/> <ci>theta</ci> </apply>
           </apply>
       </list>
    </math>
</mapping>

6.1.2 <bind> element

Attributes

Contents

For an identity mapping, if the source channel has a different name than the channel being defined, this can be specified using a <bind> element with a source attribute. In the following markup, the <traceFormat> channel X contains unmanipulated data from the device's devX channel.

<channel name="X">

  <mapping type="identity">
     <bind source="devX">
  </mapping>
</channel>

Within a mapping formula (type="mathml"), the variable names in the formula need to be bound to particular channel names. This is specified using a combination of source and variable attributes for binding inputs of the formula, and target and variable for the output of the formula. This is useful if the same mapping formula is to be reused across multiple channels, like X and Y for example.

   <mapping xml:id="m06" type="mathml">
      <bind type="setvar" target="X" variable="Q" />
      <math mlns=" http://www.w3.org/1998/Math/MathML ">
         <apply>
            <plus/>  
            <ci>Q</ci>
            <cn>10</cn>
         </apply>
      </math> 
   </mapping> 

The example shown above means that the channel X is referred to by the variable name Q in the mapping expression "Q+10".

For a lookup table, each index column must be bound to the channel that provides the input for the lookup operation. This is done with a <bind> element that specifies source and column attributes. Similarly, each value column must be bound to the channel that receives the output of the lookup. Its <bind> element specifies target and column.

<mapping xml:id="m07" type="lookup">
   <bind target="X" column="1"/>
   <bind source="OTx" column="2"/>
   <bind source="P" column="3"/>
   <table apply="relative" interpolate="floor">
      10    45    512,
      9     45    400,
      8     45    372,
      7     45    418,

      10    50    510,
      9     50    403,
      8     50    302,
      7     50    407,

      10    55    512,
      9     55    410,
      8     55    303,
      7     55    405,

      10    60    512,
      9     60    420,
      8     60    355,
      7     60    401,
   </table>
</mapping>

<channelDef name="X"...>
...
<crossCoupling>
<mapping xml:id="m03" type="lookup" apply="relative" lookup="floor">
        <bind target="X"/>
        <bind source="E"/>
        <table>
             45  10,
             50   9,
             55   8,
             60   7
        </table>
</mapping>
</crossCoupling>
...
</channelDef> 

X += 10  if 45 <= E < 50,
X += 9   if 50 <= E < 55,
...

X += 10  if E <= 47.5,
X += 9   if 47.5 < E <= 52.5, 
...

X += 10  if E <= 45, 
X += 9  if 56 < E <= 50, 
...

6.2 Definitions

6.2.1 <definitions> element

Attributes:

Contents:

<ink>
   <definitions>
      <brush xml:id="redPen"/>
      <brush xml:id="bluePen"/>
      <traceFormat xml:id="normal"/>
      <traceFormat xml:id="noForce"/>
      <context xml:id="context1"
           brushRef="#redPen"
           traceFormatRef="#normal"/>
      <context xml:id="context2"
           contextRef="#context1"
           brushRef="#bluePen"/>
   </definitions>
   <context contextRef="#context2" traceFormatRef="#noForce"/>
   <context xml:id="context3"/>
</ink>

<ink xmlns:dc="http://dublincore.org/documents/2001/10/26/dcmi-namespace/">
  <annotation type="description">A Sample of Einstein's Writings</annotation>
  <annotation type="writer">Albert Einstein</annotation>
  <annotation type="contentCategory">Text/en</annotation>
  <annotation type="language" encoding="ISO639">en</annotation>
  <annotation dc:language="en"/>    

  <trace id="trace1">
    ...
  </trace>
  <traceGroup id="tg1">
    <annotation type="truth">Hello World</annotation>
    <traceGroup>
      <annotation type="truth">Hello</annotation>
      <trace> ... </trace>
      ...
    </traceGroup>
  <traceGroup>
    <annotation type="truth">World</annotation>
      <trace> ... </trace>
        ...
      </traceGroup>
    </traceGroup>

  <traceView href="#tg1">
    <annotation type="style">Cursive</annotation> 
  </traceview>
</ink>

<ink>
  <annotation type="description">A Sample of Einstein's Writings</annotation>
  <annotationXML type="metadata" encoding="rdf">
    <rdf:RDF xmlns:rdf = "http://www.w3.org/1999/02/22-rdf-syntax-ns#"
      xmlns:dc = "http://purl.org/dc/elements/1.1/" >
      <rdf:Description about="" 
        dc:language="en"
        dc:date="2004-04-11"
        dc:creator="InkML Maker v0.1"
        dc:publisher="Famous Handwritings Ltd."/>
    </rdf:RDF>
  </annotationXML>

  <trace> ... </trace>
  ...
  <trace> ... </trace>
</ink>

<ink>
  <annotationXML type="truth" encoding="'application/xhtml+xml">
    <html>
      <body>
        <div xml:id="Ch1">
          <h1 xml:id="T1"><span xml:id="W1">Weather</span></h1>
          <p xml:id="P1">
            <span xml:id="W2">The</span> <span xml:id="W3">rain</span>
            ... more words
          </p>
          <p xml:id="P2">...</p>
          ... more paragraphs
          </div>
        </div>
        ... more chapters
      </body>
    </html>
  </annotationXML>
  
  <traceGroup>
    <annotationXML href="#Ch1"/>
    <traceGroup>
      <annotationXML href="#T1"/>
      <traceGroup>
        <!-- Weather -->
        <annotationXML href="#W1"/>
        <trace>...</trace>
    </traceGroup>
  </traceGroup>
  
  <traceGroup>
    <annotationXML href="#P1"/>
  
    </traceGroup>
      <!-- The -->
      <annotationXML href="#W2"/>
      <trace>...</trace>
    </traceGroup>
  
    </traceGroup>
      <!-- rain -->
      <annotationXML href="#W3"/>
      <trace>...</trace>
    </traceGroup>
    ... more words in paragraph
  </traceGroup>
  
  <traceGroup>
    <annotationXML href="#P2"/>
    ... words in paragraph
  </traceGroup>
  ... more paragraphs in chapter.
  </traceGroup>
</ink>

unitExpr ::=
      unit
    | "1"      "/" unit
    | unitExpr "/" unit
    | unitExpr "*" unit

unitPrimitive ::= unit | "(" unitExpr ")"

unit ::= one of the units from the table above,      
         with the exception of em, ex, % and dev.

7 Streams and Archives

The ink markup is expected to be utilized in many different scenarios. Ink markup data may be transmitted in substantially real time while exchanging ink messages, or ink documents may be archived for later retrieval or processing.

These examples illustrate two different styles of ink generation and usage. In the former, the markup must facilitate the incremental transmission of a stream of ink data, while in the latter, the markup should provide the structure necessary for operations such as search and interpretation. In order to support both cases, InkML provides archival and streaming modes of usage.

7.1 Archival Applications

In archival usage, contextual elements are defined within a <definitions> element and assigned identifiers using the id attribute. References to defined elements are made using the corresponding brushRef, traceFormatRef, and contextRef attributes. The following example:

<definitions>
   <brush xml:id="penA"/>
   <brush xml:id="penB"/>

   <traceFormat xml:id="fmt1">
      
      <channel name="X" type="integer"/>
      <channel name="Y" type="integer"/>
      <channel name="Z" type="integer"/>
      
   </traceFormat>

   <canvas xml:id="canvasA">
      <traceFormat>
         <channel name="X" type="decimal" min="0" max="200" units="mm">
         <channel name="Y" type="decimal" min="0" max="150" units="mm">
      </traceFormat>
   </canvas>
   <canvasTransform xml:id="trans1">
      <mapping type="affine">1 0 0 0,0 1 0 0</mapping>
   </canvasTransform>
   <canvasTransform xml:id="trans2">
      <mapping type="affine">2 0 0 0,0 -2 0 0</mapping>
   </canvasTransform>

   <context xml:id="context1"
      canvasRef="#canvasA"
      canvasTransformRef="#trans1"
      traceFormatRef="#fmt1" 
      brushRef="#penA"/>

   <context xml:id="context2"
      canvasRef="#canvasA"
      canvasTransformRef="#trans2"
      traceFormatRef="#fmt1" 
      brushRef="#penB"/>
</definitions>

defines two brushes ("penA" and "penB"), a traceFormat ("fmt1"), and two contexts ("context1" and "context2") which both refer to the same canvas ("canvasA") and traceFormat ("fmt1"), but with different canvas transforms and brushes. Note the use of the brushRef, traceFormatRef, canvasRef and canvasTransformRef attributes to refer to previously defined <brush>, <traceFormat> <canvas> and <canvasTransform> elements.

Within the scope of a <definitions> element, unspecified attributes of a <context> element are assumed to have their default values. This <definitions> block:

<definitions>
   <brush xml:id="penA">
   <context xml:id="context1"  
      canvasRef="#canvasA"
      brushRef="#penA"/>
</definitions>

defines "context1", which is comprised of "canvasA" with the default canvasTransform and traceFormat (the identity mapping and a traceFormat consisting of decimal X-Y coordinate pairs), and "penA".

A <context> element can inherit and override the values of a previously defined context by including a contextRef attribute, so:

<definitions>
   <brush xml:id="#penA"/>
   <context xml:id="context1"
      canvasRef="#canvasA"
      canvasTransformRef="#trans1"/>
   <context xml:id="#context2"
       contextRef="#context1"
       canvasTransformRef="#trans2"
       brushRef="#penA"/>
</definitions>

defines "context2" which shares the same canvas ("canvasA") and traceFormat (the default format) as "context1", but has a different canvasTransform and brush.

Within archival ink markup, traces can either explicitly specify their context through the use of contextRef and brushRef attributes, or they can have their context provided by an enclosing traceGroup. In the following:

<trace xml:id="t001" contextRef="#context1"/>...</trace>
<trace xml:id="t002" brushRef="#penA"/>...</trace>

<traceGroup contextRef="#context1">
   <trace xml:id="t003">...</trace>
</traceGroup>

traces "t001" and "t003" have the context defined by "context1", while trace "t002" has a context consisting of the default canvas, canvasTransform and traceFormat, and "penA".

Traces within a <traceGroup> element can also override the context or brush specified by the traceGroup. In this example:

<traceGroup contextRef="#context1">
   <trace xml:id="t001">...</trace>
   <trace xml:id="t002" brushRef="#penA">...</trace>
   <trace xml:id="t003">...</trace>
</traceGroup>

traces "t001" and "t003" have their context specified by "context1" while trace "t002" overrides the default brush of "context1" with "penA".

A trace or traceGroup can both reference a context and override its brush, as in the following:

<trace xml:id="t001" contextRef="#context1" brushRef="#penA">...</trace>
<traceGroup contextRef="#context1" brushRef="#penA">
   <trace xml:id="t002">...</trace>
</traceGroup>

which assigns the context specified by "context1" to traces "t001" and "t002", but with "penA" instead of the default brush.

In archival mode, the ink markup processor can straightforwardly determine the context for a given trace by examining only the <definitions> blocks within the markup and the enclosing traceGroup for the trace.

7.2 Streaming Applications

In streaming ink markup, changes to trace context are expressed directly using the <brush>, <traceFormat>, and <context> elements. This corresponds to an event-driven model of ink generation, where events which result in contextual changes map directly to elements in the markup.

In the streaming case, the current context consists of the set of canvas, canvasTransform, traceFormat and brush which are associated with subsequent traces in the ink markup. Initially, the current context contains the default canvas, an identity canvasTransform, the default traceFormat, and a brush with no attributes. Each <brush>, <traceFormat>, and <context> element which appears outside of a <definitions> element changes the current context accordingly (elements appearing within a <definitions> block have no effect on the current context, and behave as described above in the archival section).

The appearance of a <brush> element in the ink markup sets the current brush attributes, leaving all other contextual values the same. Likewise, the appearance of a <traceFormat> element sets the current traceFormat, and the appearance of a <context> element sets the current context.

Outside of a <definitions> block, any values which are not specified within a <context> element are taken from the current context. For instance, the <context> element in the following example changes the current brush from "penB" to "penA", leaving the canvas, canvasTransform, and traceFormat unchanged from trace "t001" to trace "t002". That is, each context element is taken to inherit from the previously established context.

<brush xml:id="penA"/>
<brush xml:id="penB"/>
<trace xml:id="t001">...</trace>
<context brushRef="#penA"/>
<trace xml:id="t002">...</trace>

In order to change a contextual value back to its default value, its attribute can be specified with the value "". In the following:

<context  canvasRef="#canvasA" brushRef="#penA"/>
<trace xml:id="t001">...</trace>
<context canvasRef="" brushRef=""/>
<trace xml:id="t002">...</trace>

Trace "t001" is on "canvasA" and has the brush specified by "penA", while trace "t002" is on the default canvas and has the default brush.

Brushes, traceFormats, and contexts which appear outside of a <definitions> block and contain an id attribute both set the current context and define contextual elements which can be reused (as shown above for the brushes "penA" and "penB"). This example:

<context xml:id="context1" 
   canvasRef="#canvasA"
   canvasTransform="#trans1"
   traceFormatRef="#fmt1"
   brushRef="#penA"/>

defines a context which can be referred to by its identifier "context1". It also sets the current context to the values specified in the <context> element.

A previously defined context is referenced using the contextRef attribute of the <context> element. For example:

<context contextRef="#context1"/>

sets the current context to have the values specified by "context1". A <context> element can also override values of a previously defined context by including both a contextRef attribute and one or more of the canvasRef, canvasTransformRef, traceFormatRef or brushRef attributes. The following:

<context contextRef="#context1" brushRef="#penB"/>

sets the current context to the values specified by "context1", except that the current brush is set to "penB" instead of "penA".

A <context> element which inherits and overrides values from a previous context can itself be reused, so the element:

<context xml:id="context2" contextRef="#context1" brushRef="#penB"/>

defines "context2" which has the same context values as "context1" except for the brush.

Finally, a <context> element with only an id has the effect of taking a "snapshot" of the current context which can then be reused. The element:

<context xml:id="context3"/>

defines "context3", whose values consist of the current canvasRef, canvasTransform, traceFormat, and brush at the point where the element occurs (note that since "context3" does not specify any values, the element has no effect on the current context).

An advantage of the streaming style is that it is easier to express overlapping changes to the individual elements of the context. However, determining the context for a particular trace can require more computation from the ink markup processor, since the entire file may need to be scanned from the beginning in order to establish the current context at the point of the <trace> element.

7.3 Archival and Streaming Equivalence

The following examples of archival and streaming ink markup data are equivalent, but they highlight the differences between the two styles:

Archival

<ink>
   ...
   <definitions>
      <brush xml:id="penA"/>
      <brush xml:id="penB"/>
      <context xml:id="context1" canvasRef="#canvas1"
         canvasTransform="#trans1" traceFormatRef="format1"/>
      <context xml:id="context2" contextRef="#context1"
         canvasTransform="#trans2"/>
   </definitions>
   <traceGroup contextRef="#context1">
      <trace>...</trace>
      ...
   </traceGroup>
   <traceGroup contextRef="#context2">
      <trace>...</trace>
      ...
   </traceGroup>
   <traceGroup contextRef="#context2" brushRef="#penB">
      <trace>...</trace>
      ...
   </traceGroup>
   <traceGroup contextRef="#context1" brushRef="#penB">
      <trace>...</trace>
      ...
   </traceGroup>
   <traceGroup contextRef="#context1" brushRef="#penA">
      <trace>...</trace>
      ...
   </traceGroup>
</ink>

Streaming

<ink>
   ...
   <definitions>
      <brush xml:id="penA"/>
      <brush xml:id="penB"/>
   </definitions>
   <context xml:id="context1" canvasRef="#canvas1"
      canvasTransform="#trans1" traceFormatRef="#format1"/>
   <trace>...</trace>
   ...
   <context xml:id="context2" contextRef="#context1"
      canvasTransform="#trans2"/>
   <trace>...</trace>
   ...
   <context brushRef="#penB"/>
   <trace>...</trace>
   ...
   <context contextRef="#context1"/>
   <trace>...</trace>
   ...
   <context brushRef="#penA"/>
   <trace>...</trace>
   ...
</ink>

In the archival case, the context for each trace is simply determined by the <trace> element, its enclosing traceGroup, and contextual elements defined in the <definitions> block, while in the streaming case, the context for a trace can depend on the entire sequence of context changes up to the point of the <trace> element.

However, the streaming case more simply expresses the changes of context involving "penB", "context1", and "penA", whereas the archival case requires the restatement of the unchanged values in the successive traceGroups.

The two styles of ink markup are equally expressive, but impose different requirements on the ink markup processor and generator. Tools to translate from streaming to archival style might also be of use to applications which work on stored ink markup.