SVG Accessibility/ARIA chart taxonomy discussion

From W3C Wiki

A single page gathering up all the discussion on the mailing lists (in August-September 2015) about Fred Esch's Chart Taxonomy Proposal and Amelia Bellamy-Royds' alternative proposal. Comments have been roughly divided between the main issues: what information is required, what roles are needed, and how should data be communicated. However, the division isn't perfect.

Note: There has also been discussion in the teleconferences; I haven't copied anything from the minutes yet.

What information is required?

Jason White wrote up a summary of Graphics Types used in Educational Applications which we would want to support.

Léonie Watson's email of 14 August:

I think it would be helpful if we could establish some of the basic things and relationships someone might want to know about SVG content - specifically data visualisations. Chaals has put together a useful collection of SVG charts [1], which seem like a good place to start looking for typical things and relationships.

>From the call today, a few things were suggested...

What type of data visualisation is it? A pie chart, bar chart, scatter plot etc.

What units are represented in the data visualisation? Temperature (degrees), rainfall (millimetres), Dollar value etc.

How is data arranged? By product type and Dollar value, or by colour, shape and volume for example.

What is known about the data? 10 data-points grouped into 2 data-sets for example.

After which it's likely that someone would want to drill down for more information. Depending on the complexity and nature of the data visualisation, the way someone drills down it likely to change. Possible strategies include drilling down for more information about scale/axes, or drilling straight down into the data-points.

If we can identify the generic things and relationships in a typical data visualisation, we can then look at how those things can best be represented using ARIA. From there we can look at the challenges we're up against - like managing complexity for example.

Jason White's email of 14 August, referring to the NCAM Effective Practices document:

As I understand it, the NCAM research only affects the order in which the information is presented to the user, not its content.

Another way of handling data visualization in ARIA that we haven’t considered would be

1. Define each “dimension” of the data so that its type is clear to the UA/AT.

2. Use an ARIA property to declare its visual representation. Thus, for a numerical value, the visual presentation could be declared as, e.g., “bar height”, “bar width”, “point” (in a coordinate plane), “sector” (as in a pie chart), etc.

This would separate information about the data from information about its graphical presentation, while making both available to the user agent and, ultimately, to the user in a consistent way.

Note that I’m not advocating this design here but rather adding it to the options under discussion.

Fred Esch's reply on 18 August:

I like the idea of enabling authors to present information to their users. Graphics present information very well to users with perfect vision and without cognitive impairments. Providing additional channels of information will only work well when authors can customize information for the intended audience.

1. Define each “dimension” of the data so that its type is clear to the UA/AT.

fe: I think the proper place to define a dimension is on a visible scale - either an visible axis or a legend. If there is no axis/legend for a dimension, then we should not try to define that dimension for the audience as the visual user doesn't get that information and the author may not intend the scale to be metric.

For example, an application may use a proprietary algorithm to rank order a list. Instead of presenting a list, they choose to present the results as a bar chart with the largest bar on the left and the bars in descending order as you go right. In this case, the author does not want you to compare categories using value ratios - so they do not have a Y axis on the chart. The visual user looks at the left most category on the X axis and knows this category has the highest value, they don't know what that value is. The user can't compare a ratio value for two categories as they are not given a Y axis as a scale - which is what the author intends.

2. Use an ARIA property to declare its visual representation. Thus, for a numerical value, the visual presentation could be declared as, e.g., “bar height”, “bar width”, “point” (in a coordinate plane), “sector” (as in a pie chart), etc.

I think this is valuable and interesting. However, it think it can be difficult for an author to articulate and thus won't be used much. How does an author articulate a color in a continuous color scale? ( ColorBrewer 2, has great categorical color scales. ) How does an author provide names for custom shapes?

There will also be domain considerations, such as symbol names on blueprints. Would an aria property add confusion if the author provided a description?

Jason White's response of 18 August:

I think it depends on the level of detail required to make the representation comprehensible to the user. The advantage of a property is that the vocabulary is standardized and can be localized by the rendering software when presented to the user. Author-supplied descriptions are variable in quality and not standardized, so there’s a decision to be made regarding (1) what level of detail is needed concerning the visual presentation, and (2) whether having standard terms defined in ARIA properties is desirable. My suggestion assumed that the trade-off was worth making, but not that the exact visual representation of each data item needs to be recoverable from a non-visual rendering. Rather, the idea was to represent the visual property that is being used to denote a data dimension without necessarily describing or classifying the appearance of the individual data items. How useful that would turn out to be is a question that merits further thought, of course.

Fred Esch's email of 19 August compares his proposal with Jason's Notes on Graphics Types and Educational Applications

What specific features of charts should be reflected in a role taxonomy (and with what terminology)?

Fred Esch's summary of his proposal, 19 August:

The taxonomy takes it's hierarchy from the Grammar of Graphics. The taxonomy splits a chart into 3 aspects data, guides and facets (aka views, diagrams) and uses roles to designate the aspect. By splitting the chart this way, any visualization can be described using the taxonomy. Well known chart features are subclassed from their parent aspect. Subclassing is a recognition that users recognize the feature by it's role and may have a name for the feature/role. For instance, axis is a well known scale feature in a chart. The taxonomy leaves lesser known uses, using the parent role and leaves any explanation to the author through aria properties or a title or desc child; keeping the taxonomy compact.

Charts are very diverse and it is not always possible to know how features relate to each other without author guidance. Authors can distinguish between compound objects and two objects with the same role using the aria-gtype property. For instance axes can be configured in several ways in a chart and without author guidance it would not be possible to know the actual usage. Here are cases of multiple objects with an axis role: The axis line and tick marks are drawn before the scatter plot points. The axis label and title are in a different group element and drawn after the scatter plot points, but because of (legacy) drawing order the two groups with the role axis are part of the same object. There are two y axes on the chart. The y axis on the left is in meters, the y axis on the right is in feet. There are multiple sets of x axes (with the same y coordinate), because the clustered axes are repeated for every cluster There are multiple sets of x axes (with different y coordinates) because the chart is clustered and the nested axes are shown. There is an radial axis at 45 degrees and an axis around the outer circle in a chart that uses polar coordinates. A panelled bar chart with paneling by sex and geographic region has axes on each panels chart and axes for sex and geographic region.

Without author guidance, there would be no way to know which objects with role axis should be treated as a single axis vs different axes. Author guidance is also needed for AT to convey what the differences between axes objects are. Using aria-gtype and appropriate aria/title/desc the author can guide the user through the axes.

Data The data aspect are for things that represent data in the chart - the scatter plot points, bars, columns, box and whiskers, areas, lines, paths and polygons. The data aspect is split into a group and item roles. There are a couple of proposed subclasses for the item role.

Guides Guides are the non data things that are there to help a user read and interpret the chart. Scales, a subclass of guides are used for measuring. There are are several subclasses of scales in the taxonomy and we could add many more domain specific scales. Annotations are another subclass of guides.

Facets (aka views or diagrams) The grammar of graphics uses the term facets, but some domains such as technical drawing uses the more understandable term view. In the Graphics Module we use the term diagram. A view represents an area of the graphics that serves a specific purpose and could stand on its own. Strictly speaking, a facet is about splitting data into subsets and drawing a chart for each subset. An example is splitting the results of a survey by male and female respondents. Each chart for each subset can stand on its own. With paneled charts, you can make a matrix of charts when you facet by two categorical variables. You can also facet by geographic place and place the chart at the center of the geographic place. For example, you can make bar charts for unemployment for each US state and place each bar chart within the state. Paneled charts they often have an axis for each panel dimension.

Requirements and best practices While it may seem like each chart type makes new requirements for reporting information, in reality they do not. The requirements and best practices for all charts are given below.

The primary requirements are to present the data, guides and views to the user and allow them to visit and investigate them in a logical order. Providing summary information should be a best practice For instance, axes should report information about the range they cover or the number of categories. A categorical axes should have a list of categories that may be read by the user. The data group object should provide a data summary such as the data range within the group and possibly point out which are the highest and lowest valued items. Stack object in stacked bar charts should provide the sum (total height) of the stack. And so on. Data items should contain their values for visible dimensions/aesthetics. Views should be identified and visitable and their children visitable. Panel axes should be visitable. When connectors and edge roles are part of SVG, relationships in node/edge graphs should be available and moving focus based on relationship possible. Much like aria-flowto works.

Jason White's email of 3 August, commenting on Fred's proposal:

There are roles in the proposal that largely correspond to existing ARIA roles: graphics-document is similar to the ARIA document role; graphics-object is similar to the group role. The principal difference seems to be that the content is designated as graphical. If assistive technologies can easily infer that the content is graphical, then it may be feasible to use the existing ARIA roles without defining new (and similar) ones, as here.

I think the semantics of the aria-gtype property should be clarified. In particular, it may duplicate the functionality of the aria-roledescription property of ARIA 1.1, in which case, reusing the latter would be preerable to introducing a new property. In general, if the type of entity to which ARIA roles, states and properties are applied needs to be distinguished in a systematic, machine-readable fashion, then this really ought to be accomplished by defining a specific role rather than by using a more general role and a property associated with the element to do so. The purpose of aria-roledescription is simply to give a localized string to the AT that can be presented to the user to characterize the type of element, without implying any difference in purpose or behavior from the underlying ARIA role. This mechanism is desirable, of course, and respects the principle articulated above.

I also concur with Doug’s misgivings about the term “guide”, though I don’t have a markedly superior alternative to offer for the envisaged superclass role.

Fred's response/comparison of Amelia's proposal, 28 August:

Overall, I believe with the current taxonomy we can express everything to AT that the authors want conveyed by relying on liberal use of <desc> and similar modes. A missing component in the current taxonomy that Ameilla addresses, is machine readable value tuples. The most controversial roles in the current taxonomy are graphics-dimensionline, graphics-graticule and graphics-timedata. Dimension lines are widely used in blueprints and technical drawings. Graticule has a different role than grid because both graticule and grids commonly appear on the same maps. And maps with both graticule and grids are widely used - for land navigation, flight, in cadastral systems, surveying and are in maps produced and used by local governments.

Comments on Amelia's roles and properties below

Amelia subclasses graphics types into maps, networks and datacharts which get further subclassed. I don't believe we need to subclass graphics using a role. A <desc> on the SVG element allows the author to convey that information. Amelia splits chart data elements based on primitive type - point, line, region. I do not think we need to subclass data based on primitive type. Again, the data element can provide information through a <desc> if the author wishes to. Amelia has the graphics-data as an abstract role, however I believe a concrete role is needed for the <g> which is the parent of the data items. Structurally, you need a <g> object to parent data, so navigation can walk through the features (axes, data, legends) without dropping into the features. I would strip 'data' from any name having to do with scales. Scales are independent of 'data'. An axis can be fixed, and a legend can show members that are not on the current chart or a legend might not show all the members on the chart (think of charts with republican candidates for president in the legend).

Amelia Bellamy-Royds' response to Fred, 31 August

It is for these alternative 2D presentations that it is really important to distinguish between maps, data charts, network graphs, and generic graphics or diagrams. When we declare a section of content as a graphical document, we are saying that layout is no longer a matter of style and formatting, it is part of the informational content. The document types for maps, charts, and network graphs distinguish *which* aspects of the layout are important for information. In combination with the other roles and properties, it provides the AT with the information to make judicious alterations to the layout as required. For example:

  • A legend may be moved around so long as it doesn't obscure data items

and the user can still easily consult it.

  • An axis may be repeated if the chart needs to be split across pages,

or the axis may remain fixed while the data (and other axis) scrolls by perpendicular to it.

  • A network graph could be extensively re-arranged so long as the

connections between components are preserved.

In contrast, if the group consensus is that authors should provide all data in the form of text labels and descriptions, then there is no potential for software to re-encode data or offer a query feature. The only purpose for a complex role taxonomy of parts of a chart would be to support layout-reorganizing changes, so a much simpler set of roles should be used. The label and description ARIA properties, including the new aria-roledescription property, would be sufficient to distinguish other features or sub-types of features. Navigation options (both 2D and hierarchical) would be the same as for more generic diagrams and structured drawings, but with the option for non-visual users to jump to axes or to data.

How much machine-readable data should be included for automated chart summaries or alternative presentations?

Jason White's email of 3 August, commenting on Fred's proposal:

It was widely agreed at the meeting two weeks ago that the author should be permitted, though not required, to specify the value of each data item explicitly. For example, a property could be defined to take comma-separated values to specify each data point. I think this is desirable functionality that should be integrated into the draft.

Fred's response/comparison of Amelia's proposal, 28 August:

I prefer to see the text from a visible label when conveying information rather than using an aria property. I worry about authors providing 'real data values' in a graphic. I prefer to rely on authors to provide formatted values through <desc> and similar means. The graphics engine producing the SVG probably does not have the original data and can't put it in a chart. It would be wrong for a user agent to round or change data through formatting that is different than the method used in the statistical package requesting the chart (and to do it right, would require the original value and knowledge about the statistical package).

Fred Esch's assessment of the difficulty of automated chart summaries, 31 August:

This is my view as a former chart rendering engine developer on units and data types. There is several datasets that may be used or produced by a product. raw data - from an original source, which may be used for analysis by the product summarized data - which may be used for analyis by the product data for rendering the chart

A charting engine only gets the last dataset - data for rendering the chart. Products create this data knowing how big they want their chart to be how they want visible labels to appear on the chart the chart renderer's API for formatting what data precision the rendering engine needs to accomplish 1 & 2 given 3

A product has no reason to provide a chart renderer more information about the data than is needed to draw the chart. This will be repeated throughout the discussion and will seem like a party pooper for many potentially cool features - like user agents formatting values. Anything a user agent does that does not match the way the author wants items expressed in the chart is unacceptable.

Units In charts, when important, units will appear in labels, descriptions and/or surrounding context. When the units are implied/expressed by context an author may choose not provide unit information to the chart renderer.

Units should be tied to an axis. Units should also be tied directly to data. For example, a chart may have temperature as the dependent variable (Y axis) and the chart has two Y axes - a left axis in Centigrade and a right axis in Fahrenheit. Any temperature on a data point, must have the units tied to the data as it would be unknown which of the two axis related Y scales a value referred to.

As mentioned in another email, I favor organizing charts so data items have a group parent, so navigation through the chart can navigate from feature to feature without dropping into features. The data group parent is a good place where group summary information can be defined. Some of the summary information that is useful are the value tuple information, printable names for the variables; variable data range (min, max) in the group and I like to provide indicators as to which of the child data items in the dataset are the mins and maxs. I prefer much of the summary information to be in a text description that a user can easily skip over.

I do not like the idea of user agents trying to format values as the formatted values may not follow the same formatting as visible labels. Any formatting differences a user agent has compared to what the author defined in making the chart are unacceptable.


Data types As mentioned above - the chart rendering engine only gets a minimally sufficient data set to draw the chart. For drawing charts, data types reduce to categorical (equally spaced) or continuous scales. And the process by which the product converts from raw/summary data to categorical or continuous scales is not known by the rendering engine.

In products, charts do not appear in isolation, so the context may explain how missing data is treated, may explain generalizations that appear in the chart or tabular data and provide context for interpreting the chart. For example, a product may find and grade query results and show a bar chart of strength of the result's match. The product explains taller bars are better matches and the user is left to assume two bars nearly the same height are almost equally good (but this is never explicitly stated). Any written description of the results match and the tooltips on the bars simply state - best match, 2nd best match, good match, poor match, poor match and poor match. As you might guess none of the bars are of equal size, nor are the drops between bars even. And for this discussion, you can ask what data type am I? And would I want AT to know about the data type? Charts like this example are not rare.

My preference is to have the author provide interesting information (ie. I am the max value in the chart!) and formatted values as descriptions. I think it is OK to provide machine readable values for data items. But it would not be OK for a user agent to do statistics on values, summarize values or format values independently of an authors descriptions. The charting engine does not have the needed information (original data type, missing data, conversion process, assumptions, rounding/precision in the chart data) to pass on to the user agent that would allow a user agent to do an acceptable job.

Amelia Bellamy-Royds' response to Fred, 31 August

I whole-heartedly agree that we should avoid any suggestion that user agents should perform inappropriate statistical analyses of chart data. We do not want a browser declaring trendlines or the significance thereof when that measure may be quite inappropriate for the dataset. We need to recognize that data used to generate a chart is often a summary of a much larger raw data set, and is often cleaned up in ways that reduce precision.

However, I do not believe the alternative is to rely solely on author-provided descriptions of the data and its patterns. There are multiple problems with this approach:

- It expects the author to identify all features worth describing. It does not allow the user to *choose* which details they are interested in. - If the chart is generated by an automated tool, the text descriptions will not necessarily be any better than text descriptions created by an automated tool on the user agent/AT. - Unlike a user-agent tool, an authoring tool will not be customizable to the language requirements and other needs of the user.

For these reasons, I would like author-supplied descriptions to be an

  • enhancement* or complement to user-controlled automated summaries. The

automated summaries would be based on metadata that can easily be inserted by the authoring tools that generate the chart. (Because, let's face it, most SVG charts are created with authoring tools or scripting libraries.)

The automated summaries I am proposing are not analysis, although they would technically be called summary statistics. They are intended to describe the features that a sighted user would gather from looking at the presentation of the data in a chart. For example:

  • The number of axes and other data scales, and the range of values they

can encode (regardless of whether the data uses this full range).

  • The number of data groups, and the number of data items within each

group.

  • Which data measurements, on which scales, are available for each group

or each data item

  • The maximum and minimum data values on each scale, either overall or

for each group, and maybe other non-parametric statistics such as median or quartiles.

  • The entire data points as a sorted list, according to values on any of

the scales.

More advanced non-visual AT could perhaps offer a query format, allowing users to slice the data into subsets based on categorical variables or ranges of quantitative variables. For example, the user may want to know the max, min, and median y-values when the x-value is within a certain range. Or the user may want to know at which x-values the y-value falls within a given range. Other software, as Doug has demonstrated, could use auditory encodings to directly represent the data values in sound.

The details of data relationships, as expressed by the properties I proposed, would also be used for alternative visual presentations (or other 2D presentations, such as embossed books).

  • Software that replaces color encodings with different colors,

patterns, or shading would need access to the data used in the existing encoding. This could be useful for monochrome printing, embossing, high-contrast mode, or color-blindness-friendly mode.

  • AT that changes the layout of some features needs to know which parts

of the layout encode information, and how different parts are connected. This could be useful for large print or magnification mode, braille/embossed documents, automated translations, and maybe even automated adaption to small screens.

Jason White responding to Amelia's comment that author-supplied descriptions should enhance automated summaries, 31 August:

I strongly agree here. Also, certain techniques important for accessibility benefit from the ability to perform (even rudimentary) interpretation of the data as they are presented. For example, it isn’t clear how one could produce appropriate sonification without determining the range of data values so that they can be assigned to appropriate audio frequencies.

Fred Esch's reply, Sept 1:

We have a diverse group and some folks may not be aware of the dangers of letting user agents interpret data, so I wanted to articulate what I believe to be acceptable and unacceptable behaviors and provide reasons for it.

I generally like your proposed automated summaries, most of the charts will be produced by chart rendering engines that could provide the summary information. I love the idea of providing users sorted lists, and that would have to be left to a user agent to create the lists. Except for situations like box and whisker plots, I don't believe a median and quartiles will be provided by a chart engine. A user agent should never calculate statistics since it does not have access to the original data. Who would be responsible for fixing a reported bug when a user agents statistic does not match a statistical package's statistic used by a product?

I don't think it is appropriate for a user agent to slice and dice chart data. Products that used the chart engines I worked on, let you slice and dice data via interactivity on the chart, sortable table or through other means. It was up to the product whether you could drill down using chart interactivity. Slicing and dicing data by a user agent has the same problem as creating statistics - the user agent does not have access to the original data.

I think Amelia and I differ on how far user agents should be able to alter charts. I favor paths to support colorblind and low vision users, however I do not believe a user agent needs to understand a graphic's data to do so. To support changing an aesthetic palette used, a chart could put the text color and the visualized aesthetic values that appear in data in the chart and in legends in <defs>. These would functionally be palettes for the aesthetics and text. Perhaps a media query would tweak the palettes for the user. Two advantages of this approach are: you can adjust the palettes without knowledge of the actual data values; and the media query can adjust things just enough to meet the goals (like improving contrast) without radically changing the appearance (reds can still look red, blues look blue) of the chart. You might be able to use a similar technique to create palettes for sounds and haptic devices. If you want a user agent to alter palettes (not using a media query), you may have to identify palettes and significant styles (ie dataPalette, textStyle). In general, a palette does not have to be ordered (ie low to high) for it to be used, but you may want an ordered palette if you want to derive a palette for sound or a haptic devices on the fly.

I think it is fine to provide shortcuts for navigating within charts, like being able to jump to axes or data without walking every feature. I am not in favor of user agents visually moving stuff around. Moving panels in a SPLOM would not be acceptable. I do not favor axes that float and stay visible when you scroll, that sounds like an application should do that. Maps can have longitude and latitude marks on the neat line, but floating latitude/longitude marks may violate the projection and give the user false information. I believe moving features around or decoupling parts of a graphic is something for an application to do not a user agent.

I think roles and properties are valuable for creating names that consistent across authors. For example a role of axis and an aria-gtype of y should consistently create the same name (my guess would be 'axis y'). The benefit to the user is - they can access charts from different authors and still know what is an axis. Even when one author uses an aria-gtype of 'y' and another uses an aria-gtype of 'dependent' the user won't have any more confusion than when one encompassing article talk about a 'y axis' and another talks about the 'axis for the dependent variable'. Charts do not have that many well known roles -titles/notes, axis, data, legend, grids, figures (aka panels or facets), labels, edge (connector) and symbols. Everything else can be handled ad hoc through descriptions. Likewise technical drawings have a recognized role for dimension lines and maps have a recognized role of graticule and can still be covered with the chart taxonomy.

The aria properties can be valuable in charts. In my examples I used aria-valuemin and aria-valuemax on axes, datagroups and dataitems.

Jason's follow-up, 1 September

The underlying principle which comes to mind here is that the user agent/assistive technology should perform the analyses that an informed observer could intuitively and easily carry out by inspecting the data as visually presented. The level of accuracy should be the same as what an informed recipient of the visual representation would be able to achieve.

Of course, if the author provides the actual data (as we’ve proposed to allow but not to require), then more accurate analyses can be undertaken.

To which Fred replied, 1 September:

Of course, if the author provides the actual data (as we’ve proposed to allow but not to require), then more accurate analyses can be undertaken.

A good visualization will allow a sighted user to get the important information in a few milliseconds, via the the visual process. If done correctly, the important information is easy to see and a very low level part of the brain does low level feature analysis and provides your brain the cue to look at the important data instantaneously. Often the important information is an overall impression - "oh the bars in the chart are all about even - not a lot of difference in this data set" or "wow look at that one in the upper right corner, it is not near any other data point" or " the tread is always downward". These things can be hard to covey to a user by providing numbers and it will certainly take a user longer to dig out the information. Note, if the important information were easy to see by comparing numbers, then the intelligent author would have used a table instead of a chart.

So what can sighted users get from a chart? I am giving WAG's for a sighted user error of when estimating a continuous value. Sighted users don't extrapolate values from locations on empty fields with any precision - my guess is 10% error is normal. Using color with continuous scales gives you a touchy-feely sense of relative value, but it is pretty difficult to interpolate, my guess is 15% error. Size is difficult to estimate, especially if the size is tied to area and not length. My estimate for visually guessing linear size of symbol is 15% error and for areas 40% error - people are really bad at comparing areas - especially circles. Are folks really that bad at guessing and extrapolating, YES and it gets worse as we get older or more impatient and it is a real pain to to do too. In creating SVG examples I tried to interpolate values to add the information in descriptions and in every case, I gave up and went back to the chart's spec and read the data. For charts that I could not find the chart spec for, I abandoned the effort and those charts do not appear in the examples. What makes it really bad is a person's error in estimations are not systematic and precise, rather they are random.

Another factor that affects interpretability of a chart is how well does work with the data set to convey the information the author wants conveyed? Choosing the right visualization is an art and for a few folks a science. Scientific article authors will search for the optimal visualization for their data and audience. But mostly what a business user will get are automated reports that have a bar chart or whatever kind of chart an admin decided on months ago, in your report and the chart may not be the optimal choice for the particular data set. So the obvious message in the chart may be "things are normal" and "this guy is the worst of the lot and this gal is the best of the lot".

I think we should empower the author to pass along the information they believe should be important to the user as they see fit. After all the author knows better than the user agent, what the context for the chart is and what the role of the chart in the product is.