As languages evolve, it is possible to speak of backwards and forwards compatibility. A language change is backwards compatible if newer processors can process all instances of the old language. Backwards compatibility means that a newer version of a consumer can be rolled out in a way that does not break existing producers. A producer can send an older version of a message to a consumer that understands the new version and still have the message successfully processed. A software example is a word processor at version 5 being able to read and process version 4 documents. A schema example is a schema at version 5 being able to validate version 4 documents. This means that a producer can send an old version of a message to a consumer that understands the new version and still have the message successfully processed. In the case of Web services, this means that new Web services consumers, ones designed for the new version, will be able to process all instances of the old language.

A language change is forwards compatible if older processors can process all instances of the newer language. Forwards compatibility means that a newer version of a producer can be deployed in a way that does not break existing consumers. Of course the older consumer will not implement any new behavior, but a producer can send a newer version of an instance and still have the instance successfully processed. An example is a word processing software at version 4 being able to read and process version 5 documents. A schema example is a schema at version 4 being able to validate version 5 documents. This means that a producer can send a newer version of a message to an existing consumer and still have the message successfully processed. In the case of Web services, this means that existing Web service consumers, designed for a previous version of the language, will be able to process all instances of the new language.

In general, backwards compatibility means that existing texts can be used by updated consumers, and forwards compatibility means that newer texts can be used by existing consumers. Another way of thinking of this is in terms of message exchanges. Backwards compatibility is where the consumer is updated and forwards compatibility is where the producer is updated, as shown below:

With respect to consumers and producers, backwards compatibility means that newer consumers can continue to use existing producers, and forwards compatibility means that existing consumers can be used by newer producers.

We need to be more precise in our definitions of what parts of our definitions are compatible with what other parts. Every language has a Defined Text set, which contains only Texts that contain the texts explicitly defined by the language constraints. Typically, a language will define a mapping from each of the definitions to information. Each language has an Accept Text set, which contains texts that are allowed by the language constraints. Typically, the Accept Text set contains Texts that are not in the Accept Text set and do not have a mapping to information. For example, a language that has a syntax that says names consists of given followed by family followed by anything. A text that consists of a name with only a given and a family falls in the Defined and Accept Text set. A text that consists of a name with a given, a family and an extension such as a middle falls in the Accept Text set but not the Defined text set. By definition, the Accept Text set is a superset of the Defined Text set.

We have discussed backwards and forwards compatibility in general, but there other flavours of compatibility, based upon compatibility between the Accept Text set, Defined Text set and Information conveyed. Syntactic compatibility is compatibility that is wrt the Texts only, not the information conveyed. Because languages have Accept and Defined Text sets, some producers will adhere to the Defined Text set, and others may generate extensions that fall in the Accept Text set. Compatibility with Producers that produce only Defined Text sets is called "strict" compatibility. Compatibility with Producers that may produce Texts in the Accept Text Set that are not in the Defined Text Set is called "full" compatibility.

A more precise definition of compatibility is with respect to the texts, that is whether all the texts in one language are also texts in another language. Another precise form of compatibility is with respect to the information conveyed, that is whether the information conveyed by a text in one language is conveyed by the same text interpreted in another language. The texts could be compatible but the information conveyed is not compatible. For example, the same text could mean different and incompatible things in the different language. Most systems have different layers of software, each of which can view a text differently and affect compatibility. For example, the XML Schema PSVI view is different from the actual text. We can also differentiate between language compatibility and application compatibility. While it is often the case that they are directly related, sometimes they are not, that is 2 languages may be compatible but an application might be incompatible with one of them.

We provide mathematical definitions of a text's compatibility based up on our terminology.

Let L1 and L2 be Languages, where L2 is introduced "after" L1.

We can also provide mathematical definitions of language compatibility:

L2 is "fully backwards compatible" with L1 if every text in L1 Accept Text set is fully compatible with L2.

We can draw a few conclusions. Given L2 is strictly backwards incompatible with L1 if any text in L1 Defined Text set is incompatible with L2, the only way that L2 can be backwards compatible with L1 is if the L2 Defined Text Set is a superset of L1 Defined Text set. Roughly, that means the addition of optional items in L2. Given L1 is "fully forwards compatible" with L2 if every text in L2 Accept Text set is fully compatible with L1, the only way that L1 can be forwards compatible with L2 is if the L1 Accept Text is is a superset of the L2 Accept Text set. Roughly, that means L1 allows all of L2 and more. It is this superset relationship that is a key to forwards compatibility, the allowing of texts by L1 that will become defined in L2.

Compatibility can be restated in terms of superset/subset relationships.

We have shown that forwards and backwards compatibility is only achievable through extensibility, and compatible versioning is a process of gradually increasing the Defined Text Set, reducing the Accept Text Set and ensuring the information conveyed is compatible, If ever the set relationships defined earlier do not hold, then the versions are not compatible.

We have defined compatibility for all possible expressions of the language, that is full compatibility. There are many scenarios where a consumer will consume only part of the information set. Partial understanding affects the Text set and the Information conveyed. Partial understanding usually results in a subset of the information (because only part of the information is understand). Interestingly, partial understanding is an increases or supersets the Accept Text Set and a parallel decrease or subset of the Defined Text Set. This is because the process of extracting a part of the text means that extra content, even those illegal under V1 syntax, becomes part of the Accept Text Set.

We can imagine an application that only looks at given names and everything else is ignored. My favourite example of this is a "Baby Name" Wizard. The application might use a simple XPath expression to extract the given name from inside the name. This is a different version of the Name Language, which we will call the Given Name Language. The Accept Text set for the Given Name Language is anything, given, anything. The Defined Text set for the Given Name Language is given. The information set for the Given Name language is given. Because the Given Name Language syntax set is more relaxed that the Name Language V1, an addition of the middle name between the given and family is a compatible change for the Given Name Language. There are a variety of other now acceptable names in the Given Name Language.

Our principles with respect to compatibility and language versioning need no change to deal with partial understanding. Partial understanding a language is the creation of Language L1' that is compatible with Language L1. This is true if L1' Accept Text set > (superset) Language L1 Accept Text set > (superset) L1 Defined Text set > (superset) L1' Defined Text Set AND every text in L1' Defined Text set is compatible with L1 AND every text in L1' Accept Text set is compatible with L1'.

Interestingly, partially understanding a language is creating a language V1', such that the V1 language is a compatible change with the V1'. There may be many different versions that are all partial understandings of a language. We call these related languages "flavours". It may be very difficult for a language designer to know how many different language flavours are in existence. However, a language designer can sometimes use the different flavours to their advantage in designing for a mixture of compatible and incompatible changes. Some changes could be compatible with some flavours but not other. It may be very useful to have some changes be compatible with some flavours, that is those consumers do not need to be updated or changed.

It is crucial to point out that the consumers of partially understood versions of the language are not also producers of the partially understood language. They have relaxed the restrictions on the consuming side, but should not do so on the production side of the language. If a flavour of a language was also used for production, it should have to create an instance that is valid according to the Language V1 rules, not the Language V1'. Perhaps the only exceptions are if they are guaranteed that they will be producing for compatible flavours. Typically this is not the case and hard to determine, so the safest course is to produce according to the Language V1 rules.

We have shown how relaxing the constraints on a language when consuming instances of it can turn an otherwise incompatible change into a compatible change. We have also shown that abiding by the language constraints when producing instances is the safest course. Said more eloquently is the internet robustness principle, "be conservative in what you do, be liberal in what you accept from others" from .

We will call this style of versioning the "liberal" style of versioning. The "liberal" style of versioning is codified in:

Use Least Partial Languages for "liberal" versioning: Consumers should use a flavour of a language that has the least amount of understanding.

The least amount of understanding will be the most liberal or have the largest syntax set possible.

The "liberal" style of versioning has a significant downside in that it can lead to very fragile and hard to evolve software because the "liberal"ness is difficult to code and it does not force producers to be correct in what they produce, causing a vicious cycle of complexity.

There is an opposite style of versioning that says the most effective way of evolving is to force producers to be correct by having strict consumers. We will call this the "conservative" style of versioning. The "conservative" style of versioning is codified in:

Use No Partial Languages for "conservative" versioning: Consumers should fully use and validate a language.

The fullest amount of understanding will find the most errors.

In either "liberal" or "conservative" versioning of consumers, the advice to a producer is the same:

Produce no partial languages: Producers should use the complete version of a language and no partial flavours of a language.

However, the difference between them is that "liberal" consumers might allow producers that aren't fully compatible with the Language whereas "conservative" consumers will be less fault tolerant.

EdNote: I think related to principle of least power. The least powerful the language, the easier to have partial understanding?

Compatibility is defined for the producer and consumer of an individual text. Most messaging specifications, such as Web Services, provide inputs and outputs. Using these definitions of compatibility, a Web service that updates its output message is considered a newer producer because it is sending a newer version of the message. Conversely, updating the input message makes the service a newer consumer because it is consuming a newer version of the message. All systems of inputs and outputs must consider both when making changes and determining compatibility. For full compatibility, any output messages changes must be forwards compatible (for the older receivers aka consumers) and any input message changes must be backwards compatible (for the older senders aka producers).

Our treatise so far has described a fairly straightfoward evolution of a language, from a first version to a next version. However, extensibility and interoperability are usually directly related. It is an axiom in computing that the lower the optionality (which includes extensibility), the higher the chance of interoperability. Each and every place that extensibility is allowed in a language is also a place for a lack of interoperability. The interoperability problems can arise when producers and consumers do not agree on which version is being used in a text.

In addition to the explicit extensibility defined in a language, there is the actual extensibility and defnition of a language used in an agent. One significant way that divergent understanding can happen is when the actual language definition used is different between the agents. A classic example of this is "HTML TAGSoup". Much of the HTML software, particularly browsers, have an Accept Text Set that is larger than the definition of HTML. For example, many situations of missing end tags are processed without generating an error. This ensures the user experience, at least in the short term, is of higher quality. However, it does suffer long term problems with interoperability when the illegal texts are copied by mechanisms such as "view source". The reason is that the more undocumented strings that are in an Accept Text Set, the more difficult it is to achieve interoperability. The more liberal an agent in accepting texts by increasing the Accept Text Set through expanding the definition of the language, the more difficult interoperability is because not every agent may have the same Accept Text Set.

On the other extreme is XML. XML allows almost no extensibility in it's constructs. Name characters, Tag closures, attribute quoting and attribute allowed values are all very fixed. This has increased interoperability between implementations of XML. However, it has also made it very difficult to move to XML 1.1 because almost all changes are incompatible because of the lack of extensibility. The XML language design was very specifically trying to avoid the "HTML TAGSoup" problem, and it has arguably done that, at a cost of inability to version. These two extremes of design of extensibility exist because of well-thought design. The trade-off between extensibility, interoperability and the Accept Set was planned in advance. Language designers should do the same with their languages.

Analyze Trade-offs for Language: Language designers should analyze the trade-offs between extensibility, interoperability, and actual language Accept Set.

The cost of changes that are not backward or forward compatible is often very high. All the software that uses the language must be updated to the newer version. The magnitude of that cost is directly related to whether the system in question is open or closed.

A closed system is one in which all of the producers and consumers are more-or-less tightly connected and under the control of a single organization. Closed systems can often provide integrity constraints across the entire system. A traditional database is a good example of a closed system: all of the database schemas are known at once, all of the tables are known to conform to the appropriate schema, and all of the elements in the each row are known to be valid for the schema to which the table conforms.

From a versioning perspective, it might be practical in a closed system to say that a new version of a particular language is being introduced into the system at such and such a time and all of the data that conforms to the previous version of the schema will be migrated to the new schema.

An open system is one in which some producers and consumers are loosely connected or are not controlled by the same organization. The internet is a good example of an open system.

In an open system, it's simply not practical to handle language evolution with universal, simultaneous, atomic upgrades to all of the affected software components. Existing producers and receivers outside the immediate control of the organization that has publishing a changed language will continue to use the previous version for some (possibly long) period of time.

Finally, it's important to remember that systems evolve over time and have different requirements at different stages in their life cycle. During development, when the first version of a language is under active development, it may be valuable to pursue a much more aggressive, draconian versioning strategy. After a system is in production and there is an expectation of stability in the language, it may be necessary to proceed with more caution. Being prepared to move forward in a backwards and forwards compatible manner is the strongest argument for worrying about versioning at the very beginning of a project.

From NoahM:The draft is on pretty firm ground when it talks about the information that can be determined from a given input text per some particular language L. I think there are important compatibility statements we can and should make at just that level (see suggestions above), and we should separate them from statements about the compatibility of a particular pair of applications that may communicate using the language. Both are important to include, I think, but they should be in separate chapters, one building on the other. Once you've cleanly told a story about which information can be reliably communicated when sender and receiver interpret using different language versions, you can go on to tell a separate story about whether the applications can indeed work well together. To illustrate what I mean, here are examples at each of the two levels.

Language level incompatibility: Consider a situation in which the same input connotes different information in one version of a language or another. Without reference to any particular application, we can say that the languages are in that respect incompatible. For example, we might imagine a version of a language in which array indexing is 1-based, and a later version in which 0-based indexing is used; the information conveyed by any particular array reference is clearly in some sense incompatible, regardless of the consuming application's needs.

Application-level incompatibility: Now consider two applications designed render the same version of the HTML language. The same tags are supported, with the same layout semantics, etc. One of the applications, however, has a sub-optimal design. Its layout engine has overhead that grows geometrically with the number of layout elements. If you give it a table with 50 rows, it takes 3 seconds to run on some procesor. If you give it a table with 5000 rows it runs for 3 days. Question: is the second application "compatible" with the 5000 row input? In some ways yes, and in some no. It will eventually produce the correct output, but in practice a user would consider it incompatible. This illustrates that compatibility of applications ultimately has to be documented in terms meaningful to the applications. In this case, rendering time is an issue. I think we should not try in this finding to document specific levels of compatibility at the application level and we should especially not fall into the trap of trying to claim it's a Boolean compatible/incompatible relation; in the performance example, it's a matter of degree. So, the terminology needs to be specific to the application and its domain. I do think we can talk about some meta-mechanisms that work at the application level, such as mustUnderstand, but they should be in a section that's separate from the exposition of texts, information, and the degree to which information may be safely extracted from a given text when sender and receiver operate under differing specifications.

The current draft tries to take the approach that we will model application compatibility by defining a new language that is the flavor of (in this case HTML) that a particular consumer will successfully process, but the point is that "success" is sometimes a fuzzy concept. Do we have two languages for this example, one for the documents that completely break application #2 and another for those that just make it run slowly? That seems to be what the finding is doing today, and I'm not convinced it's the right approach. My proposal would be that we just point out the distinction and say: "This first section of the finding for the most part restricts its analysis to the limited question of: what information can be reliably conveyed when a producer and a consumer operate using different versions of what purport to be the same or similar languages? The later sections explore some techniques that can be used by applications to negotiate means of safe interoperation when sender and receiver are written to differing versions of a language specification."