Serenoa Final Review
Browser Based Editing

Brussels, 26 November 2013

Dave Raggett <dsr@w3.org>

This is an exploration of the design requirements, challenges and applicable technologies, rather than a finished editor.

This talk is available at http://www.w3.org/2013/Talks/quill/

Serenoa is an EU FP7 research project focusing on model-based user interface design, and dynamic adaptation to changes in the context of use.

This project has received funding from the European Commission’s Seventh Framework Programme under grant agreement nº 258030 (FP7-ICT-2009-5. Internet of Services, Software and Virtualization – STREP).

Note: many of the slide bullet points expand to reveal further details, click on the text to expand or contract the bullet point.

Outline of talk

Target Users
Separation of Design Concerns
Browser-Based Editing
Domain Models
Task Models
Abstract UI Models
Concrete UI Models
Adaptation Rules
Live Editing By Remote Team Members
Synchronization Across Levels of Abstraction
Demo
Future work?

Target Users

Web developers facing the challenge of targeting desktop, smart phone, tablet and TV
Distributed Project Teams working at different locations and often for different companies
Team members with different roles
- Business experts - who set business requirements
- Systems Analysts - who design domain models
- Systems engineers - who design application scripts
- Brand managers - who specify look and feel
- User interface designers - who design the UI
- Device experts - with intimate knowledge of particular devices
- Graphics designers - who craft the visual appearance
- Usability experts - who critique the design
- Localization experts - who customize for language and culture

Separation of Design Concerns

with rules for dynamic adaptation to changes in the context of use

Browser-Based Editing

Ubiquity of HTML5
Targetting Desktop and Tablet as developer tools
HTML, Scripting and Style Sheets
- CSS for rich control over appearance
- HTML5 Canvas for scripted graphics
- Mouse clicks and touch events
HTTP and Web Sockets
- XMLHttpRequest for RESTful client/server interfaces
- Web Sockets as better solution for asynchronous messaging
- JSON-RPC as basis for remote method invocation
- Asynchronous delivery of events, e.g. signalling changes by other team members, or for changes in the context of use
Node.js for prototyping server side behaviour

Domain Models

Defines the interface to the back-end
- Named properties
- Basic data types, e.g. number, boolean, string, array
- Structured data
- Methods, arguments, return values and call backs
- The challenges of asynchronous programming
Domain model annotations
- Default values
- Knowing when a property has been updated by user
- Read-only properties
- Validity constraints on individual properties
- Validity constraints spanning multiple properties
- Relevancy constraints across related properties
- Properties with Getter methods with their own UI

Example domain model as XML Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
elementFormDefault="qualified"
xmlns:ctic="http://idi.fundacionctic.org/mobSerenoaWS/application.wadl/xsd0.xsd">
  <xs:include
  schemaLocation="http://idi.fundacionctic.org/mobSerenoaWS/application.wadl/xsd0.xsd">
  </xs:include>    
  <xs:complexType name="uiDataType">
    <xs:sequence>
      <xs:element ref="login"/>
      <xs:element ref="user"/>
      <xs:element ref="agencies"/>
      <xs:element ref="cars"/>
      <xs:element ref="subscribe"/>
    </xs:sequence>
    <xs:attribute name="user_id" type="xs:string"/>
    <xs:attribute name="selected_agency" type="xs:string"/>
    <xs:attribute name="selected_car" type="xs:string"/>
  </xs:complexType>
  <xs:element name="uiData" type="uiDataType"/>
</xs:schema>

Richer domain model as WebIDL like format

@import "clauses.txt"
@import "vehicles.txt"

interface customer {
  first_name string;
  last_name string;
  email email_address;
  phone phone;
  young_driver boolean;
  marketing_opt_in boolean;
}

interface card {
  type {MasterCard, VISA, AMEX};
  number string;
  expiry_month month;
  expiry_year year;
}

interface itinerary {
  pickup_date date_time;
  return_date date_time;
  pickup_location location using find_location();
  return_to_origin boolean;
  return_location location using find_location();
  pay {now, at_location};
  #relevant return_location !return_to_origin
  #default return_location pickup_location
}

interface agreement {
  customer customer;
  itinerary itinerary;
  pay_with card;
  readonly included : clause[];
  readonly excluded : clause[];
  vehicle from vehicles(itinerary: itinerary) : vehicle[];
  extras extras;
  #relevant pay_with itinerary.pay=now;
}

interface address {
  name string;
  street string;
  town_city string;
  state_province string;
  zip_postal_code string;
  country string;
}

interface extras {
  gps boolean;
  infant_seat integer;
  booster_seat integer;
  child_seat integer;
}

abstract interface clause {
  name string;
}

abstract readonly interface location {
  type {normal, airport, station};
  airport_code string;
  address address;
  #relevant airport_code type=airport
}

abstract readonly interface vehicle {
  category {economy, compact, intermediate, standard};
  type {car, van};
  model string;
  photo image;
  doors {"2-4", "4-5"};
  passengers number;
  shift {manual, auto};
  luggage string;
  aircon boolean;
  miles_per_gallon number;
  price_per_day number;
  #relevant doors type=car
  #relevant luggage type=car
}

XML Schema can be hard to understand and lack the annotations
XPath and XSD are hard to reversibly map to object models

Task Models

Serenoa uses ConcurTaskTrees (CTT)
- Now available as W3C MBUI Task Models specification

Car-Rental demo Task model as XML

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<TaskModel xmlns:coop="http://giove.isti.cnr.it/cttcoop"
xmlns="http://giove.isti.cnr.it/ctt"
NameTaskModelID="Make Car Reservation">
  <Task Identifier="Make Car Reservation"
  Category="abstraction"
  Iterative="true" Optional="false"
  PartOfCooperation="false" Frequency=" ">
    <Name>name</Name>
    <SubTask>
      <Task Identifier="Enter Info" Category="interaction"
      Iterative="false" Optional="false"
      PartOfCooperation="false" Frequency=" ">
        <Name>name</Name>
        <TemporalOperator name="Disabling"/>
        <Parent name="Make Car Reservation"/>
        <SiblingRight name="Submit Request"/>
        <SubTask>
          <Task Identifier="Enter Car Info"
          Category="interaction" Iterative="false" Optional="false"
          PartOfCooperation="false" Frequency=" ">
            <Name>name</Name>
            <TemporalOperator name="Interleaving"/>
            <Parent name="Enter Info"/>
            <SiblingRight name="Enter User Info"/>
            <SubTask>
              <Task Identifier="Enter Category"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false"
              Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter Car Info"/>
                <SiblingRight name="Enter Colour"/>
              </Task>
              <Task Identifier="Enter Colour" Category="interaction"
              Iterative="true" Optional="false"
              PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter Car Info"/>
                <SiblingLeft name="Enter Category"/>
                <SiblingRight name="Enter Model"/>
              </Task>
              <Task Identifier="Enter Model" Category="interaction"
              Iterative="true" Optional="false"
              PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter Car Info"/>
                <SiblingLeft name="Enter Colour"/>
                <SiblingRight name="Enter Engine"/>
              </Task>
              <Task Identifier="Enter Engine"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <Parent name="Enter Car Info"/>
                <SiblingLeft name="Enter Model"/>
              </Task>
            </SubTask>
          </Task>
          <Task Identifier="Enter User Info"
          Category="interaction" Iterative="false"
          Optional="false" PartOfCooperation="false" Frequency=" ">
            <Name>name</Name>
            <TemporalOperator name="Interleaving"/>
            <Parent name="Enter Info"/>
            <SiblingLeft name="Enter Car Info"/>
            <SiblingRight name="Enter Other Info"/>
            <SubTask>
              <Task Identifier="EnterName"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter User Info"/>
                <SiblingRight name="EnterSurname"/>
              </Task>
              <Task Identifier="EnterSurname"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter User Info"/>
                <SiblingLeft name="EnterName"/>
                <SiblingRight name="EnterAddress"/>
              </Task>
              <Task Identifier="EnterAddress"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter User Info"/>
                <SiblingLeft name="EnterSurname"/>
                <SiblingRight name="EnterCity"/>
              </Task>
              <Task Identifier="EnterCity"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter User Info"/>
                <SiblingLeft name="EnterAddress"/>
                <SiblingRight name="EnterZIPCode"/>
              </Task>
              <Task Identifier="EnterZIPCode"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter User Info"/>
                <SiblingLeft name="EnterCity"/>
                <SiblingRight name="EnterCountry"/>
              </Task>
              <Task Identifier="EnterCountry"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter User Info"/>
                <SiblingLeft name="EnterZIPCode"/>
                <SiblingRight name="SelectGender"/>
              </Task>
              <Task Identifier="SelectGender"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Selection/Single_Choice</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter User Info"/>
                <SiblingLeft name="EnterCountry"/>
                <SiblingRight name="EnterBirthDate"/>
              </Task>
              <Task Identifier="EnterBirthDate"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter User Info"/>
                <SiblingLeft name="SelectGender"/>
                <SiblingRight name="EnterEmail"/>
              </Task>
              <Task Identifier="EnterEmail"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Monitoring</Type>
                <Parent name="Enter User Info"/>
                <SiblingLeft name="EnterBirthDate"/>
              </Task>
            </SubTask>
          </Task>
          <Task Identifier="Enter Other Info"
          Category="interaction" Iterative="false"
          Optional="false" PartOfCooperation="false" Frequency=" ">
            <Name>name</Name>
            <TemporalOperator name="Interleaving"/>
            <Parent name="Enter Info"/>
            <SiblingLeft name="Enter User Info"/>
            <SiblingRight name="Set Preferences"/>
            <SubTask>
              <Task Identifier="Enter Comments"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <TemporalOperator name="Interleaving"/>
                <Parent name="Enter Other Info"/>
                <SiblingRight name="EnterMaxBudget"/>
              </Task>
              <Task Identifier="EnterMaxBudget"
              Category="interaction" Iterative="true"
              Optional="false" PartOfCooperation="false" Frequency=" ">
                <Name>name</Name>
                <Type>Edit</Type>
                <Parent name="Enter Other Info"/>
                <SiblingLeft name="Enter Comments"/>
              </Task>
            </SubTask>
          </Task>
          <Task Identifier="Set Preferences"
         Category="interaction" Iterative="true"
         Optional="false" PartOfCooperation="false" Frequency=" ">
            <Name>name</Name>
            <Type>Selection/Multiple_Choice</Type>
            <Parent name="Enter Info"/>
            <SiblingLeft name="Enter Other Info"/>
          </Task>
        </SubTask>
      </Task>
      <Task Identifier="Submit Request"
      Category="interaction" Iterative="false"
      Optional="false" PartOfCooperation="false" Frequency=" ">
        <Name>name</Name>
        <Type>Control</Type>
        <TemporalOperator name="SequentialEnablingInfo"/>
        <Parent name="Make Car Reservation"/>
        <SiblingLeft name="Enter Info"/>
        <SiblingRight name="AccessService"/>
      </Task>
      <Task Identifier="AccessService"
      Category="application" Iterative="false"
      Optional="false" PartOfCooperation="false" Frequency=" ">
        <Name>name</Name>
        <TemporalOperator name="SequentialEnablingInfo"/>
        <Parent name="Make Car Reservation"/>
        <SiblingLeft name="Submit Request"/>
        <SiblingRight name="ShowResults"/>
      </Task>
      <Task Identifier="ShowResults"
      Category="application" Iterative="false" 
      Optional="false" PartOfCooperation="false" Frequency=" ">
        <Name>name</Name>
        <Type>Visualize</Type>
        <Parent name="Make Car Reservation"/>
        <SiblingLeft name="AccessService"/>
      </Task>
    </SubTask>
  </Task>
</TaskModel>

Car-Rental demo Task model as simpler text format

task "Make car reservation" {
  enables {
    task "Enter info" {
      task "Enter car info" {
        task "Enter category" using car.category;
        task "Enter colour" using car.colour;
        task "Enter model" using car.model;
        task "Enter engine" using car.engine_size;
      }
      task "Enter user info" {
        task "Enter name" using user.given_name;
        task "Enter surname" using user.family_name;
        task "Enter address" using address.street;
        task "Enter city" using address.city;
        task "Enter Zip Code" using address.zip_code;
        task "Enter country" using address.country;
        task "Select gender" using user.gender;
        task "Enter date of birth" using user.birth_date;
        task "Enter email" using user.email;
      }
      task "Enter other info" {
        task "Enter comments" using misc.comments;
        task "Enter max budget" using misc.max_budget;
      }
      task "Set preferences" using rental.preferences;
    }
    task "Access service" application;
    task "Show results" application;
  }
}

Revised Car-Rental demo Task model

task "make a reservation" {
  task "Customer details" using customer;
  enables {
    task "Pick up and return" {
      task "pick up details" {
        concurrent {
          task "find rental location" using itinerary.pickup_location;
          task "enter date and time" using itinerary.pickup_date;
          task "Return to pickup location" using itinerary.return_to_origin;
        }
      }
      task "return details" {
        concurrent {
          task "find location" using itinerary.return_location
            #precondition !itinerary.return_to_origin;
          task "enter date and time" using itinerary.return_date;
        }
      }
      task "Pay now or at pickup" using itinerary.pay;
    }
    task "pick car model" using agreement.vehicle;
    task "Choose extras" using extras;
    task "Review and book" using agreement;
  }
}

Graphical solutions for editing Task Models

Car-Rental demo Task model as CTT diagram
Conventional CTT diagrams are cumbersome to browse
- We could provide node expand/collapse with dynamic re-layout
- We could try to make the diagrams more compact with less need for scrolling
Voronoi Cells
- The partitioning of a plane with n points into convex polygons such that each polygon contains exactly one generating point and every point in a given polygon is closer to its generating point than to any other.
Nested Voronoi cells for Tree structured graphs
- Voronoi cells as a basis for keeping different sub-trees separate
- Start with root node which occupies rectangular diagram boundaries
- Layout its immediate child nodes, e.g. using force directed layout where nodes are charged particles connected by springs
- Child nodes constrained to be within the parent node's Voronoi cell
- Determine Voronoi cells for the child nodes
- Recursively process their child nodes in turn
Requires extension of Fortune's algorithm to polygonal boundaries
- Fortune's algorithm is a sweep line algorithm for generating a Voronoi diagram from a set of points in a plane using O(n log n) time and O(n) space.

Abstract UI Models

Independent of the device characteristics and modes of interaction
- To a first approximation, but breaks down for richer speech based UI where a different design methodology is required
- Some parts of the UI may not be appropriate for constrained devices like phones, and likewise, mobile devices offer additional choices in functionality
- That can be expressed in terms of annotations at the Task and Abstract UI level, e.g. as conditions ranging over the context of use

Car Rental demo as an XML Abstract UI model (ASFE-DL)

<?xml version="1.0" encoding="UTF-8"?>
<abstractUIModel xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     currentUnit="login_iu"
    xmlns="http://www.serenoa-fp7.eu/">
    <externalModelList>
        <externalModel URI="dataModel.xsd" id="data"/>
    </externalModelList>
    <!-- External Functions from http://idi.fundacionctic.org/mobSerenoaWS/application.wadl -->
    <externalFunctionList>
        <externalFunction name="login" >
            <rest resource="http://idi.fundacionctic.org/mobSerenoaWS/login/{user}/password/{password}" method="POST" definitionURL="http://idi.fundacionctic.org/mobSerenoaWS/application.wadl"/>
            <input name="user">
               <simpleType type="string"/>
            </input>
            <input name="password">
                <simpleType type="string"/>
            </input>
            <input name="id">
                <simpleType type="string"/>
            </input>
            <output name="login">
                <complexType xPath="/xs:complexType[name='login']" externalModelId="data" definitionURL="http://idi.fundacionctic.org/mobSerenoaWS/application.wadl"/>
            </output>
        </externalFunction>
        <externalFunction name="putUser">
            <rest resource="http://idi.fundacionctic.org/mobSerenoaWS/user/{id}" method="PUT" definitionURL="http://idi.fundacionctic.org/mobSerenoaWS/application.wadl"/>
            <input name="id">
                <simpleType type="string"/>
            </input>
            <input name="user">
                <complexType xPath="/xs:complexType[name='user']" externalModelId="data"/>
            </input>
        </externalFunction>
        <externalFunction name="getUser" >
            <rest resource="http://idi.fundacionctic.org/mobSerenoaWS/user/{id}" method="GET" definitionURL="http://idi.fundacionctic.org/mobSerenoaWS/application.wadl" />
            <input name="id">
                <simpleType type="string"/>
            </input>
            <output name="user" >
                <complexType xPath="/xs:complexType[name='user']" externalModelId="data" definitionURL="http://idi.fundacionctic.org/mobSerenoaWS/application.wadl"/>
            </output>
        </externalFunction>
        <externalFunction name="getAgencies" >
            <rest resource="http://idi.fundacionctic.org/mobSerenoaWS/agencies/user/{id}" method="GET" definitionURL="http://idi.fundacionctic.org/mobSerenoaWS/application.wadl"/>
            <input name="id">
                <simpleType type="string"/>
            </input>
            <output name="agencies">
                <complexType xPath="/xs:element[name='agencies']" externalModelId="data"/>
            </output>
        </externalFunction>
        <externalFunction name="getCars" >
            <rest resource="http://idi.fundacionctic.org/mobSerenoaWS/cars/user/{id}" method="GET" definitionURL="http://idi.fundacionctic.org/mobSerenoaWS/application.wadl"/>
            <input name="id">
                <simpleType type="string"/>
            </input>
            <output name="cars">
                <complexType xPath="/xs:element[name='cars']" externalModelId="data"/>
            </output>
        </externalFunction>
        <externalFunction name="getCarsByAgency" >
            <rest resource="http://idi.fundacionctic.org/mobSerenoaWS/cars/agency/{id}" method="GET" definitionURL="http://idi.fundacionctic.org/mobSerenoaWS/application.wadl"/>
            <input name="id">
                <simpleType type="string"/>
            </input>
            <output name="cars">
                <complexType xPath="/xs:element[name='cars']" externalModelId="data"/>
            </output>
        </externalFunction>
    </externalFunctionList>
    <!--Login-->
    <abstractInteractionUnit id="login_iu" name="Login" role="Init">
        <connections>
            <connection id="registration">
                <target interactionUnitId="registration_iu"/>
            </connection>
            <connection id="login_home">
                <target interactionUnitId="home_iu">
                    <condition operator="neq">
                        <entityReference xPath="/uiData/user/user_id" externalModelId="data"/>
                        <constant value="-1" type="integer"/>
                    </condition>
                </target>
            </connection>
            <back id="back_login"/>
        </connections>
        <grouping id="login_group">
            <edit id="login_username"/>
            <edit id="login_password"/>
            <activator id="login_activator" connectionReference="login_home">
                <events>
                    <activation>
                        <eventHandler>
                            <invokeFunction name="login">
                                <input name="username">
                                    <value>
                                        <entityReference xPath="/descendant::edit[@id='login_username']/@value"/>
                                    </value>
                                </input>
                                <input name="password">
                                    <value>
                                        <entityReference xPath="/descendant::edit[@id='login_password']/@value"/>
                                    </value>
                                </input>
                                <input name="id">
                                    <value>
                                        <constant value="-1" type="string"/>
                                    </value>
                                </input>
                                <output name="login" xPath="/uiData/user/login" externalModelId="data"/>
                            </invokeFunction>
                            <!-- if login is successful, initialize the user data -->
                            <if>
                                <condition operator="eq">
                                    <entityReference xPath="/uiData/login/valid" externalModelId="data"/>
                                    <constant value="true" type="boolean"/>
                                </condition>
                                <then>
                                    <invokeFunction name="getUser">
                                        <input name="id">
                                            <value>
                                                <entityReference xPath="/uiData/user/@id" externalModelId="data"/>
                                            </value>
                                        </input>
                                        <output name="user" xPath="/uiData/user" externalModelId="data"/>
                                    </invokeFunction>
                                </then>
                            </if>
                        </eventHandler>
                    </activation>
                </events>
            </activator>
            <!-- [davide] added a navigation grouping here -->
            <grouping id="login_navigation" role="navigation">
                <navigator id="registration_navigator" connectionReference="registration"/>
                <navigator id="login_back" connectionReference="back_login"/>
            </grouping>
        </grouping>
    </abstractInteractionUnit>
    
    <!--CTIC: According to the Data Structure section available at the wiki
		(http://serenoa.morfeo-project.org/wiki/index.php/Demonstrator_M18#Data_structure) there are some mandatory
		fields to be filled in by the user as part of the registration process
        [davide] The data model has minOccurs="1" for the mandatory fields...
    -->
    
    <!--Registration-->
    <abstractInteractionUnit id="registration_iu" name="Registration" role="Init">
        <connections>
            <connection id="registration_home">
                <target interactionUnitId="home_iu">
                    <condition operator="neq">
                        <entityReference xPath="/uiData/user/@id" externalModelId="data"/>
                        <constant type="integer" value="-1"/>
                    </condition>
                </target>
            </connection>
            <connection id="registration_login">
                <target interactionUnitId="login_iu"/>
            </connection>
            <back id="back_registration"/>
        </connections>
        <grouping id="register_main">
            <!-- [davide] I added username and password for managing the login -->
            <edit id="registration_username">
                <binding xPath="/uiData/user/preferences/preference[@name='username']/value" externalModelId="data"/>
            </edit>
            <edit id="registration_password">
                <binding xPath="/uiData/user/preferences/preference[@name='password']/value" externalModelId="data"/>
            </edit>
            <edit id="registration_name">
                <binding xPath="/uiData/user/preferences/preference[@name='first name']/value" externalModelId="data"/>
            </edit>
            <edit id="registration_lastName">
                <binding xPath="/uiData/user/preferences/preference[@name='last name']/value" externalModelId="data"/>
            </edit>
            <singleChoice id="registration_gender">
                <binding xPath="/uiData/user/preferences/preference[@name='gender']/value" externalModelId="data"/>
                <choiceElement value="male"/>
                <choiceElement value="female"/>
            </singleChoice>
            <edit id="registration_email">
                <binding xPath="/uiData/user/preferences/preference[@name='email']/value" externalModelId="data"/>
            </edit>
            <edit id="registration_birth_date">
                <binding xPath="/uiData/user/preferences/preference[@name='birth date']/value" externalModelId="data"/>
            </edit>
            <grouping id="registration_address">
                <edit id="registration_street">
                    <binding xPath="/uiData/user/preferences/preference[@name='street']/value" externalModelId="data"/>
                </edit>
                <edit id="registration_poBox">
                    <binding xPath="/uiData/user/preferences/preference[@name='PO box']/value" externalModelId="data"/>
                </edit>
                <edit id="registration_zip_code">
                    <binding xPath="/uiData/user/preferences/preference[@name='zip Code']/value" externalModelId="data"/>
                </edit>
                <edit id="registration_location">
                    <binding xPath="/uiData/user/preferences/preference[@name='location']/value" externalModelId="data"/>
                </edit>
                <singleChoice id="registration_country">
                    <binding xPath="/uiData/user/preferences/preference[@name='country']/value" externalModelId="data"/>
                    <choiceElement value="Italy"/>
                    <choiceElement value="Poland"/>
                    <choiceElement value="Spain"/>
                    <!-- many more -->
                </singleChoice>
            </grouping>
            <activator id="registration_activator" connectionReference="registration_home">
                <events>
                    <activation>
                        <eventHandler>
                            <invokeFunction name="putUser">
                                <input name="id">
                                    <value>
                                        <entityReference xPath="/uiData/user/@user_id" externalModelId="data"/>
                                    </value>
                                </input>
                                <input name="user">
                                    <value>
                                        <entityReference xPath="/uiData/user" externalModelId="data"/>
                                    </value>
                                </input>
                            </invokeFunction>
                        </eventHandler>
                    </activation>
                </events>
            </activator>
            <!-- [davide] added a navigation grouping here -->
            <grouping id="registration_navigation" role="navigation">
                <navigator id="login_navigator" connectionReference="registration_login"/>
                <navigator id="registration_back" connectionReference="back_registration"/>
            </grouping>
        </grouping>
        <events>
            <onRender>
                <eventHandler>
                    <!-- assumption: call getUser with id = -1 in order to get an empty one -->
                    <invokeFunction name="getUser">
                        <input name="id">
                            <value>
                                <entityReference xPath="/uiData/user_id" externalModelId="data"/>
                            </value>
                        </input>
                        <output xPath="/uiData/user" externalModelId="data"/>
                    </invokeFunction>
                </eventHandler>
            </onRender>
        </events>
    </abstractInteractionUnit>
    
    <abstractInteractionUnit role="main-menu" id="home_iu" name="Main menu">
        <connections>
            <connection id="home2preferences">
                <target interactionUnitId="preferences_iu"/>
            </connection>
            <connection id="home2userInfo">
                <target interactionUnitId="userInfo_iu"/>
            </connection>
            <connection id="home2searchInfo">
                <target interactionUnitId="searchInfo_iu"/>
            </connection>
            <connection id="home2login">
                <target interactionUnitId="login_iu"/>
            </connection>
            <back id="back_home"/>
        </connections>
        <!-- [davide] added a the role navigation here -->
        <grouping id="home_main" role="navigation">
            <navigator id="home_preferences" connectionReference="home2preferences"/>
            <navigator id="home_userInfo" connectionReference="home2userInfo"/>
            <navigator id="home_searchInfo" connectionReference="home2searchInfo"/>
            <navigator id="home_back" connectionReference="back_home"/>
            <activator id="home_logout" connectionReference="home2login">
                <events>
                    <activation>
                        <eventHandler>
                            <update>
                                <entityReference xPath="/uiData/user_id" externalModelId="data"/>
                                <value>
                                    <!-- resets the user id -->
                                    <constant value="-1" type="string"/>
                                </value>
                            </update>
                        </eventHandler>
                    </activation>
                </events>
            </activator>
        </grouping>
    </abstractInteractionUnit>
    
    <abstractInteractionUnit id="preferences_iu" name="Preferences" role="Preferences">
        <connections>
            <connection id="preferences2home">
                <target interactionUnitId="home_iu"/>
            </connection>
            <connection id="preferences2userInfo">
                <target interactionUnitId="userInfo_iu"/>
            </connection>
            <back id="back_preferences" />
        </connections>
        <grouping id="preferences_main">
            <grouping id="serenoa_preferences">
                <singleChoice id="location_awareness">
                    <binding xPath="/uiData/user/preferences/preference[@name='location_awareness']/value"/>
                    <choiceElement value="yes"/>
                    <choiceElement value="no"/>
                </singleChoice>
                <singleChoice id="color_blindness">
                    <binding xPath="/uiData/user/preferences/preference[@name='color_blindness']/value"/>
                    <choiceElement value="yes"/>
                    <choiceElement value="no"/>
                </singleChoice>
            </grouping>
            <grouping id="application_preferences">
                <singleChoice id="choose_by_agency">
                    <binding xPath="/uiData/user/preferences/preference[@name='choose_by_agency']/value"/>
                    <choiceElement value="yes"/>
                    <choiceElement value="no"/>
                </singleChoice>
                <singleChoice id="ordering">
                    <binding xPath="/uiData/user/preferences/preference[@name='ordering']/value"/>
                    <choiceElement value="price"/>
                    <choiceElement value="name"/>
                </singleChoice>
                <singleChoice id="location_selection">
                    <binding xPath="/uiData/user/preferences/preference[@name='location_selection']/value"/>
                    <choiceElement value="map"/>
                    <choiceElement value="list"/>
                </singleChoice>
            </grouping>
            <activator id="preferences_userInfo" connectionReference="preferences2userInfo">
                <events>
                    <activation>
                        <eventHandler>
                            <!-- save preferences -->
                            <invokeFunction name="putUser">
                                <input name="id">
                                    <value>
                                        <entityReference xPath="/uiData/user/@user_id" externalModelId="data"/>
                                    </value>
                                </input>
                                <input name="user">
                                    <value>
                                        <entityReference xPath="/uiData/user" externalModelId="data"/>
                                    </value>
                                </input>
                            </invokeFunction>
                        </eventHandler>
                    </activation>
                </events>
            </activator>
            <!--CTIC: We have added a new home role in order to semantically annotate the navigators
				pointing to the main-menu interaction unit
				[davide] such information is redundant, it is possible to calculate it from the
				connection + the role of the referenced abstract interaction unit
            -->
            <!-- [davide] added a navigation grouping here -->
            <grouping id="preferences_navigation" role="navigation">
                <navigator id="preferences_home" connectionReference="preferences2home"/>
                <navigator id="preferences_back" connectionReference="back_preferences"/>
            </grouping>
        </grouping>
        <events>
            <onRender>
                <eventHandler>
                    <invokeFunction name="getUser">
                        <input name="id">
                            <value>
                                <entityReference xPath="/uiData/user_id" externalModelId="data"/>
                            </value>
                        </input>
                        <output xPath="/uiData/user" externalModelId="data"/>
                    </invokeFunction>
                </eventHandler>
            </onRender>
        </events>
    </abstractInteractionUnit>
    
    <abstractInteractionUnit id="userInfo_iu" role="User" name="User Information">
        <connections>
            <connection id="userInfo2home">
                <target interactionUnitId="home_iu"/>
            </connection>
            <connection id="userInfo2searchInfo">
                <target interactionUnitId="searchInfo_iu"/>
            </connection>
            <back id="back_userInfo"/>
        </connections>
        <!-- more groupings ? -->
        <grouping id="userInfo_main">
            <!-- [davide] this interaction unit is actually really similar to the 
                 registration one... Can we collapse them?
            -->
            <edit id="userInfo_password">
                <binding xPath="/uiData/user/preferences/preference[@name='password']/value" externalModelId="data"/>
            </edit>
            <edit id="userInfo_name">
                <binding xPath="/uiData/user/preferences/preference[@name='first name']/value" externalModelId="data"/>
            </edit>
            <edit id="userInfo_lastName">
                <binding xPath="/uiData/user/preferences/preference[@name='last name']/value" externalModelId="data"/>
            </edit>
            <singleChoice id="userInfo_gender">
                <binding xPath="/uiData/user/preferences/preference[@name='gender']/value" externalModelId="data"/>
                <choiceElement value="male"/>
                <choiceElement value="female"/>
            </singleChoice>
            <edit id="userInfo_email">
                <binding xPath="/uiData/user/preferences/preference[@name='email']/value" externalModelId="data"/>
            </edit>
            <edit id="userInfo_birth_date">
                <binding xPath="/uiData/user/preferences/preference[@name='birth date']/value" externalModelId="data"/>
            </edit>
            <grouping id="userInfo_address">
                <edit id="userInfo_street">
                    <binding xPath="/uiData/user/preferences/preference[@name='street']/value" externalModelId="data"/>
                </edit>
                <edit id="userInfo_poBox">
                    <binding xPath="/uiData/user/preferences/preference[@name='PO box']/value" externalModelId="data"/>
                </edit>
                <edit id="userInfo_zip_code">
                    <binding xPath="/uiData/user/preferences/preference[@name='zip Code']/value" externalModelId="data"/>
                </edit>
                <edit id="userInfo_location">
                    <binding xPath="/uiData/user/preferences/preference[@name='location']/value" externalModelId="data"/>
                </edit>
                <singleChoice id="userInfo_country">
                    <binding xPath="/uiData/user/preferences/preference[@name='country']/value" externalModelId="data"/>
                    <choiceElement value="Italy"/>
                    <choiceElement value="Poland"/>
                    <choiceElement value="Spain"/>
                    <!-- many more -->
                </singleChoice>
            </grouping>
            <activator id="userInfo_activator" connectionReference="registration_home">
                <events>
                    <activation>
                        <eventHandler>
                            <!-- save preferences -->
                            <invokeFunction name="putUser">
                                <input name="id">
                                    <value>
                                        <entityReference xPath="/uiData/user/@user_id" externalModelId="data"/>
                                    </value>
                                </input>
                                <input name="user">
                                    <value>
                                        <entityReference xPath="/uiData/user" externalModelId="data"/>
                                    </value>
                                </input>
                            </invokeFunction>
                        </eventHandler>
                    </activation>
                </events>
            </activator>
            <!-- [davide] added a navigation grouping here -->
            <grouping id="userInfo_navigation" role="navigation">
                <navigator id="userInfo_home" connectionReference="userInfo2home"/>
                <navigator id="userInfo_searchInfo" connectionReference="userInfo2searchInfo"/>
                <navigator id="userInfo_back" connectionReference="back_userInfo"/>
            </grouping>
        </grouping>
        <events>
            <onRender>
                <eventHandler>
                    <invokeFunction name="getUser">
                        <input name="id">
                            <value>
                                <entityReference xPath="/uiData/user_id" externalModelId="data"/>
                            </value>
                        </input>
                        <output xPath="/uiData/user" externalModelId="data"/>
                    </invokeFunction>
                </eventHandler>
            </onRender>
        </events>
    </abstractInteractionUnit>
    
    <abstractInteractionUnit id="searchInfo_iu" role="Search" name="Car Search">
        <connections>
            <connection id="searchInfo2home">
                <target interactionUnitId="home_iu"/>
            </connection>
            <connection id="submit_data">
                <target interactionUnitId="agencyList_iu">
                    <condition operator="and">
                        <condition operator="eq">
                            <entityReference xPath="/uiData/user/preferences/preference[@name='choose_by_agency']/value" externalModelId="data"/>
                            <constant value="yes" type="string"/>
                        </condition>
                        <condition operator="eq">
                            <entityReference xPath="/uiData/user/preferences/preference[@name='location_selection']/value" externalModelId="data"/>
                            <constant value="list" type="string"/>
                        </condition>
                    </condition>
                </target> 
                <target interactionUnitId="geoAgencyList_iu">
                    <condition operator="and">
                        <condition operator="eq">
                            <entityReference xPath="/uiData/user/preferences/preference[@name='choose_by_agency']/value" externalModelId="data"/>
                            <constant value="yes" type="string"/>
                        </condition>
                        <condition operator="eq">
                            <entityReference xPath="/uiData/user/preferences/preference[@name='location_selection']/value" externalModelId="data"/>
                            <constant value="map" type="string"/>
                        </condition>
                    </condition>
                </target>
                <target interactionUnitId="resultsList_iu" >
                    <condition operator="eq">
                        <entityReference xPath="/uiData/user/preferences/preference[@name='choose_by_agency']/value" externalModelId="data"/>
                        <constant value="no" type="string"/>
                    </condition>
                </target>
            </connection>
            <!-- [davide] added a navigation grouping here -->
            <grouping role="navigation" id="searchInfo_navigation">
                <back id="back_searchInfo"/>
            </grouping>
        </connections>
        <grouping id="searchInfo_main">
            <!-- I assume that all the search preferences are saved as user properties and that
                the get agencies/cars operations will take into account them
                Maybe the cars and agencies resources should have the user id as parameter -->
            <edit id="searchInfo_location">
                <binding xPath="/uiData/user/preferences/preference[@name='location']/value" externalModelId="data"/>
            </edit>
            <edit id="accepted_distance">
                <binding xPath="/uiData/user/preferences/preference[@name='accepted_distance']/value" externalModelId="data"/>
            </edit>
            <singleChoice id="color">
                <binding xPath="/uiData/user/preferences/preference[@name='color']/value" externalModelId="data"/>
                <choiceElement value="red"/>
                <choiceElement value="green"/>
                <choiceElement value="blue"/>
                <!-- many more -->
            </singleChoice>
            <!-- [davide] I removed the grouping because there was only one child -->
                <activator id="request_submit" connectionReference="submit_data">
                    <events>
                        <activation>
                            <eventHandler>
                                <!-- save the currently edited preferences -->
                                <invokeFunction name="putUser">
                                    <input name="id">
                                        <value>
                                            <entityReference xPath="/uiData/user/@user_id" externalModelId="data"/>
                                        </value>
                                    </input>
                                    <input name="user">
                                        <value>
                                            <entityReference xPath="/uiData/user" externalModelId="data"/>
                                        </value>
                                    </input>
                                </invokeFunction>
                            </eventHandler>
                        </activation>
                    </events>
                </activator>
            <!-- [davide] added a navigation grouping here -->
            <grouping id="searchInfo_navigation" role="navigation">
                <navigator id="submitRequest_home" connectionReference="searchInfo2home"/>
                <navigator id="searchInfo_back" connectionReference="back_searchInfo"/>
            </grouping>
        </grouping>
        <events>
            <onRender>
                <!-- refresh the current user preferences -->
                <eventHandler>
                    <invokeFunction name="getUser">
                        <input name="id">
                            <value>
                                <entityReference xPath="/uiData/user/user_id" externalModelId="data"/>
                            </value>
                        </input>
                        <output xPath="/uiData/user" externalModelId="data"/>
                    </invokeFunction>
                </eventHandler>
            </onRender>
        </events>
    </abstractInteractionUnit>
    
    <abstractInteractionUnit id="agencyList_iu" role="Search" name="Agency List">
        <connections>
            <connection id="agencyList2home">
                <target interactionUnitId="home_iu"/>
            </connection>
            <connection id="agencyList2results">
                <target interactionUnitId="resultsList_iu"/>
            </connection>
            <back id="back_agencyList"/>
        </connections>
        <grouping id="agencyList_main">
            <!-- list reference and list item refence needed here -->
            <repetition id="agencyTemplate" listElementAlias="ag">
                <listReference xPath="/uiData/agencies/agency" externalModelId="data"/>
                <onlyOutput id="agencyList_itemName">
                    <binding xPath="ag/@name"/>
                </onlyOutput>
                <navigator id="agency_detail" connectionReference="agencyList2results">
                    <events>
                        <activation>
                            <eventHandler>
                                <update>
                                    <entityReference xPath="/uiData/@selected_agency" externalModelId="data"/>
                                    <value>
                                        <entityReference xPath="ag/@id" externalModelId="data"/>
                                    </value>
                                </update>
                            </eventHandler>
                        </activation>
                    </events>
                </navigator>
            </repetition>
            <!-- [davide] added a navigation grouping here -->
            <grouping id="agencyList_navigation" role="navigation">
                <navigator id="agencyList_home" connectionReference="agencyList2home"/>
                <navigator id="agencyList_back" connectionReference="back_agencyList"/>
            </grouping>
        </grouping>
        <events>
            <onRender>
                <eventHandler>
                    <invokeFunction name="getAgencies">
                        <input name="id">
                            <value>
                                <entityReference xPath="/uiData/user/@user_id" externalModelId="data"/>
                            </value>
                        </input>
                        <output xPath="/uiData/agencies" externalModelId="data"/>
                    </invokeFunction>
                </eventHandler>
            </onRender>
        </events>
    </abstractInteractionUnit>
    
    
    <!-- CTIC: Geolocated information -->
    <abstractInteractionUnit id="geoAgencyList_iu" role="Search" name="Agency Map">
        <connections>
            <connection id="geoAgencyList2home">
                <target interactionUnitId="home_iu"/>
            </connection>
            <connection id="geoAgencyList2results">
                <target interactionUnitId="resultsList_iu"/>
            </connection>
            <back id="back_geoAgencyList"/>
        </connections>
        <!-- CTIC: role added to support geolocation -->
        <grouping role="geo" id="geoAgencyList_main">
            <!-- list reference and list item refence needed here -->
            <repetition id="geoAgencyTemplate">
                <listReference xPath="/uiData/agencies/agency" externalModelId="data"/>
                <navigator id="geo_agency_detail" connectionReference="geoAgencyList2results">
                    <!-- [davide] please check if it is ok -->
                    <events>
                        <activation>
                            <eventHandler>
                                <update>
                                    <entityReference xPath="/uiData/@selected_agency"/>
                                    <value>
                                        <entityReference xPath="ag/@id" externalModelId="data"/>
                                    </value>
                                </update>
                            </eventHandler>
                        </activation>
                    </events>
                </navigator>
            </repetition>
            <!-- [davide] added a navigation grouping here -->
            <grouping id="geoAgencyList_navigation" role="navigation">
                <navigator id="geoAgencyList_home" connectionReference="geoAgencyList2home"/>
                <navigator id="geoAgencyList_back" connectionReference="back_geoAgencyList"/>
            </grouping>
        </grouping>
        <events>
            <onRender>
                <eventHandler>
                    <invokeFunction name="getAgencies">
                        <input name="id">
                            <value>
                                <entityReference xPath="/uiData/user/@user_id" externalModelId="data"/>
                            </value>
                        </input>
                        <output xPath="/uiData/agencies" externalModelId="data"/>
                    </invokeFunction>
                </eventHandler>
            </onRender>
        </events>
    </abstractInteractionUnit>
    
    <abstractInteractionUnit id="resultsList_iu" role="Search" name="Car Results">
        <connections>
            <connection id="resultList2home">
                <target interactionUnitId="home_iu"/>
            </connection>
            <connection id="resultList2CarDetail">
                <target interactionUnitId="carDetail_iu"/>
            </connection>
            <back id="back_resultsList"/>
        </connections>
        <grouping id="resultList_main">
            <repetition id="carTemplate" listElementAlias="car">
                <listReference xPath="/uiData/cars/car" externalModelId="data"/>
                <onlyOutput id="car_model">
                    <binding xPath="car/@model"/>
                </onlyOutput>
                <onlyOutput id="car_price">
                    <binding xPath="car/@price" />
                </onlyOutput>
                <navigator id="car_detail" connectionReference="resultList2CarDetail">
                    <events>
                        <activation>
                            <eventHandler>
                                <update>
                                    <entityReference xPath="/uiData/@selected_car" externalModelId="data"/>
                                    <value>
                                        <constant value="car/@id" type="string"/>
                                    </value>
                                </update>
                            </eventHandler>
                        </activation>
                    </events>
                </navigator>
            </repetition>
            <!-- [davide] added a navigation grouping here -->
            <grouping id="resultList_navigation" role="navigation">
                <navigator id="resultList_home" connectionReference="resultList2home"/>
                <navigator id="resultsList_back" connectionReference="back_resultsList"/>
            </grouping>
        </grouping>
        <events>
            <onRender>
                <eventHandler>
                    <if>
                        <!-- [davide] there was a not needed eq operator here. -->
                        <condition>
                            <entityReference xPath="/uiData/user/preferences/preference[@name='choose_by_agency']/value" externalModelId="data"/>
                        </condition>
                        <then>
                            <!-- [davide] changed to use the new car service -->
                            <invokeFunction name="getCarsByAgency">
                                <input name="id">
                                    <value>
                                        <entityReference xPath="/uiData/agencies/agency[@id=/selected_agency]" externalModelId="data"/>
                                    </value>
                                </input>
                                <output xPath="/uiData/cars" externalModelId="data"/>
                            </invokeFunction>
                        </then>
                        <else>
                            <!-- get the resource from the web service -->
                            <invokeFunction name="getCars">
                                <input name="id">
                                    <value>
                                        <entityReference xPath="/uiData/user/@user_id" externalModelId="data"/>
                                    </value>
                                </input>
                                <output xPath="/uiData/cars" externalModelId="data"/>
                            </invokeFunction>
                        </else>
                    </if>
                </eventHandler>
            </onRender>
        </events>
    </abstractInteractionUnit>
    
    <abstractInteractionUnit id="carDetail_iu" role="Search" name="Car Detail">
        <connections>
            <connection id="carDetail2home">
                <target interactionUnitId="home_iu"/>
            </connection>
            <back id="back_carDetail"/>
        </connections>
        <grouping id="carDetail_main">
            <onlyOutput id="carDetail_description">
                <binding xPath="/uiData/cars/car[@id=selected_car]/@description"/>
            </onlyOutput>
            <onlyOutput id="carDetail_price">
                <binding xPath="/uiData/cars/car[@id=selected_car]/@price"/>
            </onlyOutput>
            <!-- [davide] added a navigation grouping here -->
            <grouping id="carDetail_navigation" role="navigation">
                <navigator id="carDetail_home" connectionReference="carDetail2home"/>
                <navigator id="carDetail_back" connectionReference="back_carDetail"/>
            </grouping>
        </grouping>
    </abstractInteractionUnit>
</abstractUIModel>

Car Rental demo as a simplified text-based model,
generated automatically from the XML, see http://localhost/quill/asfe_dl/

dialog "Login" login_iu {
  connections {
    "Registration";
    "Main menu" when user.user_id != -1;
    $Back;
  }
}
dialog "Registration" registration_iu {
  connections {
    "Main menu" when user.id != -1;
    "Login";
    $Back;
  }
  event onRender{
    call getUser();
  }
}
dialog "Main menu" home_iu {
  connections {
    "Preferences";
    "User Information";
    "Car Search";
    "Login";
    $Back;
  }
}
dialog "Preferences" preferences_iu {
  connections {
    "Main menu";
    "User Information";
    $Back;
  }
  event onRender{
    call getUser();
  }
}
dialog "User Information" userInfo_iu {
  connections {
    "Main menu";
    "Car Search";
    $Back;
  }
  event onRender{
    call getUser();
  }
}
dialog "Car Search" searchInfo_iu {
  connections {
    "Main menu";
    "Agency List" when user.preferences.choose_by_agency == "yes" &&
      user.preferences.location_selection == "list";
    "Agency Map" when user.preferences.choose_by_agency == "yes" &&
      user.preferences.location_selection == "map";
    "Car Results" when user.preferences.choose_by_agency == "no";
    $Back;
  }
  event onRender{
    call getUser();
  }
}
dialog "Agency List" agencyList_iu {
  connections {
    "Main menu";
    "Car Results";
    $Back;
  }
  event onRender{
    call getAgencies();
  }
}
dialog "Agency Map" geoAgencyList_iu {
  connections {
    "Main menu";
    "Car Results";
    $Back;
  }
  event onRender{
    call getAgencies();
  }
}
dialog "Car Results" resultsList_iu {
  connections {
    "Main menu";
    "Car Detail";
    $Back;
  }
  event onRender{
    if user.preferences.choose_by_agency then {
      call getCarsByAgency();
    } else {
      call getCars();
    }
  }
}
dialog "Car Detail" carDetail_iu {
  connections {
    "Main menu";
    $Back;
  }
}

Visual representations of Abstract UI models
- ASFE-DL makes it easy to distinguish dialogues and interactors
- Dialogue as a graph of nodes connected by navigation links
- Interactors as the set of grouped Abstract UI interaction controls for each dialogue
Layout of Abstract UI diagrams
- Different people will have different preferences for the visual representations of models
- In addition, the browser windows will differ in size
- This motivates browser based automatic layout
- Persisting user changes to layout on a per user basis
Force based layout
- Treat nodes as charged particles that repel each other according to Coulomb's Law
- Treat links between nodes as springs that behave according to Hooke's Law
- Additional force that draws the diagram into the centre of the window
- Implemented with the HTML5 2D Canvas and using animation for fun, and to disguise the time it takes to compute the final layout

Concrete UI

The Concrete UI involves a commitment to a class of device and to modes of interaction
For a given Abstract UI, you need multiple Concrete UI models, with one CUI model for each class of device
- Desktop - Windows, Mouse Pointer, physical Keyboard
- Smart Phone - Small display, touch gestures, on-screen keypad
- Tablet - like smart phone but with a larger display
- TV - viewed at a distance, hence similar to small display, remote control, no pointer
- Plain Phone - interaction via speech and touch-tone keypad
- Others, e.g. for multimodal interaction
UI controls are device dependent, e.g. for desktop
- Rich text output with headings, paragraphs and lists, etc.
- Buttons with icons and captions
- Radio buttons
- Checkboxes
- Drop-down menus
- Images
- Single-line text input
- Multiline/rich text input
- Layout controls, e.g. horizontal, vertical, grid, etc.
Libraries of pre-built or custom controls for flexibility
Drag and drop editors with property sheets
UI skins for generating the Final UI for target devices
Testing - emulation of devices, e.g. smart phone emulator

Adaptation Rules

Dynamically adapting the user interface to match the context of use
AAL-DL rules take the general form
```
on event if condition then actions
```
- Actions may include further conditions (if-then-else)
- Requires a mechanism for resolving conflicting rules
- AAL-DL assumes need for event, but these aren't needed!
- Describe the UI for a given context of use rather than how to change the UI for any given change in the context of use
- This assumes the context manager can determine which rules to apply in any context

Example rules as XML format

<?xml version="1.0" encoding="UTF-8"?>
<!--
    Document   : london_rules.xml
    Created on : 14 giugno 2013, 11.06
    Author     : Marco
    Description:
        Purpose of the document follows.
-->
<ruleModel xmlns="http://www.serenoa-fp7.eu/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.serenoa-fp7.eu/ file:../ruleModel.xsd">
  <ext_model_ref
  URI="http://urano.isti.cnr.it/cm/schema1.xsd" model_id="ctx"/>
  <ext_model_ref
  URI="http://urano.isti.cnr.it/cm/schema2.xsd" model_id="cui"/>    
  <rule priority="0" name="outputGraphical" id="rule1">        
    <event>
      <simple_event event_name="onEnvironmentNoiseLevelIncrease"
      xPath="/user/environment/@noise_level" externalModelId="ctx"/>
    </event>
    <condition operator="gt">
      <entityReference xPath="/user/environment/@noise_level"
      externalModelId="ctx"/>
      <constant value="1000" type="int"/>
    </condition>        
    <action>
      <foreach>
        <in>
          <entityReference xPath="//*" externalModelId="uiModel"/>
        </in>
        <do>
          <if>
            <condition operator="eq">                            
              <entityReference xPath="class"
	      externalModelId="uiModel" />
              <constant value="inputElement" type="string" />
            </condition>
            <then>
              <update>
                <entityReference xPath="./@input"
		externalModelId="uiModel" />
                <value>
                  <constant value="graphical_assignment" type="string"/>
                </value>
              </update>
            </then>					
          </if>
          <if>
            <condition operator="eq">
              <entityReference xPath="class" externalModelId="uiModel"/>
              <constant value="tts" type="string"/>
            </condition>
            <then>
              <update>
                <entityReference xPath="./@output"/>
                <value>
                  <constant value="graphical_assignment" type="string"/>
                </value>
              </update>
            </then>
          </if>
        </do>
      </foreach>
      <update>               
        <entityReference
        xPath="/descendant::single_choice[@id='order_status']/@input"
        externalModelId="cui"/>
        <value>
          <constant value="graphical_assignment" type="string"/>
        </value>
      </update>
      <update>               
        <entityReference
        xPath="/descendant::single_choice[@id='order_status']/@prompt"
        externalModelId="cui"/>
        <value>
          <constant value="graphical_assignment" type="string"/>
        </value>
      </update>
      <update>               
        <entityReference
        xPath="/descendant::single_choice[@id='order_status']/@feedback"
        externalModelId="cui"/>
        <value>
          <constant value="graphical_assignment" type="string"/>
        </value>
      </update>            
    </action>        
    </rule>
    <rule priority="0" name="outputRedundant" id="rule2">        
      <event>
        <simple_event event_name="onUserMoving"
        xPath="/user/userActivity/@activityType" externalModelId="ctx"/>
      </event>
      <condition operator="or">
        <condition operator="eq">
          <entityReference xPath="/user/userActivity/@activityType"
          externalModelId="ctx"/>
          <constant value="walking" type="string"/>
        </condition>        
        <condition operator="eq">
          <entityReference xPath="/user/userActivity/@activityType"
          externalModelId="ctx"/>
          <constant value="running" type="string"/>
        </condition>        
      </condition>
      <action>
        <foreach>
          <in>
            <entityReference xPath="//*" externalModelId="uiModel"/>
          </in>
          <do>
            <if>
              <condition operator="eq">                            
                <entityReference xPath="class"
                externalModelId="uiModel" />
                <constant value="inputElement" type="string" / >
              </condition>
              <then>
                <update>
                  <entityReference xPath="./@input"
	          externalModelId="uiModel" />
                  <value>
                    <constant value="equivalent" type="string"/>
                  </value>
                </update>
              </then>					
            </if>
            <if>
              <condition operator="eq">
                <entityReference xPath="class" externalModelId="uiModel"/>
                <constant value="no_tts" type="string"/>
              </condition>
              <then>
                <update>
                  <entityReference xPath="./@output"/>
                  <value>
                    <constant value="redundancy" type="string"/>
                  </value>  
                </update>                             
              </then>
            </if>                    
          </do>
        </foreach>
        <update>               
          <entityReference
          xPath="/descendant::single_choice[@id='order_status']/@input"
          externalModelId="cui"/>
          <value>
            <constant value="equivalence" type="string"/>
          </value>
        </update>
        <update>               
          <entityReference
          xPath="/descendant::single_choice[@id='order_status']/@prompt"
          externalModelId="cui"/>
          <value>
            <constant value="redundancy" type="string"/>
          </value>
        </update>
        <update>               
          <entityReference
          xPath="/descendant::single_choice[@id='order_status']/@feedback"
          externalModelId="cui"/>
          <value>
            <constant value="redundancy" type="string"/>
          </value>
        </update>
      </action>
    </rule>
    <rule priority="0" name="outputGraphical" id="rule3">        
      <event>
        <simple_event event_name="onUserStill"
        xPath="/user/userActivity/@activityType" externalModelId="ctx"/>
      </event>
      <condition operator="eq">
        <entityReference xPath="/user/userActivity/@activityType"
        externalModelId="ctx"/>
        <constant value="standing/seating" type="string"/>
      </condition>                        
      <action>
        <foreach>
          <in>
            <entityReference xPath="//*" externalModelId="uiModel"/>
          </in>
          <do>
            <if>
              <condition operator="eq">                            
                <entityReference xPath="class"
                externalModelId="uiModel" />
                <constant value="inputElement" type="string" />
              </condition>
              <then>
                <update>
                  <entityReference xPath="./@input"
                  externalModelId="uiModel" />
                  <value>
                    <constant value="graphical_assignment" type="string"/>
                  </value>
                </update>
              </then>					
            </if>
            <if>
              <condition operator="eq">
                <entityReference xPath="class" externalModelId="uiModel"/>
                <constant value="tts" type="string"/>
              </condition>
              <then>
                <update>
                  <entityReference xPath="./@output"/>
                  <value>
                    <constant value="graphical_assignment" type="string"/>
                  </value>
                </update>
              </then>
            </if>                    
          </do>
        </foreach>
        <update>               
          <entityReference
            xPath="/descendant::single_choice[@id='order_status']/@input"
            externalModelId="cui"/>
          <value>
            <constant value="graphical_assignment" type="string"/>
          </value>
        </update>
        <update>               
          <entityReference
          xPath="/descendant::single_choice[@id='order_status']/@prompt"
          externalModelId="cui"/>
          <value>
            <constant value="graphical_assignment" type="string"/>
          </value>
        </update>
        <update>               
          <entityReference
          xPath="/descendant::single_choice[@id='order_status']/@feedback"
          externalModelId="cui"/>
          <value>
            <constant value="graphical_assignment" type="string"/>
          </value>
        </update>
     </action>        
  </rule>
</ruleModel>

Example rules as simplified text format,
generated automatically from the XML, see http://localhost/quill/aal_dl/

rule 1 "outputGraphical"

on EnvironmentNoiseLevelIncrease (noise_level)
if user.environment.noise_level > 1000 then
begin
  foreach *
  begin
    if class == "inputElement" then
    begin
      input = "graphical_assignment";
    end
    if class == "tts" then
    begin
      output = "graphical_assignment";
    end
  end
  single_choice.order_status.input = "graphical_assignment";
  single_choice.order_status.prompt = "graphical_assignment";
  single_choice.order_status.feedback = "graphical_assignment";
end

rule 2 "outputRedundant"

on UserMoving (activityType)
if user.userActivity.activityType == "walking" ||
  user.userActivity.activityType == "running" then
begin
  foreach *
  begin
    if class == "inputElement" then
    begin
      input = "equivalent";
    end
    if class == "no_tts" then
    begin
      output = "redundancy";
    end
  end
  single_choice.order_status.input = "equivalence";
  single_choice.order_status.prompt = "redundancy";
  single_choice.order_status.feedback = "redundancy";
end

rule 3 "outputGraphical"

on UserStill (activityType)
if user.userActivity.activityType == "standing/seating" then
begin
  foreach *
  begin
    if class == "inputElement" then
    begin
      input = "graphical_assignment";
    end
    if class == "tts" then
    begin
      output = "graphical_assignment";
    end
  end
  single_choice.order_status.input = "graphical_assignment";
  single_choice.order_status.prompt = "graphical_assignment";
  single_choice.order_status.feedback = "graphical_assignment";
end

Local scoping: allowing conditions and actions to refer to data passed by event, or bound by conditions

Live Editing by Remote Team Members

See the changes made by others as they make them
- Analogous to experience with Google Docs
Requires near real-time revision control and merging of changes
- Merge algorithms for changes to tree structured data, e.g. Domain, Task, Abstract UI, Concrete UI, and context of use models
- Merge algorithms for changes within text strings, e.g. static text controls
- Resolving conflicting changes, e.g. when I insert a child node and you delete its parent
- Bill Ritcher's paper "A Trustworthy 3-Way Merge"
Simplest approach is to manage this on the server
- Clients send proposed changes to server for review, server sends updates for the official (trunk) version
- Alternatively, elect one client in current editing session to act as the manager (senior editor) that merges changes and generates the new version
Editor serializes changes as mutation messages
- Messages formed from sequence of add, delete, move, and copy operations on trees or strings
- Tumbler notation for nodes in tree of structured data, e.g. 2.1.5 where each number indicates which child node to select and so on until you reach a leaf node
- Messages include a version number identifying the version of the model they apply to
- Manager transforms proposed changes to reflect their effect on current version of model
- Clients responsible for reverting to previous official version when applying changes sent by manager
- Each Client's undo/redo history only covers the changes made by that client
- Clients responsible for updating their local undo/redo history for consistency with current version
Web Sockets as basis for asynchronous messaging - avoiding the need for long polling with HTTP

Synchronization Across Levels of Abstraction

Research work on Model-Based UI has focused on top-down design
- e.g. with use of XSLT to generate Abstract UI from Task models, and Concrete UI from Abstract UI
- These transforms are hard to maintain, curtailing flexibility, and discouraging real-world developers from using model-based UI as a design paradigm
But many UI designers prefer to focus on Concrete UI
- Many people are poor at abstract thinking and prefer to discuss concrete UI design proposals
Relying on manual updates to synchronize higher levels of abstraction is error prone and costly -- as a result it is likely to be done badly, or not done at all
We need the editing tool to function as a collaborative assistant that actively helps you to maintain consistency across different levels of abstraction
- For example, if you are editing a Concrete UI model for the desktop, the assistant automatically updates the abstract UI and domain models, and then updates the Concrete UI models for the other platforms in your current project
But how to make this work in practice?

Collaborative Assistants for Model-Based UI Design

This is essentially a class of Expert Systems
- Software that emulates the behaviour of human experts
We can thus apply knowledge engineering as a methodology working together with UI design experts
- Identify a range of use cases
- Identify strategies and tactics for UI design for each use case
- Implement, test and revise, to cover an expanding range of use cases
The Car Rental Use Case

Note that the desktop and smart phone interfaces offer different capabilities -- one is not a subset of the other!
Initially considered using event-condition-action rules
- This approach requires a rule for every combination of changes and conditions, resulting in the need for large numbers of rules, that then become hard to maintain
Now believe that a generative approach is better
- Rules for generating next level down in abstraction stack
- Changes made by human designer serve as design constraints
- Limit search space through constraint propagation
- Dependency based explanation of conflicts
Abduction as basis for managing consistency
- Consistency can be expressed in terms of relationships across different levels of abstraction
- If you know certain facts and also that certain relationships hold true, then it is possible to infer additional facts that must be true if the relationship is to hold
- In practice, this can be treated as a generalization of relational table joins to include unification
- Testbed for abduction over relationships, see: http://www.w3.org/2013/01/abduction/
Lessons learned
- This is very much harder than originally anticipated!
- The XML representations of domain models, task models, and abstract UI models make it harder still due to the complexity of reasoning over XML schemas and the full range of XPath expressions
- Found it easier to use a simplified object model and easy to understand concise text representations for serializing the models
- A rich domain model makes it practical to generate proposals for task models
- It is then straightforward to generate the abstract UI models
- But the concrete UI involves lots of choices and the need for information on the likely size of UI controls as a basis for layout decisions
- There is a need for a way to rank choices to limit the UI designs that are presented to the human user for review
- The generative approach can record a decision tree for generating dependency based explanations
- A lot more work is needed to evolve a practical solution

Demo

The demo requires the server to be started first.
- cd ~/Projects/quill/server node wsd.js
- Then open the browser at http://localhost/quill/client/
Demo project settings, domain, task, abstract UI and concrete UI
Quill provides a tabbed interface
- Design patterns providing a library of UI patterns for the designer to browse through
- Low fidelity UI providing the means for discussing the UI design at a broad brush level
- Design Agenda for design tasks proposed by the user or the system
- Adaptation Rules for reviewing and modifying the rules that determine how the design adapts to changes in the context of user
- Domain Models that describe the information system underlying the application
- Task Models for high level description of the tasks to be carried out by the end user or the system
- Abstract UI for models that are independent of the target platforms and interaction modalities
- Concrete UI for models for specific target platforms
Quill screen shots

Project properties dialog

Abstract UI Tab

Concrete UI Tab
Other potential demos

Future Work?

An obvious market need for better tools to address increasing variety of devices people use to access services
We live in a world where design teams work remotely, and often for different companies
This stresses the value of tools that support remote collaboration
People vary in their ways of thinking and their personal preferences
Model-based UI design won't be widely adopted without support for collaborative design assistants
There is an opportunity for further work that focuses on the needs of real-world developers!
The source code for Quill is available for anyone to use under an open source licence
The W3C Ubiquitous Application Design Community Group provides an open forum for further discussion

Serenoa Final ReviewBrowser Based Editing