Authors (add yourself after contributing): Florian Daniel
- 1 Context and concept
- 2 iAPI annotation convention
- 3 Data sources
- 4 Forms
- 5 UI widgets
- 6 iAPI middleware
- 7 References
Context and concept
In this context, this proposal is motivated by the following observations:
- The Deep Web generally provides access only to a subset of what is provided through the Surface Web (e.g., there are many good Web sites without programmable APIs);
- Wrapping (extracting, scraping) is therefore a practice that will always be needed (e.g., because one nevertheless wants to reuse parts of that Web site);
- Wrapping of Web pages is currently neither supported technically nor expected by the Web site providers. Wrapper logic is therefore hard to implement and not robust against even minimal changes to a source page's HTML markup.
The idea pursued by the interactive API proposal is to design a simple annotation format adding meta-data to HTML markup, so as to enable the identification of reusable parts inside Web pages and their programmatic reuse. Annotations may come from the providers of the Web pages (directly inside the source markup) or from third parties (via external annotations of existing Web pages). The goal is to support the dynamic extraction and reuse of data, functionality and pieces of user interfaces (UI widgets), as well as the automated operation/navigation of Web sites in a stable and principled fashion. The artifact that provides these capabilities is called interactive API, short iAPI.
The name "interactive APIs" further hints at the second goal pursued by this initiative, i.e., that of providing an API that is interactive, that can be inspected, browsed and navigated just like a common Web page, and that - more importantly - can be profitably used also by non-programmers inside their Web browser. The vision is the foundation of a composition/reuse paradigm that is entirely based on UIs and the interactive manipulation of iAPIs: UI-oriented computing .
Making iAPIs operative requires the development of a new layer in the Web architecture stack, which makes UIs accessible programmatically and mediates between the Deep Web and the Surface Web. The approach of iAPIs is to split this layer into two complementary aspects: an annotation format that tells what can be reused and an iAPI middleware that knows how to reuse iAPIs. That is, the goal is to develop an approach that does not just annotate content, dumping the burden of the actual reuse on the developer, but that provides concrete, actionable development support to both developers and users.
The focus of this document is the annotation format. The iAPI middleware and development support is developed in parallel as open-source project hosted on GitHub (see the section "Parsing and using iAPIS: iAPI middleware").
iAPI annotation convention
iAPIs are inspired by microformats 2 (http://microformats.org/), which use the HTML class attribute to add meta-data to HTML markup. Tags inside the class attribute (separated by a space character) are interpreted by the browser as CSS classes to be applied to the respective HTML element, provided that the page's style definition contains a class with the tag as name. Otherwise, the tag is ignored.
The iAPI annotation proposes a set of pre-defined tags, i.e., properties, for the class attribute to specify a iAPIs. The root of any iAPI is the h-iapi property, which identifies the scope of an iAPI. For instance, the following example identifies the HTML table as iAPI:
<table id="1" class="h-iapi"> ... </table>
There may be more than one iAPIs per page. Each iAPI inside a page must therefore be scoped by an h-iapi property and uniquely identified by an HTML id attribute.
|h-iapi||Qualifies the annotated HTML element as iAPI (requires a valid HTML id attribute)|
In line with the convention proposed by the microformats 2 proposal, the iAPI annotation makes use of prefixes to facilitate the implementation of parsers. Specifically, the prefixes used in this document are:
- h-* = root classname that identifies the microformat (as for microformats 2)
- e-* = element to be parsed as HTML (as for microformats 2)
- p-* = text property (as for microformats 2)
- u-* = URL (as for microformats 2)
- i-* = input element (indicates input elements that allow filling content, not extracting content)
The goal of annotating data inside HTML markup is to support the extraction of structured data from Web pages, so as to enable their reuse inside other Web pages.
- A user wants to use the data rendered inside a table of a page authored by a provider inside his own web page.
- A user wants to use the data underlying a paginated table, without having to browser through the individual pages.
- A user wants to re-format fetched data inside his own page (e.g., using a list instead of a table).
Extracted data are independent of the HTML/CSS formatting of the source page and follow the data model shown in the following figure, which is essentially based on items, attributes and values, where values can be structured again with attributes and items, and so on.
Annotating custom data structures
Annotating data requires identifying a data source inside a page, that is, the HTML element from which data can be extracted, and describing the structure of the contained data. Data sources are identified using the h-iapi property together with a e-data property, which also allows to label the data source. Data structures are defined by identifying the data items and their attributes and labeling them. This technique allows the provider of the data source to specify arbitrary data structures and to provide them with the necessary semantics (the labels).
<table id="1" class="h-iapi e-data:Publications"> <tr class="e-item:Publication"> <td class="p-attr:Authors">F. Daniel and A. Furlan</td> <td class="p-attr:Title">The Interactive API (iAPI)</td> <td class="p-attr:Event">ComposableWeb 2013</td> </tr> ... </table>
|e-data:label||Qualifies the iAPI as data source; label provides a human-readable name for the data|
|e-item:label||Identifies data items inside a feed of data; label gives a name to data items|
|p-attr:label||Structures data items into attributes; label gives names to data attributes|
If the data to be made available for reuse complies with any of the existing microformats (see http://microformats.org), it is possible to use the respective microformat properties to annotate data inside an iAPI qualified as data source. There is no need for own data structures and/or semantics.
For example, the following properties make use of the h-card microformat to identify business contacts (URL and organization):
<div id="1" class="h-iapi e-data:Organization h-card"> <a class="u-url p-org" href="http://interactive-apis.org">interactive-apis.org</a> </div>
|h-card||Identifies the h-card microformat (http://microformats.org/wiki/h-card)|
Ideally, all existing microformats can be used inside an iAPI to annotate data. Which exact microformat is supported depends on the microformat parsing capabilities of the adopted iAPI middleware.
Linking external data sources
The use of custom annotations and microformats enables the extraction of data from HTML markup. Another option to make data available is to provide access to a remote data API, such as an XML or JSON web service. This allows the user to fetch data directly from the provider's back-end, instead of extracting them from the HTML markup, and may provide access to the full data available without, for instance, having to extract data from multiple pages (e.g., for paginated tables). An iAPI may contain both markup annotations and a link to an data API. It is up to the data provider to choose how to make data available.
<table id="1" class="h-iapi e-data:Publications u-json:http://source"> ... </table>
|u-json:url||Identifies a JSON data source; url specifies the URL of the data source|
|u-rss:url||Identifies an RSS data source; url specifies the URL of the data source|
|u-xml:url||Identifies an XML data source; url specifies the URL of the data source|
As for the structure of the data fetched from remote web services, this can be specified directly inside the root element of the iAPI with the addition of the element/key/query that uniquely identifies the respective element in the data source.
<table id="1" class="h-iapi e-data:Publications u-json:http://source e-item:Publication:pubs p-attr:Author:auth p-attr:Tile:title p-attr:Event:event"> ... </table>
|e-item:label:key||Identifies data items inside an external data source; label gives a name to data items; key tells how to identify the item in the data source, i.e., how to query the data source for data items|
|p-attr:label:key||Structures data items into attributes; label gives names to data attributes; key tells how to identify the attribute in the data source|
Again, which source data format is supported depends on the implementation of the adopted iAPI middleware. Annotation-wise, there is flexibility.
Forms provide access to remote application logic. They allow one to submit data (inputs), to get them processed remotely on the provider's web server, and to obtain responses (outputs) that depend on the submitted data. Reusing forms can therefore be seen as reusing remote application logic, just like when using conventional web services.
- A user wants to automate a set of search tasks.
- A user wants to extend a given data set with data accessed through a form.
- A user wants to integrate two or more applications by passing data among them.
The figure below illustrates the basic form model: a typical form interaction consists of filling a set of input fields/elements (e.g., text fields or checkboxes), submitting the entered data for processing, and inspecting the result. This latter may be provided inside the same page of the form or it may be provided in another page. In addition, forms may be split into multiple steps, i.e., pages, and may therefore require to traverse multiple pages to reach a final result.
Reusing a form as a programmatic function therefore requires annotating the form and input fields, which allow one to provide inputs, as well as the data produced by submitting the inputs for processing. Forms are annotated using the properties explained in the following; data are annotated as described above.
|e-form:label||Identifies a form inside a web page and assigns a label to it|
|i-text:label||Identifies a text input with name label|
|i-password:label||Identifies a password input with name label|
|i-file:label||Identifies a file input with name label|
|i-button:label||Identifies a button with name label|
|i-radiobutton:label||Identifies a radiobutton with name label|
|i-image:label||Identifies a image input with name label|
|i-reset:label||Identifies a reset button with name label|
|i-checkbox:label||Identifies a checkbox with name label|
|i-result:label||Identifies the result of the computation and assigns a label|
|i-submit||Identifies the form submission button inside the form|
A sample form that could be used for a log in:
<form class="h-iapi e-form:login" id="1" method="post" action="Login"> <input class="i-text:username" type="text" id="username" name="username"/> <input class="i-password:password" type="password" id="password" name="password" /> <input class="i-submit" type="submit" id="btnSubmit" value="submit"> </form>
The idea of UI widgets is to enable the extraction of complete pieces of UI (the so-called widgets) from web pages, however without the need to follow complex widget implementations and in a fashion that is almost for free to the developer sharing the widget.
- A user wants to copy and paste a given table into his own website.
- A user wants to include a full-fledged payment widget (e.g., PayPal) into his website.
- A user wants to construct a new UI out of existing pieces of UI.
While making available data or form functionality can be achieved by suitably annotating the respective HTML markup, making UI widgets reusable requires the provider of the UI widget to both annotate the widget and to follow some (simple) implementation guidelines, which guarantee that the widget can correctly be extracted at runtime:
- HTML markup: this can easily be extracted via copy/paste;
- CSS styles: these can be extracted by reading the browser's computed style properties for each element included in the HTML markup;
- Linked resources: these can be either not extracted at all (and the cloned UI widget refers to the original resources) or they can be copied locally (and the cloned widget refers to the local copies of the resources);
|e-widget:label||Identifies a UI widget inside a web page and assigns a label to it|
|u-css:url||References a possible CSS style sheet for the widget|
 F. Daniel and A. Furlan. The Interactive API (iAPI). Proceedings of ComposableWeb 2013 / ICWE 2013 Workshops, LNCS 8295, pp. 3–15, 2013, Springer.