W3C Augmented Reality Workshop (Day 2) -- 16 Jun 2010

<phila> scribe: phila

<scribe> scribeNick: Phila

Data, Realities, Things

Portholes and Plumbing: how AR erases boundaries between 'physical' and 'virtual'

Chris Burman, Connected Environments

Chris Burman's paper http://www.w3.org/2010/06/w3car/portholes_and_plumbing.pdf

background in building industry

Chris: I make the mobile app Pachbe
... Demos pachube
... see http://www.pachube.com/
... shows 8K feeds of electricity meters

Pachube written in C++ but plan is to make it mobile and Web based ASAP

Not based on gelocation, based on markers and buildings

Shows picture of online gaming den for 2

Need markers or hooks for orientin content

See "What Do We Need" slide

Should be able to process images showing parts of buildings

Markers on buildings Ok as a gimmick but not real world

Seen lots of demos of generic sufaces mixed with location data

Suggests we might start planning our cities using AR (ARban planning)

Mobile Web 3D "Still a bit rubbish"

Central question for me: If we're using a browser that sort of works and then someone creates a standalone APP that beats its quality which will people use?

One marker might lead to different experiences for differnet people based on context info

Lots of data about cities and objects within cities that we could interact with

<Mohit> people tend to use a uniform browser that a particular APP

<Mohit> *than

Chris: Dangers of full AR experience may impact negatively on society - may be too easy to change stuff, change the look of the person you're talking to etc.

follow @pachube

Pachube is struggling with its success - people keep thinking of new things to upload data about. Which is good!

Corporate and enterprise clients upload their data - they are our clients

Why should I upload my data? What's in it for the user?

It's somewhere to put data you may be recording anywhere

Also it's a way of targeting people creating devices that can talk to each other

We provide the infrastructure for people and organisations to talk to each other

people like to network their energy data to see what other people are doing

it's another form of social metadata

privacy aspects are clearly important. Currently we're open

we plan to add privacy layer

CP: You uses markers, tags and hooks as synonyms - deliberate?
... And triggers?

Chris burman: I loosely said hooks and I think that's what you call a trigger - orientation etc.

CP: Comment on openess/proprietary?

Chris: People who want the data don't care about the standard, and the people consuming the data aren't too concerned, they just want an API
... so not much interest in standards
... but the need to extend the platform/design means that we tend to throw JSON around

Exploiting Linked Open Data for Mobile Augmented Reality Vinny reynolds

paper is at http://www.w3.org/2010/06/w3car/exploiting_lod_for_ar.pdf

VR: I'm from DERI, a Sem Web institute

largest Sem Web institute in the world

90% working on SW

10% on sensor networks, mobile phones etc.

scribe: my job is to take the SW ideas and see if I can apply it to mobile

Describes the scenario from slide

Played with the tools. Thought about what wed like to do

Can we bring in government data sets on crime for e.g. to an AR experience in the street

we'd like to do stuff on recommendations, discovery etc.

Describes idea of static data. The data set is provided. Want to change the data, change provider

Doesn't scale

puts all responsibility on the content provider - it's not very Web-like

Current browsing experience doesn't support discovery. hard to find more related stuff

If I look at a historical building, I might also like info about other historical buildings

There's a lot of context info on your mobile device other than location that could be very useful in AR

Gives crash course in linked data

Shows LOD cloud slide

20 billion triples

Highlights existing important Geo-located data (GeoNames etc.)

DBPedia (the RDf version of Wikipedia) contains a lot of geolocated data

Can LOD address some of the AR issues? It's one way certainly

Expands on idea of following links from one item to another and how a user can explore and discover

Need some sort of balance between what you want to know and what's available.

Lots of context available from the user's device

Summarises discussion points

Highlights provenance XG

We're building all sorts of SW tools, incl. search

Talks through slide showing technology stack and what DERI is doing

WE look at your PIM< your FOAF file, accelerometer, Skype handle etc to work out if you're in a meeting, at play, walking, driving etc.

Dirk: LOD assumes every object has a URI
... what do you do where there aren't any URIs?

VR: we're trying to encourage folk to migrate to this way of thinking

Message from DERI and TimBL is 'jet get the data out there' - then we'll work on the representation

Chris Burman: How to filter out the spam if everyone's linking everything?

VR: we're not denying that there's a huge amount of data. Provenance XG is tackling that - it's a key area clearly

RJ: There's a lot of unstructured data - it needs more structure

<francois> Scribe: francois

Basic Concepts in AR Audio

jacques: [showing AR demo on the iPhone, with audio associated with buildings in Marseille]
... Working on audio and on MW4D (Mobile Web for Dev)
... We put sensors everywhere in the buidling, to help blind people find their ways.
... It's a bit different from what we've seen so far, we're working on guidance systems.
... Using 3D sounds, combined with an XML description of the building (Open Street Maps).
... Interesting because you can use the same software indoors and outdoors
... Depending on the size of the room, we change the audio as a consequence
... [demo with voice and 'heartbeat' at various frequencies, door knocking sounds]
... It works well on iPhones because the audio is good. Other phones have more crappy audio right now, so it's more difficult to do for the time being on other phones.
... Users are blind people, and visual-impaired people to help them confirm that there's a door somewhere, for instance.
... On the new iPhone, there are gyroscopes that we're missing so far for precision.
... Two kinds of AR systems, POI browsers and navigation systems.
... There's some link between the two so there will probably be convergence between the two.
... My definition of AR is real-time rendering of gro info through a tag-based dispatching languages with multimedia objects
... What we're doing on audio is very close to SVG. It's complex, but we have a player for the iPhone.

<phila> An audio SVG - fascinating idea IMO

jacques: [showing their AR platform]
... One language that is very important is Open Street Maps.
... It's an open format.
... the tag-based dispatching language I was talking about.
... For the audio objects, the language we use is A2ML, an XML Interactive Audio format with sound object models (cues).
... Open SL will be on Android and later on on Symbian phones it seems.
... The coold thing would be to have OpenSL everywhere and an API on top of that.
... We're using 3D sounds to speed up object discoveries.
... One problem is that you cannot isolate blind people from real sounds, so we can use bone conduction headsets or headphones with integrated microphones.
... We need formats.
... We need some audio stylesheet to associate sounds with geolocalized objects.
... We've started from iXMF from IAsig which has never really been implemented. We basically translated it into XML.
... People think about audio as just files, but you can go way beyond that. The idea is to put small chunks of audio put in a sequential container such as SMIL, make loops
... You can think of sound objects as small iPods.
... In audio, you need to "instantiate" models
... When you bring all of these ideas together, you get something similar to what happens in audio for games.
... The mixing stage could be viewed as CSS rendering for sounds.
... The content hierarchy definition format is really close to SVG.
... [demo of a small grain-based music that plays differently each time]
... Duration of each chunk can be 50ms.
... [demo of interactive jungle with a gun shot that makes the jungle quiet for some time]
... For indoor localization, we use all sorts of sensors, but the most important thing we need is a precise map

Dirk: What is your reference point?

Jacques: you start walking, make a right, a left, and we can determine where you are in the building

Rittwick: Thanks!

phila: That opens up so many possibilities with AR.

Dirk: yes. In Layar, you can already have sounds layers. Only streaming files for the time being, but that's interesting.

Panel discussion

[introducing panelists]

Rittwick: let me start with a quick overview of what Thomas Wrobel proposed in his position paper.
... He proposes some architecture to facilitate server to server exchange of geolocated data
... Email is distributed, one server that crashes is not the end of the world. It should be the same thing with geolocated data. It should be decentralized.
... He tries to leverage the WAVE Federation procotol from Google. A client associates with a server. Another client associates with another server. Servers exchange with each other.
... The good things is that there is no lock-in.
... [presenting WAVE]
... [example of a blip. A collection of blips is wave. A collection of waves is what the user sees]
... It's a proposal for how to exchange information, not how the information is structured.
... I have a feeling that GIS folks have already worked on that.
... [example of how a WAVE AR client could look like]
... It gives us interesting ideas.

Marc: I'm going to talk about Vuvuzelas to start with since we were talking about sounds.
... We're at a moment where people are tagging content.
... We're moving to a different environment in a near future. We're trying to think about how AR can interact with real objects in real-time.
... There are some people who need to interact with content with special needs.
... We need to think about how AR can be accessible.

Jordi: [Soundcapes from virtual environments to AR]
... We work on content-based analysis.
... Trying to analyse content automatically.
... Soundscapes relate to AR because it improves the immersion of the user.
... The example of the Berlin Wall yesterday could be completed with sounds of the revolution in 1989 for instance.
... Similar to the Ray-Ban example, we could think of an air guitar game.
... Demo is about soundscape generation.
... User uploads sounds to a repository.
... We can get info from the real world (traffic, weather), that influence the soundscape that is generated.

Nathaniel: [presenting the prototype]
... As opposed to a statically generated soundscape, you can generate a soundscape that is dynamically generated based on various factors included the user preferences.
... [demo that uses traffic info among other things to create a soundscape]
... [over Google Street Maps]
... The church that is in the background may not be playing when you drive by, but some user might have uploaded the sound.

Chris: how do you do spatialization when user record audio?

Nathianel: there is no magic solution. You can have a panning system for instance, so you can play the sound like that.

<phila> CP: Introduces final talks session

<phila> ... we've been getting towards applications for AR

<phila> ... intent for this session is to show that we need to create productive applications and not just impressive demos

Augmented Reality at IBM

<phila> Alex Phillips, IBM

<phila> paper is at http://www.w3.org/2010/06/w3car/augmented_reality_at_ibm.pdf

<phila> Alex: I've been looking around IBM at what's been going on. I spend most of my time working with Vodafone

<phila> Some of these examples are ones we've done with clients and others are research projects

<phila> Shows tag Cloud - Wimbledon and Vodafone strong!

<phila> Talks through goals and focus slide

<phila> Wimbledon Seer

<jjaner> This is the link to our soundscape generation demo: http://dev.mtg.upf.edu/soundscape/media/StreetView/streetViewSoundscaper2_0.html

<phila> App is really only useful if you're at the All England Club!

<phila> Last year's app was extremely popular during last year's competition

<phila> IBM works at every Grand Slam match

<phila> We have people at the court side (tennis experts) entering data in real time

<phila> Each point in the match creates around 200 data points!

<phila> Poi9nt the app at Centre Court and it will tell you what's going on all lots more

<phila> people walking around the site entering data

<phila> Will be on Android and iPhone this year

<phila> We provide the infrastructure to push the hige amount of data for each of the Grand Slam events

<phila> ACME project - doing this with Nokia and VTT

<phila> Shows video

<phila> Room sensors track gestures and so on

<phila> people can control their Second Life avatar from their laptops

<phila> Combination of VR and AR

<phila> Can pick up objects from the table and interact withinn the space

<phila> MAR Project

<phila> Augmenting the retail experience...

<phila> Use the handset within a store to give you more product info, hooks into social networks, find out reviews etc.

<phila> represented by avatars with different roles

<phila> See range of people: one may be the author, a reviewer, professional review etc.

<phila> Also useful for tech support/help desk

<phila> MAR is a research project, not a released product

<phila> Finally VIRA (Virtual Agent)

<phila> originally centred on remote support

<phila> Help agent can point at things and guide you

<phila> Outside the computing environment, a Web cam can be used and point to real objects

<phila> Vinny: Do you have plans to use the data, such as on queue length?

<phila> Alex: Good idea...

<DKA> Christine: One feedback from February - (AR summit) - we need feedback from trials and prototypes - [we need to be able to take learnings away from these in a public way.]

<DKA> ... it would be a great thing for the community to share some kind of statistics that could be revealed...

Manel Medina's talk on Payment / Security

<DKA> Manel: We are trying to apply knowledge of secure communications to AR.

<DKA> ... we have to consider user privacy, anonymity, etc...

<DKA> ... how users pay for information. We were discussing last night over dinner - how to do business with AR - who should pay for what?

<DKA> ... [presents slide] Telecom provider provides payment infrastructure...

<DKA> ... this makes it easier to launch and pay for new applications.

<DKA> ... [next slide] Four roles in the ecosystem: sponsor; advertiser; content provider, infrastructure provider

<DKA> ... advertiser can be partly a sponsor and partly a content provider...

<DKA> ... infrastructure provider - e.g. telecom operator - wants to launch services to make products (e.g. flat tariff) more attractive...

<DKA> ... since we are in a W3C workshop I think we have to consider the interfaces between these different roles. E.G. we have seen this morning how to interchange location info,...

<DKA> ... [next slides] different payment methods currently available on mobile - e.g. premium SMS, ...

<DKA> ... with WAP premium subscriptions we can get more, larger payments that could be used for subscriptions...

<DKA> ... [next slide] case study : geobuyme

<DKA> ... allow the users to find products they like if they identify them in a store ...

<DKA> ... [conclusion slide] triggering discussion: the components we need to consider if we want to build business on top of AR.

<DKA> ... Payment options: Bulk payment (sponsor); subscription (user); item payment (user)

<phila> CP: Christine invites questions for Manel

<phila> CP: Plugs her talk at Mobile 2.0 tomorrow

Augmented Reality and standards in the Building and Medical Industries

<phila> (no paper)

<phila> Building Information Modelling (BIM)

<phila> Very complex 3D Models

<phila> FIM is Fabrication Information Modelling (shows Norman Foster TED Talk)

<phila> using robotics you can use more organic shapes

<phila> Models also being used for Urban planning

<phila> Can see where the shadows will fall if builds were to be realised

<phila> Accoustic and thermal predictions

<phila> Building Industry AR

<phila> "Standard AR" - it's so last year (I paraphrase)

<phila> 2D drawings -> 3D model -> specific applications

<phila> Moving on to the medical industry

<DKA_> Scribe: Dan

<DKA_> ScribeNick: DKA

<DKA_> Damon: Issues with interoperability in the medical AR space.

<DKA_> ... cost need to come down.

<DKA_> ... MIAR 2010 - 5th year of this conference -

<DKA_> ... DiCOM - setting standards for medical visualisation.

<DKA_> ScribeNick: DKA_

Nathianel: presents youtube video of MedX3D
... using MedX3D format they can take a digital avatar of you taken using 3d scans to run surgical simulations, pharma simulations, etc..
... educational applications as well - premed - interactive skeleton embedded on san mateo medical center's web site. Uses scripting. Can be overlaid on a marker...
... [concludes]

Christine: Introduces panelists for "real world real business session.
... Pascal is VP of R&D at Total Immersion...

Pascal: don't work on standards - I think it's important for us and everybody. I could explain how standards could solve some needs.
... our focus is on general public applications.
... we started with maintenance applications... now online marketing and online retail ... also publishing ... AR books, AR trading cards, toys.
... currently we don't see needs for standards because it works - but if we look at the needs of our customers...
... applications must be robust and must work in various contexts. We developed apps for matel and McDonalds - quality is important.
... it needs to be cross-platform - PC, Mac, iPhone, Flash, Set Top Boxes...
... content creation - we must provide efficient tools to allow for content creation.
... security is another need - content must be secure. this relates to the image of our clients...
... maybe other needs as well - but these are important needs.

Christine: these are people who pay for the development of the applications...

Pascal: Yes.
... standardization could answer most of these issues. For example, we developed markerless tracking based on Flash platform. We could replace this with an HTML5 solution. Less easy to do [but wider platform deployment]...

Christine: [introduces Dan Romescu]

DRomescu: I was at the AR conference in the U.S. two weeks ago - some are not so open for standardization and collaboration.
... from my PoV - one problem for the Augmented Citizen will be privacy and security. An open augmented reality stack will help.

Christine: [next speaker]

Iban: I am from EUVE - European Virtual Engineering Foundation - We have a graphics engine...
... we adapt it to the needs of each project we develop. We are trying to make a step ahead - mix what we are doing with augmented reality...
... we have done projects with cultural heritage, virtual cities, tourism... user can go inside buildings, etc...
... now we are trying to research location inside of buildings...
... libraries, airports, these kinds of buildings...
... we are also researching about future internet and the content to include in the future internet...
... the contents in the internet are contributed by the users.

Christine: [next]

Martin: I Martin Gonzalez Rodriguez HCI Research Group, University of Oviedo - http://www.w3.org/2010/06/w3car/are_accessible_to_disabled_users.pdf
... augmented reality for trying to guide blind users in a transport system [in a city]
... on a bus stop, the bus will have a GPS antenna - when the right bus stops in front of the user we inform the user with audio - once inside we inform them when to get off..
... next step - a new project for children with CP or Autism - to know about the surrounding world using Augmented Reality.
... example - 3 years old - she can only move her neck and hand - we can use augmented animated figures...
... next example: child with Autism - we don't know what is reality for this child...

Christine: [introduces Nzube Ufodike - http://www.w3.org/2010/06/w3car/modelling_language_interface.pdf]

Nzube: My academic interest - I am interested in healthcare - monitoring and assisting geriatric patients....
... many of the use cases I've heard at this event are related...
... we don't have standards in this space.

Christine: This subject of medical applications and healthcare - in 18 months at ISMAR 2011 we could have a program on this topic as well as intelligent cities.
... any questions?

Rittwik: Can we talk about smart cities / augmented citizen?
... without the need to deploy sensors everywhere - how can a service provider help?

Dromescu: the social impact of AR... A strong point coming from the games - how will the future organization of cities be?
... how can we organize our cities and urban life in the future? Bruce Sterling said (in SF event) we need more information - more virtual information ...
... [we could build models on top of that on what will happen when people move from one part of the city to another]

Christine: How can service providers help with senors - ...

Chris Burman: From our experience - that's very segregated - construction industry people have no interest in making this data public. Closed industry - they have their own closes system.

scribe: we need to get as much of that data as possible...

Christine: I think Jaques was talking about navigation in buildings...

Vinny: Smart spaces - taking sensors and making that info available to mobile users... pervasive computing...
... smart spaces...
... I am involved in a project combining smart spaces with semantic web...

Christine: In the city of Basel - there is an AR research project going on - detecting the environment and overlaying on top of it...

Damon: Touching the cities - Autodesk has a "digital cities" initiative - cities are going digital.
... energy usage, policy, where are the pipes under the street...

<phila> Damon: AR - SIM Cities for the Real World

Christine: I was advocating to make all AR in rural areas free.. when we reach into more dense urban environments that is a richer environment to explore..

Dromescu: I talk to artists - they want to have tools to expose their work and ideas --
... Many artists are thinking "how can I give more value to my environment?"
... for advertisers it will be a very hard time because people don't want to see it - so advertisers have to think differently.

Iban: I have noticed this as well...

[some discussion on other models for montizing AR besides advertising]

Dromescu: We are still in a broadcast mentality. In the future we will see other kinds of advertising...

<phila> Nzube: Where there is no precedent, it's important that industry goes ahead and standards can catch up

<phila> ... you don't need to wait for the standards to be created

<phila> DRomescu: Companies need to be sustainable now, yes

<phila> CP: Closes the session and thanks the panel

<phila> CP: Explains the structure of the rest of the day

<phila> Discussion about patents. Offer to let chairs know about possible patents. This can be done anonymously. The existence of such patents would be included in the report but not who mentioned them.

<phila> CP: Picks on people to lead the 3 sub groups

<phila> (Vinny, Damon and Chris Burman)

What are the known standards, where are the gaps

<phila> Break for lunch

<DKA> SmellML!

<DKA> [We are asking "team leaders" Damon, Vinny and Chris to summarize their lunch discussions - this is being captured by Christine on flip chart pages which will be transcribed into the record]

<phila> DKA: Summaries the workshop...

<phila> scribeNick:PhilA

DKA: Again refers to Jonghong's 12 steps
... My take away is that there is a clear consensus around the need for a POI data standard
... Not minimising 3D or otehr stuff we have talked about
... the geolocated description of those points of interest seems to me to be the clearest gap
... the 3D stuff seems to be ticking along quite nicely thank you

<DKA> PhilA: Presents slides on "ways forward" - 3 options: do nothing; charter an XG; charter a WG

<inserted> Scribe: francois

DKA: Yesterday, before the workshop started, I thought I would be supporting option 2. I would have never thought there would be enough momentum to go for a WG.
... I changed my mind based on the interest I heard here.
... This WG, if created, should follow the steps of the geolocation API, do something precise and simple.

Timo: I think that's a good idea. I would certainly try to participate.

Damon: I definitely confirm that I would join such a WG. I can't speak for the Web3D consortium right now, although I know there's interest there either.

DKA: Wolfgang, in principle, would you support such an effort?

Wolfgang: Definitely. The intent was for ARML to be open.

Jens: This would open up the possibilities. What if people take your content away?

Wolfgang: Right, we see content as crucial to our business, so there has to be some protection, for sure.

DKA: Nothing presents anyone from adding proprietary extensions on top of a standard.

Alex: It seems to me that the format is irrelevant here. The problem is setting the data as public data or not, nothing to do with the standard.
... I would go for option 3.

Karl: SonyEricsson would support option 3 as well.

Claudio: Options 2 and 3 are good. Do we expect HTML5 to support some geo feature in the near future?

DKA: I dont' think we can jump in and say that HTML5 is going to be an AR platform.
... But what this group could do is give advice as to how HTML5 could be used to make this possible.
... This group could publish a note describing how this could work.

phila: Right. I think the POI format is definitely the normative document that should be in the charter, but the group should also publish notes on other points mentioned today: 3D, audio, ...

Claudio: Let me clarify, I cannot come back and say that W3C recommends a POI format, but then AR then renders these POIs through proprietary systems.

phila: I think that's kind of application-specific. Browsers could add this functionality if they so wish. People might develop plug-ins as well.

DKA: I think developing an AR experience within the browser will be possible once the appropriate APIs are developed. The only missing part so far is the POI thing.
... Since there's a concrete proposal on the table, I think it's a good idea to kick-off the work ASAP, while still thinking about how it fits within the architecture of the Web and producing notes.

Jonathan: I think ETRI would support that work, yes, and hope I can participate.

Rittwick: I think there's something similar going on on AR in OMA.
... I don't know what the overlap is, so need a concrete agenda.

Thomas: [telefonica] I need to check what other efforts are happening in Telefonica to figure out what our involvement would be.

<KlasH> What was the login for the wifi again?

Thomas: So maybe more as an observer.

Vinni: My initial gut feeling would be to go for Option 2, but provided we can do that exploratory thing with option 3, then I think that's fine.

Jacques: I agree the work could be interesting, we could have some ideas.

phila: Does anyone think it's a bad idea?

chris: I thought the outcome of this workshop would be Option 1. I would now go for Option 2.
... What could be interesting is groups of POIs.
... How you aggregate POIs. It's much more tricky.

phila: There are also people in this room who are not members but who could contribute. Would that help for you?

Margherita: How would you normally ensure that people in this room do not decide for the rest of the world?

phila: If we go forward, I'll probably need to draft a charter for the working group, with some chairs.
... This charter has to go through the membership, and we'll publicize it as much as we can.
... We usually reach people we need to reach for a specific topic.

Damon: Note I would almost be against Option 3 if we don't take a look at existing standards and make sure that things work well together.
... Would it make sense to include people from [scribe missed that]?

phila: Definitely. We need to get back to people behind KML, KARML and the like.

<JonathanJ> LG electronics developing the Mobile AR in OMA. If we will ogarnize the new AR WG in W3C, they will join us.

Damon: OK, industries are looking for something that is more robust. KML doesn't integrate our needs.

Mohit: From the point of view of universities, having standards would help PhD and the like, yes.

<JonathanJ> Mobile AR in OMA - http://www.w3c.or.kr/~hollobit/ARKR/20100603-workshop/arsw2-3.pdf

phila: In short, I believe I have your approval to get back to my hierarchy about the creation of a WG, then.

[Discussions about having a mailing-list and a Wiki to create a charter]

phila: OK, create a mailing-list which will need to be public, and that will be used as starting point to write a charter.
... It will take some time, it's summer, the membership has to have time to review the charter, and so on.

DKA: Our first F2F could be in Seoul for the next AR summit, actually.

phila: Anything before I wrap up?

[Thank you from the crowd]

<JonathanJ> please refernece it. 2nd AR Standardization Workshop Report (in Korea) - http://bit.ly/dgBfis #w3car

- DRAFT -

W3C Augmented Reality Workshop (Day 2)

16 Jun 2010

Attendees

Contents