See also: IRC log
<chunming> scribe: chunming
[phila give an overview of CEO-LD and agenda]
Geoffrey Boulton, CODATA and Edinburgh University
Jianhui Li, CNIC, CAS , CODATA China
Xuezhi Wang, CNIC, CAS
Xianqian Chai, China Mobile (28th morning)
Jinwei Wang, China Mobile
Xiaohai Li, China Mobile (28th morning)
Bill Roberts, SWIRRL
Maik Riechert, Reading University/MELODIES project
Chongping Wang, AC Rep of China Mobile, W3C Members
Qinghua Zhao, from CESI. Big data national standardization
Qun Zhang, CESI, W3C Member
Jitao Yang, Beijing Language and Culture University/RADI
Guoqing Li, Institute of Remote Sensing and Digital Earth, CAS
scribe: remote sensing scenario, discuss what is coverage, and apps from China
Jeremy Tandy, UK Met office
scribe: W3C/OGC WG on geospatial
data on the web
... coverage is one of type of data, that we can't use well on
the web
Denise Elizabeth McKenzie, OGC
scribe: the joint wg on
geospatial data on the web
... coverage data is piece of data collected golbally
Tianyu Wo, Beihang University/W3C
scribe: share some report of usecase on UCar services, a internet car rental data sharing
Phil: W3C Data Activity
[feel free to say in both english/chinese, ask for slow down if needed]
<scribe> [new coming: Yingying Chen from W3C/China]
<scribe> [new coming: Xueyuan Jia from W3C/China]
phila: introduction to spatial
data on the web wg
... coverage in linked data
... CEO-LD project: we believe that it will soon be a big
project
... do some significant development work
... setting the framework, do development, testing, ...
... in the moring : collecting information
... presentations from chinese colleges
... talk about use cases
W3C OGC Spatial Dat aon the Web Use Cases & Requirements
http://www.chinaw3c.org/archives/1208/
https://www.w3.org/TR/2015/NOTE-sdw-ucr-20151217/
phila: requirements on coverage
data
... section 4.4, https://www.w3.org/TR/2015/NOTE-sdw-ucr-20151217/
... an usecase
<phila> scribe: phila
chunming: Offers a summary in Chinese
Guoching: I want to introduce my
project. It's a very technical one in China, it's the biggest
approach here to use the next gen in sensing applications
... Lots of things, lots of data challenges
... Lots of data questions are not resolved. I can use just one
slice in my report to communiate our topic with coverages
... Coverages is not a topic in our peoject at the moment
<jtandy_> S/communicate/communicate/
Guoching: China remote Sensing
Network (CRSNet) is a project supported throughout
2011-2016
... Two stages - break down what we have and then
integrate
... How to use sensors together to resolve the problems at
global scale
... Forst level is at cyclic level, second is at data level,
connecting different data centres
... Third level is the workflow
... Quantitative common products and thematic products
... Want to transfer the service of remote sesing data to
service of remote sensing product
... All observation is based on the common product
... Integrating satellute, airt and ground sensors
... More than 40 types of multi scale products (now 58)
... Things loike mineral paramaters, radiation budgets,
vegtation structure, global ice change
... product can be generated at different geo and temporal
scales
<jtandy_> S/like/like/
Guoching: 1km layer, 100m layers and 10 m layers
<jtandy_> S/loike/like/
Guoching: Can match each other
automatically
... For the end product, the semantic layer... different land
use types
<chunming> s/otehr/other/
Guoching: Talks through flow
diagram of the system
... Needs online processing
... can't do online processing with muti-dimensions
... Geospatial and temporal is not eneough. Need waveband,
semantics... Services only provide one dimension
... The challenge is how to connect the data together.
... We want to process it on the provider side, and in the
middle (brokers) - but they don't have the knowledge to do
it.
... We have a broker in the middle, who understands the
different datasets, the geoloc, the waveband, the
semantics
... We don't cover all granularities, so 30m might be
normalised to 10 m, for eg
... Then there's the temporal dimension. We may get data twice
a day but we may only want it every 8 days. The broker will
take care of that
... Another aspect is the standards. So the water users, the
agricultural users etc. it should be the same.
... The current data model may have one or two dimensions, but
we need multi-dimensions
... IF there were a technology for the Web user and the
providers so the host base can make the data available for our
workflow..
... We have decided on a very complex workflow because there is
no such service from the provider
GB: One of the reasons we have
this project is because the use of Satellite data is not used
as much as it shoiuld be - because you need to be an
expert.
... We want the data to be better used.
... The data processing etc. is traditional - but is RADI
interested in making the imagery Web accessibkle
GQ: It is my intention, yes
jtandy_: I heard it's not just
remote sensing - it's airborne, UAVs, in situ
... I think here's we're looking at space borne here but the
SDW is looking more broadly
GQ: I haven't mentioned too much
about airborne, it's more complex. We just use the airbonre
data, but it's really complex.
... So when you take your raw data and trun it into your common
product, are you using s atandard grid, standard CRS etc?
... Yes
JT: So the data can be staked and used together
GQ: Yes
JT: I think there's a lot of similarity between what you're doing and what Maik is doing, combining different datasets.
MR: I'll try and cover a bit of everything in my time slot.
<chunming_> wiki page: https://www.w3.org/2015/ceo-ld/wiki/Beijing_Workshop
<chunming_> scribe: chunming
<chunming_> phila: different source of data --> common services
<chunming_> ... something here, magic happens
<chunming_> ... do you see web technology mediate as a good media to make data to different people
<chunming_> ... how much of the satalitte available to public, the world.
<chunming_> GB: make use to other body, other countries, without go through formal process
<chunming_> Guoqing: first factor: capability ... to make the data accssed
<chunming_> ... second., the policy.. at this moment, china user is fine
<chunming_> ... that's why we collaborate with Jibo, with GCR team (?), a italian parter
<chunming_> ... to make a portal to geo data
<chunming_> ... as Chinese as a data services in RADI
<chunming_> phila: for UK china collaboration ?
<chunming_> ... but it is not a unique chinese thing
<phila> GQ: Study is OK, operational may be more of a problem.
<chunming_> Xuezhi Wang: The public services of Geospatial data cloud
<chunming_> ... why this project? in 2008, USGS opened all geospatial datasets to public for free
<chunming_> ... platform objectives: construct a one-stop data services
<chunming_> ... three ways:search by address name, geo-loction, administrative unit
<chunming_> ... data resources including LANDSAT, MODIS, ..., 350+ TB data entities, 7 million data records, coveraging China area
<chunming_> ... some screen snapshots
<chunming_> ... result page, shows the datasets using a tabular view
<phila> DEM is Digital Elevantion Model
<chunming_> ... traditional way to get dataset: download DEM data, mosaic DEM data with ARcGIS, and clip DEM data with Beijing shape mask
<chunming_> ... better way, ... @@@
<chunming_> .... overall framewok: tasks to a VM cloud on data processing and storage
<chunming_> ... provide online category for spatiality tools and dataset.
<chunming_> ... use can submit tasks on geting, processing datasets according to the special task (filters)
<chunming_> ... after task finished, user can download the data
<chunming_> ... have over 100k chinese users now
<chunming_> ... average download over 200GB/day
<chunming_> Xuezhi: Geospatial Data Visualization tools in the platform
<chunming_> ... overall framework, using web client HTML5 map client
<chunming_> ... provide dynamical cache , and map styles editors
<chunming_> ... web client communicate with backends cloud VMs
<chunming_> ... in this system, user can render multi-bands raster layer with RGB false color
<chunming_> ... combine diffenent data , and rendering in different layer
<chunming_> ... user can edit the vectors on web page
<chunming_> ... publish landsat TM dataset
<chunming_> ... TM semantic mapper
<chunming_> [xuezhi show demo on the system]
<chunming_> ... search TM satalitte data with selecting Beijing area
<chunming_> ... user can changed the waveband
<chunming_> http://giscloud.cn
<chunming_> phila: if i collect on the result, we just see images?
<Zakim> Jeremy, you wanted to ask if processing is server-side
<chunming_> Jeremy: is all of the processing on server side? (on VMs)
<chunming_> Xuezhi: yes
<chunming_> Jeremy: can you do some of simple task (analysis task) on browser? so it will reduce latency
<chunming_> ... we are thinking on how to do the process in browser
<chunming_> ... all image data, but if you want to combine vector data, the image data don't work with that
<phila> GB: So what's the relevance of this to our project. Might we help create something more useful to others?
<phila> Jeremy: I think we're trying to go furtehr than where this project is, but it serves as validation that the problem exists.
<phila> billroberts: This looks like a good subset of the probelsm we're looking at
<phila> ... We can think about what the additional problems might be.
<phila> GB: So what? What does this inspire us to think about, change, etc?
<phila> phila: Talked a bit about sharing data.
<phila> GB: Several years ago, USGS made the data available. Now they have a lot of international relations and partnerships.
<phila> ... It must be a powerful economic driver.
<phila> billroberts: Web Map Services etc are useful, but we want to be able to access subsets of the data
<phila> ... So we need IDs for subsets, so you can get it, annotate it etc.
<phila> GB: The fundamental argument is that making the data available has enormous potential.
<phila> Jeremy: At the moment you have a portal that has lots of contextual info in the portal that allows you to browse and discover. For the data to be easier to find...
<phila> ... you need the context
<phila> ... How do you make the data discoverable> What standardiused metadata are you able to publish that allows otehrs to see what you have available.
<phila> ... Those are the initial steps that need to be taken.
<phila> ... That's what adds value
<phila> ... When people reuse it
<phila> ... The 2nd and 3rd reuse benefits everyone. So... do you publish standard metadata that allows other systemns to know what data you have? If no, then we can help.
<phila> GQ: We are users of their cloud. We can get their metadata and use their data in my project.
<phila> Jeremy: What form is that metadata published in?
<phila> GQ: We have a standard that we extended from another
<phila> ... Base standrad is CSW
<phila> Tianyu_Wo: Looking at Chaufferred car services
<phila> ... We have a student from Sino-French group who can pronounce Chauffeur
<phila> ... taxi services being killed by this
<phila> Tianyu_Wo: U-Car used to be a car rental company. There's the company owned fleet who hire drivers
<phila> ... Other kind of service is private owner services like Uber
<phila> ... provides a platform to give their services.
<phila> ... Both need a strong ICT platform to manage the cars and drivers
<phila> ... Both customers and service provider need safe, steady services
<phila> ... Always want to know where my cars are etc.
<phila> ... Building a platform to collect the sensor data
<phila> ... I have written a use case. My UCs may be a little far from todfay's topic but the requirements for coverage data come later
<phila> ... Also processing data in real time
<phila> ... providing flexible product for the company mostly.
<phila> ... Want metrics about customer waiting time etc. Want more efficiency. better efficiency is important (petrol etc.)
<phila> ... big fleet, growing fast
<phila> ... Services is > 30 cities, > 10M users
<phila> ... data platform needs to work 24/7, receives 120M messages a day
<phila> ... Business requirements change over time
<phila> ... Want to develop IoV system
<phila> ... tried integarating data from diff sources
<phila> ... Want to get some facts from the cars, like abrupt turns, acceleration etc.
<phila> ... Can feedback to the driver
<phila> ... Can judge the driver of course
<phila> ... Collecting GPS data for speed, direction etc. Also detailed data from the car itself
<phila> ... On board diagnostics etc.
<phila> ... Around 25K cars in the fleet, trying to expand to 60K
<phila> ... data platform
<phila> ... 99% of requests are for current data
<phila> ... We put the data into a SQL like system, put the whol lot into an H-Base big data storage. Lots of data redundancy, did some work to reduce redundancy
<phila> Tianyu_Wo: Shows system diagram
<phila> ... Most work is on the back end, streaming and batch processing
<phila> ... We have spatio-temporal data storage system
<phila> .. For real time streaming processing, we are focusing on data quality analysis
<phila> ... Data collected from different cars. May get into a tunnel and not transmit, then send later
<phila> ... Portal tries to give easier interface to the company to understand what's heppening.
<phila> ... Single route of a car, we can see the petrol consumption. Is it in the same trend as others? If not, maybe there's something wrong.
<phila> ... Doing sometehing about prediction of where orders are coming from, combuned with other datasets.
<phila> ... For e.g. the weather may be a factor, but we're not using that now. We can maybe send drivers to an area where we expect lots of orders.
<phila> ... Looking for wasted mileage etc.
<phila> ... Want to open the data in future.
<phila> ... Only use of satellite data is GPS, but this kind of data needs to be opened, and service needs to evolve.
<phila> ... Want to tackle data privacy
<phila> ... Can't do individual data
<phila> ... Not a good idea to process all the data in the browser
<phila> ... In our portal, we show the vehicle distribution, but if we load all the car location data into the browser, it will crash. Need to pre-process
<phila> ... Challenge is that the business is changing
<phila> ... Need some non-functional but important business opportunities.
<phila> GQ: You said there are 120M messages per day. What's in those messages?
<phila> Tianyu_Wo: The status of the GPS, OBD, etc.
<phila> ... cars report every 10 seconds
<phila> Tianyu_Wo: One car sends about 10K packages per day
<phila> phila: Asked a bit about putting sensor data into coverages, predictions, air quality etc.
<phila> Jeremy_1: We're focussed on satellite rather than in situ sensors
<phila> phila: Yes, but they're not mutually exclusive
<phila> chunming: I wanted to talk about moving data processing to the browser
<phila> ... Of course you should only download the data you need, on demand, as the application needs
<phila> ... We have a big chunk of pre-processing but then you get the relevant data
<phila> ... Looks like we might need some client side tools, encouraging people to do more client side processing
<phila> Tianyu_Wo: Coverage is not only from satellites - air borne, in situ etc.
<phila> ... SO I think your work is related to our toipic today.
<phila> yingying: I think Tianyu_Wo is really related to our work in UbiWeb and Automotive
<phila> chunming: I think in WoT there is an overlap with coverage data. data formats, data sharing
<phila> yingying: Location based services and telematics next.
<phila> MaikRiechert: There are some things you must do server side, but post processing of compact data can be done.
<phila> ... Takes a land cover map
<phila> ... Ina GIS cloud you can do what a server supports.
<phila> ... We want to move some to the browser, where it makes sense
<phila> ... Suppose I want to select a region
<phila> ... I find a raw JSON file that's compact, I load the data
<phila> ... Paste in the URL
<phila> ... It loads, and I can inspect it and see what it says.
<phila> ... Suppose the data didin't come from a GIS peson
<phila> ... So I wanted to have different formats, GeoJSON and our own CoverageJSON format
<phila> ... I created a small subset just be combining these two files.
<phila> ... It looks like an image but it';s not it's data
<phila> ... All that data is in the browser
<phila> MaikRiechert: Core functionality that should be supported is what the browser supoprts, but we need to be able to select bounding boxes, different resolutions
<phila> ... If your data is too big for common web clients to handle, then you need to pre-process.
<phila> ... If data goes over a few MB then need to pre-process
<phila> billroberts: So what are we seeing here?
<phila> MaikRiechert: That's about 500KB, 500 x 500 Pixels
<phila> ... 300MB compressed JSON gets you the whole UK
<phila> Jeremy_1: In the old days, 40 MB in the browser was too much, but not now.
<phila> MaikRiechert: But you need to be careful. a few MB of JSOn can expand to hundreds of MB
<phila> ... We also experiemented with ...
<phila> ...You can assign different colours to different types
<phila> ... Maybe the server side doesn't support that. In the browser we can do this - mapping different colours to different categories.
<phila> ... There are different kinds of land cover schemes. we mapped one to another. All I loaded was the defintions, in the browser I can make the mapping.
<phila> ... Which means researchers can play with it easiy, with drag and drop etc.
<phila> ... Lots of post process data.
<phila> ... Once you've made these kinds of derived datasets available, then you can let others easily experiment.
<phila> ... And inspire them to take the data and do what they want.
<phila> ... Really just show others how they can create interesting new ideas.
<phila> ... When you look at ways of doing this, you see the OGC standards, WxS etc. But if you have custom solutions, it's hard to implement by yourself
<phila> ... This is all simple REST-based JSON APIs etc.
<phila> ... Don't look too closely at the specific inplementations, but the concepts.
<phila> MaikRiechert: So how do we publish some data and not just offer images for download.
<phila> MaikRiechert: Dept of Environment, Food and Rural Affairs like the idea of getting rid of all the water as they're not interested in stats for that.
<phila> ... Which I did
<phila> ... In effect creating a new dataset - in the browser.
<phila> ... Affects things like the percentages of land cover types of course, which is what was wanted on that occasion.
<phila> billroberts: There are lots of well extablished bits of software in the GIS world, nothing wring with any of that. What I like here is that it uses a different set of software that is more familiar to a larger group of developers than GIS specialists.
<phila> JT: And because it's being done in the browser, you can use otehr Web tech, like service workers, to take it offline etc.
<phila> ... It's a browser application that's also a GIS application.
<phila> ... It's the use of regaular Web techniques that is the power here. Not NetCDF and otehr specialist formats.
<phila> ... This is about accessibility because people can use it., nothing to do with licences or metadata.
<phila> GQ: So this is working at the browser level. Our approach is to take the coverage... to try to develop sometehing to move to the browser. I think this demo is good.
<phila> ... I'm thinking about the performance. If the data size and process model is not too much then it can work there.
<phila> ... If the data size is too big, can this architecture hand;e it
<phila> GQ: You have a map with the different layers, maybe this can reduce the number of layers
<phila> GQ: I think there must be someething that can be done to alter the percentage configuration... to, for example, generate the forest layer.
<phila> MaikRiechert: This dataset is produced by a scientist, using a model, it takes time.
<phila> ... These things are not a good fit for doing in thr browser, at least for now.
<phila> ... When you think about vector data. For example, the Netherlands has an open dataset about all their trees (open)
<phila> ... And you can easily fcombine it with other data. it's simple and compact.
<phila> ... On the otehr hand, if you want to focus on, say, just London, you'd ask for just that.
<phila> ... There's no clear answer tyet for where it's suitable.
<phila> ... May not good for all the channels from a satellite image. Maybe you ask for some of the wavelengths, not all.
<phila> ... So for me its about what is the funtion we need the server to handle. By bounding box, by temporal etc.
<phila> MaikRiechert: Briefly shows CoverageJSON
<phila> billroberts: An API for retrieving data in a particular format sounds good, but how complex is the server side code for that?
<phila> MaikRiechert: That depends on what your existing services are like. If you have an OGC Web Coverage Service, it's simpler to implemente it over the existing one
<phila> ... It's all experimental for now.
<phila> acj j
<Zakim> Jeremy_, you wanted to ask about result pagination
<phila> Jeremy_2: When we talk about providing a RESTful API for gathering data from the server, have you talked about how to paginate the data into tiles?
<phila> ... There are lots of options
<phila> MaikRiechert: Yes, there are lots of options
<phila> ... Pagination is something we have implemented. Gives example
<phila> ... Salinity and pressure
<phila> ... How would you filter such a big collection by bounding box or whatever/
<phila> ... You shouldn't get the full collection in the browser. So instead we just serve the first page
<phila> MaikRiechert: One page was 10 points on the map. And the idea that the client is clever enough to know that it's just a page
<phila> Jeremy_2: And that uses Link headers?
<phila> MaikRiechert: yes, that's the simplest.
<phila> Jeremy_2: Let's imagine the Uk set was too big for the browser... could you turn it into a collection of data tiles that was served as a collection of coverages?
<phila> ... So you could ask for a cube of data over London
<phila> Jeremy_2: The difference with subsetting is that you might be able to identify fixed tiles upfront so you start by serving 10MB at a time and the client can 'say when'
<phila> MaikRiechert: Since we have a server that can do subsetting, we haven't needed to do that.
<phila> billroberts: x and y might not be the only dimensions - time etc.
<phila> MaikRiechert: Each time slice is about 3 MB
<phila> ... It only loads when I click on something
<phila> ... When I do statistics over a time series, I can show a progress bar, so it collects one at a time
<phila> ... I can get rid of one as I load the next
<phila> phila: Emphasises UK-China collaboration aspect
<phila> ... We want to bring the projects that we've heard about together
<phila> ... over the Denise for how that might work.
<phila> DM: We've had several examples, but we have that commonality about how we find and use that coverage data
<phila> ... What Jeremy and I will talk about is how we see that UK-China collaboration develop
<phila> ... to see if we can describe gthe common experience, and how we might do some work together that leads to the spec that we produce.
<phila> s/guoching/Guoqing/ig
<phila> DM: So thinking about a common story - sommon tools, looking at the MELODIES work Maik taledd about, but also looking at those data themes that are commn. Water, agriculture, disadter management
<phila> ... fire management.
<phila> ... We'd like to get agreement on the themes. We need Chinese input.
<phila> DM: So a quick question...
<phila> ... How many are looking at the IRC chat?
<phila> ... Because in order for this collaboration to work, we need to makea sure that we can all access the same tools.
<phila> ... So that's things like GitHub, the wiki
<phila> ... This afternoon can only be successful if everyoine contributes to the conversation.
<phila> ... Important for us to understand the barriers that may exist between the two nations.
<phila> DM: So rather than asking about those barriers in detail, I'll ask Jeremy to talk about the ideas we have had about how to move forward. What I ask is that you think about the ideas and please make other suggestions if you can.
<TianyuWo> 如何识别共性需求,水资源管理?土地管理?
<phila> Jeremy_3: What common themes can we find between UK and China. Land usage, Water? Agriculture
<phila> Jeremy_3: This morning when I heard about the CNIC & RADI projects - those projects were allowing other people to derive value from data. Lots of data for lots of purposes.
<phila> jianhui: We are a data centre. We provide the service to scientists, we focus on the data infrastructure.
<TianyuWo> CNIC的project和Reading的项目都是允许其他用户提供增值服务。没有看到具体的应用领域,需要利用大数据输入和应用。
<phila> ... You mention the different themes. I remember 2 months ago there was a UK/China joint call for proposals focussing on using the remote sensing data to solve agricultural problems.
<phila> jianhui: Maybe our project can help with that
<phila> geoffrey: We can look at what cateches the most attention? I guess it's agriculture, water etc.
<TianyuWo> 中国有什么具体的应用,更关心那些领域?是否这些公共数据库(平台)方案是有意义的。
<phila> Guoqing: I think maybe there is a problem with the people round the table now.
<phila> ... jianhui and I are from the data centre, we're not application scientists
<phila> ... I know the agriculture area is very important and there are big research terms working on this.
<phila> ... A model developed here is one of hte most used around the world.
<phila> ... important contribution to GEO. But they're not here.
<phila> ... Only he can get a project in this call, not the team here.
<TianyuWo> ...农业是具体的应用,有专家利用遥感数据,形成重要的数据模型,指导农业
<phila> ... Maybe tomorrow night we'll have a dinner with Geoffrey and others and maybe we can address the question there.
<phila> ... Agriculture is a very important topic, as is air pollution.
<TianyuWo> ...可以和华东大学专家联系。
<phila> ... The most important thing for air pollution is the model, not the data processing.
<phila> ... We need cooperation on the model. I think we can focus on agriculture.
<phila> ... If we can move the interest from the end application to a basic level...
<TianyuWo> ...污染方面的数据共享,科学数据模型更被关注,农业应用更加关键
<phila> ... They won't really understand why we're talking about coverage here.
<phila> ... We have limited time etc.
<phila> geoffrey: Does Ping Fan use the kind of tech that Maik was showing earlier? If so is that a future route to coppoeration.
<phila> guoqing: We should talk to him and his team.
<phila> billroberts: Can I ask a similar question... there aren't many users of the data around the table. Is there enough capacity in the UK to bring in representaives of the community that we'd like?
<phila> Geoffrey: I'#d love to have Maik working on my project on ice caps
<phila> ...The problem that we all suffer from... people deliver images to me but I'm not an image deliverer. getting to a state where I can manipulate the data like that would be good.
<phila> scribe: phila
Jeremy_3: What I take from the
conversation... we're not end users ourselves, generally, nor
are we experts in the domain of application.
... None of us are agronomists
<TianyuWo> ...我们不是最终用户,我们想象的需求可能和真正的领域专家不同。
Jeremy_3: We also know that if we
spend all our time trying to develop a new scientific model, or
new application, we'll quickly run out of time and
resources.
... So rather than demonstrate new science, can we do something
with the combination of coverage data and discrete data that
would show what we can do with a follow on project?
chunming: We have seen 2 infrastructure projects - can we choose one and find collaborators to work with us and do some development. Maybe connected car can be a candidate
<TianyuWo> ...我们不应关注到具体领域内,而应该关注coverage数据集的组合,关注后续可能的应用
Jeremy_3: So we look at the
existing projects and the necessary collaborators today,
because we need to fast track this project, but we're looking
for the follow on project.
... It's an application of the tech, not just the tech
itself.
... Which is a better way to get the attention of the
politicians and funders.
<TianyuWo> ...需要一个快速上路的项目目标和应用技术需求,快速吸引资助方关注。
Guiqing: I'd like to suggest that
we can simplify this work, and can select a common product
level.
... Maybe a @@@ index, a vegetable index. These are commoin
indicators for agriculture. And we can use JSON for that.
<TianyuWo> ...选择简单易理解的应用,民众可以直接理解,下载应用。例如是否可以公开风云卫星的部分数据,让民众思考如何处理和应用
Guiqing: Jianhui can provide
MODIS data, we can put the processing of this data, maybe in
the browser level.
... It will bring some challenges
... China is many times bigger than the UK so the amount of
data is much bigger. For MODIS, 3 passes covers China. More
that 600 MB per pass
... twice per day so that's a lot of data per day.
... So it's c ahhelnge for this architecture.
<TianyuWo> ...挑战:数据主要关注中国、亚洲。数据量很大。风云数据量大,600M每次穿越,3次穿越覆盖中国
Guiqing: @@@ data is not well
organised. It's not as well organised as MODIS
... There are problems with the data format and service
... MODIS is very well organised
... So it's well used.
... I want to find some new challenges
... We can get feedback from the user community.
... Trying to integrate everything together - we don't need too
much time to do this.
<TianyuWo> ...可以关注应用社区的反馈。不需要开始就针对应用投入大量资源。
MaikRiechert: How would it work together... would it be some internal development. How would we actually work together?
Guoqing: WE can discuss this.
Jeremy: Any development done in this project needs to be openly available. I think we'd be prepared to commit to publishing the software and using data that is openly available so that we can demonstrate what we've done.
<TianyuWo> ...确保我们开放的数据都能很好的被访问。
Jianhui: I wonder whether in this
project we can produce common products, maybe with DBI, but the
mature goal for this project - should we try to demonstrate a
new style for our service.
... The key point is producing ... a new style data
services.
<TianyuWo> ...项目的最终目标,是否应该产生新形式的数据服务
jeremy: It's not too difficult
for produce a new product for NVDI or ?? We can then
investigate the deliveray of new services following the kind of
patrtern Maik demonstrated earlier.
... So we'd first create a new common product., Then we'd try
and provide some new services that the users currently don't
get, something they don't have already.
Jianhui: If we just want to try some new services, they're there already?
<TianyuWo> ...新的数据服务可以尝试利用上午Maik的数据开放平台组织应用
Guoqing: I don't think so. This
is a new workflow for me
... To reorganise this common product.
... Accroding to this geotemporal area
... WE can try to get the limited data from the data resource.
Precise data set
... I don't think distrubution is a real problem, it's the
reorganising of hte workflow.
<TianyuWo> ...新的数据处理工作流。重构数据处理流程
<chunming> +1 to guoqing
GB: Thinking about the longer
term. There's an opportunity for collaboration in the long
term. Including the kind of Maik approach in your (GQ's)
work.
... One we coujld take an application domain, pref with a wide
set of phenomena associeated with it. Partly as a demo and
partly as an app to get funding.
... We have W3C and OGC who are promoting a generic approach.
That's less like to be productive.
<TianyuWo> ...2个选择:1. 一个具体的展示领域。e.g.减灾工作
GB: There are 2 programmes that
would love to see this. IRDR Disaster reduction prog - pushed
into gathering data and integrating it, including in places
here there is no nigh order tech,
... The Future Earth Programme is the other. It has offices in
all major continents. Thgere's a geospatial base but it's
pretty limited.
... I can think of how we might approach both those for
something serious and ambitious.
<TianyuWo> ...2. 未来地球项目
GB: But to get to that stage, we
need ther kind of collaboration we're talking about.
... There's potential for some serious funding in the thematic
domains.
<TianyuWo> ...有很多的机会
dmckenzie: In terms of the
purpose of the project, the SDWWG has its Best practice doc
that is generic. Focussing on one domain at th eexpense of the
generic might be problematic.
... We need the involvement of a domain, but how how do you
make this easy to use cross domain.
<TianyuWo> ...需要领域参与,但不应该太关注单独的领域。更多关注跨领域的需求
dmckenzie: Whilst yes, you can
get funding, we need to be careful how we play it.
... Our SDOs is quite domain related. OGC has a domain WG for
oceans, agriculture etc. but we standardise across domains.
GB: The Belmont Forum is about working acorss domains.
<TianyuWo> ...OGC有农业等多个领域工作,更加有助于理解跨领域的需求
billroberts: I agree that we need
to create something that is generic.
... In liaison with end user groups - I'm guilty of creating
something that they think is easy for end users may not
be.
... The other value of the end user approach is you can often
point to real savings.
dmckenzie: There will be people
here who don't participate on OGC, but we're structured to try
and help with that. We don't just have groups that look only at
WcS etc.
... We have domain WGs to keep the connection with the
users.
... In picking a domian or thematic area, there's certainly
scope for using the domain WGs.
<TianyuWo> ...和最重用户要有紧密的联系
dmckenzie: There's a TC within a
week in Washington. The Met Office has staff going to
that.
... Thyere's no reason why ideas generated today can't be taken
to that.
<Zakim> chunming, you wanted to comment on generic vs. domain specific
chunming: For standardisation, we
need common things to be defined. In the London meeting we
talked about the semantics of the datasets. And APIs between
the client and the slices of the data
... We need a framework for handling the different
dimensions.
... For some specific dimensions, e.g. waveband, we can give an
extension mechanism to dfine their own dimensions.
<MaikRiechert> +q
<TianyuWo> ...共性需求需要被定义。数据的共性维度,具体应用领域可以定义自己的维度,可扩展
chunming: If we are successful,
then diff developers may use that to define domain-specific
tools.
... I'm thinking that for coverages on the web we need to think
long term and for now we just show what might happen,
<guoqing> 时间-空间-频谱-主题
<TianyuWo> ...核心的Web相关标准,应该关注核心共性数据,可以快速展现给用户
JT: For this project, we want to develop those generic ideas. We'll use NVDI and Leaf Index because they're interesting for the Chinese groups.
<guoqing> coverage就是需要给一个从全要素空间中快速取得多时空子集的技术框架
DM: I know that the UK govt. is working on something very similar and we can ask the relevant people.
JT: In terms of the following project, we could pick on two domains and deliver against those.
MaikRiechert: Can I suggest that
we look at an inter comparison of 2 datasets.
... Then you can find out why you can't inter compare. Units is
an issue, for example (Celcius/Kelvin)
... Measurements from bouys often have different units.
... We need to work through that.
... How do you say that 2 things that you measure are the same
concept. You can have 2 different temperatiures but at
different hieghts etc., they mean different things.
GB: One thing we should be clear
about is what we mean by domain.
... For example, we can think about lots of things like oceans,
air etc. But questions are asked differently.
... Disaster management will want real time info. Future Earth
works in decades.
... A domain is not just a physical phenomenon.
DM: I don't know that we think
about it in that way. Those are different users, but the data
underlying it will be the same.
... You're asking a more user-centric question. Froma domain
space it's the same.
... Met oceans etc. You can still think people might want
different questions. Bush fires happening now, bush fire trends
- same data.
GB: It's a spectral view
billroberts: When does this project have an end?
phila: May
... And no one has time allocated to it.
JT: So a couple of mini projects
around NVDI and Leaf Index...
... If we can show that the Chinese people who have projects on
that now, can take Maik's work and try it out for their work,
that is interestung.
... Benefit for China is potential new services/methods
... benefit for UK is validation/highlighting problems with
CoverageJSON approach
<TianyuWo> ...向国人展示一种构造应用的新方法
GB: The key thing is motivation. Is that sufficiently interesting for our Chinese colleagues and Uk colleagues?
Jianhui: I think it's possible
that we can try and implement the CoverageJSON idea and maybe
make a demo.
... but maybe there are some differences.
... And we can give feedback to you.
... I think it's possible.
XW: I thinkw e can try that. The big problem is how to use the raster data
<TianyuWo> ...可以尝试CNIC的数据json化,共享出来。需要明确技术挑战是什么?
XW: How to retrieve and organise the grid data is the challenge
<TianyuWo> ...如何获取grid数据?
JT: I think Guoqing was saying
that developing a new workflow was of interest to you.
... If your (GQ's) team coiuld develop the new workflow and
Jianhui's team can explose it.
... You said that the UK is very small and compact, whereas you
need 3x600MB passes to cover China.
... And you have data that is quite poorly organised, helping
people work through that would be helpful.
... But by providing new services you can get feedback from new
users.
<MaikRiechert> +q
<TianyuWo> ...如何让用户参与?
JT: One essential feedback loop
is to engage users.
... And maybe this would help?
<TianyuWo> q?
<chunming> to divide the workload (computing) between backends (offline)processing, and in browser (realtime, asy, on-line) processing, is a interest thing in the workload.
MaikRiechert: Can you define
users?
... Researchers? Others
GQ: Researchers, yes.
MaikRiechert: Combinging data from raw sources is hard. Researchers won't combine datasets etc.
JT: I think you're right. If we want to engage technical users, cf. research users, we'd need to do more outreach.
GB: One quick one about users... what I mean by users are people who have a scientific or preactical problem they want to solve and they don't need to employ someone else to help them
billroberts: +1
== Lunch==
<scribe> scribe: phila
denise: let's summarise next
steps and check it is feasible and practical
... what should be the outcome in 3 months, then work back to
identify what we have to do
... data flowing into a service
jtandy: GQ identified 3 things. 1 Workflow for NVDI leaf index data, 2) much of that dataset is not well organised so from a data centre perspective, how can that be structured to support better discovery and usage
3) develop services for the end users (researchers) allowing them to do analysis without help from an intermediary
scribe: this will need develop of simple web applications to show what is possible
jtandy: summary 1) workflow 2) data organisation 3) implement simple applications
phila: that soudns like a lot to do by May
jtandy: some could be done in parallel
phila: need to check later with
GQ (when he returns) that his team can do what we expect in the
avaalable time
... idea is that RADI would deliver data via the CoverageJSON
approach to Jianhui's group at CAS so that they can use it
jianhui: this could lead to providing feedback to Maik on how well CoverageJSON works in this context
maik: yes, that is the intention. The work is not complete and feedback is welcome. Things can be changed if necessary
jianhui: can we use another style to publish services?
xuezhi: 'BISON' (not sure of spelling)
maik: interesting to compare JSON
and a binary approach
... it's just a different container
jtandy: not much difference in practice between binary form and compressed JSON. Is that right?
maik: for a lot of data, parsing
JSON takes longer than processing binary. Compactness is
similar but processing is slower with JSON
... motivation was to use common web formats that can be easily
used inside a browser
chunming: compare with web
mapping service - browser applicatoin with a lot of data in the
backend. It's about easy to use protocols to make the data easy
to use in web applications
... text based web friendly formats are interesting because of
ease of use, even if there is a performance penalty
... pagination work in the data activity is also relevant
phila: that's in the Linked Data Platform work
maik: if you have GB of data,
then it will be too big whatever approach you use
... let's pick an example that works with a manageable amount
of data, so maybe use lower resolution data
... so do some work on the browser. If it's necessary to get
more detail, then that might involve downloading data and using
desktop or server side tools
phila: talking about easy ways to use web map services - is that just leaflet.js
maik: ...sort of
... good for use of tiled data
jtandy: also openlayers
maik: WMS can have several layers - more flexibility but heavier to use. Not built into leaflet, though there is a plugin. Maybe not that relevant
jtandy: today applications are
stuck with dumb images, even if tiled. Want to work with
meaningful data, so that for example you can calculate
interesection with a polygon of interest, like the boundary of
Beijing say
... we don't want to be stuck with server side processing
maik: shows an example of what cannot be done with just an image
<chunming> https://www.windyty.com/
maik: download sets of coverage
data, then create an animation in the browser
... because it is the browser, the user experience can be much
more interactive
jtandy: as you zoom out, the user
is shown lower resolution data. Coverage object consists of the
decsription of the domain, the grid and the data
... server can decide at what resolution to serve coverage data
according to the type of call - to assist client side
applications to work effectively
phila: this makes me think of an accept header where you could specify the maximum amount of data you want to get.
jtandy: [waffling]
chunming: so some kind of content negotiation?
phila: let's bring it back to
deciding who is going to do what
... what is it reasonable to ask CNIC to do? what support is
needed from the rest of the group
... what can CNIC commit to? Only two months so it needs to be
quite small
maik: what kind of server side data formats are used at CNIC?
xuezhi: GeoTIFF
maik: could it be converted to NetCDF? (because we already have software for exposing arbitrary NetCDF in CoverageJSON, so that would allow re-use of existing tools
xuezhi: no problem to convert GeoTIFF to JSON
maik: but if convert to NetCDF can use my library
jtandy: some advantages for the
project if CNIC do not use maik's library, but instead
re-implement the specification themselves
... a better test of whether the specification is implementable
if CNIC reimplement, rather than using Maik's software
directly
... separate step is developing a method to deliver subsets of
it
maik/phila: sounds like too much, that could be in follow-on project
phila: just the first bit about
delivering geotiff in CoverageJSON would be a big step
forward
... can we write down what CNIC is going to do, to check
everyone has same understanding
maik: would it be possible for me to access CNIC's implementation so I can see it?
jianhui: server not usually public, but we could give you access
denise: [writes summary of the plan on screen]
Who is doing what:
CNIC:
- Library that converts GeoTIFF to CoverageJSON
RADI:
- Something that is moderately too large for browser
- workflow for NVDI & Leaf Index (National Vegetation Diversity Index?)
[Guoginq (GQ) re-joins the meeting]
jtandy: CNIC should create a GeoTIFF that is moderately too large for the browser, so we know that the demo can't 'cheat' by just loading everything
RADI: to create 2 new content products as GeoTIFF (NVDI and leaf area index)
University of Reading (maik) will provide support and can help with new features if required
jianhui: xuezhi can liaise directly with Maik
jtandy: this will lead to collaborative improvement of the restful API.
maik: we really want to know about anything that needs improved
jtandy: other side of the restful API collaboration is CNIC (and RADI)
Guoqing: also interested in the API work
maik: so will involve RADI in discussion as well as CNIC
jtandy: so now we have a server but no client...Maik, you have some client applications already. Are you able to do more client app work?
maik: I need to understand better what the data is
jtandy/denise: just indices, scalar values
Guoqing: it's imagery. Each pixel has a simple value
Guiqing: it's a continuous scale, not discrete values
Guoqing: NVDI is a different concept, also continuous
Maik: yes, I can display it. But
would need a use case for what it's for.
... we can compare imagery to observations. Are there
observations to compare with?
jtandy: what time period is there data available for?
Guoqing: different options available including seasonal
jtandy: for example, are there areas suffering drought that affects agricultural output, so we could look for anomalies of 'greenness' in comparison to seasonal norms?
maik: is it enough to compare the annual data with monthly data?
Guoqing: no, not really suitable
for that
... data comes as HDF5. If converted to GeoTIFF then it can't
be continuous - only integer values possible in geotiff
... so let's change the plan to use HDF instead of
GeoTIFF
... could probably do both HDF5 and GeoTIFF
maik: already working on comparing data between grids
jtandy: so we can add to the plan that UoReading can also offer a client application to support grid to grid comparison
phila: that looks like a good plan for CEO-LD
denise: how will the end of CEO-LD be communicated? Opportunity to present at Dublin OGC TC meeting in June.
phila: is there also a chance to present to a Chinese audience
<phila_> denise: there are other OGC opportunities, for example meeting in Taiwan in December
<phila_> phila: most important deliverable for the UK Embassy will be a report (that can mention the demonstration sessions)
<phila_> denise: Dublin meeting can involve web streaming and recordings to make it accessible globally
<phila_> phila: can we confirm this is doable within 2 months? If this is too much, then say
<phila_> denise: should we take a week to think about it and confirm?
<phila_> phila: yes, we should have a 'cooling off' period!
<phila_> phila: Maik, will your examples work with CHinese data?
<phila_> maik: what is the coordinate system used?
<phila_> Guoqing: coordinate system is standard, no problem
<phila_> agreed: data will be provided as WGS84 lat/long
<phila_> phila: also tasks to be added for W3C and OGC. W3C to be responsible for report (phila and chunming)
<scribe> scribe: phila
The plan of action will be added to the CEO-LD project wiki
jtandy: the Restful API will be
hosted on the gscloud and will be accessible in UK (and
Dublin!)
... the web client can be hosted somewhere else
maik: the stuff i demonstrated earlier is public and can be used for testing
jianhui: CNIC will provide infrastructure for the demo. They have ample hosting resources they can access
jtandy: this is a good example of the collaboration: in the UK it is easy to experiment and bring concepts together. In China there are many opportunities to deploy large scale infrastructure. The collaboration allows things to be achieved which could not easily be achieved by either group on its own
RSSAgent, pointer
[pause for break]
<chunming> for those who want to print the boarding pass, please send e-mail with PDF borading pass to me: hucm@w3.org
<scribe> scribe: chunming
<phila> Date for the demo - will be Wednesday 22nd June at 08:00 Ireland time.
<phila> Timing to be as convenient as possible for Chinese colleagues
summer time (GMT+1)
<phila> Therefore 15:00 Beijing Time
<phila> We should make sure that official invitations go to the relevant people so they can join in if they so wish.
phila: we have two items to
discuss
... subsetting, sharing ideas, find subset of coverages, what
kind of things we need.
... negatiation, big workload
... and what's next, where's the next project
GB: think about application, the
potential to delivery to scientists, more functional, that's
the goal that i am concerning.
... what data is make available.
... climate changes, long term, world wide.
... polutions, impact locally, and globally
... how to provide fancy maps, eg. satallite maps
... if we could use sharing data, showing the change from time
to time, it will be valuable.
... the demo we hope, globally capability to do clever
things
... in relation to global change issues, to build big
application, and geo-relavent.
... a really good demonstration, to see what is possible., June
22. then 6-8 month , come to a serious proposal.
... it will take 5 years or longer that really products will
using the coverage data standards. the question is moving into
right direction.
DM: global earth observation(?) , will on board.
GB: in paris conference, world health organization, ... lots of organization want to join.
DM: how to dive into standards things, the demon of coverages, may help the government to answer the application questions on the data
GB: key things is , small but ambitious, to support future development.
DM: quite diverse range of
domains.
... with research orgs, government.
... pick common languages
Bill: @@@
DM: current push from world bank, data issues. meeting on washington on Monday.
phila: focusing on satellite, where is this project comes.
Jeremy: easy to use as the usb
stick. really easy to use.
... we are doing is to make it easy to use.
... tracking what have been done to the data
... publishing of the data should be locally, from local to
region level, to global level
... it also could be decentralized
... people have to subscribe to themself.
... big challenges here.
phila: Who else do we need in the new project? Who from China? Who from elsewhere?
<phila> DM: GEO, can't do it without them. Maybe try Stuart Minchin.
DM: 2nd round stage, merge into the data cubes. a long history, build query on the pixal.
Bill: data cube structure.
... on the slicing of data
phila: commercial person could be involved, to make money
Bill: tools, not only scientific
research, but also policy. to get more information to do
that.
... offer commercial services
GB: december, international data
science, policy for science, the big thing is "open data"
... next year, pushing government, research concils, to an open
databases.
... something is moving to the same direction.
Bill: space @@@
DM: involved in London meeting, simon?
Phila: jianhui, guoqing, what's you think most useful things to do next?
jianhui: NSF of China, and UK Newton foundation, may sometimes support the collaboration projects in china and UK
guoqing: funding is always the
problem
... china is changging the funding policy, NSF of China is the
channel, any topic could be submit as the proposal. We could
have a proposal on coverage, but not guarantee on if get or
not
... there are lots of call
<scribe> ... new policy conduct a very big one project from MOST (ministry of science and technology)
UNKNOWN_SPEAKER: also small
funding for the institute
... try to integrate of our idea, apply for the institute
funding internally.
Phila: What's you want to achieve.
guoqing: jianhui and chunming , we talked about the topic on coverage data store from NSF
Bill: the possible international funding.
GB: important issues - water, environmental, ... what need to be done.
phila: HTTP range functions,
possible already support what we want
... coverage 5 megabytes
Maik: it is useful on the data download?
phila: about chunking data
jeromy: we have whole coverage,
and need to support download.
... we need to download a piece of data in it.
... to use the URL to fetch the slice data
phila: even the slice is too big, i think HTTP range would be used.
Maik: not just APIs but @@@, to use range functions. not sure if it is the right use.
Jeremy: URI to the coverage, query primitives, saying for x from 0-10, y from 9-6, t from 7-14, the api is currently simple, but a good start
<dmckenzie> Information on Geoscience Australia Datacube from earlier discussion http://www.datacube.org.au/
[Maik to show coverage JSON restful api core specificatoin]
Maik: time, according to RDF
3339
... opensearchtime
... index based dimension , or subset index.
... collection of the coverages.
phila: allow the dataset provider to define the slices?
[maik show API description]
client use the api to describe the range on each dimension. like x[start:stop:step] for grid coverage
phila: server side code handling
a very big set of coverage
... the template define the pattern of query
Jeremy: if you want to reliably get the same data (using a dedicate URI)...
phila: that is the identifying
(of the data).
... coverage link data?
bill: if we want a polygon of
beijing
... how do we define it in the URL
phila: subsetting is still an
open issue.
... computation complexity
Maik: boundary handling. two time dimensions.
jeremy: rather than have one service that doies index based and polygon - have 2 separate services, one that generates the bbox coords that can become part of the index-based query
<scribe> scribe: phila
Bill: A job I'm not volunteering for is to manage the process of chasing the work on the to do list.
Phil to communicate with Chinese colleagues on progress throughout coming two months
Denise to work on logistics to support that, esp connectivity
JT: Standarising the RESTful API
is future work.
... We are testing the coverageJSON aspects
... We are also pushing the coverageJSON encoding as a formal
encoding as a Coverage Service encoding within OGC
... In London, GB had the idea of a citizen science game.
That's something we could potentially think about.
GB: Citizen Science is seen as a route in...
RADI
- 2 new common products - NVDI & Leaf Area Index (HDF5 & GeoTIFF)
- Collaborative development of the RESTful API
- Provided with Lat/Lon WGS84
CNIC
- Library that converts HDF5 & GeoTIFF to CoverageJSON + REST API
- Implement RESTful API to expose the CoverageJSON content
- Handle/process CoverageJSON that is moderately too large for browser
- Collaborative development of the RESTful API
- Lat/Lon WGS84
- Host on the Geospatial Data Cloud
University of Reading
- Provide spec for CoverageJSON
- Implementation support to CNIC
<chunming> https://github.com/neothemachine/floodit
- Integrate new features to the spec
- Collaborative development of the RESTful API
- Client application to support grid to grid comparison
Beihang
- Investigate an independent application utilising services developed by CNIC & RADI
OGC
- Demonstration in Dublin TC 0800GMT time 22nd June 2016
- Registration and management of invitations
W3C
- Report
DM: I think there's scope for involvement in the NASA-backed hackathon.
GB: There's a game out soon where people can play out the scenario of Greenland and Antarctic ice sheets melting and the effect of flooding
JT: Earlier on, people were interested in an overview of the MELODIES project - http://melodiesproject.eu/
MaikRiechert: Talks through the Melodies website