14:43:46 <RRSAgent> RRSAgent has joined #csvw
14:43:46 <RRSAgent> logging to http://www.w3.org/2015/02/18-csvw-irc
14:43:48 <trackbot> RRSAgent, make logs public
14:43:48 <Zakim> Zakim has joined #csvw
14:43:50 <trackbot> Zakim, this will be CSVW
14:43:50 <Zakim> ok, trackbot; I see DATA_CSVWG()10:00AM scheduled to start in 17 minutes
14:43:51 <trackbot> Meeting: CSV on the Web Working Group Teleconference
14:43:51 <trackbot> Date: 18 February 2015
14:53:59 <jtandy> jtandy has joined #csvw
14:57:09 <gkellogg> gkellogg has joined #csvw
14:57:44 <JeniT> JeniT has joined #csvw
15:00:10 <danbri> danbri has joined #csvw
15:00:13 <gkellogg> zakim, code?
15:00:13 <Zakim> the conference code is 2789 (tel:+1.617.761.6200 sip:zakim@voip.w3.org), gkellogg
15:00:28 <Zakim> DATA_CSVWG()10:00AM has now started
15:00:34 <Zakim> +??P11
15:00:39 <gkellogg> zakim, I am ??P11
15:00:39 <Zakim> +gkellogg; got it
15:00:42 <ivan> zakim, dial ivan-voip
15:00:42 <Zakim> ok, ivan; the call is being made
15:00:43 <Zakim> +Ivan
15:01:20 <Zakim> +[IPcaller]
15:01:24 <danbri> zakim, IPcaller is danbri
15:01:24 <Zakim> +danbri; got it
15:01:41 <Zakim> +[IPcaller]
15:01:49 <DavideCeolin> zakim, IPcaller is me
15:01:50 <Zakim> +DavideCeolin; got it
15:02:02 <Zakim> +[IPcaller]
15:02:02 <danbri> zakim, who is talking?
15:02:13 <Zakim> danbri, listening for 10 seconds I heard sound from the following: gkellogg (33%), [IPcaller] (19%)
15:03:08 <jumbrich> jumbrich has joined #csvw
15:03:55 <Zakim> +??P2
15:04:04 <jumbrich> zakim ??P2 is me
15:04:18 <danbri> Q: how many of us are Skyping in? considering Ivan's notification that Zakim will be shut down later this year
15:04:27 <danbri> (I'm skyping)
15:04:40 <gkellogg> I’m on skype
15:05:13 <danbri> Dan, Ivan (called back to skype-in), JeniT,  Gregg, …
15:05:34 <danbri> scribenick: danbri
15:05:39 <danbri> jenit: 6 issues listed for discussion
15:05:42 <JeniT> https://lists.w3.org/Archives/Public/public-csv-wg/2015Feb/0020.html
15:06:16 <JeniT> https://github.com/w3c/csvw/issues/195
15:06:16 <danbri> … some quick hopefully, others need a bit more discussion
15:06:26 <danbri> "Effect of tableSchema on both Table and TableGroup #195"
15:06:30 <danbri> rrsagent, pointer?
15:06:30 <RRSAgent> See http://www.w3.org/2015/02/18-csvw-irc#T15-06-30
15:06:50 <danbri> jenit: I proposed 2nd of the suggested options
15:07:05 <danbri> namely that the one in table completely overides that in the table group
15:07:07 <danbri> no fancy merging
15:07:09 <danbri> any objections?
15:07:17 <danbri> ivan: in spite of the normalization of the metadata?
15:07:18 <danbri> jenit: yes
15:07:27 <danbri> gkellogg: it's more a search path
15:07:41 <danbri> when you're going through the cols you look at the first schema you find
15:07:50 <danbri> you don't take into effect what might also be represented in the table gorup
15:08:02 <danbri> you might imaging using table group … matching cols in diff langs
15:08:06 <danbri> that would not be supported here
15:08:17 <danbri> jenit: or you can imagine this as part of the normalization process
15:08:33 <danbri> e.g. copying in where missing [approximately what jeni said]
15:08:40 <danbri> jenit: any objections to proposed closure?
15:09:10 <danbri> [none heard - resolved]
15:09:21 <danbri> [updated to indicate editor action status]
15:09:28 <JeniT> https://github.com/w3c/csvw/issues/226
15:09:30 <danbri> rrsagent, pointer?
15:09:30 <RRSAgent> See http://www.w3.org/2015/02/18-csvw-irc#T15-09-30
15:09:35 <danbri> "Support for totalDigits and fractionDigits #226"
15:09:58 <danbri> jenit: suggestion is that we remove these
15:10:07 <danbri> ivan: +1
15:10:10 <jumbrich> +1
15:10:11 <danbri> +1
15:10:19 <danbri> (gkellogg +1'd in git)
15:10:25 <danbri> resolved - moved to editor action
15:10:25 <DavideCeolin> +1
15:10:41 <JeniT> https://github.com/w3c/csvw/issues/220
15:10:50 <danbri> rrsagent, pointer?
15:10:50 <RRSAgent> See http://www.w3.org/2015/02/18-csvw-irc#T15-10-50
15:10:51 <JeniT> http://w3c.github.io/csvw/metadata/#processing-tables
15:10:53 <danbri> "Move Processing Tables section from metadata to model document #220
15:11:18 <danbri> jenit: this is about what processors can do with tables - displayed, converted, etc etc
15:11:25 <danbri> suggestion is that we move that section into the Model doc
15:11:33 <danbri> …because it is about actions over the table model.
15:11:48 <danbri> … that means that the metadata doc is purely about how to annotate a table model / generate a table model
15:11:51 <danbri> +1
15:12:03 <danbri> gkellogg: might require creating a couple of term definitions in the syntax doc
15:12:16 <danbri> … juggling those between specs; makes a lot of sense to keep that in a single place.
15:12:19 <danbri> jenit: yep
15:12:35 <danbri> … probably a bit more editorial juggling to do also, cross-refs, where terms are defined etc.
15:12:43 <danbri> +1s from gkellogg, ivan on issue
15:12:46 <danbri> +1 from danbri here
15:12:47 <jumbrich> +1
15:12:49 <DavideCeolin> +1
15:13:02 <danbri> resolved - moved to editorial
15:13:19 <JeniT> https://github.com/w3c/csvw/issues/212
15:13:27 <danbri> jenit: the other 3 issues arose from possible use case offered in #212
15:13:38 <danbri> … looking at real life edu data, school performance stats
15:14:11 <danbri> … one thing interesting here is that looking at the data in depth, you see particular codes eg. SUPT, NE (= not entered) that can take the place of normal statistics
15:14:29 <danbri> i.e. cols have typically got numeric content but can have such values as alternate content
15:14:39 <danbri> one way of viewing those is to view them as null values
15:14:55 <danbri> jenit: I'll summarize this set of issues as they are linked
15:14:56 <JeniT> https://github.com/w3c/csvw/issues/218
15:15:02 <danbri> one approach is to say they're all kinds of nulls
15:15:23 <danbri> see #218, … which says the value interpreted as null could be written multiple ways
15:15:27 <danbri> … give a bit more structure, etc
15:15:39 <JeniT> https://github.com/w3c/csvw/issues/223
15:15:50 <danbri> (aside https://en.wikipedia.org/wiki/Null_(SQL) … scary world)
15:16:05 <danbri> jenit: #223 explores possibility of union values
15:16:13 <danbri> cols are numbers or else strings like NA
15:16:40 <gkellogg> q+
15:16:42 <JeniT> https://github.com/w3c/csvw/issues/224
15:16:44 <danbri> in order to support union-based types, where you'd need to list set of datatypes that the cells needed to comply with you would really need a different structure for datatypes
15:17:01 <danbri> … which brings us to #224, "Reworking structure for datatypes #224".
15:17:04 <JeniT> ack gkellogg
15:17:04 <danbri> q?
15:17:10 <ivan> q+
15:17:34 <danbri> gkellogg: another use case I've often seen: col might have different date stamps in it. Dates and Date-times intermingled. This would allow a super datatype there
15:17:44 <danbri> maybe allowing different dates in different formats
15:17:48 <JeniT> ack ivan
15:18:14 <danbri> ivan: original poster explicitly asked for data type unions, not sure it was just w.r.t. null.
15:18:30 <danbri> jenit: maybe it is helpful then to talk about the requirement for union types and whether we want to support them
15:18:31 <ivan> q+
15:18:42 <JeniT> ack ivan
15:18:44 <danbri> thoughts on supporting union types?
15:18:56 <danbri> ivan: I was wondering about the opposite direction?
15:19:15 <danbri> (jumbrich - is there anything from your study of actual CSV files to guide us here?)
15:19:16 <Zakim> -jumbrich
15:19:29 <danbri> jenit: any objections to restructuring the datatypes?
15:19:44 <Zakim> +??P2
15:19:48 <danbri> … basically they become their own little object, including base datatype, … e.g. decimal, … then you have extra properties
15:19:53 <jumbrich> zakim, ??P2 is jumbrich
15:19:53 <Zakim> +jumbrich; got it
15:20:01 <danbri> …  we could imagine in future naming those
15:20:01 <ivan> +1
15:20:10 <danbri> jenit: i personally think it is the right way to structure it
15:20:25 <jumbrich> q+
15:20:42 <danbri> jenit: you can still say 'decimal'
15:20:45 <JeniT> ack jumbrich
15:20:48 <ivan> ack m
15:20:49 <danbri> but you could also use a structure to set max/min etc
15:20:51 <ivan> ack mr
15:20:54 <ivan> ack ivan
15:21:30 <danbri> jumbrich: re dan's q, … in our study we used a simple heuristic, tried to guess cell type by using regexes, … and we found a couple of times that a col had multiple types in it. We just went with the majority.
15:21:47 <danbri> I could try to look deeper, find what kinds of types did this
15:22:08 <danbri> jenit: the fact that you noticed that that was happening is enough to know it's out there in real data
15:22:24 <danbri> jumbrich: other case is decimals … maybe excel export
15:22:35 <danbri> … maybe more interesting when strings vs numericals;  or strings vs urls
15:22:50 <danbri> I can try to look a bit into detail, what kind of different datatypes we observed in 1 col, and report back
15:23:05 <danbri> ivan: you were asking for usecases. would it help to get that?
15:23:19 <danbri> s/ivan:/jenit: ivan,/
15:23:33 <danbri> ivan: what if a cell can be interp'd by several of the datatypes
15:23:38 <danbri> jenit: suggest using array order
15:24:04 <danbri> ivan: yeahbut, … i know pathological, but imagine 2 alternatives. One is JSON, the other is a string.
15:24:09 <danbri> How do I decide that a string is JSON
15:24:18 <danbri> … or say XML
15:24:24 <danbri> do I have to parse the whole thing to know?
15:24:31 <danbri> I'm not sure we have a clear idea about these ugly edge cases
15:24:49 <danbri> jenit: that is a separate issue about validation of xml and json and html
15:25:05 <danbri> … this arises regardless because we say values have to be valid against whatever the datatype is
15:25:28 <danbri> …we can go one of two ways. We say you have to go all the way and really parse it. Or else say that typing on these cols is just a hint, ...
15:25:43 <danbri> (which has implics for union types in case that the markup isn't as valid as intended)
15:26:02 <danbri> gkellogg: we could say that datatypes explicitly have a regex form, then that is used to match, otherwise it is the first found. That would basically get everything.
15:26:10 <danbri> There are a couple of areas in xsd whjere you have confusions
15:26:16 <danbri> datatime stamps are datetimes, etc
15:26:21 <danbri> otherwise they are largely in diff spaces
15:26:38 <danbri> ivan: but then, … for time being at least, a convertor to json or to rdf can be lax
15:26:57 <danbri> … meaning that it does not do any validation. it will just believe what is there, and produce whatever is the datatype that is signalled.
15:27:12 <danbri> if we introduce this, strictly speaking a converter cannot be lax, it'll have to make a decision.
15:27:18 <danbri> jenit: depends on your meaning of 'lax'
15:27:34 <danbri> … text we have currently says that an errror must be generated if not valid against a particular datatype
15:27:39 <danbri> e.g. a decimal vs the word 'foo'
15:27:56 <danbri> …means that the value of the cell is set to the string 'foo' rather than a decimal number. The conversion
15:28:04 <danbri> …can then do whatever with that value. It could be lax and …
15:28:11 <danbri> or strict and raise error.
15:28:13 <danbri> various options.
15:28:26 <danbri> jenit: the checking of the value and the generation of the value for the cell happens regardless.
15:28:39 <danbri> it has to, otherwise you get in real messes around parsing of dates etc.
15:29:02 <danbri> gkellogg: say i have a set of datatypes, e.g. date, boolean, … listed, …
15:29:23 <danbri> as a convertor i need to check lexical form to see if value matches date, or then boolean, … then if it doesn't, what do i use, the last one? string?
15:29:35 <danbri> jenit: it (i.e. "foo" in example) is  a string
15:29:44 <danbri> … if you had datatype: boolean, and string value was foo
15:29:50 <danbri> …then value of the cell is the string 'foo'
15:30:15 <danbri> gkellogg: if the type was xml literal, … because we don't have a defined format for detecting it, i'd just go ahead and say it was an xml literal
15:30:53 <danbri> … could say default comes from def of that datatype
15:30:58 <danbri> … format/pattern
15:31:18 <JeniT> https://github.com/w3c/csvw/issues/236 << validation of html/xml/json datatypes
15:31:21 <danbri> we could then have same datatype diff times with diff patterns/formats
15:31:43 <danbri> ivan: we've moved away from the q of whether we want a structure for datatypes to be an array
15:31:46 <JeniT> https://github.com/w3c/csvw/issues/224
15:31:52 <ivan> +1
15:32:03 <danbri> jenit: let's try closing 224. any objections to sturctured datatypes?
15:32:10 <gkellogg> +1
15:32:10 <danbri> [tumbleweed]
15:32:17 <danbri> +1 is for it, right? :)
15:32:22 <gkellogg> Right
15:32:28 <danbri> resolved -> editor action
15:32:30 <danbri> rrsagent, pointer?
15:32:30 <RRSAgent> See http://www.w3.org/2015/02/18-csvw-irc#T15-32-30
15:32:47 <JeniT> https://github.com/w3c/csvw/issues/236
15:32:49 <danbri> #226
15:33:10 <danbri> jenit: if we have a cell that is marked as being e.g. json do we want to validate that it is actually json
15:33:16 <danbri> similarly for xml, html, …
15:33:34 <danbri> ivan: I agree w/ not validating
15:33:53 <danbri> danbri: +1 for not needing to
15:34:01 <danbri> gkellogg: validation isn't the right word
15:34:05 <danbri> (xml-wf?)
15:34:20 <danbri> q+ to suggest 'wellformedness'
15:34:37 <danbri> q-
15:34:50 <danbri> ivan: not even WF, as an xml segment needn't have a top level element
15:35:02 <Zakim> -jumbrich
15:35:08 <danbri> all we could find in rdf discussion of this was some DOM function
15:35:14 <danbri> gkellogg: just wording choice
15:35:18 <Zakim> +??P2
15:35:25 <danbri> i don't think we want/need detection on these 3
15:35:26 <jumbrich> zakim, ??P2 is jumbrich
15:35:26 <Zakim> +jumbrich; got it
15:35:34 <danbri> just a note to say that pattern/format  can be used
15:35:38 <danbri> to help discriminate
15:35:41 <danbri> jenit: can you clarify?
15:36:11 <danbri> gkellogg: I mean that if someone wanted to try to discriminate, based automatically on datatype, … could put a format in there which looked for <html
15:36:20 <danbri> so that they could distinguish
15:36:26 <danbri> ivan: i'd keep it simple
15:36:30 <JeniT> Proposal: we won’t built-in recognise/validate html/xml/json, but add a note to say that authors can add a pattern if they want
15:36:49 <danbri> (that's a heavy rider tacked on the end)
15:36:59 <danbri> ivan: so if there is a pattern i have to use it?
15:37:48 <RRSAgent> RRSAgent has joined #csvw
15:37:48 <RRSAgent> logging to http://www.w3.org/2015/02/18-csvw-irc
15:37:51 <JeniT> file:///Users/user/Documents/projects/w3ctag/csvw/metadata/index.html#formats-for-other-types
15:38:05 <JeniT> http://w3c.github.io/csvw/metadata/#formats-for-other-types
15:38:11 <danbri> rrsagent, please make logs public
15:38:25 <danbri> jenit: "format property provides a regex ...."
15:38:41 <danbri> gkellogg: in formats for data/time the format is yyyy-mm-dd in which case that is not a regex
15:38:47 <danbri> but there is still a pattern property
15:39:12 <danbri> gkellogg: what are consequences of having both format and pattern?
15:39:20 <danbri> jenit: pattern only on a format for a numeric type
15:39:26 <danbri> not at the top level, alongside format
15:39:29 <JeniT> http://w3c.github.io/csvw/metadata/#formats-for-numeric-types
15:39:30 <danbri> never clashes
15:39:45 <danbri> gkellogg: ok
15:40:12 <danbri> jenit: back to HTML/XML/JSON people can already use format property to constrain the value as gregg described
15:40:18 <danbri> so the note is just a pointer to existing functionality
15:40:19 <danbri> ivan: ok
15:40:19 <JeniT> Proposal: we won’t built-in recognise/validate html/xml/json, but add a note to say that authors can add a pattern if they want
15:40:25 <danbri> +1
15:40:28 <gkellogg> +1
15:40:32 <DavideCeolin> +1
15:40:36 <ivan> +1
15:40:39 <jumbrich> +1
15:40:45 <JeniT> +1
15:40:46 <danbri> rrsagent, pointer?
15:40:46 <RRSAgent> See http://www.w3.org/2015/02/18-csvw-irc#T15-40-46
15:40:57 <danbri> resolved
15:41:55 <danbri> https://github.com/w3c/csvw/issues/218 - Categories of null values #218
15:42:17 <danbri> gkellogg: instead of treating these multiple values as diff version of null, could treat them as …[missed]
15:42:21 <danbri> tokens
15:42:26 <danbri> ivan: what is current situation?
15:42:34 <danbri> … it looked as if null was already an array
15:42:47 <JeniT> https://github.com/w3c/csvw/issues/136
15:42:54 <danbri> jenit: it is but we decided from #136 that null would become a single value
15:43:06 <danbri> … so the doc hadn't been updated to reflect that resolution
15:43:26 <danbri> effectively #218 reopens #136 but with more of a rationale for why you might want multiple null values
15:43:38 <danbri> ivan: what's the merge? do we concat the arrays?
15:43:42 <danbri> jenit: atomic
15:43:46 <danbri> so you do not merge the arrays
15:43:57 <danbri> if you have two metadata files, the null list from A overides from B
15:44:00 <danbri> ivan: fine w/ that
15:44:07 <danbri> … and ok w/ several null values that way
15:44:10 <danbri> jenit: ok
15:44:33 <JeniT> Proposal: allow several null values, but merge in an atomic way (don’t merge arrays)
15:44:39 <gkellogg> +1
15:44:45 <jumbrich> +1
15:44:46 <danbri> +1
15:44:46 <DavideCeolin> +1
15:44:47 <ivan> +1
15:44:49 <JeniT> +1
15:45:06 <ivan> q+
15:45:10 <danbri> rrsagent, pointer?
15:45:10 <RRSAgent> See http://www.w3.org/2015/02/18-csvw-irc#T15-45-10
15:45:29 <JeniT> https://github.com/w3c/csvw/issues/223
15:45:36 <JeniT> ack ivan
15:45:44 <danbri> https://github.com/w3c/csvw/issues/223 - Allowing "unions" of datatypes? #223
15:46:02 <danbri> ivan: if we move to datatypes being these objects then the q of merge arises for those as well, regardless of the union issue
15:46:05 <danbri> jenit: true
15:46:14 <danbri> ivan: we merge atomically or property by property
15:46:17 <danbri> [someone said 'yes']
15:46:20 <danbri> jenit: i agree
15:46:30 <danbri> gkellogg: general trend is to make a small set of things which merge.
15:46:38 <danbri> ivan: maybe adding that note to the issue?
15:47:35 <danbri> jenit: propose that we allow arrays of datatypes to be provided and the first datatype wins in terms of labelling a particular value
15:47:39 <JeniT> Proposal: we allow arrays of datatypes to be provided, and the first matching datatype wins in terms of assignment of datatype to a particular value
15:47:48 <JeniT> (and atomic merge)
15:48:15 <danbri> ivan: at this moment my vote is "If we do it, then yes that's the way we should do it" (but) I would like to see jumbrich's measures before we make a decision on this rather than rush a new feature based on 1 new use case
15:48:20 <danbri> jenit: ok
15:48:25 <danbri> gkellogg: leave a week?
15:48:34 <danbri> … we need to decide
15:48:55 <danbri> jenit: what do you think is your measure of what would be persuasive. How many cases? 10, 20? 3?
15:49:01 <danbri> ivan: not a number
15:49:27 <danbri> … he goes through a certain amount of usecases for scientific data. If only 5% have this feature I would go against it. Obviously if it goes up to 30% then yes.
15:49:41 <jumbrich> info: we should have around 80K+ documents, from which 60k we could parse.
15:49:48 <jumbrich> q+
15:49:55 <danbri> jenit: I'd say 5% is too high a bound. We should support features that are in 1 in 20 docs
15:49:57 <JeniT> ack jumbrich
15:50:00 <danbri> gregg/dan: agree
15:50:21 <danbri> jumbrich: i will have a look and report on how many cols per doc we found at least 2 or 3 datatypes
15:50:27 <danbri> then try to present them in a reasonable way
15:50:29 <danbri> q+
15:50:41 <Zakim> -jumbrich
15:50:49 <danbri> [click]
15:50:55 <Zakim> +??P0
15:51:11 <jumbrich> zakim, ??P0 is jumbrich
15:51:11 <Zakim> +jumbrich; got it
15:51:14 <danbri> (sip client problems)
15:51:32 <danbri> q-
15:51:37 <danbri> ivan: I'll be away next week
15:51:41 <danbri> (dan away too)
15:51:52 <danbri> jumbrich: I'll email around beforehand
15:52:12 <danbri> ivan: fear of feature creep
15:52:13 <JeniT> ACTION: jumbrich to do an analysis on union types to see if they are prevalent in real data
15:52:13 <trackbot> Created ACTION-64 - Do an analysis on union types to see if they are prevalent in real data [on Jürgen Umbrich - due 2015-02-25].
15:52:23 <JeniT> ack danbri
15:53:31 <danbri> jenit: fine, leave til next week
15:53:52 <danbri> jenit: jumbrich's github id?
15:54:00 <jumbrich> jumbrich is my github id
15:54:17 <danbri> ...access on repo?
15:54:34 <danbri> ivan: no
15:55:03 <danbri> ivan to add jumbrich to our github group
15:55:13 <danbri> jumbrich: if we write code, should it be hosted here?
15:55:20 <danbri> e.g. to extract metadata files etc?
15:55:26 <danbri> ivan: no need to, can host wherever
15:56:02 <danbri> AOB?
15:56:36 <danbri> (github admin details)
15:58:13 <danbri> Adjourned.
15:58:32 <danbri> KUTGW - please try to vote on proposals etc on github.
15:58:42 <danbri> rrsagent, please draft minutes
15:58:42 <RRSAgent> I have made the request to generate http://www.w3.org/2015/02/18-csvw-minutes.html danbri
15:58:46 <Zakim> -DavideCeolin
15:58:47 <Zakim> -gkellogg
15:58:48 <Zakim> -JeniT
15:58:48 <Zakim> -jumbrich
15:58:51 <Zakim> -Ivan
15:58:53 <Zakim> -danbri
15:58:53 <Zakim> DATA_CSVWG()10:00AM has ended
15:58:53 <Zakim> Attendees were gkellogg, Ivan, danbri, DavideCeolin, JeniT, jumbrich
15:59:38 <ivan> trackbot, end telcon
15:59:38 <trackbot> Zakim, list attendees
15:59:38 <Zakim> sorry, trackbot, I don't know what conference this is
15:59:46 <trackbot> RRSAgent, please draft minutes
15:59:46 <RRSAgent> I have made the request to generate http://www.w3.org/2015/02/18-csvw-minutes.html trackbot
15:59:47 <trackbot> RRSAgent, bye
15:59:47 <RRSAgent> I see 1 open action item saved in http://www.w3.org/2015/02/18-csvw-actions.rdf :
15:59:47 <RRSAgent> ACTION: jumbrich to do an analysis on union types to see if they are prevalent in real data [1]
15:59:47 <RRSAgent>   recorded in http://www.w3.org/2015/02/18-csvw-irc#T15-52-13