16:28:24 <RRSAgent> RRSAgent has joined #coremob
16:28:24 <RRSAgent> logging to http://www.w3.org/2012/06/26-coremob-irc
16:28:26 <trackbot> RRSAgent, make logs 25
16:28:26 <Zakim> Zakim has joined #coremob
16:28:28 <trackbot> Zakim, this will be 
16:28:28 <Zakim> I don't understand 'this will be', trackbot
16:28:28 <Dong-Young> Dong-Young has joined #coremob
16:28:29 <trackbot> Meeting: Core Mobile Web Platform Community Group Teleconference
16:28:29 <trackbot> Date: 26 June 2012
16:28:43 <Josh_Soref> Scribe: Josh_Soref 
16:28:48 <nghanavatian> nghanavatian has joined #coremob
16:28:54 <Josh_Soref> topic: Testing
16:29:56 <chrisramos> chrisramos has joined #coremob
16:29:57 <ytsai> ytsai has joined #coremob
16:29:58 <ming> ming has left #coremob
16:30:05 <ming> ming has joined #coremob
16:30:39 <Josh_Soref> darobin: topic for today is Testing Testing Testing
16:30:53 <Josh_Soref> ... with maybe a little on vendor prefixing
16:30:59 <Josh_Soref> ... yesterday we talked about QoI tests
16:31:04 <Josh_Soref> ... conformance tests
16:31:10 <Josh_Soref> ... prioritizing interop issues
16:31:41 <Josh_Soref> ... testing the untestable
16:31:48 <Josh_Soref> ... we had a notion of testing for areas
16:31:57 <Josh_Soref> ... "categorizing testing/levels"
16:33:18 <Eunjoo> Eunjoo has joined #CoreMob
16:33:43 <jfmoy> jfmoy has joined #coremob
16:34:30 <wesj> wesj has joined #coremob
16:34:34 <Josh_Soref> [ darobin live edits a text file ]
16:34:51 <Wonsuk> Present+ Wonsuk_Lee
16:34:58 <Josh_Soref> rob: you might be interested in building a web app that's primarily an audio player
16:35:18 <ming> present+ Ming_Jin
16:35:31 <Josh_Soref> ... you might really care about ring 2+3 and only ring 1 of typography
16:36:05 <Josh_Soref> tobie: rob's point goes in the direction of the point that Josh_Soref made yesterday
16:36:12 <Josh_Soref> ... leveling doesn't make sense for extra features
16:36:22 <wseto> wseto has joined #coremob
16:36:45 <Josh_Soref> dehgan: polling app developers
16:36:52 <Josh_Soref> ... "what features do you need for these themes"
16:36:59 <wseto> q+
16:37:50 <Josh_Soref> DanSun: we might want a video category
16:38:03 <Josh_Soref> [ Scribe isn't going to transcribe the text file ]
16:38:06 <darobin> q?
16:38:14 <darobin> ack wseto 
16:39:08 <chihiro> chihiro has joined #coremob
16:40:14 <mattkelly> mattkelly has joined #coremob
16:40:26 <tobie> q+
16:40:32 <mattkelly> q+
16:40:49 <julianshen> julianshen has joined #coremob
16:40:58 <Josh_Soref> mattkelly: the need to automate tests....
16:41:35 <tobie> ack mattwkelly
16:41:46 <tobie> ack mattkelly
16:42:12 <Josh_Soref> [ chairs bicker at eachother over testing the untestable ]
16:42:45 <Josh_Soref> tobie: categorization is useful
16:42:53 <Josh_Soref> ... but a goal of this project is to fight fragmentation
16:43:04 <Josh_Soref> ... having a device that's a good fit for some apps and not others
16:43:06 <Josh_Soref> ... is a problem
16:43:10 <darobin> q?
16:43:12 <Josh_Soref> ... i want to raise a flag about this
16:43:13 <darobin> ack tobie 
16:43:17 <mattkelly> q+
16:43:32 <Josh_Soref> jo: surely it's legitimate to have devices with a specific purpose in mind
16:43:50 <Josh_Soref> tobie: for the vast majority of mobile devices people are interested in
16:43:54 <Josh_Soref> ... i'd argue it's less so
16:43:58 <bejram> bejram has joined #coremob
16:44:19 <Josh_Soref> jo: say you're building a navigation - car app
16:44:24 <Josh_Soref> tobie: it's not mobile
16:44:33 <Josh_Soref> jo: it's "mobile scoped, not mobile specific"
16:44:48 <mattkelly> ack mattkelly
16:45:23 <Josh_Soref> jo: rob, why don't you lead us on QoI?
16:45:27 <Josh_Soref> rob: i don't know how to do this
16:45:47 <Josh_Soref> ... it's the thing that causes us the most problems:
16:45:53 <Josh_Soref> ... browsers not quite behaving right
16:45:58 <Josh_Soref> jo: give us an example
16:46:09 <Josh_Soref> rob: there are 2 examples that sum up the problems
16:46:13 <Josh_Soref> ... 1. password field
16:46:32 <Josh_Soref> ... if it has lots of dom elements before it, it hangs when you press backspace
16:46:44 <Josh_Soref> ... we attach a dom listener and clear it if it had one character
16:46:58 <Josh_Soref> ... 2. browser crashes if you have a thing to define a schema
16:47:10 <Josh_Soref> ... 3. browser clears local storage if you get a large calendar invite
16:47:25 <Josh_Soref> ... it took us 6 months to reach what we think is a reproducible test case for that last one
16:47:45 <Josh_Soref> darobin: some of the tests you mention are egregious corner cases of one browser
16:47:51 <Josh_Soref> ... hopefully in a single version of the browser
16:47:57 <Josh_Soref> ... we could have a test suite for that
16:48:06 <Josh_Soref> ... but it would require automation driving
16:48:12 <Josh_Soref> ... and it's more in the field of regression testing
16:48:15 <Josh_Soref> ... than QoI
16:48:20 <Josh_Soref> tobie: i agree w/ darobin 
16:48:34 <Josh_Soref> ... you end up w/ test suites targeted at existing browser bugs
16:48:38 <Harrison> Harrison has joined #coremob
16:48:42 <mansoor> mansoor has joined #coremob
16:48:44 <Josh_Soref> ... and browser vendors don't like that
16:48:48 <Josh_Soref> rob: absolutely
16:48:58 <Josh_Soref> ... and it makes the browsers you build for look like they're the worst
16:49:07 <Josh_Soref> rob: conformance to spec is something we don't pay attention to
16:49:13 <Josh_Soref> ... we need to focus on real devices
16:49:20 <Josh_Soref> ... nuances that don't quite work
16:49:26 <Josh_Soref> ... we need to deliver now
16:49:33 <Josh_Soref> ... waiting for things to improve isn't an option
16:49:48 <Josh_Soref> darobin: conformance testing brings a lessening
16:49:53 <Josh_Soref> ... of problems with time
16:50:14 <Josh_Soref> ... there's a reason no one's asking about GIFs or Tables
16:50:27 <Josh_Soref> Josh_Soref: only in the last 5 years (gifs were crashing before)
16:50:33 <Josh_Soref> ... (tables may have been problematic more recently)
16:50:44 <Josh_Soref> darobin: performance... not hardware accelerated graphics
16:50:47 <Josh_Soref> ... css animations
16:51:00 <Josh_Soref> ... where the frame rate suddenly drops to 1/5 s
16:51:04 <Josh_Soref> ... those are more common
16:51:13 <Josh_Soref> ... i think fixing those things can help
16:51:26 <Josh_Soref> rob: i think we're close to the problem of defining what a device is capable of
16:51:32 <Josh_Soref> ... and detecting if it's doing well enough
16:51:34 <Josh_Soref> ... or doing badly
16:51:42 <Josh_Soref> ... we have flags to detect "fastish" or "slowish"
16:51:54 <Josh_Soref> ... and vary how much we do based on how fast we perceive the device to be
16:52:04 <Josh_Soref> ... that isn't correlated to the absolute performance of the hardware
16:52:09 <Josh_Soref> ... it correlates to the browser
16:52:14 <Josh_Soref> darobin: there's a relationship
16:52:16 <Josh_Soref> q+
16:52:33 <Josh_Soref> ... part of what we've talked about before wrt QoI
16:52:37 <Josh_Soref> ... is whether it's doable
16:52:45 <Josh_Soref> ... and people get performance testing wrong most of the time
16:52:53 <Josh_Soref> ... i'd like to find out if this group wants to do it
16:52:56 <girlie_mac> girlie_mac has joined #coremob
16:53:01 <Josh_Soref> ... and has the right resources to do it right
16:53:18 <darobin> q?
16:53:30 <darobin> q+ lars erik
16:53:40 <darobin> ack jo
16:53:48 <darobin> ack lars eris
16:53:51 <darobin> ack lbolstad 
16:53:57 <darobin> ack l 
16:53:59 <darobin> ack e
16:54:03 <darobin> q+ lbolstad 
16:54:19 <Josh_Soref> Josh_Soref: i want to praise FT for doing the right thing
16:54:26 <Josh_Soref> ... namely to detect performance
16:54:33 <Josh_Soref> ... and then adjusting what they do based on it
16:54:35 <Josh_Soref> q?
16:54:53 <darobin> ack lbolstad 
16:55:10 <Josh_Soref> tobie: among the QoI issues
16:55:19 <Josh_Soref> ... are those that i added to the spec yesterday
16:55:25 <Josh_Soref> ... asked on and on again by game makers
16:55:28 <Josh_Soref> ... speed of canvas
16:55:32 <Josh_Soref> ... speed of css animation
16:55:40 <Josh_Soref> ... multiple sounds together
16:55:42 <Josh_Soref> ... latency
16:55:49 <Josh_Soref> ... - which is really terrible on some devices
16:55:57 <Josh_Soref> ... -- close to a second on some devices
16:56:10 <Josh_Soref> ... things which prevent the game industry from building html games
16:56:15 <Josh_Soref> mattkelly: i'd add physics performance
16:56:17 <Josh_Soref> ... and GC pauses
16:56:26 <Josh_Soref> ... what i was focusing on in Ringmark early
16:56:28 <Josh_Soref> ... was page scrolling
16:56:32 <Josh_Soref> ... which affects everyone
16:56:41 <Josh_Soref> ... i'd assume including FT
16:57:27 <Josh_Soref> darobin: page scrolling performance
16:57:39 <Josh_Soref> ... touch responsiveness is delayed to handle clicks
16:57:48 <Josh_Soref> jo: people use native for touch reasons
16:59:03 <Josh_Soref> darobin: it's deliberate and can be hackiliy disabled
16:59:16 <Josh_Soref> rob: can you talk about testing video output
16:59:22 <Josh_Soref> s/can/yet: can/
16:59:31 <Josh_Soref> jet: mozilla has backdoors into firefox to do testing
16:59:36 <Josh_Soref> ... for fps
16:59:51 <Josh_Soref> ... for e.g. animations
16:59:59 <Josh_Soref> darobin: there's the Browser Testing and Tools WG
17:00:05 <Josh_Soref> jet: it may well be
17:00:10 <Josh_Soref> ... i haven't seen a proposal from them
17:00:23 <Josh_Soref> darobin: the scope is anything related to testing a browser
17:00:31 <Josh_Soref> ... they'd be allowed to produce technology we're not
17:00:41 <Josh_Soref> tobie: we could write a note to that group
17:00:48 <Josh_Soref> darobin: if you have requirements around that
17:00:52 <Josh_Soref> ... then talk to them
17:01:02 <Josh_Soref> jet: for our needs, are requirements are largely met
17:01:10 <Josh_Soref> ... for this group you want to be able to test across all
17:01:19 <Josh_Soref> ... browsers
17:01:42 <jo> q?
17:01:48 <Josh_Soref> itai: just wondering if the answer to these tests is highly dependent on the hardware perf
17:01:57 <Josh_Soref> ... to test one compared to another
17:02:09 <Josh_Soref> ... maybe we need a way to have a combined grade for a hardware platform
17:02:15 <tobie> q+
17:02:19 <Josh_Soref> ... combining memory bandwidth, computing power, ...
17:02:24 <Josh_Soref> ... say "i'm a class B platform"
17:02:30 <mattkelly> q+
17:02:32 <Josh_Soref> darobin: that's possible, but it's hard
17:02:39 <Josh_Soref> ... we talked about yesterday
17:02:49 <Josh_Soref> ... to draw a line and say "this is a typical platform"
17:03:01 <Josh_Soref> ... on anything like this or better, you need to do this or better
17:03:21 <Josh_Soref> ... if you do something piggishly on a high end hardware, good for you
17:03:36 <tobie> ack tobie
17:03:42 <Josh_Soref> ... for feature phones, you can say you're below that
17:03:49 <Josh_Soref> itai: the idea is captured
17:04:03 <Josh_Soref> ack mattkelly 
17:04:16 <Josh_Soref> mattkelly: my opinion is in line with darobin 
17:04:21 <Josh_Soref> ... we should have a baseline and go from there
17:04:40 <tobie> q+
17:04:41 <Josh_Soref> ... for level 1, 50 sprites @30fps, any phone should run
17:04:45 <Josh_Soref> ... even an iPhone 3
17:05:03 <Josh_Soref> ... no Device Capabilities are in the fold
17:05:05 <Josh_Soref> ... e.g. NFC
17:05:08 <koichi> koichi has joined #coremob
17:05:10 <Josh_Soref> ... no one is building apps for that
17:05:22 <Josh_Soref> darobin: we're about to get an NFC WG
17:05:33 <tobie> ack tobie
17:05:34 <Josh_Soref> ... i hear interest in this
17:05:39 <Josh_Soref> ... how do we make it actionable
17:05:54 <Josh_Soref> ... does someone want to pick a baseline hardware
17:06:00 <Josh_Soref> ... i want speed of cpu/gpu
17:06:13 <Josh_Soref> bkelley: you can't quantify performance with a couple of numbers
17:06:17 <Josh_Soref> ... different architectures
17:06:23 <tobie> g+
17:06:26 <Yan> Yan has joined #coremob
17:06:29 <tobie> q+
17:07:15 <mattkelly> q+
17:07:16 <Josh_Soref> q?
17:07:19 <Josh_Soref> s/g+//
17:07:23 <Josh_Soref> q+ Dong-Young 
17:07:39 <Josh_Soref> ... memory bandwidth
17:07:42 <Josh_Soref> ... cache size
17:07:52 <Josh_Soref> darobin: can we cut corners in a way to be meaningful
17:08:05 <Josh_Soref> ... we know it's wrong, but good enough for our purposes
17:08:22 <Josh_Soref> bkelley: by establishing that baseline, we exclude devices
17:08:25 <Josh_Soref> q+ jo
17:08:30 <Josh_Soref> ack tobie 
17:08:51 <Josh_Soref> tobie: one issue at the bottom of this is whether we can look at a browser outside the device it's running on
17:09:02 <Josh_Soref> ... as an end user, i care about how quickly it runs on my browser on my phone
17:09:13 <Josh_Soref> ... they're tied together in a way much deeper than on desktop
17:09:21 <Josh_Soref> .... the other aspect is who the audience of these tests is
17:09:26 <Josh_Soref> s/..../.../
17:09:35 <Josh_Soref> ... for browser vendors, being able to compare matters
17:09:44 <Josh_Soref> ... for developers, it matters whether you can build to a phone
17:09:48 <Josh_Soref> ack mattkelly 
17:09:57 <Josh_Soref> mattkelly: 500mhz, no memory
17:10:13 <Josh_Soref> ... and completely awesome browser, and does 50fps, and it passes
17:10:26 <Josh_Soref> ... maybe we can go w/ numbers for individual target bits
17:10:31 <Josh_Soref> ... don't worry about hardware
17:10:39 <Josh_Soref> q+ to say target UCs
17:10:41 <jo> ack d
17:10:51 <Josh_Soref> darobin: say targets for browser-device
17:11:02 <Josh_Soref> Dong-Young: what matters is the combination of browser-hardware
17:11:05 <Josh_Soref> darobin: we can test that
17:11:10 <Josh_Soref> ... it just makes more test results
17:11:22 <Josh_Soref> tobie: you can do analysis to compare browsers on 200 different devices
17:11:26 <Josh_Soref> q?
17:11:27 <Josh_Soref> ack jo 
17:11:37 <Josh_Soref> jo: this conversation is going in the direction i want to talk about
17:11:47 <Josh_Soref> ... setting a particular hardware spec is the road to ruin
17:11:57 <Josh_Soref> ... many a young man has fallen on that road
17:12:20 <Josh_Soref> ... it's important to not talk about mobile phone 
17:12:29 <Josh_Soref> ... say your purpose is to make a "video player"
17:12:32 <marcos_lara> marcos_lara has joined #coremob
17:12:43 <Josh_Soref> ... it should be testable
17:12:59 <Josh_Soref> ... relativistic measures
17:13:06 <Josh_Soref> ... are probably the only sensible way of testing
17:13:28 <Josh_Soref> ... if i produce a thing and it works abysmally on a device
17:13:32 <Josh_Soref> ... it's not useful
17:13:55 <Josh_Soref> mattkelly: i'd argue we need very clear focus
17:13:59 <Josh_Soref> ... at least short term
17:14:08 <jo> q?
17:14:16 <Josh_Soref> ... my opinion is the group should focus on where the market is
17:14:20 <Josh_Soref> ... to catch up w/ native
17:14:23 <Josh_Soref> ... enable 2d games
17:14:32 <Josh_Soref> ... and where people will buy new markets
17:14:36 <Josh_Soref> s/new/in new/
17:14:41 <Josh_Soref> ... when we hit critical mass
17:14:49 <Josh_Soref> ... then it's much easier to talk about more aspirational issues
17:14:53 <Josh_Soref> ... focus on current market
17:14:57 <Josh_Soref> ... where they're sold and why
17:15:00 <Josh_Soref> ... 2d games
17:15:02 <Josh_Soref> ... a/v apps
17:15:04 <Josh_Soref> ... camera apps
17:15:21 <Josh_Soref> jo: i don't disagree
17:15:33 <Josh_Soref> ... i'd say categorizing in a limited and extensible way is a good thing
17:15:41 <Josh_Soref> ... i think relativistic measures is a good way
17:15:43 <Josh_Soref> q?
17:15:44 <Josh_Soref> ack me
17:15:44 <Zakim> Josh_Soref, you wanted to say target UCs
17:16:32 <fantasai> Josh_Soref: I don't know if it's technically possible to count how many sprites are on the screen in Angry Birds, but a survey of the top N apps in the market, 2d games, video players...
17:16:43 <fantasai> Josh_Soref: Top 3 devices, top 10 apps for a thing, see what they're using
17:16:52 <fantasai> Josh_Soref: Maybe 25 sprites at 30 frames per second
17:17:04 <fantasai> Josh_Soref: You test at 15 frames, 30 frames, 60 frames
17:17:10 <fantasai> Josh_Soref: Figure out how many sounds, test for that
17:17:20 <fantasai> Josh_Soref: you build tests so it can test more than the target, so it can report that
17:17:26 <fantasai> Josh_Soref: then the tests can naturally scale up
17:17:36 <jo> q+
17:17:37 <fantasai> Josh_Soref: you can go back and say "This year, we need twice as many sprites"
17:17:51 <fantasai> Josh_Soref: we don't need to rewrite the tests, just change the benchmarks
17:18:14 <fantasai> Josh_Soref: I don't think it's very hard to do most of this. Might be boring. Might be fun
17:18:25 <Josh_Soref> jo: mattkelly you have done sprite counting, or you haven't don sprite counting?
17:18:30 <Josh_Soref> mattkelly: we did this 8 months ago
17:18:35 <Josh_Soref> ... we were building jsgamebench
17:18:40 <Josh_Soref> ... we built a 2d game bench
17:18:50 <Josh_Soref> ... we launched sprite counting in ringmark about 2 weeks ago
17:18:57 <Josh_Soref> ... we measure sprites rendering @30fps
17:19:01 <Josh_Soref> ... bare minium
17:19:04 <Josh_Soref> s/nium/nimum/
17:19:10 <Josh_Soref> ... high games need @60fps
17:19:16 <Josh_Soref> ... but that's rare, even on xbox
17:19:21 <Josh_Soref> ... it's definitely testable
17:19:30 <Josh_Soref> ... but on devices, push notices inbound can lead to a pause
17:19:37 <Josh_Soref> ... causing a fail, same for gc()
17:19:48 <Josh_Soref> ... from my perspective, if the pause happens, fail the test anyway
17:19:54 <Josh_Soref> ... we're definitely doing sprite counting
17:20:18 <Josh_Soref> tobie: jo, you were asking about type of sprites in a game
17:20:26 <Josh_Soref> darobin: jo was asking if sprite counting was done
17:20:33 <Josh_Soref> tobie: the answer to that was "yes"
17:20:39 <Josh_Soref> jo: mattkelly just answered that at more length
17:20:48 <Josh_Soref> tobie: a point of mattkelly 's research for jsgamebench
17:20:55 <Josh_Soref> ... was to define types of games and sprites per game
17:21:05 <Josh_Soref> ... cards have max of 5 sprites per game
17:21:12 <Josh_Soref> s/per game/concurrently/
17:21:27 <Josh_Soref> ... 25 for 2d platform games
17:21:39 <Josh_Soref> jo: action to tobie to chat this into the public domain
17:21:45 <darobin> ACTION: Tobie to provide numbers for required sprites/fps in games
17:21:45 <trackbot> Created ACTION-26 - Provide numbers for required sprites/fps in games [on Tobie Langel - due 2012-07-03].
17:21:51 <Josh_Soref> ... it seems publishing the numbers you're talking about
17:22:04 <Josh_Soref> ... it tells developers you need to target this
17:22:08 <Josh_Soref> ... and to browser vendors
17:22:10 <Josh_Soref> ... the test's job
17:22:19 <Josh_Soref> ... is to see if you can do 1fps, 2fps, 6...
17:22:22 <Josh_Soref> ... until it barfs
17:22:41 <Josh_Soref> ... at that point, you say "you did 25fps", "but you can't do X/Y/Z @fps"
17:22:46 <Josh_Soref> ... that's all it should say, not pass/fail
17:22:54 <Josh_Soref> ... but there are external qualifiers
17:23:04 <Josh_Soref> ... it doesn't matter if you haven't reached that
17:23:06 <tobie> q+
17:23:11 <tobie> ack jo
17:23:11 <Josh_Soref> ack jo
17:23:23 <Josh_Soref> ... external contemporaneous events on a device
17:23:30 <Josh_Soref> ... in the event you get an SMS during audio, what happens
17:23:41 <Josh_Soref> ... ok, you can do 60fps
17:23:44 <Josh_Soref> ... but what happens to the battery
17:23:51 <Josh_Soref> ... there's a range of metrics that are testable
17:23:58 <Josh_Soref> ... no Pass/Fail criteria
17:24:02 <Josh_Soref> ... but perfectly testable
17:24:05 <Josh_Soref> ack tobie 
17:24:15 <Josh_Soref> tobie: corie's jsgamebench
17:24:20 <Josh_Soref> ... brought to this discussion
17:24:29 <Josh_Soref> ... to have anything smooth enough, you need 30fps
17:24:37 <Josh_Soref> ... you don't need more than that, except hard core 3d games
17:24:41 <Josh_Soref> ... and less doesn't work
17:25:30 <Josh_Soref> tobie: about Battery
17:25:42 <Josh_Soref> ... how badly running a game drains the battery
17:25:50 <Josh_Soref> ... it goes back to browser-hardware combo
17:25:55 <Josh_Soref> ... good browser on bad hardware
17:26:04 <Josh_Soref> ... will have the same perf on bad browser on good hardware
17:26:14 <Josh_Soref> ... but good browser will probably drain the battery less than bad browser
17:26:20 <Josh_Soref> ... adding that would be good to test
17:26:41 <jo> q?
17:27:16 <mattkelly> s/a point of mattkelly 's research for jsgamebench/a point of cory's research for jsgamebench
17:27:18 <Josh_Soref> jo: and you can directly compare to find 'good' / 'bad' browser on a single device
17:27:36 <Josh_Soref> RRSAgent, draft minutes
17:27:36 <RRSAgent> I have made the request to generate http://www.w3.org/2012/06/26-coremob-minutes.html Josh_Soref
17:27:37 <mattkelly> s/corie's jsgamebench/cory's jsgamebench
17:27:53 <Josh_Soref> darobin: trying to summarize to reach actions
17:28:06 <Josh_Soref> ... anyone want to write tests?
17:28:13 <Josh_Soref> ... since you joined this group to do testing
17:28:19 <Josh_Soref> jo: i joined this group to talk about testing
17:28:25 <Josh_Soref> mattkelly: the question is who wants to write these tests
17:28:35 <Josh_Soref> ... i'm happy to port over what we've done w/ ringmark
17:28:35 <bejram1> bejram1 has joined #coremob
17:28:53 <Josh_Soref> jo: can we reverse out the underlying bits
17:29:02 <Josh_Soref> ... to codify the tests we want to accomplish
17:29:14 <Josh_Soref> mattkelly: we've done a bit of research for jsgamebench
17:29:17 <girlie_mac> an interesting study on browser battery consumption: http://www2012.org/proceedings/proceedings/p41.pdf
17:29:24 <Josh_Soref> RRSAgent, make logs public
17:29:30 <Josh_Soref> RRSAgent, draft minutes
17:29:30 <RRSAgent> I have made the request to generate http://www.w3.org/2012/06/26-coremob-minutes.html Josh_Soref
17:29:58 <Josh_Soref> Chair: Jo Rabin, Robin Berjon
17:30:17 <Josh_Soref> mattkelly: GC pauses can be guessed based on dramatic framerate drops
17:30:26 <Josh_Soref> vidhya: what's a GC pause
17:30:26 <jo> ACTION: mattkelly to document JSGameBench and the approach behind it
17:30:26 <trackbot> Sorry, couldn't find user - mattkelly
17:30:32 <Josh_Soref> mattkelly: sorry, Garbage Collection pause
17:31:06 <Josh_Soref> Josh_Soref: GC pauses run a bit on the main thread
17:31:12 <Josh_Soref> ... historically heavily there, recently less so
17:31:28 <Josh_Soref> mattkelly: for <audio>, we're testing from areweplayingyet
17:32:03 <Josh_Soref> ... you can't detect a pop, except w/ your ear
17:32:11 <Josh_Soref> ... page scrolling
17:32:20 <Josh_Soref> ... you need a high speed camera and a robot that flicks it
17:32:47 <Josh_Soref> darobin: for audio testing
17:32:57 <Josh_Soref> ... we could have a background audio track
17:33:06 <Josh_Soref> ... and whenever you're supposed to have a file overlay
17:33:10 <Josh_Soref> ... you have a visual queue
17:33:16 <Josh_Soref> tobie: it's doable to write a test
17:33:23 <Josh_Soref> ... it's harder to automate
17:33:37 <Josh_Soref> ... i wanted to add about physics testing
17:33:40 <Josh_Soref> ... and gc pauses
17:33:47 <Josh_Soref> ... the guy impact.js
17:34:01 <Josh_Soref> ... wrote extensively about it
17:34:09 <Josh_Soref> ... he had a 1 minute game with pre-controlled movements
17:34:12 <Josh_Soref> ... measuring movements
17:34:20 <Josh_Soref> ... to recognize GC pauses
17:34:24 <Josh_Soref> ... he explained why
17:34:31 <Josh_Soref> ... for physics, it's raw js engine perf
17:34:52 <Josh_Soref> ... it's not very difficult to script a physics scene and measure how many loops it does
17:34:53 <jo> q?
17:34:53 <Josh_Soref> q?
17:34:57 <Josh_Soref> ... in a given time
17:35:15 <Josh_Soref> jo: in the category of external interrupts
17:35:21 <Josh_Soref> ... sms, calls, gc
17:35:27 <Josh_Soref> ... anyone have a list?
17:35:32 <Josh_Soref> jo: there are 2 different categories
17:35:37 <Josh_Soref> ... gc isn't really external
17:35:44 <Josh_Soref> ... it's part of what you want to test
17:35:55 <Josh_Soref> ... you have accidental external events
17:35:59 <Josh_Soref> ... it's QoI
17:36:06 <Josh_Soref> ... it's stupid if receiving an SMS busts gameplay
17:36:15 <Josh_Soref> ... but it isn't fair if it impacts results of the test
17:36:19 <Josh_Soref> ... it's hard to reproduce
17:36:29 <Josh_Soref> ... gc pauses will get similar count if you run it a number of times
17:36:37 <Josh_Soref> q+ to mention phone calls
17:36:52 <Josh_Soref> jo: i want to scope this down
17:37:02 <Josh_Soref> ... tests in terms of Sprites, FPS
17:37:12 <Josh_Soref> ... don't want to characterize testing as what else is going on
17:37:15 <Josh_Soref> ... which will have an impact
17:37:24 <Josh_Soref> darobin: but it wouldn't be fair
17:37:31 <Josh_Soref> jo: let's decide SMS is out of scope
17:37:33 <Josh_Soref> ... objections?
17:37:35 <Josh_Soref> [ None ]
17:37:48 <Josh_Soref> q-
17:38:00 <Josh_Soref> jo: are we talking about Steady state perf or burst
17:38:05 <darobin> RESOLUTION: Interruptions and slowdowns due to factors external to the browser engine are out of scope for our tests
17:38:28 <Josh_Soref> ... sustained rate of 30fps but a burst of 60fps
17:38:35 <Josh_Soref> ... for 5s
17:38:41 <Josh_Soref> ... useful in network testing
17:38:55 <Josh_Soref> Josh_Soref: offhand, not this year
17:39:10 <Josh_Soref> mattkelly: are there UCs for this where things happen differently?
17:39:17 <Josh_Soref> ... e.g. drawing perf in canvas
17:39:24 <Josh_Soref> ... birds just sitting in slingshot
17:39:30 <Josh_Soref> ... there's 1 sprite
17:39:41 <Josh_Soref> ... when he hits the blocks+pigs, there are 50 sprites
17:39:51 <Josh_Soref> ... we should just test for 50 sprites steady
17:40:01 <Josh_Soref> jo: a good example is network interface performance
17:40:07 <Josh_Soref> ... queuing effects
17:40:22 <Josh_Soref> rob: some new devices have cameras that capture in burst mode
17:40:36 <Josh_Soref> Josh_Soref: can we rule it out until the end of 2012?
17:40:39 <Josh_Soref> [ Yes ]
17:40:53 <Josh_Soref> jo: no one has mentioned dom manipulation performance
17:41:00 <Josh_Soref> darobin: we have test suites for dom perf
17:41:12 <Josh_Soref> mattkelly: i think every game developer's opinion is canvas is the future
17:41:18 <Josh_Soref> ... it has a very granular api
17:41:33 <Josh_Soref> ... but some game developers use dom manipulation is faster than canvas on Android
17:41:41 <Josh_Soref> ... but let's eliminate that from gaming perspective
17:42:14 <Josh_Soref> Josh_Soref: do we care about accessibility?
17:42:20 <Josh_Soref> jo: we need to put dom manipulation in scope
17:42:26 <Josh_Soref> q+
17:42:54 <marcos_lara> additional info on Benchmarking canvas.
17:43:05 <Josh_Soref> ack me
17:43:11 <marcos_lara> "Benchmark Info: Tests the 2D canvas rendering performance for commonly used operations in HTML5 games: drawImage, drawImage scaling, alpha, composition, shadows and text functions."
17:43:17 <Josh_Soref> Josh_Soref: canvas doesn't have an accessibility story today
17:43:25 <Josh_Soref> ... but there's an accessibility story coming to html5
17:43:30 <Josh_Soref> ... which doesn't have performance tests
17:43:39 <marcos_lara> http://www.kevs3d.co.uk/dev/asteroidsbench/
17:43:54 <Josh_Soref> tobie: saying you can do games fast enough with dom manipulation on a mobile phone
17:43:55 <marcos_lara> test it out and it's open source 
17:44:02 <Josh_Soref> ... means there's no need to test it
17:44:07 <Josh_Soref> darobin: if people aren't complaining about it
17:44:10 <Josh_Soref> ... then it's not an issue
17:44:16 <Josh_Soref> tobie: it's no longer a real performance issue
17:44:23 <Josh_Soref> ... coming from a company that builds timeline
17:44:29 <Josh_Soref> ... which has a huge amount of dom nodes
17:44:36 <Josh_Soref> ... it's not something we've heard as an issue
17:44:41 <Josh_Soref> mattkelly: i'd agree
17:44:43 <jo> q?
17:44:50 <Josh_Soref> ... there are more important things to push
17:44:57 <Josh_Soref> ... it's not dom manipulation that's important
17:45:00 <Josh_Soref> ... it's position:fixed
17:45:31 <Josh_Soref> mattkelly: i think it's when it's combined with other things
17:45:35 <Josh_Soref> ... that leads to problems
17:45:41 <Josh_Soref> ... and more important to focus on
17:45:48 <Josh_Soref> ... we have a massive feed in timeline
17:45:56 <Josh_Soref> ... but position:fixed killed timeline
17:46:08 <Josh_Soref> rob: i was going to echo mattkelly 's point
17:46:10 <jet> jet has joined #coremob
17:46:15 <Josh_Soref> ... momentum scrolling+position:fixed
17:46:23 <Josh_Soref> ... they aren't well implemented
17:46:30 <Josh_Soref> ... you end up fiddling with them yourself
17:47:09 <Josh_Soref> DanSun: video is an important thing too
17:47:12 <Josh_Soref> ... for perf
17:47:18 <Josh_Soref> ... do we want to test for video too?
17:47:23 <Josh_Soref> ... resolution/fps...
17:47:27 <Josh_Soref> darobin: it's difficult
17:47:34 <tobie> q+
17:47:37 <Josh_Soref> ... one thing to test is battery consumption
17:47:44 <Josh_Soref> ... testing fps on <canvas> is easy
17:47:53 <jo> q+
17:47:58 <Josh_Soref> ... i'm not sure we can do it for <video> w/o underlying engine helping
17:47:59 <Josh_Soref> q+
17:48:11 <Josh_Soref> ... i think it's a good idea, not sure how
17:48:19 <Josh_Soref> ack tobie 
17:48:25 <Josh_Soref> tobie: your comment on video reminded me
17:48:29 <Josh_Soref> ... i heard from folks @orange
17:48:34 <Josh_Soref> ... that on a lot of devices, especially iPhone
17:48:39 <Josh_Soref> ... playing video isn't done in DOM
17:48:42 <Josh_Soref> ... but as a native plugin
17:48:46 <Josh_Soref> ... you can't overlay it with stuff
17:48:49 <Josh_Soref> ... like commercials
17:48:55 <Josh_Soref> darobin: video controls
17:48:58 <Josh_Soref> tobie: that's an issue
17:49:01 <Josh_Soref> ... but it's QoI
17:49:09 <mattkelly> q+
17:49:11 <jo> q?
17:49:23 <Josh_Soref> ack jo 
17:49:35 <Josh_Soref> jo: XXX+rob made a point
17:49:40 <Josh_Soref> ... about consistency/flow
17:49:50 <Josh_Soref> ... it may pass a 70fps test
17:49:53 <Josh_Soref> ... but not smoothly
17:50:04 <Josh_Soref> ... do we need to look out for it in QoI
17:50:21 <Josh_Soref> rob: yes, but i'm not sure how other than using an external camera
17:50:23 <Josh_Soref> q?
17:50:33 <Josh_Soref> jo: if it turns out to be impractical, it can drop out
17:50:41 <Josh_Soref> rob: i'm happy to take an action to see if it's practical
17:51:12 <Josh_Soref> jo: it would be nice to indicate to vendors that it's important for animations to be smooth
17:51:15 <jo> q?
17:51:16 <Josh_Soref> q?
17:51:24 <jo> ack j
17:51:25 <Josh_Soref> s/XXX/jet/
17:51:58 <darobin> ACTION: Shilston to expeditiously check whether it is practical to measure consistency of framerate
17:51:58 <trackbot> Created ACTION-27 - Expeditiously check whether it is practical to measure consistency of framerate [on Robert Shilston - due 2012-07-03].
17:52:04 <fantasai> Josh_Soref: So, on video I think that most browsers are starting to have APIs not standardized to check FPS in their enignes
17:52:19 <fantasai> Josh_Soref: Don't know when you'll b eable to do it formally, but think sometime early next year it might be possible at least non-standardly
17:52:34 <fantasai> Josh_Soref: For other forms of testing, a lot of devices have HDMI or displayport or something else
17:52:52 <fantasai> Josh_Soref: Now that might not match the display output, but might be able to write blackbox tester that uses that
17:52:55 <fantasai> Josh_Soref: instead of a camera
17:53:33 <fantasai> Josh_Soref: Also, some devices while they have platform access, might be a debug tool that lets you capture video
17:53:41 <fantasai> Josh_Soref: at RIM we have something that captures 1fps
17:53:56 <fantasai> Josh_Soref: I think it may be possible at least on some platforms to capture frame buffers and store that to a file for testing later
17:54:00 <jo> q?
17:54:15 <fantasai> Rob: Wondering Jet whether you were able to explain your HDMI capture etc.
17:54:27 <fantasai> Jet: I wouldn't hold that up as a best practice. Largely non-deterministic.
17:54:31 <fantasai> Jet: We try to get close
17:54:47 <fantasai> Jet: but in practice all the browser implementations upload a X to the GPU and ask the hardware to draw
17:54:57 <fantasai> Jet: Beyond that we can't measure
17:54:58 <tobie> GC test: http://html5-benchmark.com/ and related blog post: http://www.phoboslab.org/log/2011/08/are-we-fast-yet by ImpactJS author.
17:55:07 <fantasai> Jet: ... impacts our ability to get 60Hz
17:55:25 <fantasai> Jet: Definitely room for innovation, but need hardware vendors to come back with methods to measure hardware
17:55:26 <jo> ack m
17:55:45 <fantasai> mattkelly: Not sure how important to measure things like fps, given most devices defer to the native layer
17:55:53 <fantasai> mattkelly: But need things like adaptive streaming
17:56:08 <fantasai> mattkelly: They have a video that's 2 hours long, can actually dial up and down the bandwidth
17:56:09 <Josh_Soref> q?
17:56:13 <fantasai> mattkelly: important for audio as well
17:56:28 <fantasai> mattkelly: can then queue up the next bit at the correct rate 
17:56:29 <Josh_Soref> q+ to note that WebRTC / HTML are talking about that
17:56:56 <darobin> RESOLUTION: We are not going to specify baseline hardware, instead we will test device+browser combos
17:57:10 <fantasai> Josh_Soref: we're not just testing device combos, we're testing to targets
17:57:36 <darobin> RESOLUTION: We will specify a number of metrics that will be used to assess the limits of performance of specific device+browser targets
17:57:49 <Josh_Soref> RRSAgent, draft minutes
17:57:49 <RRSAgent> I have made the request to generate http://www.w3.org/2012/06/26-coremob-minutes.html Josh_Soref
17:57:51 <darobin> RESOLUTION: We will not be testing burst performance for now
17:58:47 <wseto> wseto has joined #coremob
17:58:49 <wesj> http://wiki.whatwg.org/wiki/Video_Metrics
17:58:58 <darobin> RESOLUTION: We will be testing in isolation
17:59:15 <Josh_Soref> [ Break ]
18:24:44 <jfmoy> jfmoy has joined #coremob
18:26:08 <lbolstad> lbolstad has joined #coremob
18:26:24 <marcos_lara> marcos_lara has joined #coremob
18:31:59 <betehess_> betehess_ has joined #coremob
18:33:46 <Josh_Soref> darobin: we covered QoI
18:33:59 <Josh_Soref> ... i'm somewhat concerned we have Actions for things
18:34:05 <Josh_Soref> ... but not Actions to write actual tests
18:34:06 <jo> q+
18:34:13 <Josh_Soref> q-
18:34:42 <Josh_Soref> ... writing tests is welcome
18:34:46 <Josh_Soref> jo: can we clarify
18:34:52 <Josh_Soref> ... do we want text
18:34:55 <Josh_Soref> ... or bits of JS?
18:34:59 <Josh_Soref> darobin: i mean actual code
18:35:11 <Josh_Soref> ... you may care about <audio> latency and parallelism
18:35:17 <Josh_Soref> ... and submit a proposed test to the grou
18:35:21 <Josh_Soref> s/grou/group/
18:35:50 <Josh_Soref> jo: i wonder if there's scope for people who don't write JS to write text to write JS to implement it
18:36:04 <Josh_Soref> darobin: it may be useful, but it's hard to describe the JS w/o knowing how to write it
18:36:25 <Josh_Soref> mattkelly: what i'd like to avoid is that people start writing random tests that add no value
18:36:36 <Josh_Soref> ... i think it's important to get consensus on level 1
18:36:42 <Josh_Soref> ... and the framework to produce them
18:37:04 <Josh_Soref> ... and get consensus on the harness
18:37:13 <Josh_Soref> ... and get a clear way to coordinate writing these tests
18:37:16 <Josh_Soref> ... preferably not the ML
18:37:33 <Josh_Soref> ... from my perspective, it's something like github
18:37:45 <Josh_Soref> darobin: i'm not sure we need the samework for QoI and Conformance tests
18:38:06 <Josh_Soref> jo: do you have a harness you'd like to propose?
18:38:17 <Josh_Soref> mattkelly: i think keeping a lot of the things in mind that we're trying to acheive
18:38:24 <Josh_Soref> ... particularly the ability to automate these things
18:38:32 <Josh_Soref> ... in Ringmark, we're using QUnit
18:38:40 <Josh_Soref> ... it may not be the right thing
18:38:44 <Josh_Soref> ... but people know how to use it
18:38:53 <Josh_Soref> ... QUnit can compile to w3c test frame
18:38:59 <Josh_Soref> ... but not back the other way
18:39:06 <jet> q+
18:39:09 <Josh_Soref> ... it may be a potential thing we can use
18:39:15 <jo> ack j
18:39:16 <Josh_Soref> fantasai: what is ringmark?
18:39:19 <Josh_Soref> ... it's a bunch of tests?
18:39:22 <Josh_Soref> ... is it a harness?
18:39:31 <Josh_Soref> mattkelly: Ringmark uses
18:39:33 <jo> q+ jet
18:39:36 <Josh_Soref> ... a lot of QUnit methodology
18:39:36 <jo> q- jo
18:39:40 <Josh_Soref> ... it has a runner, a results page
18:39:43 <Josh_Soref> ... all of the tests
18:39:53 <Josh_Soref> ... and it's built so you could add in automatable tests
18:40:01 <Josh_Soref> ... so long as they don't require single page instances
18:40:11 <Josh_Soref> ... and you can run it through the QUnit test runner as well
18:40:26 <Josh_Soref> fantasai: so it's a framework for running JS that has to be in the same top level page
18:40:43 <Josh_Soref> mattkelly: they can use iframe fixtures
18:40:56 <Josh_Soref> ... if you go to http://rng.io
18:41:05 <Josh_Soref> fantasai: if you put 10,000 iframes in a page
18:41:10 <Josh_Soref> ... that's a major perf test on iframes
18:41:21 <Josh_Soref> darobin: you test memory leaking fairly efficiently
18:41:25 <Josh_Soref> q+ to talk about flaws in tests
18:41:41 <Josh_Soref> darobin: one thing i'm unclear about the differences between QUnit and testharness
18:41:44 <Josh_Soref> ... i've used both
18:41:45 <jo> q+ tobie
18:41:48 <Josh_Soref> ... i can do the same thing in both
18:41:50 <Josh_Soref> mattkelly: you can
18:42:04 <Josh_Soref> ... we ran into a lack of documentation + direction in how you write these things
18:42:08 <Josh_Soref> ... these are fixable things
18:42:12 <Josh_Soref> ... there might be some overhead
18:42:19 <Josh_Soref> ... documentation is a big thing
18:42:24 <Josh_Soref> ... how tests are set up
18:42:33 <Josh_Soref> ... it's a lot harder to run in an automated fashion
18:42:41 <Josh_Soref> ... each test is meant to have an entire page defined
18:42:54 <Josh_Soref> ... for a <canvas> test, you have to have <head>, <body>
18:43:02 <Josh_Soref> darobin: the reason i'm pushing back here
18:43:08 <Josh_Soref> ... we need to integrate with existing test suites
18:43:15 <Josh_Soref> ... we have thousands of tests using testharness
18:43:22 <Josh_Soref> ... i'd like to avoid conversion
18:43:33 <Josh_Soref> mattkelly: there are probably tens of thousands of tests
18:43:41 <Josh_Soref> ... they are of varying quality/implementations
18:43:46 <Josh_Soref> ... they're all over the map
18:43:51 <Josh_Soref> ... some include other harnesses
18:44:01 <Josh_Soref> ... it seems like tests were of mixed quality
18:44:15 <Josh_Soref> darobin: one thing that would be useful would be to have documentation on these issues
18:44:31 <Josh_Soref> ... testharness is THE STANDARD for HTML, WebApps, etc., etc.
18:44:46 <Josh_Soref> s/s, etc./s, DAP, etc./
18:45:02 <Josh_Soref> ... even if we agreed there was a better alternative, i don't think we could convince them to convert
18:45:23 <Josh_Soref> mattkelly: from ringmark's perspective, it was about moving fast
18:45:30 <Josh_Soref> ... we had limited resources
18:45:44 <Josh_Soref> ... we had a goal of automating these things
18:45:51 <Josh_Soref> ... from OEMs and vendors I talked to
18:45:59 <Josh_Soref> ... none seem to run these
18:46:07 <Josh_Soref> ... they don't run testharness when they do device QA
18:46:27 <Josh_Soref> ... a goal should be to have Vendors run these so they can fail them
18:46:37 <Josh_Soref> jet: in general, we don't go running the entire W3 test suite
18:46:41 <jo> q?
18:46:42 <Josh_Soref> ... before we ship a browser
18:46:46 <Josh_Soref> ... it takes more than 24 hours
18:47:05 <Josh_Soref> ... to the other extent, anything that claims to test the browser in 60s isn't trustworthy
18:47:23 <Josh_Soref> ... ringmark could be useful for something in the middle
18:47:43 <Josh_Soref> ... for Mozilla, we can't commit to a third, fourth or fifth harness
18:47:58 <darobin> http://w3c-test.org/framework/app/suite
18:48:11 <Josh_Soref> darobin: you have a list of test suites
18:48:15 <Josh_Soref> ... suites test specifications
18:48:23 <Josh_Soref> ... you can look at results
18:48:32 <Josh_Soref> ... you can run tests
18:48:40 <Josh_Soref> ... you can load a runner
18:49:11 <Josh_Soref> ... there's a JSON API on this Database
18:49:26 <Josh_Soref> ... if you can have a Requirements Doc of what you'd like to see
18:49:39 <Josh_Soref> ... it would be possible for us, you, or a third party, to get a list of these tests
18:49:41 <Josh_Soref> ... run them, etc.
18:49:54 <Josh_Soref> ... to get something that could run in 15 minutes
18:50:02 <Josh_Soref> ... running 10,000 tests. and you could cherrypick
18:50:04 <Josh_Soref> jet: sure
18:50:13 <Josh_Soref> darobin: you could find bugs in the test
18:50:16 <Josh_Soref> s/test/tests/
18:50:19 <Josh_Soref> ... and presumably file them
18:50:26 <Josh_Soref> ... and hopefully find more bugs in the browsers
18:50:38 <Josh_Soref> fantasai: i don't think cherrypicking a bunch of tests
18:50:46 <Josh_Soref> ... and saying here's a test of the web stack
18:50:55 <Josh_Soref> darobin: i meant cherrypicking whole suites
18:51:01 <Josh_Soref> fantasai: like ACID tests, 
18:51:07 <Josh_Soref> ... we shouldn't build an ACID test
18:51:15 <Josh_Soref> darobin: i meant more the ones you can run automatically
18:51:21 <Josh_Soref> jet: a basic need i ran into
18:51:25 <Josh_Soref> ... i'm hacking firefox
18:51:30 <Josh_Soref> ... i put it on my phone
18:51:42 <Josh_Soref> ... i couldn't find a way to run the w3c suite against us
18:51:54 <Josh_Soref> fantasai: importing the suite into tinderbox
18:52:02 <jo> q?
18:52:06 <Josh_Soref> jet: that works for us, but we're trying to address everyone
18:52:11 <Josh_Soref> q- jet 
18:52:13 <Josh_Soref> ack me
18:52:13 <Zakim> Josh_Soref, you wanted to talk about flaws in tests
18:52:39 <fantasai> Josh_Soref: I wanted to talk about flaws in tests
18:52:58 <fantasai> Josh_Soref: Most of browser tests have laughed at tests they've looked at for the flaws they've found in the tests
18:53:09 <fantasai> Josh_Soref: But I don't think anyone has made a list of common mistakes
18:53:34 <fantasai> Josh_Soref: e.g. not scoping variables
18:53:58 <fantasai> Josh_Soref: Would be helpful to have a list for new test authors to write better tests
18:54:09 <jo> q?
18:54:13 <darobin> ACTION: Josh to survey people and compile a list of common errors in test writing
18:54:13 <trackbot> Sorry, amibiguous username (more than one match) - Josh
18:54:13 <trackbot> Try using a different identifier, such as family name or username (eg. jkerr, jsoref)
18:54:33 <darobin> ACTION: Robin to write documentation for testharness.js
18:54:34 <trackbot> Created ACTION-28 - Write documentation for testharness.js [on Robin Berjon - due 2012-07-03].
18:54:40 <darobin> ACTION: timeless to survey people and compile a list of common errors in test writing
18:54:40 <trackbot> Sorry, couldn't find user - timeless
18:54:44 <darobin> ACTION: Soref to survey people and compile a list of common errors in test writing
18:54:45 <trackbot> Created ACTION-29 - Survey people and compile a list of common errors in test writing [on Josh Soref - due 2012-07-03].
18:55:16 <fantasai> Robin: Problems I saw in Ringmark were feature tests, not perf tests
18:55:33 <fantasai> Robin: First 5% of testing something
18:55:43 <fantasai> Robin: rest was missing
18:55:47 <darobin> s/perf/conformance/
18:55:51 <fantasai> mattkelly: goal of Ringmark isn't surface area testing
18:56:03 <fantasai> mattkelly: To be successful, you'd need 100s of thousands of tests
18:56:12 <fantasai> mattkelly: we were just trying to provide a framework for thinking about things
18:56:19 <fantasai> mattkelly: need to get consensus around that
18:56:24 <Josh_Soref> q?
18:56:25 <jo> ack t
18:56:42 <fantasai> fantasai noted above that testharness.js can't test CSS other than its parsing
18:56:47 <tobie> http://test262.ecmascript.org/
18:56:58 <Josh_Soref> tobie: i'm not sure you're familiar w/ test262
18:57:02 <mattkelly> q+
18:57:09 <Josh_Soref> ... it's probably a good idea to know about performance
18:57:21 <Josh_Soref> ... running 27k tests takes about a quarter of an hour
18:57:24 <Josh_Soref> ... each in its own frame
18:57:35 <Josh_Soref> ... having been responsible for the architecture of ringmark
18:57:39 <Josh_Soref> ... about testharness.js and qunit
18:57:50 <Josh_Soref> ... the idea behind the original architecture
18:58:02 <Josh_Soref> ... having written for Prototype
18:58:11 <Josh_Soref> ... having JS test separated from the page in which it would run
18:58:14 <Josh_Soref> ... was extremely useful
18:58:19 <Josh_Soref> ... and a good architectural choice
18:58:24 <Josh_Soref> ... there's a lot of boilerplate
18:58:35 <Josh_Soref> darobin: fantasai might have something to add to that
18:58:41 <Josh_Soref> ... i know the CSS WG uses a build tool
18:58:49 <Josh_Soref> ... notably for multiformat
18:59:00 <Josh_Soref> fantasai: the tests we write @CSS WG have a bunch of metadata
18:59:04 <Josh_Soref> ... a lot of the boilerplate is XML
18:59:10 <Josh_Soref> ... a goal was tests be standalone
18:59:16 <Josh_Soref> ... that you could load in your browser
18:59:28 <Josh_Soref> ... rather than having to run a build to be able to see the results of your tests
18:59:33 <Josh_Soref> s/build/build system/
18:59:40 <Josh_Soref> ... it made it easier to work on tests
18:59:52 <Josh_Soref> ... it was harder when we had the build system required for Selectors
19:00:02 <Josh_Soref> ... it's only a little more work to have <!DOCTYPE> at the top
19:00:11 <Josh_Soref> tobie: i guess it makes more sense to have doctype in CSS
19:00:20 <Josh_Soref> ... that explains about how you did that
19:00:27 <Josh_Soref> ... fortestharness, it's in github
19:00:33 <Josh_Soref> s/for/for /
19:00:39 <Josh_Soref> ... it's easy to submit patches
19:00:45 <Josh_Soref> ... the documentation exists
19:00:49 <Josh_Soref> ... but it's included in the comments
19:00:53 <Josh_Soref> ... i submitted a patch a while back
19:01:06 <Josh_Soref> ... to turn that documentation into markdown
19:01:10 <Josh_Soref> ... to be turned into a readme
19:01:11 <marcos_lara> marcos_lara has joined #coremob
19:01:12 <jo> q?
19:01:15 <Josh_Soref> ... it was turned down
19:01:25 <Josh_Soref> ... AFAIK, the plan is to move the documentation into the wiki
19:01:41 <Josh_Soref> ... i don't think there's more overhead in testharness than any other Open Source project
19:01:42 <Josh_Soref> q?
19:01:44 <Josh_Soref> ack mattkelly 
19:01:49 <Josh_Soref> mattkelly: i had something of value to add
19:02:19 <Josh_Soref> mattkelly: i'd like to stress pragmatism
19:02:24 <Josh_Soref> ... about building practical web apps
19:02:34 <Josh_Soref> ... which is the reason people buy smartphones these days
19:02:38 <Josh_Soref> ... we need lots of tests
19:02:43 <Josh_Soref> ... but it's easy to go overboard
19:02:49 <Josh_Soref> ... including very strict testing
19:02:54 <Josh_Soref> ... is something to consider not testing
19:02:59 <Josh_Soref> ... e.g. ecmascript
19:03:09 <Josh_Soref> ... we shouldn't go overboard
19:03:23 <jo> q?
19:03:24 <Josh_Soref> jo: we could take a RESOLUTION not to go overboard
19:03:49 <Josh_Soref> ... so requirements for testharness
19:04:01 <Josh_Soref> rob: how could it be made more friendly to newcomers
19:04:06 <Josh_Soref> ... a vm image?
19:04:20 <Josh_Soref> darobin: like a git-clone of template project
19:04:37 <Josh_Soref> tobie: it requires Node
19:04:43 <Josh_Soref> darobin: who does not have Node.js?
19:04:49 <Josh_Soref> rob: it has a bunch of dependencies
19:04:58 <Josh_Soref> ... the entry barrier could be lowered
19:05:01 <tobie> q+
19:05:16 <Josh_Soref> darobin: the only thing you need is testharness.js and a test page
19:05:25 <Josh_Soref> ... you probably tried ringmark
19:05:39 <Josh_Soref> rob: there are dependencies like php for AppCache
19:05:43 <Josh_Soref> darobin: oh, right
19:05:56 <jo> ack tob
19:05:58 <Josh_Soref> tobie: there was a design disagreement about how that was done
19:06:11 <Josh_Soref> ... to write a testharness test, if you don't need server side stuff
19:06:16 <Josh_Soref> ... you don't need anything but an html page
19:06:24 <Josh_Soref> ... the coremob stuff on coremob's github repo
19:06:29 <Josh_Soref> ... requires both Node and a php runtime
19:06:34 <Josh_Soref> ... and that's stupid and should be fixed
19:06:50 <Josh_Soref> ... if all it requires is PHP, there are 1 click installers for it
19:07:10 <Josh_Soref> ... the existing code needs to be fixed
19:07:15 <Josh_Soref> ... and then documentation eeds to be updated
19:07:56 <Josh_Soref> ACTIONS?
19:08:04 <darobin> ACTION: Matt to remove the dependency on Node to get Ringmark running, and help make it easier to set up
19:08:05 <trackbot> Created ACTION-30 - Remove the dependency on Node to get Ringmark running, and help make it easier to set up [on Matt Kelly - due 2012-07-03].
19:09:17 <Josh_Soref> jo: who has requirements?
19:09:27 <Josh_Soref> ... i have a requirement that we not create another system for doing this
19:09:41 <Josh_Soref> darobin: to address rob's point
19:09:49 <Josh_Soref> ... is whether it'd be useful to have something similar to jsFiddle
19:09:55 <Josh_Soref> ... but to have it preloaded w/ testharness
19:10:02 <Josh_Soref> ... and then be able to save it online
19:10:08 <Josh_Soref> Josh_Soref: sounds useful
19:10:12 <Josh_Soref> tobie: sounds like a good idea
19:10:17 <Josh_Soref> ... you just volunteered
19:10:22 <Josh_Soref> darobin: if i can get time+budget...
19:10:33 <Josh_Soref> ... i can look into it
19:10:49 <Josh_Soref> jo: jet , you expressed requirements earlier
19:10:56 <darobin> ACTION: Robin to look into something like jsFiddle for test writing
19:10:56 <trackbot> Created ACTION-31 - Look into something like jsFiddle for test writing [on Robin Berjon - due 2012-07-03].
19:11:13 <Josh_Soref> jet: something that takes more than 60s but less than 24hrs
19:11:17 <Josh_Soref> ... proper scoring of tests
19:11:23 <Josh_Soref> ... not green/gray
19:11:30 <Josh_Soref> ... some depth to tests as well
19:11:35 <Josh_Soref> ... it's too easy to cheat on green
19:12:12 <Josh_Soref> jo: i volunteer darobin to write requirements
19:12:22 <Josh_Soref> darobin: i'll implement, but not write
19:12:34 <Josh_Soref> darobin: it wouldn't hurt if an OEM or Carrier did it
19:12:37 <Josh_Soref> ... how about jfmoy ?
19:12:47 <Josh_Soref> ... wouldn't that be helpful?
19:12:50 <Josh_Soref> jfmoy: for sure.
19:12:52 <Josh_Soref> ... i don't knkow
19:12:55 <Josh_Soref> s/kow/ow/
19:13:03 <Josh_Soref> darobin: what would you need to run automated tests
19:13:11 <Josh_Soref> jfmoy: for now, we're working on automation tests
19:13:18 <Josh_Soref> ... which we committed to give back to the group
19:13:24 <Josh_Soref> ... we're going down that road
19:13:29 <Josh_Soref> tobie: so it
19:13:39 <Josh_Soref> s/it/it must be easy to write requirements, since you did that/
19:13:49 <Josh_Soref> ... if it's sharable, then you should be able to give it
19:14:15 <darobin> ACTION: Moy to provide requirements for an automated test runner of all tests
19:14:15 <trackbot> Created ACTION-32 - Provide requirements for an automated test runner of all tests [on Jean-Francois Moy - due 2012-07-03].
19:14:25 <Josh_Soref> jfmoy: some of our tests are interactive
19:14:32 <Josh_Soref> darobin: if you'd like to present that
19:14:44 <Josh_Soref> jfmoy: we compared 3 test platforms
19:14:52 <Josh_Soref> ... ours, html5test.com, rng.io
19:14:53 <jo> ISSUE what are the requirements for a test framework?
19:15:07 <jo> s/ISSUE what are the requirements for a test framework?//
19:15:10 <Josh_Soref> ... sometimes interaction is needed for things
19:15:12 <Josh_Soref> ... like forms
19:15:19 <jo> ISSUE: what are the requirements for a test framework?
19:15:19 <trackbot> Created ISSUE-29 - What are the requirements for a test framework? ; please complete additional details at http://www.w3.org/community/coremob/track/issues/29/edit .
19:15:39 <Josh_Soref> ... all form bits for html5test/ringmark
19:15:45 <Josh_Soref> ... the proper keyboard display isn't tested
19:15:52 <Josh_Soref> ... for video, it isn't tested usefully
19:16:02 <jo> q?
19:16:13 <Josh_Soref> ... ringmark is more automated than ours
19:16:46 <Josh_Soref> mattkelly: starting a conversation in the group
19:17:00 <Josh_Soref> ... how QA processes work @ OEMs, Carriers, Browser Vendors
19:17:05 <Josh_Soref> ... making it as flexible as possible
19:17:11 <Josh_Soref> ... action to Orange, Mozilla, Qualcomm
19:17:21 <Josh_Soref> ... what would be the best way to get information out of ringmark
19:19:27 <darobin> ACTION: matt to document JSGameBench and the approach behind it
19:19:27 <trackbot> Created ACTION-33 - Document JSGameBench and the approach behind it [on Matt Kelly - due 2012-07-03].
19:20:35 <darobin> ACTION: matt to talk to OEMs/carriers about what they would most usefully need to get out of Ringmark results
19:20:35 <trackbot> Created ACTION-34 - Talk to OEMs/carriers about what they would most usefully need to get out of Ringmark results [on Matt Kelly - due 2012-07-03].
19:21:15 <darobin> COREMOB TESTING:
19:21:17 <darobin>     - Quality of Implementation tests
19:21:17 <Josh_Soref> Topic: todo today
19:21:18 <darobin>         - speed of canvas
19:21:20 <darobin>         - speed of CSS transitions
19:21:21 <darobin>         - audio latency
19:21:23 <darobin>         - audio parallelism
19:21:24 <darobin>         - physics performance (just raw JS performance)
19:21:26 <darobin>         - GC pauses (see ImpactJS)
19:21:27 <darobin>         - page scrolling performance
19:21:29 <darobin>         - touch responsiveness
19:21:31 <darobin>         ✓ DOM manipulation (not a real issue)
19:21:33 <darobin>     - Conformance tests
19:21:34 <darobin>         - Ringmark
19:21:36 <darobin>         - blockers for test writing
19:21:37 <darobin>         - test automation
19:21:39 <darobin>         - things that have perceptual outcomes (reftests, audio reftests…)
19:21:40 <darobin>         - Prioritising interoperability issues
19:21:42 <darobin>             - overlaying atop video
19:21:43 <darobin>         - integration with the W3C Test Framework facilities
19:21:45 <darobin>     - Categorising testing/levels (but fragmentation is evil)
19:21:46 <darobin>         - Gaming 2D
19:21:48 <darobin>         - Gaming 3D
19:21:50 <darobin>         - Device-Aware functionality
19:21:52 <darobin>         - e-books
19:21:54 <darobin>         - Multimedia playback (Audio, Video…)
19:21:56 <darobin>         - Core (networking, application packaging & configuration, HTML…)
19:21:58 <darobin>     - Testing the untestable
19:22:01 <darobin>         - things that don't have adequate test specs of their own (e.g. HTTP)
19:22:32 <Josh_Soref> Topic: Testing
19:22:39 <Josh_Soref> s/Testing/Testing Goals
19:22:56 <Josh_Soref> darobin: if you could get 5 test suites, what would you like
19:24:16 <fantasai> Rob: I wonder if we could put a survey up
19:24:41 <fantasai> Rob: e.g. Tobie's been talking with people buliding apps, maybe he has some idea of what people need most
19:25:00 <fantasai> mattkelly: In ringmark we focused on audio, 2d gaming, and camera apps
19:25:08 <fantasai> mattkelly: And then going from there, dirlling down into what features are missing
19:25:24 <fantasai> mattkelly: how can you test those features extensively to make sure they work well; that was the goal of Ringmark v1
19:25:43 <Josh_Soref> RRSAgent, draft minutes
19:25:43 <RRSAgent> I have made the request to generate http://www.w3.org/2012/06/26-coremob-minutes.html Josh_Soref
19:25:44 <fantasai> darobin: Ringmark tries to cover a lot of ground, covers some of it very thinly
19:25:56 <fantasai> mattkelly: Whatever we agree on L1 is not that big
19:26:06 <fantasai> mattkelly: In Ring 1 it's only about 14 features
19:26:19 <fantasai> mattkelly: 1-2 that are large: one is DRM
19:26:36 <fantasai> mattkelly: I think the feature set is reasonably small, and feedback I'm hearing is we just don't hae deep enough tests for each of those areas
19:26:37 <Josh_Soref> s/Robin:/darobin:/G
19:26:44 <fantasai> mattkelly: want to go through the features and see if group agrees on them
19:26:52 <jfmoy> +q
19:27:00 <tobie> +q
19:27:01 <fantasai> mattkelly: features were determined by us working with developers
19:27:14 <fantasai> mattkelly: I think I have an action to put more research in the group on how we qualified what's in ring 1
19:27:19 <fantasai> mattkelly: based on what apps are out there today
19:27:39 <fantasai> mattkelly: that would be my proposal, to start with what we've done in Ringmark and figure out if we have any pieces missing or should be removed, an dfocus our test writing effort there
19:28:53 <fantasai> ...
19:29:04 <fantasai> mattkelly: probably makes sense to have deeper consensus on categories in L1
19:29:17 <Josh_Soref> q?
19:29:23 <darobin> q+ fantasai 
19:29:26 <jo> ack jf
19:29:37 <fantasai> jfmoy: I put two links to our comparison
19:29:57 <fantasai> jfmoy: That's our results
19:30:17 <fantasai> jfmoy: We're pretty happy with L1 right now
19:30:50 <jo> q?
19:30:54 <jo> ack tob
19:31:04 <fantasai> tobie: Missed part of conversation
19:31:17 <fantasai> tobie: Robin, you wanted a couple areas of focus to work on?
19:31:24 <fantasai> tobie: Why not looking at what holes exist?
19:31:49 <fantasai> tobie: If what we want to do is to reuse existing tests and run those, makes sense to have a good understanding of what exists
19:32:06 <fantasai> tobie: and go through tests we want but aren't writen, might not need to prioritize
19:32:19 <jo> ACTION: tobie to carry out a gap analysis of existing W3C test suites
19:32:19 <trackbot> Created ACTION-35 - Carry out a gap analysis of existing W3C test suites [on Tobie Langel - due 2012-07-03].
19:32:46 <Josh_Soref> q+ to note that w3c test suites rarely test perf
19:33:15 <fantasai> Robin: a lot of work for one person, could split by section
19:33:44 <jo> ACTION-35?
19:33:44 <trackbot> ACTION-35 -- Tobie Langel to carry out a gap analysis of existing W3C test suites -- due 2012-07-03 -- OPEN
19:33:44 <trackbot> http://www.w3.org/community/coremob/track/actions/35
19:34:03 <fantasai> Robin: HTML5!
19:34:13 <fantasai> Robin: There are gaps we know aren't tested
19:34:32 <fantasai> Robin: Are there missing tests on things we care about there? Does someone want to look into that?
19:35:16 <fantasai> tobie: could be it's not a concern for companies/ppl
19:35:18 <jo> ACTION-35?
19:35:18 <trackbot> ACTION-35 -- Tobie Langel to carry out a gap analysis of existing W3C test suites -- due 2012-07-03 -- OPEN
19:35:18 <trackbot> http://www.w3.org/community/coremob/track/actions/35
19:35:33 <Josh_Soref> fantasai: what's the relationship between the tests we want to write
19:35:38 <Josh_Soref> ... in ringmark
19:35:41 <Josh_Soref> ... and level 1
19:35:54 <Josh_Soref> ... there's no way to get a solid set of tests for things in level 1 in any time reasonable
19:36:10 <Josh_Soref> ... if you can do 2% testing
19:36:13 <Josh_Soref> ... how is that representing
19:36:17 <Josh_Soref> ... showing interop
19:36:19 <jo> q?
19:36:22 <jo> ack f
19:36:26 <Josh_Soref> ... testing 5% of features at 50% effectiveness
19:36:32 <Josh_Soref> ... but you want this to be level 1
19:36:40 <Josh_Soref> ... and show interop by the end of the year
19:36:46 <Josh_Soref> tobie: what's your proposed solution
19:36:55 <Josh_Soref> fantasai: pick a few features, and prioritize those
19:37:02 <mattkelly> q+
19:37:02 <Josh_Soref> ... what's the goal of this document wrt testing?
19:37:12 <Josh_Soref> tobie: that's true of every
19:37:28 <Josh_Soref> different wg's aren't building test suites for the specs they're publishing
19:37:43 <Josh_Soref> s/different/... different/
19:38:04 <Josh_Soref> fantasai: i can't figure out how reporting results would relate to this
19:38:22 <Josh_Soref> tobie: as a group, as editing that spc
19:38:24 <Josh_Soref> s/spc/spec/
19:38:32 <Josh_Soref> fantasai: yes we want to contribute tests
19:38:35 <Josh_Soref> ... to a bunch of WGs
19:38:41 <jo> q+
19:38:42 <Josh_Soref> ... and also have some other way to report things
19:38:45 <Josh_Soref> ... as in ringmark
19:38:51 <Josh_Soref> ... that's the main advantage of it, right?
19:38:57 <Josh_Soref> jo: that's an assumption that needs to be verified
19:39:03 <Josh_Soref> ... it isn't an assumption of mine
19:39:16 <Josh_Soref> ... it isn't an assumption that this CG will produce a reporting framework
19:39:24 <Josh_Soref> darobin: i'd like to get the bottom of it
19:39:31 <Josh_Soref> ... fantasai has a good point
19:39:43 <Josh_Soref> ... the relationship between this document and the testsuite is unclear
19:39:55 <Josh_Soref> ... we should be able to reach consensus by the end of the year
19:40:04 <Josh_Soref> ... but how does that document relate to the testing effort it requires
19:40:09 <Josh_Soref> ... in the referenced specifications
19:40:16 <Josh_Soref> ... which we can't possibly accomplish by January
19:40:22 <Josh_Soref> ... unless aliens arrive
19:40:28 <Josh_Soref> ... that's where XO planet research helps
19:40:37 <Josh_Soref> darobin: plus we have to verify those tests
19:40:56 <Josh_Soref> ... we shouldn't produce a testsuite and say "This fully tests level 1"
19:41:06 <Josh_Soref> ... we need to articulate this clearly
19:41:15 <Josh_Soref> ... what i'd like to get out is an improvement
19:41:24 <Josh_Soref> ... if we test 5% where before we tested 2%, then i'm happy
19:41:32 <Josh_Soref> ... not as happy as if we could test 10%, but happier
19:41:42 <Josh_Soref> ... the test suite for this will never be final in under 10 years
19:41:52 <Josh_Soref> ... but i wanted to focus on high value targets for interop
19:42:07 <Josh_Soref> ... maybe html5 parsing is mostly interoperable
19:42:16 <Josh_Soref> ... maybe it's tested at 2% and that's ok
19:42:30 <Josh_Soref> ... but maybe shades of red, green, or pink doesn't work in <canvas>
19:42:42 <Josh_Soref> ... but maybe it's more important to get matching on blue by January
19:42:49 <Josh_Soref> ... does that make sense to people?
19:43:12 <Josh_Soref> dehgan:  one thing that would make tests solid
19:43:18 <Josh_Soref> ... if we make tests a moving target
19:43:29 <Josh_Soref> ... i get a result today
19:43:39 <Josh_Soref> ... and a result tomorrow, and my score goes down
19:43:44 <mattkelly> q?
19:43:49 <Josh_Soref> fantasai: i think it's great that people want to contribute to the testing effort @w3c
19:44:01 <Josh_Soref> ... but the goal of this CG seems to be to push for specific things to be fixed
19:44:07 <Josh_Soref> darobin: we want to defrag the web
19:44:09 <Josh_Soref> fantasai: right
19:44:14 <Josh_Soref> ... you want those fixed
19:44:24 <Josh_Soref> ... and to push for vendors to implement or fix those
19:44:35 <Josh_Soref> ... one thing that has not been done well
19:44:40 <Josh_Soref> ... at w3c
19:44:44 <Josh_Soref> ... is getting tests we've done
19:44:54 <Josh_Soref> ... and getting people excited
19:45:00 <Josh_Soref> ... ringmark did that
19:45:05 <Josh_Soref> ... well, making it a game
19:45:26 <Josh_Soref> ... the psychological pressure is lost if you won't seem to go somewhere in 10 years
19:45:27 <tobie> q+
19:45:35 <Josh_Soref> ... this is gamification of testing
19:45:41 <Josh_Soref> ... but if level up takes 10 years
19:45:47 <Josh_Soref> ... then it isn't going to work
19:45:56 <Josh_Soref> jo: it's impractical to do a suite for level 1
19:46:04 <jet> q+
19:46:10 <Josh_Soref> fantasai: one thing to think about is
19:46:14 <Josh_Soref> ... to break it down and prioritize
19:46:23 <Josh_Soref> ... to avoid spreading yourself too thinly
19:46:30 <Josh_Soref> ... and to focus communication effort
19:46:48 <Josh_Soref> ... more than even adding 3 CSS testing volunteers
19:47:03 <Josh_Soref> darobin: focus on making things pretty
19:47:21 <Josh_Soref> ... probably having ringmark 1, 2, 3, ... 17 in the next few years
19:47:29 <Josh_Soref> ... making it identifiable
19:47:42 <Josh_Soref> ... having conformance targets to have PR
19:47:50 <Josh_Soref> ... to avoid getting lost
19:47:54 <Josh_Soref> fantasai: it would be good
19:47:59 <Josh_Soref> ... to have a goal to release testing wise
19:48:03 <Josh_Soref> ... this document is a 10 year road map
19:48:12 <Josh_Soref> ... what will you get done by the end of the year
19:48:20 <Josh_Soref> ... and getting them involved and excited about
19:48:23 <jo> ?
19:48:26 <Josh_Soref> ... if all you have is an extra 200 tests
19:48:27 <jo> q?
19:48:31 <Josh_Soref> ... to the html5 parsing algorithm
19:48:36 <Josh_Soref> ... that won't get anyone excited
19:48:46 <darobin> q?
19:49:00 <darobin> ack Josh_Soref 
19:49:00 <Zakim> Josh_Soref, you wanted to note that w3c test suites rarely test perf
19:49:06 <jo> ack josh
19:51:38 <fantasai> Josh_Soref: if we want perf tests, we either need to find someone whose written them and steal them
19:51:48 <fantasai> Josh_Soref: or write them ourselves
19:51:53 <Josh_Soref> Zakim, close the queue 
19:51:53 <Zakim> ok, Josh_Soref, the speaker queue is closed
19:51:55 <fantasai> Josh_Soref: current w3c tests are conformance/interop tests
19:52:20 <fantasai> Josh_Soref: Wrt ringmark, don't like that failing the first ring prevents runnign the second ring
19:52:23 <darobin> ACTION: Robin to draft a test suite release strategy based on what fantasai and Josh_Soref described
19:52:23 <trackbot> Created ACTION-36 - Draft a test suite release strategy based on what fantasai and Josh_Soref described [on Robin Berjon - due 2012-07-03].
19:52:32 <fantasai> Josh_Soref: HTMl5 tests can have bonus points -- you can get them even if you didn't pass 
19:52:33 <Josh_Soref> s/runnign/running/
19:52:37 <fantasai> Josh_Soref: people like getting points
19:52:42 <Josh_Soref> s/HTMl5/HTML5/
19:52:45 <fantasai> Josh_Soref: different tests run on different tracks
19:52:58 <fantasai> Josh_Soref: same engineer doesn't work on all the different aspects of the web platform
19:53:13 <fantasai> Josh_Soref: can race up one track while another engineer works on other track
19:53:30 <jo> ack matt
19:53:41 <fantasai> mattkelly: it boils down to focus
19:53:51 <fantasai> mattkelly: earlier point around the hesitation and concern that L1 spec can get unweildy and large
19:53:54 <fantasai> mattkelly: I share the same concern
19:54:06 <fantasai> mattkelly: I feel strongly that for L1 spec we should focus on 14 different features, like we are in Ringmark
19:54:11 <fantasai> mattkelly: and focus intensely on that batch
19:54:23 <fantasai> mattkelly: and feel comfortable about our coverage of those 14 features by end of year
19:54:32 <fantasai> mattkelly: if we try to test all of HTMl5, we'll go down a rabbithole
19:54:38 <fantasai> mattkelly: and will not ship a coherent suite of tests
19:54:48 <fantasai> mattkelly: another point wrt bonus points, and why ringmark stops running if it fails
19:55:00 <fantasai> mattkelly: primary reason it does that is to make the browser look like it failed
19:55:14 <fantasai> mattkelly: goal is to reduce the fragmentation
19:55:26 <jo> q?
19:55:43 <fantasai> mattkelly: don't want to reward browser for jumping out and implementing WebGL from L2 when core features are not implemented
19:55:52 <fantasai> mattkelly: Think we should have many releases, and have different levels
19:56:02 <fantasai> mattkelly: L1 should have small amount of functionality, with ample test coverage
19:56:11 <fantasai> mattkelly: Ultimately, we don't know what the unknowns are until we start building this stuff
19:56:36 <fantasai> mattkelly: if we do small bite-size chunks, can cover more ground faster
19:57:04 <fantasai> mattkelly: I do feel that without having test suite in this group, we'd just have a nother doc, have no impact on industry
19:57:23 <fantasai> mattkelly: need a product that encapsulates our vision. test suite is how we make this happen
19:57:34 <fantasai> mattkelly: Do think group should have some work aorund crafting message
19:57:43 <fantasai> mattkelly: .. need to own that message
19:57:51 <fantasai> mattkelly: sharing of message, where group formulates what the structure of the message is
19:57:59 <fantasai> mattkelly: OEMs figure out how you message that to end users, end developers
19:58:06 <fantasai> mattkelly: Unclear if that should be part of group's goal
19:58:17 <fantasai> mattkelly: wrt focus, should focus on structure of that message, not necessarily delivering it
19:58:22 <jo> ack me
19:58:27 <fantasai> Jo: I agree with everything said before
19:58:48 <fantasai> Jo: I think the whole thing would be more tractable if there was a L0 which was smaller in scope than L1
19:58:56 <fantasai> timeless: if there was a smaller level 1...
19:59:29 <fantasai> Robin: There's no useful reduction of the current document for which we would have sufficient tests
20:00:09 <fantasai> Jo: I am not convinced this group should present a flashy state of things
20:00:20 <fantasai> Jo: But to present tests that other people can show under a UI
20:00:24 <fantasai> Robin: But we already have that
20:00:36 <fantasai> Robin: we already have a number of test suites that can report results that can be reused by others
20:00:41 <fantasai> Robin: Why would we do that?
20:00:53 <fantasai> Robin: We have frameworks to do that
20:01:13 <fantasai> Robin: One thing missing so far is packaging the rsults in a way that creates market pressure to improve the situation
20:01:31 <fantasai> Jo: What's the point of making a pretty interface?
20:01:39 <fantasai> Jo: Let's make the tests reusable by anybody
20:01:49 <fantasai> Robin: But we already have that in the W3C test frameworks
20:02:26 <fantasai> discussion between Jo and Robin of whether we should use w3c test frameworks or not
20:02:29 <fantasai> ~_~
20:02:30 <jo> ?
20:02:34 <jo> q?
20:02:40 <jo> ack tob
20:02:48 <fantasai> tobie: I think it would be reasonably easy for Ringmark to pull out tests from other WGs
20:02:59 <fantasai> tobie: Either by unbuilding a stage to existing tests
20:03:23 <fantasai> tobie: or by just changing ringmark so that it actually uses iframes and pulls existing tests into it
20:03:35 <fantasai> tobie: not a hard problem to solve
20:03:43 <fantasai> tobie: fantasai talked aobut test the web forward
20:04:02 <Josh_Soref> fantasai: Test The Web Forward effort
20:04:06 <Josh_Soref> ... Adobe is spearheading it
20:04:14 <Josh_Soref> ... and teaching people to write tests for CSS and SVG
20:04:17 <Josh_Soref> ... primarily
20:04:24 <jo> q?
20:04:25 <Josh_Soref> ... it's complementary to what you're doing here
20:04:33 <Josh_Soref> ... it isn't quite the same
20:04:47 <Josh_Soref> ... it's getting broader community to write tests
20:04:56 <Josh_Soref> ... and they're w3c contributions
20:05:06 <Josh_Soref> darobin: also something Mosquito did
20:05:33 <Josh_Soref> jo: can we have you talk to eachother?
20:05:44 <Josh_Soref> fantasai: there's events and you're welcome to attend
20:05:51 <jo> q?
20:05:51 <Josh_Soref> tobie: do you explain how testharness works
20:06:05 <Josh_Soref> fantasai: we talk people through the process of creating tests
20:06:08 <Josh_Soref> ... and submittting them
20:06:14 <Josh_Soref> ... and reviewing each-other's tests
20:06:20 <Josh_Soref> darobin: how did it go?
20:06:34 <Josh_Soref> fantasai: it takes writing 20 tests to get good at it
20:06:45 <Josh_Soref> darobin: i made a presentation similar to yours 2 weeks before
20:07:00 <Josh_Soref> ... about half an hour in, i realized no one had written tests for anything before
20:07:24 <Josh_Soref> fantasai: i started in the mozilla project doing this with very little guidelines/guidance
20:07:24 <darobin> q?
20:07:29 <darobin> ack jet
20:07:29 <Josh_Soref> ack jet 
20:07:29 <jo> ack j
20:07:36 <Josh_Soref> jet: comments on testharness
20:07:41 <Josh_Soref> ... about depth of a test
20:07:49 <Josh_Soref> ... and fail on a test v. continue on a test
20:07:59 <Josh_Soref> ... there are very basic features that if you add them to ringmark
20:08:04 <Josh_Soref> ... no browser will pass ring 0
20:08:08 <Josh_Soref> ... i don't think that's the goal of testing
20:08:23 <Josh_Soref> ... i don't think complete css 2.1 every single thing is good
20:08:34 <Josh_Soref> ... and by definition not testing other features
20:08:45 <tobie> +q
20:08:54 <Josh_Soref> ... we'd like /all to be the default config for ringmark
20:08:56 <Josh_Soref> s/+q//
20:09:01 <Josh_Soref> ... you can timebox
20:09:06 <Josh_Soref> ... level 1 december
20:09:26 <Josh_Soref> ... what you have in your tests at december is your level 1
20:09:38 <Josh_Soref> darobin: if we did this today
20:09:46 <Josh_Soref> ... we'd have html3 and some level of scripting and styling
20:09:50 <Josh_Soref> jet: right
20:09:57 <Josh_Soref> ... browsers claim support for html5
20:10:06 <Josh_Soref> ... we'll try to turn things green
20:10:11 <Josh_Soref> ... but that won't solve interop
20:10:47 <Josh_Soref> darobin: one thing to do is testing
20:11:15 <Josh_Soref> ... every six months we release a new set of tests we'd like to turn things green
20:11:23 <Josh_Soref> fantasai: so like an acid test with more tests?
20:11:24 <mattkelly> q+
20:11:36 <Josh_Soref> darobin: right, with a lot more tests
20:11:50 <Josh_Soref> ... anything we have tests already, we take
20:11:58 <Josh_Soref> fantasai: seems like a reasonable goal to me
20:12:25 <Josh_Soref> DanSun: whatever we do
20:12:30 <Josh_Soref> ... testing, quality is the key
20:12:38 <Josh_Soref> ... ringmark testing, in 5s there's no chance at all
20:12:43 <fantasai> darobin^: wherever there's major gaps or interop problems, we add more tests, and package this all up nicely
20:13:14 <Josh_Soref> ... can we list test suites
20:13:19 <Josh_Soref> ... and which are the most trusted
20:13:23 <Josh_Soref> ... and maybe leverage that?
20:13:39 <Josh_Soref> ... and integrate w/ ringmark to show the results
20:13:46 <Josh_Soref> tobie: the problem is, that needs towrk
20:13:52 <Josh_Soref> s/twork/work/
20:14:00 <Josh_Soref> ... and that requires resources
20:14:11 <Josh_Soref> s/towork/work/
20:14:17 <Josh_Soref> mattkelly: +1 for small packets of tests
20:14:23 <Josh_Soref> ... -1 on a 10 year plan for a doc
20:14:37 <Josh_Soref> s|s/twork/work/||
20:15:09 <Josh_Soref> darobin: i'd find that weird
20:15:21 <Josh_Soref> [ Lunch ]
20:15:37 <darobin> ACTION: Robin to assess which existing test suites can be reused and at what level of coverage they stand
20:15:38 <trackbot> Created ACTION-37 - Assess which existing test suites can be reused and at what level of coverage they stand [on Robin Berjon - due 2012-07-03].
20:15:48 <darobin> ISSUE: should the document track the testing effort or not
20:15:48 <trackbot> Created ISSUE-30 - Should the document track the testing effort or not ; please complete additional details at http://www.w3.org/community/coremob/track/issues/30/edit .
21:14:52 <lbolstad> lbolstad has joined #coremob
21:15:31 <Wonsuk> Wonsuk has joined #coremob
21:23:25 <chrisramos> chrisramos has joined #coremob
21:28:27 <Dong-Young> Dong-Young has joined #coremob
21:28:30 <andrewhubbs> andrewhubbs has joined #coremob
21:29:33 <ming> ming has joined #coremob
21:29:43 <Eunjoo> Eunjoo has joined #CoreMob
21:29:43 <tobie> tobie has joined #coremob
21:30:43 <fantasai> Jo: Various side discussions happened over lunch in an attempt to break the logjam
21:30:53 <fantasai> Jo: Starting point is having concrete deliverables by end of year
21:31:05 <fantasai> Jo: Document seems perfectly achievable, but what are we going to deliver in terms of tests by the end of the year
21:31:29 <fantasai> Jo: So here's a plan, taking fantasai's point on board,
21:31:45 <fantasai> Jo: Yes, we need something nice and visual that ppl can rally around. But doesn't have to be this group
21:32:01 <jfmoy> jfmoy has joined #coremob
21:32:05 <fantasai> Jo: So we should provide infrastructure to do that,
21:32:17 <fantasai> Jo: What we need to do is an existing proof of an actual implementation of such a thing
21:32:32 <fantasai> Jo: Facebook is happy to refactor their existing ringmark output to fit in with what I'm about to say
21:32:53 <fantasai> Jo: In terms of meeting objective of having visual output, FB will provide that existence proof
21:33:03 <fantasai> Jo: Would be good fo others to provide similar things
21:33:17 <fantasai> Jo: Browser vendors might want to work headless testing into ..
21:33:39 <fantasai> Jo: So objective of this group then is to produce a framework within which tests can be run and can be incorporated into other things
21:33:59 <fantasai> Jo: Next thing is what tests should be done by the end of hte year
21:34:09 <fantasai> Jo: Well, actually, we have a whole slew of tests that exist today
21:34:26 <andrewhubbs> andrewhubbs has joined #coremob
21:34:31 <fantasai> Jo: If we said what we want by the end of the year is what exists today, could be done
21:34:40 <fantasai> Jo: But have som notion of prioritization, want to influence things
21:34:56 <fantasai> Jo: to influence browser vendors, device manufacturers, and users
21:35:06 <fantasai> Jo: some tests in ringmark, and lots of tests in WG
21:35:11 <fantasai> Jo: But we have to do some gap analysis
21:35:18 <fantasai> Jo: All that is so clear so far
21:35:25 <fantasai> Jo: What is the framework these tests are to b eexecuted in?
21:35:42 <fantasai> Jo: Seems clear to me that there is only one option, and that is to use the existing W3C infrastructure
21:36:04 <fantasai> Jo: Sounds like doing that in tobie's output is not simple, but doable
21:36:21 <fantasai> Jo: So what we'll have by end of year, is a framework document that says what we're trying to do in some timeframe writ large
21:36:34 <fantasai> Jo: Then a prioritized list of features that goes into our initial test stuff
21:36:44 <fantasai> Jo: Won't be whole of HTML5, but HTML5 things that people find problematic
21:36:57 <fantasai> Jo: And then at least 1 visual representation of those results
21:37:14 <fantasai> Jo: If you don't like FB's version, can create your own!
21:37:21 <wesj> wesj has joined #coremob
21:37:26 <fantasai> Jo: So I think that's it. HOpe it made some kind of sense
21:37:43 <darobin> q?
21:38:06 <fantasai> Rob: So, you've got a test suite for all tests which is already capturing data
21:38:16 <fantasai> rob: Ringmark is a nice way of showing the data, and the idea is to combine the two?
21:38:19 <fantasai> Robin: Bit more than that
21:38:25 <fantasai> Robin: To summarize,
21:38:48 <fantasai> Robin: 1. Keep document for L1, it's the shopping list of what devs need today, and guidance for finding gaps
21:39:01 <fantasai> Robin: 2. Write a smaller document, list of things to test for 2013
21:39:12 <fantasai> Robin: that document will match the release of the test system
21:39:21 <fantasai> Robin: That test system would ideally be able to use tests in W3C databases
21:39:36 <fantasai> Robin: Talked with matt wrt separating Ringmark visual representation from running the tests
21:40:05 <fantasai> Robin: Could also compare the test results across browsers
21:40:18 <fantasai> Robin: Has advantage that nonautomated test results can be included
21:40:30 <fantasai> Vidya: I did not understand what you said.
21:41:04 <fantasai> Vidya: You said, Ringmark will do what it does today plus it will show me other stuff that's in the database about my browser?
21:41:13 <fantasai> Robin: I don't know if this was clear in earlier explanation
21:41:22 <fantasai> Robin: There is a W3C existing system on w3-test.org
21:41:31 <fantasai> Robin: Many of the tests by W3C WGs have been integrated
21:41:43 <fantasai> Robin: This contains a test runner, you can take your browser and run the tests
21:41:55 <fantasai> Robin: If the tests are automated, the results in your browser are automatically submitted
21:42:12 <fantasai> Robin: But for non-automated tests, the person looking at the test can say Pass/Fail/Can't Tell/etc
21:42:17 <fantasai> Robin: All that info is stored
21:42:33 <fantasai> Robin: So for all browsers we have stored data on pass/fail results on all these tests
21:42:39 <fantasai> Robin: You can query this data, it's in a database
21:42:54 <fantasai> Robin: Some of the things we want to test are not automatable, can't be used in Ringmark
21:43:13 <fantasai> Robin: But we can pull all that data and display it in a similar way to Ringmark
21:43:38 <jo> q?
21:43:43 <fantasai> ...
21:43:49 <jo> q+ tobie
21:43:53 <fantasai> Robin: The visual representation would be cleanly abstracted
21:44:04 <jo> zakim, open the queue
21:44:04 <Zakim> ok, jo, the speaker queue is open
21:44:05 <fantasai> Vidhya: The output here is what? Someone is going to define this api
21:44:12 <fantasai> Robin: That's up to FB
21:44:14 <jo> q+ tobie
21:44:19 <fantasai> Robin: need to talk about what we need to feed into it
21:44:52 <fantasai> Vidhya: I think the reality is that we see a lot of browsers that people out there don't see
21:45:04 <fantasai> Vidhya: We'll see them before they're commercial
21:45:13 <fantasai> Vidhya: Would be great to go in and see that
21:45:39 <tobie> ack tobie
21:45:52 <fantasai> Josh_Soref: Is there an action on you to fix the JSON to help people?
21:46:09 <fantasai> DanSun: So this team, or ringmark, is going to connect to test harness to get results?
21:46:25 <fantasai> mattkelly: Yes, the goal would be to integrate with the test harness
21:46:32 <fantasai> mattkelly: would need to make changes to do that, but that would be the goal
21:46:44 <fantasai> mattkelly: Ringmark would just be a results page, rather than a runner and a test suite and all that stuff
21:47:42 <fantasai> mattkelly: we would just sit on top of what the group produces
21:47:48 <fantasai> some confusion over test harness and testharness
21:47:53 <fantasai> Robin: We want 2 things
21:48:09 <fantasai> Robin: go to a page, and it tells you your browser sucks, what we have today
21:48:18 <fantasai> Robin: that would run the tests right there, automated tests
21:48:34 <fantasai> Robin: Other thing is to use the same visual component to get results from the W3C database (or some private database)
21:48:50 <fantasai> Robin: Idea is to produce multiple reports that are buzzword-compliant
21:49:04 <fantasai> DanSun: Two step process or one step?
21:49:14 <fantasai> DanSun: Run first in W3C harness, then Ringmark?
21:49:17 <bejram> bejram has joined #coremob
21:49:33 <fantasai> Robin: Depends. One thing will run those automated tests directly and show you your results
21:49:49 <fantasai> Robin: Other thing will pull data from W3C test database, that will be 2 step process
21:50:10 <fantasai> DanSun: Are there documents to run the tests?
21:50:23 <fantasai> Robin: Ideally it should be user friendly enough that you won't need documentation to run the tests
21:50:57 <fantasai> Jo: Note there isn't any one method of running tests, or one visual representation, we're just outlining what FB would like to achieve
21:51:09 <fantasai> Jo: If anyone wants to volunteer for something else, that's great.
21:52:21 <fantasai> Jo gives some history of the mobileOK testing 
21:52:59 <fantasai> Jo: Proposal is not to limit how reporting and test results happen, but just to make a start of it
21:53:24 <fantasai> Robin shows off
21:53:32 <fantasai> Robin: Let's imagine you want to run some tests
21:53:36 <marcos_lara> marcos_lara has joined #coremob
21:53:38 <fantasai> Robin: you go here, click on the button to run tests
21:54:05 <fantasai> Robin: It reports your UA, lets you choose which tests, and then starts running tests
21:54:28 <fantasai> Robin: Shows you the test with some buttons to choose the results, and some metadata about the test
21:54:52 <fantasai> Robin: The results you produce here, will appear in the results table
21:55:13 <fantasai> Robin shows off the table
21:55:26 <fantasai> Robin: The data used here you can have access to
21:56:03 <fantasai> Robin: in a JSON dump from the system
21:56:47 <fantasai> Robin: Are we all in agreement here?
21:56:56 <fantasai> s/Robin:/darobin:/g
21:57:07 <fantasai> Rob: The idea I was talking about was to create a short-term hit list
21:57:15 <fantasai> Rob: We can choose our own reporting and visualization
21:57:33 <fantasai> Rob: Everybody can take whatever data they like and show it off
21:57:38 <fantasai> Rob: But we can share the data
21:57:43 <fantasai> darobin: so long as there's a test suite
21:57:52 <fantasai> Rob: And we contribute our tests to the main W3C test suites, so it's valuable all around
21:58:04 <fantasai> Rob: And people can theoretically run private instances of this
21:58:13 <fantasai> tobie: and run the tests on their own devices, yes
21:59:27 <darobin> TAKEAWAY:
21:59:29 <darobin>     - target: end of year
21:59:30 <darobin>         - Level 1 document
21:59:32 <darobin>             - this is the aspirational documentation of what developers
21:59:33 <darobin>               need to produce applications today
21:59:35 <darobin>         - specific test suite nice and visual
21:59:36 <darobin>             - this is pretty, can run atop testharness.js
21:59:38 <darobin>         - document for the specific test suite
21:59:40 <darobin>             - this is the subset of the Level 1 document that describes
21:59:42 <darobin>               the interoperability hitlist that we are targeting for the
21:59:44 <darobin>               current test release
21:59:46 <darobin>     - refactoring Ringmark to be able to place the visual component
21:59:47 <darobin>       atop results from a test run, or stored runs
22:00:24 <darobin> PROPOSED RESOLUTION: the target for this group for EOY 2012 is the above summary
22:00:30 <Josh_Soref> this is the aspirational documentation of which apis are needed by developers to produce applications today
22:01:09 <Josh_Soref> fantasai: this CG is going to focus on which things need to be worked on
22:01:12 <Josh_Soref> ... by the end of the year
22:01:17 <Josh_Soref> darobin: mattkelly indicated he had 14 features
22:01:22 <Josh_Soref> ... in ringmark
22:01:30 <Josh_Soref> ... and those might be what we focus on
22:01:34 <Josh_Soref> ... or maybe we trim things out
22:03:34 <darobin> [the test bundle could be called Hit List Zero]
22:03:36 <darobin> RESOLUTION: the target for this group for EOY 2012 is the above summary   
22:05:43 <darobin> RESOLUTION: the primary input for Hit List Zero is the list of fourteen features currently focused upon by Ringmark
22:08:21 <darobin> ACTION: Tobie to make a fluffy picture out of the architecture described by Robin for the test system
22:08:21 <trackbot> Created ACTION-38 - Make a fluffy picture out of the architecture described by Robin for the test system [on Tobie Langel - due 2012-07-03].
22:10:31 <darobin> RESOLUTION: The group will not try to boil the ocean nor make a perfect system for the first release — which only care about rough consensus and running code
22:13:34 <darobin> ACTION: Robin to draft the architecture of the test system
22:13:34 <trackbot> Created ACTION-39 - Draft the architecture of the test system [on Robin Berjon - due 2012-07-03].
22:13:44 <Josh_Soref> Topic: Vendor Prefixes
22:13:50 <Josh_Soref> jo: No!!!!!!!!!!!!!!!!!
22:13:59 <Josh_Soref> darobin: I think we're done w/ Testing
22:14:03 <Josh_Soref> [ No ]
22:14:11 <Josh_Soref> darobin: we had vendor prefixes on the agenda
22:14:17 <Josh_Soref> ... we agreed as chairs to drop the discussion
22:14:21 <jo> jo has left #coremob
22:14:33 <Josh_Soref> ... the reason is that the proponent for text in that area isn't in attendance
22:14:34 <vgholkar> vgholkar has joined #coremob
22:14:43 <Josh_Soref> ... i think it's a solved problem in CSS WG
22:15:04 <Josh_Soref> jet: I think it becomes a topic for the last question
22:15:10 <Josh_Soref> ... will our tests have prefixes?
22:15:12 <Josh_Soref> darobin: they won't
22:15:22 <Josh_Soref> ... I believe the current opinion is that our tests won't have prefixes
22:15:25 <Josh_Soref> ... opinions mattkelly ?
22:15:29 <Josh_Soref> mattkelly: can of worms
22:15:35 <Josh_Soref> ... we want to strike two balances
22:15:42 <Josh_Soref> ... give ability for vendors to move quickly
22:15:44 <Josh_Soref> ... and implement things
22:15:57 <tobie> tobie has joined #coremob
22:15:57 <Josh_Soref> ... move fast
22:16:03 <Josh_Soref> ... prefixes introduce fragmentation
22:16:15 <Josh_Soref> ... for ringmark we thought about allowing prefixes but marking as yellow
22:16:31 <Josh_Soref> ... passing but non standard
22:16:38 <Josh_Soref> ... for developers, they just want the feature
22:16:49 <Josh_Soref> ... long term, there needs to be long term stigma if you continue to use prefixes
22:17:00 <Josh_Soref> ... we need to move quickly and get features in
22:17:05 <Josh_Soref> ... but also remove fragmentation
22:17:31 <Josh_Soref> darobin: anyone want to react to that?
22:17:49 <Josh_Soref> Josh_Soref: I think vendors who have employed prefixes shouldn't be punished for supporting prefixes
22:17:51 <Josh_Soref> ... in their code
22:17:58 <Josh_Soref> s/Josh_Soref:/jo:/
22:18:07 <Josh_Soref> ... but they shouldn't get credit for implementing the feature
22:18:16 <Josh_Soref> ... since they did what the CSS WG asked them to do
22:18:25 <Josh_Soref> ... what we should do is test for conformance to the spec as finally agreed
22:18:30 <Josh_Soref> Josh_Soref: +1
22:19:06 <Josh_Soref> Topic: Beyond Level 1
22:19:20 <Josh_Soref> darobin: tobie, you wanted to talk about your UCs and Reqs doc
22:19:26 <Josh_Soref> tobie: not really that ready
22:19:36 <Josh_Soref> ... i'm working on a document for UCs and Reqs for level 1
22:19:48 <Josh_Soref> ... i'm hoping to have something to share w/ the group in the near future
22:20:17 <Josh_Soref> ... i'm also going to bring UCs for AppConfig and Chromelessness
22:20:48 <Josh_Soref> Topic: QoI Testing
22:21:38 <Josh_Soref> darobin: we have a fairly clear plan for Conformance testing
22:21:40 <Josh_Soref> ... for QoI testing
22:21:46 <Josh_Soref> ... we have agreement that it's cool
22:21:52 <Josh_Soref> ... and ideas of what we would like to test
22:21:58 <Josh_Soref> ... but no commitment to producing tests
22:22:39 <Josh_Soref> jo: have we enough on our plate
22:22:45 <Josh_Soref> ... to do something in that area
22:22:47 <Josh_Soref> ... but not yet
22:22:51 <Josh_Soref> ... at least not before december
22:22:58 <Josh_Soref> darobin: i have too many Action items
22:23:07 <Josh_Soref> jo: that's largely my feeling
22:23:12 <Josh_Soref> ... absent volunteers
22:23:18 <Josh_Soref> ... i think it won't be worked on yet
22:23:30 <Josh_Soref> darobin: anytime someone feels like jumping into it
22:23:35 <Josh_Soref> ... we welcome that contribution
22:23:49 <Josh_Soref> mattkelly: we have a giant action item for a testrunner-testresults thing
22:24:02 <Josh_Soref> ... seems like we can do general compliance testing in parallel
22:24:23 <Josh_Soref> ... testing things like speed of canvas is highly important to goals of the group
22:24:29 <Josh_Soref> ... it feels like we should dip our toes in the water
22:24:42 <Josh_Soref> ... w/o perf tests on things like <canvas>
22:24:47 <Josh_Soref> ... even if we get a feature in
22:24:55 <Josh_Soref> ... if it's slow and crappy, it defeats the purpose
22:25:00 <Josh_Soref> darobin: <canvas> is the easy one to test
22:25:07 <Josh_Soref> mattkelly: 2d <canvas> perf
22:25:19 <Josh_Soref> ... should be something we could tackle by the end of the year
22:25:34 <Josh_Soref> jo: i think that's fairly generous of you to think of doing
22:25:41 <Josh_Soref> rob: for us, we aren't using <canvas>
22:25:51 <Josh_Soref> ... about our own ports
22:25:57 <Josh_Soref> ... by being able to cherrypick things
22:26:15 <Josh_Soref> ... we can use them to prove bugs to vendors
22:26:22 <Josh_Soref> ... but if we know there are multiple browsers failing
22:26:43 <Josh_Soref> ... then we know of places where we should suggest pain points for future devices
22:26:53 <Josh_Soref> ... but we can't do that until we can see where we are at the moment
22:27:08 <Josh_Soref> jo: that's a tentative offer of contributing something in the future
22:27:10 <Josh_Soref> rob: i think it's slightly firmer than that
22:27:19 <Josh_Soref> darobin: any other offers on QoI testing?
22:27:33 <Josh_Soref> http://arewefastyet.com
22:29:12 <fantasai> Josh_Soref: This is essentially a QoI test
22:29:19 <fantasai> Josh_Soref: comparse FF and Chrome with v8
22:29:29 <fantasai> s/parse/pares/
22:29:40 <fantasai> Josh_Soref: I don't actually use this thing, I just knows it exists
22:30:56 <fantasai> Mozilla rep explains the tests
22:31:13 <fantasai> which are used internally to monitor performance
22:31:23 <tobie> q+
22:31:53 <Josh_Soref> bkelley: it seems JS benchmarking has been done to death
22:31:59 <Josh_Soref> ... i think we should stay away from that
22:32:10 <Josh_Soref> ... unless there's something we can do that addresses a UC more directly
22:32:21 <Josh_Soref> ... maybe a physics computation benchmark
22:32:35 <Josh_Soref> ... just stealing + rebranding won't add value
22:32:44 <Josh_Soref> jo: i feel inspired
22:33:03 <tobie> ack tobie
22:33:33 <ytsai> ytsai has left #coremob
22:34:19 <darobin> RESOLUTION: For QoI testing, we're open to input, but we won't move on it before someone proposes something specific (FT & FB have tentatively suggested they might think about it)
22:34:34 <Josh_Soref> Topic: Wrap
22:34:44 <Josh_Soref> jo: AOB
22:34:47 <Josh_Soref> darobin: is there AOB?
22:34:51 <Josh_Soref> ... next F2F?
22:35:00 <Josh_Soref> jo: proposal for group telecoms?
22:35:16 <Josh_Soref> ... darobin isn't enamored of the idea
22:35:21 <Josh_Soref> ... i'd like to try it
22:35:39 <Josh_Soref> ... meetings are difficult to coordinate based on time zones
22:35:53 <Josh_Soref> rob: could we try dual-location F2F?
22:36:03 <Josh_Soref> darobin: jo was talking about Phone Bridges
22:36:18 <Josh_Soref> ... separately to plan a single location F2F
22:36:23 <Josh_Soref> ... probably close to London
22:36:38 <Josh_Soref> jo: if not @Orange, perhaps @FT
22:36:50 <Josh_Soref> darobin: we know others in London, perhaps @Vodafone
22:36:57 <Josh_Soref> q+ jfmoy 
22:37:03 <Josh_Soref> ack jfmoy 
22:37:12 <Josh_Soref> jfmoy: I'll try to do my best if we can host
22:37:21 <Josh_Soref> ... but if it's more people than today, that'll be tough in London
22:37:24 <Josh_Soref> ... 40 people Max
22:37:30 <Josh_Soref> ... i need to check
22:37:48 <Josh_Soref> ... if we had to do it in Orange, we could do it in Paris
22:37:54 <Josh_Soref> ... I prefer London
22:37:58 <Josh_Soref> ... but we have more space in Paris
22:38:17 <Josh_Soref> ACTION jfmoy check on hosting @Orange Oct 2-3
22:38:17 <trackbot> Sorry, couldn't find user - jfmoy
22:38:28 <Josh_Soref> ACTION moy check on hosting @Orange Oct 2-3, in London (alt Paris)
22:38:28 <trackbot> Created ACTION-40 - Check on hosting @Orange Oct 2-3, in London (alt Paris) [on Jean-Francois Moy - due 2012-07-03].
22:38:38 <Josh_Soref> s/ACTION moy/ACTION: moy/
22:39:01 <darobin> ACTION: Jo to figure out teleconference logistics, timing, and critical mass
22:39:01 <trackbot> Created ACTION-41 - Figure out teleconference logistics, timing, and critical mass [on Jo Rabin - due 2012-07-03].
22:39:10 <Josh_Soref> jo: AOB?
22:39:14 <Josh_Soref> [ None ]
22:39:19 <Josh_Soref> darobin: many thanks to everyone for coming
22:39:31 <Josh_Soref> ... special thanks for Josh_Soref and fantasai (who got dragged in) for scribing
22:39:34 <Josh_Soref> [ Applause ]
22:39:35 <lbolstad> lbolstad has left #coremob
22:39:49 <Josh_Soref> darobin: thanks to FB for hosting in this cool location with great logistics
22:39:52 <Wonsuk> Wonsuk has left #coremob
22:39:53 <Josh_Soref> [ Applause ]
22:40:10 <Josh_Soref> Josh_Soref: thanks for calling in lgombos 
22:40:17 <darobin> RESOLUTION: The CG thanks Facebook for great organisation, location, and logistics
22:40:32 <darobin> RESOLUTION: The CG thanks Josh and fantasai for their outstanding scribing
22:40:57 <Josh_Soref> tobie: thanks to the chairs
22:40:59 <Josh_Soref> [ Applause ]
22:41:03 <Josh_Soref> RRSAgent, draft minutes
22:41:03 <RRSAgent> I have made the request to generate http://www.w3.org/2012/06/26-coremob-minutes.html Josh_Soref
22:41:41 <Josh_Soref> trackbot, end meeting
22:41:41 <trackbot> Zakim, list attendees
22:41:41 <Zakim> sorry, trackbot, I don't know what conference this is
22:41:49 <trackbot> RRSAgent, please draft minutes
22:41:49 <RRSAgent> I have made the request to generate http://www.w3.org/2012/06/26-coremob-minutes.html trackbot
22:41:50 <trackbot> RRSAgent, bye
22:41:50 <RRSAgent> I see 18 open action items saved in http://www.w3.org/2012/06/26-coremob-actions.rdf :
22:41:50 <RRSAgent> ACTION: Tobie to provide numbers for required sprites/fps in games [1]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T17-21-45
22:41:50 <RRSAgent> ACTION: mattkelly to document JSGameBench and the approach behind it [2]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T17-30-26-1
22:41:50 <RRSAgent> ACTION: Shilston to expeditiously check whether it is practical to measure consistency of framerate [3]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T17-51-58
22:41:50 <RRSAgent> ACTION: Josh to survey people and compile a list of common errors in test writing [4]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T18-54-13
22:41:50 <RRSAgent> ACTION: Robin to write documentation for testharness.js [5]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T18-54-33
22:41:50 <RRSAgent> ACTION: timeless to survey people and compile a list of common errors in test writing [6]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T18-54-40
22:41:50 <RRSAgent> ACTION: Soref to survey people and compile a list of common errors in test writing [7]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T18-54-44
22:41:50 <RRSAgent> ACTION: Matt to remove the dependency on Node to get Ringmark running, and help make it easier to set up [8]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T19-08-04
22:41:50 <RRSAgent> ACTION: Robin to look into something like jsFiddle for test writing [9]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T19-10-56
22:41:50 <RRSAgent> ACTION: Moy to provide requirements for an automated test runner of all tests [10]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T19-14-15
22:41:50 <RRSAgent> ACTION: matt to document JSGameBench and the approach behind it [11]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T19-19-27
22:41:50 <RRSAgent> ACTION: matt to talk to OEMs/carriers about what they would most usefully need to get out of Ringmark results [12]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T19-20-35
22:41:50 <RRSAgent> ACTION: tobie to carry out a gap analysis of existing W3C test suites [13]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T19-32-19
22:41:50 <RRSAgent> ACTION: Robin to draft a test suite release strategy based on what fantasai and Josh_Soref described [14]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T19-52-23
22:41:50 <RRSAgent> ACTION: Robin to assess which existing test suites can be reused and at what level of coverage they stand [15]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T20-15-37
22:41:50 <RRSAgent> ACTION: Tobie to make a fluffy picture out of the architecture described by Robin for the test system [16]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T22-08-21
22:41:50 <RRSAgent> ACTION: Robin to draft the architecture of the test system [17]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T22-13-34
22:41:50 <RRSAgent> ACTION: Jo to figure out teleconference logistics, timing, and critical mass [18]
22:41:50 <RRSAgent>   recorded in http://www.w3.org/2012/06/26-coremob-irc#T22-39-01
22:43:24 <ming> ming has left #coremob