IRC log of testing on 2012-10-31

Timestamps are in UTC.

10:03:57 [RRSAgent]
RRSAgent has joined #testing
10:03:57 [RRSAgent]
logging to http://www.w3.org/2012/10/31-testing-irc
10:04:10 [ArtB]
Scribe: NotMe
10:04:17 [ArtB]
ScribeNick: uh
10:04:25 [ArtB]
Date: Halloween
10:04:42 [ArtB]
Meeting: #testing gathering @ TPAC 2012
10:04:46 [ArtB]
Chair: PLH
10:04:53 [plinss]
plinss has joined #testing
10:05:10 [AnssiK]
AnssiK has joined #testing
10:05:15 [fantasai]
fantasai has joined #testing
10:05:28 [plh]
plh has joined #testing
10:05:46 [fantasai]
plh: I'm in the digital publishing room, if you think you need me, feel free to pull me
10:05:52 [ArtB]
Present+ Art_Barstow
10:06:00 [krisk]
krisk has joined #testing
10:06:12 [adambe]
adambe has joined #testing
10:06:45 [ArtB]
https://github.com/w3c/testing-how-to
10:07:39 [Wonsuk]
Wonsuk has joined #testing
10:07:49 [ArtB]
plh: there are two other testing related sessions today
10:07:58 [ArtB]
http://www.w3.org/wiki/TPAC2012#Session_Grid
10:08:55 [krisk]
I can scribe...
10:09:02 [ArtB]
scribenick: krisk
10:09:02 [SimonPieters]
SimonPieters has joined #testing
10:09:10 [Marcos]
Marcos has joined #testing
10:09:14 [caribou]
caribou has joined #testing
10:09:18 [bryan]
bryan has joined #testing
10:09:18 [ArtB]
Scribe: Kris
10:09:19 [arronei]
arronei has joined #testing
10:09:26 [giuseppe]
giuseppe has joined #testing
10:09:26 [ArtB]
RRSAgent, make minutes
10:09:26 [RRSAgent]
I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB
10:09:26 [AnssiK]
Present+ Anssi_Kostiainen
10:09:29 [jet]
jet has joined #testing
10:09:35 [bryan]
bryan has left #testing
10:09:48 [plh]
http://w3c.github.com/testing-how-to/
10:09:49 [ArtB]
RRSAgent, make log Public
10:09:52 [jgraham]
10:09:56 [ArtB]
RRSAgent, make minutes
10:09:56 [RRSAgent]
I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB
10:10:20 [sms]
present + Simon Stewart
10:10:21 [krisk]
present+ kris_krueger
10:10:22 [rhauck]
present +rhauck
10:10:23 [darobin]
darobin has joined #testing
10:10:24 [giuseppe]
present+ giuseppe
10:10:25 [Wonsuk]
Present+ Wonsuk_Lee
10:10:26 [jet]
present + Jet Villegas
10:10:26 [plinss]
Present+ Peter Linss
10:10:28 [bryan]
bryan has joined #testing
10:10:35 [jgraham]
present+ jgraham
10:10:35 [lmclister]
present + Larry McLister
10:10:35 [SimonPieters]
Present+ Simon_Pieters
10:10:36 [JohnJansen]
present+ JohnJansen
10:10:38 [adambe]
present+ adambe
10:10:39 [wilhelm]
present+ Wilhelm Joys Andersen
10:10:42 [bryan]
present+ Bryan_Sullivan
10:10:43 [Mark_Vickers]
=present+ Mark_Vickers
10:10:44 [arronei]
present +Arron Eicholz
10:11:10 [krisk]
plh talking about overview - starting with 'what do you test in a specification?'
10:11:15 [Zakim]
Zakim has joined #testing
10:11:20 [ht]
ht has joined #testing
10:11:40 [JohnJansen]
JohnJansen has joined #testing
10:11:57 [jet]
jet has changed the topic to: TPAC testing sessions
10:11:59 [darobin]
Present+ Robin Berjon
10:12:08 [ht]
present+ Henry S. Thompson
10:12:14 [JohnJansen]
present+ JohnJansen
10:12:15 [krisk]
jgraham - the #github testing talk will talk about improvements in our test infrastructure, requirements for tests, etc..
10:12:24 [Marcos]
Present+ Marcos Caceres
10:12:31 [ArtB]
RRSAgent, make minutes
10:12:31 [RRSAgent]
I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB
10:12:32 [MichaelC_]
MichaelC_ has joined #testing
10:12:33 [krisk]
plh - "what do you test in a specification?"
10:12:40 [MichaelC_]
present+ Michael_Cooper
10:12:51 [robin]
robin has joined #TESTING
10:13:03 [krisk]
plh - it's always not obvious what needs to be testing, for example webidl exists that needs to be tested
10:13:06 [acolwell]
acolwell has joined #testing
10:13:20 [krisk]
plh - if it's going to have an impact on implementations needs to be tested
10:13:43 [krisk]
plh - MUST, MUST NOT, SHOULD , SHOULD NOT, MAY are all items that should be tested
10:13:44 [kotakagi]
kotakagi has joined #testing
10:14:38 [Marcos]
Lachy: just pinged him
10:14:46 [Marcos]
Lachy: what room are you in?
10:14:50 [krisk]
plh - mutliple tests can be created for a single statement from the spec
10:14:56 [Lachy]
Marcos: RHONE 2
10:15:18 [jeff_]
jeff_ has joined #testing
10:15:19 [krisk]
plh - it's complex and not automated....
10:15:33 [krisk]
Part #2 for the talk "How to write a test?"
10:15:52 [krisk]
First mechanism is javascript test
10:16:06 [krisk]
second test mechaism is what is known as a REF test
10:16:19 [krisk]
...used for rendering tests
10:16:35 [krisk]
third mechanism is 'seff-describing' tests
10:16:58 [krisk]
..you don't want to do this by default, since it's expensive
10:17:09 [krisk]
...though at times this is the only mechanism
10:17:22 [MikeSmith]
MikeSmith has joined #testing
10:17:31 [MikeSmith_]
MikeSmith_ has joined #testing
10:17:59 [krisk]
For script based tests we use testharness.js
10:18:06 [ArtB]
http://w3c-test.org/resources/
10:18:15 [JohnJansen]
s/seff-describing/self-describing
10:18:25 [krisk]
Check out the api examples...
10:18:39 [krisk]
http://w3c-test.org/resources/apisample.htm
10:18:51 [ArtB]
http://darobin.github.com/test-harness-tutorial/docs/using-testharness.html (testharness.js tutorial by Robin Berjon)
10:18:54 [krisk]
http://w3c-test.org/resources/apisample2.htm
10:18:56 [krisk]
http://w3c-test.org/resources/apisample3.htm
10:19:55 [tobie]
present+ Tobie_Langel
10:20:09 [krisk]
Two main use cases - asynchronous (e.g. events aka onload) and synchronous
10:20:32 [krisk]
..a number of assert_* exist to validate the results
10:20:45 [krisk]
plh showing the test examples for each...
10:22:32 [krisk]
some cool asserts...
10:22:35 [glenn]
glenn has joined #testing
10:22:50 [krisk]
assert_unreachable - makes sure an event DOESN'T fire
10:22:51 [sms]
wonder what approx_equals does
10:23:16 [krisk]
assert_readonly, assert_idl_attribute!
10:23:39 [krisk]
assert_array is also very helpful as wel...
10:23:46 [krisk]
s/wel/well/
10:24:04 [sms]
krisk: how do you assert the negative?
10:24:14 [krisk]
assert_false
10:24:15 [sms]
Just assume it doesn't fire within a certain amount of time?
10:24:24 [sms]
Sorry: "assert_unreachable"
10:25:40 [krisk]
you can timeout at the 'page' level and at the test level (you can have more than one test per page)
10:26:28 [krisk]
..adding metadata to the test is very good when you want to get an understanding on what parts of the spec have coverage
10:27:06 [krisk]
see slide 18 from the deck on https://github.com/w3c/testing-how-to
10:27:17 [krisk]
..now on slide 19...
10:27:26 [krisk]
..what are the pitfalls?
10:27:47 [krisk]
It important that we get the same number of test pass/fails
10:28:06 [krisk]
DO NOT stop running tests because something is not supported
10:28:30 [SimonPieters]
documentation in the source: http://w3c-test.org/resources/testharness.js
10:28:58 [krisk]
PLH see slide #20 if you can follow along just see the link above
10:29:04 [krisk]
http://darobin.github.com/test-harness-tutorial/docs/using-testharness.html
10:29:38 [krisk]
Now on Reftest
10:29:43 [krisk]
what is a reftest?
10:29:56 [krisk]
It's two pages that should result in the same rendering
10:29:58 [jet]
are there docs on how each browser runs reftests?
10:30:58 [simonstewart]
http://www.w3.org/TR/webdriver/
10:31:17 [simonstewart]
http://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-spec.html
10:31:45 [tpacbot]
tpacbot has joined #testing
10:31:56 [jgraham]
jet: I dont think we have docs, but our system is a little different to yours
10:32:03 [jgraham]
We can talk about it
10:32:16 [jet]
jgraham: thx
10:33:27 [krisk]
The test web server has php support, so you can add a delay or headers, etc..
10:35:14 [hta]
hta has joined #testing
10:35:29 [krisk]
We have media and media files as well that can be used...
10:35:41 [krisk]
we have codec agnostic support files as well
10:35:51 [jalvinen]
jalvinen has joined #testing
10:39:50 [krisk]
W3C Test Suite Licenses
10:40:09 [JohnJansen]
JohnJansen has joined #testing
10:40:17 [ArtB]
RRSAgent, make minutes
10:40:17 [RRSAgent]
I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB
10:40:44 [krisk]
See slide #33
10:41:15 [krisk]
Now on to Mark Vickers
10:41:20 [Mark_Vickers]
http://lists.w3.org/Archives/Public/public-web-and-tv/2012Oct/0017.html
10:41:31 [krisk]
Who will talk abou the Web and TV Testing Task Force
10:42:04 [krisk]
The recharter is going to be updated so that this WG has a Testing Task Force
10:42:57 [krisk]
Testing Use Cases
10:43:16 [krisk]
#1 Currently Verification of the specification
10:43:40 [krisk]
#2 New Improve the consistency of the open web platform (OWP)
10:44:14 [krisk]
#3 New support external testing/certification organizations
10:45:24 [krisk]
Number of group do this today - for example DLNA
10:45:49 [krisk]
When it comes from HTML5 we want to have these tests come from the w3c and not 'fork' the tests
10:46:18 [krisk]
#4 Support testing devices
10:46:54 [seo]
seo has joined #testing
10:47:13 [acolwell]
acolwell has joined #testing
10:47:27 [sungok_you__]
sungok_you__ has joined #testing
10:56:24 [krisk]
summary of this session!
10:56:44 [krisk]
tobin can you give summary?
10:56:57 [krisk]
One person states - no testing is enough
10:57:17 [rhauck]
uh, no "amount" of testing is enough?
10:58:31 [Wonsuk]
Wonsuk has left #testing
10:58:54 [ArtB]
RRSAgent, make minutes
10:58:54 [RRSAgent]
I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB
11:00:08 [jalvinen]
jalvinen has left #testing
11:01:07 [shepazu]
shepazu has joined #testing
11:04:01 [giuseppe]
giuseppe has left #testing
11:04:19 [ArtB]
RRSAgent, bye
11:04:19 [RRSAgent]
I see no action items
13:35:46 [RRSAgent]
RRSAgent has joined #testing
13:35:46 [RRSAgent]
logging to http://www.w3.org/2012/10/31-testing-irc
13:35:51 [Mark_Vickers]
Mark_Vickers has left #testing
13:35:51 [ph]
ph has joined #testing
13:35:53 [Mark_Vickers]
Mark_Vickers has joined #testing
13:35:58 [Abel]
Abel has joined #testing
13:36:01 [divya]
divya has joined #testing
13:36:04 [ArtB]
Title: #github session @ TPAC 2012
13:36:10 [ArtB]
Date: Halloween
13:36:18 [ArtB]
Chair: James Graham
13:36:25 [plh]
plh has joined #testing
13:36:26 [bryan]
bryan has joined #testing
13:36:26 [divya]
[missed a bit]
13:36:27 [ArtB]
ScribeNick: divya
13:36:34 [ArtB]
Scribe: Divya
13:36:34 [Lachy]
Lachy has joined #testing
13:36:39 [divya]
Starting with Source control which was the original motivation for this session
13:36:41 [Mark_Vickers]
present+ Mark_Vickers
13:36:50 [Lachy]
Lachy has joined #testing
13:36:51 [divya]
jgraham: atm we are running custom w3c instance of mercurial
13:36:54 [ArtB]
present+ Art_Barstow
13:37:25 [divya]
…we ahve all these repos, in order to contribute tests to that you need to learn how to use mercurial, find the repo, add your tests, commit & send a mail to mailing list
13:37:42 [divya]
RRSAgent: make logs public
13:37:58 [divya]
…one suggestion has come up is if we are better off outsourcing part of problem, in particular github
13:38:20 [ArtB]
(http mirror for webapps' test repo: http://w3c-test.org/webapps/)
13:38:23 [divya]
…its git whichi is very similar to mercuirial hosted service nicer UI, and has lot of developer buy-in
13:38:32 [divya]
…all big libraries are hosted on Github
13:38:42 [divya]
…has a different workflow which is slightly better
13:38:53 [isra]
isra has joined #testing
13:39:02 [darobin]
darobin has joined #testing
13:39:05 [divya]
[shows example of testharness.js] http://github.com/w3c/testharness.js
13:39:19 [divya]
…they have this mechanism called pull request instead of sending mail off to a mailing list
13:39:34 [divya]
…it gives you a way of tracking various people who want to make changes
13:39:43 [divya]
…you do not have to go out to mail, mailing list and different systems
13:40:06 [divya]
…do sy yhid poiny it would be interesting to have a discussion about whether people think this is a good idea.
13:40:14 [divya]
s/so at this point
13:40:33 [divya]
kris: there has this whole patent policy stuff that needs to be addressed
13:40:36 [ArtB]
q+
13:40:53 [divya]
darobin: 1st there is contributions to spec, chair makes a call on whether the contributions has done IP properly
13:41:25 [divya]
darobin: separate from contributions to test suite. they are fairly standard open source license and you can get anyone contributing to test suite without IP consequences for the royalty free aspect ot the spec
13:41:38 [divya]
kris: do we make people sign to this to contribute to the spec.
13:41:48 [divya]
kris: cable companies think there would be a lot of money in it.
13:42:02 [divya]
darobin: W3C is not allowed to publish test suites in anything other htan open source license
13:42:12 [divya]
kris: today you cant add tests to w3c without signing some stuff
13:42:24 [divya]
darobin: that is manageable, you just need a list of people who signed off.
13:42:33 [divya]
darobin: i do not see it as an issue.
13:42:34 [Mark_Vickers]
q+
13:42:36 [darobin]
q?
13:42:51 [Zakim]
Zakim has joined #testing
13:42:53 [Marcos]
Marcos has joined #testing
13:42:58 [darobin]
q+ ArtB
13:43:00 [masinter]
masinter has joined #testing
13:43:00 [Marcos1]
Marcos1 has joined #testing
13:43:03 [darobin]
q+ Mark_Vickers
13:43:06 [masinter]
q?
13:43:11 [darobin]
ack ArtB
13:43:20 [rhauck]
rhauck has joined #testing
13:43:20 [divya]
ArtB: i am indifferent, beggers cant be choosy might dominate here. by we i mean webapps anyhow. we will take tests
13:43:27 [divya]
ArtB: we should provide a way for them to contribute
13:43:47 [divya]
ArtB: if we are chasing another shiny object. in this case github. i am not convinced what we have right now is broken enough.
13:44:05 [divya]
darobin: how often do you use the mercurial system
13:44:09 [divya]
ArtB: probably a few times permonth
13:44:17 [divya]
darobin: i think that maybe why you do not see the problem
13:44:44 [divya]
kris: i dont believe everyone moved ot github, we still have stuff coming out of CVS. we have to learn multiple systems
13:44:49 [divya]
kris: that is a valid thing
13:44:59 [divya]
darobin: if we want outside contributions we can give up on using mercurial.
13:45:10 [divya]
kris: i do not think it is really very inviting to contribute.
13:45:35 [divya]
kris: even geeks have a hard time to contribute to github(?)
13:45:47 [divya]
jgraham: i think robin said on wiki page, it is about github not git
13:46:28 [divya]
jgraham: the point is because there is an existing community around github and much broader than w3c, the existing pool of users who know what the workflow is, then they make pull requests.
13:46:41 [divya]
darobin: in developer world, if it does not exist on github, it does not exist.
13:47:08 [divya]
Marcos: i have seen this in responsive images. amazing to get that workflow, getting the intial flow is hard, once you ahve the community happening, it is hard.
13:47:52 [divya]
marcos: what our community here does is important. are they comfortable going through this. it is not as tedious as doing the manual merge on the command line. github does this automatically, and transparent. commenting appears real time.
13:48:23 [divya]
darobin: the way we use mercurial in w3c setup is not correct. we have central repo for something that is supposed to be distributed, we are using it as cvs and it is easy to destroy content this way.
13:48:52 [divya]
[random discussion]
13:48:53 [ArtB]
q?
13:48:53 [darobin]
q?
13:48:57 [darobin]
ack Mark_Vickers
13:49:21 [divya]
Mark_Vickers: on the legal thing, my understanding is that github repo is you can put a CLA that covers a particular project.
13:49:26 [AnssiK]
AnssiK has joined #testing
13:49:36 [plinss]
q+
13:49:38 [divya]
darobin: there is an approval stamp the 1st time you contribute.
13:49:58 [divya]
Mark_Vickers: it has value to have a large number of tests in terms of saving money for peoople who create content.
13:50:02 [MikeSmith]
btw the way dvcs.w3.org is set up right now it's possible for a push to create multiple heads
13:50:06 [divya]
Mark_Vickers: not interms of saving money.
13:50:11 [MikeSmith]
q?
13:50:16 [divya]
kris: i dont think we can say it is all free and open.
13:51:09 [darobin]
ack plinss
13:51:12 [divya]
jgraham: you can see who the contribution is coming from, so you can say if they should fill the form or not. it is no different than when people send us a patch via email, and that you commit the patch without checking if they signe the right agreement
13:51:45 [divya]
plinss: i am in favour of crowdsourcing the tests, i certainly get the fact that github does that with devs, for some definition of devs. The devs that github targets are they contributing to test suites.
13:52:07 [divya]
plinss: the lot of folks who show up at test the web forward are designers not coders.
13:52:28 [divya]
darobin: my experience of test the web forward is paris. They tried to use dropbox, it didnt work and then they switched to github.
13:52:45 [divya]
stearns: at SF #1 question was why we are not using github.
13:53:06 [divya]
lmclister: or what is the mercurial equivalent for the git command
13:53:09 [masinter]
q+ to ask whether this is necessarily a choice, or whether there's some workflow where things can be staged
13:53:19 [divya]
plinss: i am concerned like ArtB that we are chasing the next shiny thing.
13:53:35 [divya]
plinss: i would rather see w3c be the cool place for this stuff.
13:53:44 [divya]
marcos: cost of setting up infrastructure is huge.
13:53:59 [MikeSmith]
I wish somebody from the w3c systems team were here to comment
13:54:00 [divya]
Marcos1: how many years do we keep doing the same thing.
13:54:09 [divya]
plinss: i have snever seen a migration step not lose something
13:54:20 [divya]
plinss: i have never seen a pre-baked thing that delivers to all of our needs.
13:54:26 [krisk]
q+
13:54:26 [divya]
darobin: we may have fewer points of pain
13:54:35 [divya]
plinss: something else might be the point of pain
13:54:37 [darobin]
q?
13:54:45 [divya]
jgraham: i am not claiming github is perfect at least it has an api that we can build on
13:54:55 [odinho_]
q+
13:55:01 [divya]
…if it does turn out it is not optimal we can just replace that bit of it.
13:55:08 [divya]
plinss: what if github goes down
13:55:14 [ArtB]
q+ how would this work http://w3c-test.org/framework/app/suite in the github world?
13:55:20 [divya]
jgraham: certainly we should have w3c hosted read only clones of repo
13:55:35 [jet_]
jet_ has changed the topic to: github for w3c tests
13:55:36 [divya]
plinss: github has the apis to developp tools, if we spend times building tools on top of infrastructure
13:56:09 [divya]
jgraham: someone at opera wrote a code review tool that was open sourced on monday. I think it is orders of magnitude better than github one, and it might be worth looking at that.
13:56:19 [darobin]
q?
13:56:21 [odinho_]
https://github.com/jensl/critic/ -> Critic code review tool (from Opera)
13:56:29 [divya]
…might be worth making it transparent such that we can use that tool rather than whats on github
13:56:41 [divya]
[missed convo]
13:56:43 [masinter]
q-
13:56:53 [odinho_]
s|https://github.com/jensl/critic/">https://github.com/jensl/critic/ -> Critic code review tool (from Opera)|-> https://github.com/jensl/critic/ Critic code review tool (from Opera)|
13:56:59 [divya]
jgraham: there may be different design goals
13:57:10 [divya]
plinss: shepherd was designed to be a test review system
13:57:23 [divya]
jgraham: if it gets us an awesome user experience then, but it is not there yet.
13:57:25 [ArtB]
q+ to ask: how would this work http://w3c-test.org/framework/app/suite in the github world?
13:57:29 [divya]
plinss: anything we build is going to take time
13:57:36 [divya]
plinss: rather than rely on someone else's infrastructure
13:57:45 [divya]
plinss: i have no predictability on github infrastructure
13:57:57 [divya]
darobin: as long as the people who wrote the code are alive & around to maintain it
13:57:59 [isra]
q+
13:58:08 [divya]
kris: stuff on w3c lasts longer than it lasts on github
13:58:21 [divya]
darobin: 10 years ago sourceforce was the only option and people hated it.
13:58:35 [divya]
darobin: anything out there would be synchronized tow3c
13:58:52 [divya]
darobin: 1. one side is git 2. other side of tooling is github api, which maps very closely to git.
13:59:13 [divya]
darobin: i am not too worried about that, there exists at least 1 open source implementation that exposes same api on top of git repository
13:59:15 [odinho_]
q-
13:59:31 [divya]
krisk: how long have we had stuff on this site so far?
13:59:34 [divya]
jgraham: which site?
13:59:36 [divya]
krisk: github
13:59:41 [odinho_]
Present+ Odin_Hoerthe_Omdal
13:59:51 [divya]
jgraham: well the htmlwg is readwrite. that is what they are using as primary interface
13:59:59 [divya]
jgraham: test stuff is mostly readonly. the html test suite is readonly.
14:00:05 [divya]
jgraham: testharness.js is readonly here.
14:00:19 [jdurand]
jdurand has joined #testing
14:00:20 [divya]
…webperforamnce is read only event source is read-write doesnt exist anywhere else.
14:00:29 [divya]
…amaya is readwrite nowadays
14:00:48 [divya]
…this is the official w3c github repo there is a lot more w3c related github things that are not part of the official one
14:00:58 [divya]
darobin: there is actually a lot more content than this.
14:01:07 [divya]
krisk: isnt that a problem? wont people have hard time giguring out
14:01:17 [divya]
darobin: ideally i want all the official stuff to go to the official thing.
14:01:32 [ArtB]
RRSAgent, make minutes
14:01:32 [RRSAgent]
I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB
14:01:36 [divya]
darobin: i would encourage groups be consistent and make their info more easy to find.
14:01:43 [bryan]
q+
14:01:51 [divya]
jgraham: it is a problem we would solve if we say this is the prefered user interface
14:02:04 [MichaelC_]
MichaelC_ has joined #testing
14:02:23 [divya]
darobin: what has happened today is people join WG they are told mercurial is the way to do, lose data once, and hten they get rustrated and they do not know w3c has a github account.
14:02:27 [darobin]
q?
14:02:30 [darobin]
ack krisk
14:03:02 [divya]
Marcos1: i am one of the people who has test suites in public account not on w3cc account. It looked like i didnt know who to contact.
14:03:09 [divya]
jgraham: the process at the moement is to ask MikeSmith
14:03:11 [divya]
darobin: or me
14:03:33 [divya]
darobin: if you look at the members anyone there who is w3c should normally have the ability to add you to the organization
14:03:49 [darobin]
ack ArtB
14:03:50 [Zakim]
ArtB, you wanted to ask: how would this work http://w3c-test.org/framework/app/suite in the github world?
14:03:53 [ArtB]
http://w3c-test.org/framework/app/suite
14:04:02 [divya]
ArtB: i am oppossed to [] work.
14:04:25 [divya]
ArtB: we have some tests in webapps. in the scenario where webapps would have some tests in mercurial etc
14:04:34 [divya]
ArtB: would framework accomodate both?
14:04:37 [divya]
darobin: not a problem
14:04:48 [MikeSmith]
s/[] work/makework/
14:04:54 [MikeSmith]
RRSAgent, make minutes
14:04:54 [RRSAgent]
I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html MikeSmith
14:05:01 [divya]
ArtB: do you have WG that are like ground zero, that have already gone to github
14:05:08 [divya]
darobin: the htmlwg did that for spec editing
14:05:19 [divya]
Marcos1: DAP did it
14:05:33 [divya]
krisk: is it not on CVS for html spec?
14:05:45 [wilhelm]
q+
14:05:50 [divya]
darobin: there is a tool that takes the github source, compiles into html thing and then checks into cvs
14:06:00 [divya]
…sam wrote the tool in under 15 mins
14:06:18 [divya]
krisk: if someone starts from ground zero, we have other infrastructure how does w3test.org work
14:06:36 [divya]
darobin: i maintain respec on github, whenever i ship a build, it automatically gets synched to w3c server
14:06:52 [divya]
jgraham: w3test.org does perl update at the moment changing to git pull owuld be trivial
14:07:00 [divya]
SimonPieters: whatwg specs are developed on github.
14:07:09 [darobin]
q?
14:07:11 [darobin]
ack isra
14:07:40 [divya]
isra: should w3c ever decide to use github, i think we should protect our members find some sort of legal protection in case they change user agreements, what happens when they change agreements.
14:07:48 [divya]
isra: it is a fertile ground for some issues to come up.
14:08:00 [divya]
krisk: i would +1 that, for microsoft, it is not a normal thing.
14:08:07 [divya]
darobin: plenty of ms people put stuff on github
14:08:13 [divya]
krisk: there are some things in place that is different tho
14:08:41 [divya]
isra: w3c has enough power to impose some conditions.
14:08:49 [divya]
darobin: disappearing data is not a problem
14:09:06 [divya]
darobin: you are thinking the case what happens when they become evil.
14:09:12 [divya]
masinter: if you can sync why is there an issue
14:09:41 [divya]
plinss: it is not about the repo itself. the issue is tooling around the repo, we will be married to what we are building on top of
14:09:50 [divya]
masinter: github has an api.
14:09:58 [divya]
plinss: dont confuse git and github and mercurial
14:10:08 [divya]
isra: its about legal terms of use
14:10:12 [divya]
isra: not about synching data
14:10:19 [divya]
[side conversations]
14:11:00 [divya]
masinter: most tools seem to be for running tests than submitting them
14:11:04 [divya]
plinss: in csswg we have both
14:11:08 [SimonPieters]
q?
14:11:28 [divya]
plinss: we have sheperd code it runs, finding errors giving data right back into client.
14:11:48 [divya]
plinss: comments reviews happen on shpeherd. if push comes to shove i can move those to github.
14:11:53 [darobin]
q?
14:12:15 [divya]
…so i dont defend the code per se, there is a lot of sense that the methodology runs on same server
14:12:24 [divya]
…validates test files, etc that will soon move to CI system
14:12:34 [divya]
…there is lot that is either built or not too far away from being built
14:12:38 [divya]
masinter: based on mercurial?
14:12:41 [divya]
plinss: yes
14:13:05 [davidburns]
davidburns has joined #testing
14:13:08 [divya]
plinss: at that time w3c was doing to svg, and then it appeared like w3c was going to standardize on mercurial
14:13:18 [divya]
darobin: it is not w3c the orgn it is just feedback the people
14:13:41 [darobin]
q?
14:13:46 [darobin]
ack bryan
14:13:55 [divya]
bryan: i am using github to ocllaboratively edit specs
14:13:58 [odinho_]
s/doing to svg/using svn/
14:14:02 [divya]
bryan: should that be under w3c account
14:14:17 [odinho_]
s/ocllaboratively/collaboratively/
14:14:23 [divya]
bryan: should we try to harmonize the least, sign some sort of community power agreement with github.
14:14:30 [divya]
bryan: it is definitely easy
14:14:37 [divya]
bryan: compared to cvs which was horrible
14:14:52 [plinss]
BTW, Shepherd: http://test.csswg.org/shepherd/
14:14:52 [divya]
…mercurial is okay, i dont know about people stepping on top of each other. certainly github is easy.
14:15:27 [divya]
…if we didnt get tests here from outside community. we still need some kind of system that encourages participation. as long as we have some sort of scripted commands that allow tests to come in.
14:16:06 [divya]
wilhelm: i am mildly in favor of this change. all the front end devs i know use this service and github has 2 million members. how many people in this room have a github account
14:16:09 [divya]
[majority]
14:16:25 [SimonPieters]
q?
14:16:29 [divya]
wilhelm: that is an interesting datapoint
14:16:32 [SimonPieters]
ack wilhelm
14:17:07 [divya]
plinss: it doesnt matter what the people in this room are using. if we are trying to engage a broader audience, what are they using? is this tool a barrier to entry to people
14:17:26 [divya]
stearns: if crowdsourcing the test is the point, then we are going to be chasing the shiny 2 years from now
14:17:45 [divya]
jgraham: it is hard to become the shiny thing because we do only 1 thing. github you can do all the other htings.
14:18:01 [divya]
jgraham: it has a lot of uses so it spreads through the community quickly
14:18:01 [odinho_]
q+
14:18:32 [divya]
jgraham: so one other problem we have had sort of had solutions, is we have not had good code review of tests or testing tools
14:18:44 [divya]
…csswg's sheperd does not really work for code review
14:19:02 [divya]
…demo the tool we use @ Opera which was open sourced on monday on github, maybe it is interesting to other people
14:19:08 [divya]
[demos]
14:19:21 [darobin]
q+ to ask if critic has an API
14:19:43 [divya]
jgraham: each review is a branch in the repository
14:19:54 [MikeSmith]
q?
14:20:05 [divya]
[shows eg of a review for testharness.js]
14:20:20 [divya]
jgraham: shows if anyone has reviewed it or what % of files have been reviewed.
14:20:42 [divya]
…diff view old code on left new code on right
14:21:14 [divya]
…can create an issue on each line
14:21:30 [darobin]
Opera Critic is https://github.com/jensl/critic
14:21:32 [divya]
…i can then see the issues that have been raised
14:22:11 [divya]
…the author would fix the issue and push another commit to the branch that fixes the issue. The Critic tool would notice that the change in the line was for the issue, and then notify people on the update.
14:22:14 [ph]
ph has joined #testing
14:22:15 [divya]
(paraphrasing htere)
14:22:36 [divya]
…when you have changes you keep pushing htem, when review is done you merge the branch
14:23:02 [divya]
…without trying it, it is hard to explain how efficient that is. we tried other tools at opera. We tried reviewboard for e.g. it was a disaster and that is why we ended up with this tool
14:23:21 [divya]
…for test cases we may have slightly different process that may not be supported by this tool. e.g commit to main repo rather than commit to a branch
14:23:43 [divya]
…for testharness or other tools it might work really well. I am going to try it on github and use it instead of github review tool.
14:24:02 [divya]
…github does not allow you to comment across multiple lines, squash multiple commits and review together.
14:24:13 [divya]
SimonPieters: we use this for opera source and test suites that we want to release at opera.
14:24:55 [divya]
jgraham: until the issue is addressed it will not markt he code as ready to be merged.
14:25:11 [divya]
jgraham: probably enough sales pitch if people want to look at it. it is on github like everything else.
14:25:27 [divya]
darobin: does it have an api, so if you want to list issues somewhere else…
14:25:38 [divya]
jgraham: it has ability to write extensions, so you could write an API
14:25:42 [divya]
darobin: it would be logical
14:25:53 [divya]
jgraham: i think someone wrote an extension that creates a JSON dump of the issues
14:26:06 [divya]
…that is all i wanted to say. Does anyone else have anything to say?
14:26:15 [SimonPieters]
q?
14:26:35 [divya]
odinho_: another thing is when review is accepted, push a button and takes it where it is supposed to go.
14:26:41 [divya]
darobin: merges it to master?
14:26:48 [masinter]
q+ to ask about testing infrastructure for HTTP, URL tests needed more well-known sites, etc.
14:26:56 [darobin]
ack odinho_
14:26:57 [divya]
odinho_: merges to specific branch
14:27:06 [divya]
SimonPieters: …and resolves a bug in the bug system
14:27:07 [darobin]
ack darobin
14:27:07 [Zakim]
darobin, you wanted to ask if critic has an API
14:27:12 [isra]
isra has left #testing
14:27:22 [masinter]
http://redbot.org/
14:27:33 [divya]
odinho_: it is the same git server we use for everything. when it was put on github some people started pushing patches there because it was very visible.
14:27:33 [ArtB]
RRSAgent, make minutes
14:27:33 [RRSAgent]
I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB
14:27:58 [divya]
jgraham: there were people who had access to the sourcecode for over a year who never made a patch, and suddenly after it was on github for a day, made the patch
14:28:04 [divya]
darobin: same thing for respec
14:28:14 [divya]
jgraham: did anyone want to write up a summary of this session?
14:28:34 [bryan]
q+
14:28:40 [divya]
masinter: i want to ask about other parts of testing infrastructure.
14:28:53 [divya]
masinter: e.g. testing uris, nice to have DNS wildcard
14:29:05 [divya]
masinter: i see there is more to testing the web than just testing css.
14:29:32 [divya]
jgraham: i agree with you and i think we need to have conv about server side testing infra. i do not think we have the right people in the room to have that conversation
14:29:52 [divya]
masinter: i sent an email to public-test-infra mailing list and didnt get any response.
14:30:19 [divya]
masinter: i think assumption seems to be that testing happens in wg
14:30:32 [Marcos]
Marcos has left #testing
14:30:38 [divya]
krisk: the websocket one was extremely painful.
14:31:10 [divya]
jgraham: for specific tests, we invoke the MikeSmith protocol again
14:31:31 [divya]
[krisk says something about vm]
14:31:48 [SimonPieters]
q?
14:31:52 [divya]
bryan: how close are we to clone this framework thing and run behind our firewall?
14:31:56 [divya]
darobin: we are not there yet.
14:32:00 [SimonPieters]
ack masinter
14:32:00 [Zakim]
masinter, you wanted to ask about testing infrastructure for HTTP, URL tests needed more well-known sites, etc.
14:32:08 [divya]
darobin: can i give an update on this in 2 months?
14:32:26 [divya]
bryan: a proposal in coremob take a half-step forward by hosting coremob test somewhere else.
14:32:36 [divya]
darobin: github might help with as it has an api that makes it easy to pull data out.
14:32:45 [divya]
darobin: we will figure that out.
14:32:54 [divya]
bryan: esp for network operators its very important.
14:33:05 [SimonPieters]
ack bryan
14:33:18 [divya]
RRSAgent: make minutes
14:33:18 [RRSAgent]
I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html divya
14:37:30 [ArtB]
RRSAgent, bye
14:37:30 [RRSAgent]
I see no action items