<ArtB> Scribe: NotMe

<ArtB> ScribeNick: uh

<ArtB> Date: Halloween

<fantasai> plh: I'm in the digital publishing room, if you think you need me, feel free to pull me

<ArtB> https://github.com/w3c/testing-how-to

<ArtB> plh: there are two other testing related sessions today

<ArtB> … http://www.w3.org/wiki/TPAC2012#Session_Grid

<krisk> I can scribe...

<ArtB> scribenick: krisk

<ArtB> Scribe: Kris

<plh> http://w3c.github.com/testing-how-to/

<Mark_Vickers> =present+ Mark_Vickers

plh talking about overview - starting with 'what do you test in a specification?'

jgraham - the #github testing talk will talk about improvements in our test infrastructure, requirements for tests, etc..

plh - "what do you test in a specification?"

plh - it's always not obvious what needs to be testing, for example webidl exists that needs to be tested

plh - if it's going to have an impact on implementations needs to be tested

plh - MUST, MUST NOT, SHOULD , SHOULD NOT, MAY are all items that should be tested

<Marcos> Lachy: just pinged him

<Marcos> Lachy: what room are you in?

plh - mutliple tests can be created for a single statement from the spec

<Lachy> Marcos: RHONE 2

plh - it's complex and not automated....

Part #2 for the talk "How to write a test?"

First mechanism is javascript test

second test mechaism is what is known as a REF test

scribe: used for rendering tests

third mechanism is 'self-describing' tests

scribe: you don't want to do this by default, since it's expensive
... though at times this is the only mechanism

For script based tests we use testharness.js

<ArtB> … http://w3c-test.org/resources/

Check out the api examples...

http://w3c-test.org/resources/apisample.htm

<ArtB> http://darobin.github.com/test-harness-tutorial/docs/using-testharness.html (testharness.js tutorial by Robin Berjon)

http://w3c-test.org/resources/apisample2.htm

http://w3c-test.org/resources/apisample3.htm

Two main use cases - asynchronous (e.g. events aka onload) and synchronous

scribe: a number of assert_* exist to validate the results

plh showing the test examples for each...

some cool asserts...

assert_unreachable - makes sure an event DOESN'T fire

<sms> wonder what approx_equals does

assert_readonly, assert_idl_attribute!

assert_array is also very helpful as well...

<sms> krisk: how do you assert the negative?

assert_false

<sms> Just assume it doesn't fire within a certain amount of time?

<sms> Sorry: "assert_unreachable"

you can timeout at the 'page' level and at the test level (you can have more than one test per page)

scribe: adding metadata to the test is very good when you want to get an understanding on what parts of the spec have coverage

see slide 18 from the deck on https://github.com/w3c/testing-how-to

scribe: now on slide 19...
... what are the pitfalls?

It important that we get the same number of test pass/fails

DO NOT stop running tests because something is not supported

<SimonPieters> documentation in the source: http://w3c-test.org/resources/testharness.js

PLH see slide #20 if you can follow along just see the link above

http://darobin.github.com/test-harness-tutorial/docs/using-testharness.html

Now on Reftest

what is a reftest?

It's two pages that should result in the same rendering

<jet> are there docs on how each browser runs reftests?

<simonstewart> http://www.w3.org/TR/webdriver/

<simonstewart> http://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-spec.html

<jgraham> jet: I dont think we have docs, but our system is a little different to yours

<jgraham> We can talk about it

<jet> jgraham: thx

The test web server has php support, so you can add a delay or headers, etc..

We have media and media files as well that can be used...

we have codec agnostic support files as well

W3C Test Suite Licenses

See slide #33

Now on to Mark Vickers

<Mark_Vickers> http://lists.w3.org/Archives/Public/public-web-and-tv/2012Oct/0017.html

Who will talk abou the Web and TV Testing Task Force

The recharter is going to be updated so that this WG has a Testing Task Force

Testing Use Cases

#1 Currently Verification of the specification

#2 New Improve the consistency of the open web platform (OWP)

#3 New support external testing/certification organizations

Number of group do this today - for example DLNA

When it comes from HTML5 we want to have these tests come from the w3c and not 'fork' the tests

#4 Support testing devices

summary of this session!

tobin can you give summary?

One person states - no testing is enough

<rhauck> uh, no "amount" of testing is enough?

<ArtB> Title: #github session @ TPAC 2012

<ArtB> Date: Halloween

<ArtB> Chair: James Graham

<divya> [missed a bit]

<ArtB> ScribeNick: divya

<ArtB> Scribe: Divya

Starting with Source control which was the original motivation for this session

jgraham: atm we are running custom w3c instance of mercurial

…we ahve all these repos, in order to contribute tests to that you need to learn how to use mercurial, find the repo, add your tests, commit & send a mail to mailing list

RRSAgent: make logs public

…one suggestion has come up is if we are better off outsourcing part of problem, in particular github

<ArtB> (http mirror for webapps' test repo: http://w3c-test.org/webapps/)

…its git whichi is very similar to mercuirial hosted service nicer UI, and has lot of developer buy-in

…all big libraries are hosted on Github

…has a different workflow which is slightly better

[shows example of testharness.js] http://github.com/w3c/testharness.js

…they have this mechanism called pull request instead of sending mail off to a mailing list

…it gives you a way of tracking various people who want to make changes

…you do not have to go out to mail, mailing list and different systems

…do sy yhid poiny it would be interesting to have a discussion about whether people think this is a good idea.

s/so at this point

kris: there has this whole patent policy stuff that needs to be addressed

darobin: 1st there is contributions to spec, chair makes a call on whether the contributions has done IP properly
... separate from contributions to test suite. they are fairly standard open source license and you can get anyone contributing to test suite without IP consequences for the royalty free aspect ot the spec

kris: do we make people sign to this to contribute to the spec.
... cable companies think there would be a lot of money in it.

darobin: W3C is not allowed to publish test suites in anything other htan open source license

kris: today you cant add tests to w3c without signing some stuff

darobin: that is manageable, you just need a list of people who signed off.
... i do not see it as an issue.

ArtB: i am indifferent, beggers cant be choosy might dominate here. by we i mean webapps anyhow. we will take tests
... we should provide a way for them to contribute
... if we are chasing another shiny object. in this case github. i am not convinced what we have right now is broken enough.

darobin: how often do you use the mercurial system

ArtB: probably a few times permonth

darobin: i think that maybe why you do not see the problem

kris: i dont believe everyone moved ot github, we still have stuff coming out of CVS. we have to learn multiple systems
... that is a valid thing

darobin: if we want outside contributions we can give up on using mercurial.

kris: i do not think it is really very inviting to contribute.
... even geeks have a hard time to contribute to github(?)

jgraham: i think robin said on wiki page, it is about github not git
... the point is because there is an existing community around github and much broader than w3c, the existing pool of users who know what the workflow is, then they make pull requests.

darobin: in developer world, if it does not exist on github, it does not exist.

Marcos: i have seen this in responsive images. amazing to get that workflow, getting the intial flow is hard, once you ahve the community happening, it is hard.
... what our community here does is important. are they comfortable going through this. it is not as tedious as doing the manual merge on the command line. github does this automatically, and transparent. commenting appears real time.

darobin: the way we use mercurial in w3c setup is not correct. we have central repo for something that is supposed to be distributed, we are using it as cvs and it is easy to destroy content this way.

[random discussion]

Mark_Vickers: on the legal thing, my understanding is that github repo is you can put a CLA that covers a particular project.

darobin: there is an approval stamp the 1st time you contribute.

Mark_Vickers: it has value to have a large number of tests in terms of saving money for peoople who create content.

<MikeSmith> btw the way dvcs.w3.org is set up right now it's possible for a push to create multiple heads

Mark_Vickers: not interms of saving money.

kris: i dont think we can say it is all free and open.

jgraham: you can see who the contribution is coming from, so you can say if they should fill the form or not. it is no different than when people send us a patch via email, and that you commit the patch without checking if they signe the right agreement

plinss: i am in favour of crowdsourcing the tests, i certainly get the fact that github does that with devs, for some definition of devs. The devs that github targets are they contributing to test suites.
... the lot of folks who show up at test the web forward are designers not coders.

darobin: my experience of test the web forward is paris. They tried to use dropbox, it didnt work and then they switched to github.

stearns: at SF #1 question was why we are not using github.

lmclister: or what is the mercurial equivalent for the git command

plinss: i am concerned like ArtB that we are chasing the next shiny thing.
... i would rather see w3c be the cool place for this stuff.

marcos: cost of setting up infrastructure is huge.

<MikeSmith> I wish somebody from the w3c systems team were here to comment

Marcos1: how many years do we keep doing the same thing.

plinss: i have snever seen a migration step not lose something
... i have never seen a pre-baked thing that delivers to all of our needs.

darobin: we may have fewer points of pain

plinss: something else might be the point of pain

jgraham: i am not claiming github is perfect at least it has an api that we can build on

…if it does turn out it is not optimal we can just replace that bit of it.

plinss: what if github goes down

jgraham: certainly we should have w3c hosted read only clones of repo

plinss: github has the apis to developp tools, if we spend times building tools on top of infrastructure

jgraham: someone at opera wrote a code review tool that was open sourced on monday. I think it is orders of magnitude better than github one, and it might be worth looking at that.

<odinho_> Critic code review tool (from Opera)

…might be worth making it transparent such that we can use that tool rather than whats on github

[missed convo]

jgraham: there may be different design goals

plinss: shepherd was designed to be a test review system

jgraham: if it gets us an awesome user experience then, but it is not there yet.

plinss: anything we build is going to take time
... rather than rely on someone else's infrastructure
... i have no predictability on github infrastructure

darobin: as long as the people who wrote the code are alive & around to maintain it

kris: stuff on w3c lasts longer than it lasts on github

darobin: 10 years ago sourceforce was the only option and people hated it.
... anything out there would be synchronized tow3c
... 1. one side is git 2. other side of tooling is github api, which maps very closely to git.
... i am not too worried about that, there exists at least 1 open source implementation that exposes same api on top of git repository

krisk: how long have we had stuff on this site so far?

jgraham: which site?

krisk: github

jgraham: well the htmlwg is readwrite. that is what they are using as primary interface
... test stuff is mostly readonly. the html test suite is readonly.
... testharness.js is readonly here.

…webperforamnce is read only event source is read-write doesnt exist anywhere else.

…amaya is readwrite nowadays

…this is the official w3c github repo there is a lot more w3c related github things that are not part of the official one

darobin: there is actually a lot more content than this.

krisk: isnt that a problem? wont people have hard time giguring out

darobin: ideally i want all the official stuff to go to the official thing.
... i would encourage groups be consistent and make their info more easy to find.

jgraham: it is a problem we would solve if we say this is the prefered user interface

darobin: what has happened today is people join WG they are told mercurial is the way to do, lose data once, and hten they get rustrated and they do not know w3c has a github account.

Marcos1: i am one of the people who has test suites in public account not on w3cc account. It looked like i didnt know who to contact.

jgraham: the process at the moement is to ask MikeSmith

darobin: or me
... if you look at the members anyone there who is w3c should normally have the ability to add you to the organization

<Zakim> ArtB, you wanted to ask: how would this work http://w3c-test.org/framework/app/suite in the github world?

<ArtB> http://w3c-test.org/framework/app/suite

ArtB: i am oppossed to makework.
... we have some tests in webapps. in the scenario where webapps would have some tests in mercurial etc
... would framework accomodate both?

darobin: not a problem

ArtB: do you have WG that are like ground zero, that have already gone to github

darobin: the htmlwg did that for spec editing

Marcos1: DAP did it

krisk: is it not on CVS for html spec?

darobin: there is a tool that takes the github source, compiles into html thing and then checks into cvs

…sam wrote the tool in under 15 mins

krisk: if someone starts from ground zero, we have other infrastructure how does w3test.org work

darobin: i maintain respec on github, whenever i ship a build, it automatically gets synched to w3c server

jgraham: w3test.org does perl update at the moment changing to git pull owuld be trivial

SimonPieters: whatwg specs are developed on github.

isra: should w3c ever decide to use github, i think we should protect our members find some sort of legal protection in case they change user agreements, what happens when they change agreements.
... it is a fertile ground for some issues to come up.

krisk: i would +1 that, for microsoft, it is not a normal thing.

darobin: plenty of ms people put stuff on github

krisk: there are some things in place that is different tho

isra: w3c has enough power to impose some conditions.

darobin: disappearing data is not a problem
... you are thinking the case what happens when they become evil.

masinter: if you can sync why is there an issue

plinss: it is not about the repo itself. the issue is tooling around the repo, we will be married to what we are building on top of

masinter: github has an api.

plinss: dont confuse git and github and mercurial

isra: its about legal terms of use
... not about synching data

[side conversations]

masinter: most tools seem to be for running tests than submitting them

plinss: in csswg we have both
... we have sheperd code it runs, finding errors giving data right back into client.
... comments reviews happen on shpeherd. if push comes to shove i can move those to github.

…so i dont defend the code per se, there is a lot of sense that the methodology runs on same server

…validates test files, etc that will soon move to CI system

…there is lot that is either built or not too far away from being built

masinter: based on mercurial?

plinss: yes
... at that time w3c was using svn, and then it appeared like w3c was going to standardize on mercurial

darobin: it is not w3c the orgn it is just feedback the people

bryan: i am using github to collaboratively edit specs
... should that be under w3c account
... should we try to harmonize the least, sign some sort of community power agreement with github.
... it is definitely easy
... compared to cvs which was horrible

<plinss> BTW, Shepherd: http://test.csswg.org/shepherd/

…mercurial is okay, i dont know about people stepping on top of each other. certainly github is easy.

…if we didnt get tests here from outside community. we still need some kind of system that encourages participation. as long as we have some sort of scripted commands that allow tests to come in.

wilhelm: i am mildly in favor of this change. all the front end devs i know use this service and github has 2 million members. how many people in this room have a github account

[majority]

wilhelm: that is an interesting datapoint

plinss: it doesnt matter what the people in this room are using. if we are trying to engage a broader audience, what are they using? is this tool a barrier to entry to people

stearns: if crowdsourcing the test is the point, then we are going to be chasing the shiny 2 years from now

jgraham: it is hard to become the shiny thing because we do only 1 thing. github you can do all the other htings.
... it has a lot of uses so it spreads through the community quickly
... so one other problem we have had sort of had solutions, is we have not had good code review of tests or testing tools

…csswg's sheperd does not really work for code review

…demo the tool we use @ Opera which was open sourced on monday on github, maybe it is interesting to other people

[demos]

jgraham: each review is a branch in the repository

[shows eg of a review for testharness.js]

jgraham: shows if anyone has reviewed it or what % of files have been reviewed.

…diff view old code on left new code on right

…can create an issue on each line

<darobin> Opera Critic is https://github.com/jensl/critic

…i can then see the issues that have been raised

…the author would fix the issue and push another commit to the branch that fixes the issue. The Critic tool would notice that the change in the line was for the issue, and then notify people on the update.

(paraphrasing htere)

…when you have changes you keep pushing htem, when review is done you merge the branch

…without trying it, it is hard to explain how efficient that is. we tried other tools at opera. We tried reviewboard for e.g. it was a disaster and that is why we ended up with this tool

…for test cases we may have slightly different process that may not be supported by this tool. e.g commit to main repo rather than commit to a branch

…for testharness or other tools it might work really well. I am going to try it on github and use it instead of github review tool.

…github does not allow you to comment across multiple lines, squash multiple commits and review together.

SimonPieters: we use this for opera source and test suites that we want to release at opera.

jgraham: until the issue is addressed it will not markt he code as ready to be merged.
... probably enough sales pitch if people want to look at it. it is on github like everything else.

darobin: does it have an api, so if you want to list issues somewhere else…

jgraham: it has ability to write extensions, so you could write an API

darobin: it would be logical

jgraham: i think someone wrote an extension that creates a JSON dump of the issues

…that is all i wanted to say. Does anyone else have anything to say?

odinho_: another thing is when review is accepted, push a button and takes it where it is supposed to go.

darobin: merges it to master?

odinho_: merges to specific branch

SimonPieters: …and resolves a bug in the bug system

<Zakim> darobin, you wanted to ask if critic has an API

<masinter> http://redbot.org/

odinho_: it is the same git server we use for everything. when it was put on github some people started pushing patches there because it was very visible.

jgraham: there were people who had access to the sourcecode for over a year who never made a patch, and suddenly after it was on github for a day, made the patch

darobin: same thing for respec

jgraham: did anyone want to write up a summary of this session?

masinter: i want to ask about other parts of testing infrastructure.
... e.g. testing uris, nice to have DNS wildcard
... i see there is more to testing the web than just testing css.

jgraham: i agree with you and i think we need to have conv about server side testing infra. i do not think we have the right people in the room to have that conversation

masinter: i sent an email to public-test-infra mailing list and didnt get any response.
... i think assumption seems to be that testing happens in wg

krisk: the websocket one was extremely painful.

jgraham: for specific tests, we invoke the MikeSmith protocol again

[krisk says something about vm]

bryan: how close are we to clone this framework thing and run behind our firewall?

darobin: we are not there yet.

<Zakim> masinter, you wanted to ask about testing infrastructure for HTTP, URL tests needed more well-known sites, etc.

darobin: can i give an update on this in 2 months?

bryan: a proposal in coremob take a half-step forward by hosting coremob test somewhere else.

darobin: github might help with as it has an api that makes it easy to pull data out.
... we will figure that out.

bryan: esp for network operators its very important.

RRSAgent: make minutes

This is scribe.perl Revision: 1.137  of Date: 2012/09/20 20:19:01  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/seff-describing/self-describing/
Succeeded: s/wel/well/
WARNING: Bad s/// command: s/so at this point
Succeeded: s|https://github.com/jensl/critic/ -> Critic code review tool (from Opera)|-> https://github.com/jensl/critic/ Critic code review tool (from Opera)|
Succeeded: s/[] work/makework/
Succeeded: s/doing to svg/using svn/
Succeeded: s/ocllaboratively/collaboratively/
Found Scribe: NotMe
Found ScribeNick: uh
WARNING: No scribe lines found matching ScribeNick pattern: <uh> ...
Found ScribeNick: krisk
Found Scribe: Kris
Found ScribeNick: divya
Found Scribe: Divya
Inferring ScribeNick: divya
Scribes: NotMe, Kris, Divya
ScribeNicks: uh, krisk, divya

WARNING: No "Topic:" lines found.

Present: Art_Barstow Anssi_Kostiainen Simon Stewart kris_krueger rhauck giuseppe Wonsuk_Lee Jet Villegas Peter Linss jgraham Larry McLister Simon_Pieters JohnJansen adambe Wilhelm Joys Andersen Bryan_Sullivan Arron Eicholz Robin Berjon Henry S. Thompson Marcos Caceres Michael_Cooper Tobie_Langel Mark_Vickers Odin_Hoerthe_Omdal
WARNING: Date not understood: Halloween
Got date from IRC log name: 31 Oct 2012
Guessing minutes URL: http://www.w3.org/2012/10/31-testing-minutes.html
People with action items: 

WARNING: No "Topic: ..." lines found!  
Resulting HTML may have an empty (invalid) <ol>...</ol>.

Explanation: "Topic: ..." lines are used to indicate the start of 
new discussion topics or agenda items, such as:
<dbooth> Topic: Review of Amy's report