See also: IRC log
<ArtB> Scribe: NotMe
<ArtB> ScribeNick: uh
<ArtB> Date: Halloween
<fantasai> plh: I'm in the digital publishing room, if you think you need me, feel free to pull me
<ArtB> https://github.com/w3c/testing-how-to
<ArtB> plh: there are two other testing related sessions today
<ArtB> … http://www.w3.org/wiki/TPAC2012#Session_Grid
<krisk> I can scribe...
<ArtB> scribenick: krisk
<ArtB> Scribe: Kris
<plh> http://w3c.github.com/testing-how-to/
<Mark_Vickers> =present+ Mark_Vickers
plh talking about overview - starting with 'what do you test in a specification?'
jgraham - the #github testing talk will talk about improvements in our test infrastructure, requirements for tests, etc..
plh - "what do you test in a specification?"
plh - it's always not obvious what needs to be testing, for example webidl exists that needs to be tested
plh - if it's going to have an impact on implementations needs to be tested
plh - MUST, MUST NOT, SHOULD , SHOULD NOT, MAY are all items that should be tested
<Marcos> Lachy: just pinged him
<Marcos> Lachy: what room are you in?
plh - mutliple tests can be created for a single statement from the spec
<Lachy> Marcos: RHONE 2
plh - it's complex and not automated....
Part #2 for the talk "How to write a test?"
First mechanism is javascript test
second test mechaism is what is known as a REF test
scribe: used for rendering tests
third mechanism is 'self-describing' tests
scribe: you don't want to do this
by default, since it's expensive
... though at times this is the only mechanism
For script based tests we use testharness.js
<ArtB> … http://w3c-test.org/resources/
Check out the api examples...
http://w3c-test.org/resources/apisample.htm
<ArtB> http://darobin.github.com/test-harness-tutorial/docs/using-testharness.html (testharness.js tutorial by Robin Berjon)
http://w3c-test.org/resources/apisample2.htm
http://w3c-test.org/resources/apisample3.htm
Two main use cases - asynchronous (e.g. events aka onload) and synchronous
scribe: a number of assert_* exist to validate the results
plh showing the test examples for each...
some cool asserts...
assert_unreachable - makes sure an event DOESN'T fire
<sms> wonder what approx_equals does
assert_readonly, assert_idl_attribute!
assert_array is also very helpful as well...
<sms> krisk: how do you assert the negative?
assert_false
<sms> Just assume it doesn't fire within a certain amount of time?
<sms> Sorry: "assert_unreachable"
you can timeout at the 'page' level and at the test level (you can have more than one test per page)
scribe: adding metadata to the test is very good when you want to get an understanding on what parts of the spec have coverage
see slide 18 from the deck on https://github.com/w3c/testing-how-to
scribe: now on slide 19...
... what are the pitfalls?
It important that we get the same number of test pass/fails
DO NOT stop running tests because something is not supported
<SimonPieters> documentation in the source: http://w3c-test.org/resources/testharness.js
PLH see slide #20 if you can follow along just see the link above
http://darobin.github.com/test-harness-tutorial/docs/using-testharness.html
Now on Reftest
what is a reftest?
It's two pages that should result in the same rendering
<jet> are there docs on how each browser runs reftests?
<simonstewart> http://www.w3.org/TR/webdriver/
<simonstewart> http://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-spec.html
<jgraham> jet: I dont think we have docs, but our system is a little different to yours
<jgraham> We can talk about it
<jet> jgraham: thx
The test web server has php support, so you can add a delay or headers, etc..
We have media and media files as well that can be used...
we have codec agnostic support files as well
W3C Test Suite Licenses
See slide #33
Now on to Mark Vickers
<Mark_Vickers> http://lists.w3.org/Archives/Public/public-web-and-tv/2012Oct/0017.html
Who will talk abou the Web and TV Testing Task Force
The recharter is going to be updated so that this WG has a Testing Task Force
Testing Use Cases
#1 Currently Verification of the specification
#2 New Improve the consistency of the open web platform (OWP)
#3 New support external testing/certification organizations
Number of group do this today - for example DLNA
When it comes from HTML5 we want to have these tests come from the w3c and not 'fork' the tests
#4 Support testing devices
summary of this session!
tobin can you give summary?
One person states - no testing is enough
<rhauck> uh, no "amount" of testing is enough?
<ArtB> Title: #github session @ TPAC 2012
<ArtB> Date: Halloween
<ArtB> Chair: James Graham
<divya> [missed a bit]
<ArtB> ScribeNick: divya
<ArtB> Scribe: Divya
Starting with Source control which was the original motivation for this session
jgraham: atm we are running custom w3c instance of mercurial
…we ahve all these repos, in order to contribute tests to that you need to learn how to use mercurial, find the repo, add your tests, commit & send a mail to mailing list
RRSAgent: make logs public
…one suggestion has come up is if we are better off outsourcing part of problem, in particular github
<ArtB> (http mirror for webapps' test repo: http://w3c-test.org/webapps/)
…its git whichi is very similar to mercuirial hosted service nicer UI, and has lot of developer buy-in
…all big libraries are hosted on Github
…has a different workflow which is slightly better
[shows example of testharness.js] http://github.com/w3c/testharness.js
…they have this mechanism called pull request instead of sending mail off to a mailing list
…it gives you a way of tracking various people who want to make changes
…you do not have to go out to mail, mailing list and different systems
…do sy yhid poiny it would be interesting to have a discussion about whether people think this is a good idea.
s/so at this point
kris: there has this whole patent policy stuff that needs to be addressed
darobin: 1st there is
contributions to spec, chair makes a call on whether the
contributions has done IP properly
... separate from contributions to test suite. they are fairly
standard open source license and you can get anyone
contributing to test suite without IP consequences for the
royalty free aspect ot the spec
kris: do we make people sign to
this to contribute to the spec.
... cable companies think there would be a lot of money in
it.
darobin: W3C is not allowed to publish test suites in anything other htan open source license
kris: today you cant add tests to w3c without signing some stuff
darobin: that is manageable, you
just need a list of people who signed off.
... i do not see it as an issue.
ArtB: i am indifferent, beggers
cant be choosy might dominate here. by we i mean webapps
anyhow. we will take tests
... we should provide a way for them to contribute
... if we are chasing another shiny object. in this case
github. i am not convinced what we have right now is broken
enough.
darobin: how often do you use the mercurial system
ArtB: probably a few times permonth
darobin: i think that maybe why you do not see the problem
kris: i dont believe everyone
moved ot github, we still have stuff coming out of CVS. we have
to learn multiple systems
... that is a valid thing
darobin: if we want outside contributions we can give up on using mercurial.
kris: i do not think it is really
very inviting to contribute.
... even geeks have a hard time to contribute to github(?)
jgraham: i think robin said on
wiki page, it is about github not git
... the point is because there is an existing community around
github and much broader than w3c, the existing pool of users
who know what the workflow is, then they make pull
requests.
darobin: in developer world, if it does not exist on github, it does not exist.
Marcos: i have seen this in
responsive images. amazing to get that workflow, getting the
intial flow is hard, once you ahve the community happening, it
is hard.
... what our community here does is important. are they
comfortable going through this. it is not as tedious as doing
the manual merge on the command line. github does this
automatically, and transparent. commenting appears real
time.
darobin: the way we use mercurial in w3c setup is not correct. we have central repo for something that is supposed to be distributed, we are using it as cvs and it is easy to destroy content this way.
[random discussion]
Mark_Vickers: on the legal thing, my understanding is that github repo is you can put a CLA that covers a particular project.
darobin: there is an approval stamp the 1st time you contribute.
Mark_Vickers: it has value to have a large number of tests in terms of saving money for peoople who create content.
<MikeSmith> btw the way dvcs.w3.org is set up right now it's possible for a push to create multiple heads
Mark_Vickers: not interms of saving money.
kris: i dont think we can say it is all free and open.
jgraham: you can see who the contribution is coming from, so you can say if they should fill the form or not. it is no different than when people send us a patch via email, and that you commit the patch without checking if they signe the right agreement
plinss: i am in favour of
crowdsourcing the tests, i certainly get the fact that github
does that with devs, for some definition of devs. The devs that
github targets are they contributing to test suites.
... the lot of folks who show up at test the web forward are
designers not coders.
darobin: my experience of test the web forward is paris. They tried to use dropbox, it didnt work and then they switched to github.
stearns: at SF #1 question was why we are not using github.
lmclister: or what is the mercurial equivalent for the git command
plinss: i am concerned like ArtB
that we are chasing the next shiny thing.
... i would rather see w3c be the cool place for this
stuff.
marcos: cost of setting up infrastructure is huge.
<MikeSmith> I wish somebody from the w3c systems team were here to comment
Marcos1: how many years do we keep doing the same thing.
plinss: i have snever seen a
migration step not lose something
... i have never seen a pre-baked thing that delivers to all of
our needs.
darobin: we may have fewer points of pain
plinss: something else might be the point of pain
jgraham: i am not claiming github is perfect at least it has an api that we can build on
…if it does turn out it is not optimal we can just replace that bit of it.
plinss: what if github goes down
jgraham: certainly we should have w3c hosted read only clones of repo
plinss: github has the apis to developp tools, if we spend times building tools on top of infrastructure
jgraham: someone at opera wrote a code review tool that was open sourced on monday. I think it is orders of magnitude better than github one, and it might be worth looking at that.
<odinho_> Critic code review tool (from Opera)
…might be worth making it transparent such that we can use that tool rather than whats on github
[missed convo]
jgraham: there may be different design goals
plinss: shepherd was designed to be a test review system
jgraham: if it gets us an awesome user experience then, but it is not there yet.
plinss: anything we build is
going to take time
... rather than rely on someone else's infrastructure
... i have no predictability on github infrastructure
darobin: as long as the people who wrote the code are alive & around to maintain it
kris: stuff on w3c lasts longer than it lasts on github
darobin: 10 years ago sourceforce
was the only option and people hated it.
... anything out there would be synchronized tow3c
... 1. one side is git 2. other side of tooling is github api,
which maps very closely to git.
... i am not too worried about that, there exists at least 1
open source implementation that exposes same api on top of git
repository
krisk: how long have we had stuff on this site so far?
jgraham: which site?
krisk: github
jgraham: well the htmlwg is
readwrite. that is what they are using as primary
interface
... test stuff is mostly readonly. the html test suite is
readonly.
... testharness.js is readonly here.
…webperforamnce is read only event source is read-write doesnt exist anywhere else.
…amaya is readwrite nowadays
…this is the official w3c github repo there is a lot more w3c related github things that are not part of the official one
darobin: there is actually a lot more content than this.
krisk: isnt that a problem? wont people have hard time giguring out
darobin: ideally i want all the
official stuff to go to the official thing.
... i would encourage groups be consistent and make their info
more easy to find.
jgraham: it is a problem we would solve if we say this is the prefered user interface
darobin: what has happened today is people join WG they are told mercurial is the way to do, lose data once, and hten they get rustrated and they do not know w3c has a github account.
Marcos1: i am one of the people who has test suites in public account not on w3cc account. It looked like i didnt know who to contact.
jgraham: the process at the moement is to ask MikeSmith
darobin: or me
... if you look at the members anyone there who is w3c should
normally have the ability to add you to the organization
<Zakim> ArtB, you wanted to ask: how would this work http://w3c-test.org/framework/app/suite in the github world?
<ArtB> http://w3c-test.org/framework/app/suite
ArtB: i am oppossed to
makework.
... we have some tests in webapps. in the scenario where
webapps would have some tests in mercurial etc
... would framework accomodate both?
darobin: not a problem
ArtB: do you have WG that are like ground zero, that have already gone to github
darobin: the htmlwg did that for spec editing
Marcos1: DAP did it
krisk: is it not on CVS for html spec?
darobin: there is a tool that takes the github source, compiles into html thing and then checks into cvs
…sam wrote the tool in under 15 mins
krisk: if someone starts from ground zero, we have other infrastructure how does w3test.org work
darobin: i maintain respec on github, whenever i ship a build, it automatically gets synched to w3c server
jgraham: w3test.org does perl update at the moment changing to git pull owuld be trivial
SimonPieters: whatwg specs are developed on github.
isra: should w3c ever decide to
use github, i think we should protect our members find some
sort of legal protection in case they change user agreements,
what happens when they change agreements.
... it is a fertile ground for some issues to come up.
krisk: i would +1 that, for microsoft, it is not a normal thing.
darobin: plenty of ms people put stuff on github
krisk: there are some things in place that is different tho
isra: w3c has enough power to impose some conditions.
darobin: disappearing data is not
a problem
... you are thinking the case what happens when they become
evil.
masinter: if you can sync why is there an issue
plinss: it is not about the repo itself. the issue is tooling around the repo, we will be married to what we are building on top of
masinter: github has an api.
plinss: dont confuse git and github and mercurial
isra: its about legal terms of
use
... not about synching data
[side conversations]
masinter: most tools seem to be for running tests than submitting them
plinss: in csswg we have
both
... we have sheperd code it runs, finding errors giving data
right back into client.
... comments reviews happen on shpeherd. if push comes to shove
i can move those to github.
…so i dont defend the code per se, there is a lot of sense that the methodology runs on same server
…validates test files, etc that will soon move to CI system
…there is lot that is either built or not too far away from being built
masinter: based on mercurial?
plinss: yes
... at that time w3c was using svn, and then it appeared like
w3c was going to standardize on mercurial
darobin: it is not w3c the orgn it is just feedback the people
bryan: i am using github to
collaboratively edit specs
... should that be under w3c account
... should we try to harmonize the least, sign some sort of
community power agreement with github.
... it is definitely easy
... compared to cvs which was horrible
<plinss> BTW, Shepherd: http://test.csswg.org/shepherd/
…mercurial is okay, i dont know about people stepping on top of each other. certainly github is easy.
…if we didnt get tests here from outside community. we still need some kind of system that encourages participation. as long as we have some sort of scripted commands that allow tests to come in.
wilhelm: i am mildly in favor of this change. all the front end devs i know use this service and github has 2 million members. how many people in this room have a github account
[majority]
wilhelm: that is an interesting datapoint
plinss: it doesnt matter what the people in this room are using. if we are trying to engage a broader audience, what are they using? is this tool a barrier to entry to people
stearns: if crowdsourcing the test is the point, then we are going to be chasing the shiny 2 years from now
jgraham: it is hard to become the
shiny thing because we do only 1 thing. github you can do all
the other htings.
... it has a lot of uses so it spreads through the community
quickly
... so one other problem we have had sort of had solutions, is
we have not had good code review of tests or testing tools
…csswg's sheperd does not really work for code review
…demo the tool we use @ Opera which was open sourced on monday on github, maybe it is interesting to other people
[demos]
jgraham: each review is a branch in the repository
[shows eg of a review for testharness.js]
jgraham: shows if anyone has reviewed it or what % of files have been reviewed.
…diff view old code on left new code on right
…can create an issue on each line
<darobin> Opera Critic is https://github.com/jensl/critic
…i can then see the issues that have been raised
…the author would fix the issue and push another commit to the branch that fixes the issue. The Critic tool would notice that the change in the line was for the issue, and then notify people on the update.
(paraphrasing htere)
…when you have changes you keep pushing htem, when review is done you merge the branch
…without trying it, it is hard to explain how efficient that is. we tried other tools at opera. We tried reviewboard for e.g. it was a disaster and that is why we ended up with this tool
…for test cases we may have slightly different process that may not be supported by this tool. e.g commit to main repo rather than commit to a branch
…for testharness or other tools it might work really well. I am going to try it on github and use it instead of github review tool.
…github does not allow you to comment across multiple lines, squash multiple commits and review together.
SimonPieters: we use this for opera source and test suites that we want to release at opera.
jgraham: until the issue is
addressed it will not markt he code as ready to be
merged.
... probably enough sales pitch if people want to look at it.
it is on github like everything else.
darobin: does it have an api, so if you want to list issues somewhere else…
jgraham: it has ability to write extensions, so you could write an API
darobin: it would be logical
jgraham: i think someone wrote an extension that creates a JSON dump of the issues
…that is all i wanted to say. Does anyone else have anything to say?
odinho_: another thing is when review is accepted, push a button and takes it where it is supposed to go.
darobin: merges it to master?
odinho_: merges to specific branch
SimonPieters: …and resolves a bug in the bug system
<Zakim> darobin, you wanted to ask if critic has an API
<masinter> http://redbot.org/
odinho_: it is the same git server we use for everything. when it was put on github some people started pushing patches there because it was very visible.
jgraham: there were people who had access to the sourcecode for over a year who never made a patch, and suddenly after it was on github for a day, made the patch
darobin: same thing for respec
jgraham: did anyone want to write up a summary of this session?
masinter: i want to ask about
other parts of testing infrastructure.
... e.g. testing uris, nice to have DNS wildcard
... i see there is more to testing the web than just testing
css.
jgraham: i agree with you and i think we need to have conv about server side testing infra. i do not think we have the right people in the room to have that conversation
masinter: i sent an email to
public-test-infra mailing list and didnt get any
response.
... i think assumption seems to be that testing happens in
wg
krisk: the websocket one was extremely painful.
jgraham: for specific tests, we invoke the MikeSmith protocol again
[krisk says something about vm]
bryan: how close are we to clone this framework thing and run behind our firewall?
darobin: we are not there yet.
<Zakim> masinter, you wanted to ask about testing infrastructure for HTTP, URL tests needed more well-known sites, etc.
darobin: can i give an update on this in 2 months?
bryan: a proposal in coremob take a half-step forward by hosting coremob test somewhere else.
darobin: github might help with
as it has an api that makes it easy to pull data out.
... we will figure that out.
bryan: esp for network operators its very important.
RRSAgent: make minutes
This is scribe.perl Revision: 1.137 of Date: 2012/09/20 20:19:01 Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: RRSAgent_Text_Format (score 1.00) Succeeded: s/seff-describing/self-describing/ Succeeded: s/wel/well/ WARNING: Bad s/// command: s/so at this point Succeeded: s|https://github.com/jensl/critic/ -> Critic code review tool (from Opera)|-> https://github.com/jensl/critic/ Critic code review tool (from Opera)| Succeeded: s/[] work/makework/ Succeeded: s/doing to svg/using svn/ Succeeded: s/ocllaboratively/collaboratively/ Found Scribe: NotMe Found ScribeNick: uh WARNING: No scribe lines found matching ScribeNick pattern: <uh> ... Found ScribeNick: krisk Found Scribe: Kris Found ScribeNick: divya Found Scribe: Divya Inferring ScribeNick: divya Scribes: NotMe, Kris, Divya ScribeNicks: uh, krisk, divya WARNING: No "Topic:" lines found. Present: Art_Barstow Anssi_Kostiainen Simon Stewart kris_krueger rhauck giuseppe Wonsuk_Lee Jet Villegas Peter Linss jgraham Larry McLister Simon_Pieters JohnJansen adambe Wilhelm Joys Andersen Bryan_Sullivan Arron Eicholz Robin Berjon Henry S. Thompson Marcos Caceres Michael_Cooper Tobie_Langel Mark_Vickers Odin_Hoerthe_Omdal WARNING: Date not understood: Halloween Got date from IRC log name: 31 Oct 2012 Guessing minutes URL: http://www.w3.org/2012/10/31-testing-minutes.html People with action items: WARNING: No "Topic: ..." lines found! Resulting HTML may have an empty (invalid) <ol>...</ol>. Explanation: "Topic: ..." lines are used to indicate the start of new discussion topics or agenda items, such as: <dbooth> Topic: Review of Amy's report[End of scribe.perl diagnostic output]