10:03:57 RRSAgent has joined #testing 10:03:57 logging to http://www.w3.org/2012/10/31-testing-irc 10:04:10 Scribe: NotMe 10:04:17 ScribeNick: uh 10:04:25 Date: Halloween 10:04:42 Meeting: #testing gathering @ TPAC 2012 10:04:46 Chair: PLH 10:04:53 plinss has joined #testing 10:05:10 AnssiK has joined #testing 10:05:15 fantasai has joined #testing 10:05:28 plh has joined #testing 10:05:46 plh: I'm in the digital publishing room, if you think you need me, feel free to pull me 10:05:52 Present+ Art_Barstow 10:06:00 krisk has joined #testing 10:06:12 adambe has joined #testing 10:06:45 https://github.com/w3c/testing-how-to 10:07:39 Wonsuk has joined #testing 10:07:49 plh: there are two other testing related sessions today 10:07:58 … http://www.w3.org/wiki/TPAC2012#Session_Grid 10:08:55 I can scribe... 10:09:02 scribenick: krisk 10:09:02 SimonPieters has joined #testing 10:09:10 Marcos has joined #testing 10:09:14 caribou has joined #testing 10:09:18 bryan has joined #testing 10:09:18 Scribe: Kris 10:09:19 arronei has joined #testing 10:09:26 giuseppe has joined #testing 10:09:26 RRSAgent, make minutes 10:09:26 I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB 10:09:26 Present+ Anssi_Kostiainen 10:09:29 jet has joined #testing 10:09:35 bryan has left #testing 10:09:48 http://w3c.github.com/testing-how-to/ 10:09:49 RRSAgent, make log Public 10:09:52 10:09:56 RRSAgent, make minutes 10:09:56 I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB 10:10:20 present + Simon Stewart 10:10:21 present+ kris_krueger 10:10:22 present +rhauck 10:10:23 darobin has joined #testing 10:10:24 present+ giuseppe 10:10:25 Present+ Wonsuk_Lee 10:10:26 present + Jet Villegas 10:10:26 Present+ Peter Linss 10:10:28 bryan has joined #testing 10:10:35 present+ jgraham 10:10:35 present + Larry McLister 10:10:35 Present+ Simon_Pieters 10:10:36 present+ JohnJansen 10:10:38 present+ adambe 10:10:39 present+ Wilhelm Joys Andersen 10:10:42 present+ Bryan_Sullivan 10:10:43 =present+ Mark_Vickers 10:10:44 present +Arron Eicholz 10:11:10 plh talking about overview - starting with 'what do you test in a specification?' 10:11:15 Zakim has joined #testing 10:11:20 ht has joined #testing 10:11:40 JohnJansen has joined #testing 10:11:57 jet has changed the topic to: TPAC testing sessions 10:11:59 Present+ Robin Berjon 10:12:08 present+ Henry S. Thompson 10:12:14 present+ JohnJansen 10:12:15 jgraham - the #github testing talk will talk about improvements in our test infrastructure, requirements for tests, etc.. 10:12:24 Present+ Marcos Caceres 10:12:31 RRSAgent, make minutes 10:12:31 I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB 10:12:32 MichaelC_ has joined #testing 10:12:33 plh - "what do you test in a specification?" 10:12:40 present+ Michael_Cooper 10:12:51 robin has joined #TESTING 10:13:03 plh - it's always not obvious what needs to be testing, for example webidl exists that needs to be tested 10:13:06 acolwell has joined #testing 10:13:20 plh - if it's going to have an impact on implementations needs to be tested 10:13:43 plh - MUST, MUST NOT, SHOULD , SHOULD NOT, MAY are all items that should be tested 10:13:44 kotakagi has joined #testing 10:14:38 Lachy: just pinged him 10:14:46 Lachy: what room are you in? 10:14:50 plh - mutliple tests can be created for a single statement from the spec 10:14:56 Marcos: RHONE 2 10:15:18 jeff_ has joined #testing 10:15:19 plh - it's complex and not automated.... 10:15:33 Part #2 for the talk "How to write a test?" 10:15:52 First mechanism is javascript test 10:16:06 second test mechaism is what is known as a REF test 10:16:19 ...used for rendering tests 10:16:35 third mechanism is 'seff-describing' tests 10:16:58 ..you don't want to do this by default, since it's expensive 10:17:09 ...though at times this is the only mechanism 10:17:22 MikeSmith has joined #testing 10:17:31 MikeSmith_ has joined #testing 10:17:59 For script based tests we use testharness.js 10:18:06 … http://w3c-test.org/resources/ 10:18:15 s/seff-describing/self-describing 10:18:25 Check out the api examples... 10:18:39 http://w3c-test.org/resources/apisample.htm 10:18:51 http://darobin.github.com/test-harness-tutorial/docs/using-testharness.html (testharness.js tutorial by Robin Berjon) 10:18:54 http://w3c-test.org/resources/apisample2.htm 10:18:56 http://w3c-test.org/resources/apisample3.htm 10:19:55 present+ Tobie_Langel 10:20:09 Two main use cases - asynchronous (e.g. events aka onload) and synchronous 10:20:32 ..a number of assert_* exist to validate the results 10:20:45 plh showing the test examples for each... 10:22:32 some cool asserts... 10:22:35 glenn has joined #testing 10:22:50 assert_unreachable - makes sure an event DOESN'T fire 10:22:51 wonder what approx_equals does 10:23:16 assert_readonly, assert_idl_attribute! 10:23:39 assert_array is also very helpful as wel... 10:23:46 s/wel/well/ 10:24:04 krisk: how do you assert the negative? 10:24:14 assert_false 10:24:15 Just assume it doesn't fire within a certain amount of time? 10:24:24 Sorry: "assert_unreachable" 10:25:40 you can timeout at the 'page' level and at the test level (you can have more than one test per page) 10:26:28 ..adding metadata to the test is very good when you want to get an understanding on what parts of the spec have coverage 10:27:06 see slide 18 from the deck on https://github.com/w3c/testing-how-to 10:27:17 ..now on slide 19... 10:27:26 ..what are the pitfalls? 10:27:47 It important that we get the same number of test pass/fails 10:28:06 DO NOT stop running tests because something is not supported 10:28:30 documentation in the source: http://w3c-test.org/resources/testharness.js 10:28:58 PLH see slide #20 if you can follow along just see the link above 10:29:04 http://darobin.github.com/test-harness-tutorial/docs/using-testharness.html 10:29:38 Now on Reftest 10:29:43 what is a reftest? 10:29:56 It's two pages that should result in the same rendering 10:29:58 are there docs on how each browser runs reftests? 10:30:58 http://www.w3.org/TR/webdriver/ 10:31:17 http://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-spec.html 10:31:45 tpacbot has joined #testing 10:31:56 jet: I dont think we have docs, but our system is a little different to yours 10:32:03 We can talk about it 10:32:16 jgraham: thx 10:33:27 The test web server has php support, so you can add a delay or headers, etc.. 10:35:14 hta has joined #testing 10:35:29 We have media and media files as well that can be used... 10:35:41 we have codec agnostic support files as well 10:35:51 jalvinen has joined #testing 10:39:50 W3C Test Suite Licenses 10:40:09 JohnJansen has joined #testing 10:40:17 RRSAgent, make minutes 10:40:17 I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB 10:40:44 See slide #33 10:41:15 Now on to Mark Vickers 10:41:20 http://lists.w3.org/Archives/Public/public-web-and-tv/2012Oct/0017.html 10:41:31 Who will talk abou the Web and TV Testing Task Force 10:42:04 The recharter is going to be updated so that this WG has a Testing Task Force 10:42:57 Testing Use Cases 10:43:16 #1 Currently Verification of the specification 10:43:40 #2 New Improve the consistency of the open web platform (OWP) 10:44:14 #3 New support external testing/certification organizations 10:45:24 Number of group do this today - for example DLNA 10:45:49 When it comes from HTML5 we want to have these tests come from the w3c and not 'fork' the tests 10:46:18 #4 Support testing devices 10:46:54 seo has joined #testing 10:47:13 acolwell has joined #testing 10:47:27 sungok_you__ has joined #testing 10:56:24 summary of this session! 10:56:44 tobin can you give summary? 10:56:57 One person states - no testing is enough 10:57:17 uh, no "amount" of testing is enough? 10:58:31 Wonsuk has left #testing 10:58:54 RRSAgent, make minutes 10:58:54 I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB 11:00:08 jalvinen has left #testing 11:01:07 shepazu has joined #testing 11:04:01 giuseppe has left #testing 11:04:19 RRSAgent, bye 11:04:19 I see no action items 13:35:46 RRSAgent has joined #testing 13:35:46 logging to http://www.w3.org/2012/10/31-testing-irc 13:35:51 Mark_Vickers has left #testing 13:35:51 ph has joined #testing 13:35:53 Mark_Vickers has joined #testing 13:35:58 Abel has joined #testing 13:36:01 divya has joined #testing 13:36:04 Title: #github session @ TPAC 2012 13:36:10 Date: Halloween 13:36:18 Chair: James Graham 13:36:25 plh has joined #testing 13:36:26 bryan has joined #testing 13:36:26 [missed a bit] 13:36:27 ScribeNick: divya 13:36:34 Scribe: Divya 13:36:34 Lachy has joined #testing 13:36:39 Starting with Source control which was the original motivation for this session 13:36:41 present+ Mark_Vickers 13:36:50 Lachy has joined #testing 13:36:51 jgraham: atm we are running custom w3c instance of mercurial 13:36:54 present+ Art_Barstow 13:37:25 …we ahve all these repos, in order to contribute tests to that you need to learn how to use mercurial, find the repo, add your tests, commit & send a mail to mailing list 13:37:42 RRSAgent: make logs public 13:37:58 …one suggestion has come up is if we are better off outsourcing part of problem, in particular github 13:38:20 (http mirror for webapps' test repo: http://w3c-test.org/webapps/) 13:38:23 …its git whichi is very similar to mercuirial hosted service nicer UI, and has lot of developer buy-in 13:38:32 …all big libraries are hosted on Github 13:38:42 …has a different workflow which is slightly better 13:38:53 isra has joined #testing 13:39:02 darobin has joined #testing 13:39:05 [shows example of testharness.js] http://github.com/w3c/testharness.js 13:39:19 …they have this mechanism called pull request instead of sending mail off to a mailing list 13:39:34 …it gives you a way of tracking various people who want to make changes 13:39:43 …you do not have to go out to mail, mailing list and different systems 13:40:06 …do sy yhid poiny it would be interesting to have a discussion about whether people think this is a good idea. 13:40:14 s/so at this point 13:40:33 kris: there has this whole patent policy stuff that needs to be addressed 13:40:36 q+ 13:40:53 darobin: 1st there is contributions to spec, chair makes a call on whether the contributions has done IP properly 13:41:25 darobin: separate from contributions to test suite. they are fairly standard open source license and you can get anyone contributing to test suite without IP consequences for the royalty free aspect ot the spec 13:41:38 kris: do we make people sign to this to contribute to the spec. 13:41:48 kris: cable companies think there would be a lot of money in it. 13:42:02 darobin: W3C is not allowed to publish test suites in anything other htan open source license 13:42:12 kris: today you cant add tests to w3c without signing some stuff 13:42:24 darobin: that is manageable, you just need a list of people who signed off. 13:42:33 darobin: i do not see it as an issue. 13:42:34 q+ 13:42:36 q? 13:42:51 Zakim has joined #testing 13:42:53 Marcos has joined #testing 13:42:58 q+ ArtB 13:43:00 masinter has joined #testing 13:43:00 Marcos1 has joined #testing 13:43:03 q+ Mark_Vickers 13:43:06 q? 13:43:11 ack ArtB 13:43:20 rhauck has joined #testing 13:43:20 ArtB: i am indifferent, beggers cant be choosy might dominate here. by we i mean webapps anyhow. we will take tests 13:43:27 ArtB: we should provide a way for them to contribute 13:43:47 ArtB: if we are chasing another shiny object. in this case github. i am not convinced what we have right now is broken enough. 13:44:05 darobin: how often do you use the mercurial system 13:44:09 ArtB: probably a few times permonth 13:44:17 darobin: i think that maybe why you do not see the problem 13:44:44 kris: i dont believe everyone moved ot github, we still have stuff coming out of CVS. we have to learn multiple systems 13:44:49 kris: that is a valid thing 13:44:59 darobin: if we want outside contributions we can give up on using mercurial. 13:45:10 kris: i do not think it is really very inviting to contribute. 13:45:35 kris: even geeks have a hard time to contribute to github(?) 13:45:47 jgraham: i think robin said on wiki page, it is about github not git 13:46:28 jgraham: the point is because there is an existing community around github and much broader than w3c, the existing pool of users who know what the workflow is, then they make pull requests. 13:46:41 darobin: in developer world, if it does not exist on github, it does not exist. 13:47:08 Marcos: i have seen this in responsive images. amazing to get that workflow, getting the intial flow is hard, once you ahve the community happening, it is hard. 13:47:52 marcos: what our community here does is important. are they comfortable going through this. it is not as tedious as doing the manual merge on the command line. github does this automatically, and transparent. commenting appears real time. 13:48:23 darobin: the way we use mercurial in w3c setup is not correct. we have central repo for something that is supposed to be distributed, we are using it as cvs and it is easy to destroy content this way. 13:48:52 [random discussion] 13:48:53 q? 13:48:53 q? 13:48:57 ack Mark_Vickers 13:49:21 Mark_Vickers: on the legal thing, my understanding is that github repo is you can put a CLA that covers a particular project. 13:49:26 AnssiK has joined #testing 13:49:36 q+ 13:49:38 darobin: there is an approval stamp the 1st time you contribute. 13:49:58 Mark_Vickers: it has value to have a large number of tests in terms of saving money for peoople who create content. 13:50:02 btw the way dvcs.w3.org is set up right now it's possible for a push to create multiple heads 13:50:06 Mark_Vickers: not interms of saving money. 13:50:11 q? 13:50:16 kris: i dont think we can say it is all free and open. 13:51:09 ack plinss 13:51:12 jgraham: you can see who the contribution is coming from, so you can say if they should fill the form or not. it is no different than when people send us a patch via email, and that you commit the patch without checking if they signe the right agreement 13:51:45 plinss: i am in favour of crowdsourcing the tests, i certainly get the fact that github does that with devs, for some definition of devs. The devs that github targets are they contributing to test suites. 13:52:07 plinss: the lot of folks who show up at test the web forward are designers not coders. 13:52:28 darobin: my experience of test the web forward is paris. They tried to use dropbox, it didnt work and then they switched to github. 13:52:45 stearns: at SF #1 question was why we are not using github. 13:53:06 lmclister: or what is the mercurial equivalent for the git command 13:53:09 q+ to ask whether this is necessarily a choice, or whether there's some workflow where things can be staged 13:53:19 plinss: i am concerned like ArtB that we are chasing the next shiny thing. 13:53:35 plinss: i would rather see w3c be the cool place for this stuff. 13:53:44 marcos: cost of setting up infrastructure is huge. 13:53:59 I wish somebody from the w3c systems team were here to comment 13:54:00 Marcos1: how many years do we keep doing the same thing. 13:54:09 plinss: i have snever seen a migration step not lose something 13:54:20 plinss: i have never seen a pre-baked thing that delivers to all of our needs. 13:54:26 q+ 13:54:26 darobin: we may have fewer points of pain 13:54:35 plinss: something else might be the point of pain 13:54:37 q? 13:54:45 jgraham: i am not claiming github is perfect at least it has an api that we can build on 13:54:55 q+ 13:55:01 …if it does turn out it is not optimal we can just replace that bit of it. 13:55:08 plinss: what if github goes down 13:55:14 q+ how would this work http://w3c-test.org/framework/app/suite in the github world? 13:55:20 jgraham: certainly we should have w3c hosted read only clones of repo 13:55:35 jet_ has changed the topic to: github for w3c tests 13:55:36 plinss: github has the apis to developp tools, if we spend times building tools on top of infrastructure 13:56:09 jgraham: someone at opera wrote a code review tool that was open sourced on monday. I think it is orders of magnitude better than github one, and it might be worth looking at that. 13:56:19 q? 13:56:21 https://github.com/jensl/critic/ -> Critic code review tool (from Opera) 13:56:29 …might be worth making it transparent such that we can use that tool rather than whats on github 13:56:41 [missed convo] 13:56:43 q- 13:56:53 s|https://github.com/jensl/critic/ -> Critic code review tool (from Opera)|-> https://github.com/jensl/critic/ Critic code review tool (from Opera)| 13:56:59 jgraham: there may be different design goals 13:57:10 plinss: shepherd was designed to be a test review system 13:57:23 jgraham: if it gets us an awesome user experience then, but it is not there yet. 13:57:25 q+ to ask: how would this work http://w3c-test.org/framework/app/suite in the github world? 13:57:29 plinss: anything we build is going to take time 13:57:36 plinss: rather than rely on someone else's infrastructure 13:57:45 plinss: i have no predictability on github infrastructure 13:57:57 darobin: as long as the people who wrote the code are alive & around to maintain it 13:57:59 q+ 13:58:08 kris: stuff on w3c lasts longer than it lasts on github 13:58:21 darobin: 10 years ago sourceforce was the only option and people hated it. 13:58:35 darobin: anything out there would be synchronized tow3c 13:58:52 darobin: 1. one side is git 2. other side of tooling is github api, which maps very closely to git. 13:59:13 darobin: i am not too worried about that, there exists at least 1 open source implementation that exposes same api on top of git repository 13:59:15 q- 13:59:31 krisk: how long have we had stuff on this site so far? 13:59:34 jgraham: which site? 13:59:36 krisk: github 13:59:41 Present+ Odin_Hoerthe_Omdal 13:59:51 jgraham: well the htmlwg is readwrite. that is what they are using as primary interface 13:59:59 jgraham: test stuff is mostly readonly. the html test suite is readonly. 14:00:05 jgraham: testharness.js is readonly here. 14:00:19 jdurand has joined #testing 14:00:20 …webperforamnce is read only event source is read-write doesnt exist anywhere else. 14:00:29 …amaya is readwrite nowadays 14:00:48 …this is the official w3c github repo there is a lot more w3c related github things that are not part of the official one 14:00:58 darobin: there is actually a lot more content than this. 14:01:07 krisk: isnt that a problem? wont people have hard time giguring out 14:01:17 darobin: ideally i want all the official stuff to go to the official thing. 14:01:32 RRSAgent, make minutes 14:01:32 I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB 14:01:36 darobin: i would encourage groups be consistent and make their info more easy to find. 14:01:43 q+ 14:01:51 jgraham: it is a problem we would solve if we say this is the prefered user interface 14:02:04 MichaelC_ has joined #testing 14:02:23 darobin: what has happened today is people join WG they are told mercurial is the way to do, lose data once, and hten they get rustrated and they do not know w3c has a github account. 14:02:27 q? 14:02:30 ack krisk 14:03:02 Marcos1: i am one of the people who has test suites in public account not on w3cc account. It looked like i didnt know who to contact. 14:03:09 jgraham: the process at the moement is to ask MikeSmith 14:03:11 darobin: or me 14:03:33 darobin: if you look at the members anyone there who is w3c should normally have the ability to add you to the organization 14:03:49 ack ArtB 14:03:50 ArtB, you wanted to ask: how would this work http://w3c-test.org/framework/app/suite in the github world? 14:03:53 http://w3c-test.org/framework/app/suite 14:04:02 ArtB: i am oppossed to [] work. 14:04:25 ArtB: we have some tests in webapps. in the scenario where webapps would have some tests in mercurial etc 14:04:34 ArtB: would framework accomodate both? 14:04:37 darobin: not a problem 14:04:48 s/[] work/makework/ 14:04:54 RRSAgent, make minutes 14:04:54 I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html MikeSmith 14:05:01 ArtB: do you have WG that are like ground zero, that have already gone to github 14:05:08 darobin: the htmlwg did that for spec editing 14:05:19 Marcos1: DAP did it 14:05:33 krisk: is it not on CVS for html spec? 14:05:45 q+ 14:05:50 darobin: there is a tool that takes the github source, compiles into html thing and then checks into cvs 14:06:00 …sam wrote the tool in under 15 mins 14:06:18 krisk: if someone starts from ground zero, we have other infrastructure how does w3test.org work 14:06:36 darobin: i maintain respec on github, whenever i ship a build, it automatically gets synched to w3c server 14:06:52 jgraham: w3test.org does perl update at the moment changing to git pull owuld be trivial 14:07:00 SimonPieters: whatwg specs are developed on github. 14:07:09 q? 14:07:11 ack isra 14:07:40 isra: should w3c ever decide to use github, i think we should protect our members find some sort of legal protection in case they change user agreements, what happens when they change agreements. 14:07:48 isra: it is a fertile ground for some issues to come up. 14:08:00 krisk: i would +1 that, for microsoft, it is not a normal thing. 14:08:07 darobin: plenty of ms people put stuff on github 14:08:13 krisk: there are some things in place that is different tho 14:08:41 isra: w3c has enough power to impose some conditions. 14:08:49 darobin: disappearing data is not a problem 14:09:06 darobin: you are thinking the case what happens when they become evil. 14:09:12 masinter: if you can sync why is there an issue 14:09:41 plinss: it is not about the repo itself. the issue is tooling around the repo, we will be married to what we are building on top of 14:09:50 masinter: github has an api. 14:09:58 plinss: dont confuse git and github and mercurial 14:10:08 isra: its about legal terms of use 14:10:12 isra: not about synching data 14:10:19 [side conversations] 14:11:00 masinter: most tools seem to be for running tests than submitting them 14:11:04 plinss: in csswg we have both 14:11:08 q? 14:11:28 plinss: we have sheperd code it runs, finding errors giving data right back into client. 14:11:48 plinss: comments reviews happen on shpeherd. if push comes to shove i can move those to github. 14:11:53 q? 14:12:15 …so i dont defend the code per se, there is a lot of sense that the methodology runs on same server 14:12:24 …validates test files, etc that will soon move to CI system 14:12:34 …there is lot that is either built or not too far away from being built 14:12:38 masinter: based on mercurial? 14:12:41 plinss: yes 14:13:05 davidburns has joined #testing 14:13:08 plinss: at that time w3c was doing to svg, and then it appeared like w3c was going to standardize on mercurial 14:13:18 darobin: it is not w3c the orgn it is just feedback the people 14:13:41 q? 14:13:46 ack bryan 14:13:55 bryan: i am using github to ocllaboratively edit specs 14:13:58 s/doing to svg/using svn/ 14:14:02 bryan: should that be under w3c account 14:14:17 s/ocllaboratively/collaboratively/ 14:14:23 bryan: should we try to harmonize the least, sign some sort of community power agreement with github. 14:14:30 bryan: it is definitely easy 14:14:37 bryan: compared to cvs which was horrible 14:14:52 BTW, Shepherd: http://test.csswg.org/shepherd/ 14:14:52 …mercurial is okay, i dont know about people stepping on top of each other. certainly github is easy. 14:15:27 …if we didnt get tests here from outside community. we still need some kind of system that encourages participation. as long as we have some sort of scripted commands that allow tests to come in. 14:16:06 wilhelm: i am mildly in favor of this change. all the front end devs i know use this service and github has 2 million members. how many people in this room have a github account 14:16:09 [majority] 14:16:25 q? 14:16:29 wilhelm: that is an interesting datapoint 14:16:32 ack wilhelm 14:17:07 plinss: it doesnt matter what the people in this room are using. if we are trying to engage a broader audience, what are they using? is this tool a barrier to entry to people 14:17:26 stearns: if crowdsourcing the test is the point, then we are going to be chasing the shiny 2 years from now 14:17:45 jgraham: it is hard to become the shiny thing because we do only 1 thing. github you can do all the other htings. 14:18:01 jgraham: it has a lot of uses so it spreads through the community quickly 14:18:01 q+ 14:18:32 jgraham: so one other problem we have had sort of had solutions, is we have not had good code review of tests or testing tools 14:18:44 …csswg's sheperd does not really work for code review 14:19:02 …demo the tool we use @ Opera which was open sourced on monday on github, maybe it is interesting to other people 14:19:08 [demos] 14:19:21 q+ to ask if critic has an API 14:19:43 jgraham: each review is a branch in the repository 14:19:54 q? 14:20:05 [shows eg of a review for testharness.js] 14:20:20 jgraham: shows if anyone has reviewed it or what % of files have been reviewed. 14:20:42 …diff view old code on left new code on right 14:21:14 …can create an issue on each line 14:21:30 Opera Critic is https://github.com/jensl/critic 14:21:32 …i can then see the issues that have been raised 14:22:11 …the author would fix the issue and push another commit to the branch that fixes the issue. The Critic tool would notice that the change in the line was for the issue, and then notify people on the update. 14:22:14 ph has joined #testing 14:22:15 (paraphrasing htere) 14:22:36 …when you have changes you keep pushing htem, when review is done you merge the branch 14:23:02 …without trying it, it is hard to explain how efficient that is. we tried other tools at opera. We tried reviewboard for e.g. it was a disaster and that is why we ended up with this tool 14:23:21 …for test cases we may have slightly different process that may not be supported by this tool. e.g commit to main repo rather than commit to a branch 14:23:43 …for testharness or other tools it might work really well. I am going to try it on github and use it instead of github review tool. 14:24:02 …github does not allow you to comment across multiple lines, squash multiple commits and review together. 14:24:13 SimonPieters: we use this for opera source and test suites that we want to release at opera. 14:24:55 jgraham: until the issue is addressed it will not markt he code as ready to be merged. 14:25:11 jgraham: probably enough sales pitch if people want to look at it. it is on github like everything else. 14:25:27 darobin: does it have an api, so if you want to list issues somewhere else… 14:25:38 jgraham: it has ability to write extensions, so you could write an API 14:25:42 darobin: it would be logical 14:25:53 jgraham: i think someone wrote an extension that creates a JSON dump of the issues 14:26:06 …that is all i wanted to say. Does anyone else have anything to say? 14:26:15 q? 14:26:35 odinho_: another thing is when review is accepted, push a button and takes it where it is supposed to go. 14:26:41 darobin: merges it to master? 14:26:48 q+ to ask about testing infrastructure for HTTP, URL tests needed more well-known sites, etc. 14:26:56 ack odinho_ 14:26:57 odinho_: merges to specific branch 14:27:06 SimonPieters: …and resolves a bug in the bug system 14:27:07 ack darobin 14:27:07 darobin, you wanted to ask if critic has an API 14:27:12 isra has left #testing 14:27:22 http://redbot.org/ 14:27:33 odinho_: it is the same git server we use for everything. when it was put on github some people started pushing patches there because it was very visible. 14:27:33 RRSAgent, make minutes 14:27:33 I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html ArtB 14:27:58 jgraham: there were people who had access to the sourcecode for over a year who never made a patch, and suddenly after it was on github for a day, made the patch 14:28:04 darobin: same thing for respec 14:28:14 jgraham: did anyone want to write up a summary of this session? 14:28:34 q+ 14:28:40 masinter: i want to ask about other parts of testing infrastructure. 14:28:53 masinter: e.g. testing uris, nice to have DNS wildcard 14:29:05 masinter: i see there is more to testing the web than just testing css. 14:29:32 jgraham: i agree with you and i think we need to have conv about server side testing infra. i do not think we have the right people in the room to have that conversation 14:29:52 masinter: i sent an email to public-test-infra mailing list and didnt get any response. 14:30:19 masinter: i think assumption seems to be that testing happens in wg 14:30:32 Marcos has left #testing 14:30:38 krisk: the websocket one was extremely painful. 14:31:10 jgraham: for specific tests, we invoke the MikeSmith protocol again 14:31:31 [krisk says something about vm] 14:31:48 q? 14:31:52 bryan: how close are we to clone this framework thing and run behind our firewall? 14:31:56 darobin: we are not there yet. 14:32:00 ack masinter 14:32:00 masinter, you wanted to ask about testing infrastructure for HTTP, URL tests needed more well-known sites, etc. 14:32:08 darobin: can i give an update on this in 2 months? 14:32:26 bryan: a proposal in coremob take a half-step forward by hosting coremob test somewhere else. 14:32:36 darobin: github might help with as it has an api that makes it easy to pull data out. 14:32:45 darobin: we will figure that out. 14:32:54 bryan: esp for network operators its very important. 14:33:05 ack bryan 14:33:18 RRSAgent: make minutes 14:33:18 I have made the request to generate http://www.w3.org/2012/10/31-testing-minutes.html divya 14:37:30 RRSAgent, bye 14:37:30 I see no action items