SV_MEETING_TITLE -- 29 Jan 2013

Robin: One thing I can easily do is list tests-per-section and contrast that with simple heuristics (normative statements, lines of algorithms).
... And probably a few other metrics.

plh: What I'd like to do with the data when it comes back to us is go around this group again and see who can sign up to do what.

<Mark_Vickers> Please add these six specs to the Common spec list: CSS Selectors, CSS Media Queries, Web Messaging, CSS Image Values and Replaced Content, CSS Text, CSS Values and Units

Bryan: Assessment and prioritization, a 2 step process.

Jeff: What was the task Robin just got? How many tests HTML5 needs total? Or how many we have?

Robin: Both. Find what we have, and what relationship to an estimate of what we need.

Jeff: Expect in 4 weeks 2 numbers of the form: need X tests, have Y already, most in browser test repositories

Robin: I wouldn't expect to go through browser repositories in 4 weeks.
... We can compare number of tests per section against the size of section.

Jeff: Can browser vendors help by looking at tests that already exist.

Alan: It would be easier for the browser vendors to identify tests for a specific section and ask if they have tests for a particular section.

Jeff: Would you be able to do the identification early enough that you can hand browser vendors the list in 2 weeks?

Robin: If I put it at the top of my list I can come up with a rough and ready estimate in 2 weeks.

Rebecca: If it's per-spec, I could do transforms.

plh: We were going to start with HTML5....

Jeff: But you had a volunteer...

plh: that would be great
... Jet, Mike, Paul, David: could you answer whether you have tests for X in a few weeks?

Jet: We can respond to that email.

plh: ...

mcham: What are we assuming about shims or conversions?

Robin: Really depends on the browser vendor

lbolstad: Should also add opera to the list of people to send email to

Jeff: In your response, you could say that we have tests to cover the section, but we'd need the following types of shims to adapt it.

Robin: Could, e.g., send sample tests and we could figure out what's needed for shim.
... Opera has made all of their tests available, but the vast majority not integrated into the current repository.
... Some are in submitted section and not in pull requests ,others on public server that Opera has made available but not integrated into repo.
... IT's a lot of work to go through all Opera's tests and integrate into the repo.
... They don't necessarily map onto the structure we have even though they do map on using testharness. Would be great to have time from James for doing that.

lbolstad: Opera will happily assist in converting our tests

plh: Any volunteers for any other specifications that were on the list?

Bryan: We have a list, I have a working analysis of where we have tests against coremob 2012. But they all have to be covered.

Robin: While I'm doing html I can do canvas.

plh: I'm wondering if you're going to come back and say "we have 0 tests for X" or "we need Y tests"

Bryan: Both. Find the state, then talk about the need.
... Then what we tackle is prrioritization

Tobie: Format... what do we need to measure? per-section vs global across spec, percentage vs. ???

plh: I think needs to be per-section at the minimum.

Tobie: We need to agree what it is and standardize it.

plh: How many tests we have and what we think we need, per-section.

Tobie: How deep within sections?
... high-level sections or each heading?

plh: I think the deeper the better

Tobie: agree on a format and a place to store it?

Robin: now?

Tobie: Before everyone goes and does it

Alan: I think we should try starting and iterate, on the list.

Bryan: Earlier point about thing on prioritization. All CSS stuff in a bundle, and I'll need to talk to you about it... whole bunch of thing sfor which we have no tests.

Tobie: I'll see if I can have a look at workers (soft commitment).

plh: One thing I'm curious about: we talked to vendors about being able to produce some tests, and they came back with various numbers.
... How realistic would it be, once we know what we need to produce, should we try throwing money at them to write some tests? Or hiring some engineers to write tests?
... And probably browser vendors were exposed to tests produced by such a vendor.
... Was it a good experience?

<Zakim> kaz, you wanted to ask whether we should ask the other browser vendors about their tests or not

kaz: I was wondering if we should include ???

Jeff: [list of other people who might have tests]
... I wanted to support the idea of having multiple tricks in our arsenal and of managing outsourced vendors very carefully.

<inserted> kaz: will try to create a list

Jeff: I suspect that by the time we're done there will be a large number of testcases that have to be written or transformed.

<fantasai> plh, it depends highly on who you're hiring.

Jeff: With a large programming task... if members want to create a ... I don't see W3C staffing this in the long term. Help from the outside is something we could do (maybe from lower cost countries). Don't want to take off-the-street programmer who's never heard of the technology before.

<fantasai> plh, if they're competent, then it's good! if not, then it's a lot of time spent reviewing and rewriting the tests. (I've had both experiences with contractors hired by HP and Mozilla, respectively.)

Jeff: I'd also want to report on experiences at CES; met with some companies, and discovered that it's not the case that we're just finding unvarnished programmers off the street who don't know about Web technology. They shared description of some testing they'd done with a number of vendors across the world related to Web technology.
... I was surprised that in each case they showed me the HTML5 set top box that they had each developed. They're starting to invest in our technology, not just doing outsourced testcases. But also every organization has it's stars/needs to be managed carefully.

<fantasai> plh, familiarity with Web technology is not enough; you also have to be able to think like a tester

<fantasai> plh, and read specs meticiulously

plh: Thing I forgot earlier -- improving testharness.js. Bottom line is working with whoever from Mozilla, Microsoft??,

mcham: Who'd step up with improvements, to make it easier for you to submit tests.

?: and James from Opera

Tobie: I'm happy to do a more-specific effort on testharness.js

Jet: sounds fine.

Tobie: or fine within other task force

plh: Also people from WebKit side...
...

Alan: Two issues: Make it easier to upstream existing tests, and make it good enough that vendors want to use it for their internal tests.

Tobie: Could we have commitment from browser vendors to help with this?

plh: jet said yes, paul will try to find someone

Tobie: Can I have poinst of contact, Jet, Paul, hober, ...

fil: ... cordova ...

mcham: Kris for MS

Mark: Look at this whole project... key resources, engineers. Other companies who may not have the right engineers, but might be able to provide some money.
... We have a large number of members, I expect more who could contribute money than engineers.

Robin: And we know of at least 2-3 freelance people who would be good.

Mark: I can't go and ask for money unless there's a plan.
... I'd encourage that.

Bryan: If we have a proposal for specific things that could be sponsored, we can consider them.

plh: That's the point of these task forces.

Jeff: I appreciate offer from Mark and Bryan for financial support; agree we need to size the proble mfirst. Typically what we do when we have an additional effort above and beyond the normal effort, we'll send out notice to the AC and get a sense for that. Preparatory work is helpful.

<glenn> 10M is probably a better estimate

Jeff: Earlier today, Philippe said it's going to cost $1M. Then people said there are tests around, or added other things we need to do. How you approach that from a funding perspective...
... If there's any conversation among the membership that's useful in the next few weeks, or should we hold off after reconvening in about 4 weeks?
... In my mind you represent large industries that have an interest in this but not necessarily a lot of engineers to contribute.

Mark: I've asked up already some, want to see a plan.

Jeff: Talking about communicating more among membership.

Mark: I have a list of overlapping members who might be interseted.

Bryan: If we want to outline a program, and begin to share that with stakeholders to get their feedback, I think that'd be valuable.

Jeff: To be very practical, I think that from the day that some AC rep wants to help with such a fund... that's when they start the conversation inside their compnany, but that takes time.

Dan: I talked with my management... willing to help with test running... want proposal and ballpark estimate.
... Some vendors not able to give reasonable proposals. So looking to W3C for a concrete plan we can take to management.

<darobin> [I wonder if some companies that have engineers but not necessarily with the required knowledge would be interested in training around this]

plh, we know some vendors we could get price estimate from... not saying we should go to vendors for the plan

<fantasai> Wrt any infrastructure we build... let's make sure to get some UX people involved up front!

<fantasai> We've got people like plinss who can think through backend architecture, but the front-end architecture is also important...

Yosuke: Quick comment about financial support: test ??? project. Several options: hiring programmers, money to hire. We'd also like to see and discuss task force plan and if it's a good option we'd support.

<Zakim> darobin, you wanted to mention training

plh: ...

Robin: An idea in case it's interesting: I've noticed in speaking to companies about testing, that in some cases companies do have engineers that they'd be willing to dedicate to testing, but those engineers understand Web tech but don't have knowledge about how to write spec tests.
... Wondering if we could provide some training to train engineers... send somebody to company to do training for writing tests?
... Would make it possible to contribute tests afterwards.

Tobie: An enterprise edition of test the web forward?

Bryan: Train the trainers?

Robin: Need a course that people can follow.
... There's a difference between documentation and a course.

<fantasai> Robin: Much cheaper to come up with $5000 to bring over an expert than $50,000 to hire a test-writer. And increases the capacity to contribute tests.

plh: nede documentation first

<fantasai> Robin: [ facilitates cross-company collaboration, etc ]

Dan: What's going to be the next step after the 4 weeks?

plh: Report results to the participants. Then depends on the propsed actions from the task forces.
... Figure out how much it's going to cost.
... ... resources to improve what we have...
... Similar one for test mgmt system.
... for documentation, less clear
... ... what conclusion the documentation tf can come up with?
... What do you expect docs tf to do in 4 weeks?

Robin: I'd expect, maybe in 2 days, to figure out what topics we need docs on as a priority
... And where we're going to host the docs
... Anything that allows someone who wants to write docs to know what to do

plh: One other thing to explore: relationship of this project with webplatform.org

Tobie: I can take that action item

mcham: One reason webplatform.org is not w3.org/webplatform is because it's broader than w3c
... At least webplatform.org is somewhat neutral.
... could be a test subtree parallel to docs subtree; I don't have strong opinions.

Dan: A question: for test framework, were talking about ??? profile. We should be able to repeat tests if there is a failure, vendor may need to rerun to verify. Can take snapshot to repeat the same tests at a different time?

Tobie: I think that's technical details for within the TF.

Mark: On webplatform.org, I think you have to think about communities. For both of these we're trying to get Web app developers as part of the community.
... If we get people to come for Web platform... think about it as a community. Thus at least integrated if not the same thing.

plh: Going back to platform...if we need X for documentation, next task force needs to figure out who is going to write those ??? tests.

Tobie: some people who gravitate around w3c and might be available

plh: Regarding ??? on coverage itself. It's a matter of going around the table and seeing who can contribute tests.
... Then estimating what it takes to write tests.

Tobie: Or maybe people in here willing to lead crowdsourcing.

plh: Sure. I don't want to prejudge methods for coming up with tests.
... Anything else we need to discuss / conclude?

Bryan: I was going to send a note... describe approach.
... We talked about HTML5, CSS, other things.
... As part of coremob CG... we need to determine based on that work how it fits into prioritization in this effort.
... My assumption is it's guiding prioritization of this effort.
... To clarify prioritization is important.

plh: We're gonig to have WGs looking at the results of this meeting.
... Maybe we motivate the WGs to write more tests?

<plh> ack

kaz: Minor question-- which group do the TFs belong to?

plh: Not attached to a WG
... the Web testing IG?

Tobie: what's the point in doing that?
... That would mean I'd need to get legal approval to join the group I'd be going to chair.

plh: I wouldn't expect IP commitments. Not a group under the patent policy.

<bryan> here is a link to the email that I mentioned, further describing the priority approach I recommend for test asset development: http://lists.w3.org/Archives/Public/public-test-infra/2013JanMar/0005.html

plh: Not sure if there's value in doing this. Concept of TF is outside the process. And not working on a REC-track document.

Robin: I think we should use the public-test-infra mailing list but not make it part of the group.

Bryan: We're interested in helping that IG; don't want to chair an empty group.

<Zakim> kaz, you wanted to wonder which group do the TFs belong to

Bryan: I dropped link to the email in IRC.

<andreatrasatti> thank you everyone

plh: Thanks to everyone for coming, Mozilla for hosting at short notice, and to scribes.

Robin: Does anyone object to making the minutes public?

[no objections]

<chaals> ok, have a good evening folks.

<fantasai> bye~

- DRAFT -

SV_MEETING_TITLE
29 Jan 2013

Attendees

Contents

Summary of Action Items