Research and Development Working Group Teleconference

Welcome and logistics

<shadi> http://www.w3.org/WAI/RD/wiki/

saz: please remember Wiki, and please try to put your ideas, coontribute to existing topics, or extend existing topics

sac: we have a new EC funded project, which will be announced soon

saz: it will provide support to this group
... this project will support additional human resources to this group

<giorgio> what is the purpose of that projet?

saz: please provide support
... primarily guidance on accessibility implementation, and guidance on evaluation, related to evaluation task force
... additionallly to contribute to this group
... so there will be additional sources
... more details will follow in the announcements

<giorgio> ok, thanks

saz: happy to answer questions offline
... questions, comments?

CFP Finalisation and Release

<shadi> http://www.w3.org/WAI/RD/2011/metrics/

saz: this is a place holder page

<shadi> http://www.w3.org/WAI/RD/2011/metrics/cfp.html

saz: this is where we announce agenda, cfp, etc

<shadi> http://www.w3.org/2002/09/wbs/1/metrics/

saz: this page has a link to the cfp which we have been working on
... there is also a link to call for participation
... it has details of paper submission

<shadi> http://www.w3.org/WAI/RD/2011/metrics/template.html

saz: it also has a link to the template, which marke, giorgio and josh provided
... I have been getting them reviewed internally, talked to legal person about the licensing
... we will be working with them, taking their contribution and add them to a W3C note, so wanted to make sure we use the right licensing
... it will not be an exclusive right, people can submit it to somewhere else, they can put it on their website, publish it another journal, etc
... those all set

<giorgio> do you have alink to he license?

<shawn> questions on participation, etc.: http://lists.w3.org/Archives/Public/public-wai-rd/2011Aug/0032.html

saz: I also have comments received similar to Shawn's comments, including participation as a speaker, as a non-speaker, etc, we need to look at that more closely
... some people already said they are fine with it
... but I would like to get feedback from all, as we have to announce it soon
... any other comment beside the participation aspect?

mv: why we cannot release it today or tomorrow?

saz: we are ready to annmounce it soon, we might need to polish some small things before we publish it

<markel> ok, thanks

saz: I do expect that we can announce it after the call
... anything else?
... has anybody not looked at the template? I hope you all have

<vivienne> Yes, I've looked at it

shawn: it would be nice to put the license before the submission
... I don't know if it will be clutter the cfp

saz: are you referring to the submission form or cfp?

<shadi> [[put licensing information in the CfP]]

shawn: when I look at the cfp, I have all the info but when I try to submit it, first time I come across with the license, so might not have time to look into this license
... so it would be useful to put it in the cfp, so people can see more about it before they submit

<shadi> [[The Contributor hereby grants to the W3C, a perpetual, non-exclusive, royalty-free, world-wide right and license under any Contributor copyrights in this contribution to copy, publish, use, and modify the contribution and to distribute the contribution under a BSD License or one with more restrictive terms, as well as a right and license of the same scope to any derivative works prepared by the W3C and based on, or incorporating all or part of the contribution.

<shadi> The Contributor further agrees that any derivative works of this contribution prepared by the W3C shall be solely owned by the W3C.

<shadi> The Contributor states, to the best of her/his knowledge, that she/he, or the company she/he represents, has all rights necessary to contribute the Materials.

<shadi> W3C will retain attribution of initial authorship to the Contributor. The W3C makes no a-priori commitment to support or distribute contributions.

<markel> cool

<shadi> Please see the RDWG Frequently Asked Questions (FAQ) for more information on copyrights and attribution.]]

GB: I agree with Shawn, it would be good to put it at the end of CFP, but if the license is not ready I do not want to delay the release of the CFP

saz: license is ready
... we can do this and add it before the cfp is released
... anything else?
... another comment, cfp is long....so how much we can put in the overview page?
... show stopper is the participation, lets discuss that
... participation and format, they are closely related

Participation and Format

GB: format I had in mind is the conference format
... depending on the overall papers, we can have a panel, a panel proposed by different authors, where we discuss a number of discussion points raised by the group

<markel> it would depend on the submissions

<markel> but at most 10 minutes

<markel> In my opinion

saz: what do others think?

<Mario-Batusic> In my opinion also at most 10 minutes

saz: how do we run the actual telecon?
... full day, half a day, 2 hours

?

mv: I agree with the proposed format by Giorgio, I think at most 10 minutes, doing this online is not like the real conference
... we cannot force people to read the abstracts before the meeting
... so the panel would be good
... I agree with the authors and observers

<giorgio> yes

mv: they can both participate in the panel
... two hours is good, I think

saz: can you please explain more?

gb: the idea is that we as the editors, would like compare the results, etc.
... during this panel, I would like to see the differences in perspectives, etc
... sort of lively discussion
... every presented presents for 10 minutes that would be an hour

<markel> presentations would focus more on open research lines

<markel> as the us have read the papers in advanced

<giorgio> could be

<markel> yes

saz: then we will have a panel, editors would have specific questions, then an half an hour for an open QA

<giorgio> panel=guided discusion around a set of predefined and new questions

saz: when you say panel, you mean specific questions come from the chair

I cannot hear vivienne well

:(

<markel> me neither

vivienne: the editors will have the papers/abstracts so the chairs can direct the questions

<giorgio> vivienne, the PC will have read all the submittd papers.

<shawn> vivienne: I think 10 minutes should be sufficient. If we have a lot of people who want to pose questions, could be complicated. Maybe better if discussion question could be decided ahead of time.

<Mario-Batusic> She said: 10 minutes should be enough because the paper is limited at 1000 words only.

<giorgio> no: my was an answer to vivienne

<markel> thank you for the summaries

<vivienne> yes, that's fine

great

saz: as a working we can have input to the questions, I am not sure if we will have one or more panel chairs
... we will also have a lot of input from the group

Mario, I think 10 minutes is more than enough as the papers are short

Mario, we can also limit the no. of accepted papers so we can control the length of the webinar

saz: no. of authors is important

<markel> what's that?

saz: on the one side the way we phrase the participation, we only encourage, may see offputting

<markel> :-S

<markel> :-D

saz: presenters are important
... for example, somebody who is not very strong in academia but has good contribion
... but we have to be careful with the no. of speakers
... with already 10 minutes each, and 5 speakers, we hit 1 hour
... we have to reflect this in the wording of the cfp

<pthiessen> (sorry noise was probably me - in office - overzealous programmer getting a bit to excited about code - hiding in room with door now)

<Zakim> shawn, you wanted to say the expectation that participants read the abstracts beforehand could be part of the registration process. then can spend less time on presentations of

<Mario-Batusic> We should state the acceptance guidleines clearly in the CFP.

shawn: we could say that the participants have to read the abstracts before the seminar, so we could have shorter presentations

<markel> agree

shawn: online presentations are not complelling so 10 minutes would be long

<pthiessen> (agreed about attention span :)

<vivienne> shawn, I like those ideas

<shadi> [[participants to send questions ahead of time]]

shawn: we could even offer participants to send the questions before the event, so that they can organise the panel better

I like these ideas as well

<markel> yeah, good idea: making questions in advance

gb: I agree with shawn
... asking participants to submit questions beforehand would be very good
... selection of the papers, we have already clearly explain in the cfp

<shadi> [[require participants to read contributions in the registration form]]

<vivienne> what happens if one of us wants to submit a paper? Should we abstain in hope of encouraging others?

gb: we send out the cfp, depending on the contributions, depending on the no. of submissions, we can make re-arrengements on the timeslots
... ex, 10 papers 5 minutes each vs. 5 papers, 10 min

mv: I think anybody can submit, as long as they are good quality, it does not matter if they are from academia or not

<giorgio> agree

<Mario-Batusic> agree

saz: some wording here that we need to add to cfp

<markel> shadi, I think the template forces (because of the sections we put) you to put mature work, doesn't it?

<giorgio> i don't agree it would not be useful to authors anyway-

mario: in the cfp, we have said according to which guidelines, but one of the criteria should be no. of papers that could be accepted

<shawn> wording brainstorms: well-developed, mature, well-thoughout, ...

<giorgio> sayning the numbr of acceptd papers

saz: I think it should not be a disadvantage being in the group

<pthiessen> (perhaps external reviewers for group members?)

<markel> again, I would encourge anybody to submit

<shawn> +1 that group participants should be encouraged to submit papers, and then think about how to handle it if there is an issue, e.g., that there are more good papers than there are timeslots

gb: anybody can be an observer, anybody is free to join as the observer

<markel> I don't agree either in setting a number of papers

<shawn> +1 to leave it open how many papers will be accepted -- it depends how many good ones we get :)

gb: regarding the no. papers that can be accepted, I don't think it is a good idea to include this information on the cfp

I agree as well

<vivienne> I agree, putting the number of papers to be accepted may discourage authors

gb: I would invite everybody to submit

<Mario-Batusic> agree

<markel> fully agree with GB

<vivienne> ok

<pthiessen> agree

and then we should trust to the scientific committee to do a good job

<christos> agree

<giorgio> we do not need to chang thecfp

<shadi> [[We particularly, encourage densely referenced, concise contributions based on sound scientific evidence covering work already accomplished, works in progress, and future avenues of investigation. We also seek reports and guides, which can be theoretical or practical in nature, that will help us to form our opinions and educate future readers.]]

saz: I would like to propose this as a small change

mv: first sentence, I agree with that
... not sure about "theoretical or practical" aspect?
... for theoretical work, they need to have strong support
... I would remove "also" from "we also seek reports and guides"

<giorgio> I would leave the cfp as it is now

YY: I would also leave it as it is

<vivienne> I like the first sentence as you changed it, but maybe leave the second sentence alone.

saz: I am concerned about submissions for example a paper that is scientifically sound, but it is not what the scientific committee want
... what would be the justification for rejection then?

<Mario-Batusic> first quality, afterwards first in -> first accepted.

gb: we wil never have 100 objectivity, for example, we all have different criteria in mind
... relevance for example would be low, etc
... we have not organising the only, and the most importance conference on accessibility, we will at most receve 10 or 15
... there will be plenty of uncertainty, etc.
... I don't think it worths to discuss this

can u hear me?

<giorgio> no

sorry

tehcnical difficulty

:(

Shawn, please go ahead

until I sort this out

<markel> it's not contradictory i think

<markel> anybody can submit

<markel> but the acceptance is based on the criteria of the SC

saz: people outside academia might turn away when they read that

<giorgio> that's too strong a statement, shadi. people outside academia do also research.

<Zakim> shawn, you wanted to say do we need be sure to encourage submissions? (can reject them or require that they be modified before acceptance)

saz: how can we protect ourselves from that

shawn: I think we should encourage submissions, there are a lot of people outside academic research that would like to include, etc
... I am little concerned with the current wording, could exclude people that are outside academia

<giorgio> here i disagree with shawn!

shawn: I think the first thing should be welcoming

<markel> I dont' understand what open means

<markel> in this context

<shawn> :-)

saz: we do want high quality input

<markel> http://www.w3.org/WAI/RD/2011/metrics/template.html

mv: I do expect good work. good quality work from practitions
... people who have experience in metrics, they could put their contributions

<giorgio> agree 100%

mv: to me open means anybody can submit as long as its good quality

<shadi> [[We encourage densely referenced, concise contributions based on sound scientific evidence covering work already accomplished ...]]

<markel> that's for position papers

<markel> those that provide empirical evidence shouldn't have so much references

saz: I think this sentence can be scary to practitioners, this could sound too academic
... how do we find a god balance

?

<giorgio> yes

<markel> references are for those that submit theoretical papers

<Zakim> shawn, you wanted to say like template!, but "densely referenced" & sound scientific evidence is scarey for some of us :-)

<Mario-Batusic> Then you need 2 paper styles, Shadi: practitioner and scientific?

shawn: I have to confess, I am one of those
... it will put some people off

<markel> yes I can

<giorgio> ok

<vivienne> yes

<pthiessen> sure

<christos> ok

I have to go

:(

<shawn> scribe: Shawn

<yeliz> thank you Shawn

gb: can provide scientific evidence in different ways. so OK to change text in CfP

<Mario-Batusic> agree

<vivienne> how about "well-referenced"?

<pthiessen> (agree with Shawn from my past work in OS community on a11y work - I suspect a lot of developers would be afraid of many of the mentioned terms)

<christos> Maybe we could use the wording "scientifically sound"...?

saz: agree with shawn that template looks really good

<giorgio> not "well referenced" but "with references"

<shadi> s/sax:/saz:

markel: why did we shoose "densely referenced". we were afraid we would get lots of position papers. we want solid papers, empirical foundations.
... focus on referenced was for people who wanted to submit purely theoritical papers

saz: agree, want high quality. want to be enviting, too. find sweet spot.

<christos> adequately referenced?

<giorgio> i would like to resolve it now

<giorgio> yes yes yes :-)

<markel> "Theoretical papers should be densely referenced while empirical ones are subjected to the data their provide"

perhaps point to the template clearly?

<pthiessen> :)

<giorgio> We encourage concise contributions that are scientifically sound ...

<christos> scientifically sound says it all I think

<markel> +1

<vivienne> +1

<shadi> We encourage contributions that are scientifically sound and that are adequately referenced]]

<shadi> [[We encourage concise contributions that are scientifically sound and that are adequately referenced]]

<christos> we encourage coincise scientificlly sound contibutions

<christos> or maybe properly referenced

<christos> ?

<shadi> [[We encourage concise contributions that are scientifically sound with adequate references]]

+1 to giorgio that we don't want papers full of references, we want papers full of good material :-)

<Mario-Batusic> thats true, Giorgio!

<Mario-Batusic> Better with adequate references

<vivienne> appropriate references?

<giorgio> yes.

<markel> sound references

<christos> apropriate

<shadi> [[We encourage concise contributions that are scientifically sound with appropriate references]]

<Mario-Batusic> appropriate is better.

<Mario-Batusic> ok

<vivienne> ok

<markel> where the adequacy will be measured by the SC

<vivienne> yes

<giorgio> yes

<christos> yes

<giorgio> no

<pthiessen> magnifique :)

yes

<Mario-Batusic> yes

<giorgio> no other tweaks

<markel> it's fine

<giorgio> except for the license

<giorgio> we need to add the license link

<giorgio> sorry: we need what kind of copyleft we think is appropriate.

<shadi> ACTION: shadi to explain copyright terms of the publications and to link to the permission to use (from the WBS) [recorded in http://www.w3.org/2011/09/15-rd-minutes.html#action01]

<trackbot> Created ACTION-3 - Explain copyright terms of the publications and to link to the permission to use (from the WBS) [on Shadi Abou-Zahra - due 2011-09-22].

<giorgio> yes, go ahead.

<pthiessen> sure

saz: do I have permission from the group to add that and publish and announce? or do people want to review it first?

<vivienne> no problems

<Zakim> shawn, you wanted to suggest template

<christos> no problem

<shadi> [[Papers should follow the [template] to clearly explain:]]

<markel> yes, why not

<christos> yes

<giorgio> yes

<Mario-Batusic> yes

<vivienne> yes

<shadi> [[The review process will ensure that each paper will get at least three independent reviews, based on criteria including relevance, clarity, soundness and power of the arguments, generality of results/claims, novelty]]

<shadi> [[The review process will ensure that each paper will get at least three independent reviews, based on criteria including relevance, clarity, soundness and power of the arguments, generality of results/claims, novelty, and time available]]

<giorgio> no, i don't agree

saz: "The review process will ensure that each paper will get at least three independent reviews, based on criteria including relevance, clarity, soundness and power of the arguments, generality of results/claims, novelty." maybe add and space or time available

<markel> no

gb: saying there isn't enough time wouldn't help in getting better papers

<markel> if we get many quality papers we will have to squeeze them

saz: it is one of the criteria why we choose to accept or reject the paper. it's a safeguard if we need to say "we got too many paper"

<Mario-Batusic> agree

gb: the setnece talks aboput review. committee will decide what to accept based no other critieria, such as time. if we send a reject notice, then we say it was good paper but we had to reject due to limitations of time. why need to say?

saz: because we do get challenged. people have very high expectations of transparency, [missed], and such.

<giorgio> it'll be the program committee task to justify its choices. having this sentence would not help.

markel: if get many high quality, we will rank them, set a threshold. it's pity if can't accept all good ones.

saz: issue is this is under W3C WAI.

<giorgio> is this a research and dev group?

<markel> -q

<markel> another option is to accept all papers that have a minimum of quality and all will be part of the proceedings

saz: be spelled out clearly that paper might not be accepted based on limitations of time

<markel> but only the best will be presented at the weminar

gb: we can write it with rejection letter
... if get lots, we do more seminars :)

<giorgio> I would agree on putting it only if we can do it now.

<giorgio> suggstions?

<markel> I still don't get it, are we suggesting that all papers should be accepted because WAI is afraid of criticism because of rejections?

shawn: ...

<markel> but this is what happens in all conferences/workshops...

<vivienne> let's put the statement at the end of the "all submissions will be reviewed" paragraph

saz: no, it's so people don't criticize the reviewers. set expectation form the beginning that time limitations is a legitimate reason for rejection

All submissions will be reviewed by the Scientific Committee, and each paper will get at least three independent reviews, based on criteria including relevance, clarity, soundness and power of the arguments, generality of results/claims, novelty. Papers will be accepted based on this critieria and space availability. Accepted papers will be published - in an attributable form - as part of the

proceedings and in the ensuing publication (see more information about RDWG Publications and RDWG Practice for Writership and Credits). Authors of accepted papers will be invited to present their work during the symposium.

<Mario-Batusic> Unfortunately I have to leave, bye to all!

<vivienne> sounds good to me

<giorgio> I would go for: "All submissions will be reviewed by the Scientific Committee, and each paper will get at least three independent reviews, based on criteria including relevance, clarity, soundness and power of the arguments, generality of results/claims, novelty. Papers will be accepted based on this critieria and space availability."

markel: papers are evaluated against quality criteria by scientific committee. people should trust this

<giorgio> not for the additional Accepted papers will be published - ..

<markel> can we diseminate a summary of the CFP in mailing lists?

saz: think about for future call: how much info from call for papers should go on the main page -- for future

<markel> once it is released?

markel: can we do summary and send to mailing list

saz: we will do announcements on wai hompage, w3c, wai ig maillnig list, twitter/identi.ca

<markel> ok

saz: you are welcome to forward e-mail & retweet

<markel> excellent,

<giorgio> good

saz: encouraged to!

<christos> nice

<markel> i though it was our duty

<markel> yes, of course

shawn: prefer that you use wording from announcements or CfP as is

saz: have been getting simon's consensus on the announcements

<markel> thank you shadi

<pthiessen> cheers - ttyl (look forward to forwarding around the cfp)

<vivienne> no problem, good night all

<christos> good bye all

<giorgio> bye