mobileOK Pro Task Force Face to Face Day 1

Paul: I have to step out in 20 minutes or so

Kai: Does anyone have anything more to add to the agenda?
... I think we'll take the BPs and bang out tests that are not covered

DKA: I agree
... Let's get the draft ready rather than worry about some of the process issues at this stage
... we may find obstcles on the way

Kai: In terms of expectation management - the results of the tests will not be 100% objective
... We can bracket the tests to help different evaluators to come up with differing answers
... Some answers will be subjective - is this OK?

Paul: A conformance company can't afford to assert conformance if they're not deterministic
... To a degree, every test is ambiguous, but it needs to be very limited

Kai: Take no. links on a page - it depends what the page is for. A sitemap is nothing but links

Paul: There may be some but not that many

Adam: How are these things to be asserted?
... is it self-certification? External?

Paul: That's a different discussion. A content provider may make the claim themselves or they may want to farm it out to rubber stamp them...

Adam: The amibiguity depends how the tests are being carried out

Paul: It might become a legal requirement to be mobileOK due to the accessibility issue

Adam: So we need a rule of thumb to decide whether tests are going to be carried out by the CP or an independent

Paul: I think... "It is not necessary to seek independent verification for any type of assertion" - it's up to a CP to decide whether they want such independent verification

Kai: A well known company's claim might be believed, but it may want to seek third party verification - it changes from a claim to trustmark in that situation.

Paul: I'm hoping we can avoid getting bogged down in these issues
... How we make the assertion, POWDER or otherwise, and so on - that's not for now

Kai: I want to talk about bracketing. We created the DDC - we can also set certain values that bracket the tests. This gives us something that can be worked on
... Unless we put down the brackets, we'll have too subjective a test

Paul: We have machine testable, we have deterministic human tests, then we have the ones that are too ambiguous to ever come up with an agreement, but there are others that specialists could make a judgement call upon

Kai: I would suggest that we move on and not turn in circles
... So let's address the issues the group has presented

Note message from Jo: http://lists.w3.org/Archives/Member/member-bpwg/2008Feb/0004.html

Kai: Goes through the e-mail from Jo, who is now present
... Jo - your view is that we shouldn't be doing this?

Jo: No - I am not sure that NOW is the time to do it and I'm not sure that you, Kai, have time to do it

Kai: We're focussing on the tests today and tomorrow. We want to be done with it as quickly as we can

Jo: You're asked to come back to the Group as a whole with a charter
... that means deciding to create a doc by the end of the year
... so there are questions to answer other than come up with tests

Kai: I don't think it's up to us to determine the public interest in MOK

Jo: But you're asking the group to support this
... The questions are meant to be hard, but not unhelpful
... the WG doesn't want to annoy everyone and doesn't want the TF to spend time on something it can't support
... there are serious doubts that it is achievable

Paul: It's not 'them and us'
... it seems a watse of time to write a charter when the people in the room are willing to do the job
... I thought the only argument against it was the use of the WG's time

Kai: We have to take the concerns seriously

Paul: Can't we just write the spec and get it done
... it's a case of documenting the test cases

Alan: How much feedback was there on the Basic tests?

Jo: On the one hand there were quite a few, on the other there were not enough
... The worst thing that can happen is that you don't get any comments

Alan: in WCAG 2 the time has been taken up by handling all the comments

Jo: If you don't get any commetns that means no one has read it. Lost of comments means lots of works

Alan: And what has been the feedback and/or implementation

Jo: It's been largely internal - me, Sean, CTIC etc. We're criticising our own work a lot of the time
... We've had 3 Last Calls on it - so an extreme lavel of detail
... it took 18 months so you could argue that each test takes 2 weeks

Alan: What about the label?

Jo: I've tried not to drag that into the discussion. We have a logo that will be shown next week at mobile world
... We havn't done the labelling side
... we're unclear what the labelling paradigm will be

Kai: Does the WG have any +ve proof that what we're doing that what we're doing is undesirable?

Jo: I don't think we know
... People are using the Basic Checker and we're getting bug reports
... What's less encouraging is the crawls of various TLDs and there are very few mOK Basic sites

Kai: I think the achivability is there given the people around the room

Jo: Then I think the point to make is that neither Sean nor I think it is achivable

Kai: Desirability - that's hard to judge

Jo: I have no view about its desirability

Kai: I think most people think it's desirable
... I remember at FT Boston there seemed to be support

Jo: Minds can change

Kai: Utility to the community at large? This TF thinks its quite large. mOK makes no sense without mOK Pro - it's not complete without it
... But that's an opinion at the table. The same could be said for the BPWG in the first place?
... Does that naswer the concerns, even though it may not be satisfactory?

Jo: It's not my call, it's the group's call
... The TF can't decide to do something off its own bat - it need agreement from the WG as a whole

Alan: The time available is?

Jo: If the TF is going to produce a Rec or a Note? If Rec Track it needs to be in CR around October
... If the process is not complete by the end of the BPWG charter (Dec 08) then it looks bad.

Alan: I assume that these tests will be more complex and more controversial than machine tests
... My enthusiasm is waning

Jo: I suggest you come up with a timetable to suggest

<Kai> ACTION: Kai to put together a reasonble time table for completion of mobileOK Pro by the end of BPWG [recorded in http://www.w3.org/2008/02/05-bpwg-minutes.html#action01]

<trackbot-ng> Created ACTION-636 - Put together a reasonble time table for completion of mobileOK Pro by the end of BPWG [on Kai Scheppe - due 2008-02-12].

Kai: Point 2 in Jo's mail - the lack of compliance with mOK Basic
... Basic is a pre-requisite for Pro. We have no implementation experience of the easy bit before we start on the hard bit
... It's half the battle. We don't have any experience - but that doesn't invalidate the existence of Basic

Jo: repeats his argument

<Kai> Phil: I very much disagree with your point

<Kai> Jo: Ideally one would wait

<Kai> Phil: that is what I disagree with

<Kai> ...until the full picture is there, a complete system with all the bits and pieces, then it makes sense. It has no legs

DKA: Can I suggest that we focus on the content? We're here to get the tests done, so let's get on with it
... I'm actually neutral on whether we do this or not. We're talking in the abstract - we need something concrete to work om

Kai: But we have these concerns to answer

Jo: And the WG has asked for the charter to be elaborated upon

DKA: We can't spend to days on the charter

Jo: Yes, but the charter as drafted has not been aceepted

Kai: It was modelled on the CT TF

Jo: It needs to clarify what the deliverables are?

Kai: A timetable
... Deliverables and scope will be determiend today

Jo: I believe that will be fine

Kai: Point 3 - consumer pull

Jo: I think we've covreed that

Kai: Deterministic tests?
... I think this is covered by bracketing - assigining value ranges to subjective tests

Jo: There are things in WCAG 1 that have been dropped because they are not subject to repeatable tests

Kai: Yes, we're in dangerous territory
... Do we think that the subjectivity of these tests, with bracketing, makes them unworkable

Dave: It's an issue

Alan: I don't think we have to test 100% of the BPs - just go further than Basic

Kai: We have a variety of tools available - the question being asked is still valuable

Dave: I think we might have to introduce a warn

Kai: But the consensus is that there will be tests that we cannot apply because they are intrinsically not repeatable

Jo: I suggest you take note of Alan's points about WCAG 1 tests being dropped for non-repeatability

<achuter> ACTION: Alan to check on which WCAG 1.0 checkpoints were dropped in 2.0 due to untestability. [recorded in http://www.w3.org/2008/02/05-bpwg-minutes.html#action02]

<trackbot-ng> Sorry, amibiguous username (more than one match) - Alan

<trackbot-ng> Try using a different identifier, such as family name or username (eg. achuter, atai)

Kai: Point 5 - we've covered

<achuter> ACTION: achuter to check on which WCAG 1.0 checkpoints were dropped in 2.0 due to untestability. [recorded in http://www.w3.org/2008/02/05-bpwg-minutes.html#action03]

<trackbot-ng> Created ACTION-637 - Check on which WCAG 1.0 checkpoints were dropped in 2.0 due to untestability. [on Alan Chuter - due 2008-02-12].

Dom says that the implication is that testing should be carried out on the DDC - but is that clear?

Jo: That would be my expectation

Kai: It seems to me that the BPs we're talking about are the way content is presented so that's outside the DDC?

Jo: Either you're going to test all possible experiences or you're going to limit to a single environment

Kai: I think the tests are expected to work on any device, including the DDC

Jo: But you need to be clear that you're testing the mobile experience, not the desktop, for example
... Imagine a site has 3 completely separate representations - Basic, iPhone and desktop
... you need to specify which you're testing or you test them all

Kai: I think we're testing a mobile environment

Jo: Basic only covers the DDC
... So if I have those 3 representations, can I calim mOK Pro for all three?

<Kai> ISSUE: Does the TF need to create device which emulates the DDC for testing?

Issue surrounding testing environment. Since DDC doesn't exist, how can a human carry out tests using it

Jo offers http://rabin.mobi/dmplbit

<Kai> ACTION: Kai to raise an issue on ISSUE: Does the TF need to create device which emulates the DDC for testing? [recorded in http://www.w3.org/2008/02/05-bpwg-minutes.html#action04]

<trackbot-ng> Created ACTION-638 - Raise an issue on ISSUE: Does the TF need to create device which emulates the DDC for testing? [on Kai Scheppe - due 2008-02-12].

<Kai2> ACTION: Kai to create a more elaborate charter with times, deliverables [recorded in http://www.w3.org/2008/02/05-bpwg-minutes.html#action05]

<trackbot-ng> Created ACTION-639 - Create a more elaborate charter with times, deliverables [on Kai Scheppe - due 2008-02-12].

PhilA: Is there a Test Suite for Basic?

Jo: Yes - part of the checker

<achuter> scribenick achuter

<achuter> scribenick: achuter

Deliverables

Kai: At least we know that we can follow the model of the MOK Basic tests.

Jo: It contains the subset of aspects that are machine-testable.

Kai: So we're extending MOK basic.
... We need to go through all the BPs in Basic and add what's missing.

[On screen all see thre format agreed in Boston meeting]

Jo: Basic used a kind of pseudocode if-then pass/warn/fail.

Kai: We can modify that.
... It's not so easy because we are going to bracket some aspects.

Phil: We should give examples if we can.

Kai: Test questions, then yes or no.

Jo: In Basic we provided [IDs] for the fail or warns so tools can link to them.
... If you aggregate results this is not possible.
... You can decide if a test is failed because because one fail fails the whole test.

Phile: Testing is only for failures.

Jo: Checker shouldn't stop.
... ... on fail.

Phil: We should test for fail. Just extend the Basic tests.

<PhilA> RESOLUTION: A deliverable will be the mobileOK Pro tests document

Phil: We should provide test suite.
... I am prepared to do it if necessary.
... These are individual pages linked from an [index] page.

Kai: Do we all agree it is necessary?

Adam: For some it would not be necessary.

Kai: Perhaps they should only be for tests that are otherwise ambiguous.
... There's a danger that it could be interpreted as limiting the applicability of the tests.
... Have reservations about test cases.

Dan: This looks like it could be more than the task force can handle.
... We should just facilitate creation of these test cases by the community.
... Do we want to limit the interpretations?

Phil: Yes, we do, for example, to mobile context.
... A test suite would make it easier to write the tests.

Kai: It would be better not to include test suite in deliverables, but allow others to do so.

<PhilA> ACTION: Phil to draft test suite document to complement Test Document - such a draft may or may not be completed depending on its usefulness in the Test Document creation process [recorded in http://www.w3.org/2008/02/05-bpwg-minutes.html#action06]

<trackbot-ng> Created ACTION-640 - Draft test suite document to complement Test Document - such a draft may or may not be completed depending on its usefulness in the Test Document creation process [on Phil Archer - due 2008-02-12].

RESOLUTION: Pro document scope is all tests not covered by MOK Basic, against the default delivery context (DDC).

Phil: We need an implementation of the DDC for the human tester to use.

Jo: I did an implementation of the DDC, which the TF can use.

[discuss whether emulator should be a deliverable]

Dave: We have some people who could do it.

RESOLUTION: Jo will donate emulator code, to be tweaked by TF and become a deliverable.

<Kai2> ACTION: Dave to ask Paul if Segala can tweak Jo's emulator [recorded in http://www.w3.org/2008/02/05-bpwg-minutes.html#action07]

<trackbot-ng> Created ACTION-641 - Ask Paul if Segala can tweak Jo's emulator [on Dave Roberts - due 2008-02-12].

<Kai2> CLOSE ACTION-641

<trackbot-ng> ACTION-641 Ask Paul if Segala can tweak Jo's emulator closed

<Kai2> ACTION: Rooks to ask Paul if Segala can tweak Jo's emulator [recorded in http://www.w3.org/2008/02/05-bpwg-minutes.html#action08]

<trackbot-ng> Created ACTION-642 - Ask Paul if Segala can tweak Jo's emulator [on David Rooks - due 2008-02-12].

Kai: We need to avoid [scope creep].

Jo: [about template] Problem description would repeat the BP.
... Is a prescriptive document, not explanatory or tutorial.

Kai: We shouldn't quote or adapt the description of the BP. Danger of confilcting version.

Alan: Danger that document will be unreadable, as it links to at least two others.

Kai: This is a web document, we need to use hyperlinks rather than duplicating content.

Alan: Users can adapt the content and merge it if they want to.

Kai: Should we go on to evaulate Pro if Basic is failed.
... Passing basic test is a prerequisite.

Jo: Don't have to run basic first.

Dave: Have to pass basic test first.

Jo: It's unlikely that people will do pro without running basic first.
... We need to reward not to punish.
... Should provide helpful results.
... Structure should be as granular as possible, provide most helpful feedback possible.

Kai: Should parameterise the tests.
... To make them less subjective.

http://www.wabcluster.org/uwem/tests/#guideline-1

Alan: Would like to see XPath expressions of applicability criteria.

Kai: Yes, but can do that later.

Jo: Better no to be so precise at this stage.
... Warn should not be on cannot be determined, only on dodgy content.

Kai: Test is passed unless there is an explicity fail. Warn for things that author needs to look at.

Phil: Should avoid warnings, and aim to have none by completion.

<Kai2> ACTION: Kai to post the test format to the list [recorded in http://www.w3.org/2008/02/05-bpwg-minutes.html#action09]

<trackbot-ng> Created ACTION-643 - Post the test format to the list [on Kai Scheppe - due 2008-02-12].

Dave: Where we find machine-testable things we should provide feedback to checker group.

Alan: We must avoid adding new requirements not in the BPs.

<DKA> scribenick: DKA

<scribe> scribe: Dan

Yo-ho

Kai: we left off with access key test. We provided a format: a link to the best practice, a statement , a link to bp, pseudocode (optional) and pass/fail/warn criteria.
... First thing we noticed in accesskey is that in the BP we didn't see any list of access keys.
... [taking through earlier attept of test written in an earlier document]

Adam: re-word that to say "usage of access keys should be consistent across the site".

Phil: explaining how decoration consistent with access key annotation doesn't necesarily mean link decoration.
... ...should include 2 examples of how to do it and 1 of how not to do it.

Kai: Should we write the examples in here? If so should they be prose, pictures?

Dave: I think it should be prose.

+1

Phil: I like the idea of examples. Especially in this soft area of human tests. If we can think of examples of what we mean other people will better be able to understand...

Kai: [adds examples section to template]

[contuing to edit access keys test]

Kai, Phil: [discussion of numeric vs. alpha access keys and whether we should strongly encourage numeric ones]

Dan: We shouldn't try to refine the BP -- this might get into that territory -- we should just note the ambiguity.

Paul: I think it's reasonable to do some clarification.

Kai: I'd like to pound these out and note the open points.
... [agreeing with Dan]
... Should access keys be numeric? Should we take a stand here?

Dave: No

Dan: No

Paul: We should mark it as something we might want to come back to.

Kai: Yes.

Paul: If it's a slightly ambiguous assertion then we could give an example, as specialists, of what that might mean.

<PhilA> PROPOSED EXAMPLE: Hyperlinks may be easily decorated with access keys by presenting them in an ordered list where the access code is equivalent to the list position (and is usually numeric).

Phil: I think producing an example helps to define the test.
... [pastes example]
... by putting it in brackets it becomes a should not a must, but strongly recommends a certain approach.

Kai: should we paste it in?

+1

<Paul> +1

Kai: What are we writing here? It doesn't fit into our format. Is it pseudocode?

Phil: [suggests some pseudo-code]

Dave: Suggest we don't do it right now.

Adam: Is it a given it should be pseudo-code? Because the way it is now appears clearer to me than the pseudocode [as used in MOK Basic document]

Kai: If we use pseudocode then we know what results in pass/fail/warn.

Dan: suggests we note the tests now and decide on whether it's to be worded in pseudocode later. Prose may be more appropriate for human-verifiable tests.

Dave: our first question should be: are there any access keys defined?

Alan: First question should be: are there any items on the page for which accesskeys are appriate?

Kai: Agree.
... We need to guide people towards correct usage.

Alan: It's not only links, it's form controls as well.

Kai: so "are there elements that require access keys"?

[discussion on wording of tests for access keys]

Kai: Any other discussion on access keys?

Dave: What about multiple access keys with same value on the same page?

<PhilA> ACTION: me to test assumption that access key assignments must be unique in a given HTML instance [recorded in http://www.w3.org/2008/02/05-bpwg-minutes.html#action10]

<trackbot-ng> Created ACTION-644 - Test assumption that access key assignments must be unique in a given HTML instance [on Marcos Eguillor Fernandez - due 2008-02-12].

Kai: that's it for access keys.

<PhilA> close action 644

[auto-refresh]

s/[auto-referesh]/[AUTO_REFRESH]/

Kai: a human test is noted in the best practice.
... Can we think of any limitations to this test?

[documenting difference to MobileOK Basic test]

[waiting for microsoft word to respond to a paste command]

[going over the already written test in previous draft version]

Kai: [getting on to writing pseudo-code for this test]

Adam: Finding the structure of the sentences confusing.

Dan: it should be "if not, [fail]" instead of "[fail]"

Kai: [implements]

Dan: I have added value here today.

[AVOID_FREE_TEXT]

Dan: Because it's an "avoid" does that mean we can't "fail"?

Kai: One example would be zip codes. You can encode it as a list of zip codes, but this doesn't make sense any more because there are too many to put in a drop-down.
... Should we codify that?

Dave: This isn't just about lists though -- it's also about things like firstname, surname which migth be populated automatically.

Kai: In particular intsnaces, where info about the user is know you could require that but this would be a small number of cases.

Phil: this would be an easy way to determine if they've passed (if they have default text).

Kai: If they don't have default text then you need to look at why they don't - which can be subjective.
... My example was about a numerical line [for lists
... ] where if you are under that, you fail if you don't use a menu instead of a free text.

[formulating that into a test]

[discussion on when you can have legit free text entry vs. should have a picklist]

<PhilA> Suggestion that if there is a finite number of possible values for a text input and if that number is <= 20 then a pick list is reasonable

<PhilA> Ended up with suggesting a limit of x selection box options and y radio buttons/check boxes

<PhilA> Also noting the option to pre-fill forms with data alraedy known to website

<PhilA> Examples noted: selecting Zip codes in a limited geographical area

<PhilA> Offering predetermined values and an 'other' option with free text field

[BACKGROUND_IMAGE_READABILITY]

<PhilA> Is there a test for colour contrast?

<PhilA> Alan: Yes - from WCAG

<PhilA> Paul: It's very harsh

<PhilA> Kai: Do we need to specifically talk about colour blindness?

<PhilA> Paul: I don't think we should deal with the issue of disabled users

<PhilA> Kai: SO the human test just says "test the readability"

<PhilA> Kai: The most common case is a flowery background and a fancy-coloured font

<PhilA> Kai: We could just say "it must be readable" and then link to WCAG for those points

<PhilA> PhilA: We should also refer to lighting conditions, especially outdoor lighting

<PhilA> Checking - DDC does appear to support background images since it's in CSS level 1

<PhilA> Adam: I think this is a case where an example would be useful

<PhilA> ... we shouldn't squeeze objectivity in too far

<PhilA> Kai: Yes, but we should provide brackets

<PhilA> Add in some examples - maybe a floral background with a yellow font

<scribe> scribenick: DKA

<scribe> scribe: Dan

[BALANCE]

Kai: 20 links, warn; 30 links fail

Phil: Site maps have lots of links, home pages have lots of links...

Kai: Not home pages on the mobile side.

Adam: These numbers are going to be arbitrary... Do we need some explanatory text?

[discussion on tests]

Phil: "For general content" there should be no more than 30 links per page.

DKA: We need to consider this in the context of a device where one press down on the rocker-switch means moving from one link to the next link (rather than thinking about more advanced devices).

[CAPABILITIES]

Kai: This point is about not using the DDC.

Alan: You can't test on every last device...

Kai: This is a big can of worms. Every signal device has some limitations. You can't exploit the capabilities of every device fully.

Phil: You could test if you get something different on a higher-spec device than in the DDC. SO test it on the emulator, and test it on the N-70 and test if it's different.

Adam: Don't think that does it because it's about user experience.

Kai: [writes out test for Capabilities - not meaningfully testable]

Dan: I think we should try to do more here - and particularly focus on scripting, for example for form input controls.

[discussion on the merits of javascript]

Kai: basic functionality of the page should be usable without scripting.

Dan: Split this up into a few capability areas: scripting one of them...
... We need to take into account a more subjective evaluation of the content and whether it has effectively adapted to advanced capabilities of the device. For example, does it use scripting to provide a nice UI on the N95 - does it fill the screen and detect orientation-change on the iPhone? These are subjective measures but important and should be taken int account.

Kai: [writes a pseudo-code statement to cover this and writes a few examples]

+1

[CENTRAL_MEANING]

[CLARITY]

Phil: You can do deterministic tests on clarity but they don't take into account the audience.

Kai: if you bother the user with extra stuff they didn't ask for then you fail this best practice

[discussion on what constitutes clarity]

Kai: easiest part to deal with is [limited]...

Phil: if you ask for a time of a show - you get some specific information back - curtain up time, etc...

[working out tests]

Dan: should we consider evaluating the content by N human readers each of which could give a score?

Phil: We could use the "FOG" index which can be calculated by machines.
... Mumbles something about algorithms...

<PhilA> I was mumbling about a Fog Index: http://process.umn.edu/groups/ppd/documents/information/writing_tips.cfm

<PhilA> Alan notes that this applies to English

[discussion on cultural differences to clarity of language]

Dan: Suggest we drop this one.

Dave: +1

Paul: +1 it was dropped from WCAG

[some continued discussion on the wording of the potential pseudo-code]

Paul: I don't think it should be a pass or a fail.

Kai: Could be a warn.

[re-writes test accordingly]

[COLOR_CONTRAST]

Paul: there are specific success criteria for this in WCAG. We could use the same or similar. Then we could use the same or modified tools.

Kai: should we make use of this tool?

Alan: The tool hasn't been produced by W3C - the algorithm has been produced by W3C.

<Kai> http://www.w3.org/TR/2007/WD-WCAG20-TECHS-20070517/Overview.html#G18

Kai: Current test (5 to 1 contrast) is based on the WCAG technique.

Dan: Is it the right WCAG version?

Kai: yes.

<Kai> http://juicystudio.com/services/luminositycontrastratio.php

<adam> ScribeNick: adam

<PhilA> scribeNick: ADam

[references WCAG luminosity test]

[CONTENT_FORMAT_PREFERRED]

Phil: This is machine testable test but not in basic.
... We will define human test that could be feasibly automated.

Paul: Concerned that DDC doesn't appear to support png.

Kai: Should be consider devices that support other formats?

Phil: No.

[ some confusion between PREFERRED and SUPPORT ]

Phil: The Content Format Preferred BP says that content should be sent in preferred format "where possible".

[ discussion on how to formulate this into a test ]

[CONTROL_LABELLING]

[CONTROL_POSITION]

Kai: Current wording is too prescriptive, can envision cases where you would want to do something different.

[ Group support to reword the test to capture the need for labels to be positioned appropriately to the form elements without actually specifying their relative positions. ]

Phil: Do we need to explicitly forbid positioning using absolute css positioning which wouldn't be supported by the DDC?

Kai: Feels wrong.
... Added this point to the test requirements, and explicitly explained it in test limitations.

[COOKIES]

[DEFICIENCIES]

<PhilA> This is a difficult one. Meeting adjourned to the Porterhouse

<PhilA> Tune in tomorrow