Meeting minutes
Slideset: https://
tidoust: Agenda.. .Scribe, picked, code of conduct, we're not recording
… Goal is to briefly present what we've been playing with in the WebDX community group and exchange on ideas, how we could use the research in a more configurated manor
… This is a continuation of a discussion started in TPAC last year in 2022
… There are different research needs that go with it, we're going to focus on the middle part. When you're a group you want to focus on the API design and want feedback from developers
… how do you get that feedback in a meaningful way
tidoust: one of the workstreams is the research workstream, its stated goal is to accelerate and improve the decision making.
tidoust: This group is composed of browser vendors, but also MDN, open Web Docs, plus some others
… We could make some MDN surveys that we've made. I won't present the results but these are available online
… one was for browser, input for the interopt 2023, a more recent one was web components (prioritisation), that survey was conducted in collaboration with WebCG, the last one was web security (pain points devs may face)
… the WebDX community group has been running these surveys to get feedback from developers.
… the CG have contact points in different places to collaborate and try to exchange/promote surveys
… that's the context of what the CG have been doing. But the question today is "how could this be useful to a standardization group"
… Sometimes they have more formal surveys, they run under the name of someone in the group. We wonder whether there's benefit of providing a package to the grioup
tidoust: saying "ok you have this question on API design, this is how we would set it up so you can run a survey and get the feedback you need"
… So that's it for now. Could someone here report back on if its been useful?
kadirtopal: In addition to the questions you had, does anyone have a situation recently of handling research or user interviews that would have made the decision easier
… where its added more colour, or more detail than a survey on twitter has anyone had that?
Florian Scholz: I'm hoping to use that research to figure out what sort of documentation we need to write to help with security concepts. So that survey immediately was useful to me
Francois: The surveys we run are more quantitive in a sense, they can be more qualitive in the sense of interviews where you get more feedback (especially where you're asking a specific export).
… Have you found something useful there?
<Francois> Shares security survey on screen
Francois: Is there any approach/guidelines/handholding we could do so people don't struggle with this any more.
… Are you looking for a quick guide that says "this is what you should take into account? these are the channels you should use"? Is this a first thing you would be looking for??
Jon Kuperman: Not W3C related, event at a meta level learning how to craft good survey questions, we're worried if we put out a tweet how do we phrase it so its fair. I feel like there's a whole art to it so we understand and do it effectively
Francois: It could be a group that prepare a survey and say "well, you're introducing bias here etc", ok that's good feedback
… Im curious. Have you in TC39 had to deal with surveys where "yes its technically survey but we don't want to use it etc"
??3: We've had Github issues where they've been linked to from a social network. But the bias can be where it came from and who linked it originally etc
Francois: [When working on Web Components] We were a bit worried we might get responses that didn't make any sense and there's no way to filter it out.
… You could sort somethings out to some extent
… Any other questions?
… What were you hoping to get from this session?
Kadirtopal: I was interested in figuring out, where do most people have issues? Bringing in research that is largly "unsound" can be more damaging. Its a bit difficult to get hold of UX research, so i was curious, where's the biggest painpoints.
Francois: Are there ideas for channels, where surveys can be run? Thank you for that channel that reaches developers in broad succession. Any other channels or venues? I'm thinking that W3C may publish a survey but that may not attract people in the end and miss our intended audience
… Things like blogs, publishing platforms but then you might be falling into the same trap (as twitter) because it is the same audience, it's hard. MDN is great but I can't really think of anything else
??1: I think one way for the problems you have, could be helpful if you create your own panel of responses. Have people opt in to be contacted and have them describe themselves, then when we do contact them we provide an incentive
… Its a special population who are paid quite well
Francois: what's the incentive??
??1: Usually it's money
Francois: that's the one incentive we cannot provide
… Doesn't that introduce a lot of biases?
??1: yes, at least you have a chance to understand more about who you're getting, how good is that? Maybe better than sending something out on twitter
Chris: I'm CoChair of the media WG. We're used to APIs that have a very particular audience. There are places devs congregate, if we can identify particular communities where the technology is likely to be used then target those communities with a survey that could be a good way. It's not something we've done in the WG initially but that could be a good way, for our particular group that's something that could work
… I get the point about social media, but I wonder about stack overflow, people who are building websites they might just go there to ask questions, it makes me wonder if that's a place to run these kind of surveys to get a broader range of people
Mike: I don't personally have a good relationship with SO management right now, they went on strike months ago, the rest of the moderators were involved with SO but I was not. If we have something we need from them then I'm happy to talk to them
… If we have something concrete to take to them then I can do that
… Can you tell me more about why? (to Chris)
Chris: A lot of people coming into the WG are through relationships, occasionally because the technologies are quite niche, Stack Overflow are not the best place to ask a question, so people will come to our repo instead. We get a certain amount through github issues in the spec
… I guess we haven't run surveys because... It hasn't been something we've needed to use it for, well the case most recently we were making a prioritisation decision where we're trying to set a scope for what the Next API feature should be, I think that would be a valuable thing to do a survey.
James: You're much better off focusing on a small set of people than going through ?? and taking a subset.
Francois: I'm wondering if we're not trying to create something too complex perhaps? And what's needed is something some groups already do? (Audio WG invite some guest to their meetings). The GPU also start face to face and invite developers into the meetings to try out some things too
… I think its still valuable to explore more formal research things
Michael: I do think we're lucky we have developers participating in the working group, one thing im concerned about is our sample size is small, so one of the reasons I came here is to see if we can increase the sample size.
Francois: Good input
Robert: I agree with what James said. The design of web rpc might be the reason WHY people aren't using it. So if you don't know the target group today, you don't always know what the target group will be in the future
James: Technologies are specialised, if you ask a developer what do they think of our low level graphics APIs, their opinion isn't really relevant in that respect. They're not the target audience. With more context you would get better feedback, but it does feel like there are some bits of the web stack that are not intended for general developers. The low level stuff is really for people who are implementing those libraries
Francois: We've been discussing re-running surveys too. the security survey. Its easy to re-run on the periodic basis. It's a much lager undertaking for the larger surveys it requires a lot of investment
… There is high level sentiment, there is a need for edge level data. Interopability is a big issue so the response was "lets address that" and we were like "what do we do"? We have the satisfaction survey now, which google is contributing. But then you need to look at what actual solutions need to be prioritised in this space
… State Of HTML is the first time we're doin it, we just want to know if the sentiment is going up or down
Florian: I was thinking of points in times they have an API design question, i guess once you've solved that API design, do you run another survey to see if that design actually works. but it's going to be a different survey probably with different options.
Francois: Trying to summarise what I've been hearing: im hearing different levels that can be useful (developers is one level but it's not enough). Which means surveys that can reach a broader range of developers can be a good tool to have in your toolstack. Which means you need this evolution over time. Survey experts can help giving a "101" on main biases for each and every survey. In terms of where to run the survey, MDN seems to remain the main place but we've heard about Stack Overflow.
??2: In regards to MDN surveys. With Poll style surveys to help people make decisions, this isn't a developer satisfaction its more "the working group don't know what to work on next, you can vote"
Francois: We'll try to organize our thoughts and come up with some kind of proposal or WG chats.
Chris N: Even if a WG saw a need to run a survey, is that something you're suggesting the CG can help with? Would they help us form the questions?
Francois: the Web Components is a good example, so yes getting in touch with the Web CG is a good starting point
Francois: Don't expect we're going to write the questions down, we don't have those particular expertise. But with Web Components we provided feedback
???: Even just support with creating good surveys that are methodologically sound would be helpful.
JaseW: Some questions can be loaded where you already dictate the answer in the question itself
... People who follow a particular person may be more skewed towards voting in a certain way.
<dom__> [participants can also require a chair election]