Skip

MDN Developer Need Assessments: results and next steps - TPAC 2020 breakout

Facilitator: Philip Jägenstedt, Dominique Hazael-Massieux, Sheila Moussavi, Robert Nyman

Review outcome of the MDN DNA survey 2019, incl recently released MDN Browser Compat Report and early results from MDN DNA Survey 2020

Slides

Minutes (including discussions that were not audio-recorded)

Previous: All your spec are belong to us! Irrigating dev resources from specs All breakouts Next: Making math a first class citizen on the Web

Skip

Skip

Transcript

Okay, so while we're waiting for the slide situation to be worked out, I'll get started with the background a little bit.

I'm (mumbles) okay, there they are.

Excellent, thanks.

All right, so most of you know this, I think a lot of you were probably in this session last year as well, this is the second year that MDN is running a big web developer survey called the Developer Needs Assessment ran for the first time last year, that year we had 28,000 complete responses and the findings at least in terms of the needs ranking, which we'll get to in a bit this year as well was a lot to do with browser compatibility testing and slightly the item that mystified us is number two, their outdated or inaccurate documentation for frameworks and libraries.

So, off the back of those results, we designed a follow up survey, we call it the browser compatibility survey and we said that was also run an MDM and then we did interviews with 13 participants to try to figure out what might be the most promising thing to go and fix in terms of browser compatibility and largely speaking, it was layout and styling issues, CSS, Flexbox and Grid came up a lot and then we have a long list of other things you can go out and look at the report for that review wanna learn more.

And the the URL for both of those reports is insights, I will paste it here.

It's insightstartdevelopment.org So this year 2020, we have done the Developer Needs Assessment again with some modifications, my next slide please?

I forgot the hundreds of slides, it's been building since Monday last week or two weeks ago, I don't remember so I couldn't finish.

It's been failing for a while now, it's gonna be up for another week or so I think we have over 5000 complete responses but compared to last year that's a lot fewer and also the completion rate is lower.

But we have still quite a bit of results, so I will go through not all of the results.

If anyone is looking at the slide deck then there are a lot of hidden slides where I've included most of the questions not all of them but most of them.

And anyone...

No.

Okay.

So let's just go through them one by one, the ones that I thought especially worth calling out.

So this is just to say, who is taking the survey?

Or rather what is the target audience of this survey?

It's web developers who do some amount of coding on the web, so this is sort of the gating question for this survey.

It doesn't target people who have abandoned the web, it doesn't target people who only design but writes no code but people who do some amount of coding.

Next please.

We have included this your new question about what kinds of environments because obviously web technologies are used outside of the browser as well but you know mostly, perhaps not surprisingly it's mostly still web applications and webpages.

Next please.

This is again really to show who's taking this survey and I think worth noting is like how, what a big proportion of very experienced developers take the survey.

I have to assume this is not representative of web developers at large but the people who end up taking and completing this survey are not your typical web developer.

But exactly what the difference between the populations are I don't dare guess but I have a hard time believing that 30% of all web developers have been doing it for more than 10 years.

Next please.

In terms of representation this is not a new question for this year, I don't know.

I'm not gonna compare this to last year, so I can't say if it's different but the next question is new though.

Next please.

Were we asked about whether or not you identify as being in a minority group?

I don't know what we were expecting other than it being probably less than 50% which it is and we do have subsets of the results by the people who responded yes to this question, where one can go and look for things to look into to form hypothesis about what's going on and dig deeper, which I have not done yet.

Next please.

So people from all over the world took this survey, I think it's notable that the second largest, Germany and Russia are now tied for the second largest but when I looked at it last time it was Russia at number two, I don't recall it was the case last year but indicates the point being here that it's quite a broad representation and the server was translated into, I forget how many languages this year, if someone remembers please speak up.

Was it seven or something like that?

Yep.

And next please.

So this is partly new compared to last year, this is sort of where we're taking the temperature of how different parts of the web platform are doing or like different parts of web developers experience are doing and it's a little tricky to interpret the results, so I sort of put them in this spreadsheet and did some things.

This is based on the 5000 or so complete responses and what we can see is...

So last year we asked only about overall satisfaction and that number is not changed like we gonna confidence interval of whatever it was I tried.

It's basically the same because there are fewer responses, we're less sure of the number, so we can't really see a change.

Unsurprisingly people are not happy about browser compatibility the second row here, that's exactly in line with (chuckles) what we saw last year of course but I think the interesting things note are the last four rows, where both people to a large degree, say they are neither satisfied nor dissatisfied with tooling in various areas and I have to think that some of these are people who perhaps don't know of any tooling or don't particularly care about the thing because there is no, does not apply option you had to pick either on this five point scale how satisfied you are but this even given that, I think the levels of dissatisfaction here for the last four rows are still notable and something is going on with privacy and security, that's interesting, maybe that will be a discussion topic and perhaps I could speculate it has to do with all the changes happening with privacy in browsers where basically web developers have to adapt to changes coming down the line and they aren't the same behaviors across browsers either.

Alright, next please.

So the meet of the survey, both last year and this year was this ranking exercise.

If you took the survey you know what this was, you got I think six options at a time and you were asked, please pick the most frustrating and the least frustrating and you get about 18 or so sets of these.

And based on that in aggregate, there's an algorithm that sort of tries to rank these and we get a good ranking of sorts.

The top five are quite similar to last year, I think notably the frameworks and libraries documentation that is no longer number two but I don't really know if this is like how significant of a change it is?

It's sort of hard to reason about the change in rank if there's been an actual change or we can't really be sure it's not the change in population that's done it.

But I think these are probably the same five, certainly four of these are the same as last year, I'm not sure about number three there.

Next please.

So what's at the very bottom of the list is also interesting I think 'cause documentation for the platform itself, HTML, CSS and JavaScript, people seem to be pretty satisfied with that and that also showed up in the previous slide about my spreadsheet with colors there but people are actually pretty happy with the documentation on the web and I guess this maybe means people like MDN.

So they're coming to this survey mostly from MDN and they maybe interpret this question as being about MDN and so they say yes like MDN perhaps but also of course, it's a testament to the fact that the the documentation for these areas, like the base of the web platform, is really quite good but this is lower than it was last year and I don't know how to explained that.

Okay, next please.

This was new for this year, we ask people like in what way they participate in building the web platform?

Nobody is surprised that none of the above sort of one out and they're very few, for example, contributed directly to webpack from tests, the absolute numbers don't show up here but then I don't know less than 100 certainly, for that WPT contribution.

So I don't know what to make of the results other than saying that's interesting.

Next please.

This is not a new questions since last time but if the second question following this is new since last year.

So last year we asked people, what are the browsers you support?

And then we asked them to rank the browsers they support, I think in terms of frustratingness say, which made the results sort of tricky to interpret all the notes not impossible but sort of hard to read.

So this year we asked it in a different way.

Next please.

Right, so we ask people, whether you support it or not, which browser cause you the most issues?

And fascinating to me is that while only 27 or so percent said they support IE, more than 60 something percent say that IE causes issues.

I guess that means it's not officially supported but they would maybe try to fix things, I don't know exactly how to interpret it but that difference is pretty stark and and interesting.

Next please.

All of the following are questions that only a subset of the participants got based on their answers to other questions, which I've omitted here and so take a look at the the slides, you can see it more of this but what I thought was interesting about this in particular, like large majority of people say they do use CSS unsurprisingly but if you had to pick the biggest pain points then its challenges is creating allowed specified which sort of wins that question.

And that's different from for example, JavaScript people aren't gonna say that they couldn't solve the problem with JavaScript, it's gonna be about lack of support for new JavaScript features or performance but not failure to get the job done.

So, I don't know what's going on there but it's very interesting and the answers were HTML sort of point in a similar direction.

That's about the customizability and some maybe it has it's form control starting basically, so that's something that possibly do a follow up on.

Next please.

All right.

Again new for this year was two sections on blood testing and then accessibility.

So this was blood testing, the main unanswered question from last year at least for me, was when people said that testing across browsers was a frustration, how much of that just means that?

It's work but it's worth, like manually opening different browsers and testing them versus how hard is it to set up a CI infrastructure to test browsers because you could imagine it mostly being either of those.

So these questions added this year I think shed some light on that and that in fact, it's not close to 24% say that it's the time spent on manual testing.

So that's significant but it's not all that for sure, the difficulty of writing tests and running the tests and just setting up the environment is also takes large slices of this pie.

So a bit of both, I guess, is the conclusion.

Next please.

So for accessibility also new for this year, it looks like just knowing how to do it is a big part of the problem and then you might also notice the big pink slice there is lack of support from my employer.

So last year that's in the needs ranking, accessibility rank fairly low but still it was pretty clear that the right conclusion isn't that it's not a problem and people can easily write accessible sites, it's more like it does not apply sort of situation where not all developers have the mandate to try to do it or know how to do it, so that makes it perhaps not frustrating.

So again, this was added this year to add some color to that and I haven't dug deep into this myself.

Next please.

Still on accessibility.

Yeah, mostly people I guess, just don't test more than half and I guess testing with screen readers which is the actual interface that people are gonna use is pretty low and again remember that I'm pretty sure a subset of the respondents be 100% here would not mean all 5000 people who took the survey.

There were some getting question for this I think.

Okay, next please.

I wonder why I included this slide (chuckles), this is the same as last year, I guess the point is that the amount of influence that developers have over the tools they choose is sort of matters if people say that they don't have a choice at all, then improving, you would have to prioritize differently depending on the answer to this question simply but I think it's interesting that only a small minority rarely or never get to make their own choices.

So I would suggest that web developers at large still have that mix of people who took the survey, have a lot of agency in and how they do their work.

Alright, I hope that wasn't too boring.

Next please, because we have reached the discussion part of this and I threw together some suggestions.

How about we build four topics for a minute or two and then we'll pick a few and just spend 10 minutes on each?

I guess we could get through three topics or so.

Skip

Sponsors

Platinum sponsor

Coil Technologies,

Media sponsor

Legible

For further details, contact sponsorship@w3.org