Skip ⬇

The MDN Web Developer Needs Assessment

See meeting agenda

Video

slides - minutes

Video hosted by Web Castor on their StreamFizz platform.

Transcript

Dominique Hazael-Massieux: I'm Dominique Hazael-Massieux from W3C staff. I'm also called Dom. I'm the representative of the W3C on the MDN Product Advisory Board.

As many of you will know MDN is one of the most popular documentation portal for web developers out there, and W3C sits on the product advisory board there, along with Google, Samsung, Boku, Microsoft, and of course Mozilla, who is leading MDN project overall.

Thanks to the work that the Mozilla team has done, a survey was organized over the past few months.

The W3C community was involved: we got lots of input from Chairs and others on what that survey needed to ask to the developers out there.

And then we have the pleasure to have Kadir from the MDN team to give us some early insight as what was learned from the survey, both in the process of doing the survey, and then collecting answers on the survey, and I'll leave the room to Kadir.

Kadir Topal: Thank you, Dom. So, yeah as Dom said, these days I'm the product lead for MDN Web Docs, but I started as the, I started my career as a web developer back in 1997, and this is my first TPAC. So, seeing how the sausage is made was really interesting. And you all are really amazing.

So this project that I'm going to talk about started started shortly after CSS Grid was shipped in 2017, in late 2017.

CSS Grid was a massive success, so in the last developers relations team we ask ourselves: how can we have more of this, how can we duplicate the success of this, and how can we support something like this again?

And CSS Grid was a success because it addressed the real user need, it was shipped in roughly all the browsers, in roughly the same time, and it came with tooling - in our case, the CSS Grid inspector. And all of that was seriously awesome.

That said, CSS Grid was years in the making, and layout had been an issue for web developers at least since the time we abused tables to implement our UI designs back in the 90s.

And so, looking at this we had a few questions.

• Why did it take until 2012 to address this problem specifically?
• Why did it take until 2017 to ship?
• And how was this coordinated to roughly ship at the same time in different browsers?
• And what where the roadblocks to that?
• And what were the pivoting elements to moving this forward.

So lot's of, lot's of questions.

And so we took a step back, to see what the whole process looks like from Mozilla's perspective.

And after interviewing numerous people involved in different stages of the process, we identified three distinct phases, research, then standardization and implementation, and adoption at the end.

Now, as you can probably see, this is very simplified version that smooths out many things and looks like a pipeline when in reality it's actually more like a loop, or like a loop of loops, and loops that loop back on themselves, or something - you get the idea.

But, it was good enough for us to get started.

Now most of us had a pretty good idea about the adoption side, and that's also where MDN really shines, and we had some idea about the standardization and implementation phase, but we really wanted to know more about the research phase.

So, how do we learn about developer pain points, what they need, and how do we prioritize what we are going to ship on the Web platform.

And we interviewed more than ten people involved in different stages of this process, and it became clear that there was no formal research really.

At Mozilla, we have an amazing user research team for Firefox, but the same is not really true when it comes to developer research - not even close.

And it is important because at a place like Mozilla we have limited resources - we can't go after everything. And there are serious opportunity costs that we have to pay when we decide to do something, because it usually means that we cannot do something else.

And this is what we heard from people about how things are prioritized.

And there's one thing that really stood out from almost every single person saying the same thing: we need to hear more from developers, more from people who are not already in our channels.

That was something that we heard across the board, and it makes so much sense, because none of us can be successful without that part.

It's hard to prioritize the right thing without knowing about developer pain points, and it's hard to find the right solution, and it's hard to get people to adopt something if it's not solving one of their needs, or it's not solving it in the right way.

So for all of these reasons, we've proposed the Developer Needs Assessment.

So, what is that exactly? It's a prioritized list of designer and developer needs.

It's a simple tool for harsh prioritization representing diverse populations and a huge feature space.

And it's published on MDN: it's not owned by a single browser vendor, so we initially proposed this under the umbrella of our product advisory board where we have representation from browser vendors, W3C and industry stakeholders.

And as a community, we thought we need to have at least a common understanding of the facts when it comes to needs, even if you draw different conclusions from them.

And then finally, it's reproduced annually, and I think this is really important, as an industry or as a community, we need to know whether we are addressing developer pain points, and to understand whether we are prioritizing correctly, we need to track those needs and pain points over time, so that we can see the impact of our efforts.

So if you're prioritizing correctly, the needs list should look differently next year, and the thing that's at the top shouldn't be at the top anymore.

And if you do this well, we think the MDN Web Developer Needs Assessment, or MDN Web DNA for short, it can be the voice of designers and developers working on the web.

And asking about web developer designer needs and frustrations is not trivial. The problem space is so vast and the audience is extremely diverse.

So let me tell you a little bit about the process we went through to do that - and I apologize up front, I am not a researcher.

We worked with the very talented Allison McKeever, from Pinpoint Research, to do the research design and to conduct this, and I'll do my best to channel her in this.

So in January this year, we started the process by fielding the survey to all product advisory board members, asking for data they would use for decision making.

And we then had almost twenty in-depth interviews with web developers and designers before putting the initial version of the survey together.

And this is important, that's because we wanted the survey to be rooted in the voice of developers and designers from the very get go, and had we run the survey without those interviews, the questions would have been based on the internal perspectives of browser vendors.

By conducting the interviews, we made sure to derive the survey questions from the stories of developers and designers about what is important to them in their work, and on the web, but also things that cause them frustrations, so it was an outside in process, instead of from the inside out, and that was really important to us.

So to go from the interview findings to the survey we followed Pinpoint's analysis process, from observations, what we saw and heard in the interviews, to insights, what it means, and then up to critical themes, why it matters.

And we are continuing that process to incorporate the survey findings into what we learned from the pilot interviews.

What this means in practice is that we transcribed sixteen hours of interviews.

We then coded them.

And then we bucketed them to derive insights for the items in our survey.

And that is a very hands-on process, so still very analog.

And so based on these themes, we constructed the central survey question around the frustration of web developers and designers.

And then we then continued to work with our product advisory board on the overall survey with pilot interviews at every stage where we watched multiple people take the survey to ensure that survey would be as unambiguous as possible.

So with the help of more than thirty stakeholders from various MDN product advisory board members, member companies, we have put together the final version of the survey, and it was fielded in July this year, on MDN but also through the channels of our product advisory board members, through W3C, Google, Microsoft, Samsung, and Boku.

And not just in English like here, we also localized it, the survey and the call to action into 8 languages: this is the Japanese one, but also into Arabic, simplified Chinese, English, French, Korean, Brazilian Portuguese, Russian and Spanish.

And I'm very happy to announce that after fielding the survey for four weeks, more than 76,000 developers and designers took us up on that, and more than 28,000 developers and designers took the 20 minutes necessary to complete the full survey. And that's from a 173 countries total.

So, yeah, absolutely amazing.

I just to underline this, just from the complete responses alone, that's more than ten, or almost 10,000 hours of time contributed from the community, from developers and designers to help us understand what their needs and frustrations are.

So we believe that this makes the MDN Web Developer Needs Assessment the biggest developer and designer focused survey ever conducted.

And with that scale come benefits.

So we can segment the data by experience level, by developer role, country, by satisfaction level, and we can still retain a high level of confidence for the results that we get.

So after talking about the survey in the abstract for so long, I'd like to share some questions with you aside from the main question of the survey about frustrations.

And these are in no particular order, so the first one we have here is: how would you rate your overall satisfaction with the Web, as a platform and set of tools, to enable you to build what you need or want?

Another one we had was, What are the things that you would like to be able to do on the Web but lack web platform features to do?

And then, What are the biggest pain points for you when it comes to JavaScript development?

And we had the same question for HTML, SCSS and Web Assembly as well.

And because it's boring to just see the questions, let me share a little more.

So the first one is, List of the top five items that cause developers the most frustration. And this is across all countries, roles, experience levels, et cetera. So this is the main question of the survey.

1. First one on the list is interoperability, having to support specific browsers, and yes we do have a follow up question about which browsers.
2. And the second on the list is documentation, but this is very specific. It's documentation for frameworks and libraries, not web standards - we've asked for that too, but that's further down the list.
3. The third on the list, is once again, an interoperability issue - and this time, the fact that developers have to avoid using a feature because it's not available broadly.
4. Fourth one the list is testing across browsers.
5. And fifth on the list, is once again, an interoperability issue. And this one about making a design look and work the same across browsers.

I don't know about you, but I see a theme here.

So in our final report, we'll go back to our sixteen hours of interviews, and add detail and color from those interviews to these findings.

The second item I want to share is this one.

When a new web technology relevant to your needs becomes available, what are the main barriers to adoption?

Usually, when this question is asked, regardless of platform, it's documentation and sample code that is, that's what developers really care about, that's usually the number one spot. And it is listed here, but it's in second position.

The number one barrier to adoption on the web platform is broad interoperability across browsers.

It's not all bad though.

And here is another question that I wanted to share with you.

What are the biggest pain points for you when it comes to HTML development.

And the answer is, nothing.

For at least one-third of the audience, for the people we've asked, they don't have pain points when it comes to HTML, which is pretty amazing.

And finally, How would you rate your overall satisfaction with the Web, as a platform and set of tools, to enable you to build what you need or want?

And a full 76 percent are either satisfied or very satisfied, and that's something to be proud of, here of all places, yeah.I hope you keep that in mind when you hear complaints about the web developer experience.

So this is pretty awesome to see, and this and so much more will be in the full report that we intend to publish in November this year.

And we'll give a sneak preview of more of the results at the breakout session at 11:00 a.m.

But as I said, we want this to be an annual thing.

So the next iteration, we will start in March 2020, and we'll field the surveys in May 2020, and then hopefully publish the results in September, and again hopefully in time for TPAC 2020, so we can share the results there.

And this is where we need your help and input.

This is the first iteration of the survey, and its far from perfect, but its a start, and you are the target audience for the results. So we want the next iteration to be much better than this. We need your help for that.

We need you to tell us how you want to reuse the results, and how useful this is for you, and what the granularity level is that you need to make decision based on this data set, and tell us what segmentations you need when you need to prioritize.

We want to make the survey as actionable as possible, and with your help we hope that we can iterate on this to really make it as actionable as possible.

And the goal is to hear the voice of designers and developers working on the web, and to have a single and simple tool for harsh prioritization representing diverse populations in a huge feature space, and I'm extremely excited that we have made this first step in this direction, and many thanks to everyone who has contributed to that, and I'm looking forward to many more.