Credibility Group – The Next Chapter

• This group is going to change, at least temporarily, from one building credibility indicators, to one that tests efforts to improve the information ecosystem, based in part on the credibility indicators documented previously by this group.
• Building a “test suite” for anti-misinformation efforts will help all those efforts, we hope, improve.
• We need a fresh start and some new expertise to make this happen. We hope that you will re-engage with this group, and invite others who can help make this group even stronger.

Below is an edited version of a note that I wrote recently to the group email. I’ll expand on that and have the next steps below:

In the years since I first met Sandro and then joined this group, I’ve seen what happened in this email thread happen a number of times. I’ve also seen it happen in CredCo threads, MisinfoCon discussions, and other groups.

What happens is essentially this: Someone proposes a new thing, let’s call it a MacGuffin. The person who proposes this MacGuffin explains it in great detail, yet is a bit hand-wavy on some of the aspects, especially those about how it will be supported, or how it would get adoption. That doesn’t bother me at all. If we didn’t have people dreaming up new things we wouldn’t have anything new. But the reality is that these are just ideas, not actual initiatives.

Then in Act II there is some discussion about the MacGuffin, talking about the pros and the cons, etc.

Then there is a pause. This pause comes because the group is made up of people who all have full-time jobs. People who have jobs can’t just drop everything and put in the work needed to launch a new MacGuffin. In some cases they can to some degree, which is what Sandro has done with TrustLamp. He would be the first to tell you, I think, how hard that is.

After the pause comes Act III, in which some people who are in the group realize that much of the MacGuffin is a lot like what they are already doing, and so they promote their own thing. Or sometimes they provide a link to something else that exists that is similar.

After that, the play is over, everyone goes home. And then after a while it happens again.

My suggestion (and the reason that I ran for the chair) is that we reverse the order of this play. Rather than: 1. Idea, 2. Evaluation, 3. Look at related existing initiatives. We do this: 1. Look at existing initiatives, 2. Evaluation, 3. (with luck) Propose new ideas that fill an existing and yet un-served need.

To evaluate the current initiatives effectively, I would propose that first we come up with some guidelines. To do that, we start with the documents we have and that we have all agreed to. Then we turn those into the start of a framework for evaluation. Once we agree to that framework, we publish it. 

That will give this group the relatively quick win of publishing something that can be used by anyone as they are looking at existing initiatives, or are thinking about starting something new. It essentially puts this group in the middle of many conversations happening about misinformation. It will help everyone to clarify what can actually help, whom it helps, how much it helps, and how much downside there may be.

Once we have that document, we can then decide if we want to meet regularly and evaluate initiatives based on that document, or create a new group to do that, or examine the whole landscape and figure out if there’s something that would be appropriate for this W3C group to try to do next.

I say all this not to say that the original idea doesn’t have merit, or that any idea discussed here (including my own trust.txt) is great or sucks. I’m just saying it would help the world, and each of us individually, if we could evaluate ideas based on a common vocabulary.

That was the note that I sent to all the current members of the group after a round of emails, and I was indeed elected chair, along with the incomparable Aviv Ovadya. Here’s what I am now proposing that this group does, more specifically:

Members

I’m first going to propose that we expand the membership. I will be reaching out to some journalists that I know, and I’d encourage all of the current members to reach out to at least one other person as well. We have a lot of technical firepower in this group now when it comes to technical standards, but we don’t have journalists, even though this group’s goals are inextricably bound up with journalism. We are talking about systems of credibility for people, but we don’t have deep expertise in user interface, psychology, or any other discipline related to distribution and consumption of credible information online. If we want to fix a problem that big, we need a lot more tools in the toolbox.

If you are reading this and are not a current member of the group, here is the link with instructions on how to join.

Meetings

It’s been so long since we had regular meetings, that as soon as we’ve grown the group to the size and with the profile that meets the above goal, we should send out a new doodle poll to find a time to meet that works as well as it can work. My suggestion is that we shoot for a monthly cadence at the start.

Process for a new working document

There are several ways we could do this, but I think perhaps for the sake of expediency, the best might be this: I will create a first working draft of the document. It will be rough, have giant holes, and probably will bear no qualities that will be in the final draft. But at least we’ll have a document.

I’ll send it out to the members of the group at least two weeks before the first meeting. The agenda for the first meeting will be to get a sense from the group that the document is indeed the direction that the Credibility Group wants to move in. We’ll also come up with a target for when we’ll be ready to publish a public draft, and then the final document. My hope is that the final document will come about six months after the first meeting.

At that first meeting, my hope is that we’ll get at least a couple of volunteers who will be willing to take a stab at either writing the bits to fill in the holes, or even running small sub-committees to hash through the thorny issues and present potential language to the larger group.

Long-range goals

The document I’m talking about here is somewhat different from most W3C documents. It won’t be a standard in the sense that it will be a reference document. It will be, I hope, a document that can be used to evaluate just about anything that hopes to work on the web to fight misinformation in general. So, in that way, it’s a bit like a tool that a W3C group might come up with. For instance, W3C has a long history of building and advocating for tests to drive interoperability, like this suite of tests for CSS, and these for internationalization. But instead of testing code, this test will be one to check misinformation efforts.

Indeed, some of the questions we have will be similar, I’m guessing. For example, if there’s a proposed effort to to find credibility signals on a page, will it work in other languages? Will it be available to people with visual or other disabilities? Will it work in all browsers/platforms/devices?

While the test that we create may not be able to be run as software, I would think we’d want it to be as close to software as possible. For instance, if some new effort claims that it works internationally, the person making that claim should have some evidence to make that claim verifiable by anyone.

What will this document look like?

This will be up to the group, but as one of the co-chairs, a guiding principle for me be the idea that this testing document be as universal as possible. That is, I don’t want any document to get too far away from its implementation. Perhaps the document that we create isn’t a document at all, but instead is a Google Form that is filled out answering questions about misinformation-fighting efforts, and the resulting data is available for all.

I envision us creating what educators would call a rubric, a guide that will allow us to grade efforts that are built to improve the information infrastructure of the open web.

My goal is that we create something that builds a new body of information that will be really useful in this fast-changing world. What exactly will happen with it is unknown, but I do know that when I talk to people in the world of misinformation I find that the conversation often turns to trying to remember which effort does what exactly. If we can provide a quick common basis for analysis and familiar and easily searchable repository, it will help us that are inside of the efforts first. From there, it should make all of us just a bit stronger and better informed.

I hope you will join us on this journey.

-Scott Yates