Improving Web Advertising BG

02 February 2021


AramZS, arnaud_blanchard, arnoldrw, blassey, bleparmentier, bmay, br-rtbhouse, brendan-eyeo, btsavage, charlieharrison, cpn, DGill_Criteo, dialtone, dinesh, dkwestbr, ErikAnderson, eriktaubeneck, GarrettJohnson, gendler, hober, hong, imeyers, jrobert, jrosewell, Karen, kleber, lbasdevant, Mike_Pisula, Mikjuo, pbannist, pedro_alvarado, pl_mrcy, seanbedford, shigeki, wbaker, wseltzer
Wendy Seltzer
Karen, Karen Myers

Meeting minutes

<wseltzer> https://github.com/WICG/turtledove/blob/master/FLEDGE.md

<wseltzer> https://w3c.github.io/web-advertising/dashboard/

Wendy: I hope our connection will be uneventful today.
… please "present+" yourself as you join the irc channel and "q+" to ask questions or make comments

Wendy: Let's start looking at the agenda
… continuation of our Q&A on FLEDGE; dashboard highlights
… any other business, and some people asking about other relevant meetings
… Do a go-round on other places where people are discussing these issues
… and a brief note on repository management
… We seem to have a good number of people, so let's get started

Agenda-curation, introductions

Wendy: with our first agendum
… agenda curation and introductions?
… new business or new participants who would like to introduce themselves?

Amelia Wellington, VP Product from Captify

Wendy: Welcome

Eric: Just agenda item question
… I think last week we were asking questions about FLOC
… hoping we could ask more questions about FLOC; not sure if that is also in the Q&A

Wendy: yes, continuing the discussion when we got cut off when our connections went down
… hope they are stronger today

Michael: I am happy to talk about the FLOC topic
… great if we can not intermix the FLEDGE and FLOC discussions

<eriktaubeneck> +1 to Michael's suggestion

Michael: since they are two quite different proposals
… minor preference there

Wendy: So since we started with presentation on FLEDGE, invite Q&A on that
… and then capture FLOC as another topic

any other agenda suggestions?
… are we queuing for the discussion...


Brian: I appreciate you putting together the FLEDGE proposal
… first question, there are a number of instances in IG attributes which refer to minimum populations

<btsavage> I'd love to discuss Webkit's announcement: https://webkit.org/blog/11529/introducing-private-click-measurement-pcm/

Brian: wondering how you figure out what those population sizes are

Michael: Are you asking from technical POV
… how many people are in browser, or policy POV how to pick?

Brian: First one

Michael: on tech side, in FLEDGE, all the infrastructure Chrome plans to build out for aggregate reporting and measurement
… is a system still being designed

<wseltzer> FLEDGE

Michael: a way for parties to learn how stuff is happening on different browsers
… even if too sensitive to happen on an individual browser
… the reporting we have been talking about....we talked about using aggregate reporting
… the underlying mechanism that relies on is crypto reports and sends to serve
… only possible to get aggregate info out
… and discussion on how to do multiparty computation
… same sort of aggregate server used by adtech companies can be used by the browser itself
… to report how many people...and browser to render...

Brian: So browser submits data and queries the browser reporting in a separate interaction?

Michael: Don't have to be separate interactions
… if we show ad to 100 people in last hour
… browser can take something hash of the rendering URL
… or whatever designates a particular ad and do some encrypted thing
… and talk to server
… please increment this thing
… and tell me what you see in last hour
… and return an aggregate answer for yes or not if happened a hundred times in last day
… that increments and lets browser know if it's over the threshold

Brian: How does the server know it's receiving from 100 or a single browser?

Michael: rests with the browser
… each copy of Chrome...can reach out to this server
… and say here is an ad I'd like to render
… first time in 24 hours, so please add to counter
… or browser talks to server, and says I already asked, so do not add to counter, but add to threshold

Brian: As I was going through on-device auction

<brendan-eyeo> This is interesting, but seems to be a tangent of educating on published information.

Brian: thinking that it's not always readily apparent to me what the trade-offs are
… as we move to browser based functionality
… while proposals are being put together, are people considering what kind of parity is needed
… and add a section on how FLEDGE models compare and contrast with existing model
… Is that something you guys to put together?

Michael: yes, I could put something together that explores that
… I think the explainer by the NextRoll folks does a good job

<wseltzer> TERN

Michael: TERN does not have the contacted trusted server and do budget constraints
… but TERN would be a good starting place for some of the questions you are asking

Brian: Would be helpful to have that sort of what is happening now and how to create parity in the new model
… I used to be involved in DSP
… less involved now, so would be helpful to see where tradeoffs are happening
… I have watched the process in W3C and other groups
… Looks like we are waiting for the Chrome team to know what model we will have going forward
… I am a little concerned that we are tying our businesses to decisions that Chrome team does, independent of the ecosystem
… Wondering what kind of effort Chrome team is making
… as we redevelop our businesses around the new model to get support from other browser vendors as well

Michael: concerned that is question for other browser vendors

Brian: We are making significant commitments, rebuilding to work with FLEDGE, FLOC and Chrome team
… we have to ask ourselves if we are building just to Chrome team or with a broader ecosystem
… want to get some sense of how much effort Chrome team is working with other browser vendors to adopt what you are doing

Michael: as with all web development
… Chrome does not do anything unless we see a path to become a widely adopted cross-browser standard
… we would push for something that all browsers would adopt
… not looking to do Chrome only

<jrosewell> +1 to bmay - a separate web for Chrom(ium) is contrary to the W3C mission and purpose as encapsulated by "One Web" which all W3C members accept. Google are asking the eco system to make a significant investment in this experiment.

Michael: and help the web platform in general
… that said, I am not in position to speak for any other browser
… historically, things end up in web platform
… but there is a time skew when things show up
… some things like service workers take years to be cross-browser
… a long time when they were not
… not sure if that answers your question, but we do want to be cross-browser

Brendan_eyeo: [missed] plans for profiling

<AramZS> Interesting on the chrome extension - can you link it if it is public?

Brendan_eyeo: looking at same problems outside of main browser space

Brian: What I would like to get some guidance on
… as we move forward with our work around Chrome, is there some commitment, not nec official
… to make best efforts to get adoption of things you are working with; or are you saying you hope others will follow along

<brendan-eyeo> https://crumbs.org/

Brian: are you actively engaging with them on a broad-stakeholder basis

Michael: yes we are engaging with the entire web platform ecosystem

Wendy: yes, and reflect that is a goal of the whole W3C process

<AramZS> Thanks Brendan!

Wendy: to gather community discussion, id points of consensus, and have forums to build that consensus
… Pleased to hear the discussion here, and things going into incubation groups, and as things might move to Working Groups
… that is where we look at broad, cross-platform implementation

<Zakim> wseltzer, you wanted to react to bmay

Brian: makes sense, thank you

Ben: I jumped on queue when you were asking for agenda items
… not sure if there will be time for it
… John Wilander published a blog post
… on App to Web Measurement
… we have our first browser announcing support for a new convergence measurement API
… that supports app to web measurement
… not sure if anyone else on calls knows about it, or hear from Charlie and Michael about it

Wendy: I note that as an agenda item for today if time or future call

Erik: hey, Michael, thanks for the update for FLEDGE
… enjoy seeing how it's coming together and want to play around with it
… You addressed this earlier
… once you hit 100 hits to the URL without anything else, you can start targeting...
… maybe do for only the last hour
… have you thought about how you decide what that policy looks like
… how do we pick that number and the rules around that number?

Michael: That seems like something should be an eco-system wide discussion

<wseltzer> https://github.com/WICG/turtledove/blob/master/FLEDGE.md#12-interest-group-attributes

Michael: with other browsers...reports on research...adtech and impact of economic decisions
… like to discuss publicly
… I used 100 times in a day just to have a number there
… not because it's more right than any other number

Erik: great, look forward to that

Lbasdevant: Question about FLEDGE
… think we should include some KPIs
… agree on some indicators we want to measure through the experiement to know if...
… we should agree on this KPI first and the way to ... events
… and agree on measurement across the ecosystem
… if we do that, each adtech company and publisher...otherwise we end up with widely different sytems
… would you agree to set some KPIs and set steps on how to measure?

Michael: Thank you, Lionel, that is a good question
… I am hesitant
… I worry that
… Chrome should not try to be the boss on how people evaluate these proposals
… I would be supportive of people in adtech picking KPIs and running some experimentation
… Folks on ad side working on RTB

<jrosewell> lbasdevant: Agree. Given the investment everyone is making agreeing on the success / evaluation criteria for the experiment seems essential before any work starts. This forum is a great place to define those KPIs.

Michael: are in a better place to do that kind of thing
… I would really...
… Problem is that our goal is not to build a solution, but rather to build a solution for others to build solutions in different ways
… how good a job FLEDGE supports use cases would be different for third-party cookies
… versus some based on FLOC, TD, or IG proposals
… I worry about pushing for something that is too one-size-fits-all for evaluation

Lionel: On the other end, my worry is
… we go down the technical route
… we will have many different players using the FLEDGE APIs

… and other players...with cookies
… at that point we don't have a view of what impact


Michael: I understand

<jrosewell> lbasdevant asked for evaluation criteria to be defined. He didn't ask for a "one size fits all" outcome.

Michael: different people in adtech doing their own experiments and investigations
… and reporting to this group in general seems to be the right way to handle it
… I worry about Chrome deciding how they should be evaluated

Lionel: thank you

Arnaud: just to jump on your exchange with Lionel
… how do you reconcile that you said several times
… you will phase out cookies when you have a satisfying solution
… how do you measure the satisfaction?
… can you elaborate on that?

Michael: yeah
… that is a good question
… So I think that if there are
… use cases
… that we collectively have identified, or individual people have identify and we think it's possible to address those use cases
… to fit inside the privacy model
… then we will spend time trying to add support for those use cases
… that is the approach we have been doing together for the past year

<jrosewell> What does "appropriately preserves" privacy mean? That's key to the evaluation criteria.

Michael: That is what we are looking for here; id use cases that can be supported in a private way

Arnaud: So we need to revisit the list of use cases
… and on publisher side, see which proposal ticks most of the boxes to define what it means
… we end up with criteria to determine if solution is acceptable to the community, right?

Michael: figuring out if we have addressed all the use cases
… and figure out all things to add, and come up with additional proposals

<lbasdevant> For the minutes "I worry that if we go down the technical route, at some point many companies will be using the FLEDGE APIs, and that will be enough for deciding to retire 3p cookies, without evaluating first the economic / user impact".

Michael: I see my job as have we addressed collectively all the things we can do, and address all the use cases of the ecosystem
… seems feasible to do
… ,that is the standard by which I am trying to evaluate proposals

Arnaud: once we set a proposal, how do you define the options in those proposals
… so TD, Sparrow, different approaches...for size of IG for example
… and different impacts on use case
… how can we help you guys in determining

Michael: Similar to Erik's question, there should be a wide ranging discussion on parameters
… and discuss utility and trade-offs
… those don't have to be set in stone
… adjustable parameters over time

Arnaud: How can we raise the fact that cohorts are too big or too small
… or we cannot do IGs fast enough
… today we discuss as general concepts, but once we start doing them, what is protocol to improve the setup

Michael: I think putting Github repository issues
… and make some change in future issue

Arnaud: the policy and steering of the parameter will be done in Github only?
… there won't be another way to do so?

mvale: Marshall, product manager for privacy sandbox, goal is to get proposals out, get trials out, get ecosystem to participate in the experiments
… and we can look at the parameters here in these forums

Arnaud: so an outcome is a defnition of these parameters

<joshua_koran> to ensure we all look at the "right" results, don't we need to agree on which metrics we will be measuring?

Marshall: there is always a range of parameters to be explore
… and not determine...there are a variety of people coming from diff perspectives
… important to show the whole range; that is what will help to explore across the broadest range
… if you anchor too quickly on one target, that becomes the solutions
… but important to have different use cases and perspectives on how the parameters will be used across ecosystem

Arnaud: what will be the metric?

Marshall: that is also part of the discussion; important to bring those to the table
… feels like a broad range of scope, but it's important to see how folks are measuring it

Arnaud: important to know what will solve a business problem
… we are going to implement stuff if it's going to work, and don't know if it allows us to measure anything of value
… these are design decisions
… only way to do so is to experiment independently and raise as a Github issue
… don't take this wrong
… but you are the common denominator of these discussions, and you should facilitate a sharing of measurement practice for this test and the goals we want to reach
… by testing these proposals instead of asking if it satisfies you
… no one @ that side completely
… unless we define goals beforehand, I think we are going to get stuck

Wendy: I see queue...

James: I totally agree with previous speakers about the big investment being asked
… and the term privacy was asked
… the entity that constructs and consumes a cohort; what is appropriate privacy for them; where and who defines that
… anyone?

Marshall: I will go back to earlier response
… this is part of that evaluation through experiments that folks are doing
… with the proposals and origin trials

<joshua_koran> An experiment suggests we will measure something to determine if it is successful - what will we measure?

Marshall: and experiments along utility...and look at things collectively

James: the question is what privacy looks like

Marshall: that's how I am responding to it

Julien: basically I was interested to
… define metrics of success
… why the general principal from beginning is clear
… privacy...I did not find a clear definition of how to be measured
… and how to separate use cases
… l am a bit worried we are debating merits of technical implementations without defining clear KPIs

<wseltzer> https://github.com/michaelkleber/privacy-model

Marshall: early on we published information on general privacy objectives we are trying to achieve in the sandbox

Wendy: thanks

<jrosewell> Marshall: does consensus exist concerning those privacy objectives? If so where?

Apireno: maybe a more high level question
… have you thought in terms of model size
… how...to address privacy part
… in exchange
… or [missed]

Alessandro: in terms of model bias
… when someone wants to determine if bias is part of decision; how do you see that shaking out
… is it in the browser, or how do you go about testing or eliminating the bias

Michael: sounds like question is about model bias
… in the FLEDGE proposal the browser is not making decisoins
… in the business of preserving privacy around decisions
… perhaps you are thinking of FLOC proposal where browser does some clustering
… and that can be used by adtech as input to their own decision-making
… true there is a possibility of bias anytime with machine learning
… browser needs to be aware of potential bias; and users of FLOC
… need to be aware, just as anyone dealing with ML understands today

Alessandro: testing decision making
… you could not do; not enough granularity
… or probability...modeling limited for lack of a better term

Michael: not sure I understand question; perhaps better to put this on the Github repo

Wendy: last for this item today

Viraj: one thing I wanted to understand
… in the new proposal opening for a trusted entity for @
… how in using this cluster
… see it's being used... for server
… for campaign budget
… while bidding happens on the browser
… is that sufficient
… where are using that cluster for @
… ratings might be simpler to implement
… can you elaborate?

Michael: sure
… the underlying privacy model of FLEDGE
… requires us not leaking certain information
… requires having a guarantee that publishers cannot see what individuals...[missed]
… on device and trusted server are in pursuit of those variants and not leak
… Answer is what info or tasks to hand off to a trusted server
… it is a complicated question that depends upon the trust model
… FLEDGE is picking a trusted server model, and least amount of trust to address the use cases like budget
… that are impossible to do with an entirely on-browser solution
… so the trusted model I was thinking of using multi-party computation or other oblivious data base lookups
… or other well understood pieces of tech
… taking something that seems plausible for browser to have trust in
… might be future in which there is future to have a much stronger trust model
… where the browser has the confidence to offload more computation on to server
… because browser has a way to guarantee how data is used
… better a trust model we come up with for the trusted server, the more info the browser would be confident to hand to the trusted server
… Trying to start as simply as possible

Viraj: so more from the feasibility POV, and we can build on top of that
… thanks

Wendy: Look over things in queue
… we wanted to come back to FLOC discussion
… I will put a call on the list
… for pointers to other relevant meetings
… we hear that FLEDGE will be discussed in WICG
… and some people asking about the difference in discussion on those calls v what comes back to Web Adv calls

<kleber> Yes! FLEDGE meeting tomorrow at this hour! https://github.com/WICG/turtledove/issues/88

Wendy: invite those leading those discussions to share a bit on the list
… as to what is happening where
… and reminder of how to get invoved
… Michael notes a FLEDGE meeting tomorrow
… assume that is to address what is in GIthub and diving into the details
… anyone from those discussions is welcome to bring them back at relevant points
… The PCM issue that Ben raised at beginning
… I saw that Tess was on call earlier, but she had to leave
… So let's bring that back when Apple can answer questions you might have
… And I wanted to note briefly on the repository
… we have been looking to switch term from "master" to "main"

<kleber> Re. /master/ -> /main/ — thank you!

Wendy: we will be making that change for people who have references to things in repository
… and make sure links will be updated
… it's because of connotations of "master" and want to avoid that term
… want anyone contributing to our code to see a more neutral term
… those are my administrative announcements


Wendy: Happy to turn the discussion back to FLOC and open the queue

Erik: on the FLOC statistic that was published last week
… there was mention of an experiment on the call
… want to get clarity if that was offline or online test?
… actively informing ad delivery, or on a previous delivery? And how FLOC would change that

Deepak: I work on Google ad system
… answer it was an A-B experiment done on light traffic
… conducted on data from Google's DSP system
… we looked at all the third party history browsing behavior on all systems and put into buckets for FLOC ID
… small persons...allow third party cookie systems
… for third party system you redact the PID and use the user's FLOC
… this experiment was done orthogonally to our retargeting system which was supposed to be addressed by FLEDGE
… we have inmarket and affinity systems
… inmarket for @ and affinity for branding
… we tested these against ROI
… and measures for audience quality
… measure was against ROI stats or conversion per dollar
… across multiple advertisers
… we do some stats...mental Mantel Haenszel
… we do have a white paper we are writing up

<joshua_koran> Does that mean (retargeting + cohorts) vs (retargeting + other attributes)?

Deepak: what I said will be published in Github or next week
… we are going through some checks and balances right now

Erik: that is helpful
… did you use the simhash method described in paper, or some other clustering system?

Deepak: We did simhash
… sorting is way to make sure wide cluster...has guarantees...for min. of X users
… so we used that combination


Erik: Do you anticipate that being used in FLOC
… when in Chrome

Michael: simhash + sorting LSH...will go into Chrome 89 coming in March

Erik: Awesome, thank you

Brian: First, thank you very much to you, Michael for all of your efforts and all you are doing for this group
… is there a possibility to have an explainer for origin trials on FLOC
… have spec directions on how to do that

Michael: absolutely, we will have developer relations do report on how to do origin trials
… even people used to origin trials will need some guidance on this one

<mjv> https://web.dev/origin-trials/

<mjv> https://web.dev/third-party-origin-trials/

Brian: Thank you very much

Wendy: We are at the end of our queue and the end of our time
… as we heard, a FLEDGE meeting tomorrow for those interested in more technical details
… and follow-up on WICG threads
… and join the Web Platform CG to participate
… Community Groups are open to all, anyone welcome to participate
… We will see you back here next week
… and please share questions or comments with the list
… thank you from rainy/snowy Boston
… at least connections held up

<wseltzer> [adjourned]

Wendy: and thanks Karen for scribing

Minutes manually created (not a transcript), formatted by scribe.perl version 127 (Wed Dec 30 17:39:58 2020 UTC).


Succeeded: s/options/auction/

Succeeded: s/@/NextRoll/

Succeeded: s/answered//

Succeeded: s/@/Brendan_eyeo/

Succeeded: s/Millender/Wilander/

Succeeded: s/It's "John Wilander" not "John Millender"//

Succeeded: s/APIs/KPIs/

Succeeded: s/API/KPI/

Succeeded: s/APIs/KPIs/

Succeeded: s/Marshall, product manager for sandbox:/mvale: Marshall, product manager for privacy sandbox,/

Succeeded: s/open entity/opening for a trusted entity/

Succeeded: s/@/Deepak/

Succeeded: s/@/hassle/

Succeeded: s/hassle/Mantel Haenszel/

Succeeded: s/LH/LSH/

Succeeded: s/we will need more metrics on this one/even people used to origin trials will need some guidance on this one/

No scribenick or scribe found. Guessed: Karen

Maybe present: Alessandro, Apireno, Arnaud, Ben, Brendan_eyeo, Brian, Deepak, Eric, Erik, James, Julien, Lionel, Marshall, Michael, mvale, Viraj, Wendy