14:48:52 <RRSAgent> RRSAgent has joined #web-speech
14:48:56 <RRSAgent> logging to https://www.w3.org/2024/09/25-web-speech-irc
14:48:56 <tpac-breakout-bot> RRSAgent, do not leave
14:48:57 <tpac-breakout-bot> RRSAgent, make logs public
14:48:58 <tpac-breakout-bot> Meeting: Web Speech API Improvements
14:48:58 <tpac-breakout-bot> Chair: evanbliu
14:48:58 <tpac-breakout-bot> Agenda: https://github.com/w3c/tpac2024-breakouts/issues/21
14:48:58 <Zakim> Zakim has joined #web-speech
14:48:59 <tpac-breakout-bot> Zakim, clear agenda
14:48:59 <Zakim> agenda cleared
14:48:59 <tpac-breakout-bot> Zakim, agenda+ Pick a scribe
14:49:00 <Zakim> agendum 1 added
14:49:00 <tpac-breakout-bot> Zakim, agenda+ Reminders: code of conduct, health policies, recorded session policy
14:49:00 <Zakim> agendum 2 added
14:49:00 <tpac-breakout-bot> Zakim, agenda+ Goal of this session
14:49:01 <Zakim> agendum 3 added
14:49:01 <tpac-breakout-bot> Zakim, agenda+ Discussion
14:49:01 <Zakim> agendum 4 added
14:49:01 <tpac-breakout-bot> Zakim, agenda+ Next steps / where discussion continues
14:49:02 <Zakim> agendum 5 added
14:49:02 <tpac-breakout-bot> tpac-breakout-bot has left #web-speech
21:46:16 <dom> dom has joined #web-speech
21:49:08 <tidoust> tidoust has joined #web-speech
21:54:42 <hgo> hgo has joined #web-speech
21:56:34 <nigel_> nigel_ has joined #web-speech
21:56:49 <nigel_> Present+ Nigel_Megitt
21:58:19 <nigel> scribe+ nigel
21:58:32 <nigel> Early introductions (no scribe)
21:58:51 <nigel> Chair: Evan_Liu
21:59:03 <mjwilson> mjwilson has joined #web-speech
21:59:06 <nigel> dom: Is there a link to LLMs?
21:59:26 <nigel> Evan_Liu: We might use LLMs in Google internally, but the API won't change
21:59:49 <nigel> dom: Signal to abort processing?
21:59:59 <nigel> Evan_Liu: Probably out of scope.
22:00:11 <nigel> .. We're not talking about changing the core capabilities beyond introducing this new functionality
22:00:38 <nigel> Topic: Offline Speech Recognition
22:00:52 <jcraig> jcraig has joined #web-speech
22:01:07 <jcraig> https://github.com/WICG/speech-api/issues/108
22:01:23 <nigel> [slide]
22:01:33 <nigel> Evan_Liu: These proposals have been around for a while,
22:01:44 <nigel> .. for on-device speech recognition
22:02:18 <nigel> .. Proposing to support by introducing two new attributes on the SpeechRecognition interface
22:02:25 <nigel> .. localService attribute
22:02:31 <nigel> .. allowCloudFallback attribute
22:02:38 <nigel> .. both Booleans. Names may change.
22:02:54 <nigel> .. Cloud speech to text service can support more options than on device
22:03:04 <nigel> .. Might combine the bools into a 3-value enum
22:03:14 <nigel> .. Idea is to control where the speech recognition is allowed to happen.
22:03:20 <nigel> .. New methods
22:03:30 <nigel> .. allow triggering download of a language pack
22:03:41 <nigel> .. query if on-device speech recognition is available
22:03:48 <nigel> .. Privacy concerns
22:04:06 <nigel> .. Chrome planning to only allow websites to install a language pack if it matches the user's
22:04:27 <nigel> .. primary language, or if not, ask permission
22:04:39 <nigel> Jer: What's the use case for disabling local speech recognition?
22:05:00 <nigel> Evan_Liu: More about allowing server-based STT, but also users may
22:05:14 <nigel> .. not want to use the CPU on their machine
22:05:25 <nigel> Jer: When would you ever set local service to false.
22:05:42 <nigel> .. Sounds like only if the website knows it may be doing other cpu intensive tasks
22:06:39 <nigel> discussion of on/off device use cases, including device capabilities
22:07:19 <smaug> smaug has joined #web-speech
22:07:24 <smaug> asdf
22:07:36 <smaug> (smaug == Olli Pettay)
22:07:40 <TylerWilcock> TylerWilcock has joined #web-speech
22:07:49 <solis> solis has joined #web-speech
22:07:50 <nigel> smaug: [missed]
22:07:57 <ningxin> ningxin has joined #web-speech
22:08:12 <nigel> michael_wilson: Why not an enum?
22:08:12 <eric_carlson> eric_carlson has joined #web-speech
22:08:22 <nigel> Evan_Lie: No reason, seems like it would be simpler based on this discussion
22:08:36 <jcraig> s/discussion of on/HN: discussion of on/
22:08:39 <nigel> Jer: An enum with 3 values would mean each option is clearer to understand
22:09:03 <nigel> smaug: the "on" prefix...
22:09:11 <mjwilson> mjwilson has joined #web-speech
22:09:15 <nigel> Evan_Liu: Good point maybe should be "is"
22:09:25 <nigel> Jer: Seems like a property, so a getter would make sense
22:09:54 <nigel> solis: On Device might make more sense than Local
22:10:07 <nigel> Evan_Liu: It's matching existing spec text, I prefer On Device too. Any preferences?
22:10:19 <nigel> dom: The boundary of a device might evolve over time.
22:10:31 <nigel> .. Trying to communicate the privacy aspect of the choice here.
22:10:46 <nigel> .. Instead of focusing on this abstract boundary, focus on what is driving the developer's choice.
22:11:00 <nigel> .. Not sure it matters, as much as the privacy implications based on which environment is shown.
22:11:14 <nigel> .. Not sure how to formulate this.
22:11:17 <nigel> .. Hybrid models might exist, so this distinction might not be what we need.
22:11:27 <nigel> tidoust: The notion of "parties" comes to mind, but also not right.
22:11:40 <nigel> .. Could say it's First party for the UA, or 3rd party for someone else.
22:11:53 <nigel> .. Not sure that's the right way either.
22:11:59 <nigel> .. Google has plenty of hats here.
22:12:41 <nigel> Evan_Liu: installOnDeviceSpeechRecognition() - returns a boolean,
22:12:54 <nigel> .. could take minutes to download, so we just return if the fetch has been initiated.
22:13:06 <nigel> .. Could have event listeners to reveal when the installation is complete
22:13:16 <nigel> .. Or return true when the installation is complete
22:13:24 <nigel> eric_carlson: Some of the language packs are quite large.
22:13:38 <nigel> .. Do we need to be concerned about allowing a page to use a lot of user data in this way?
22:13:47 <nigel> .. e.g. people on limited data plans
22:14:06 <nigel> Evan_Liu: Another Chrome criterion is if the user is on cellular net or on WiFi/ethernet.
22:14:23 <nigel> .. Could be in the spec, or could be in the browser, depending on what people want.
22:14:37 <nigel> smaug: Feels v scary this install, needs to be async, and behind permissions.
22:14:45 <nigel> .. May never get past the installation.
22:14:51 <nigel> .. Maybe return a promise
22:15:13 <nigel> Evan_Liu: This would be async, and would return as soon as the user signals their preference
22:15:20 <nigel> smaug: User may not say anything
22:15:36 <nigel> Evan_Liu: Any concerns about that API?
22:15:46 <nigel> eric_carlson: Should be a promise
22:15:54 <nigel> .. May not ever return, or the download may time out
22:16:08 <nigel> .. To prevent polling, it should resolve once it's been downloaded and is available for use
22:16:14 <nigel> .. Polling is an anti-pattern
22:16:43 <tidoust> scribe+
22:16:44 <tidoust> nigel: Do you have a scheme in mind for caching if two pages want to fetch the same language?
22:17:11 <tidoust> Evan_Liu: For Chrome, there will always be only one package per language
22:17:28 <tidoust> smaug: There may be a privacy issue here
22:17:59 <tidoust> Evan_Liu: True. If it requires a pop-up that scares people away, we feel the fingerprinting issue goes away
22:18:14 <tidoust> eric_carlson: Same issue with fonts. Recurring issue.
22:19:15 <tidoust> s/privacy issue here/privacy issue here: where a page polls for installed languages. Fingerprinting problem.
22:19:55 <nigel> Jer: Not sure if this is something that could be partitioned
22:20:05 <nigel> .. Always ask the user for every page even if the language pack is already available
22:20:25 <nigel> eric_carlson: Interesting idea: there was a way to get microphone permissions, which was used
22:20:33 <nigel> .. for fingerprinting.
22:20:47 <nigel> .. We returned a fixed list until the user started to capture, and only then return the correct list
22:21:02 <nigel> .. Do something similar here, signal only the user's language signalled as available,
22:21:10 <nigel> .. until the API actually asks to download something.
22:21:19 <nigel> .. If the pack is already there, return more quickly.
22:21:32 <nigel> .. User would get prompted every time, which would be wierd.
22:21:42 <nigel> .. If you prompt on download request and then reveal what's actually available,
22:21:47 <nigel> .. that could be one way to handle it.
22:22:03 <nigel> dom: If you ignore media stream track at the minute, right now the speech api does go through
22:22:31 <nigel> .. a prompt. Agree that default language makes sense, but there may be an opportunity
22:22:38 <nigel> .. for bringing something into the prompt.
22:22:47 <nigel> .. It's super privacy invasive to do recognition in the first place
22:23:03 <nigel> eric_carlson: Could be that microphone permission is enough of a barrier
22:23:22 <nigel> mjwilson: Is it assumed to only work on mic, or other audio resources?
22:23:34 <nigel> eric_carlson: Wouldn't work for other audio streams
22:23:50 <nigel> Evan_Liu: Could have auto-download as soon as recognition begins
22:23:56 <nigel> .. Should this be in the spec?
22:24:08 <nigel> eric_carlson: The spec must have at least recommendations.
22:24:15 <nigel> .. Leave room for UAs to figure out other ways
22:24:28 <nigel> dom: If you leave privacy mitigations and let them have interop impact then its a race to the bottom
22:24:38 <nigel> .. so agree put something in the spec
22:24:55 <nigel> jer: Best way would be to only allow user's language STT
22:25:06 <nigel> .. Could imagine foreign language STT and then translation though
22:25:15 <nigel> .. Or could have a list of preferred language
22:25:27 <nigel> dom: Except for duolingo!
22:25:36 <nigel> jer: that already knows the text though.
22:25:59 <nigel> dom: hard to separate this from whether it only applies to live capture or not
22:26:15 <nigel> .. as soon as you cut that link it's not an effective mitigation, so need other mitigation for non-live capture
22:26:32 <nigel> jer: Misunderstood the duolingo example - the user is speaking and the page wants to know if
22:26:37 <nigel> .. they spoke correctly
22:26:40 <nigel> [slide]
22:27:13 <nigel> s/[slide]/
22:28:15 <tidoust> nigel: I used text to speech features of browsers and experienced divergence in performance across browsers. Especially during starting times.
22:28:33 <tidoust> ... Any plan to signal "readiness"?
22:28:39 <jcraig> q+
22:28:44 <tidoust> ... To avoid missing bits?
22:29:14 <tidoust> Evan_Liu: No real requirement in the spec that things must happen within a specified amount of time.
22:29:39 <tidoust> ??1: Presumably, that's a bug, not supposed to drop bits.
22:30:37 <tidoust> Nigel: What about extending to other types of sounds? Sometimes, it's not only speech you're interested in.
22:30:52 <tidoust> ... There are machine-based products that recognize sound.
22:31:43 <tidoust> ??1: My understanding of the proposal is that the main interest is converting speech to text.
22:31:56 <nigel> s/??1/jcraig
22:32:06 <nigel> Topic: MediaStreamTrack support
22:32:13 <jcraig> s/??1:/jcraig:/g
22:32:15 <nigel> [slide] github issue #66
22:32:46 <nigel> Evan_Liu: start() method requests permission to use microphone, and starts using it if allowed
22:33:15 <nigel> jer: If you send noise to a speech recognition API, you can figure out what platform is being used
22:33:29 <nigel> .. so its a big entropy change for fingerprinting
22:33:40 <nigel> .. needs to be listed in the spec, exposing arbitrary speech recognition
22:34:11 <tidoust> nigel: I haven't understood the use case properly. Why does the page need the API?
22:34:39 <eric_carlson> eric_carlson has joined #web-speech
22:34:43 <tidoust> Evan_Liu: For video conferencing tools, streams may be coming from elsewhere.
22:35:15 <tidoust> nigel: It does not seem very sustainable to do the work on all clients.
22:35:51 <nigel> s/API? Why not have the user trigger the microphone and provide a text stream? Why does the page need to know?
22:36:02 <nigel> s/API?/API? Why not have the user trigger the microphone and provide a text stream? Why does the page need to know?
22:36:38 <nigel> [discussion of use cases] - live translate and dub
22:36:56 <nigel> jcraig: Sender side could have better sound quality from local mic
22:37:13 <nigel> dom: Completely muted conference, only sending speech text
22:37:33 <nigel> jer: Mitigation could be a mandated delay so a website can't immediately get an answer.
22:37:55 <nigel> .. Require min 30s before producing text mitigates by forcing website to wait to get fingerprinting data
22:38:11 <nigel> .. One possible mitigation to privacy impacts. Not a solution, just increases cost for the page.
22:38:22 <nigel> Evan_Liu: Concern is profiling performance capabilities?
22:38:48 <mjwilson> q+
22:38:49 <nigel> jer: Not perf, each implementation might give a different answer, so tell devices apart
22:39:18 <nigel> eric_carlson: Might get fine grained information about the machine
22:39:35 <nigel> jer: Could be less risky for Chrome having a large number of users with the same characteristics
22:40:00 <nigel> mjwilson: This isn't theoretical, it actually happened with Web Audio, allows fine grained identification
22:40:11 <nigel> jer: which is why we bring this up, it's been an active area of attack
22:40:12 <nigel> q?
22:40:18 <jcraig> ack me
22:40:19 <nigel> ack mjwilson
22:40:31 <nigel> Topic: Spoken Punctuation Parameter
22:40:59 <nigel> Evan_Liu: boolean attribute - if true, uses the punctuation, if false spells out e.g. "comma"
22:41:07 <ningxin> q+
22:41:12 <nigel> .. If you're using it for captioning you might want to spell it out
22:41:27 <nigel> dom: i18n questions about what constitutes punctuation
22:41:39 <nigel> .. likely to regret a boolean, there will be other choices
22:41:51 <nigel> Evan_Liu: Could start with an enum to make it more extensible
22:42:02 <nigel> jcraig: Agree, verbosity of screen readers is use case dependent
22:42:23 <nigel> Nigel: Does this control unspoken punctuation?
22:42:36 <nigel> Evan_Liu: No, Google supports it but we haven't had a request for that yet
22:42:40 <nigel> tidoust: Could be an array
22:42:58 <nigel> ack ningxin
22:43:16 <nigel> ningxin: For mass education, want to speak a formula, and have that appear.
22:43:36 <nigel> .. Send it to the cloud. Previously used Web Speech API with a different backend, but got a different answer
22:43:43 <jcraig> s/HN:/ningxin:/g
22:43:52 <mjwilson> q+
22:43:54 <nigel> .. Would be helpful to do mathematical representation output
22:43:59 <nigel> ack mjwilson
22:44:15 <nigel> mjwilson: Last year MathML WG had presentation on spoken math, very interesting and deep topic
22:44:24 <nigel> Topic: Remove SpeechGrammar
22:44:34 <nigel> Evan_Liu: Not implemented, or well defined, requests to remove it.
22:44:43 <jcraig> that would have been Neil Soiffer discussing spoken MathML and the MathJax project
22:44:51 <nigel> .. Seems to be consensus to remove, not controversial.
22:44:58 <nigel> .. Intended to do biasing support
22:45:01 <nigel> Topic: Biasing support
22:45:16 <nigel> Evan_Liu: Add bias to certain phrases, depends on recognition support.
22:45:30 <nigel> .. Chrome's recognition supports this. It's pretty generic, not tied to specific use cases.
22:45:53 <nigel> dom: Guess this would be super useful, would want to include language info and substream tagging
22:46:08 <nigel> .. If the phrase is used in several languages, you need to tag that, so need a different structure
22:46:19 <nigel> Evan_Liu: Each recogniser only works for a single language
22:46:39 <nigel> dom: Does the API need to support multiple language audio? Need to surface in the API.
22:46:51 <nigel> Evan_Liu: Multi-lang recognition in the same phrase not supported
22:47:10 <nigel> dom: If I put my French name into an English phrase, the pronunciation would change
22:47:10 <jcraig> q+ to mention musical artists like Ke$ha... technical or domain that don't align well terms (not a perfect example but LaTeX is "la-tek" not "latex")
22:47:14 <nigel> q+ to mention pages
22:47:24 <nigel> dom: If I put my Chinese name, that wouldn't work either.
22:47:33 <nigel> .. Fair, needs to be clear what language the stream is in
22:47:44 <nigel> .. Don't know how you take it back
22:48:10 <nigel> jcraig: Blind colleague laughed because Ke$ha was pronounced "Key dollar har"
22:48:24 <nigel> .. Also other words get pronounced differently by domain experts.
22:48:28 <nigel> .. Need phonetic spelling
22:48:31 <nigel> ack jcraig
22:48:31 <Zakim> jcraig, you wanted to mention musical artists like Ke$ha... technical or domain that don't align well terms (not a perfect example but LaTeX is "la-tek" not "latex")
22:49:03 <jcraig> s/Need phonetic spelling/Need phonetic hinting, perhaps IPA "International Phonetic Alphabet"/
22:50:39 <nigel> ack n
22:50:39 <Zakim> nigel, you wanted to mention pages
22:50:42 <mjwilson> mjwilson has left #web-speech
22:50:58 <nigel> Nigel: was the case that "pagers" got misrecognised as "pages" recently in the news.
22:51:36 <nigel> .. So what jcraig said: need to know the phonetic details
22:51:56 <nigel> .. Other point: need to know how you would layer in speaker recognition, or changes of speaker,
22:51:59 <nigel> .. again for the captioning use case
22:52:10 <nigel> mjwilson: Also language recognition
22:52:19 <nigel> Topic: Meeting close
22:52:27 <nigel> Evan_Liu: We're 2 minutes over, let's close
22:52:31 <nigel> rrsagent, make minutes
22:52:33 <RRSAgent> I have made the request to generate https://www.w3.org/2024/09/25-web-speech-minutes.html nigel
22:54:43 <nigel> Present+ Evan_Liu, James_Craig, Jer_Noble, Michael_Wilson, Francois_Daoust, Dom, Ninxing, smaug, solis, Eric_Carlson
22:54:47 <nigel> Chair: Evan_Liu
22:54:48 <nigel> rrsagent, make minutes
22:54:50 <RRSAgent> I have made the request to generate https://www.w3.org/2024/09/25-web-speech-minutes.html nigel
23:08:56 <dom> dom has left #web-speech