16:51:51 <RRSAgent> RRSAgent has joined #publishingcg
16:51:55 <RRSAgent> logging to https://www.w3.org/2025/03/20-publishingcg-irc
16:52:13 <wolfgang> Zakim, start meeting
16:52:13 <Zakim> RRSAgent, make logs Public
16:52:15 <Zakim> Meeting: Publishing Community Group
16:52:34 <wolfgang> rrsagent, start
16:52:51 <wolfgang> Meeting: W3C Publishing Community Group Plenary (20-03-2025)
16:53:05 <wolfgang> Chair: wolfgang
16:53:22 <wolfgang> Agenda: Nick Brown (VP Product Vitalsource): From Principles to Practice - Responsible AI for Enhanced Student Engagement in Reading Systems
16:54:54 <wolfgang> present+
16:58:26 <gautierchomel> gautierchomel has joined #publishingcg
17:00:59 <liisamk> liisamk has joined #publishingcg
17:01:11 <liisamk> present+
17:01:25 <gautierchomel> present+
17:02:10 <rickj> rickj has joined #publishingcg
17:02:23 <rickj> present+
17:03:02 <rickj> scribe+
17:05:39 <rickj> wolfgang: welcome to all
17:06:19 <rickj> Nick: From Principles to Practice overview
17:06:37 <rickj> ... VitalSource overview... talking about 'the engagement challenge'
17:08:09 <rickj> ... who are we?  28 m units delivered last year, 19+m users served, users around the world world, localized in 37 languages
17:08:41 <rickj> ... we act as a 'learning delivery network' sitting between learning providers and institutions/students/bookstores
17:09:39 <rickj> ...How do we do this?  1. Driving day 1 affordable access for millions.  2. Helping students stay engaged on Day 2 and beyond
17:10:37 <rickj> ...https://research.vitalsource.com/research researching what works
17:10:59 <rickj> ... more than 30 published papers about learning science
17:11:10 <rickj> ... engagement is step 1
17:12:52 <rickj> https://dl.acm.org/doi/10.1145/3576050.3576086 referenced
17:13:21 <rickj> ... shows students not reading
17:14:17 <rickj> 14% average number of assigned textbook pages read by students
17:14:50 <rickj> ... one solution: VitalSource CoachMe.
17:15:16 <rickj> feature inside Bookshelf reader, adds AI generated low-stakes formative practice questions inside the textbook reading experience
17:15:45 <rickj> generated from the textbook, not a general LLM model
17:16:10 <rickj> instructors can assign engagement with these questions as a part of their grade
17:16:54 <rickj> uses VitalSource proprietary AI for automatic question generation.  Based on the Doer Effect from Carnegie Mellon OLI work
17:17:10 <rickj> 'practice while reading causes 6x more learning gain than reading alone'
17:18:01 <rickj> bring active learning to a 'passive medium'
17:18:54 <rickj> (deeper dive into how it works)
17:21:14 <rickj> timeline of deployment and achievement
17:22:00 <rickj> 21 million AI generated questions answered in a learning context
17:22:09 <rickj> published 35 peer reviewed papers
17:22:17 <rickj> 3 best paper awards
17:24:58 <rickj> examination at Iowa State University course with A/B test with the same book, looking at how many times students opened their book showing dramatic improvement in engagement when CoachMe questions are assigned as a part of the grade
17:26:48 <rickj> multiple school study showing increase in engagement by students
17:28:34 <rickj> ... 'but what about Gen AI?'
17:29:25 <rickj> https://www.aacu.org/research/leading-through-disruption discussed
17:30:01 <rickj> explosion of GenAI usage
17:30:21 <rickj> Key opportunities: things we can do for students and faculty that could not be done before
17:31:25 <rickj> Pro's and Con's of using AI
17:33:07 <rickj> The importance of responsible AI use:  https://get.vitalsource.com/ai-principles
17:34:36 <rickj> referenced 1EdTech rubric https://www.1edtech.org/standards/ai-rubric
17:34:48 <rickj> Why do these principles matter?
17:35:24 <rickj> avoid "AI for AI's sake', focus on real learning gains not hype, maintain strong publisher and institutional partnerships
17:37:41 <rickj> where we are heading next...
17:38:30 <rickj> High quality AI answers aligned with textbook content, no model training, no IP leakage, SOC2 compliant use with LLMs, DRM protected
17:41:54 <rickj> Q&A
17:42:14 <gautierchomel> q+
17:42:48 <wolfgang> ack gautier
17:43:46 <rickj> gautierchomel: when content/questions are generated, we need an evaluation methodology and a way to advise the user this is AI generated.  Have you been thinking about this?  Is it possible?
17:44:48 <rickj> uptownnickbrown: that concern drove a lot of our development and decisions.  'generation' is a misnomer, as nothing is 'generated'.  There is zero risk, as the sentence for the 'fill in the blank' is from the book.
17:45:22 <rickj> ... there are feedback mechanisms inside the book to give positive/negative feedback on the questions
17:45:52 <rickj> ... recommend: have clear disclaimers that this is AI generated content.  It's the right ethical thing to do.
17:46:14 <rickj> ... measuring quality is also critical, and hard.
17:46:51 <Michalis0> Michalis0 has joined #publishingcg
17:46:55 <rickj> ... automated judges that use LLMs to judge LLM responses, trained to do different tasks in different ways.  Judge a few different things at the same time (factually accurate, ...)
17:47:18 <wolfgang> q?
17:47:18 <rickj> ... reduce hallucinations by having the concrete source material
17:47:48 <rickj> ... comfortable with a 'the book does not address that question' response
17:48:00 <rickj> ... also trying to evaluate the underlying pedagogy behind the model
17:48:24 <rickj> ... 'can answer that... best I can do is recommend you read page 62 of the book' type of answers
17:48:40 <liisamk> q+
17:49:10 <wolfgang> ack liisamk
17:49:10 <rickj> liisamk: have you thought about using AI for 'other things' with your reading system?
17:49:42 <rickj> uptownnickbrown: a few ways to tie this into the reading system.  Things like making good flashcards (hard to do now).
17:50:08 <rickj> ... also looking at ways to evolve search beyond a 'find' function
17:50:34 <rickj> ... search with mixed languages...
17:50:52 <rickj> ... alt-text for screen readers
17:51:22 <rickj> ... lots of places to pervade the reading system
17:51:26 <Michalis0> q+
17:51:37 <wolfgang> ack Michalis
17:52:02 <rickj> Michalis0: are questions created in real time, or pre-loaded?
17:52:14 <rickj> uptownnickbrown: yes, the CoachMe questions are pre-generated
17:52:37 <rickj> ... also aligned with where to ask the question in the book flow
17:52:53 <rickj> ... may evolve over time as AI improvements come
17:53:07 <rickj> ... over 1,000,000 questions in production at scale
17:53:23 <wolfgang> q+
17:53:56 <rickj> wolfgang: am I right that you mainly use NLP and not LLMs?
17:54:34 <rickj> uptownnickbrown: Yes.  That's exactly how we built CoachMe
17:54:54 <wolfgang> q?
17:55:53 <wolfgang> ack wolfgang
17:57:03 <rickj> wolfgang: did you take into account the AI legislation from the EU?
17:57:33 <rickj> uptownnickbrown: we are starting to think more about that now, as it was not in place at the time.
17:58:06 <wolfgang> q?
18:00:10 <rickj> rssagent, generate minutes
18:01:00 <rickj> rrsagent, generate minutes
18:01:01 <RRSAgent> I have made the request to generate https://www.w3.org/2025/03/20-publishingcg-minutes.html rickj
18:01:09 <gautierchomel> rrsagent, please make the minutes
18:01:10 <RRSAgent> I have made the request to generate https://www.w3.org/2025/03/20-publishingcg-minutes.html gautierchomel
18:01:19 <rickj> thank you gautierchomel!!
18:01:35 <gautierchomel> rickj, i guess your wording was the right one
18:01:43 <rickj> rss/rrs
18:02:06 <gautierchomel> ah!
19:09:29 <gautierchomel_> gautierchomel_ has joined #publishingcg
19:42:08 <gautierchomel> gautierchomel has joined #publishingcg
20:18:14 <Zakim> Zakim has left #publishingcg