Text and Data Mining Reservation Protocol Community Group
The goal of this Group is to facilitate Text and Data Mining (TDM) Reservation Protocol in Europe and elsewhere, by specifying a simple and practical machine-readable solution, capable of expressing the reservation of TDM rights - following the rules set by the new European DSM Directive / Art.4 - and the availability of machine-readable licenses for TDM actors.
Group's public email, repo and wiki activity over time
Note: Community Groups are proposed and run by the community. Although W3C hosts these conversations, the groups do not necessarily represent the views of the W3C Membership or staff.
Giulia and I had a quick call, to discussed a practical path to the next steps of our work. Here is our proposal:
1/ Study if “the use of data for the purposes of training an AI/ML model” is the same thing as “the use of data for the purposes of data mining”, especially in the scope of the EU DSM Directive. For that we need access to document formalizing the position of experts in this field, we need lawyers around the table, and the opinion of EU Commission people would be welcome.
2/ Discuss if we should apply evolutions to the specification in order to embed TDMRep properties in different types of resources (images, PDF and EPUB come to minds). One important question is what should be found in such resources: TDMRep properties (as we defined them) or detailed usage rights information (as the IPTC is defining them)?
3/ If the answer to issue 2 is “yes we should embed TDMRep properties in resources”, then how should we embed them? (XMP comes to minds there).
Could we start with issue 1? AI and LLMs are the big thing currently, and there is traction for protecting artistic content against free commercial use in AI products.
We therefore propose to held a Zoom call on this topic on April 4th, 14 UTC (16:00 CEST), using this URL: https://us06web.zoom.us/j/84639131689?pwd=QXl1T1R1UldjdG1XR00rWG5ZQUlZQT09
Please indicate if you’re willing to join, and if you have a specific expertise in the AI vs TDM field.
After the release of our final report, in February 2022, we had the opportunity to present the project to several publishers and publishers associations in European countries and start discussing possible pilots in order to gather feedback on the first release of the protocol.
We found that the interest for the “TDM opt-out” solution is growing fast, in particular since ChatGPT was released. GPT-3, the LLM (Large Language Model) behind ChatGPT, constitutes an impressive example of the potentiality of AI agents when processing digital publications, and rightsholders from different sectors of the content industries are looking for an effective machine readable solution for managing TDM rights on their content.
The TDMRep effort is especially interesting for the IPTC (International Press Telecommunications Council), which develops standards for the News Industry and has started exchanging ideas with us.
This group will therefore continue its work on refining the specification and study possible extensions to meet the requirements of specific sectors of the content industry and to facilitate the integration of our work with other metadata standards (such as the IPTC standard for photos) and content formats (e.g. to embed TDMRep metadata in PDF resources).
As multiple EU countries are moving towards the implementation of the DSM directive, publishers are looking for solutions. The Community Group will now focus on communication, its targets being the European Commission, TDM Actors and publishers.
In order to communicate effectively, the Community Group will have to work on its governance and define some tools (maybe a presentation website, recorded demos, a brochure, a logo …). We’ll therefore organize multiple calls in 2022.
If you are aware of events which could be good supports for presentations, please contact the CG chairs. We’re currently thinking about the IPTC Spring Meeting 2022 (May) and the Digital Publishing Summit 2022 (June).
Jean-Baptiste de Vathaire (Cairn.info) described the integration of TDM properties on the Cairn website, using both http headers and html metadata (both for the sake of precaution).
Robin Le Marois (Seraphin.legal) showed the results of the TDMRep compliant scrapping of the Cairn.info website by a scrapper they developed.
The consensus is that the specification corresponds to the requirements and is easy to implement, which corresponds to its main goals.
One remaining issue is the de-duplication of requests for licenses from TDM Actors. If many resources a linked to the same policy, on which ground should a TDM Actor send one request, or multiple requests? Not every Publisher will set a `target` property in its TDM Policy. It seems to the participant that the URL of the TDM Policy should be used as an identifier and therefore a way to send a generic email stating a “request for license to mine a set of resources pointing at the TDM Policy with URL xxxx”, optionally with the list of resource URLs the TDM Actors is willing to mine. This need to be clarified in a best practices guide.
Progress:
The CG will stay open for feedbacks about the specification until the end of the year 2021, will update it when needed, and will then release the specification as Final Report.
Communication:
The co-chairs and the FEP (Federation of European Publishers) will turn to the European Commission for planning demos or some other representation of the work done so far, and by this raise awareness about the project.
The CCC (Copyright Clearance Center) reports that WIPO is holding webinars every 2 weeks on copyright infrastructure and ne developments. It is a good avenue for promoting our work.
We hope that Press Publishers will size the project, as they are among the organizations which will benefit most from the success of the solution.
The group feels that we should create a landing page introducing the project (and therefore certainly acquire a specific domain name), from which multimedia communication can be developed, like an overview of the project and How To for the 3 techniques we propose.
An incentive for TDM Actors could be to create a label (“Clean scrapping”?) and a logo, for those who adopt the solution.
We’ll have to discuss the governance of the project after the Final Report of the CG is released. Should it be ex-nihilo, or based on an existing organization?
The proposed agenda for the TDM Reservation Protocol Community Group meeting on Thursday 26 October 2021 (15:00 UTC to 16:00 UTC) happening as part of W3C TPAC 2021, is as follows:
Introduction to the TDM Reservation Protocol CG
Progress update
Finalization of prototypes in view of presentations to interested parties
Wider communication plan: how can we make sure that TDM Actors are aware of this initiative?
Q&A
The co-chairs invite community group members to propose further agenda items that they would like to be added to the meeting by sending their suggestions via the public-tdmrep CG mailing list.
The meeting will take place via Zoom and the link for the meeting will soon be published on the TPAC Wiki.
If you are planning to attend the CG meeting, you are invited to register for TPAC 2021 so that you can attend other sessions in the conference if you are interested to do so. To register for TPAC 2021, click here.
During its early meetings, the group worked on what TDM means in practice, the vocabulary to be used during the project, the goals and requirements for a technical solution, created several use cases and compiled a set of past and existing initiatives with a similar scope. The group then agreed on three alternative technical solutions for expressing the reservation of TDM rights: one based on http headers, another based on a file hosted on the origin server, and a third based on html meta tags. These three solutions correspond to different situations and technical skills.
In May, the group defined a machine readable TDM policy which details how a rightsholder can be contacted and conditions in which a TDM license can be acquired. It was agreed that TDM Policies would be defined as a profile of ODRL 2. In June, a complete draft of specification (https://w3c.github.io/tdm-reservation-protocol/spec/) was written, which includes details like the priority by which the 3 techniques must be processed by TDM Agents and how TDM Agents should react to protocol errors.
From July to August, details of the specification were discussed and the group requested advice and prototyping from content providers and TDM Actors.
#issue 23 and #20: these issues are strongly related to possible protocol errors and how TDM Agents should react to such errors (made by content providers). As a general mechanism, in case protocol errors are detected by TDM Agents, the fallback is to consider that tdm-reservation is unset. tdm-reservation=2 with no tdm-policy property is a kind of protocol error, but applying the general rule in this case was controversial.
The decision of the group is to suppress this value 2, which suppresses the controverse and simplifies the spec at the same time.
a TDM Agent which sees tdm-reservation = 1 and NO tdm-policy property will assume that there is NO way to get rights to use the content
a TDM Agent which sees tdm-reservation = 1 and a tdm-policy property set will assume that there is a way to get rights to use the content and will decide if it tries to get more information or stop there.
The issues will be closed as soon as the modification of the specification is approved.
During its early meetings, the group worked on what TDM means in practice, the vocabulary to be used during the project, the goals and requirements for a technical solution, created several use cases and compiled a set of past and existing initiatives with a similar scope.
The group then agreed on three alternative technical solutions for expressing the reservation of TDM rights: one based on http headers, another based on a file hosted on the origin server, and a third based on html meta tags. These three solutions correspond to different situations and technical skills.
In May, the group defined a machine readable TDM policy which details how a rightsholder can be contacted and conditions in which a TDM license can be acquired. It was agreed that TDM Policies would be defined as a profile of ODRL 2.
In June, a complete draft of specification (https://w3c.github.io/tdm-reservation-protocol/spec/) was written, which includes details like the priority by which the 3 techniques must be processed by TDM Agents.
Participants: Giulia, Robin, Claudio, Laurent
Agenda: discuss specification details
Issue #18: the order in which TDM Agents must check the 3 possible techniques have been integrated in the spec. The participants to the call are confortable with the proposal, but it is important that other participants check this in details.
Issue #20 is about tdm property values in case of protocol errors. The draft specification considers that if there is a protocol error, tdm-reservation is considered unset. In this case the TDM Agent will act as if the rightsholder had not provided any information. This is especially true if tdm-reservation = 2 but tdm-profile is not set.
tdm-reservation = 2 indicates that “tdm rights are reserved and a policy is available”. If we consider that a missing (or unreachable) tdm-profile is not a protocol error, then it would be simpler to suppress this value 2 and keep only value 1 with an optional tdm-profile. In such a case, when the policy is not set or unreachable, the TDM Actor must simply not process the content during this round of scraping. Issue #23 was open to discuss this aspect.
There will be no call in August, therefore please use Github issues to discuss the draft specification, which will be updated at least once during August.
The next call of the group is planned on September 7th, at the usual time (3 pm UTC) , via the usual Zoom (ask details to the chairs if needed).
A complete version of the draft specification is now available athttps://w3c.github.io/tdm-reservation-protocol/spec/ Please review it and open Github issues or comment in existing ones as you see fit. If some of you have started developing prototypes, please check this version of the spec and advise if you miss important information. Proposal: having a call on Tuesday July 27th, at 3 pm UTC (= usual time), to discuss if the draft needs some enhancements for the completion of prototypes.
During its early meetings, the group worked on what TDM means in practice, the vocabulary to be used during the project, the goals and requirements for a technical solution, created several use cases and compiled a set of past and existing initiatives with a similar scope.
Then, the group agreed on three alternative technical solutions for expressing the reservation of TDM rights: one based on http headers, another based on a file hosted on the origin server, and a third based on html meta tags. These three solutions correspond to different situations and technical skills. The priority in which these techniques should be processed by TDM Agents is still to be decided.
During recent meetings, the group defined a machine readable TDM policy which details how a rightsholder can be contacted and conditions in which a TDM license can be acquired. It has been agreed by the group that TDM Policies will be defined as a profile of ODRL 2 and a detailed proposal has been written.
Issue #15, Should the vcard prefix be explicitly declared? It is closed, as it has been tested that re-defining the vcard prefix in Policy instances is not useful.
Issue #13, Which domain name for the ODRL profile used for TDM Policies? The W3C management accepts the use of the W3C domain name for identifying the properties defined in the Policy specification. We will now have to work on the details of which URL is chosen.
Issue #14, Should we merge the file on the origin server with the ODRL TDM Policy? The current comment do not allow to reach a consensus. During the dicussion it appears that the participants to the call are in favor to keep separate both structures, as it keeps a clean separation between the core deliverable of the group (related to the Article 4 of the DSM) and the optional & advanced feature (offering TDM Policies). The final choice will be done during the next meeting.
Issue #16, Property names and values of TDM-a and TDM-b The beauty contest is open, several ideas are exchanged during the call, we’ll make a choice during the next meeting.
Next steps:
The chairs will now start writing a detailed specification, using the W3C document style and tools (respect). The drafts will be discussed during the summer and we should be able to finalize it in October 2021. As soon as property names are chosen, interested parties can start prototyping, by setting TDM properties on one side, by developing test scripts on the other side. We hope being able to demo the result by October also.
In previous meetings, the group has worked on what TDM means in practice, the vocabulary to be used during the project, the goals and requirements for a technical solution, the creation of several use cases. They have also compiled a set of past and existing initiatives with a similar scope.
Then, the group discussed on how rightsholders can declare to TDM Agents the reservation of TDM Rights. Three different technical solutions were selected: one based on http headers, another based on a file hosted on the origin server, and a third based on html meta tags. These three solutions correspond to different situations and technical skills. The priority in which these techniques should be processed by TDM Agents is still to be decided.
Using one of these techniques, a rightsholder can express either that “TDM rights are not reserved”, “TDM rights are reserved” or “TDM rights are reserved but a TDM license can be acquired”.
In the latter case, it is interesting for both parties (rightsholder and TDM Actor) to create a machine readable TDM policy which details in which conditions a TDM license can be acquired.
The group is therefore now defining the model of such TDM policy.
Participants: Aziz Ndao, Tzviya, Leonard, Jean-Baptiste, Giulia, Laurent, Carlo Lavizzari, Mathilde
Agenda: discuss the draft of TDM Policy properties
The TDM Policy is currently drafted in a shared GDoc. No serialization format is chosen so far, but ODRL 2 is the best candidate so far.
The discussion focuses on some properties:
Content identifier.
STM is using DOI
We may decide that it takes the form of a URI, depending the serialization format.
But using this content id forbids using the same policy for N resources: this is a big issue.
Rightsholder name & contact info
We could add a rightsholder identifier.
Rights reservation (values are no, yes, yes + a TDM policy)
if a TDM agent has reached this policy, this is because the rights reservation value is “yes”. This value will always be “yes”, therefore its use if not obvious.
“where” property: this is initially meant to express “EU” vs “non-EU”.
The EU Directive already sets rules about TDM policies in the EU for research purpose, therefore no TDM policy defined by a rightsholder is applicable to “research” in the “EU”.
Participants are wondering if going to this level of details in the policy is useful.
Both “where” and “when” are at the same conceptual level: either both are present or both absent from the TDM policy model.
w3c/tdm-reservation-protocol
