Feature Wiki

Information about planned and released features

Tabs

Metadata interface for querying by referatories

Page Overview

[Hide]

1 Initial Problem

Currently, the OER Harvester collects files on an ILIAS installation that are licensed in such a way that they can be used and shared as OER. 

However, there is no mechanism implemented yet by which these OERs can be retrieved by some referatory. But a truly open educational resource should automatically be made available to the outside world.

An ILIAS installation should be able to be queried by so called OER-referatories.

  • In OER-referatories, metadata of and links to learning material are offered (in contrast to OER-repositories to which learning material is uploaded).
  • An established, widely used protocol for this relationship between LMS and referatory is the OAI Protocol for Metadata Harvesting (OAI-PMH), established and maintained by the Open Archives Initiative (OAI). Supplementary notes: The OAI protocol is e.g. compatible with the OER-Suchindex (OERSI) and edu-sharing, a widely used solution in German-speaking countries, especially in the OER environment. edu-sharing in recent versions offers an OAI service and an OAI import.

The protocol was previously implemented by the OERinForm plugin, from which this proposition borrows heavily.

A small refresher on the OER-Harvester 

OER Harvester uses a Cron Job to harvest file objects with specific licenses, and places references to these objects into a selected category.

It can be configured via its Cron Job.

Currently, the OER harvester is limited to file objects, but there is a proposition to expand it, see OER-Harvester collects more object types.

One LOM-Shortlist 

There are initiatives aiming at providing metadata for referatories on the basis of LOM. (Of course there are other shortlists / metadata specifications, too.) 

  • Their working group decided which sub-set of LOM data is necessary for repositories / referatories and prepared an XML-schema. 
  • Based on this metadata specification, the search can also be extended to other repositories ("federated search").

Searching Metadata

The project OER-Suchindex (OERSI) aims to facilitate the overarching search for OER content via a central index harvesting data from OER Repositories and LMS installations. The following diagram is intended to demonstrate this:

The goal is now to provide a local index directly via ILIAS, which can be queried by relevant central services. Special attention is to be given to simplifying the handling as much as possible in order to achieve the desired acceptance.

https://www.oer-repo-ag.de/uebergreifende_suche/

Harvesting Protocol

The widely used Open Archives Initiative (OAI) prepared the OAI Protocol for Metadata Harvesting (OAI-PMH) for collecting and processing metadata. OAI uses Dublin Core as a precursor and simpler variant of Learning Object Metadata (LOM). 

Due to the widespread use, the focus in the following will be on OAI and thus implicitly on the use of Dublin Core.

2 Conceptual Summary

External querying has to be enabled in the Administration. When it is activated, the OER Harvester Cronjob not only harvests OER on the ILIAS installation, but also compiles them as a list of records.

ILIAS will provide an OAI 2.0 compliant endpoint under {ILIAS base URL}/oai, via which these records can be queried.

The location where the OER harvester deposits the collected objects in ILIAS can be separated from the location that the endpoint operates on. This allows for a manual check of the harvested ressources, before they are made available publically. If such a check is not required, the system can also be configured in such a way that OER are pushed out via the endpoint without any human intervention.

The records are XML-encoded metadata in the format Simple Dublin Core, translated by the harvester from the LOM native to ILIAS. This includes most importantly permalinks to the harvested ILIAS objects, and a permalink to the category where published OER are deposited. For more details, see Technical Information.

3 User Interface Modifications

3.1 List of Affected Views

  • Administration > Search And Find > Metadata > Copyright
  • Administration > System Settings and Maintenance > General Settings > Cron Jobs
  • Administration > General Settings > OER Harvester > Edit

3.2 User Interface Details

In the Copyright Administration (Administration > Search And Find > Metadata > Copyright), new options are added to configure if and how the installation provides OER and their metadata externally. These options consist of:

  • A checkbox 'Allow External Querying' with info 'If enabled, harvested OER can be queried by external interested parties, e.g. OER referatories.'

As sub-inputs of that checkbox:

  • A required text field 'Repository Name' with info 'This is returned as the name of this ILIAS installation when queried.' Default is the Installation Name.
  • A required text field 'OAI Prefix' with info 'This prefix is used as a namespace in the identifiers of returned OER records.' Default is 'oai:{Client ID}_{Installation ID}'.
  • A required text field 'Contact E-Mail'. Default is the 'E-Mail' from Contact Information in the General Settings.

Further, the tab should be renamed to 'Copyright & OER'.

In the 'Edit' view of the OER Harvester (Administration > General Settings > OER Harvester > Edit), in addition to the Category where OER content is collected, a second Category can be chosen with a repository picker. This is then the Category which the OAI 2.0 compliant endpoint operates on.

Additionally, the wording for the already existing repository picker input field is adjusted.

Moving Forms to KS

Both the 'Copyright Settings' and the 'Edit: "OER Harvester"' forms shown above have dependencies before they can be moved to KS.

In 'Copyright Settings', there is a link to the Cron Job Administration. In the course of the LUI project, a strategy will be developed to translate these links to KS. When this is worked out, it will be applied here.

To be able to move 'Edit: "OER Harvester"' to the KS, the classes from Services/Cron responsible for the generic Cron Job settings need to be moved first (which in turn depends on every other Cron Job). This would be a project in its own right, and is out of scope for this feature.

Further, repository pickers need to be implemented for KS forms. This is currently in active developement, see here.

3.3 New User Interface Concepts

New User Interface Concepts are not provided.

3.4 Accessibility Implications

No foreseeable implications for accessibility.

4 Technical Information

In contrast to the OERinForm plugin, no Java/Tomcat-based jOAI server is to be required.

The OAI endpoint provided by ILIAS can partially be configured in the ILIAS Administration (repositoryName, identifier prefix and adminEmail from Copyright Administration), and is partially fixed. ILIAS will not support deleted records, meaning deleted ILIAS objects can not be queried. Further, dates are never presented with a finer granularity than YYYY-MM-DD, and no optional information like compression or an additional description will be provided.

Records will always be returned as a non-hierarchical list, sets will not be supported. The last date of change of a ressource will always be set to the date of the last harvest, as changes to LOM metadata are not tracked in ILIAS. The identifier of a record is given by the configurable prefix, followed by ':il__{object type}_{obj id}'. Records will never include an 'about' element. As the metadata format of a record, Simple Dublin Core (DC) will be used exclusively , without any qualifiers or extensions. Its elements will mostly be filled from the LOM metadata of the corresponding ILIAS object, according to the following specification, adapted from the OERinForm plugin:

DC Element

From LOM Element(s)

No. of Occurences

Additional Information

title

general: title: string

1

with general: title: language as xml:lang attribute

creator

entity of lifecycle: contribute where role: value is 'author'

any

order of authors will be respected

subject

general: keyword: string

any

with corresponding general: keyword: language as xml:lang attribute

taxonPath: taxon of classification where purpose: value is 'discipline'

any

one element per taxonPath, individual taxons appear colon-separated

description

general: description: string

any

with corresponding general: description: language as xml:lang attribute

publisher

entity of lifecycle: contribute where role: value is 'publisher'

any

order of publishers will be respected

contributor

entity of lifecycle: contribute where role: value is anything but 'publisher' or 'author'

any

order of contributors will be respected

date

first lifecycle: contribute: date

0 or 1

will be in YYYY-MM-DD format, the date of the first contribute where role: value is 'author' is preferred, then 'publisher', then any contribute

type

educational: learningResourceType: value

any

format

technical: format

any

identifier

entry of first general: identifier where catalog is 'ILIAS'

1

will be translated to a permalink to the corresponding ILIAS object

source

-

0 or 1

will be filled not from LOM but with the permalink to the category where published OER are deposited

language

general: language

any

relation

relation

any

will be assembled from the kind: value, identifier: catalog, and identifier: entry as '{kind} {catalog}: {entry}'. If the catalog is ILIAS and the entry can be translated to a permalink, then that link will be inserted instead of catalog and entry.

coverage

general: coverage: string

any

with corresponding general: coverage: language as xml:lang attribute

rights

rights: description: string

1

will be translated from an ILIAS internal identifier to a human-readable licence according to the settings in the Copyright Administration

This endpoint will be implemented in ILIAS similar to that of the calendar, or more recently WOPI: a file 'oai.php' (or alternatively 'oai/index.php') will be added to the 'public' folder, which will initialize ILIAS (the context REST should be sufficient) and handles requests according to the OAI protocol based on the DC metadata sets previously compiled by the OER Harvester. No external dependencies are required for this implementation.

5 Privacy

The only personal information that can appear in the queryable metadata are the names of contributors (creator, publisher, contributor), derived from LOM's lifeCycle > contribute > entity. These are only automatically filled by ILIAS for Tests and Surveys, which are for the time being not harvested as OER.

6 Security

The endpoint is read-only, so we don't foresee any unusual security implications. It returns previously assembled xml responses, so the server load of individual requests is minimal.

7 Contact

8 Funding

If you are interest in funding this feature, please add your name and institution to this list.

  • ...

9 Discussion

Kunkel, Matthias [mkunkel], July 19, 2017: I suggest not to add another checkbox to the metadata screen ("Enable metadata for users who don’t use the ILIAS installation?") but to make the harvesting dependent from the chosen copyright setting (here CC licence). Reason: if someone decides to set her or his resources under a CC licence, she or he accepts that these resources are accessible to others, too. Why should the metadata of such objects not harvested and offered in internal or external repositories or referatories? We need to make content sharing as easy as possible!

Kunkel, Matthias [mkunkel], 06 JUL 2023 : Thanks for picking up this suggestion and improve the feature request. It is an important step for a better OER support in ILIAS. Unfortunately, I still have some problems to understand the request completely:

  1. Does this request only tackle the interface in ILIAS? Or will it come with extensions of the objects 'Glossary', Content Page', 'SCORM...' and 'ILIAS Learning Module', too? The latter is adressed in the last sentence of chapter 2 but not picked up again in chap. 3 and later. Clarifying this would be highly appreciated. BTW: there is already a Feature Request Allow OER Harvester to collect export files that would perfectly fit to this requirement.
  2. Assumed we only get the interface with this feature request, does this mean that the records mentioned in chap 2 will only be made for harvested file objects? Or for other object types, too (and if so, how? As long as the OER Harvester can only handle file objects?)
  3. Is there already a plan how the 'OAI 2.0 compliant endpoint' in ILIAS will be implemented? Maybe I am wrong, but I assume that this endpoint is a precondition for the current feature request. It would be helpful to have an idea about the implementation of this endpoint before finally decide upon this feature request – even if it is not implemented yet.
  4. It's nice that no jOAI server is needed for implementing this request. Will any other dependencies be added to the core with this request?
  5. And last but not least: instead of adding more settings to the current tab 'Copyright', please move the settings for 'Enable Copyright Selection', 'Cron Jobs' and the new 'Allow External Querying' to the 'Settings' tab and make it a new section there. The current tab 'Copyright' should only consist of the table for the 'Available Copyrights' and renamed appropriately. At the moment, the screen consists of a form and a table – which is not conformant to our UI guidelines.

Schmitz, Tim [tschmitz], 07 JUL 2023 : Thank you for the feedback.

  1. This request tackles both the interface changes, and the extension of the harvester. We also want to establish whether ILIAS should offer an OAI compliant endpoint to begin with, or if going in a different direction with respect to OER would be preferred.
  2. All harvested objects would also be made available via the endpoint.
  3. We have some ideas for how to implement the endpoint (translate the OAI protocol to an OpenAPI specification, offer the endpoint as a REST webservice), but as mentioned above we first want to check whether there is a consensus for such an endpoint in general before going into detail.
  4. Currently we see no need to add any dependencies to the core with this request.
  5. We have already made various KS improvements to the Copyright Administration in the trunk, independently of this request. Similar to what you propose, we have split up the 'Copyright' tab into two subtabs, one for the form shown in this request, and one for the table.

I hope that answers all questions. I have modified the article accordingly.

JourFixe, ILIAS [jourfixe], 24 JUL 2023: We highly appreciate this suggestion and schedule the feature for ILIAS 9. Please get into contact with Jephte Abijuru as maintainer of the webservice component to discuss and decide upon the planned Rest webservice.

Board, Technical [technicalboard], 27 JUL 2023: Thank you for this suggestion, which we also support from the point of view of the Technical Board and would be pleased if this extension were to find its way into ILIAS. From a technical point of view, however, we still have many questions, e.g. the connection to the component "Webservices", for which the maintainership has recently changed. We therefore ask you to expand and concretise the feature request in the following points and please also discuss these with the maintainer of webservices:
- Technical implementation: How exactly is this - from our point of view - new web service implemented and how does it relate to existing web services, for example?
- Which aspects have to be taken into account in terms of security? The introduction of a new endpoint/web service almost always introduces new possible attack vectors.
- Privacy: Please update the feature request to the current template and also describe the Privacy section accordingly.
Since there is a lot to be discussed, defined and documented, we expect that this request will be discussed on the JF again.

Schmitz, Tim [tschmitz], 06 DEC 2023: I updated the article according to the comment above, I hope all open questions are now answered. The changes are as follows:
- I added sections for 'Privacy' and 'Security' - no personal data is automatically offered in the endpoint, and since the endpoint is read only there shouldn't be any unusual security implications.
- We plan to implement the endpoint similar to how the Calendar handles its endpoint: we'll place a oai.php file in public, which handles all queries to it 'by hand'. No external dependencies will be used to build the endpoint. We discussed this with the new maintainer of Webservices.
- Additionally, we expanded the feature somewhat, and plan to offer separate categories for the collection and distribution of OER. This will allow institutions to insert manual control steps into the process of publishing OER, if required.

Schmitz, Tim [tschmitz], 15 DEC 2023: I removed the proposal to extend the OER Harvester to more object types, since this is covered in a separate article, see OER-Harvester collects more object types.

JourFixe, ILIAS [jourfixe], 08 JAN 2024: We highly appreciate this suggestion and re-schedule the feature for ILIAS 10.

10 Implementation

Implemented largely as described above, for more details see the documentation.

As opposed to the original description, the sub-fields under the 'Allow Querying via OAI-PMH Interface' checkbox are not pre-filled, as we want admins configuring the OAI-PMH interface to think about what they set here.

Further, 'source' in the Simple DC metadata of objects exposed through the interface is not filled with the permalink of the public category on the installation, but rather from the object's LOM via resource > identifier > entry of relation where kind > value is 'isbasedon'. This is more in line with other similar implementations of mapping between LOM and Simple DC.

Administration > Search And Find > Metadata > Copyright & OER
Output of an 'Identify' request to the OAI PMH interface
Configuration of the OER-Harvester cron job

ADDENDUM, 16 OCT 2024: To improve compatibility with some consumers of this interface (e.g. edu-sharing), a trivial set named 'default' was implemented, which always contains all records.

Test Cases

Test cases completed at 19.08.2024 by Ahmed Elagamy

C76493 Allow Querying via OAI-PMH Interface

C76494 Activate and Excecute OER Harvester Cron Job

C76499 Delete Files from Publication Category

C76500 Ignored Objects are Not Harvested

C76501 Ensure Only Published Files are Listed in the Interface 

C76495 LOM Elements in Records

C76496 Handling of Multiple Licenses 

C76497 Metadata Export via OAI-PMH Endpoint 

C76498 Category for the Published OER Content 

C76505 Harvesting = Publishing category

Privacy

no change required

Approval

Approved at 2024-09-09 by Brauns, Johanna [jbrauns].

Last edited: 16. Oct 2024, 13:05, Schmitz, Tim [tschmitz]