Icona Wiki

Feature Wiki

Information about planned and released features

Schede

Project: ILIAS Resource Storage Service (IRSS)

This is a project page that bundles several feature wiki pages which belong to a larger development activity for the File component in ILIAS.

1 Aim of Project

«All user-uploaded files shall be stored centralized in one Service»
ILIAS uses in many components the possibility for users to upload their own files and to download or display them again at a later time. Some parts of this process are already centralized (FileUpload-Service, FileDelivery-Service). But not the central storage of files in the file system. Many components have implemented this part independently. Due to the different implementations, problems related to files occur again and again (see detailed description).

The "Resource Storage Service" project aims to have all files stored and managed centrally by a service. The components that use files from users in ILIAS should be able to use the service in a simple way.
The project includes various developments and changes in ILIAS. On the one hand the service is introduced as a central storage management in ILIAS. On the other hand, individual components that use files will have to be modified bit by bit.During these modifications files that have been stored for years have to be migrated to the Resource Storage Service.

Parallel to the migration of existing components, new features can be planned and implemented through centralisation.

The concrete objectives:
  • In ILIAS there is only one central place where uploaded files are stored
  • Existing components are converted so that they no longer have any dependencies on other objects (e.g. FileObject).
  • The implementation of features that include user uploads should become easier and cheaper.
 
 
 
Centralization and the Migration of Components to the Resource Storage Service is important because
  • Problems related to the storage of files can be solved centrally
  • Problems that occurred in the past, e.g. in connection with special characters in file names, are now a thing of the past
  • Features related to files will be easier to implement
  • Security is increased, security problems with files can be solved centrally
  • Whitelisting no longer needs to be done in the FileSystem but can be done when files are delivered
  • Storage options other than the web server hard disk will be possible in the future

1.1 Conceptual Summary

1.2 The ILIAS Resource Storage Service (IRSS)

The Resource Storage Service was introduced in a first version in ILIAS 6 where it was used to store icons for MainMenu entries. This allowed first experiences to be gathered and the service to be further expanded.
The principle of the service is simple. A file uploaded by a user is forwarded by a component (e.g. forum) to the storage service. In return, the components receive an identification (unique ID of the resource as a string). This ID is stored by the component. If the file is then needed again, the component with the ID can ask the Storage Service for a consumer (e.g. DownloadConsumer, InlineConsumer, ...) and execute it. In this way, the uploaded image is displayed, for example.

Resource Storage Service

Overview

The Resource Storage Service should be the central place where components and plugins should manage their user uploaded files and resources.

Historically ILIAS uses the file system directly in its components to manage files. There are some conventions but within the structure of a component files can be stored freely. Additional features such as versioning of files is always the responsibility of the component or the plugin.

ILIAS 7 offers the Resource Storage Service for all components and plug-ins.

In summary, the service allows to exchange a user upload (UploadResult from the Upload-Service) for a unique resource ID. During this exchange, the service takes care of the correct storage of the data (file as well as metadatan to the file). There is no direct access of the component to the FileSystem, neither by native PHP functions nor by the FileSystem service. As soon as the UserUpload is needed again, e.g. to display or download it, the Resource-ID can be exchanged for a corresponding delivery method. This process also does not directly access the file system by the component.

  Component                 +-+ Resource
                            | | Storage
  +----------+              | | Service
  | UPLOAD   +--------------> |
  +----------+              | |
                            | |
                            | |
  +----------+              | |
  | ID       <--------------+ |
  +----------+              | |
                            | |
                            | |
+---------------------------------+
                            | |
                            | |
  +----------+              | |
  | ID       +--------------> |
  +----------+              | |
                            | |
                            | |
  +----------+              | |
  | CONSUMER <--------------+ |
  +----------+              | |
                            | |
                            | |
                            +-+

Terms

+---------------------------------------------------------------------+
| RESOURCE                                                            |
+---------------------------------------------------------------------+
|                                                                     |
| +--------------------+                                              |
| | IDENTIFICATION     |                                              |
| +--------------------+                                              |
|                                                                     |
| +-------------------+  +-------------------+  +-------------------+ |
| | REVISION          |  | REVISION          |  | REVISION          | |
| +-------------------+  +-------------------+  +-------------------+ |
| |                   |  |                   |  |                   | |
| | +---------------+ |  | +---------------+ |  | +---------------+ | |
| | | INFORMATION   | |  | | INFORMATION   | |  | | INFORMATION   | | |
| | +---------------+ |  | +---------------+ |  | +---------------+ | |
| | | "FILE"        | |  | | "FILE"        | |  | | "FILE"        | | |
| | +---------------+ |  | +---------------+ |  | +---------------+ | |
| |                   |  |                   |  |                   | |
| +-------------------+  +-------------------+  +-------------------+ |
|                                                                     |
+---------------------------------------------------------------------+

Resource

A Resource is the abstract term for a "thing" uploaded by a user and passed to the Resource Storage Service for management. The Resource is the combination of

  • Identification
  • Revisions
  • Information about the revisions
  • File or its contents (of the respective revisions)
  • Stakeholders

Identification

The identification represents the unique information about which resource is involved. This information is stored by the component itself to be able to retrieve a resource later.

Revision

Revisions contain the file together with metadata (information) and a version number. This means that a resource can have multiple revisions. The management of these revisions is the responsibility of the Service. Revisions can be added, changed or deleted from outside by the components.

Information

Information holds general metadata about a revision, such as MimeType, size, title, ... . This information is stored by the service.

Consumer

Consumers allow to "consume" a resource. This is the interface closest to the FileSystem that is allowed to start an action with the file of a revision, such as

  • Download
  • Get Stream
  • Retrieve content
  • Get the absolute Path of the file (only for legagy purposes)
  • ...

A set of Comsumers is already part of the service.

Stakeholder

Although resources are uploaded to ILIAS by users, these users are not only responsible for a resource but also for the component in which the file has been uploaded. The service must be able to retrieve information from the component in order to decide whether a resource is needed at all or not, e.g. during cleanup processes. Or, if a resource is removed by the service for security reasons, the component must be informed about it, in order to carry out further cleanup steps.

StorageHandler (internal)

In order to be equipped for the future, the Storage Service internally uses so-called StorageHandlers to store files. In the first step this is of course a FileSystem-based storage handler. However, the way is open here to expand this at a later point in time.

Usage

The use of the service should be as easy as possible from the point of view of the components and Plugins. The components should only communicate with the service according to the principle shown above, i.e. exchange uploads for identifications or exchange an identification for a consumer.

Examples

In most cases most of the work is already done by the UI inputs and as a form developer you get the identification directly from the service which can then be stored in your component.

In case you want to exchange an upload for an identification yourself, proceed as follows in your component:

<?php
// ...
global $DIC;
$upload_result = $DIC['upload']->getResults()['my_uploaded_file'];
$stakeholder = new ilMyComponentResourceStakeholder();

$identification = $DIC['resource_storage']->manage()->upload($upload_result, $stakeholder);

// then store the $identification whereever I need it in my component


Suppose we already have a file stored in the Storage Service and would download it in our component. We had stored the identification, with which a corresponding consumer can now be obtained to download the file.

<?php
// ...
global $DIC;
$rid_string = $this->getMyResourceIDasString(); // we get the stored ID of the resource

$identification  = $DIC['resource_storage']->manage()->find($rid_string);
if (null !== $identification) {
    $DIC['resource_storage']->consume()->download($identification)->run();
} else {
    // there is no such resource in the storage service
}

Adding a new revision is as simple as that:

<?php
// ...
global $DIC;
$rid_string = $this->getMyResourceIDasString(); // we get the stored ID of the resource

$identification  = $DIC['resource_storage']->manage()->find($rid_string);
$upload_result = $DIC['upload']->getResults()['my_uploaded_file'];
$stakeholder = new ilMyComponentResourceStakeholder();

if (null !== $identification) {
    $DIC['resource_storage']->manage()->appendNewRevision($identification, $upload_result, $stakeholder);
} else {
    // there is no such resource in the storage service
}

Collections

In many cases a component does not only need a single resource to be stored, but wants to be able to use a collection of resources (e.g. in an exercise unit several files can be delivered per person). For this purpose Collections can be used in IRSS. A collection always contains a collection of ResourceIdentifications in a defined order. The management of collections is done via:

$DIC['resource_storage']->collection();

To use a collection, a new collection-id can be created:

$rcid = $DIC['resource_storage']->collection()->id();

Or an existing collection-id can be called:

$rcid = $DIC['resource_storage']->collection()->id("f0e564e2-5d48-4d33-a8e6-bdc2646900d7");

Retrieving the collection itself is done via:

// $rcid see example above
$collection = $DIC['resource_storage']->collection()->get($rcid);

ResourceIdentifications can now be added to such a collection, e.g. after the upload. The collection must be saved afterwards:

$irss = $DIC['resource_storage'];
$rcid = $irss->collection()->id("f0e564e2-5d48-4d33-a8e6-bdc2646900d7");
$collection = $irss->collection()->get($rcid);

$this->upload->process();
$result = array_values($this->upload->getResults())[0];
if ($result->isOK()) {
    $id = $irss->manage()->upload($result, $this->stakeholder);
    $collection->add($id); // adding a resource to the collection
}
$irss->collection()->store($collection); // saving the collection

Collections can be explicitly assigned to a user ID, and such collections can later be retrieved and modified only on the basis of this user ID:

// create new collection for user 6
$rcid = $DIC['resource_storage']->collection()->id(null, 6); // return a collection with e.g. ID "f0e564e2-5d48-4d33-a8e6-bdc2646900d7"
// ... accessing the same collection with another user-id results in an exception
$rcid = $DIC['resource_storage']->collection()->id("f0e564e2-5d48-4d33-a8e6-bdc2646900d7", 13);

To get the ResourceIdentifications assigned to a collection, they can be accessed as follows:

$rcid = $DIC['resource_storage']->collection()->id("f0e564e2-5d48-4d33-a8e6-bdc2646900d7");
$collection = $irss->collection()->get($rcid);

foreach ($collection->->getResourceIdentifications() as $rid) {
    // do something with the resource
    $file_stream = $DIC['resource_storage']->consume()->stream($rid)->getStream();
}

Collections can also be easily downloaded as a ZIP file.

$rcid = $DIC['resource_storage']->collection()->id("f0e564e2-5d48-4d33-a8e6-bdc2646900d7");
$DIC['resource_storage']->consume()->downloadCollection($rcid)->run();

Besides storing collections (store) there are also clone and remove.

Sorting and Ranges of Collections

A collection can be sorted for display, various options are available for this:

use ILIAS\ResourceStorage\Collection\Collections;
/** @var Collections $collection_services */
$collection_services = $DIC['resource_storage']->collection();

$rcid = $collection_services ->id("f0e564e2-5d48-4d33-a8e6-bdc2646900d7");
$collection = $irss->collection()->get($rcid);

// By Title
$collection_sorted_by_title_asc = $collection_services->sort($collection)->asc()->byTitle();
$collection_sorted_by_title_desc = $collection_services->sort($collection)->desc()->byTitle();

// By Creation Date
$collection_sorted_by_creation_date_asc = $collection_services->sort($collection)->asc()->byCreationDate();
$collection_sorted_by_creation_date_desc = $collection_services->sort($collection)->desc()->byCreationDate();

// By File-Size
$collection_sorted_by_filesize_asc = $collection_services->sort($collection)->asc()->bySize();
$collection_sorted_by_filesize_desc = $collection_services->sort($collection)->desc()->bySize();

These are in-memory sorts. But if you want to store the sort permanently, you can do this as follows:

$collection_services->sort($collection)->asc()->andSave()->bySize();

To display only a part of a collection, you can obtain a range of a collection. In this case you get the ResourceIdentification directly either as array or as \Generator:

use ILIAS\ResourceStorage\Collection\Collections;
/** @var Collections $collection_services */
$collection_services = $DIC['resource_storage']->collection();

$rcid = $collection_services ->id("f0e564e2-5d48-4d33-a8e6-bdc2646900d7");
$collection = $irss->collection()->get($rcid);

// Get only a range of the collection
$range_generator = $collection_services->rangeAsGenerator($collection, 10, 200);
$range_array = $collection_services->rangeAsArray($collection, 10, 200);

Resource Flavours

Flavours are derivatives of resources that can be created, managed and used alongside the original resource. A simple example are different resolutions of images: If a person uploads e.g. a profile picture with 20MB size, the IRSS can generate several flavours from it, e.g. a 500x500px flavour in an adapted quality, which can be used for the representation in a member gallery, without the full resolution with 20MB having to be loaded and displayed in the browser. Similar approaches exist in ILIAS in many places, but always the components themselves must ensure that the desired " derivatives" are available, stored and can be retrieved. With the Flavours the IRSS offers this centrally.

Structure of Flavours

For the creation of Flavours technically 3 things are needed:

  • Definition
  • Machine
  • Engine

Definition: The consoments of the flavours use in most cases a definition, which kind of flavour they would like to get from the IRSS. For many cases the IRSS already offers possible definitions that can be configured. One definition is for example "A preview image with max. 500x500px".

Machine: A machine can process such a definition and create the desired flavors. For a definition, there must always be a machine that can process this definition. For the definitions provided by the IRSS there are also corresponding machines.

Engine: A machine sometimes needs an engine to be able to create the desired flavours. Such engines are e.g. packages like Imagick or GD. Some machines also get along with a simple NoEngin.

How to use Flavours

Let's take the example of file objects: Already in older versions of ILIAS preview files could be created for file objects, different file types were supported. In the case of PDFs, for example, the first 5 pages are generated as a preview image.

The file object can now request this process from the IRSS via a definition. The IRSS knows two methods for the creation or retrieval of flavors:

global $DIC;
$irss = $DIC->resourceStorage();

$irss->flavours()->ensure(...);
$irss->flavours()->get(...);

ensure can be used to trigger the creation of the flavours, e.g. after an upload of a new file. get creates them as well, but you also get the result directly and can display the flavors.

With the following setup we generate max. 10 thumbnails for a resource. The IRSS takes all file types and generates up to 10 thumbnails, if the file type of the resource is supported by the Engine. So in case of PDF you get 10 thumbnails, in case of JPG only one. Other file types do not result in any preview image.

global $DIC;
$irss = $DIC->resourceStorage();

// the ResourceIdentification we want to get the flavours for
$rid = $irss->manage()->find('53ed29e0-7c13-49f0-a077-bcb5f87fa50c');

$definition = new PagesToExtract(
    true, // persistent, Flavours are stored in FileSystem, otherwise it's in-memory
    280, // max size (w/h) of the preview image
    10 // max number of pages to extract
);

$flavour = $irss->flavours()->get($rid, $definition);

The result is a Flavour object. These contain some meta data about the flavour and with getTokens you can access alls Tokens for StreamAccess (see below). Normally, you would now want displayable URLs for these FlavourStreams. The Consumer service of the IRSS supports this accordingly (which uses those Tokens inside):

$flavour = $irss->flavours()->get($rid, $definition);
$urls_of_flavour_streams = $irss->consume()->flavourUrls($flavour);

// create UI Images for each flavour stream 
$images = array_map(function (string $url): Image {
    return $this->ui->factory()->image()->standard($url, 'Flavour');
}, $urls_of_flavour_streams);

The flavourUrls consumer already supports the WebAccessChecker and the URLS are signed accordingly.

Tokens for StreamAccess

The Flavour object contains a list of Tokens for the StreamAccess service. These tokens can be used to access the Stream of a Flavour or of a Revision, but you hopefully won't ever need to do this directly. The Consumer service of the IRSS supports this accordingly (which uses those Tokens inside). Tokens are a mechanism to manage write (and later read) access to streams from the IRSS. In a first step, the WebAccessChecker also uses these tokens to know which revision or flavor exactly should be delivered.

As I said, if you are a consumer of the IRSS, you use the Consumers approach anyway.

Other (involved) Services

  • UploadService, see here
  • FileSystem-Service, see here

Migration

  • Migration will need at least the feature "Migrate Command"
  • A documentation how you can migrate your component is here.

Roadmap

Please see the roapmap of the service in README.md

 

1.3 Migrations

Files were historically stored by the components in a directory in the iliasdata or data directory. If a component wants to use the Storage Service to manage files, all existing files of this component have to be moved to the Storage Service once.
This task can take a long time depending on the components and the size of the installation. To make this easy ILIAS 7 offers the possibility to perform migrations in the setup. A migration can be executed in many small parts, for example file by file. Such an implementation has already been done for ILIAS 7 for the file object. But since many other components use the FileObject internally as well this has been rewritten so that the old methods still work. In a further step all these components have to be rewritten so that they directly use the storage service and no longer the file object. In a further step, all other places that work with uploaded files should also be reprogrammed.

1.4 Features in Future

The central administration of all files will enable new features in the future.
For example, it will be possible to reintroduce a quota per user, which will then work globally for all uploaded files.
ILIAS will be able to perform regular actions on all files, e.g. checking for viruses or checking if files are orphaned and delete them automatically.

2 Involved Maintainers and Stakeholders

3 Timeline

Release
Objectives and developments
With ILIAS 8 the following topics were tackled and implemented:
  • The Resource Stoage Service has been implemented and is available via DIC The service is therefore usable for all components and offers all necessary functions to store and manage files. See Introduction of Resource Storage Service
  • Migrations are possible: The CLI setup of ILIAS 7 has been extended so that large migrations can be initiated and tracked after an update. See Setup - Add migrate Command
  • The first migration was the migration of the file object. Due to the large number of file objects in ILIAS the file object was converted to work with already migrated as well as with not yet migrated files. Thus an installation can be put back into operation immediately after the update. The migration of the file objects can then be done during operation. See IRSS: Migrate File-Object-Files to IRSS
Status
With ILIAS 9 we aim to address the following points:
Some components use uploaded content that is not a single file but a collection of files, such as HTML or SCORM learning modules. For this purpose, the storage service is extended by containers that can manage such related files.

4 Related Feature Requests and Status

Feature Request
Suggested by
Funding
Planned Release
Status
7
implemented
7
implemented
7
implemented
8 + 9
ongoing
8
implemented
9
implemented
9
implemented (Forum posting files), Draft files still need to be migrated by Michael Jansen
9
implemented
9
implemented

5 Further Results

6 Additional Information

7 General Discussion

Please discuss specific questions of feature requests on the related feature wiki pages. This discussion section is only for a general discussion of the project and its realisation.
Schmid, Fabian [fschmid] 2019-03-01: A basic version of a File-Storage-Service should be implemented for the feature Customisable Main Bar - Icon Upload. The service won't be rolled out to the whole ILIAS base with 6.0. 
JourFixe, ILIAS [jourfixe], 11 MAY 2020 : We highly appreciate the suggested project. The project manager will provide a template for such project pages, soon. And he will help to adapt the content to this structure. All related feature requests and pull requests should be listed here but discussed separately. (Done in the meantime.)
JourFixe, ILIAS [jourfixe], 21 FEB 2022: Schmid, Fabian [fschmid] presented the project at the Big Projects Jour Fixe and answered questions.
  • Relevance for strategic ILIAS development: Important continuation of the conversion to IRSS in many other components.
  • Plan for ILIAS 9: Convert other components to IRSS
  • Chance of realisation: Conversion in components can also be done by the respective maintainers themselves
  • Dependencies: Cooperation of other maintainers
  • Impact on other development activities: No impact
Statement Technical Board, 2022 Apr 6: The TB believes this effort to be on a good track. First components are already migrated, and the infrastructure is ready. The remaining effort mostly lies in migrating the various components using file storage functionality. To complete this project soon, we assign a high priority to it. We ask every collaborator to focus on this effort or include according changes into other efforts. If we all pull together, we can move our system to the IRSS soon. To achieve this, we feel that Fabian will mostly need to do some further marketing and dissemination, and we will happily support him as Technical Board to convince everyone to finally use the IRSS.

Ultima modifica: 28. Apr 2023, 11:24, Haagen, Nils [nlz]