Show Advanced KnowledgeHide Advanced Knowledge

Setup - Add migrate Command

1 Initial Problem

The current implementation of the setup knows "database updates" which originally seem to have been a way to modify the schema of the database. Since these steps have not been restricted programmatically much, they have been used for other kind of operations on update as well, (e.g.) moving files, reloading the control structure or performing complex operations to shuffle data around. In other circumstances (e.g. migration of certificates), these kind of operations have been moved to other parts of the system.

To incorporate heavy and long lasting tasks in the setup, we propose to introduce a new kind of setup steps, called migrations.

2 Conceptual Summary

All migrations or specific migrations can be triggered and monitored with the migrate command. As other objectives in the setup (for update or installation) these migrations can be defined by individual components of the system. A migration is kind of objective that

• can be broken down to many small tasks
• can thus be resumed and
• has a measureable progress

The migrations will have unique names and can be monitored via the status command if we decide to implement that. Unique tasks within a migration may also be triggered from elsewhere, e.g. on demand from a user.

A migration must be build in a way that allows the installation to be available, even if the migration is not completed.

3 User Interface Modifications

3.1 List of Affected Views

This feature will be visible in the CLI of the ILIAS Setup.

3.2 User Interface Details

The CLI will get a `migrate` command. Calling `migrate` will show a list of migrations that have not been completed yet. `migrate <migration-name>` will show the status of the migration with the specified name. An Option "--run" will either run an individual migration when defined via `<migration-name>` or all migrations that are currently not completed when no individual migration is specified. When running the migration, the command will output the progress frequently. `run` will also support a `step`-parameter and will only run the defined number of steps if that parameter is supplied.

3.3 New User Interface Concepts

None.

4 Technical Information

This will be based on some new infrastructure in src/Setup that will be presented asap.

5 Privacy Information

No implications.

6 Security Implications

No implications.

7 Contact

• Author of the Request: Klees, Richard [rklees]
• Maintainer: Klees, Richard [rklees]
• Implementation of the feature is done by: CaT ILIAS Team
  

8 Funding

If you are interest in funding this feature, please add your name and institution to this list.

• CaT Concepts and Training GmbH

9 Discussion

Klees, Richard [rklees], 2020-09-25: I want to bring this up in the JF already, although I'm not sure we will have a user for this Setup-feature with ILIAS 7. I see that Fabian Schmid's changes in the file service will be an obvious candidate for this feature. I'm not sure about the status there, though. I see we had some migrations that were handled as updates or by other means in the past (some repair tasks in the T&A and the migration of the certificates come to mind immediately), so I think the general requirement for this feature should be evident. Even if we can foresee that this won't have a user with ILIAS 7 I would still advocate to include this feature in ILIAS 7 because this will allow plugin developers and service providers to try and test the infrastructure already.

Schmid, Fabian [fschmid] 2020-09-25: Many thanks for this feature suggestion! I am convinced that exactly such mechanisms are needed in ILIAS to perform major migration steps during an update. When upgrading you can choose between at least two types of actions that have to be done during the update:

• Adjustments to the database schema
• Migrations of large amounts of data (in database but also file system)

Certain actions must take place exactly at the time of the update (e.g. adaptation of the database schema and also the associated data migration in the database). Other actions can be triggered, but do not necessarily have to be completed during the update. For example, moving large amounts of data in the file system. This can be continued even after the update of the installation has finished by running, e.g.

• in the background as BackGroundTask
• As CronJob
• Ad-hoc when accessing a file
• ...

From my point of view all these variants can be combined (but at least triggering during the update, then ad-hoc in operation).
As you already write, it needs:

• A "migration"
• small migration steps (e.g. single or ranges)
• a measurable progress
• derived from it a status (completed, running, ...)

This is exactly what I need for example for the migration of the file(objects). From my side you have full support and I can also do the implementation if you want me to.

JourFixe, ILIAS [jourfixe], 28 SEP 2020 : We highly appreciate this suggestion and schedule the feature for ILIAS 7. A first use case could be the planned migration of the file system.

10 Implementation

The feature is implemented as described. Additionally, a safety measure was added: If user do not use the `--yes` option, they need to confirm to actually want to run the migration by typing its name. Please look into PR 2968 for the initial implementation. A documentation for developers is also available. For a usage example, please have a look into IRSS: Migrate File-Object-Files to IRSS.

Test Cases

Test cases completed at 2020-11-224 by Klees, Richard [rklees]:

• C42482 : Migration anzeigen
• C42483 : Migration ausführen

Approval

Approved at 2020-11-12 by Amstutz, Timon [amstutz] with kind support of Schmid, Fabian [fschmid] while working through the approval for IRSS: Migrate File-Object-Files to IRSS.