Show Advanced KnowledgeHide Advanced Knowledge

Deployment of Background Tasks as a General Service

1 Initial Problem

ILIAS has several user-driven and/or recurrent actions/tasks/workflows which can  result in a high performance impact or at least can take a long time and users have to wait for completion, such as

• Copy a Category/Course/Gruop with a huge amount of subobjects
• Apply permissions recursively
• ZIP and download a folder/course/group/ ...
• Reload/Install Languages
• ...

2 Conceptual Summary

ILIAS should provide a way to put such processes into the background (async) and provide feedback about the state and progress of the to user, which has triggered the tasks.

The service must provide a configuration to control the amount of processes/tasks which are processed simultaneously.

2.1 Service

3 User Interface Modifications

The user interface is discussed in its own feature wiki entry: Background Task Service User Interface.

4 Technical Information

Overview

The smallest element of the background tasks are called "Task". A Task can be a "Job" (e.g. zip some files) or a "User Interaction" (e.g. Download a File). Several Tasks can be bundled to a "Bucket". For an example a bucket can be: Zip some files and then offer them for the user to download.  In order for these individual tasks to interact nicely, tasks define their input and their ouput by class. For example: The DownloadFile User Interaction will need a FilePathIO instance. The input and or output elements are ending with IO and extend an abstract class ilBackgroudTaskIO.

A bucket of work can be monitored by a user and will display a progress bar or request user feedback if a user interaction is encountered while working on the bucket.

A basic worker is implemented for the first iteration: If there are less than x buckets that are currently being processed, the worker takes a bucket and tries works through it. After he finishes the bucket he repeats the process. The worker will be triggered by a cron job in the first implementation.

A basic exception handler is implemented for the first iteration: If an exception occurs that is not caught early on, the whole bucket will terminate (purged from the db) and an error is displayed to the user in the "Running Tasks" pop over.

The interfaces can be found here: https://github.com/studer-raimann/ILIAS/tree/feature/5-3/BackgroundTasks/Services/BackgroundTask/classes. The classes in the root folder of the link are NOT relevant for this feature request, only look at the classes in the sub folder.

Usage Example

Have a look at the following example to see how the developer uses the Service in the end:
https://github.com/studer-raimann/ILIAS/blob/feature/5-3/BackgroundTasks/Services/BackgroundTask/classes/Examples/SimpleExample.php

Workers
We suggest to implement two workers for the first iteration:

• A synchronous worker: This will obfuscate the background tasks as the tasks will actually be handled synchrounously in the same http request. This way the behaviour will stay the same for most tasks as in the current ILIAS version. Furthermore this worker is needed for installations which don't configure cron jobs.
• A cronjob worker: A worker activated via cron job.

UML

A simple class hierarchy (as linked above) can be seen below.

5 Contact

• Author of the Request: Schmid, Fabian [fschmid]
• Maintainer: {Please add your name before applying for an initial workshop or a Jour Fixe meeting.}
• Implementation of the feature is done by: {The maintainer must add the name of the implementing developer.}

6 Funding

If you are interest in funding this feature, please add your name and institution to this list.

• ...

7 Discussion

Amstutz, Timon [amstutz], 29. Dez. 2016: Thank you for this FR. I think this is a good example of how a PR for a general Service can look like, since you took the time to provide interfaces and example code.

Concering UI, I have two questions:

• Is the new glyph in the Topbar Sticky? Does it disappear if no BG-Tasks are running? Remark: Do not forget to add the new Glyph the the Centralized UI-Components (e.g. Glyph->Task).
• This could be a good oppurtinity to add the Popover to the new UI-Components. However, since Popovers and Progress-Bars Buttons etc. all already exist, I would not regard it as new, since almost the same layout exists already for the Chat/Awerness Tool. Nevertheless, Popovers surely could profit from a centralized solutions serving at least for the Chat, Awerness Tool and the BG-Tasks.

Klees, Richard [rklees], 09. Jan. 2017: Thanks a lot for this FR! My thoughts and questions:

• I am not sure i understand the "User Interaction". Is it that a Task that requires information from the user? Or is it that the user gets access to the result of a task? Is this really the same? Could be benefitial to make a distinction between those two. This also seems to be a bit mixed up with these IO-classes.
• Would you be willing to put (at least) the interfaces to the services in the src-folder (and thus under the rules we defined for it)? Besides the guarantees this would provide, it also would mean that you should get rid of the BT suffixes which (for me) made it hard to follow the UML.
• Is a Bucket some kind of a composed task? Why doesn't it share the Task-interface?
• I'm somehow lost between Job/Task/Bucket.
• Is *IO just some Value passed between different steps in the task?
• In general i like this a lot. Terminology seems to be worth some rework.
• I would suggest to make it possible that either every task could have more than one input or that a task could return another task instead. This will make composition of tasks easier. I would suggest to use the first option.
• How is a Task different from an Action (?) in the Workflow engine? To me it seems as if both are the same and we are really searching for some general Task or Action monad for ILIAS.

Truffer, Oskar [otruffer], 10. Jan. 2017:
I'll try to answer the questions given by Richard and Timon. There's a new pull request only containing the Interfaces but in the src folder: https://github.com/ILIAS-eLearning/ILIAS/pull/405. If you have comments or questions concerning the implementation please post them on the github pull request.

• Is the new glyph in the Topbar Sticky? Does it disappear if no BG-Tasks are running? Remark: Do not forget to add the new Glyph the the Centralized UI-Components (e.g. Glyph->Task).

I'd suggest only visible if tasks are running.

• This could be a good oppurtinity to add the Popover to the new UI-Components. However, since Popovers and Progress-Bars Buttons etc. all already exist, I would not regard it as new, since almost the same layout exists already for the Chat/Awerness Tool. Nevertheless, Popovers surely could profit from a centralized solutions serving at least for the Chat, Awerness Tool and the BG-Tasks.

Agreed.

• I am not sure i understand the "User Interaction". Is it that a Task that requires information from the user? Or is it that the user gets access to the result of a task? Is this really the same? Could be benefitial to make a distinction between those two. This also seems to be a bit mixed up with these IO-classes.

Currently user interaction is just the possibility for the user to interact: Imagine a "attention this will take a really long time, do you want to continue anyways?" or a "You're Zip Folder is ready to download here." kind of thing. 

• Would you be willing to put (at least) the interfaces to the services in the src-folder (and thus under the rules we defined for it)? Besides the guarantees this would provide, it also would mean that you should get rid of the BT suffixes which (for me) made it hard to follow the UML.

Yes, we did so in the pull request.

• Is a Bucket some kind of a composed task? Why doesn't it share the Task-interface?

Yes, it is a composed task. I'll add that shortly.

• I'm somehow lost between Job/Task/Bucket.

User Interaction -> User input needed to continue Work. Job -> Something a worker can work on. Taks -> Either User Interaction or Job. Bucket -> An ordered list of tasks.

• Is *IO just some Value passed between different steps in the task?

Exactly.

• In general i like this a lot. Terminology seems to be worth some rework.

Agreed! :-) 

• I would suggest to make it possible that either every task could have more than one input or that a task could return another task instead. This will make composition of tasks easier. I would suggest to use the first option.

Having multiple inputs will make the implementation harder to do and the interface more complicated to work with. The question is whether a pipe is enough or not.

• How is a Task different from an Action (?) in the Workflow engine? To me it seems as if both are the same and we are really searching for some general Task or Action monad for ILIAS.

Yes, a Action monad for ILIAS is exactly what we need. Additionally to the monad we need to be able serialize the order/chain of execution, the results inbetween, the specific comuptations and the values inbetween. The Task and Action from the workflow engine are close. I did not find an interface in the workflow engine that matches the typisation, though.

Killing, Alexander [alex], 11 Jan 2017: Thanks for the detailed proposal.

• I also struggle a little bit with the UserInteractions. I understand them begin a step at the beginning or the end. But the current concept allows to create Job > UserInteraction > Job sequences since both are tasks and I wonder if this is intended. How would the user experience be for these cases? And do UserInteractions allow to dynamically influence the sequence of tasks (do a) or b) task depending on UserInteraction)?
• How will the cron worker work in general? How will tasks be performed in parallel and how will be controlled how many run in parallel? What if nothing is to be done - will the cron job stop? Does it mean the user would have to wait up to 59 seconds before the next worker is triggered by the crontab?

, 13.01.2017

• I also struggle a little bit with the UserInteractions. I understand them begin a step at the beginning or the end. But the current concept allows to create Job > UserInteraction > Job sequences since both are tasks and I wonder if this is intended. How would the user experience be for these cases? And do UserInteractions allow to dynamically influence the sequence of tasks (do a) or b) task depending on UserInteraction)?

The user interaction will take place in the same place where the progress is shown. A user interaction can be between jobs. Imagine the Zip category example: A user wants to download all files of a category recursivly. The bucket may look as follows:
Collect all Files the user has read permission on under this category -> User Interaction "The category contains 1392 Files, do you want to continue downloading?" -> Zipping all the files -> User Interaction "Download here."
Currently a dynamic flow is not in the concept for complexities sake. But similar to Richards comment we could make it possible for jobs to spawn new jobs, thus effectivly allowing such dynamic flows depending on the user input.

• How will the cron worker work in general? How will tasks be performed in parallel and how will be controlled how many run in parallel? What if nothing is to be done - will the cron job stop? Does it mean the user would have to wait up to 59 seconds before the next worker is triggered by the crontab?

A maximum of tasks that can run at the same time should be defined in a ILIAS config file or the setup. The second point is a really difficult one: The easiest solution is to stop the cronjob when there are no tasks left, but as you mentioned correctly, a next task might be delayed up to 59 seconds without any real reason appart from the technical implementation. I'd suggest to either spawn a cron job task on the server, that runs the job, whena  new task is added, additionally to the regular cron job. Or to leave out the cron job and directly implement a "real" background job with e.g. RabbitMQ (https://www.rabbitmq.com/tutorials/tutorial-two-php.html).
 

, 13.01.2017
FYI: I split up the GUI and the Service implementation part. Thus we can discuss and decise separatly on the look & feel and the implementation.

AT, 2017-01-14:  Please make sure that the lable "task" is attached to exactly one concept: either background task or a task list. Please devise.

Killing, Alexander [alex], 16 jan 2017:

• "I'd suggest to either spawn a cron job task on the server, that runs the job, whena  new task is added, additionally to the regular cron job. Or to leave out the cron job and directly implement a "real" background job with e.g. RabbitMQ". How would spawning a cron job task work starting from an apache request? I think we are facing very different decisions depending on being able to use cron or something like RabbitMQ which would add a new dependency. Why is the current implementation (using SOAP calls to put things in the background) not an alternative? These processes can always be started immediately and limiting parallel tasks should be quite easy, too.
• "The bucket may look as follows: Collect all Files the user has read permission on under this category -> User Interaction "The category contains 1392 Files, do you want to continue downloading?". So in general it could be "Start bucket -> Long Job A -> User Interaction -> Long time job B", right? Will there be any notifications or does the user have to open the popover to check updates? I think we could use OSD to notify users.

Schmid, Fabian [fschmid], 16.01.2017: @Alex: Concerning the notifications we could use the suggested OSD. As a first step we would suggest to use the topbar glyph with the following indication: "orange" novelty-counter for userinteractions (because you can klick the buttons to get rid of the orange counter) and the "grey" status-counter to inform about the amaount uf running jobs. I put this also to the page Background Task Service User Interface.

JourFixe, ILIAS [jourfixe], 16 Jan 2017: We appreciate the concept in general and would like to use SOAP for starting jobs plus Cron to start workers again if open tasks still exists. This would avoid to install additional software (RabbitMQ). To schedule the feature finally, we need a clear idea about an administration interface in the setup where a system administrator can stop accepting new background tasks for all clients, see how much running and waiting tasks are still exist per client and restart the task manager again. Please add a concept for this workflow to the request.

25. January 2017

I added the requested changes in the GUI part + an explanation on why the settings are in the setup and not the administration here: Background Task Service User Interface. 

8 Implementation

{The maintainer has to give a description of the final implementation and add screenshots if possible.}

Test Cases

Test cases completed at {date} by {user}

• {Test case number linked to Testrail} : {test case title}

Approval

Approved at {date} by {user}.