Introducing Workflow Engine

1 Requirements

1.1 Description

The workflow engine is a modular embedded runtime within ILIAS that allows to host a multitude of processes which implement complex, event-based behaviour.

Main advantages of a workflow engine in comparison to "plain old php - code" are:

• Features, that require conceptual adaption to meet the customers needs can be delivered with the trunk, while customer specific changes are very local and easier to maintain against the code base.
• While this first implementation does not support GUIs, a future visual designer for workflows would allow end users to implement their own process requirements.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

<?php
/* Copyright (c) 1998-2012 ILIAS open source, Extended GPL, see docs/LICENSE */
 
require_once './Services/WorkflowEngine/classes/workflows/class.ilBaseWorkflow.php';
 
/**
 * ilPetriNetWorkflow1 is part I of the petri net based workflow engine tutorial.
 *
 * @author Maximilian Becker <mbecker@databay.de>
 * @version $Id$
 *
 * @ingroup Services/WorkflowEngine
 */
class ilPetriNetWorkflow1 extends ilBaseWorkflow
{
 
	public function __construct() 
	{
		$this->workflow_type				= 'Tutorial';
		$this->workflow_content				= 'Part 1';
		$this->workflow_subject_type		= 'none';
		$this->workflow_subject_identifier	= '0';
		$this->workflow_context_type		= 'none';
		$this->workflow_context_id			= '0';
		$this->workflow_class				= 'class.ilPetriNetWorkflow1.php';
		$this->workflow_location			= 'Services/WorkflowEngine/classes/tutorial';
 
		require_once './Services/WorkflowEngine/classes/nodes/class.ilBasicNode.php';
		$node_1 = new ilBasicNode($this);
 
		require_once './Services/WorkflowEngine/classes/detectors/class.ilSimpleDetector.php';
		$detector_1 = new ilSimpleDetector($node_1);
 
		$node_1->addDetector($detector_1);
 
		require_once './Services/WorkflowEngine/classes/detectors/class.ilCounterDetector.php';
		$detector_2 = new ilCounterDetector($node_1);
		$detector_2->setExpectedTriggerEvents(3);
		$node_1->addDetector($detector_2);
		$node_2 = new ilBasicNode($this);
 
		require_once './Services/WorkflowEngine/classes/emitters/class.ilActivationEmitter.php';
		$emitter_1 = new ilActivationEmitter($node_1);
		$detector_3 = new ilSimpleDetector($node_2);
		$node_2->addDetector($detector_3);
		$emitter_1->setTargetDetector($detector_3);
		$node_1->addEmitter($emitter_1);
 
		require_once './Services/WorkflowEngine/classes/activities/class.ilSettingActivity>php';
		$activity_1 = new ilSettingActivity($node_1);
		$activity_1->setSetting('ilWorkflowTest_Node1_transition', '1');
 
		global $ilSetting;
		$ilSetting->set('ilWorkflowTest_Node1_transition', '0');
		$node_1->addActivity($activity_1);
 
		$detector_4 = new ilSimpleDetector($node_2);
		$node_2->addDetector($detector_4);
 
		$activity_2 = new ilSettingActivity($node_2);
		$activity_2->setSetting('ilWorkflowTest_Node2_transition', '1');
 
		$ilSetting->set('ilWorkflowTest_Node2_transition', '0');
 
		$node_2->addActivity($activity_2);
 
		$this->setStartNode($node_1);
 
		$this->addNode($node_1);
		$this->addNode($node_2);
 
		$this->startWorkflow();
 
	}
}

The whole concept is around nodes, that switch when their detectors are all satisfied. When switching, attached activities are triggered. With this simple building blocks, complex features can be realised, like compliance workflows, mailings / reminders in general, course membership limitations and the like.
 
An in depth introduction to the engine is availabe in the asfinag-branch: branches/databay/asfinag/4_2/Services/WorkflowEngine/classes/tutorial, a four part series showing all the nitty gritty details.
 
The code is "ready to merge" with minor changes (e.g. taking advantage of the new event system). In regards to quality, it is worth mentioning, that the code is covered with 123 unit tests, so all developers may feel comfortable about using the service. (ILIAS trunk, rev. 35097, has 77 tests total so far.)

Future perspective for this service is the use of industry standard business process modeling tools (like BPML 2.0s' XML Standard) to customize the behaviour of ILIAS in defined parts of the system. The engine now provides "primitives", which allow to form blocks of functionality, that may well fit into widely differing modeling worlds. Currently, there are evaluations for a toolchain and the technical feasibility to use visual design tools to create and modify workflow definitions.

1.2 Use Cases for Workflow Engine

As requested by the Jour Fixe in June 2012 we should list some use cases where the workflow engine could show it's added value:
 
7 February 2014: This is a very specific use case where a workflow engine could support our processes: "Creating test questions cooperatively". We would like to have the feature "WorkflowEngine" again discussed at the next JourFixe, since we are willing to support it as a development. For further questions, please contact: Nadine Köcher / koecher@dhbw-karlsruhe.de

Here is the presentation of the WorkflowEngine as held on the 2012 DevConf in Stuttgart, containing a variety of samples for possible use cases on slide 5.

Another presentation was held for the Databay board of directors which explains the current state of the machine as well as some of the possible extensions.

The implementation of the REST Server mentioned in this file would of course massively profit from the existence of the REST Service when it comes down to implementation cost.

Stefan Schneider of University Marburg has also provided sample workflows which demonstrate use cases for the workflow engine.

1.3 Workflow Example "Membership Limitation"

In this example we investigate a well known feature request Databay AG has done serveral times in the past using scripts triggerd in the cronjob. It deals with the limitation of membership for a course.

The first diagram shows how this is achieved "now". Every week, for every user-course-relation, someone goes through the list and determines if the member has exceeded the time limit in the course and eventually removes the member.
Noone will ever do this kind of thing manually, but in theory, this is how it's done without code or an automated workflow engine. The small hand-symbol marks the tasks as "manual tasks", the tiny clock in the start event marks the starting condition.

Now someone says: Wow, this is painful and a waste of time, can we automate this? And of course, we can do that.
The second diagram shows this tiny process. The start event now shows an envelope, which means that we receive a message (call it: Event) about a user that on some way became a member of a course. The relevant information - user and course - is then put into the workflows instance store. Regularly, this would not appear in a diagram, it's just so you know why we know, "who and where to act on".
The control flow now "runs against the clock". This is a so called intermediate catch event. The control flow is stopped until the timers condition is met and then continues.
Finally, a script task (see the icon in the upper left of the box) is called to revoke the membership.

At the end of the process, the membership is terminated.
The revoke course membership task does the following: It calls a method on a given facade class and executes a method, passing in a reference to itself (the 'context') and the instances data object.
To route the script tasks execution, the following three informations need to be added to the model:

• The path to the facadeclass, e.g. Modules/Course/classes/class.ilWFECourseFacade.php
• The name of the facadeclass, e.g. ilWFECourseFacade
• The name of the method to call, e.g. revokeCourseMembership

class ilWFECourseFacade
{
  public static function revokeCourseMembership($a_context, $a_params)
  {
     include_once "Modules/Course/classes/class.ilCourseParticipants.php";
     $participants_object = ilCourseParticipants::_getInstanceByObjId($a_params['course_id']);
     $participants_object->delete($a_params['user_id']);
     return;
  }
}

This method does only one thing: It gets the object to call the given method with the parameters "reformulated" from the generic syntax.
So this saves the workflow designers from knowing about objects and instantiation, requiring files and all this.
What the workflow designer needs to know is simple "rtfm-knowledge", a description of the task, what to enter in the workflow and which instance variables should be around.
(Ok, he should also know what data comes with the starting event  - user was assigned to course - and what to enter at the event-element, so the information from the event ends up in the instances data object.)
This is not perfectly trivial but this is really not the same thing as actual programming the functionality. Stuff the power user knows or something an ilias administrator learnt.

Where does this all lead to? So far, what we have is something a programmer can hack within a short time. The advantage for workflows over custom development starts here, when requirements change.

Let's see. The end users are happy - for a while. Then the next request comes in: "We should give users a notice before their membership ends."
Lets do that:

The "send message" task is inserted between two timer events. This task will get properties like subject and text and a receiver, which may either be a literal address or a user_id.
The revoke course membership task remains unchanged, perfectly reuseable code.
We omit here a handful of variations, e.g. "also inform the tutor", "send two notices", "send a mail when the membership is revoked" and the like, the idea is the same, we have a reuseable facade and the notations elements are also reuseable.

One case, that may be interesting, would be, that the user has the opportunity to renew his course. We again extend the workflow to look like this:

New here is the "event based gateway", which is an exclusive-or (XOR) gateway, that executes on the path where the event attached is received first. So if the second timer is triggered, then the membership will end, unless the other path is invoked.
This invokation would be the receiving of an event that is raised when the renewal was done. We leave it open what kind of event this is, but it could be anything from an event raised in the shop, an event raised by a webservice-method or something happening in a plugin.
 
Again, the method in the facade is reused while we have scaled and changed the business process considerably at the same time.

This example has shown an intrinsic premise of what workflows mean. The activities - the "work" - remains the same, the "flow" to it, the way through decisions and events, differ from organisation to organisation.

Files / workflows created with the camunda Modeler (open source).

1.3.1 Specific Questions of the 17 Mar 2014 JourFixe

Doubling code / parallel worlds:
As the facade method shown above explains, there is not more or less code than in any regular class that operates on or consumes the API of a module. The sole purpose of every single instruction in the facade is to call an available method or the shortest way to being able to do that. In the past, one may have seen "actual code" in the activity libraries. This was done to overcome the absence of a certain feature - namely the deletion of learning progress. Within the maintainerships and everything "core" involved, such can be ruled out as a violation: A facade class, by the patterns definition, does not carry any business logic. Should the need for specific code arise, there are compliant ways that allow such, e.g. by plugins providing their own facade classes.
 
Internal API alignment:
The concept of the facade classes as shown above is addressing a different problem than the problems due to the absence of a concept for aligned API exposure in the project. However, due to the slender nature of the facade methods, a switching to a new standard internal API looks simple and quick.
With the above example we expect a library of 100 activities in the range of 1,000 - 1,500 lines code (NCLOC), distributed over serveral modules, that would need attention in the process of adopting a new internal API Königsweg.
Seeing and practicing the facade pattern as used in the workflow engine however may help during the process of developing a sustainable standard for the internal API alignment.
 

1.3.2 Activity Library

With the first implementation, it is proposed to add the following facade-methods to allow for a variety of workflows:

• Grant skill / revoke skill
• Assign / deassign role (in here is the course membership revocation from the above example)
• Set course online/offline
• Send Mail
• Add/Remove user as fixed test participant

1.3.3 Pluginslot

Workflow modeling methods and the BPMN2-standard know a concept called "rule engine". Such a rule engine encapsulates a complex piece of business logic (e.g. "discount calculation" or "course assignment list by user criterion"). To allow for this, a plugin slot is also proposed that can be consumed from within a workflow.

1.3.4 Sample use cases for workflows

These sample use cases for workflows in ILIAS are taken from the above attached files and represent the current state of imagination. Some new ideas are listed as well.

• Communication
  • Control about mailings for time-limited users
  • Shutdown / removal processes for content elements with early notices for users
  • Reminders about contents not consumed, tests not taken, etc.
• Escalation-Logic
  • Complaints management
  • Suggestion systems
  • Support
• Compliance Systems
  • Monitoring of course memberships
  • Enforcement of recurred learning (security advisories, workplace safety, hygienics, young person protection law, infectious disease protections)
  • Automatic management of "Joiners" logic (e.g. make sure certain content for new students/employees is consumed)
• Membership Management
  • Assignment and deassignment of course/group memberships to build learning paths or assignments by user criteria
  • Management of session participation
• Automatic Content creation
  • Creation/Duplication of repository structures (e.g. Setup of annually created courses from predefined templates.)
• Systems integration
  • Embedding ILIAS into larger corporate workflows (e.g. making course-participation part of a larger hiring process or assignment of tests and signalling the outcome for assessment centers)
  • Bridging gaps between systems (e.g. integration of DMS for archiving purposes)
• E-Assessment support
  • Scheduling, invitation and location booking (see Nadine Köchers workflow "eKlausur-Erstellung")
  • Negotiation support (e.g. developing a common body of exam questions, see Nadine Köchers workflow "eKlausur-Erstellung")
  • Quality assurance processes
  • Coordination of process-components (e.g. corrections process vs. results access, see Nadine Köchers workflow "eKlausur-Erstellung")
  • General administrative processes (see Stefan Schneiders workflow "Buchung E-Klausur")
  • Assignment of supervisors for exams (see Stefan Schneiders workflow "Buchung E-Klausur")
  • Transport and creation of tests between ILIAS instances (see Stefan Schneiders workflow "E-Klausur - Teilnehmerverwaltung")

Please note: The workflow engine is able to support both automatic as well as manual tasks. This means that not every individual activity that could be part of the above mentioned use-cases is necessarily available as an automatic script task. It can be decided on a case-by-case basis if relevant activities should be made available as script tasks. A growing library of tasks is suggested with the feature and the use of manual tasks vs. script tasks is fully backward compatible. So should an activity be available in a later version, the process can still contain a manual task until the organisation reviews the process and decides to automate a task.

1.4 Development Packages

The following packages were defined, so the funding can be distributed with working product increments:

• BPMN2-Parser Elementtranslation (funded by DHBW / optes)
• BPMN2-Parser Controlflow (9.000,00 €, funded by Universität Freiburg)
• BPMN2-Fileset for Parsertests (1.800,00 € funded by Universität Freiburg & DHBW)
• WorkflowEngine Administration - (funding required) 4.500,00
• Activity-Package "E-Exam Session Management" - (funding required) 2.700,00 €
• Activity-Package "Memberships" - (funding required) 3.600,00 €
• Activity-Package "Mailings" - (funding required) 2.700,00 €
• Activity-Package "Assessment Archive Integrity Checks" - (funding required) 3.600,00 €
• Service Discovery - (funding required) 3.150,00 €
• Service and Events Administration and Documentation System -  (funding required) 4.500,00 €
• Service and Events Developer Documentation - (funding required) 1.800,00 €
• Adaption Service and Event Discovery of the WorkflowEngine - (funding required) 3.600,00 €

2 Additional Information

• Idea / concept: Becker, Maximilian [mbecker], Databay AG
• Funding is done by Crowdfunding, see also http://www.ilias.de/docu/goto_docu_dcl_3700_489.html:
  • Mrs. Becker
  • Databay AG AG
  • DHBW
  • Universität Freiburg
• Maintainer: Maximilian Becker
• Development: Feature is to be developed by Databay AG AG / Maximilian Becker
• Test cases by: ...

3 Discussion

MB, 18 June 2012: On the JF agenda for 25 June. We will also need to discuss where to put this page to in the wiki.

JF 25 June 2012: We discussed the workflow engine today. We see that it has potential to add flexibility, especially in customizing the behaviour of ILIAS. However we also see that it adds complexity to the system. We need a good use case for the engine that shows it's added value. A use case "could" be a revision of the activities controlled by the cron job, e.g. limited/trial user accounts.

MB 14 Mar 2014: I have added some materials to the page. I would like to point out that the extensions discussed in the presentation for the databay board are all optional, since we consider the engine itself complete. We would however strongly vote to at least deliver the engine for the trunk with the there mentioned BPMN2-Parser, allowing for externally modeled workflows to be processed in the ILIAS workflow engine.

Jour Fixe, 17 Mar 2014: We had a long discussion about the integration of a workflow engine into ILIAS. We agreed that an integration of the workflow engine needs a BPMN parser and a clear roadmap of activities we are starting with.
 
We are collecting use cases for the workflow engine on this page until March 31 (last JF before Feature Freeze) to give a clearer impression of its use, as well as pros and cons. We will have a final decision about the request on the next Jour Fixe.

JF 31 Mar 2014: We highly appreciate this extension and schedule it for 4.5. The following issues must be adressed during the development:

• The documentation of all events and actions should be generated automatically based on any information in the code (or in service.xml/module.xml files). (the documentation should be presented in the administration section mentioned below)
• A list of all output (events) and input (action) parameters should be included in the documentation (names and types). The naming of parameters should be discussed (see parameter discussion in event service, "crs_id" vs. "course_id").
• A guideline/example of exception handling must be provided. We assume that "interaction" code of the components throws component related exceptions (e.g. no course id given). These exceptions must be handled in some way.
• All additional actions/events and all parameter types (beside integer and text) must be discussed in the Jour Fixe.
• The workflow is invoked by a cron job or by the general event handling.
• A new administration section "Workflows" should be added, which gives an overview on all workflows installed and their run status. Workflows are added by uploading "bpmn" files into a directory (outside of webspace).
• It should be possible to list all running instances of workflows (textbased information on running time and parameters, not necessarily a graphical representation).

JourFixe, ILIAS [jourfixe], March 14, 2016: We reschedule the feature request for 5.2 with the following requirements:

• Due to missing funding, the required Service Discovery and automatic generated documentation can not be offered with a first implementation of the WFE. As an alternative, we need to have a written and continually updated document that clarifies how a developer can introduce a new activity
• All other information that is listed above should be available (in a written form if cannot be generated automatically).
• Activities related to components of other maintainers should be offered as pull request. Maintainer may asks for changes or modifications.
• We need a service discovery within the next two years to prevent huge documentation efforts and to offer an automatic generated documentation for the future. Without a service discovery we will stop future acceptance of activities for 5.4 and later.

4 Implementation

Becker, Maximilian [mbecker], 03. Feb 2015
Implementation of the first part of the BPMN2-parser is complete. Documentation will be added here once the parser is finalized with the second order.

Becker, Maximilian [mbecker], 24. Oct 2016
Writing up details regarding implementation and the documentation (events and activities) according to JF requirements is taking place in a learning module in the SIG WorkflowEngine Group. 

Test Cases

Test cases completed at 2016-08-23 by mbecker

• C12988 : Workflow-Definition hochladen / Aufruf des Formulars
• C12989 : Workflow-Definition hochladen / Abbruch des Formulars
• C12990 : Workflow-Definition hochladen / Hochladen einer Definition
• C12991 : Benutzung Definitions-Tabelle / Aufruf der Tabelle
• C12992 : Benutzung Definitions-Tabelle / Sortierung, Paginierung, Spaltenauswahl und Zeilenlimitierung
• C12993 : Benutzung Definitions-Tabelle / Filter "Titel"
• C12994 : Benutzung Definitions-Tabelle / Filter "Instanzen"
• C12995 : Mit Workflows arbeiten / Workflow starten ohne DataInputs
• C12996 : Mit Workflows arbeiten / Workflow starten mit DataInputs
• C12997 : Mit Workflows arbeiten / Definition löschen
• C12998 : Mit Workflows arbeiten / Event-gestarteten Workflow "scharf schalten"
• C12999 : Mit Workflows arbeiten / Event-gestarteten Workflow "unscharf schalten"

Approval

Tested successfully and approved at 2016-08-25 by and Glaubitz, Marko [mglaubitz].