Introduction of Service Discovery / Internal API alignment

1 Initial Problem

ILIAS has a growing number of internal service consumer. By the time of the invention of the requirement that is addressed with this feature, it was SOAP, REST and the workflow engine as well as a perspective for more such consumers in the future, which may hail from other platforms like Android and iOS. For such components and clients, it is vital to not only implement their protocols and communications, but also their interaction with the ILIAS core, be it modules, services, the database or even plugins. Until now, this is done through implementations on a per-consumer-level, meaning there is a specific solution for SOAP and another one, REST, and at this time, for the workflow engine. These three solutions are currently located in locations relevant to the consumers, namely ./webservice/soap/classes, ./Services/WebServices/Rest/classes and ./Services/WorkflowEngine/classes/tasks.

Also, at the time of this writing, ILIAS lacks a consciously designed, clear document and well communicated standard for internal APIs. It's on each developer/maintainer, how they offer services to the rest of ILIAS and while certainly each maintainer does their best to offera meaningful and properly working interface, there is no common standard in us, that allows developers to just call a foreign service without knowledge or research if the call is correct. Even with descriptive names on classes and methods, it's relatively safe to say we've all been to "Oh, no, don't use *that* method."-land. 

This request, as required per JourFixe decision on [], aims to improve the following situations:

• Offer a way to present an internal API to other developers, webservices and end users in a machine-readable way.
• Offer documentation in the ILIAS GUI for end users to take advantage of the available services e.g. in webservice integrations or workflow definitions.
• Move APIs as facade classes back into the maintenances of the proper maintainers in a consumer-agnostic way.

• Maintainers have better insight in changes to their API and cross-communications with webservice-maintainers about changes are not required. A maintainer will be responsible for an API to their component being available and properly working.
• New API methods that allow interaction with the core will be in the control and responsibility of the affected components maintainer, making coordination and eventual subcontracting with a webservice maintainer unnecessary. (Or even multiple.)
• Newly introduced consumers - the next webservice hotness *will* come - may take advantage of existing APIs without a hefty pricetag attached to recreating existing means for their specific purposes.

2 Conceptual Summary

The Service Discovery will define a way to mark-up classes contents as discoverable services using docblock syntax.
It will introduce a new scannig-step during reloading the ILIAS control structure and preprocess definitions found in the modules.xml/service.xml files into the database for consumption by services and the UI.

The Service Discovery will introduce a browsable directory of API documentation for ILIAS Administrators and Developers, following the UI-Browser visual, which can be reused here.

The descriptions in the Service Discovery will not only describe the services itself but also error-information and description of data and data structures moving over the service discovery.

By employing the service discovery, a layer of added transparency is added to the ILIAS codebase, review of APIs with the goal to adopt proper development craftsmanship, is simplified. Patterns like CQRS can be validated as in place and then also be effectively required by the communities organs.

3 User Interface Modifications

3.1 List of Affected Views

A general concept of the Service Discovery was created here: SIG WorkflowEngine Concept Development Wiki

The Service Discovery will hook into the ILIAS Administration menu with an entry that leads to a browsable directory of API documentation.

3.2 User Interface Details

As shown in the wireframe, which is please considered abstract, class and method docblocks of designated APIs (see 4 for further details on that) are displayed in a human readable and easily navigatable way.

The selection of a class (signified with a dropdown menu) leads to a page that displays the classes documentation in full as well as a navigation on the page to allow for quick jumps to the desired paragraphs.

3.3 New User Interface Concepts

No new user interface concepts. It is intended to follow the look and feel of the kitchen sink documentation.

4 Technical Information

4.1 Announcing a Class as Internal API

A class offering an API for automatic or semi-automatic consumption must follow certain guidelines and conventions. To allow control and save ILIAS from the necessity to detect such, classes with discoverable services are listed in the object definitions (module.xml/service.xml) and plugin.php for plugins.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<?xml version = "1.0" encoding = "UTF-8"?>
<service xmlns="http://www.w3.org" version="$Id$"
	id="rep">
	<baseclasses>
		<baseclass name="ilObjPluginDispatchGUI" dir="classes" />
		<baseclass name="ilRepositoryGUI" dir="classes" />
	</baseclasses>
	<objects>		
		<object id="reps" class_name="RepositorySettings" dir="classes"
			checkbox="0" inherit="0" translate="sys" rbac="1" system="1" administration="1">
			<subobj id="rolf" max="1">rolf</subobj>
			<parent id="adm" max="1">adm</parent>
		</object>
	</objects>
	<pluginslots>
		<pluginslot id="robj" name="RepositoryObject" />
	</pluginslots>
	<servicediscovery>
		<service_api path="Services/Repository/classes/api/class.ilRepositoryTasksGroupOne.php" />
		<service_api path="Services/Repository/classes/api/class.ilRepositoryTasksGroupTwo.php" />
	</servicediscovery>
</service>

An object definition can hold more than one class, allowing complex objects to group tasks in a logical way.
during control structure reload, the classes are parsed and their documentation is stored in the database.

4.2 Database Schema Changes

The database will get one new table with columns

• Component, a varchar holding a base path to component such as Modules/Course, Service/User or the semantic plugin path.
• API, a varchar holding the name of the class
• Path, a varchar with the full path to the file in question
• Method, a varchar which is a name or "null" for the classes docblock
• Definition, a CLOB with the full docblock that will be parsed for user

4.3 Class and Method Documentation Example

Services should be exposed by means of public static methods and should not make any assumptions about the runtime except for it being barely initialised. Documentation should be made available using docblocks which is a widely known concept for code documentation. We can safely assume every core developer is familiar with it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

/**
* SomeName (1 line title)
* Description (n lines of description terminated by a blank doc-line)
*
* @moreUsefulInfo
*/
class SomeName
{
        /**
        * Read an Entity by the given entity_id
        * Reads an entity from the database and returns the
        * the object as string (for example simplicity).
        *
        * @param integer $entity_id Identification number of the entity which shall be read.
        * @return string The entity as string.
        */
        public static function readEntity($entity_id)
        {
                // implementation code omitted
                return $entity;
        }
}

Using this schema, the creation and documentation of API methods which can be used in services does not require maintainers to learn new things about the consumers, they offer services to.
However, it is necessary to follow the conventions in order to achieve a quality documentation. On the technical side, especially @params need both type and name for the system to work properly.

A new docblock-element @service_invisible, should be introduced, too. Since all methods of the API classes must be public, methods marked as @service_invisible should not be visible or callable by service consumers.

4.4 API Versioning

As with every meaningful API, the services exposed by the ILIAS core, which are presented in the service discovery, are required to offer a solid contract to the user of the API. To achieve that, a service offers a version number which remains unchanged, even over ideally a number of ILIAS releases, as long as the consequence of calls to it and the signature remains unchanged.
It is desired to offer maintainers the possibility to have multiple versions of an API active to allow for backward compatibility to extensions that rely on it.

4.5 Limitations and Outlook

In order to make a first iteration of the service discovery usable and affordable, the following limitations are in place, that may be addressed in further releases.

• Design guidelines and principles for services and the requirements of an internal API alignment are required and a most valuable development. The TB and the developer community (core and extensions) should be involved in a formal process proposing, discussing and forming these requirements and standards, so the Service Discovery can pick up these as "givens" from the start.
• Parameters and Return values are now limited to primitives and lists of primitives.
• The automatic generation of examples as shown in the mockup may be postponed and allow for an easier reachable funding target in the first step.

5 Contact

• Author of the Request: Databay AG AG / Becker, Maximilian [mbecker] (on behalf of ILIAS JourFixe)
• Maintainer: Databay AG AG / Becker, Maximilian [mbecker]
• Implementation of the feature is done by: Databay AG AG / Becker, Maximilian [mbecker]

6 Funding

If you are interest in funding this feature, please add your name and institution to this list.

• DHBW

7 Discussion

Becker, Maximilian [mbecker], 2018-03-23: Major overhaul of the article for consideration by the JourFixe for ILIAS 5.4
A workshop-date to discuss this complex feature request will be scheduled by the beginning of next week.

Kunkel, Matthias [mkunkel], 28 JUN 2018 : The workshop announced above now takes place at July 04, 2018, 10. a.m. to 4 p.m. at Databay AG AG in Würselen.

8 Implementation

{The maintainer has to give a description of the final implementation and add screenshots if possible.}

Test Cases

Test cases completed at {date} by {user}

• {Test case number linked to Testrail} : {test case title}

Approval

Approved at {date} by {user}.