File Upload Service

1 Initial Problem

Currently there is no unified way to handle the uploaded files. Because of that, each ILIAS service / module handles the wole logic which leads some times to unexpected behaviours and security vulnerabilities. There are also a large amount of code duplication which makes the ILIAS code base harder to test and maintain.

2 Conceptual Summary

A new File Upload Service could serve as a bridge between the new HTTP Service and other ILIAS modules / services. The File Upload Service enables developer to move uploaded files to a specified destination.

2.1 Terminology

2.2 Interface

The File Upload Service interface provides a simple method to move an UploadedFile to a destination relative to the ILIAS storage directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

//move all uploaded files to a new location
$upload->moveFilesTo("relative/path/to/directory");
 
//returns the upload size limit in bytes.
$upload->uploadSizeLimit();
 
//register a new post processor
$upload->register(PostProcessor processor);
 
//this will invoke all registered post processors 
//throws an exception if invoked more then once.
$upload->process();
 
//returns all upload results, which contain meta data and file information
$upload->getResults();

2.2.1 PreProcessor Interface

The pre processor are called while the process method call is running. Each processor receives the data of the previous processor until all processors are done.  The stream will be rewinded for each processor. The studer + raimann ag will provide a base class which implements all methods expect the process method to ease the development process for new PreProcessors.

1
2
3
4
5
6

interface PreProcessor {
 
//process the current request
public function process(resource $stream, Metadata $metadata)  : void;
 
}

Metadata

The metadata holds all information about the upload. Furthermore all processors can add additional information with the help of the put method. All setter methods return a self reference to enable the develpers to chain their method calls.

It's possible to interrupt the filterchain with the help of the status code which can be set via setStatus. If the upload status contains the code REJECTED, there will be no further processing attempts for the current processor chain, because the upload is not valid anymore.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

class Metadata {
 
//returns the current filename
public function getFilename() : string;
 
//overwrite the current filename with a new one.
public function setFilename($filename) : Metadata;
 
//get the current upload status described in the UploadStatus interface.
public function getStatus() : UploadStatus;
 
//get the current upload status described in the UploadStatus interface.
public function setStatus(UploadStatus $status) : Metadata;
 
//get the actual file size (this value always represents the initial size of the file)
public function getSize() : int;
 
//get the mime type of the file
public function getMimeType() : string;
 
//overwrite the mime type of the file
public function setMimeType($type) : Metadata;
 
//get additional meta information
public function get(string $key) : string;
 
//get all additional meta information
public function getAll() : array;
 
//checks if a key exists (null is not a valid key)
public function has(string $key) : string;
 
//puts a new value into the meta data (old values can not be overwritten)
public function put(string $key, string $value) : Metadata;
 
}

UploadStatus

The upload status indicates if a file upload succeeded or got rejected by a processor. This class is immutable to encourage processor developers to set a message along with the code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

class UploadStatus {
 
//upload is ok
const OK = 1;
 
//upload got rejected by a processor
const REJECTED = 2;
 
public __construct(int $code, string $message);
 
//get the current status code
public function status() : int;
 
//get the current status message
public function message() : string;
 
}

2.2.2 UploadResult

The upload results can be fetched via the file upload service, after the processors are run and the files are moved.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

class UploadResult {
 
//the filename
public function name() : string {}
 
//filesize
public function size() : int {}
 
//mime type
public function mimeType() : string {}
 
//meta data set by the processors
public function metaData() : array {}
 
//upload status
public function status() : int {}
 
//status message
public function statusMessage() : string {}
 
//the file path to the moved files.
public function path() : string {}
}

2.3 PreProcessors delivered with the Service

The studer + raimann ag creates the following pre processors to ease the integration process of the Upload File Service.
The following processors are delivered with the Service:

• AntivirusPreProcessor
• WhitelistExtentionPreProcessor
• WhitelistFileHeaderPreProcessor
• WhitelistMimeTypePreProcessor
• BlacklistExtentionPreProcessor
• BlacklistFileHeaderPreProcessor
• BlacklistMimeTypePreProcessor

2.3.1 AntivirusPreProcessor

The Antivirus pre processor uses the new Revision Virus Scanner or the old ilUtil virushandling.

2.3.2 WhitelistExtentionPreProcessor

The whitelist extension preprocessor rejects all files which don't end with the whitelisted extensions.

2.3.3 WhitelistFileHeaderPreProcessor

The whitelist file header preprocessor rejects the upload, provided that the stream don't start with the whitelisted entries.

2.3.4 WhitelistMimeTypePreProcessor

The whitelist mime type preprocessor reject the upload, unless the mime type of the upload is found in the whitelist.

2.3.5 BlacklistExtentionPreProcessor

The blacklist extention preprocessor rejects the upload if the extention matches with an entry in the blacklist.

2.3.6 BlacklistFileHeaderPreProcessor

The blacklist file header preprocessor rejects the upload, if the file stream starts with the blacklisted file starts.

2.3.7 BlacklistMimeTypePreProcessor

The blacklist mime type preprocessor rejects the upload, due to matches with the upload mime types and the mime type blacklist.

2.4 Additional PreProcessor implementations

There are several PreProcessor implementations which could be done in a future release:

• SvgSanizingPreProcessor
• FilenameNormalizingPreProcessor

2.4.1 SvgSanizingPreProcessor

The SVG sanizing preprocessor could reject SVG uploads, which contain javascript.

2.4.2 FilenameNormalizingPreProcessor

The filename normalization preprocessor normalizes filenames with the help of a blacklist or / and php encoding functions.

2.5 ILIAS DI Integration

To enable developer to use the new File Upload Service we need to introduce a new key to the DIC named "upload". 

1
2

//direct access to the move function via upload key
$DIC["upload"]->moveFilesTo("your/directory/");

2.6 ilUtil method deprication

2.7 Old FileUpload Service deprication

It exists an old Service called FileUpload which serves as a dropzone GUI element. This service will be refactored by the studer + raimann ag into a new UI element.

2.8 Dicto

We propose to create Dicto rules to aid the refactoring process. The Dicto rules should contain the deprecated ilUtil functions. Furthermore we want to add a rule which enshures that only the File Upload Service make use of the HTTP Service UploadedFile instances, which abstract an uploaded file, because the moveTo method is not jet compatible with the Filesystem Service.

3 User Interface Modifications

No UI Modifications needed.

4 Technical Information

The technical information is already described in "Conceptual Summary".

As a maintainer i highly support this FR. From the security-POV we definitely should centralize file-uploads sinc many of security-bugs are file-upload-related.

5 Contact

• Author of the Request:  (studer + raimann ag)
• Maintainer: Schmid, Fabian [fschmid]
• Implementation of the feature is done by: Schmid, Fabian [fschmid]

6 Funding

If you are interest in funding this feature, please add your name and institution to this list.

• Universität Bern
• sr.solutions

7 Discussion

Amstutz, Timon [amstutz], 4. Mai, 2017: Thank you very much for this request. I think we definitely need this to centralize the handling of file uploads.

Schmid, Fabian [fschmid], May 8, 2017: As the Maintainer a highly support this FR.

Killing, Alexander [alex], 8 May 2017: Just a note: I think lots of these things, especially the preprocessors will be needed in the unzip process as well.

JourFixe, ILIAS [jourfixe], May 08, 2017: We highly appreciate this suggestion and schedule it for 5.2. Please provide a pull request so we can discuss the final implementation, e.g. if some methods should be moved to the file system service as they are not only needed when uploading files.

8 Implementation

This features is mainly dedicated for developers, the relevant information on the interface is available here: https://github.com/ILIAS-eLearning/ILIAS/blob/release_5-3/src/FileUpload/FileUpload.php

The methods provided by ilUtil have been marked as deprecated and use the new upload service internally.

An example how to use the new FileUploadService is described here:

1
2
3
4
5
6
7

global $DIC;
 
		$upload = $DIC->upload();
		if ($upload->hasUploads() && !$upload->hasBeenProcessed()) {
			$upload->process();
			$this->moveUploadedFile($upload);
		}

A full description of the service can be found here: https://github.com/ILIAS-eLearning/ILIAS/blob/release_5-3/src/FileUpload/README.md

Test Cases

FileUpload-Service is a core-service which many Modules and Features rely on. Testing is done implicitly by many Testcases.

Approval

Approved at 1. Juli 2017 by Amstutz, Timon [amstutz].