Filesystem Service

1 Initial Problem

Currently there are several ways to access the local filesystem within ILIAS. Because of the different access method there is lot of space for security issues. A good example is the import functionality provided by various ILIAS modules. Furthermore there is currently no way to extend ILIAS to access other filesystems like an Amazone S3 Bucket.

1.1 Security

The ILIAS 5.2 import functionality havy relies on XML which contains relative filepaths, which are concatinated while the import is running. Each modul has its own import logic which copy the files directly with the help of build in php functions or the help of the utility class ilUtil. Therefore each module needs to sanitize every concatenated path which leads to an insecure import.

The table below shows an incomplete list of file operations done by ILIAS, which are roughly 500 in total. The data below were collected with the help of dicto.

1.1.1 Example

ILIAS 5.2 had a bug which allowed a user with rights to import a FileObject to move files to arbitrary locations. This bug was possible because the destination and source path was not validated at all. Because of that, it was possible to move files out of the ILIAS webroot (denial of service, Line 6) or copy files into the webroot (remote code execution, Line 10).

The paths in the example below are only for illustration and point to invalid locations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

<?xml version="1.0" encoding="utf-8"?>
<!--Generated by ILIAS XmlWriter-->
<exp:Export xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:exp="http://www.ilias.de/Services/Export/exp/4_1" xmlns="http://www.ilias.de/Modules/File/file/4_1" InstallationId="0" InstallationUrl="http://172.28.128.7" Entity="file" SchemaVersion="4.1.0" TargetRelease="5.2.0" xsi:schemaLocation="http://www.ilias.de/Services/Export/exp/4_1 http://172.28.128.7/xml/ilias_export_4_1.xsd http://www.ilias.de/Modules/File/file/4_1 http://172.28.128.7/xml/ilias_file_4_1.xsd">
    <exp:ExportItem Id="476">
        <File obj_id="il_0_file_476" version="1" size="215" type="text/html">
            <Filename>../../../../poc.php</Filename>
            <Title>poc.php</Title>
            <Description/>
            <Rating>0</Rating>
            <Content mode="COPY">../../../../Modules/File/set_1/expDir_1/poc.php</Content>
            <Versions>
                <Version id="1" date="1488790457" usr_id="il_0_usr_6"/>
            </Versions>
        </File>
    </exp:ExportItem>
</exp:Export>

2 Conceptual Summary

To eleminate security issues like path traversal, we would like to introduce a new Filesystem Service which streamlines the filesystem access for ILIAS. The service provides a modular way for extension, which enables the ILIAS community to seamless extend the service with additional supported filesystem types.

There are four directories which should be accessed via the new service:

• Data directory within the ILIAS webroot
• ILIAS data directory
• Customizing directory
• Temporary directory

2.1 Interface

The interface provides the following functionality

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

//Write Files
$filesystem->write('path/to/file.txt', 'contents');
 
//Update Files
$filesystem->update('path/to/file.txt', 'new contents');
 
//Write or Update Files
$filesystem->put('path/to/file.txt', 'contents');
 
//Read Files
$contents = $filesystem->read('path/to/file.txt');
 
//Check if a file exists
$exists = $filesystem->has('path/to/file.txt');
//NOTE: This only has consistent behaviour for files, not directories.
 
//Delete Files
$filesystem->delete('path/to/file.txt');
 
//Read and Delete
$contents = $filesystem->readAndDelete('path/to/file.txt');
 
//Rename Files
$filesystem->rename('filename.txt', 'newname.txt');
 
//Copy Files
$filesystem->copy('filename.txt', 'duplicate.txt');
 
//Get Mimetypes
$mimetype = $filesystem->getMimetype('path/to/file.txt');
 
//Get Timestamps
$timestamp = $filesystem->getTimestamp('path/to/file.txt');
 
//Get File Sizes
$size = $filesystem->getSize('path/to/file.txt');
 
//Create Directories
$filesystem->createDir('path/to/nested/directory');
 
//Directories are also made implicitly when writing to a deeper path
$filesystem->write('path/to/file.txt', 'contents');
 
//Delete Directories
$filesystem->deleteDir('path/to/directory');
//The above method will delete directories recursively
 
//NOTE: All paths used by Filesystem service are relative to the adapter root directory.
 
//List Content
$contents = $filesystem->listContents();
foreach ($contents as $object) {
echo $object['basename'].' is located at'.$object['path'].' and is a '.$object['type'];
}
 
//By default Flysystem lists the top directory non-recursively. You can supply a directory name and recursive boolean to get more precise results
$contents = $filesystem->listContents('some/dir', true);
 
//get / set visibility if the underlying filesystem supports the concept.
$filesystem->setVisibility('data/dir/example.txt', 'public');
$filesystem->getVisibility('data/dir/example.txt');
 
/*
 * Stream interface
 * 
 * Please always check if the stream got released before calling fclose() because some of the
 * internal filesystem adapter could release the resource.
 */
 
//write to a file via php stream
$stream = fopen('/path/to/database.backup', 'r+');
$filesystem->writeStream('backups/'.strftime('%G-%m-%d').'.backup', $stream);
 
// Using write you can also directly set the visibility
$filesystem->writeStream('backups/'.strftime('%G-%m-%d').'.backup', $stream, [
    'visibility' => AdapterInterface::VISIBILITY_PRIVATE
]);
 
if (is_resource($stream)) {
    fclose($stream);
}
 
// Or update a file with stream contents
$filesystem->updateStream('backups/'.strftime('%G-%m-%d').'.backup', $stream);
 
// Retrieve a read-stream
$stream = $filesystem->readStream('something/is/here.ext');
$contents = stream_get_contents($stream);
fclose($stream);
 
// Create or overwrite using a stream.
$filesystem->putStream('somewhere/here.txt', $putStream);
 
if (is_resource($putStream)) {
    fclose($putStream);
}

2.1.1 Filesystem access right management

The Filesystem Service interface shows a way to change the access right of a file or folder.
But is it really needed to manage the access rights? This point needs some further clarifications.

2.2 ILIAS DI Integration

To use the new filesystem service we need to introduce a new key to the DIC named "filesystem".
It's possible to access the 4 storage locations via the methods described bellow. Each of the 4 Methods return a filesystem object which satiffies the interface described in the section 2.1.

1
2
3
4
5
6
7
8

//new filesystem service key
$DIC["filesystem"]
 
//access the 4 predefined storage locations
$DIC["filesystem"]->web();                //Data directory within the ILIAS web root
$DIC["filesystem"]->storage();            //ILIAS data directory
$DIC["filesystem"]->customizing();        //The Customizing directory within the ILIAS web root
$DIC["filesystem"]->temp();               //Temporary directory

2.3 ilUtil method deprication

The following ilUtil methods should be depreciated with the new filesystem service.

2.4 Dicto

We propose to create dicto rules to aid the refactoring process. The new dicto rules should report direct access of php filesystem functions and the listed ilUtil methods described in section 2.3.

The table below contains all php function which should no longer be used.

3 User Interface Modifications

4 Technical Information

The technical information is arleady described in "Conceptual Summary". The FileSystemService comes with a great benefit for ILIAS in general. The migration will take some time but from the security-POV this should be done.

5 Contact

• Author of the Request:  (studer+raimann ag)
• Maintainer: Schmid, Fabian [fschmid]
• Implementation of the feature is done by: Schmid, Fabian [fschmid]

6 Funding

If you are interest in funding this feature, please add your name and institution to this list.

• sr.solutions

7 Discussion

Kunkel, Matthias [mkunkel], May 02, 2017: I added this suggestion to the next Jour Fixe agenda. Just one question: does this suggestion also include to change all current implementations of local file system accesses in all components? Or is it a concept to be used for new implemented features that need file system access?

Amstutz, Timon [amstutz], May 04, 2017: I like this and look forward to the PR describing the complete interface of this new addition to the src folder. We definitely need such a library. One feature I would strongly recommend is to add some xcopy function to recursively copy the content of a complete folder. 

Schmid, Fabian [fschmid], May 8, 2017: @Matthias: Since there are many accesses to the FileSystem, the conversion would probably take place over two releases, unless sufficient funding is available. For the file objects, we would adjust the necessary places.

@Timon: Thanks for the good note with xcopy, we will add thisd to the interface.

As the Maintainer a highly support this FR.

Klees, Richard [rklees], May 8, 2017: I really like this.

Killing, Alexander [alex], 8 May 2017: It will be great to have something like this. Some points:

• $filesystem->setVisibility('data/dir/example.txt', 'public'); Does what exactly? "public" seems to be some magic string we usually would like to avoid.
• Since the feature already "has knowledge" of the web data and "storage" data directories, I think it should take care of the components subdirectories, too. The naming of these directories still suffers from inconsistencies. Also the ID to subdirectory mapping should imo be provided by the service. So basically I think it should provide enough functionality that it can replace ilFileSystemStorage (under Services/File) completely. It would be strange to have these things being split up in two services.

JourFixe, ILIAS [jourfixe], May 08, 2017: We highly appreciate this suggestion and schedule it for 5.3. We would like to have a pull request for the file system interface and a strategy how to migrate existing ilFileSystemStorage related code

• Roadmap for 5.3: implementation of new file system service, setting file access methods in ilUtil to deprecated (2.3), implement Dicto rules
• Roadmap for 5.4: complete substitution of ilUtil and native PHP function

8 Implementation

This features is mainly dedicated for developers, the relevant information on the interface is available here: https://github.com/ILIAS-eLearning/ILIAS/blob/release_5-3/src/Filesystem/FilesystemFacade.php

The methods provided by ilUtil have been marked as deprecated and use the new service internally.

An example how to use the new FileSystemService is described here:

1
2
3
4
5
6
7
8

global $DIC;
 
		$fs = $DIC->filesystem()->storage();
 
		if($fs->has($this->getMetadataPath($idpId)))
		{
			$fs->delete($this->getMetadataPath($idpId));
		}

A full description of the service can be found here: https://github.com/ILIAS-eLearning/ILIAS/blob/release_5-3/src/Filesystem/README.md

Test Cases

FileSystem-Service is a core-service which many Modules and Features rely on. Testing is done implicitly by many Testcases.

Approval

Approved at 28.8.2017 by Amstutz, Timon [amstutz].