Feature Wiki

Information about planned and released features

Tabs

WAC: Redesign Web-Access-Checker

Page Overview

[Hide]

1 Requirements

The current Web Access Checker has three main problems:

  • The Web Access Checker is per Default disabled. See the following feature wiki entry: WAC: Enable Web-Access-Checker by default.
  • There is a lot of overhead in handling out files, as every request initializes a whole ILIAS environment. Leading to problems within sites with a lot of secured Links, e.g. the SCORM modules.
  • Implementing a new Module or Service into the Web Access Checker needs changes within the WAC itself.
This feature request is meant to discussion on a new implementation of the Web Access Checker.

2 Additional Information

3 Discussion

OT 22.01.2015: Unfortunately there was little time to discuss the Web Access Checker concept on the JF 19.01.2015. That's why I'd like discuss some concepts in this feature request.
The concept attached to this comment is not the same as presented on the JF, it is a different approach which would allow the protection on a file level, there was not enough time to discuss it in addition to the prototype. I would like to point out the following questions during the discussion:

  • Are there general flaws and/or concerns with the concept for a file based access (attached in to this comment)?
  • Are there general flaws and/or concerns with the concept for the folder based access (as discussed at the JF)?
  • Are there any problems in running these two concepts in parallel? Is it to complicated, is it necessary?
Please add any additional discussion points as needed.

Uwe Kohnle 2015-03-12: I have attached the concept for folder based web access checker using the .htaccess and .htpasswd files.

File Based Concept

CK 12.02.2015:

  • The implementation should only "loosely" depend on apache, because nginx is gaining popularity as a lightweight alternative. If PHP is run in FCGI-mode (either on nginx or apache) the php-function "apache_get_modules()" will not be available. So you should first check if it is safe to call this function and fallback to your sendFilePHP implementation. Since recently there is a utility class "ilEnvironment" / "ilRuntime"? Maybe one could add a sub-method ::isCGI() or ::isApacheModule() there. Currently it already implements ::isPHP() and ::isHHVM().
  • The token does not need to be saved in a database - especially if a database-connection could be avoided. It is fully sufficient to use simple cryptographic validation:
    • Just add a timestamp in the URL:ilias.de/wac.php/data/sec/pfad/zum/bild.png?accessToken=jl2397jkf8 & timestamp=123456789
    • Check the timestamp: If the timestamp is in the past, the accessToken has become invalid and the wac has to do its checks.
    • Check the accessToken: accessToken = CRYPTOHASHFUNCTION ( object-path  x  secret-salt  x  timestamp  x  session-id  x  request-ip-adress).  If the accessToken doesn't match, then someone either manipulated the token, path or timestamp or the session-id or request-ip has changed. Or the secret-salt has changed. In any case the wac has to do it's usual checks.
    • The secret-salt must be randomly created string and could be stored in the client.ini.php.
    • No database needed, because the timestamp is 'stored' in the URL. And it's still secure. :-)

OT 20.02.2015:
 
Thansk for your input! Both are very valid points.

  • The switch on whether to use an accelerated download was just a showcase in the concept. For the real implementation this decision will be a bit harder (apache vs. nginx, some old hhvm serves as webservice as well).
  • Sounds great! The secret should be per client I guess and would be stored in the client.ini?
 

AK 16 Mar 2015

  • The current web access checker is broken. This is the most important issue, see bugs #12124 and #12622. I think it is feasible to have one technical concept for the checker that really works, befor adding additional component related optimizations (like SCORM).

JF 16 Mar 2015: We appreciate this improvements and schedule it for 5.1, please

  • check your implementation against the current bugs listed above
  • please check if there are any alternatives for the SCORM optimization that does not need to write constantly into .htaccess files, that create a dependency to the currenty directory path of the installation.
  • If the dependency to the directory path can not be avoided, please provide a migration script that enables to move an installation to another server/directory.
The feature WAC: Enable Web-Access-Checker by default is included in this implementation.

CK 18 Mar 2015: I second the comment of Ralf Schenk (below) - relying on htaccess is not good at all, because the WAC should *decouple* from the concrete web-server as much as possible. Nginx is  (gaining popularity). If you ask me, Nginx should really be "the first choice" as a web server when running ILIAS on large installations (many concurrent users). The WAC is the only real obstacle right now (ok, may be ECS for university installations, too). We should step away from adding new tight couplings to apache (or any other concrete web server), as Ralf pointed out. Everything else would be a real step backwards, when redesigning the WAC.

CK 29 Mar 2015: Another Note following a discussion from the DevConf in Bern:

There are known cases, where client-side caching of static-data is a game-changer (wrt to performance). Especially learning modules can be rather large. Network bandwidth _can_ be a bottleneck. And there are cases, where learning modules are accessed multiple times from the same client (or more importantly, with the same _browser-cache_). In these cases it is very desirably to leverage client-side caching to ease network usage.

One _extremely effective_ approach - let's call it *cache extent* (inspired by google/mod_pagespeed) - is to include filehashes in filenames. E.g. instead of xyz/largevideo.mp4 the path is changed to xyz/largevideo_2fcab58712467eab.mp4 or 2fcab58712467eab/xyz/largevideo.mp4.

Why? When the client requests this file, the web-server can set an infinite cache-TTL without risking the usual side-effects. The client will keep the file in cache until the client-side heuristic will drop some cached data. Chances are good, that the file will still be cached, when needed the next time. If, however, the file changes server-side this is _not_ a problem, because the client will be told to fetch a file with a different hash (which will not be cached). All it needs is the server checking the files for changes (periodically and/or based on timestamps).

But: This approach will NOT work in any way with the URL-embedded access tokens sketched above. The access-tokens will change periodically by design - which will always annull client-side caches.

The dilemma then is: There are two good ideas which optimize different aspects - but exclude one another. URL-embedded access tokens can elegantly minimize server-side permission-checks - while URL-embedded filehashes can elegantly maximize client-side caching (which in return will reduce the total number of requests to be served by the web-server).

Ideally the WAC would be written in a way to be able to support both models in the future (maybe as runtime options). Which approach is best might depend on the concrete scenario at hand. This comment is not to include support for a cache-extent model from the beginning - but to keep the design open to explore this model in the future. :-)

Schmid, Fabian [fschmid] The Implementation is visible in the branch feature_5-1_WebAccessChecker_Redesign and at the bottom of this page.

The feature has been discussed and approved on JF 16 Mar 2015. Several questions have been asked and can be answered now:

  • check your implementation against the current bugs listed above --> both mentioned bugs can't be reproduced using the new implementation. baside the redesign of the WAC, also the MimeType-Determination has been optimized and the file delivery uses streming optimizations for video-files
  • please check if there are any alternatives for the SCORM optimization that does not need to write constantly into .htaccess files, that create a dependency to the currenty directory path of the installation. --> the new implementation provides token asses for single files of for folders (incl. subfolders). e.g. e HTML-Learning-Modul can sign the whole path where the module is stored. all requests in this folder can be delivered by the WAC without recheck the permissions (permissions are already checked when the learning module has been opened). A file-based token is valid for 3 seconds and attached to the client IP, the user session, a installation-specific SALT and the path itself. the folder-based token uses the same attributes. if one of the tokens gets invalid, the WAC uses a module-specific checking class to determine if the file can bedelivered. in case of a folder based token (which has become invalid), this token gets revalidated and is valid for 3 more seconds. therefore all new requests ro this folder within 3 seconds can be delivered without recheck the permissions. The new implementation doesn't use any dynamically generated htaccess files
  • If the dependency to the directory path can not be avoided, please provide a migration script that enables to move an installation to another server/directory. --> as already mentioned, the htaccess-based method is not used and therefore no migration script is needed
Currently there is one open point to discuss on JF:

The installation-specific SALT has to be stored on the webserver. There is already another SALT if the installations uses BCrypt for password protection. The WAC should absolutelly not use this salt but use it's own. 

As in the grafical representation of the workflow in the next chapter, the fasted way to deliver files is token based. with a token based delivery no instance of ILIAS has to be initialized, such a request is up to 5x faster than with the initialization of ILIAS. therefore the WAC should have the possibility wo access the SALT without initialize ILIAS. If the SALT is in the "Data Directory outside Webspace" a.k.a "iliasdata", at least the ilias.ini.php has to be read by the WAC to determine the path of the iliasdata-directory. This is an additional request couses some additional execution time. alternatively the salt-file could be stored in the "Data Directory in Webspace" a.k.a "data". When using a .php file with a variable-declaration in it, this is not e security-risk and additionally, bytecode compiler can store the file. both variants (A: using iliasdata-directory and read ilias.ini.php, B: using data-directory and include .php-salt-file) compared in duration:

A) 0.048282037 s
B) 0.002439153 s

both variants are feasible, variant B is faster and results in a faster delivery of the file and half the read-accesses to the storage.

--> JF: Please decide the location of the SALT-file.

JourFixe, ILIAS [jourfixe], August 17, 2015: We highly appreciate the improvement of the WAC and decided:

  • SALT should be stored per installation in the webspace ("data")
  • Client information should be included in token
Open question: who takes care of bugs related to WAC? The Jour Fixe would prefer that this is done by srag (as developer of WAC).

Killing, Alexander [alex], 25 Aug 2015: Please update the status of this feature.  Currently it is "ready for approval, not approved, no test cases". On the other hand it is commited in the trunk. Martin stated it has been approved.

4 Implementation

The new WebAccessChecker allows fast and secure delivery of files in the /data directory. As definied in the Data Directory Guideline, directories within the /sec-Folder are supported. Since there are folders outside the /sec-Folder which have to be secured, the new WebAccessChecker also secures those directories (e.g. lm_data, usr_images, mobs).

All requests to /data are now redirected to the WAC-Script per default, the .htaccess-File has a new entry:

RewriteRule ^data/.*/.*/.*$ Services/WebAccessChecker/wac.php [L]

The WAC delivers the file after the following decisions:

The most performant ways to deliver files are the file- or the folder-based tokens. These requests are already signed and and access-checked during the rendering of a page (e.g. display of poll-image, the user accesses a course and ILIAS renders a poll-image to the apge. at this moment the access is already checked). 

If there is no token or the token has become invalid, e previosly registred checking-instance has to decide whether the file can be delivered or not. if theres no checking instance, only files outside the /secore-Folder will bedelivered. if there's a checking instance and the instance declines delivery, access to the file is also denied.

Developers can sign files and folders using the ilWACSignedPath-Class:

// Example in Poll:
$img = $a_poll->getImageFullPath();
$this->tpl->setVariable("URL_IMAGE", ilWACSignedPath::signFile($img));

// Example in SCORM-Module
ilWACSignedPath::signFolderOfStartFile($this->slm->getDataDirectory().'/manifest.xml');

When registering a checking isntance, developers have to add a secured path to their service.xml or module.xml:

<web_access_checker>
    <secure_path path="ilPoll" checking-class="ilObjPollAccess" in-sec-folder='1'/>
</web_access_checker>

This entry will be registred with the next structure reload (add one if you want to register a new secured path). After the reload all requests in ./data/my_client/sec/ilPoll/* will be checked by the class ilObjPollAccess. The method canBeDelivered() receives the ilWACPath Object which proviedes several information about the requested file. Most Modules will look for the obj_id and check using ilAccess.

Delivering files is implemented using the ilFileDelivery-Service introduced in ILIAS 5. ilFileDelivery supports X-SendFile and introduces a updated ilMimeTypeUtil. Straming videos is now done chunked and allows (as previously in ilUtil) ranges. Methods in ilUtil are marked as deprecated.

If you want to use ilFileDeliver with X-SendFile please install and activate

sudo apt-get install libapache2-mod-xsendfile
sudo a2enmod xsendfile

In your Apache-Config or VHOST the "iliasdata"- and the "data"- directories must be unlocked, e.g.:

XSendFilePath /var/www
XSendFilePath /var/iliasdata

Additionally in the .htaccess the following rule activated the X-SendFile Module:

<IfModule mod_xsendfile.c>
    XSendFile On
</IfModule>

The following grafics show the performance-gain of the new implementation (token-based, fallback with checking-instance) and php vs. X-SendFile-Delivery:

Using nginx as Webserver: The new implementation allows nginx-based webservers to use the WebAccessChecker-Feature as well. Use a configuration like this:

server {
    [...]
    root /var/www/trunk;
    set $root $document_root;
    rewrite ^/data/(.*)/(.*)/(.*)$ /Services/WebAccessChecker/wac.php last;
    location /secured-data {
        alias $root/data;
        internal;
    }
    [...]
}

Test Cases

Test cases completed at 28.08.2015 by Theodor Truffer

  • http://testrail.ilias.de/index.php?/cases/view/6578 : C6578 Banner zum Blog hinzufügen und anzeigen
  • http://testrail.ilias.de/index.php?/cases/view/6581 : C6581 SCORM Modul erstellen und Bild anzeigen
  • http://testrail.ilias.de/index.php?/cases/view/6572 : C6572 Bild zum Poll hinzufügen
  • http://testrail.ilias.de/index.php?/cases/view/6573 : C6573 Bild vom Poll aufrufen
  • http://testrail.ilias.de/index.php?/cases/view/6579 : C6579 Banner zum Portfolio hinzufügen und anzeigen
  • http://testrail.ilias.de/index.php?/cases/view/6556 : C6556 Media-Objekt hochladen und anzeigen
  • http://testrail.ilias.de/index.php?/cases/view/6555 : C6555 Profilbild hochladen und anzeigen

Approval

August 27, 2015, Studer, Martin [mstuder]: Tested and approved by studer + raimann ag and iLUB Universität Bern

Last edited: 15. Dec 2021, 09:09, Schmid, Fabian [fschmid]