Show Advanced KnowledgeHide Advanced Knowledge

Introduction of Individual Assessment

1 Requirements

Until now it is not possible in ILIAS to integrate a personal examination of a learner as a course or group element and use it for determining the learning progress. But in practice such a alternative test is necessary, helpful - and avoids strange workarounds.

Example: User A was not able to take the online test T in course X (no matter why). Now T is one module that needs to be passed to pass the entire course. Because the test has already been taken by the rest of the course participants and result were presented, A cannot run the test. And professor P is not willing to create a new test with new questions a.s.o for A. In the classroom world P would just invite A for an oral examination and decides afterwards if A has passed or failed.

Characteristics

Features: The following options and fields are necessary for this new repository object.

1. Title and Description
2. Examinee : person who is tested (on PE object could contain several examinees; a separate record is created for each examinee).
3. Tester : person who is testing the examinee (usually but not necessarily one of the course admins or tutors)
4. Date of examination : time and date (event should appear in personal calendar of examinee and tester)
5. Location of examination : text (support of map would be nice but is not conditio sine qua non.
6. Test record : text area to describe the purpose and subjects of this examination (simple RTE might be the best format because it allows simple lists, strong and emph)
7. Grading : selection list with values passed | failed | not graded ( = default when object is created )
8. Feedback : text area to describe result and score of the examination
9. Internal note : text area for tester to make notes that should not be visible for examinee.
10. Option "Notify examinee" : Gives examinee access to test record, grading and feedback; sends a notification to examinee.

Learning Progress support: Trigger for the learning progress status of a personal examination is the value of grading:

• Grading = none => Learning progress = not started yet
• Grading = passed => Learning progress = completed
• Grading = failed => Learning progress = failed

There is no need for an "in progress" status.

Access / RBAC: The following RBAC permissions are necessary to use this object:

• Visible : object appears in ListGUI
• Read : user can read content of entries 1 - 5
• Edit settings (and content) : user can read and edit all values / entries
• Copy : user can copy object with all information
• Delete : user can move and delete object
• Change permissions : user can change permissions

Unlike other modules in ILIAS, this feature is very much user-related. Therefore, a role permission READ should not give access to personal information like the personal test record, grading or comment.

2 Additional Information

• Idea / concept: Matthias Kunkel
• Interest in funding: ILIAS-Verein
• Maintainer: (new object type - no maintainer defined yet)
• Implementation of the feature is done by Klees, Richard [rklees]
• Tested by / status: (name, e-mail), (status information set after implementation)

3 Discussion

Matthias Kunkel, 14 Mar 2014 : How we handle the access to test record, grading and comment for the single learner? Does it make sense to create a proper role for each examinee just for assigning the necessary permissions to read test record, grading and comment? Or do we handle this in a different way? Usually, among the course or group members only the examinee should even see this personal examination, so the member roles would not even get VISIBLE and READ permissions by default.

Volker Reuschenbach: Aus meiner universitären Sicht kann ich den Entwicklungsvorschlag nur unterstützen. In vielen Prüfungsordnungen sind "mündliche" Nachprüfungen vorgesehen. Würde dieses traditionelle Prüfungselement nicht oder nur umständlich in einer primären Online-Prüfung verfahrensmäßig abgebildet werden können, dürfte das auch einem ILIAS-Ausschlusskriterium für das Prüfungsamt eine Fachbereiches nahe kommen.

Unabhängig davon kann man durch die verfahrensmäßige Integration traditioneller Prüfungsabschnitte in Rahmen einer Online-Prüfung, die Akzeptanz letzterer mit Sicherheit fördern. Es ist doch gerade die Mischung aus Präsenz- und Online-Elementen, die dem E-Learning zum Durchbruch verholfen hat.

We had an effective meeting with Matthias today. Obejct was to discuss and refine this feature request. I'm going to edit this request according to the results.

CaT Concepts and Training GmbH is interested in taking the maintainership.

AT, 2016-06-04:

Main label issue
"Personal examination" has medical connotation. If this is to serve "Oral Exam" it should be called just that.  
In case this is too narrow a label, you might consider "Compensational Assessment" or "Subsitutional Assessment". 

Other label issues
Examinee : person who is tested  
It is not neccessarily a test, in it not? Why not call this person a participant. 

Tester : person who is testing the examinee (usually but not necessarily one of the course admins or tutors)
In English this person is an "exam administrator". It is: to administer an exam. 

Test record : text area to describe the purpose and subjects of this examination (simple RTE might be the best format because it allows simple lists, strong and emph)
This is not neccessariyl a test and thus should not be labeled as such, try Assessment Description or Exam Description 

Wegener, Miriam [miriamwegener] 13 Jul 2016:
As we are thinking of a new function where all the grades are summarized (see http://www.ilias.de/docu/goto_docu_wiki_wpage_2632_1357.html), we appreciate this feature request. We also thought about an option to fill in grades manually for things that users do „offline“ - e.g. oral exams, oral participation or other achievements that are not represented in other items.
Therefore, I would suggest to broaden this feature request:

Instead of calling it personal examination I would prefer something like “offline assessment” (the word offline implies that it is an achievement that users do not do online (in ILIAS)).
Date of examination should be optional (it is not useful to set a date if you want to grade oral participation).
It should be possible to add all course/group members (as in the exercise).
It should be possible to fill in points and set up a high score.

In a further step it would also be nice to add a mark scheme. Grading could then be set automatically if you fill in the points.

JourFixe, ILIAS [jourfixe], July 25, 2016: We highly appreciate this suggestion and schedule it for 5.2. We would like to have the following modifications of the object:

• Permissions:
  • To streamline this object with other objects with Learning Progress, a dedicated permission "Edit Learning Progress" should be introduced as well as an option to disable/enable LP.
  • The permission currently called "View Grades" should be extended to "View Grades and Learning Progress"
• Fields:
  • "Location of Examination" shall be removed to avoid overlaps with the session object.
• Wording:
  • We are still looking for a good title for the object. Matthias and Richard will work out a suggestion.
• Ideas for future extensions:
  • Connecting the feature with the competence management.
  • Improving the process to add members from the parent course (based on a general service)

Kiegel, Colin [kiegel] 2016-07-26: I prefer assessment over examination or evaluation. And I also prefer manual over personal or substitutional, etc. My pick would be manual assessment.

Kunkel, Matthias [mkunkel], July 26, 2016: The concept of the "manual assessment" (if we follow Colin's suggestion) aka "personal examination" comes with an "Info" page like other repository objects. But for the last two years we are discussing where we really need an "Info" page and where it is superfluous, see Revision of Info Page.

• Is there any additional information presented on the "Info" screen that justify the implementation of the "Info" screen? Would "Content" be an information that should be presented there - and is this a general information for the entire object (and not for one selected examinee)?
• And another idea: shouldn't we offer an input "Contact" on the Settings screen? It might be necessary for an examinee to contact his tester / tutor. Such an input would clarify who should be contacted. And this "Contact" could be presented on the "Info" screen as well.

Klees, Richard [rklees], 2016-08-01: I would also pick  manual assessment. I would also implement "Info" to display the Content and the testers, as I consider this information usefull for the participants. There also won't be anything left that a participant could view before he got tested if there were no Info screen. If anyone comes up with another name for that screen, that would be ok for me as well. I also support "contact", which imo should contain the fields "Name", "Telefon" and "Mail". 

JourFixe, ILIAS [jourfixe], August 01, 2016: We rename the feature to 'Manual Assessment' and support Matthias' idea to add contact information (name, telephone, mail) to the 'Info' page.

25. Aug 2016, Heyser, Björn [bheyser]:

Perhaps I will be send to hell for this suggestion, but I strongly support the Info Tab. Even when it shows very common information only. It gives me the feeling I know that there is nothing else to know. Most of other objects has important information presented there.

Perhaps i am the very confused generation, that can handle a smartphone but was registered in the social network "outside" for playing games like catching and hiding. Indeed I very often confuse about online platforms because i expect some areas I cannot find. I than have the feeling of not beeing informed. It needed a long time i started to navigate to things by typing in on facebook for example, i suddenly noticed  platform ares that are fully new to me.

For me an info screen showing no important information just shows me there is no important information, otherwise it would be presented here.
(Yes it sounds like waste of tab again, but indeed this IS an important information for me)

Regarding the discussion about labels: I support Alexandras Ideas, when we handle oral exams, we should call it like this. Manual Assessemnt involves Paper Exams that are written by a couple of people not beeing able to control a pc for example (dont know for what reasons). Another example is the manual scoring of any external project work. This all are Manual Assments in an abstract way to me.

Depending in the vision of the object it should keep this name "manual assessment" (vision of handling all kind of manual assessments) or get strongly another one (in case of oral exams only).

11 Sep. 2016: Bromberger, Norbert [bromberger] 
After the presentation at the ILIAS-Conference I suggest to rename the feature. We shouldn't use name in the software because of the technical handling.
My proposition: Individual Assessment.

JourFixe, ILIAS [jourfixe], September 26, 2016: We highly appreciate the suggestion to rename the object and call it "Individual Assessment". Richard will try to rename the service component ID to 'inas'. Page and feature category in the FW will be renamed, too.

4 Implementation

The object is now implemented in the trunk. It may be located in the repository or within a course, where it may be used for learning progress calculation within a group of LP-objects (such as test or SCORM-courses) or stand-alone.

The object has a learning progress, that may be disabled. Once it is enabled the learning progress of users, which may be booked onto a Manual Assessment via repository-search, can be set by an admin, i.e. another user with the appropriate permission. The learning progress, and information associated with it, of a user may be saved, leaving it open for later adjustment, or finalized, fixing it for years to come. Once a learning progress is finalized, the corresponding member can not be removed from the manual assessment anymore.

Permissions are controlled via RBAC. As with other objects one may assign permissions to role templates (e.g. course related roles), which will be used to define permissions in newly created objects. Also any examinee will recieve a member role on the manual assessment object.

Before finalization the examiner may choose to notify the member, in which case he will recieve a notification and may see his learning progress/status in the info-screen section of the manual assessment.

The manual assessment may also be used to control/limit/enable access to other objects within a course, depending on the learning progress/status.

Finally, as other repository objects, the manual assessment may be move/copied within the repository.
Export functionality is to be implemented in the near future.

Also still a warning is missing, if a user may not access parent objects.

Test Cases

Please refer to test suite #217 in testrail for further information about existing test cases.

Approval

Approved at August 23, 2016 by Kunkel, Matthias [mkunkel]