(Project) Individual Assessment: Use in High Volume Scenarios

1 Aim of Project

The Individual Assessment Object of ILIAS provides a way to record individual examinations, such as oral exams, and integrate them into larger and digital structures like Courses via standard mechanisms such as RBAC and Learning Progress. In its current state, however, the object is mostly suitable for low traffic scenarios. Participants have to be added manually. The written exam grade is to be given as a freely structured text. Users with appropriate permissions can view individual results and records, but there is no way to get an overview of the collected results.

The aim of this project is to enhance Individual Assessment with functionality to make it more useful and usable in high traffic scenarios. One such scenario is described in the section "Background: Assessments by Tutors" below, but we expect similar scenarios to exist in other organisations. Although on the surface it might look as if there are already other ILIAS objects to accommodate these scenarios, we demonstrate in "Background: Alternatives" that this is not the case.

1.1 Background

1.1.1 Assessment by Tutors

The idea of improving Individual Assessment for use in high volume scenarios arose during an investigation of a scenario from LVM Versicherung: For certain programmes, especially in vocational training of their sales and distribution departments, they implement a process called "Assessment by Tutors". The programme lasts for 3 years and is interspersed with part-time courses in the company’s agencies. The trainees in the programme are accompanied by their direct supervisors in the agency, as well as by senior managers in the company and by the tutors. The on-site courses are intensive all-day events where the trainees learn different aspects of the business. They receive detailed feedback on their current progress after the training events. The assessment forms filled out by the trainers contain fields and questions that are specifically tailored to align with the content of the corresponding event. Once filled out, the forms will be provided to the trainees, as well as to their direct supervisors and their higher supervisors.

The scenario is currently implemented via a custom software solution. However, LVM is exploring the possibility of migrating the scenario to its ILIAS-based LMS, LVM AkadeMe, for various reasons:

• The LVM AkadeMe already contains the necessary data about people and organisational structure to accurately represent and express the required relations and permissions via the existing Organisational Units and Positions.
• The events are already managed via the LVM AkadeMe, so the relevant objects and assignments already exist.
• The existing data provides a solid foundation for eliminating some manual processes and streamlining the overall workflow, thereby achieving greater efficiency gains.
• The scenario is an ideal fit for an LMS, as it is an integral part of the learning and education process, and the various participants in the scenario already utilise the LVM AkadeMe for other purposes.

During the investigation of the scenario, it became evident that comparable scenarios exist in other institutions, and the rationale for integrating them into an existing LMS solution is analogous. Consequently, the LVM and CaT Concepts and Training GmbH explored the possibility of implementing the scenario using standard ILIAS, and ultimately identified Individual Assessment as a suitable provider for a solution.

1.1.2 Why Individual Assessment?

From the object’s name alone, it is evident that there are similarities with the "Assessment by Tutors" scenario. Furthermore, the object’s intent and workflows appear to align with the target scenario. An examiner assesses the performance of an individual and records their findings, which may be shared the participant and relevant parties. Furthermore, the notion that the assessment is structured and a pre-fabricated response is possible is hinted at in the properties "Assessment content description" and "Record template". However, the Individual Assessment in its current form is not equipped to handle the aforementioned scenario, as evidenced by the comprehensive list of changes and features that will be necessary to implement the target scenario.

Conversely, the existing Individual Assessment is a relatively basic object that has not been developed to the same extent as other objects. This provides scope for enhancing the Individual Assessment with the necessary functionality.

1.1.3 Non-Alternative: 360°-Feedback

It is possible that other objects or features in ILIAS that could accommodate the outlined scenario, at least in part. Following an initial investigation of LVM and CaT, it appeared that the 360°-Feedback feature might be a suitable option for implementing the intended scenario. However, upon closer examination, it became evident that certain design decisions within the feature made it unsuitable for the in question. These mismatches provide valuable insights into the potential enhancements that could be made to the Individual Assessment feature.

Therefore, it is beneficial to document these observations for future reference:

• While for target scenario primarily utilises written feedback, the 360°-Feedback appears to focus on extracting numerical values that are not directly relevant to the target scenario. This is particularly evident in the presentation of results, where the 360°-Feedback displays the numerical values initially and only provides the textual results in the form of a secondary results page. Consequently, for the target scenario, the valuable feedback is hidden behind a layer that does not actually exist in the scenario.
• The same design decision to target numerical values as results is evident in the absence of a question type for free text feedback
• In the target scenario, each event and participant is to receive one piece of feedback. Even when there are multiple tutors present at one event, they will collaborate to provide a unified feedback statement for the trainee. The 360°-Feedback, on the other hand, is designed to gather results from various individuals to gain diverse perspectives.
• In the target scenario, the majority of feedback is provided by a single tutor to all participants of a specific event. However, for the 360°-Feedback, feedback givers must be manually assigned per participant, which is more suited to an actual 360°-Feedback process but not aligned with the requirements of the target scenario.

It can be seen that the 360°-Feedback is not entirely aligned with the target scenario. While there is some overlap, there are also significant differences. It may be possible to adapt the 360°-Feedback to align with the target scenario, but it seems to be designed for a different purpose, making it an imperfect fit.

2 Involved Authorities and Stakeholders

• LVM-Versicherung
• CaT Concepts and Training GmbH
• Authorities of Individual Assessment

3 Timeline

4 Related Feature Requests and Status

5 Further Results

• …

6 Additional Information

…

7 General Discussion