Feature Wiki

Information about planned and released features

Tabs

Revision of ILIAS Test Question Creation Process

Page Overview

[Hide]

1 Initial Problem

Currently, for every one of the 15 question types in the ILIAS core, the creation works differently in sometimes very subtle ways. This is the product of the long history of the Test & Assessment. Some questions types are older than others. Features have been added over time and changing paradigms have never been backported to existing questions.

This culminates in many problems:

  • Not every form is reachable by keyboard.
  • The inputs must be sanitized individually and often this goes wrong (I look at you --> Cloze Question).
  • We have 3 different editor options: Unformatted input field, TinyMCE and IPE.
  • We have different ways to integrate pictures into options for answering. * aso.

Maintaining this is nearly impossible.

Additionally, the coupling of the Test- and the TestQuestionPool-Components and the questions themselves is far to strong, requiring changes in many places for every improvement and creating doubt about the places where the actual "source of truth" is. This makes it currently impossible to implement specialised Components for different use cases, e.g. a Quiz-Component for formative assessments or an Exams-Component to only run exams, leading to a Test-Component that is over complex and conducive to configuration- and other errors.

Further, there are many feature requests that have been around for a long time and would implement understandable needs, but cannot be implemented as implementation would be very complicated and could only be done in a way that would further weaken the stability and integrity of the system.

Finally all forms in the test are still based on the unmaintained Legacy-Forms and are getting harder and harder to keep in synch with the look and feel of the rest of ILIAS. We won't be able to change this in one fell swoop, but we would like to start this process by moving the creation- and edit-forms of the questions to UI-Forms.

1.1 Timeline

It is our hope that we are able to migrate all question types with ILIAS 12 and that we then can start using the new questions in the Test, the Test Question Pool, and the Learning Modul in ILIAS 13. This is rather optimistic planning though and it is very possible that we will need an additional version to get things done.

2 Conceptual Summary

2.1 Objectives

In general the proposed changes aim at future-proofing the questions. To achieve this it the following objectives are pursued:

  • Simplifying the handling of user input when creating/editing questions. This will allow us to be sure about the type of input we receive and how to present it to the user.
  • Decoupling the three different aspects of questions in a test:
    • The question itself.
    • The place to manage questions implemented in the component "TestQuestionPool".
    • The tool to present the questions to participants implemented in the component "Test".
  • Reducing redundant code.
  • Providing an infrastructure for questions that allows to add additional capabilities to questions, so that currently proposed features can be implemented easily and in a structured way.
  • Providing a clear-cut way to add question types outside of the ILIAS-Core.

2.2 Guiding Concepts

We are trying to provide a concept that is about "questions" in general not about "questions for assessment" and we believe it is possible to do so.

The ILIAS-Community has spent a lot of time discussing the question what a question actually is. We stipulate that this question is actually not very relevant, as a question is just a message that solicits an answer and as such isn't very interesting in our context. Additionally, on a more technical level, the question is independent of the way to answer it. We propose to put the answering of the question in the center of our attention.

We thus stipulate that the question is just a way to provide stimuli to a user so that she can react to them. We use the singular "question" for convenience and clarity, but stipulate that a question can be more complex and consist of multiple stimuli and multiple opportunities for the user to react to them. This is already the case today, e.g. in the cloze-question or on a more technical level in the KPrim question and will be expanded through the proposed changes. We conclude that the presentation of the question profits from a rich environment where different types of stimuli can be provided and it is mostly the responsibility of the creator to make this input clear and concise, i.e. to manage the extraneous cognitive load of the participant.[1] The ILIAS Page Editor is the logical tool in ILIAS to provide an interface for creating this stimulus.

We thus focus our attention on the options for answering a question. These options need to be created, presented, and handed in. The answer handed in needs to be processed, persisted, and it might trigger a reaction like showing a feedback or proposing content for further learning. The answer a participants tries to hand in might also be unacceptable and thus need an instant reaction, even before it is processed. From a technical perspective we are always dealing with form-inputs, the creation of them, the presentation of them, and the reaction to their submission. Additionally, many of these actions may need to be processed in the client of the participant without being able to rely on the availability of an internet connection.

We do not believe that all types of answer form need to or even can provide all the capabilities. One type might not offer a way to calculate points as it is only relevant in surveys, another one is unable to run without an internet connection and thus cannot be answered purely in the client of the participant.

2.3 Structural Changes

  • Questions are moved to their own component called "Questions". The questions in this component are modular, i.e. they can have points for answers or not, they can have feedbacks or not. They will in this way be designed to make them usable across multiple scenarios.
  • The available types of answer forms will not be configurable on a system level. If a consumer wants to make the available types of answer forms configurable, as is the case for the test, it needs to implement its own configuration interface.
  • The units will be moved to component "Questions".
  • All code concerning the import and export of questions will be remove from the component "QTI". The aim is to also remove the rest of the code, but this depends on further changes in the component "Test".
  • The question types "Cloze", "Longmenu", "Numeric", and "TextSubset" will be merged to one answer form type.
  • The question types "Multiple Choice (Single Answer)" and "Multiple Choice (Multiple Answers)" will be merged into one answer form type.
  • The question types "Ordering Horizontal" and "Ordering Vertical" will be merged into one answer form type.
  • The administration of skills in questions is moved to the component Questions and linked to the editing of the answer forms.
  • All code about presenting questions will be centralized in the component "Questions", including code used to render a question client side currently distributed across multiple components.

2.4 Changes in the Backend

  • The ids of questions will be moved to `UUID`s, this will allow to identify questions more or less uniquely across different platforms.
  • Adding a question to a test will always create a copy with a new `UUID` the original `UUID` will still be held in the field `original_id`.
  • The field `external_id` is dropped and will not be kept any longer.
  • Only questions that have been copied into a test will have an `original_id`. The field will be emptied when the two questions diverge.
  • With this changes questions will stop being of a certain type. They may contain answer forms of different types. It is thus possible to add multiple answer forms to one page containing answers of different types. Each of the answer forms will be treated independently when handed in.
  • A question is explicitly allowed to not contain any answer forms. It then does not award any points and does not show up on e.g. the manual marking views of the test.
  • Feedback and Skills will be optional properties of answer forms. Content for Recapitulation and Taxonomies will still be an optional property of the question.
  • Consumers will not receive any information on the question upon creation or copy. The component "Question" will store the `obj_id` of the object a question belongs to and, if configured accordingly, the position of the question.
  • Consumers will receive a `response_id` upon handing in a response and are responsible for safekeeping. The component "Question" will delete responses if the corresponding question is deleted.

2.5 Changes in the Interface

  • The creation- and edit-forms of the questions will be moved to UI-Forms. If new Inputs are needed to achieve this, they will be implemented and coordinated with the UI-Coordinators.
  • The way to edit questions will be unified, replacing the three different editors by a single one per use case.
    • The ILIAS Page Editor will be used to create the question itself and to input the question-text.
    • For all other inputs Markdown will be used.
  • The current input field "Question" will be removed from the create/edit form.
  • Clicking on the title of a question in a list of questions will enable the question-editing-mode.
    • All controls to leave the screen will be removed:
      • The Meta Bar will only show the help.
      • The Breadcrumb will be disabled.
      • The Main Bar will be emptied and will only present two buttons:
        • A first one "List of Questions" toggling a list of questions in the slate.
          • Clicking on the question titles in the list in the slate will let you move from one question to the other.
        • A second button labeled "Create question" leads to a view showing a minimal form to create a question in the main content:
          •  a text-field "Title",
          • a text-field to enter the author pre-filled with the name of the current user,
          • a dropdown to select the life-cycle set to "Draft",
          • and a textarea "Remarks", containing the information currently in the text-field labeled "Description".
          • Creating a question will open the edit-view of the ILIAS Page Editor without leaving the question-editing-mode.
    • The main content area of a question will contain:
        • Two Tabs "Question" and "Statistics" same as today
        • A form to edit the basic properties of the question containing the same fields as the form to create a question
        • Underneath the form a preview of the question in a panel; the actions of the panel will contain the entries:
          • "Edit" that will open the question-page for editing (analogous to the current button "Edit Page")
          • "Reset Preview"
    • Inserting the answer forms and configuring them will be similar as it currently is in the Learning Module.
    • All the above views are presented in question-editing-mode and the Main Bar always contains the buttons to toggle the list of questions, to show the create form and potentially a tools button to toggle the controls of the Page Editor.
    • You can always close the question-editing-mode through the Mode Info to get back to where you started.
    • The view to edit answer form will have a "back"-button, labeled "Page" as in other page elements.
    • The option to directly add a question to a pool when creating it in a test will be removed.
    • Questions may contain answer forms of different types. It is thus possible to add multiple answer forms to one page containing answers of different types. Each of the answer forms will be treated independently when handed in.
    • Feedback and Skills will be edited in tabs accessible from inside the view to edit the answer form. This means that the view to edit the answer form will contain three tabs "Answer Form", "Feedback", and "Skills".
  • For filtering in Test and TestQuestionPool the options 'Contains only', allowing to select a single type of answer form, 'Does contain', and 'Does not contain', both allowing to select multiple types of answer forms, will be provided.
  • The element "Question" in the ILIAS Page Editor used in ILIAS Learning Modules will need to be adapted:
    • It will not be possible to create or edit questions directly in a page of a ILIAS Learning Module. It will be possible to add questions directly to a ILIAS Learning Module (without the need for a "QuestionPool") on the Tab "Questions". For this a button will be added to the page and each row receives actions to edit and delete the question.
    • The element "Question" will then be used to add a question to the page either from a "QuestionPool" or from the ILIAS Learning Module itself. It will display the same table as is currently used in "Add from Pool" and "Add from Test" in the component "Test". The input "Question Pool" will be replaced by the new repository picker input set by default to the ILIAS Learning Module itself. If no questions are present in the current ILIAS Learning Module an Info-MessageBox is shown informing the user that questions can be created in the tab "Questions".
    • The "Number of Tries"-input will be moved out of the question and into the form to insert the question into the page.
  • For complex forms for example the ones to create/edit "ErrorText" or "Cloze" questions, the create form will be staged (i.e. a wizard/roundtrip) and you will never be able to edit the interdependent sections of the question simultaneously. Editing will differ fundamentally from creating the answer forms. An orientation will be provided to show you what step you are in. This still needs to be specified and it should be part of a stepped/multi-step/wizard-form container in the UI-Framework. This container still needs to be developed.
  • If a complex form is used, every step of the form will be directly accessible from the start/overview screen used to edit the question.
  • Every form containing fields that could previously be entered with the tinymce or the ILIAS Page Editor and that now are Markdown fields, will provide a button above the form to fill in the previous texts stripped of all html.
  • The component "Questions" will provide a node in the administration under "System Settings and Maintenance". It will show a tab "Questions" listing all questions in the system and a tab "Units" where the units can be configured.

2.6 Migration

  • Current texts will be frozen and kept for the foreseeable future, i.e. it will be displayed as long as no changes are required, but will need to be moved to the new implementation to be updated. They will be used for presentation when no migrated version is available and deleted as soon as the text has been migrated. We will only keep rendered html-versions of the texts. The current database structure for legacy text will be restructured and simplified and thus the texts will still need to be moved around as the database is cleaned up.
  • The legacy texts will only be shown in the preview as it would be close to impossible to display this information in a consistent way in the edit-forms without huge modification to the UI-forms and an astronomical chaos on the screen.
  • The legacy question text will be shown in a custom Page Editor element until they are deleted. The only action available on it will be to delete it.
  • We will move all questions types to answer form types in one go in order to keep a consistent interface and not have to bother with presenting both versions in parallel. This will mean that the migration to the version this is introduced in (we aim at ILIAS 13), will be long and resource intensive, if you do not run the migrations in advanced (see next point).
  • We will provide the corresponding question types in in the code as soon as possible also offering migrations for the question types that are already implemented. These migrations will not delete any data but copy them into the new component "Questions".
  • We propose to add a very rudimentary viewer for the questions, so that the migration can be tested on real data without causing any loss or more fundamental pain.
  • Tests, Question Pools, and Learning Modules will NOT work without the migration being run, they will just show a message that the migration must be run.
  • One version after adapting the Test, the Test Question Pool, and the Learning Module to use the new component a migration to drop all legacy tables will be added and the migration to migrate question types will be removed.

2.7 Additional Remarks

  • We will not change the functionality of the Test or the TestQuestionPool as part of this request.
  • The function to manually mark questions will not become a function of the component "Questions", but will stay part of the component "Test".
  • All rendering of questions will be centralized in the component "Questions". Corresponding code for asynchronous rendering currently present in other components will be moved there.
  • The element "Questions" in the ILIAS Page Editor will be left as part of the component "COPage" and will not be moved to the component "Questions". The corresponding changes will be provided as a PR as will the changes to the ILIAS Learning Module.
  • An additional page element will be created to display the legacy "question".
  • A change of the scoring system for all types of answer forms that would allow to separate the correctness of an answer from the points was considered and researched (see this feature request). The move was deemed to be a step backwards as it would decrease usability but increase complexity. It is insufficient to move to a system that tags answers as right or wrong, as we believe the current implementation that provides for answers to be partially correct to be the right one and it would be impossible to migrate existing questions to a purely binary system, even if we didn't. This move would work form some question types, but for most it would call for a lot of additional input fields (imaging the answer form type "Cloze" where we would need weights on the level of the question, the answer form type and possibly also the gap) and it would sometimes completely unclear what an index of correctness would even mean.
  • We will need two additional icons: One for the "List of Questions" one for "Create Question".

3 User Interface Modifications

3.1 List of Affected Views

  •  `ilAssQuestionPreviewGUI`
  • `assQuestionGUI::edit()`
  • `assQuestionGUI::suggestedsolution()`
  • `ilAssQuestionFeedbackEditingGUI`
  • Pages of the ILIAS Page Editor offering to add questions

3.2 User Interface Details

Base Page to Edit a question, currently the List of Questions is implemented as a Panel in a Legacy Slate. This implementation will need to be discussed with the UI-Coordinators.
Base Page to Edit a question showing the create form
Edit Page with a single answer form and the option to add another

3.3 New User Interface Concepts

  • We would like to expand the KS-Forms with the option to add multiple buttons, defining the action happening after the form hast been saved. This would hugely improve the user experience e.g. for creating questions as it would allow to add an action "Save and Create new" and it would allow to keep the current behavior that allows for questions being saved with a return to the previous page or staying on the edit form.
  • We would like to add a input-container for forms with multiple steps to the UI-Components.
  • We propose to expand the KS-Forms with the option to send them asynchronously. This would allow to reduce to dynamically build forms depending on parts of their input without having to reload the whole page, we could thus e.g. show the form to edit the gaps in a "Cloze" question only after the cloze text is input.
  • We propose to allow an action on a panel to be defined as primary and to display it directly if we are not on a small screen or if it is the only one.
  • There will surely be a need for adaptations to inputs in the KS-Forms. These will be specified in the individual feature requests specifying the changes for each question type.

3.4 Accessibility Implications

The forms to create questions are based on legacy-ui-forms with a lot of custom inputs that have a huge amount of accessibility issues. By moving these forms to KS-Forms a lot of the issues should be resolved. Additionally it will be easier to address future accessibility issue.

4 Additional Information

4.1 Involved Authorities

If this request is related to multiple components, please list both authorities for all related components.

4.2 Technical Aspects

  • The component should use the new integration mechanism provided by the component "Component". This will need some additional preparatory work in other components. We aim at providing this as PRs, but depend on the collaboration of other code authorities for it to succeed.
  • Logging:
    • The Test-Logger has been refactored with ILIAS 10, but there is still some coupling of questions with the test logger. The generation of logs needs to be completely decoupled by creating a `LogInformationGenerator` for questions, that would than be injected into the `AdditionalInformationGenerator` in the test, and by moving the creation of the `TestUserInteraction`s out of the question. This will also allow other components to implement their own logging facilities with access to information from editing and answering questions.

4.3 Privacy

This feature request doesn't change the data that is collected and also doesn't change how it is presented. It clarifies the storage of answers by users answering questions, by doing so it will improve the understandability and the structure of the data thus making it easier to control the data's lifecycle and to inform the user correspondingly.

4.4 Security

By moving to KS-Forms and restructuring and clarify the storage of data, the data will be easier to control and user inputs in the KS-Forms will be more consistently validated and sanitized on output.

4.5 Contact

Person to be contacted in case of questions about the feature:  Kergomard, Stephan [skergomard]. For questions about the funding, please contact: Strassner, Denis [dstrassner]

4.6 Funding

Funding status and funding parties are listed in the block 'Status of Feature' in the right column of this page.

If you are interested to give funding for this feature, please get into contact with the person mentioned above as 'Contact'.

5 Discussion

6 Implementation

Feature has been implemented by {Please add related profile link of this person}

6.1 Description and Screenshots

{ Description of the final implementation and screenshots if possible. }

6.2 Test Cases

Test cases completed at {date} by {user}

  • {Test case number linked to Testrail} : {test case title}

6.3 Privacy

Information in privacy.md of component: updated at {date} by {user} | no change required

6.4 Approval

Approved at {date} by {user}.

[1] This consciously ignores the system-environment as it is out of scope for the proposed changes.

Last edited: Today, 13:27, Strassner, Denis [dstrassner]