Reduce Functionality in »Corrections«-tab

1 Initial Problem

The implementation of the corrections tab threatens data integrity and poses a legal threat by equal measure. Several technical issues were simply not considered with the initial implementation and are now causing this inacceptable circumstance.

Furthermore the corrections tab does provide the opportunity to edit nearly all properties of a question except that ones, that are available for every question type (title, owner, question text). This is a significant conceptional problem as questions can be changed in a manner that it presents differently to participants than before. This MUST not be possible, since participants that allready have been taken the test answered the question in the former version.

Finaly the strongly required but not provided functionality of removing a question from a test is completely missing. This need came up after the correction tab has been integrated, but up to now it is unavailable.

2 Conceptual Summary

For the situation of having a question within an exam, that is simply wrong, the need to get this question out of a running test is important. The corrections tab should offer the possibility to remove question from a test completely.

When opening the correction screen for any question, a new tab context should be set separating the different aspects of the question correction. A first tab "question" shows the question editing screen with the question type specific properties that can be edited. A second tab "solution" shows the presentation of the best possible solution. A third tab shows a table with the answer behaviour of the participant.

For a consistent concept about the test question correction itself it is neccessary to think about every change that could be done to a question separately. In general the possibilities for editing the question should restrict to scoring relevant aspects, but for open question types there are some additional content relevant aspects that can be allowed to be changed.

Within the following sections all changeable aspects are described per question type in detail. There might be additional requirements that can be implemented in the future. This concept focus on removing undesired functinality.

2.1 Single Choice

• The answer options needs to be shown but must not be changeable
• The points per answer option needs to be changeable

2.2 Multiple Choice

• The answer options needs to be shown but must not be changeable
• The points per answer option needs to be changeable

2.3 Kprim Choice

• Points for the question should remain changeable
• The setting for half-point scoring should remain changeable
• The answers needs to be shown but must not be changeable
• The status for right/wrong per answer option should remain changeable

2.4 Error Text

• Points for the error terms needs to be changeable
• Negative points for wrong selections needs to be changeable

2.5 Imagemap Question

• The imagemap areas needs to be shown but must not be changeable
• Points per imagemap area should remain changeable

2.6 Cloze Test

• Select Gaps
  • The answer options needs to be shown but must not be changeable
  • Points for the answer options needs to be changable
• Text Gaps
  • Non predefined answer options needs to be addable from the within the answer statistic
  • Points for the answer options needs to be changable
• Numeric Gaps
  • The value as well as the lower-/upper bound needs to be changable
  • The points for the gap needs to be changeable
• Gap Combinations
  • Points per combination variants needs to be changed

2.7 Numeric Question

• Lower and upper bound needs to be changeable
• Points needs to be changeable

2.8 Text Subset

• The answer options need to be shown but must not be editable
• Points per answer option needs to be changeable

2.9 Ordering Question

• Points for the question needs to be changeable
• The ordering elements need to be shown but must not be editable
• The ordering needs to be changeable
• Also the indentation needs to changeable if activated

2.10 Ordering Horizontal

• Only the points for the question need to be changeable

2.11 Matching Question

• Only the points per matching pair need to be changeable

2.12 Essay Question

• The scoring mode for the question needs to be changable
• The keywords needs to be editable
• The points per keyword / for the question need to be changeable

2.13 File Upload

• The option for scoring on upload need to be switchable
• The points for the question need to be changable

2.14 Long Menu

• Correct answer options need to be addable from the list of available answer options
• The available answer options need to be shown but must not be editable
• The points per long menu need to be changable

3 User Interface Modifications

3.1 List of Affected Views

• Corrections Tab of Test Object
  • Question List
  • Correction of Question
    • Question
    • Solution
    • Answer Statistic

3.2 User Interface Details

The corrections tab shows the list of questions supported/enabled for test question corrections. The screen also provides functionality to remove a question from the test completely.

When opening a question for correction, the screen shows up a new tab context for this question.

The following section shows up the different tabs separated for each question type:

Single Choice

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Multiple Choice

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Kprim Choice

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Error Text

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Imagemap Question

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Cloze Test

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

The answer statistic for GAP 1 - Select Gap

The answer statistic for GAP 2 - Text Gap. A non predefined answer has been given.

Numeric Question

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Text Subset

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Ordering Question (Terms)

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Ordering Question (Nested)

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Ordering Question (Pictures)

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Ordering Horizontal

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Matching Question

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Essay Question

Question Editing Screen:

Best Possible Solution:

The aggregated statistic of the answer behaviour is not supported for the essay question type.

File Upload

Question Editing Screen:

Neither the best possible solution screen nor the answer statistic screen is supported for the file upload question type.

Longmenu

Question Editing Screen:

Best Possible Solution:

Aggregated Statistic for Answer Behaviour:

Answer Statistic for the "Longmenu 1"

Answer Statistic for the "Longmenu 2"

3.3 New User Interface Concepts

none.

4 Technical Information

Several technical issues with the corrections tab will be resolved.

5 Contact

• Author of the Request: Tödt, Alexandra [atoedt]
• Maintainer: Heyser, Björn [bheyser]
• Implementation of the feature is done by: Heyser, Björn [bheyser]

6 Funding

If you are interest in funding this feature, please add your name and institution to this list.

• ...

7 Discussion

Tödt, Alexandra [atoedt] 2018-06-28: The Corrections tab serves high stakes test that need corrections after they were completed.

This is not about self-assessment tests that are offered for a long time. We are highly aware that these tests also need some kind of editing options. This is however a very different concept that is adressed in a separate Feature Request.

Please consult Allow editing of question after test has started and / or Updating Questions in running Random Tests. For general versioning please look at Quality Management of Test Questions and better Editorial support.

Rabah, Rachid [rabah] 2018-08-23: A backport of this feature for supported and maintained ILIAS- Version would be highly appreciated. 

Rabah, Rachid [rabah] 2018-08-27: Please don't forget to tackle the issue with the overlapping see: mantis-report

Heyser, Björn [bheyser] 01 October 2018:

I guess it isn't a good idea to backport the implementation to stable releases as there will be no community side testings at all for this. The implementation will be a wide intervention to the test & assessment.

The overlapping of the question editing form with the aggregated answer behaviour has been resolved by separating into different screens. The interaction of adding additional answer options to the cloze test question is now more easy but still comfortable. The understanding of the feature gets increased even for non power users.

Having all the components of the correction on one screen was simply not possible without breaking guidelines. The given example screenshot within the named mantis report shows up a screen that cannot be done using ilias standard ui components.

Kunkel, Matthias [mkunkel], 08 OCT 2018 : First of all, thanks to Björn for this very well elaborated feature request! I have just a little suggestion for the page titles:

• Editing screen, best solution and statistic are all placed in an "underworld" and use the question title as title. Therefore, there is no need to repeat the title of the question on the editing screen (tab 'Question') again. It makes more sense to use "Correction of Points" for the first form where uses can correct points instead of displaying the question title again (which is obviously presented above). The type of question (e.g. 'Multiple Choice Question (Multiple Answers)' could be placed as 'description' under the question title.
• And I would simply use the label 'Statistics' for the current tab 'Answer Statistic' and give the table a title 'Statistics of Answer' (AFAIK, 'Statistik' is translated to 'statistics')
• And last but not least: couldn't we find a better column title for 'Frequency'? This column shows how many users have selected an answer option, right?

JourFixe, ILIAS [jourfixe], 08 OCT 2018 : We highly appreciate this suggestion and schedule it for 5.4 with the following changes / modifications:

1. We prefer to use "Correction of Points" as headline for the corrections forms in tab "Questions".
2. The question type should become a description text under the question title.
3. The tab "Answer Statistic" should be re-labeled to "Statistics".
4. Instead of one "Remove" button and inputs before each question we prefer to have a "Remove" button for each entry (because you seldom remove several questions at once).
5. Concerning tab "Statistics", we prefer not to have a mix of input an submit button for questions that should be added to cloze questions with text gaps but one button "Add as Correct Answer". When clicking on this button, a modal is opened that allows to enter points for this answer. After saving the modal input, the user is back on the Statistics tab.
6. We do not want to have sub-tabs for single answer options (i.e. for cloze questions with several gaps) but tables for all gaps on one screen, only devided by headlines for Gap 1, Gap 2, etc...

Rabah, Rachid [rabah] 2018-10-16 Is the remove of a question immediately or does the user get a prompt for confirmation (like in the example below) ?

Heyser, Björn [bheyser] 18 October 2018: A confirmation screen for the removal of questions from the test will definitly be implemented as this is required by an official guideline.

8 Implementation

The feature has been implemented according to the descriptions above. The changes required by the JJour Fixe has been respected.

8.1 Question Overview and Removal of Questions

The removal of questions from the test has been implemented and comes with a single remove button for every question contained in the test. The question titles with the overview are linked with the correction interface that is provided for all supported question types.

When removing questions from a test object a confirmation screnn is asking wether the removal should really be done, because this process cannot be undone afterwards. The allready submitted solution data of participants for this question are completly removed when confirming. The pass and test scores are recalculated.

8.2 Question Correction

The actual question correction can be done by opening a single question within the question summary of the corrections tab. An underworld of a new tab context is opened for the question correction. The question tab does provide a form to change the points for questions or its answer options.

For some question types there are additional properties of the question that can be changed like the actual ordering of elements for an ordering question for example.

8.3 Question Solution

For question types having a best possible solution the correction of questions comes with a Solution tab showing this solution. The file upload question for example does not provide this tab.

8.4 Statistics of Answers

For most of the supported question types a statistics tab is provided when correcting questions. It shows up the answering frequency for all existing test runs by any participant.

Especially for the cloze test question type there is a feature to add given answers that does not relate to the pre-configured answers for text gaps. When such answers have been submitted by any participant, the statistics tab of the question correction does provide a button to open up a modal. The modal asks for a number of points and adds the answer without leaving the context.

8.5 Supported Question Types

A detailed despription of what is possible with the corrections tab for each supported question type is provided in the following chapters:

• Conceptual Summary
• User Interface Modifications

Test Cases

Test cases completed at 29 October 2018 by Heyser, Björn [bheyser]

• C24979 : Fragen aus dem Test entfernen
• C25011 : Antwort Statistik für bestehende Durchlaufergebnisse
• C24980 : Single Choice Frage - Frage korrigieren
• C24981 : Multiple Choice Frage - Frage korrigieren
• C24982 : Kprim Choice Frage - Frage korrigieren
• C24983 : Fehlertext Frage - Frage korrigieren
• C24984 : Imagemap/Hotspot Frage - Frage korrigieren
• C24985 : Numerische Frage - Frage korrigieren
• C24986 : Begriffe benennen - Frage korrigieren
• C24987 : Anordnungsfrage (vertikal) - Frage korrigieren
• C24988 : Anordnungsfrage (horizontal) - Frage korrigieren
• C24989 : Zuordnungsfrage - Frage korrigieren
• C24990 : Text Frage (Essay) - Frage korrigieren
• C24991 : Datei hochladen - Frage korrigieren
• C24992 : Longmenu Frage - Frage korrigieren
• C24993 : Lückentextfrage - Frage korrigieren
• C24994 : Lückentextfrage - Antwort aus Statistik hinzufügen

Approval

Approved at 2018-10-29 by Glaubitz, Marko [mglaubitz] as part of the funding consortium.