Extending Continuous Testing Mode: Purgatory Status of Question

1 Initial Problem

By now (version 5.2), the Continuous Testing Mode (CTM) can only count a question as (completely) successfully processed or failed.
However, there are a number of scenarios in which questions cannot be classified in such a clear way.
There might be different triggers that make questions ambiguous:

• Questions with multiple choice character that have only been partially solved.
• Questions the learner solved only by assistance of one or more hints.
• Questions the learner solved but wasting too much time.
• Questions the learner solved but only once, so it's not sure whether the result is reliable.
• Questions the learner actively wants to keep within the selection, e.g., because he/she considers to have passed only by good luck.

2 Conceptual Summary

So, my proposal is to enhance the settings of the CTM (or its selection options) by a "purgatory" status for questions that have been solved but should keep within the selection for some reason.
This status could be indicated on the question page itself and/or by listing them in a specific column in the table of the selection view.

Independent of the trigger (s. list above), we would need some adaptions of the CTM:

• A "purgatory" status for questions (a better term still to be found).
• A configuration setting for CTM tests that determines whether this status is to be used.
• A configuration sub-setting for CTM tests that determines what triggers the status.
• Another column in the table of the selection view.

With regard to specific triggers, there would be correspondingly specific additional requirements:

• Questions with multiple choice character which may be partially solved:
  This would require ...:

  • A function that clearly discerns partially solved questions.
  • An automatism that sorts those questions into the "purgatory" status if the "partially solved" trigger is selected by the CTM author.

• Questions the learner solved only by assistance of one or more hints:
  IMHO, this would not require anything specific.

• Questions the learner solved but wasting too much time:
  There is a separate page Extending Continuous Testing Mode: Consider Processing Time.

• Questions the learner solved but only once, so it's not sure whether the result is reliable:
  There is a separate page Extending Continuous Testing Mode: Threshold to Overcome Question.

• Questions the learner actively wants to keep within the selection:
  This would require a "Submit but keep in selection" button in CTM tests.

3 User Interface Modifications

3.1 List of Affected Views

{Please list all views (screens) of ILIAS that should be modified, newly introduced or removed.}

3.2 User Interface Details

{For each of these views please list all user interface elements that should be modified, added or removed. Please provide the textual appearance of the UI elements and their interactive behaviour.}

3.3 New User Interface Concepts

{If the proposal introduces any completely new user interface elements, please provide a link to separate feature wiki entries for each of them according to the kitchen sink template.}

4 Technical Information

{The maintainer has to provide necessary technical information, e.g. dependencies on other ILIAS components, necessary modifications in general services/architecture, potential security or performance issues.}

5 Contact

• Author of the Request: Suittenpointner, Florian [suittenpointner]; Wischniak, Stanislav [wischniak]
• Maintainer: Heyser, Björn [bheyser]
• Implementation of the feature is done by: Databay AG

6 Funding

If you are interest in funding this feature, please add your name and institution to this list.

• ...

7 Discussion

Florian Suittenpointner, 10 Oct 2016:
One thing I already started discussing with Björn Heyser is whether it makes sense to enhance the existing CTM by so many options or rather to introduce a completely new test mode "Card File" (Karteikasten).

Kergomard, Stephan [skergomard], 2022 Mar 9: We close this as part of the cleaning up effort undertaken in the Splitting-Up Test & Assessment Workinggroup.

8 Implementation

{The maintainer has to give a description of the final implementation and add screenshots if possible.}

Test Cases

Test cases completed at {date} by {user}

• {Test case number linked to Testrail} : {test case title}

Approval

Approved at {date} by {user}.