The following steps explain the intended behavior in detail:

Step 1: The user start the test

Step 2: An unanswered question is presented

• The answer can directly be edited
• Buttons in the toolbar (‘Show List of Questions’, ‘Dashboard’, ‘Suspend the Test’, ‘Finish the Test’ - depending on test settings) are active
• Navigation buttons above and below the question (‘Previous’, ‘Next’) are active
• Buttons in the question (‘Undo Editing’, ‘Delete Complete Answer’) are inactive
• The question status ‚Not answered‘ is shown with an ‚empty circle‘ icon in the title row of a question
• Ordering, SC, and MC questions will show an additional checkbox below the question: ‘[  ] use unchanged answer’

Step 3: The user navigates away without editing the answer

• The question will not be saved because ILIAS recognizes that the answer isn’t changed.

Step 4: The user starts changing the answer

• The button ‚Undo Editing‘ becomes active
• The button ‚Delete Complete Answer‘ becomes active
• The question status ‘Editing’ is shown with an ‚empty circle‘ icon in the title row of a question
• This can also be achieved by setting ‘[ x] use unchanged answer’.

Step 5: The user further changes an answer

• Status and button enabling stays like in step 4.
• The checkbox ‘[  ] use unchanged answer’ becomes automatically unchecked and disabled

Step 6: The user navigates away from an edited question

• A modal appears telling that all changes will be saved at navigation. A checkbox allows suppressing this modal for further navigations in the test run. The navigation can be confirmed or canceled.
• Title: ‚Save on Navigation‘
• Text: ‚Please note that all changes made to the answers are stored when navigating through the test.
  [ ] Do not show this message again for the test run’
• Buttons: ‘Ok’, ‘Cancel’

Step 7: An already answered question is presented

• The answer can directly be edited
• The button ‚Undo Editing‘ is inactive
• The button ‚Delete Complete Answer‘ is active
• The question status ‘Answered’ is shown with a ‚filled circle‘ icon in the title row of a question
• The additional checkbox ‘[ ] use unchanged answer’ for an ordering, SC and MC question is hidden, because the displayed solution is already confirmed by the user

Step 8: The user wants to change a saved answer

• He can directly edit the answer
• He can click ‘Undo Editing’ while he edits the answer

Step 9: The user wants to purge a saved answer

• He clicks ‚Delete Answer Completely’
• A confirmation modal appears
• Title: ‚Delete Complete Answer‘
• Text: ‚Your answer will be finally and completely discarded. The question could be answered later again.’
• Buttons: ‘Ok’, ‘Cancel’
• After confirming with ‚Ok‘, the questions gets the ‘Not answered’ status of step 2

Step 10: The user clicks ‚Finish test‘

• A confirmation modal appears
• Title: ‘Finish Test’
• Text: You are going to finish this test pass. You won’t be able to enter this test pass again to change your answers. Do you really want to finish the test?
• Buttons: ‘Yes, I want to finish the test’, ‘No, go back to the previous question’

Display of a question in the question list (if activated)

• Not answered questions will only show the title
• Answered questions will show the title and the ‘filled circle’ icon
• The currently edited question will be shown in bold letters

Display of question in the dashboard (if activated)

• Answered questions will show the ‘filled circle’ icon in the ‘Answered’ column

Neumann, Fred [fneumann] April 1, 2015:
The former prototype shows an explict "Save" button for a question that is blue by default and gets green when the question is edited. This was the state discussed at the SIG EA meeting in March 2016. However, when testing the prototype again, we found that this extra button will confuse the users and cause them to think that they have to click it every time they change something.  The original intention of the save button was to support the special question types that may present a valid solution from the beginning. This is now supported by the  'use displayed solution' checkbox that is only shown when needed.

Some participants of the EA meeting in March 2016 want to reduce the Interface even further, more like in ILIAS 5.0. This could theoretically be achieved by new test settings to make some elements optional:

• Availability of the "Undo Editing" button
• Availability of the "Delete Complete Answer" button
• Display of the "Saving on Navigation" modal

AT: 2016-05-09: Please at least condider alternative suggestion Is-dirty Mode for Tests

JourFixe, ILIAS [jourfixe], May 09, 2016: Postponed to next JF when Björn can attempt the meeting.

Neumann, Fred [fneumann] May 31, 2016: a new prototype is available at http://www.labs.odl.org/ilias-testnav51. The code is published here: https://github.com/fneumann/ILIAS/tree/lab51-testnav

JourFixe, ILIAS [jourfixe], June 06, 2016: Fred gave a live demonstration of his suggestion to improve separating submission and navigation. We appreciate his suggestions but have some additional wishes / requests:

• The "Next" button is considered as the primary action of the question screen and shall become primary buttons. Additionally, we introduce glyphs in Next and Previous.
• The question title should be presented in a proper line and gets a new byline for status information: "editing", "not answered yet", "3 points", ...
• The actions for "Undo Editing" and "Delete Answer Completely" shall …
  • … either be moved as buttons under the question to make these actions less important.
  • … or be moved into an Actions menu (on the same level like the status information) for the same reasons.
• "Reset Ordering" will be removed and task is carried aut by "Delete answer completely".
• As a vision for 5.3 (or higher) we want for all question types that users activily interact while answering the question.
  • For 5.2 we need a checkbox "Use unchanged answer" for ordering questions unless as we do not refactor this question type to avoid that a correct question is already given.
  • To get a common understanding for handling MC questions without correct answer, we ask the members SIG EA if their teachers are using MC questions where no answer is correct or if this question design is not used (and this setting does not need to be supported any longer). In this case, we can deactivate tests with MC questions which have no correct solution and notify the responsible user that these questions have to be edited. A survey will be prepared by Matthias.

Neumann, Fred [fneumann], June 8, 2016: Alexandra prepared two mock-ups for the new position of question related actions during the Jour Fixe:

I think a question for the preference regarding these options should be included in the survey. My personal preference is the action menu.

Board, Technical [technicalboard], 8 June 2016: We highly appreciate the feature request in general and would like to give thanks to every involved person. We veto against scheduling the feature proposal at this time. MC questions should not get the additional checkbox. The need to introduce the checkbox only arose due to technical reasons, we consider this a bad reason for such a huge semantical change of the MC question. We think this change would massively decrease the usability and meaning of existing MC questions. We ask to change the feature request in this point and to work out alternatives.

Heyser, Björn [bheyser], 08. Juni 2016:

The maintainer can live with the decision of having no "use unchanged solution" checkbox for the MCMR question.

In consequence MCMR questions, that are to be answered without any selection to get the maximum points, cannot work in a consistent way anymore and MUST be migrated to an incomplete state with the coming release - regardless of the results of the survey to the SIG EA and other institutions. Additionally tests containing these questions will be migrated to offline as it is explained in the discussion above.

Using these questions/tests in production will require a reworking for these questions. Tests containing these questions as well as participant data are not broken relating to result presentation, but cannot be used for new test passes anymore, it can just be archived, since taking the test wont be possible any longer.

The maintainer suggests to add release notes as well as sending notifications to the development list and the SIG EA. The maintainer expects related bug reports anyway, at least after the stable gets released. The maintainer will close these reports fastly and without detailed comments but of course a short hint. It is very important to avoid any discussion on this in the root, since otherwise much time is to be spend for a discussion that actually is no discussion, since the JF decided finally.

The maintainer also can live with the hiding of available functions by presenting question related buttons in form of a actions list dropdown. I just want to point on the aspects that were discussed in the original request for the SubNav feature when the decision about test navigation in toolbar, modal or actions list was to be taken. Some of the arguments against a actions list are also valid for this new context.
You can find the discussion at bottom of this page: http://www.ilias.de/docu/goto_docu_wiki_wpage_2363_1357.html

Students that make no heavy use of ilias but must use it during exams will mightly not find the possibilities for e.g. undoing the editings to a question. Other people are less nervous and will find them, this can be classified as unfair, since an exam should not check the students skills of controlling unknown userinterfaces or the skill to keep cool, instead of getting nervous.

The maintainer would actually not want to live with the newly choosen place for the not answered/answered/editing hint, since the original place was exactly the place my eyes come around when exactly this hint becomes relevant for me - when moving with my focus from the question container to the next button. The advantage of the original place was that the hint did not annoy me, while concentrating on the question itself, but jumped in my eyes from itself when become relevant.

As I am not taking exams with ilias as studen this days, I can of course develop the documented and scheduled change when all others have a preference for the newly choosen place, i just wanted to give my own feelings about this to the community. We all were tested in knowlege anytime, and I know that I am getting nervous a bit during these situations. At the new place I would never notice it actively and so it would give me no value at all.

Neumann, Fred [fneumann], June 16, 2016

New Proposal to handle MC questions in the test navigation

The MC problem can be solved, if the test navigation is sensitive for a test setting that already exists under Settings > Scoring and Results:

The following SQL queries help to find questions with points on empty answers and tests that score these points:

/** * MC questions with points on empty answer */ SELECT CONCAT('goto.php?target=',o.type, '_', MIN(r.ref_id)) link, o.title, o.create_date, q.question_id, q.title, q.description, q.author, q.created, SUM(a.points_unchecked) points FROM qpl_questions q INNER JOIN qpl_a_mc a ON a.question_fi = q.question_id INNER JOIN object_data o ON o.obj_id = q.obj_fi INNER JOIN object_reference r ON r.obj_id = q.obj_fi WHERE q.complete = '1' AND r.deleted IS NULL GROUP BY o.title, o.type, q.question_id, q.title, q.description, q.author HAVING points > 0 ORDER BY o.title, q.title, q.question_id; /** * MC questions with points on empty answer and no positive points for any option */ SELECT CONCAT('goto.php?target=',o.type, '_', MIN(r.ref_id)) link, o.title, o.create_date, q.question_id, q.title, q.description, q.author, q.created, SUM(a.points_unchecked)sum_unchecked, MAX(a.points) max_checked FROM qpl_questions q INNER JOIN qpl_a_mc a ON a.question_fi = q.question_id INNER JOIN object_data o ON o.obj_id = q.obj_fi INNER JOIN object_reference r ON r.obj_id = q.obj_fi WHERE q.complete = '1' AND r.deleted IS NULL GROUP BY o.title, o.type, q.question_id, q.title, q.description, q.author HAVING sum_unchecked > 0 AND max_checked <=0 ORDER BY o.title, q.title, q.question_id; /** * Tests that score points on mc questions with empty answer */ SELECT CONCAT('goto.php?target=tst_', MIN(r.ref_id)) link, o.title, o.create_date FROM tst_tests t INNER JOIN object_data o ON o.obj_id = t.obj_fi INNER JOIN object_reference r ON r.obj_id = t.obj_fi WHERE t.mc_scoring = '1' AND r.deleted IS NULL AND t.online_status > 0 GROUP BY o.title ORDER BY o.title; /** * Tests that score points on mc questions with empty answer and having such questions */ SELECT CONCAT('goto.php?target=tst_',MIN(r.ref_id)) link, o.title, o.create_date FROM qpl_questions q INNER JOIN qpl_a_mc a ON a.question_fi = q.question_id INNER JOIN tst_tests t ON t.obj_fi = q.obj_fi INNER JOIN object_data o ON o.obj_id = q.obj_fi INNER JOIN object_reference r ON r.obj_id = q.obj_fi WHERE t.mc_scoring = '1' AND r.deleted IS NULL AND t.online_status > 0 GROUP BY o.title HAVING SUM(a.points_unchecked) > 0 ORDER BY o.title; /** * MC questions with points on empty answer in tests scoring them */ SELECT CONCAT('goto.php?target=tst_',MIN(r.ref_id)) link, o.title, o.create_date, q.question_id, q.title, q.description, q.author, q.created, SUM(a.points_unchecked) points FROM qpl_questions q INNER JOIN qpl_a_mc a ON a.question_fi = q.question_id INNER JOIN tst_tests t ON t.obj_fi = q.obj_fi INNER JOIN object_data o ON o.obj_id = q.obj_fi INNER JOIN object_reference r ON r.obj_id = q.obj_fi WHERE q.complete = '1' AND t.mc_scoring = '1' AND r.deleted IS NULL AND t.online_status > 0 GROUP BY o.title, o.type, q.question_id, q.title, q.description, q.author HAVING points > 0 ORDER BY o.title, q.title, q.question_id;

Running these queries in our ILIAS installation at the FAU, I get the following results.

Questions:

• 58.687 complete MC questions in undeleted pools or tests
• 15.896 of them give points on an empty answer
• 71 of them are tricky questions that don’t give positive points on any checked option

Tests:

• 2.704 online tests
• 41 of them have the setting ‘Score Points for Unanswered Questions as Defined in Question’
• 18 of them have MC questions that need this setting because they give points on an empty answer

Most of the questions with points on empty answers rely on the test setting “Score 0 Points for Unchecked Answers”. The real tricky questions need the second test setting ‘Score Points for Unanswered Questions as Defined in Question’. These tests are not many but they have to be handled somehow. Fortunately the test setting gives us the purpose to distinct the cases clearly and treat the tricky tests while not affecting the standard tests.

As a consequence I propose the following changes to the original feature request:

1. The generic checkbox ‘Use unchanged answer’ is only shown for ordering questions and plugin question types that indicate their need by overwriting a function. It may be rephrased to “The preset answer is correct.” Additionally the question types can tweak the label to something more specific. The checkbox will only be shown if the question is not yet answered.
2. The test option ‘Score 0 Points for Unchecked Answers’ is renamed to ‘Don’t save and score empty answers’. In this standard case the additional checkbox is not needed for MC questions and it won’t be shown.
3. The test option ‘Score Points for Unanswered Questions as Defined in Question’ is renamed to ‘Support saving and scoring empty answers’. In this rare case an additional checkbox ‘none of the above’ is shown below the options of an MC question, separated by a line or something else. The checkbox is shown regardless of the answering status.
   • It is unchecked for not answered questions.
   • Checking the box sets the question dirty which results in saving the empty result when moving away from the question.
   • If one of the MC options is checked, the ‘none of the above’ gets automatically unchecked.
   • If the question is already answered and all MC options are unchecked, then the ‘none of the above’ gets automatically checked.

The scoring of a test with “Allow scoring if no answer is chosen”  has to distinct between saved MCs with unchecked answers (which get scored) and not-saved MCs (which don’t get scored). The appearance of the additional checkbox in such a test does not give a hint on the answer, because it’s a test setting and the test may have questions with scored empty answer and questions without.

Additionally I propose to mark the test setting regarding MC questions as deprecated and show it as read-only. Existing tests will keep their setting. New tests can only be created with the standard option” Don’t save and score empty answers”.

For ILIAS 5.3 I propose a new feature request ‘Force answering interaction in all question types’ that specifies a new screen design for ordering questions and specifies consistency checks for the definition of MC questions to abandon tricky questions completely.

JourFixe, ILIAS [jourfixe], June 20, 2016: We highly appreciate Fred's suggestions from June 16 and scheduled the feature for 5.2. A feature request ‘Force answering interaction in all question types’ for 5.3 would be appreciated as well.

Kruse, Fabian [Fabian]: I appreciate the work put into this topic and think the result looks very promising from a user perspective. That said, I still have some remarks concerning a few of the labels. Maybe it is too late already, but as language issues are often able to be fixed, I would like to make a few suggestions, all coming from an end user perspective.

First of all, the modals: In step 10, the confirmation button speaks to the user (instead of „OK“ it states "Yes, I want to finish the test"). I would like to introduce this kind of label in steps 6 and 9 as well.

Modal in step 6

CURRENT STATE:
Text: ‚Please note that all changes made to the answers are stored when navigating through the test.
[ ] Do not show this message again for the test run’
Buttons: Ok, Cancel

SUGGESTION for the buttons:
Save and Proceed / Speichern und weiter
Cancel / Abbrechen

Modal in step 9

CURRENT STATE:
Text: ‚Your answer will be finally and completely discarded. The question could be answered later again.’
Buttons: Ok, Cancel

SUGGESTION for the buttons:
Delete answer / Antwort löschen
Cancel / Abbrechen

…

Then, the German wording of the UNDO button:
CURRENT STATE: Undo Editing / Bearbeitung verwerfen
SUGGESTION: Undo Editing / Letzte Änderung zurücknehmen ODER Letzte Änderung rückgängig ODER Letzte Änderung widerrufen

Reason: „Verwerfen“ is semantically too close to „Antwort löschen“, which makes the choice confusing in my book. If we decide to change this as suggested, we would make the first button a bit larger, but could make the second one smaller:

CURRENT STATE: Delete complete answer / Antwort ganz löschen
SUGGESTION: Delete answer / Antwort löschen

…

Finally, the wording for the checkbox for answers that don’t need to be changed:

CURRENT STATE: Use unchanged answer
SUGGESTION: The preset answer is correct. / Die vorgegebene Antwort ist korrekt. (As suggested by Fred in his comment. I only want to second his proposal.)

Neumann, Fred [fneumann], July 29, 2016
It seems that the test options for a forced instant feedback are important to someone and can't be reduced.

The behavior in ILIAS 5.1 is like follows:

1a) Setting "Lock Answers and Force Feedback"

• Green button "Save Answer and Check" in the question toolbar saves the answer and shows it locked in view mode with feedback.
• Blue "Next" button in the test navigation asks if the question should be saved. A click on "Save" in the modal saves the question as locked and moves to the next question without showing the feedback.
• Coming back to the question later shows it locked in view mode and shows the feedback automatically.

• Green button "Save Answer and Check" in the question toolbar saves the answer and shows it in view mode with the feedback.  It can be changed with "Edit Answer".
• Blue "Next" button in the test navigation asks if the question should be saved. A click on "Save" in the modal saves the question unlocked and moves to the next question without showing the feedback.
• Coming back to the question shows it in view mode without the feedback. It can be changed with "Edit Answer".
• Clicking "Edit Answer" for a saved question shows it in edit mode without feedback and offers "Save Answer and Check".

• Normal button "Save Answer and Check" in the question toolbar saves the answer and shows it shows it locked in view mode with feedback. 
• Green "Next" button in the test navigation asks if a changed question should be saved. A click on "Save and Proceed" in the modal saves the question unlocked.
• Coming back to the question can show it in different states:
  • If feedback was already requested, then the question is shown locked in view mode with automated feedback.
  • If feedback was not yet requested, then the question it is shown editable without feedback, having a "Save Answer and Check" button.

• Normal button "Check" in the question toolbar shows the feedback, but keeps the question in "editing..." state.
• Green "Next" button in the test navigation asks if a changed question should be saved. A click on "Save and Proceed" in the modal saved the question unlocked and moves to the next question without showing the feedback.
• Coming back to the question always shows it in edit mode without the feedback. "Check" can be clicked again.

• The "forced feedback" settings make the button "Save Answer and Check" green instead of the "Next" button.
• The "forced feedback" settings always save the answer when feedback is requested.
• The setting "Lock Answers and Force Feedback" shows the feedback when coming back to a saved question, even it if was not explictly requested before.

1. Student has changed the answer
2.  Student clicks "Next", "Previous", "Dashboard" or another navigation link
3. ILIAS saves the answer, stays at the question and shows the feedback
4.  Student clicks "Next", "Previous", "Dashboard" or another navigation link again
5. ILIAS navigates as requested in step 4

1. Student has changed the answer
2.  Student clicks "Next", "Previous", "Dashboard" or another navigation link
3. ILIAS saves the answer, stays at the question and shows the feedback in a modal that can be closed with an "Ok" button
4. Student clicks "Ok"
5. ILIAS navigates as requested in step 2

• Variant A may disturb the student because he expects the next question to be shown. Additionally he has to click the same action again.
• The modal in variant B may be lange with scrolling, because feedback can consist of several elements: inline feedback in the question, dedicated feedback texts for the question or single answer options and a best solution that may be shown.

JourFixe, ILIAS [jourfixe], August 01, 2016: We see three options to handle the feature 'forced instant feedback':

1. Drop the feature
2. Keep the feature as it is in 5.1 (Fred's suggestion)
3. Introduce a feedback modal (variant B) for forced feedback