Feature Wiki

Information about planned and released features

Tabs

Option to Obligate Test Questions

Page Overview

[Hide]

1 Description

In some cases questions of the test tool should be obligate (verpflichtend). For example for the file upload question this feature would be nice.

Like in the survey tool the obligation button could be in the tab "question" (column "obligatory") and/or in the question itself.

2 Status

3 Additional Information

  • If you want to know more about this feature, its implementation or funding, please contact: yvonne.seiler@ilub.unibe.ch / Universität Bern

4 Discussion

JF 20 Sep 2010: So how would the behaviour be? If a student does not answer one obligatory question, the test is failed? We need Helmuts opinion on this, since we are unsure what implications this feature would have on the whole test processing.

HS 20 Sep 2010: I don't see any issues with obligatory questions. The only thing is to define the consequences of not answering obligatory questions (can a test be finished with unanswered questions, is a test failed if an obligatory question isn't answered). The implementation of such implications could be done easily in the methods that finish a test or check if the test is passed.

Jour Fixe, 01 August 2011: We appreciate the feature but request some more conceptual explanations as well as a clear scenario where and how this option could be useful.

YS, 27. Sept 11: At the moment these features are implemented (we tested it in the elba branch):
  1. questions are marked with * if they are obligate and a explanation text at the end of the question shows what this means: * Required
  2. in the list of questions you can see, which questions are required.
  3. you can finish the test, if you have answered all required question in the test.

Now we have an annotation for the pass overview of the test results. I post this in german, because it's easier to explain - perhaps someone likes to translate it in english. Please discuss this at the next jour fixe:
Dann haben wir noch eine Anmerkung zur Darstellung der Testergebnisse. Stand jetzt: Füllt ein Teilnehmer einen Test mit verpflichtenden Fragen aus, so werden seine Testergebnisse bereits gespeichert, auch wenn er den Test (noch) nicht abgeschlossen hat.

Szenario zur Erläuterung: Ein Teilnehmer klickt sich durch den Test. Er bricht ab indem er den Browser schliesst, bzw. sich aus dem Test klickt und woanders durch ILIAS navigiert. Seine Ergebnisse sind nun aber bereits gespeichert, er ist durchgefallen, weil er nicht alle Punkte erreicht hat. Der Test-Admin möchte aber sehen, wer den Test auch richtig beendet hat (mit Button "Test beenden"). Kann im Test 1 Durchlauf gemacht werden, wird im Register "Teilnehmer" in der Spalte "Beendet" ein grünes Häkchen gesetzt, wenn ein Teilnehmer auf "Test beenden" geklickt hat. Das ist gut so.
Können im Test aber mehrere Durchläufe (z.B. 2) gemacht werden, wird im Register "Teilnehmer" in der Spalte "Beendet" erst dann ein grünes Häkchen gesetzt, wenn die Person den zweiten Durchlauf absolviert hat. Es wird nicht ersichtlich, ob die Person den ersten Durchlauf oder den zweiten (oder beide) mit Button "Test beenden" abgeschlossen und somit eingereicht hat. Dies erschwert zur Zeit den Test-Admins den Überblick, wer den Test mit allen verpflichtenden Fragen ausgefüllt hat und wer nur probeweise im Test war.

Wir schlagen nun folgendes Vorgehen vor:
  1. Im Reiter "Teilnehmer" wird die Markierung "Beendet" (grünes Häkchen) gesetzt, sobald ein Durchlauf vollständig über "Test beenden" abgeschlossen wurde. Egal ob dies nun der erste, der zweite oder der dritte, usw. Durchlauf ist. In der Spalte "Bereits durchgeführte Testdurchläufe" wird aufgezählt, wie oft ein Test bereits durchgeführt wurde. (siehe Bild A)
  2. Über "Testergebnisse anzeigen" (ebenfalls Reiter "Teilnehmer") sollte man dann bei jedem einzelnen Teilnehmer nachprüfen können, welchen Testdurchlauf er beendet hat. Hier wird eine neue Spalte eingefügt, die zeigt, ob er den Testdurchlauf vollständig beendet hat. (siehe Bild B)

Ist dies ein grosser Eingriff in die bisherige Handhabung der Spalte "Beendet" oder könnte man diese so optimieren?
Oder weshalb wird bei 2 möglichen Durchläufen der Test nur als "Beendet" angesehen, wenn beide Durchläufe beendet sind?
A
B

YS, 5. Oktober 2011: but request some more conceptual explanations as well as a clear scenario where and how this option could be useful. In this file we thought about all other question types and how or if there could be implemented an option to "obligate" questions.

JF 10 Oct 2011: We currently do not support the approach as being described here:
  • It adds a lot of complexity to the T&A system.
  • It is hard to understand for learners.
  • Learners that understand the need to answer the required questions will just "rush" through them at the end of the test to ensure that their test is scored. Which is quite a useless behaviour.
JF 10 Oct 2011 wrote: We currently do not support the approach as being described here:
  • It adds a lot of complexity to the T&A system.
We need a more different feedback: Why do you think this feature adds "too much" complexity to the T&A-system? What we should change?
a) Do you think that the view of the test-results adds too much complexity?
b) or ???
  • It is hard to understand for learners.
  • Learners that understand the need to answer the required questions will just "rush" through them at the end of the test to ensure that their test is scored. Which is quite a useless behaviour.
All of our consulted teachers wish test isn't ending with "failed", if users haven't solved all obligate questions. They like to have the behaviour that a test can't be scored, if users haven't solve all obligate questions: "Ich persönlich fände es besser, wenn bei fehlenden obligatorischen Fragen der Test nicht abgegeben werden kann und nicht als "failed" zu betrachten ist. Für uns wäre die Funktion auf jeden Fall sinnvoller.".
 
We think, if the participants have required questions they haven't answered, it could be because they
  1. don't know the answer or
  2. they skip the answer (because they didn't know the answer yet and likes to answer it later) or
  3. they forget to answer the question and jumped to the next one, too fast (clicking to times almost at the same time).
For 1.) you've right it's perhaps a bit confusing if they don't know the answer and have to set an answer. So they they will "rush" through them to end the test. But they don't get any points anyway if they don't answer the question, so even if they "rush" through, they have a little chance to get some point . They were forced to give an answer, like in surveys.
 
For 2.) and 3.) they get the possibility to answer a question they have forget or skipped before. So they get a hint/an overview, which question(s) they forgot. We think this is more a help than a useless behaviour for the participants.
 
Perhaps this behaviour would be more usefull: If you have some obligate question they aren't answered yet, then at the end of the test there is a hint like: "You haven't answered all obligatory questions, that means your score will be "failed". Are you sure you want finish the test anyway?". Now you have two options: Go back to answers (with "Back"-Button or "List of questions") or "Finish the test".
 
The option to get a hint at the end of test which means "You must answer all obligatory questions before you can finish the test." is useful, because in tests you can postpone questions. So we can't use the same behaviour of required answers like in surveys. Or do you like to have all required questions at the end of test and only if you have answered all questions you get the "Finish the test"-Button?
 
What's your opinion about this?

JF 7 Nov 2011: We are sorry that this feature has been initially "be appreciated", but we have asked for conceptual clarification and we currently do not see a good reason to add the feature. There is a lot of explanation how the feature should be implemented, but not why. What is the basic need behind this idea? The importance of questions is given by the mark schema and the points that are assigned to questions: Important questions get higher points. We discussed this in a very large group today (7 persons), including people that wrote paper/pencil tests and we still have problems to see the didactical need. Should we schedule a phone call on this? We think that this could make it easier to come to a conclusion, than to go on discussion it on this wiki page.

YS & HL 30 Nov 2011: Thanks for your work and discussions. We try to explain the basic need and didactical benefit of this feature in the attached file. Please contact us if you need more information than that.

JF 9 Jan 2012: We worked through the "Konzept Obligate Test Questions" document and see added value in scenario 2. But still possible negative usability implications for test authors and learners.
  • Scenario 1: We still think that obligatory questions will not avoid that users just "click through" the test. Some of them may spent more time on these questions, others just click "any" of the possible answers which will reduce the information value of the test / results quality (no answer would be better than a random one).
  • Scenario 3: It is unclear to us, how the navigation at the end of the test will work in detail (Jan Posselt should present this in one of our next Jour Fixes). If users get a presentation of a list of unanswered obligatory questions, users may misinterpret it as a list of all unanswered questions.
  • Is it correct, that the "passed" status of the test depends on a) the points reached by the user and b) whether all obligatory questions have been answered? This would be our interpretation of the feature. This means a user can pass a test, without clicking the Finish button.
  • We think that a "obligatory" property for questions makes only sense within the test (for test questions) not within question pools, since it is related to the design of a test, not only to a single question (re-used in different tests with different designs).
  • We would like to identify at least one other institution that backs this feature and would like to use it, to ensure that this is not an uncommon need.
We will continue the discussion at the next Jour Fixe, which takes place at Databay together with the test maintainers.
JF 23 Jan 2012: Databay gave us a presentation of the current implementation of obligatory questions in the Elba patch.
 
Navigation
  • When obligatory questions are presented to the user...
    • obligatory questions get a little "*" as marker
    • the "Next" button is present and navigates to the next question
  • After the last question
    • if "show list of questions" is activated the list of question is presented, obligatory questions are marked
    • the test the "finish" button shows a message: "You must answer all obligatory questions..." and a "Previous" button that navigates back to the last question (regardless of whether it is obligatory or not)
We think that the usability of this approach is substandard. Especially the need to use lots of "Previous" clicks to get back to questions marked with "*" (especially if "show list of questions" is deactivated).
 
Supported Question Types
 
Currently only single choice and file upload are supported.
 
Open Issues
  • A navigation concept for "missed" obligatory questions that fixes the usability issues above.
  • A setting "Use Obligatory Questions" setting (default deactivated) in the test that avoids the presentation of the obligatory column in the test authoring mode.
  • The setting should also be part of the test settings templates.
  • Another institution that backs the feature should be found.
  • At least multiple choice questions should get this feature, too. If a question is marked as obligatory, that has the "all unchecked" variant as correct solution, the question should be marked in the question overview (similar to incomplete questions).
YS&HL 24 Jan 2012: Thanks for your brain work. We would like to take part of it and have some suggestions about the open issues:
  • A setting "Required Question" in the setting-tab: We agree to this point and think this is a good way to implement this feature! An additional information in the description like "If checked, questions of the test can be marked as required (*). All required questions need to be filled out to pass this test. You can set required questions in the "list view". We recommend to activate "Show List of Questions" with required questions." would be very useful.
  • A navigation concept: We think that the within a test with obligatory questions, the option "List of Question" is the best way to navigate between the questions. If an admin uses this option, then there are no usability issues. In the following file "Navigation Concept required questions" we give some suggestions about the processes to create required questions (Admin), to display useful informations before participants starts with tests they have required questions (Participant's Info), the test himself with "List of Questions" (Test) and his verifying "are all required questions answered?" and the consequences "pass/failed" (Scoring). In this navigation concept you can see also a proposal how this could be implemented in the pass overview.
  • The setting should also be part of the test settings templates: Yes, we agree to this point!
  • Another institution: Pädagogische Hochschule Zürich (contact-person: Werner Willi) supports this feature, f.e. there they have a large demand on scenario 3.
  • Other question types: We highly appreciate this :) You can see in the upper file "Obligatorische Testfragen_Umsetzung" some ideas for most question types in the test tool. We agree with you for multiple choice questions.
Please have a look at the navigation concept, if you have any questions let us know!
Navigation Concept required question (optimized for monitor; only 72dpi)
BH 11 Apr 2012: I have tried to carve out the last todos for this feature by comparing the latest comments from JF and YS/HL.
 
The following issues are clear and have to be implemented:
  • New test setting that enables the feature for a test in general (with support for test settings templates)
  • An information message, that adverts to the obligatory questions
The following issues are unclear to me:
  • Multiple Choice Questions The JF made demands on the support of this question type in that way, that the question is not answered when no checkbox of the possible answers was checked by the participant. However YS/HS described a different concept to determine wether a multiple choice question is answered or not. But they highly appriciate the support of this question type and also other types, too. Is the possibilty to set multiple choice question as obligatory to be implemented regarding to the suggestions of JF or is here still the need for conceptual discussion?
  • Navigation Concept The navigation concept YS/HS have comitted shows the detailed workflows for admins as well as for participants. But regarding to the participants I can not identify any difference to the current state. The only improvement is the description for the test setting that recommends the admin to enable the list of questions.

YS, HL (11.4.2012)
 
The following issues are unclear for Databay:
 
a) Multiple Choice Questions The JF made demands on the support of this question type in that way, that the question is not answered when no checkbox of the possible answers was checked by the participant. However YS/HS described a different concept to determine wether a multiple choice question is answered or not. But they highly appriciate the support of this question type and also other types, too. Is the possibilty to set multiple choice question as obligatory to be implemented regarding to the suggestions of JF or is here still the need for conceptual discussion?
 
Our comment: We do not found a good solution how we could improve the actual MC-form so that it works fine with this feature. We recommand not to implement MC-questions. We will continue discussing an alternative MC-question type, e.g. Kprim. So we will later discuss the new question-type Kprim AND its implementation as a possible obligatory question (ILIAS v4.4).
 
Actually only single-choice and fileupload-questions are implemented.
 
b) Navigation Concept The navigation concept YS/HS have comitted shows the detailed workflows for admins as well as for participants. But regarding to the participants I can not identify any difference to the current state. The only improvement is the description for the test setting that recommends the admin to enable the list of questions.
 
Our comment: The new improvements (not yet implemented) to the current state are:
1. The test settings

2. The description for the test setting and the advice for the participant if he starts the test
3. The pass-overview must have better information for Admins/Users - when questions are obligatory:
a) All required questions are filled out
=> a1: Failed when user don`t get required points
=> a2: Passed when user get enough required points
b) Minimum one required question is missing
=> Failed the test
4) The admin see this better pass-overview here: Participants-Tab => Dropdown-Menu "Show Marked Pass Overview" and "Show Users Detailed Results (marked pass)"
5) The user see this better pass-overview here: Info-Tab => Show test results

BH 11 Apr 2012
 
The following issues are unclear for Databay:
 
a) Multiple Choice Questions The JF made demands on the support of this question type in that way, that the question is not answered when no checkbox of the possible answers was checked by the participant. However YS/HS described a different concept to determine wether a multiple choice question is answered or not. But they highly appriciate the support of this question type and also other types, too. Is the possibilty to set multiple choice question as obligatory to be implemented regarding to the suggestions of JF or is here still the need for conceptual discussion?
 
Our comment: We do not found a good solution how we could improve the actual MC-form so that it works fine with this feature. We recommand not to implement MC-questions. We will continue discussing an alternative MC-question type, e.g. Kprim. So we will later discuss the new question-type Kprim AND its implementation as a possible obligatory question (ILIAS v4.4).
 
Actually only single-choice and fileupload-questions are implemented.
 
But we have to implement the support for at least multiple choice questions to get this feature into 4.3 and if i have understand the JF in the right way,
the implementation have to follow these rules:
 
At least multiple choice questions should get this feature, too. If a question is marked as obligatory, that has the "all unchecked" variant as correct solution, the question should be marked in the question overview (similar to incomplete questions).
(--> because an "all unchecked" answered question is to be interpreted as not-answered)
 
b) Navigation Concept The navigation concept YS/HS have comitted shows the detailed workflows for admins as well as for participants. But regarding to the participants I can not identify any difference to the current state. The only improvement is the description for the test setting that recommends the admin to enable the list of questions.
 
Our comment: The new improvements (not yet implemented) to the current state are:
1. The test settings
2. The description for the test setting and the advice for the participant if he starts the test
3. The pass-overview must have better information for Admins/Users - when questions are obligatory:
a) All required questions are filled out
=> a1: Failed when user don`t get required points
=> a2: Passed when user get enough required points
b) Minimum one required question is missing
=> Failed the test
4) The admin see this better pass-overview here: Participants-Tab => Dropdown-Menu "Show Marked Pass Overview" and "Show Users Detailed Results (marked pass)"
5) The user see this better pass-overview here: Info-Tab => Show test results
 
Yes, of course these (not yet implemented) changes are very useful improvements to the feature in general!
 
But regarding to the test pass it self (participant processes the test and wants to finish) your navigation concept just reflects the current state.
The JF mentioned some usibility issues within this process of passing the test that aren't solved I think.

@JF: What do you say? Maybe you can explain the usibility issues in a more detailed way?
 
Navigation
  • When obligatory questions are presented to the user...
    • obligatory questions get a little "*" as marker
    • the "Next" button is present and navigates to the next question
  • After the last question
    • if "show list of questions" is activated the list of question is presented, obligatory questions are marked
    • the test the "finish" button shows a message: "You must answer all obligatory questions..." and a "Previous" button that navigates back to the last question (regardless of whether it is obligatory or not)
YS&HL 11 Apr 2012
  • Multiple Choice question: We agree about an implementation of this question type. So if you see a technical way to implement it like JF mentioned, we're glad:
    • If someone add a multiple choice question, that has the "all unchecked" variant as correct solution, the question will be marked in the question overview IF it's marked as a required question. An advice should describe this symbol like "You have to set points for a checked answer to use this question as a required question".
  • Navigation Concept: Yes our navigation concept just reflects most of the current state. The aim of this concept is to visualize the whole feature. So we put this on jour fixe again!
@JF: What do you say? Maybe you can explain the usibility issues in a more detailed way?

JF 16 Apr 2012: We postpone the discussion to a JF where Björn Heyser is able to attend.

JF 2 May 2012
  • Adding a hint to the description of the "Activate Obligatory Test Questions" option that suggests to activate the question overview feature is something e should do.
  • There should be a better way to visualize obligatory test questions. Below the question title a message in red should be displayed stating "Diese Frage muss beantwortet werden, um den Test beenden zu können" / "You have to answer this question to be able to finish the test."
  • On the page where you currently get the message "Fehlermeldung Sie müssen alle verpflichtenden Fragen beantworten bevor Sie den Test abschließen können." a list of all unanswered obligatory questions must be displayed in order to allow the participant to address the issues directly.
  • We have to define clearly when a question gets the "has been answered" state and when it is evaluated in the results. BH wrote above: "(--> because an "all unchecked" answered question is to be interpreted as not-answered)". We just tried out a test including a multiple choice questions (ILIAS 4.2). If no checkbox is checked by the user and the user navigates to the next question the MC question is still "unanswered", but the question is part of the evaluation (in our case -3 points). If the user finishes the test without navigating to this question, the quesion is also "unanswered" but NOT part of the evaluation. BH: Could you please collect the current behaviour of the different question types at Revision of Test Evaluation.
  • We see that it is currently not possible to implement the "obligatory" feature for all question types (mc and ordering e.g.). @BH: Please check for which other question types the feature can be added in a feasible way (at least the essay questions should be added).

YS&HL (22.6.2012)
• Adding a hint to the description of the "Activate Obligatory Test Questions" option that suggests to activate the question overview feature is something e should do.
=> OK.
 
• There should be a better way to visualize obligatory test questions. Below the question title a message in red should be displayed stating "Diese Frage muss beantwortet werden, um den Test beenden zu können" / "You have to answer this question to be able to finish the test."
=> OK. this is better than just the "*".
 
• On the page where you currently get the message "Fehlermeldung Sie müssen alle verpflichtenden Fragen beantworten bevor Sie den Test abschließen können." a list of all unanswered obligatory questions must be displayed in order to allow the participant to address the issues directly.
=> OK
 
• We have to define clearly when a question gets the "has been answered" state. Currently questions are saved as answered when the user navigates to the question page and hits any button that navigates to another screen. (--> because an "all unchecked" answered question is to be interpreted as not-answered)

 
=> Unser Vorschlag für 4.3: Zusätzliche Implementation von MCs(Multiple Answers) wie folgt: Der Autor kann eine MC(Multiple-Answers) in der Listensicht nur dann als "Verpflichtend" markieren, wenn bei mindestens einer Antwort bei der Frage Spalte "Ausgewählt" ein Wert >0 eingegeben wurde. Falls nicht, benötigt es hier einen entsprechenden Hinweis, dass die Frage nicht als "Verpflichtend" markiert werden kann.
=> Wenn eine MC-MA Frage als Verpflichtend markiert werden kann, dann markiert ILIAS die Frage nur dann als "Beantwortet", wenn tatsächlich mind. 1 Häkchen angekreuzt wurde.

JF 25 June 2012: We support the ideas of YS&HL from 22.6.2012.

5 Implementation

Last edited: 17. Apr 2025, 15:04, Kunkel, Matthias [mkunkel]