Show Advanced KnowledgeHide Advanced Knowledge

Item preview from question pool plus feedback

1 Requirements

1.1 Problem

Preview single test items directly from the question pool (from a learner's view)
 
At this moment, test authors have to publish an item in a test first, if they want to preview the feedback to this particular item. If they want to adapt or change the item, the (bogus) test results have to be deleted first (and again and again).
 

right/wrong, feedback and/or suggested solution should be previewed directly from the question pool, e.g. as pop-up

1.2 Screenshots 4.3

The feedback shown here is:

• Answer specific feedback: "Yes, it is open source." below the chosen answer
• Best solution

1.3 Solution

Autoren von Aufgaben sollen die Möglichkeit haben, das Verhalten einer einzelnen Frage zu testen, ohne dass sie einen kompletten Test durchführen müssen und ihr Ergebnis in diesem Test wieder löschen.
 
Innerhalb eines Tests: Im Vorschau-Reiter der Frage kann die Antwort eingegeben und über einen Button abgeschickt werden. Der Autor erhält das gleiche Feedback, das ein Teilnehmer am Test auch erhalten würde (inklusive Musterlösung) auf Basis der Einstellungen, die er für diesen Test vorgenommen hat. Das Vorschau-Ergebnis wird nicht gespeichert.
 
Innerhalb eine Fragepools: Im Vorschau-Reiter der Frage kann die Antwort eingegeben und über einen Button abgeschickt werden. Außerdem kann der Autor über drop-down Menüs die Einstellungen für den Preview definieren (entsprechend den Einstellungen, die sonst für den Test gemacht werden): Anzeige Punktzahl (ja/nein), Anzeige Feedback (ja/nein), Anzeige Hinweise (ja/nein), Anzeige Musterlösung (ja/nein). Der Autor erhält das gleiche Feedback, das ein Teilnehmer am Test auch erhalten würde (inklusive Musterlösung). Das Vorschau-Ergebnis wird nicht gespeichert.

2 Status

• Scheduled for: Release 5.0
• Funding: Funded by DHBW as part of optes
• Development: Feature is to be developed by Databay AG

3 Additional Information

• Katja Derr, katja.derr@dhbw-mannheim.de optes, DHBW Mannheim, Teilprojekt Formatives Assessment

4 Discussion

JF 4 Mar 2013: We support the idea and schedule it for 4.4. Please contact the maintainer of the T&A tool bheyser (at) databay.de.

BH 10 Sep 2013: We also support the idea, but there are several open issues with the concept for this feature.
 
At the moment the questions from the question pools are useable by test and learning module objects. Since the learning module does not make use of the question's full functional range and since the questions are more often used in tests, we tend to consider the test objects context only. But it is important to keep in mind, that the questions could be used in future object types (core or plugin), too.
 
Some of the test settings are easily projectable onto the question's preview within question pools, because they only control the question's presentation behaviour (feedback/best solution). Not all ilias object types that make use of the questions have to support these presentation behaviours, but they can.
 
There are other settings, that influence the strategy of result calculation (score cutting, score only complete answers for mc). Projecting these settings onto the question pool's item preview would be a conceptional break, since the calculation of results is a very test specific process.
 
In consequence the question pool's item preview must support all settings, that influence the question's behaviour in general, of all ilias object types, that currently make use of the questions, and the object types, that possibly make use of the questions in the future. But the settings, that are provided by future object types implmented as repository object plugin must not be projected onto the question pool's item preview, since we are providing settings for the authors, that aren't useable in any object type, otherwise when the corresponding plugin isn't installed.
 
To involve the presentation of the question result and the feedback depending on the authors response should be no problem. Also the involvement of the best solution should be quite easy. But since the authors responses schould not be saved persistently, the hint feature cannot work correctly. When we want so simulate the process of taking hints, we have to keep in mind, that this process is a workflow with multiple requests to the question pool's or test's item preview. So we have to provide a hint reset mechanism for example and we have to talk about the required user experience, so we can decide how to persist the hint request information (session or database) and we can decide when to invalidate these hint request trackings passively.
 
What is about the "Assigened Suggested Solution"? It is not shown during the test pass, but in the pass details of the evaluation screens. When we want to support the preview to result/evaluation views, we are opening a very big barrel.
 
In every case we need a detailed concept that examines all these open issues and we also need mockups/screenshots that illustrate the imaginations of the authors, so we can discuss them here.

Matthias Kunkel, 11 Sep 2013:  IMHO this feature request is focussing on a slight improvement of usablitity for the test creation process by extending the existing preview feature for questions in the question pool and in the test object. In other words: it should make it easier and clearer for test authors to check if they have implemented a question the way the wanted to. Therefore we should focus on this process and try to keep concept and request as simple as possible and avoid a huge specification process.
 
What is asked for?

1. The request is addressing the preview of test questions used in the ILIAS test object. We do not need to discuss how these questions are presented in the learning module because this implementation differs from the test object anyway (and does not support feedback for example). Same for other object types and plugins that might be implemented in the future. This means: Item preview is implemented in the context of the test object only.
2. The requested extension of the presentation of a test question in the question pool and in the test object's preview should
   • present the evaluation of the answer, i.e. right / wrong - as known from the the test object
   • show the best solution
   • present feedbacks (if available) according to given answer
3. The request above is not asking for a calculation of results and presentation of points received for a given answer. So we can skip the entire discussion about result calculation. In other words: calculation and presentation of points for a given answer is not supported.
4. The request above is not asking to support the presentation of hints either. So we do not need to discuss the persistent storing of a given answer and how to support the multi-step process of giving hints and taking points. In other words: the preview is not supporting hints for technical reasons.
5. As the request wants only to offer a better presentation of the test question and its behaviour when a user has answered it, the preview does not need to show the "Assigned suggested solution" either. This information is given in the test results. But preview of the question and the presentation of the test results are two different things. In other words: Assigned suggested solution is not supported.

Concerning mockups: I would recommend to use the same presentation for this extended preview as it is used in a running test. Authors want to see how a user gets presented the test question including feedback, result and best solution. So the preview should immitate this and not offering a different presentation. Of course, there are no forward and back buttons and any other navigation elements. It is still a preview of a question and not a test simulation.
 
Are there any open questions left? Or is this concept sufficient for an offer and for implementing the feature?

BH 11 Sep 2013:
 
The request is addressing the preview of test questions used in the ILIAS test object. We do not need to discuss how these questions are presented in the learning module because this implementation differs from the test object anyway (and does not support feedback for example). Same for other object types and plugins that might be implemented in the future. This means: Item preview is implemented in the context of the test object only.
 
If our vision for the test question pool is to keep it intended for the test object only/primary, but hold it consumeable by other object types anyway, this concept quite fits, otherwise we have to think about in detail.
 
The requested extension of the presentation of a test question in the question pool and in the test object's preview should

• present the evaluation of the answer, i.e. right / wrong - as known from the the test object
• show the best solution
• present feedbacks (if available) according to given answer

With "evaluation of the answer" you mean the generic feedback, right? (all correct / not all correct - feedback)
 
The request above is not asking for a calculation of results and presentation of points received for a given answer. So we can skip the entire discussion about result calculation. In other words: calculation and presentation of points for a given answer is not supported.
 
The feature request isn't asking for the calculation or presentation of results, this is right. But we need this calculation at least for the determination of the fact right/wrong and even for the determination of the best solution. I have to declare, that the test settings relating to the calculation of results are either dealing with the aggregation of the results of all questions or with the cutting of reached points to 0. Both contexts do not influence the process of right/wrong determination.
Okay, we need to calculate results, but we are not presenting them.
 
The request above is not asking to support the presentation of hints either. So we do not need to discuss the persistent storing of a given answer and how to support the multi-step process of giving hints and taking points. In other words: the preview is not supporting hints for technical reasons.
 
The request above is asking for the hint feature. Katja Derr's additional feature description (the german part) that I got per email involves the hint feature. If we leave that feature out, we would have less conceptional issues to solve, but in consequence the author cannot preview their created hints.
 
As the request wants only to offer a better presentation of the test question and its behaviour when a user has answered it, the preview does not need to show the "Assigned suggested solution" either. This information is given in the test results. But preview of the question and the presentation of the test results are two different things. In other words: Assigned suggested solution is not supported.
 
This is okay, I do not really want to implement evaluation previews ;)
 
Concerning mockups: I would recommend to use the same presentation for this extended preview as it is used in a running test. Authors want to see how a user gets presented the test question including feedback, result and best solution. So the preview should immitate this and not offering a different presentation. Of course, there are no forward and back buttons and any other navigation elements. It is still a preview of a question and not a test simulation.
 
For the test item preview it is quite easy to imagine the screen, but for the item preview in question pools we need to know how the on demand setting switches should look like. Do we need to implement the same navigational behavior as in the test, so we have a check button and the on demand settings only control the possibility to request the feedback/best solution by the well known "check button", or should the on demand settings control the visibility directly and we do not need any "check buttons"?

Matthias Kunkel, 11 Sep 2013:

• Evaluation = generic feedback (I hadn't change my user language, sorry)
• If you need calculation to show right and wrong, then use it. It remembers me to our general change of concept, see Right and Wrong Instead of Points
• I will clarify with Katja Derr if optes really needs a hints support. I do not see this in the budget...
• Concerning the preview in the test object I suggest only to offer this "full preview" when you call the Preview from the view of a single question but not from the Preview in the list overview. I have added some screenshots above to clarify about what we are talking.

JF Sep 16 2013: All data that is available (feedback/hints) should be presented in the question pool presentation. We do not need checkbox settings to enable/disable these items in the preview. The preview should be as close to the presentation in the test as possible. We schedule this for 4.5 (implementation for 4.4 as originally planned is not possible due to missing resources).

JF 5 Jan 2015: The feedback section should not be presented, if no feedback is given. Please check the layout of the feedback (esp. the feedback in ordering questions) and make it more readable.

5 Implementation

BH 18 September 2014: Feature was implemented and ready for acceptance.

Tested and approved (works in ILIAS 4.4, optes) (Katja Derr, DHBW Mannheim)

BH 06 Jan 2014: Fixed presentation of answer specific feedback test/question wide without any change on the question preview itself. The presentation within the test player is now improved, so the requirement of an exactly equaling view is still not violated with the improved presentation within the preview feature.