Show Advanced KnowledgeHide Advanced Knowledge

Random Test: Check for Fully Stable Selection Configs

1 Initial Problem

With the current implementation "semi stable" question selection configs are possible that require a fallback selection of unintended questions. The reason is a missing exclusive question set check per selection rule that would consider any question that can be chosen by different rules

1. imagine a question pool having two taxonomy nodes A and B
2. the pool contains 3 questions (1, 2 and 3)
3. question 1 is assigned to A while question 2 is assigned to B
4. the last question, question 3 is assigned to A as well as B
5. two selection rules are configured:
   1. take one question from the taxonomy node A of our pool
   2. take two questions from our pool now, but from the taxonomy node B

• when the first rule selects question 1 - we have still enough questions for the second selection rule
• but when it selects question 2 - it won't be possible to fullfill the second selection rule

When a selection rule returns less questions than it should the difference is tracked. Any difference is solved by reiterating over the selection rules to collect questions where questions are still available. In a possible final round any other available question from the pool is taken without considering the selection rule's taxonomy filter anymore.

Additionally we depend on the need to have ALL questions from every involved question pool synchronized to the question stage in the test, regardless of any taxonomy filtering that could actually reduce the actual indented question stage, as we need to be able to serve the fallback for choosing questions from the stage that is described above.

A general check wether the config leads to succesful question selections for valid test passes does indeed do the same check, that is also done for the non promlematical mode that takes only one amount of questions to be selected shared for all selection rules. This is working for the current situation because of the fallback that does consider the other rules again when any conflict comes up.

2 Conceptual Summary

We should consider to avoid semi stable configs from beeing considered as valid senario. We indeed tried to do in the root, but there was not enough brainfuel present in the conceptional heads at this time.

Fred Neumann hinted on the mathematical solution for the problem: Do an exclusive question set check per selction rule as it is described below.

Imagine the same example from above. The main problem was that we do not know wether the first selection rule takes the question that is exclusively available for that rule or the question that is also available for the other rule. (a stable or a unstable config - a semi stable config)

But we should try to just consider exactly that questions, that are exclusively available per selection rule. When we than check for a buildable pass. We will recognize that there can occur a conflict on the second selection rule as this rule cannot be fullfilled now. (no "semi" stable configs any longer).

Concrete example:

• build exclusive question set for the FIRST selection rule
  • Question 1 -> Exclusive? -> Yes
  • Question 3 -> Exclusive? -> No
  • Exclusive Set: Question 1 only
• build exclusive question set for the SECOND selection rule
  • Question 2 -> Exclusive? -> Yes
  • Question 3 -> Exclusive? -> No
  • Exclusive Set: Question 2 only
• perform build check:
  • First Rule should select one question -> Exclusive Set Provides one total question -> OK
  • Second Rule should select two question -> Exclusive Set Provides one total question -> FAIL

The check shows up, that there is at least one non working combination. The configuration is not consideres as valid as it wont lead to valid passes in every situation.

There is a weakness left. The following scenario will fail and the configuration will be considered as invalid. The exclusive sets of questions for the second and the third selection rule are always empty, so picking ten questions is not possble for sure.

• Grundrechenarten (100 Questions)
  • Add (25 Questions)
  • Sub (25 Questions)
  • Mul (25 Questions)
  • Div (25 Questions)

1. take 10 from Grundrechenarten
2. take 10 from Multiplication
3. take 10 from Division

--> SHOULD THIS WORK ?

The solution is to extend the check from above. While checking we need to count the questions in the exclusive set as well as the non conflicting amount of questions of the question sets shared by other selection rules.

This way checking each amount of available questions per selection rule in comparison to the according required number of questions, we can guarantee stable configs only to be considered as valid. And therefore we won't depend on the dirty fallback selection of undesired questions, so it can be removed completely.

As we do not rely on the fallback selection of undesired questions to compensate conflicts when the config is semi stable, there is no need for the synchronisation of ALL questions for ALL involved question pools any longer, because only that questions will be selected that indeed belongs to the filter criteria for the question pool that are defined by the selection rule.

Therefore we limit the synchronisation of questions from a question pool to the test's question stage to exactly that questions, that match the filter criteria of any selection rule that references this pool.

Overview of changes:

• Replace the dirty check for non-fully-unstable selection configs with a qualified exclusive question set check, that considers fully stable selection configs as valid configs only. This check simply needs to not consider any question that matches the filter criteria of more than one selection rules.
• Remove the dirty question selection fallback completely, that might use questions from other rule's stages or even any other question from the pool, when there is no question left that hasn't been selected yet from the other selection rules.
• limit question pool synchronisation to the subset of questions contained in the union set of all selection rules, because we won't need any other questions from the pools for sure.

Regarding the ordering of seletion rules: This is still neccessary, although the first minds said - remove it. Without enabled shuffle setting, the test of course draws up random and therefore shuffled questions, but the groups of question selected by each rule stick together. The shuffle setting for the test can force up the fully shuffled variant of a random test pass.

3 User Interface Modifications

3.1 List of Affected Views

3.2 User Interface Details

The input fields used for order the selection rules will be removed completely.

3.3 New User Interface Concepts

There is no need for any new user interface element.

4 Technical Information

No existing depencies or technical requirements. Regarding the performance a small boost could be noticed for the synchronisation of a single question pool  when the set filters reduce the amount of relevant question significantly compared to the total amount of questions stored in this pool.

5 Contact

• Author of the Request: Heyser, Björn [bheyser]
• Maintainer: Heyser, Björn [bheyser]
• Implementation of the feature is done by: Heyser, Björn [bheyser]

6 Funding

If you are interest in funding this feature, please add your name and institution to this list.

• ...

7 Discussion

Neumann, Fred [fneumann], February 14, 2017

We use the restriction to fully stable configs as a patch since two years in our e-exams and it works fine. Of course, it reduces the possible variations of random tests but it is easily understood by the authors and can be supported with detailed information in the validation message for the rules, e.g. "Selection rule 2 provides only one exclusive question but 2 are needed. Please ensure that each rule selects enouch questions that are not matching other rules. "

For random tests in e-exams it is absolutely necessary that a test is always built with the defined rules and no "fallback" question is chosen from the whole set of questions, just to fill up the total number of questions. The test conditions for students wouldn't be comparable anymore as the fallback questions could have different maximum points or be related to different topics or difficulties.

We also need the ordering mechanism as long as the feature Test-Parts and Question-Groups is not implemented. This is the only way to construct random tests that have a logical order of questions. In most cases one taxomony will provide a thematical structure, e.g. "basic calculation/addition", "basic calculation/multiplication". Test authors can create selections rules where all questions of a rule are comparable regarding maximum points and difficulty because they just change names and numbers. With the ordering mechanism it is possible to create a random test that has a fixed order of question topics (following the the topics of a lecture or course) but a different selection of questions per topic for each student.

Heyser, Björn [bheyser] 13 March 2017:

I agree with the objection to remove the ordering of selection rules, since the building of thematical structures is indeed possible this way. A random test is a shuffled thing in general but when we do not enable the shuffling of questions in the test's settings, we are indeed dealing with shuffled groups.

JourFixe, ILIAS [jourfixe], March 13, 2017: We discussed Björn's suggestion and appreciate it in general as it improves random test selections. But we see a problem with the statement that selection rules are not valid if no exclusive questions exist. Example: 10 questions should be selected from a tax node 'Grundrechenarten' (with 100 questions) and 10 from the sub-node 'Multiplikation'. As all 25 questions assigned to 'Multiplikation' are also assigned to 'Grundrechenarten', Björn's heuristic would fail because there is no exclusive question. Björn will revise his concept and present it at the next JF again.

Neumann, Fred [fneumann], March 13, 2017: Is this a realistic, practical example? Who would build a random test where one selection rule is a complete subset of another? If "Multiplikation" is a sub node of "Grundrechenarten", then you can build a rule for this node and a rule that selects 10 questions from all other nodes (node selection is or-combined).
The current fallback mechanism makes a random test unusable for exams. A restriction to the existence of exclusive questions keeps control for the maintainers of question pools and tests. They must in any way think about what they choose for which purpose and how to organize their pools appropriately. They should not be surprised by a hidden "magic" in the selection process. If no better concept is found, then the restriction to stable configs should at least be supported by a setting "random selection for summative test". If this switch is not set, then a big "DANGER" warning should appear with the validation message.

Heyser, Björn [bheyser] 13 March 2017: I made some mathmatical experiements and extended our current algoritmus. It is a draft only, so please do not afraid when you do not understand it instantly.

Exclusive Amounts would be added in the calculation as well of course, they are not considered here, because they are all empty.

Migration:

Existing tests will be set to dirty, so a re-synchronisation including a build check will be performed for tests prior to its use. For tests with existing participant data this isnt possible. A new indicator outdated will be introduced that freezes the test, so it is not executeable any longer. It cannot be repaired and is just left for archiving purposes.

JourFixe, ILIAS [jourfixe], March 27, 2017: We would like to thank Björn for this improvement of the concept and schedule the feature for 5.3. We hope that the revised algorhythm coveres all possible combinations of random test selection rules.

8 Implementation

The implemantation has been done like described above. The behaviour the random test is configured and taken by the participants has not been changed at all. Only the mathematical way a random test pass is drawn for a participant has been enhanced. During the configuration of a test the selection config gets checked if a test pass can be drawn for sure in every random combination of questions considering every selection rule and its definition for a question amount.

When drawing a test pass for participants is not surely possible a feedback is shown that provides information about question amounts per selection rule. Even intersections of questions from different selection rules having the same pool are considered.

Test Cases

Test cases completed at 24 Aug 2017 by mbecker

• C18724 : Fragenziehung aus Fragenpools mit instabiler Selektion
• C18725 : Fragenziehung aus Fragenpools mit stabiler Selektion

Approval

Approved at 24. August 2017 by Enst Huber.