Rated question selection in variable random tests

1 Description

In the VSP context, question pools typically include 15 to 40 questions on a topic. The self-tests are configured in a way that 10 questions each are selected randomly from a pool. By random selection of questions, it is very likely that you haven't seen every question from a pool even after several test passes.
However, both the coaches and the learners have an interest in all question are processed at least one time.
 
Desired functionality:
The questions are selected randomly from a pool but the process only considers questions which were never used in a previous test pass of the user.
You would need to clarify the question of what happens when all the questions have already been selected once. That could be solved by only considering question that have been selected less often than average.
 
Idea:
A kind of "rating" is introduced which simply counts how many times a question has already been processed (either via an extra table or via a database query). The random selection could take that into account and choose questions that have been used less previously (as a result of the rating, the probability of not having been selecting for a specific user is higher for those questions).
 

2 Status

• Scheduled for: Release 5.0
• Funding: Helmut Schmidt Universität Hamburg
• Development: Feature is to be developed by Databay AG

3 Additional Information

• If you want to know more about this feature, its implementation or funding, please contact: suittenpointner (at) qualitus.de

4 Discussion

JF 26 Sep 2011: We had a longer discussion in the JF and are unsure, how the feature should be implemented in detail. Please get in contact with the test maintainers (Helmut/Databay) and ask the programmer who would like to implement this to participate in a Jour Fixe. We will discuss this together.

Florian Suittenpointner 25 Oct 2011: One additional requirement is where to activate this feature.
We would prefer to add a checkbox "Rated question selection" in the "Question Selection" section of the question pool selection screen (the "Questions" tab in tests with variable random selection). The checkbox should be accompanied by a rough description of the algorithm.

Jf 7 Nov 2011: The feature should be optional: Under "Random Selection" "Prefer least used questions". The option separates all questions in two sets. The first set contains all "least used questions" the second one the rest. ILIAS now selects questions with a probability of 0.7 from the first set.

BH, 13 Jun 2012: To my knowledge, this feature was designed to make sure, that a user will see all questions within a minimum number of passes. To achieve that, the algorithm of selecting the questions was designed without a random component. (Note: Random occurence vs. Random order of questions in the test.)
I am not sure if a 30% quota of questions repeated "prematurely" serves the purpose of the feature.
 
I am sure, though, that there will be more than two groups of weights in the question pool and this may have an undesired influence to the workings of the picking algorithm. With the 0.7-chance, it will be possible to have one question repeating in three subsequent passes while others have not been shown once, leading to groups with 0,1,2 or 3 previous occurences when pass 4 is initiated.

Florian Suittenpointner 18 Jun 2012:
Two proposals:
New logic of question drawing (probability version, JF proposal as understood by FS/BH)

1. Calculate the minimum value of "frequency of displaying a question" (= y)
2. For each question drawing, first a random number n between 0 and 1 is generated. If n < x % (default: 70%), the question is drawn from all questions that were selected with frequency y. Disadvantage: Must be calculated for every single question=> Performance issue?
3. If no questions with frequency y are left anymore, the remaining question are drawn from the group of questions selected by frequency > y.

1. Calculate the minimum value of "frequency of displaying a question" (= y).
2. From all questions that were selected with frequency y, n questions are selected randomly (n = x % of number of desired test questions). If this group is smaller than n (m), all questions of this group are selected.
3. If in step 2 less than n questions have been selected (m), so now the remaining questions (n - m) are selected randomly from the group of questions selected by frequency > y.
4. At last, the remaining questions (100 -x %) are selected randomly from the group of questions selected by frequency > y.

• The order of the selected questions in the test is determined randomly.
• Value x can be configured in the "Test & Assessment" administration.This is necessary because the customer for whom we originally created this feature wants to use it without the adaption discussed here but a purely rated mechanism.So, they can set value x to 100, thus making the test draw all questions by the rated mechanism.

JF 25 June 2012: We appreciate this development. We schedule this development for 4.3, since the open issues of the November discussion will be adressed. We do not have a strong preference for the two options listed by Florian. The threshold (here 0.7) could be an option in the test settings (suboption of "Zufällige Auswahl von Fragen").

BH 02 Aug 2013: The maintainers prefer to implement the alternative version Florian mentioned above.
 
Since the implementation for this feature wasn't contracted yet, this feature has to be scheduled again. We have proposals for a contract now.

02. Aug 2013, Heyser, Björn [bheyser]: Technical implementation in a branch:

• Manual configuration for activation and propability factor (x %) within ILIAS tests.
• Logic of question drawing:
  • Calculate the minimum value of "frequency of displaying a question" (= y).
  • From all questions that were selected with frequency y, n questions are selected randomly (n = x % of number of desired test questions). If this group is smaller than n (m), all questions of this group are selected.
  • If in step 2 less than n questions have been selected (m), so now the remaining questions (n - m) are selected randomly from the group of questions selected by frequency > y.
  • At last, the remaining questions (100 -x %) are selected randomly from the group of questions selected by frequency > y.
• The order of the selected questions in the test is always determined randomly.

BH 10 Sep 2013:
 
Since the feature Taxonomies for Test Questions involves the improvement of the questions tab for random tests (random test configuration's general and source pool settings are separated by two subtabs), I would like to have the settings dealing with the activation and propability of the rated selection on the general settings subtab screen within the questions tab for random tests.

JF 30 Sep 2013: We still appreciate the feature and accept the "VSP 100% version" without any percentage setting as a subfeature of the random test named "Prefer least used questions" (checkbox). The info text should make clear that questions are used only a second time if all other questions have been used at least one time. The option can be placed on the screen suggested by Björn above.
 
We keep the discussion open. If any user would like to make the randomizing configurable, e.g. as described by Florian above, additional funding may be needed.
 
We schedule this for 4.5.

5 Implementation