Show Advanced KnowledgeHide Advanced Knowledge

Taxonomies for Test Questions

1 Requirements

It would be very helpful to use the new taxonomy feature for test questions, too. This would allow ...

1. to structure questions within a question pool according to specific criterias (thematically, didactical level, ...)
2. to ease the assignment of the right questions when creating a test (probably to extend the random test and to use only questions of the same taxonomy node)
3. to present user solutions according to assigned taxonomy nodes

Filtering should not only be restricted to one category but allow combinations from different branches of the taxonomy tree. Example: one branch of the tree is representing topics, another the levels of difficulty and you want to select questions that are assigned to "Internation tax law" and "Expert knowledge".

1.1 Already Funded

At the time being (May 15) the following functionality is already funded and will be implemented for FH Münster:

1. Questions can be assigned to nodes of one or more taxonomies.
2. The question pool offers filters to present only questions of the selected taxonomy nodes.
3. The test object can use taxonomies in the new "Continuous Testing Mode". Here it will be possible for test participants to select taxonomy nodes so that only questions assigned to (one of) the selected nodes are presented in the test.

All other features described above are not funded yet.

1.2 Funding Needed

To complete the feature in a first version and to make it also usable in normal tests it is at least necessary to implement the following features, too.

1.2.1 New Test Setting "Taxonomy Support"

A new test setting "Taxonomy Support" allows to use the different features related to taxonomies in the test object. This is another option on the screen "Settings » General" but it reduces complexity for the author and participant on other screens (because it hides this additional functionality if it should not be used).

1.2.2 Controlling the selection of questions in random test via taxonomies

Taxonomies could be used to control the random selection of questions from a question pool in random tests. The following scenario should be supported:

Preconditions:

• Test setting "Random Selection of Questions" is selected (like now).
• Test setting "Taxonomy Support" is selected (new, see above)
• User has read access to at least one question pool (like now).
• At least one question in the question pool is assigned to a node of the pool's taxonomy.
Use Case "Selecting question according to taxonomy":
1. User enters tab "Questions" (Random Selection)
2. ILIAS presents all pools the user is allowed to use in drop-down of option "Source Question Pools".
3. User selects a first question pool from drop-down.
4. ILIAS is checking in the background (Ajax) if this pool has at least one taxonomy.
   1. If no taxonomy exists, nothing happens.
   2. If a taxonomy exists, a new drop-down for each taxonomy and the number of questions assigned to this node is shown (similar to the current presentation of question pools in the drop-down).
5. User selects one taxonomy node from the drop-down and defines the number of questions that should be taken.
6. If another taxonomy node of the same pool should be used to select more questions, the user clicks the "+" icon of the multi-input field of the taxonomy nodes and continuous with step 5 (no ajax check necessary as nodes are already loaded).
7. If no other taxonomy node in this pool should be taken to select questions, the user
   1. can continue to choose the next pool (using the "+" icon of the multi-input field and continue with step 3)
   2. or finish the pool selection (by clicking on "Save").

1.2.3 Presentation of User Solutions According to Taxonomy

If test questions are already assigned to taxonomy nodes, it would be interesting to filter the user solutions of a test according to assigned taxonomy nodes, e.g. "Show all answers for test questions assigned to «fractions». This feature is independant from the one above and would also work for other than random tests.

Precondition:

• Test setting "Taxonomy Support" is enabled. (new)
• User has completed test and has access to "Test Result" page. (like now)
 
Use Case "Show Test Results According to Taxonomy"
1. User enters page "Test Result".
2. ILIAS presents page "Pass Overview of the Test Results".
3. On this page there is an ILIAS filter "Presentation" with a checkbox:
   • Show test results according to taxonomies
4. If selected, ILIAS extends the view of this page by adding a new section below called "Test Results According to Taxonomies":
   • For each taxonomy node that is used by at least one question ILIAS presents the following:
     • {Title of taxononmy node as headline}
     • Result table (like used above but without column "Scored Pass") with all passes.
   • Each result table is showing only results for the questions related to this taxonomy node.
   • A user can click on the Actions menu in each result row to get a detailed view on the questions assigned to this taxonomy and to the given answers and reached points (like already now for each pass)
As the screen for the test results is available for the participant of a test as well as for the tutors, both groups can use the advantages of such a detailed and topic-related presentation of test results.



2 Status

• Scheduled for: Release 4.4
• Funding: FH Münster and DHBW
• Development: Feature is to be developed by Databay AG (and Leifos)

3 Additional Information

• If you want to know more about this feature, its implementation or funding, please contact: Matthias Kunkel

4 Discussion

JF 5 Feb 2013: We appreciate the feature and schedule it for 4.4.

The optes Project (and DHBW Mannheim) are very interested in this feature! If further funding is needed, please contact K. Derr, katja.derr@dhbw-mannheim.de or R. Hübl, reinhold.huebl@dhbw-mannheim.de

Matthias Kunkel, 26 May 2013: I would like two clarify two conceptual aspects of this feature:

• How is the general relation between taxonomy and tag (keyword)? I fear it is unclear which of the both features fits for which purpose best. Example: You have a question that is related to "even-toed ungulates" and classified as "advanced knowledge" for students. While "even-toed ungulates" fits perfect to a proper node of an existing taxonomy for animals, the second is more a tag (or keyword) than a taxonomy node (because attributes like easy, difficult a.s.o. are not nodes of a taxonomy but simple labels). If I am right, this would mean that we need not only taxonomy support for test questions but also tag support.
• What is the impact of assigning one question to a specific node of a taxonomy? If I select "Mammals" to filter all animal related questions in my big biology question pool, do I see only questions that have been assigned directly to the node "Mammal"? Or do I get also questions that are assigned to sub-nodes of "Mammal" - like "even-toed ungulates" (same for using taxonomies in a random selected test)?
And a last but very important question for the daily use of this feature: as far as I have understood, each question pool uses its own taxonomies and there is no re-use of "global" taxonomies yet. Does this mean that I have to create a new taxonomy for each biology related question pool? Or will it be possible to copy a taxonomy and import it in another question pool?

JF 27 May 2013: It should be possible to use one taxonomy as a left hand navigation taxonomy in the pool (like the taxonomy in glossaries). The availability of this feature is activated by the consumer component but managed by the taxonomy service. Also the possibility to use a taxonomy for ordering attached items is managed by the taxonomy service.
 
If taxonomies are used for filtering/navigation the default behaviour is that all items that are assigned to the selected node or to one of its subnodes is retrieved in the results.
 
In the long run we would like to support taxonoymies and "advanced meta data" for question. This gives the author both possibilies, to use a structured taxonomy or a list of keywords (or a one-level catgorization presented by a drop-down list).
 
Currently funding is settled for the features under point 1.1 - at the time being only these features will be developed for 4.4. The Optes project will contact Databay and clarify whether the implementation of the unfunded features (point 1.2) is possible for 4.4. We would highly prefer to have these features available with ILIAS 4.4, too.

Matthias Kunkel, 31 May 2013: I have added two use cases for the features to be funded by DHBW Mannheim.

MJ 03 Jul 2013: I have added a screenshot which should describe the question handling in ramdom tests (which was subject of the meeting on 02 Jul 2013 with Matthias Kunkel and Alexander Killing)

MJ 03 Jul 2013: Minutes of the "Taxonomies for Test Questions" meeting
 

Selection of questions in random test via taxonomies

• Taxonomies (incl. tree structure) have to be duplicated if the questions were duplicated into test.
• Taxonomies and assignments to questions cannot be changed in the test itself.
• The date of the last sychronisationhas to be stored and displayed (Toolbar, ilUtil::sendInfo()) in random tests.
• Some detailed information concerning the moment of question synchronisation (question pools to test) should be displayed.
• The selection process of question pool/taxonomy/node should be implemented according to the draft above and results in a multi step creation workflow (overview data table, forms for adding new datasets of question pool/taxonomy/node/number of questions). A cascaded ajaxified multi select form element should not be impelemented.

BH 29 Sep 2013:
 
While implementing this feature in the discussed way a significant issue regarding the question selection algorithm for a learner's test pass came up, when question amount configuration mode "per pool definition" is selected. There are no open issues for random question sets with an amount configuration mode "per test".
 
During the configuration process, on every change, it is required to check wether a random test pass for learners can be created or not, so the completion status of the test can be determined. This check have to be done in the same random way, which is used to create an actual pass. But in addition we have to be sure, that every random combination of questions leads to a valid result set of questions fullfilling each declaration of question amount per pool definition.

Example Configuration Draft

• TaxRandTestSkizze.pdf (229.86 KB)

• The first example contains two definitions with the same pool but differing taxonomy filter selections
• If the selection (check) has to be done according the order of definitions, we cannot be sure that every random combintation of questions leads to valid result
• A random selection becomes valid when question C and D is selected "randomly" for the first definition
• An implicit re-ordering of the definitions relating to the tree depth of the filtered tree node while selecting randomly could sove the problem in this example
• A general validation for a unique question stage could solve the problem

 

• The second example looks like not very senseful but there are thinkable scenarios using such a configuration
• Only the last suggestion above would solve the problem for this configuration, since the definitions involves different taxonomy trees and we cannot re-order the the definitions comparing tree depths

 

• Mentioning the aspects described above, we have to reconsider the relevance of a possible ordering mechanism for the source pool definitions as it is implemented at the moment

• There are no problems working off the source pool definitions, when the pool itself differs

 
Demo installation is available here http://ilias.heyser.ac

JF 30 Sep 2013: If questions are assigned to multiple taxonomy nodes, a heuristic to get a set of questions that fulfills all rules my be difficult or even impossible in some cases. There are two possible ways to solve this

1. The rules are respected exactly, but the overall number of questions may be lower than the sum of the number of questions of all rules (some questions fulfill more than one rule).
2. The overall number of questions in the test is equal to the sum of number of questions of all rules, but the number of seleted questions per rule may be higher (the numbers in rules is to be read as a "minimum" number of questions).

We think that only the second solution is acceptable.

Presentation of User Solutions According to Taxonomy

• A user with advanced priviledges (probably: write access) should be able to configure a list of those taxonomies, which should be availabe as a filter in the pass details' question table. The basic set of taxonomies results from the question pools selection.
• The pass details user interface has to be revised:
  • In the first step, the user has to choice the pass from the list of passes in the data table.
  • After selecting a pass, a new table with all questions appears, followed by the detailed list of answers.
    • The question table has a filter panel offering the available taxonomies configured in the test settings.
    • A back button should be offered to return to the list of passes.
    • The final remark (mark corresponding to the mark schema) should be presented more prominently.

General

• Maybe the advanced metadata feature could be added to test questions, because easy, medium, hard is more a metadata than a taxonomy. (Update: No desired in the first implementation)
• We do not support import/export of taxonomies and the corresponding assigments to questions (postponed to future releases).

Matthias Kunkel, 07 July 2014: Tests with a fixed selection of questions are currently not supporting taxonomies (optes branch). Therefore, a presentation of user solutions is missing in such tests. IMHO, this helpful feature should be available for all test modes as it makes sense in all of them.

Jour Fixe, 7 Jul 2014: We postpone the discussion to the next JF where Mattthias will participate.

JF 21 July 2014: @MK and BH: Please provide a screenshot of a test result screen /mode that supports taxonomies and illustrates how they are used and a screenshot of a test result screen / mode that lacks this support. How should the result screen be changed? We postpone the discussion to a JF where Matthias and Björn participate.

JF 6 Aug 2014: We envisage the following workflow for fix question tests using taxonomies:

• The taxonomy feature must be activated in the general test settings screen.
• If activated a new taxonomy tab occurs.
• Initially the tab includes an option (button) to load the taxonomy information from all pools related to questions in the test.
• Loading this information will make it impossible to change the question set of the test, without removing the taxonomy information before again.
• Additionally it is possible to "reload" the taxonomy information at any time.
• Information messages should make the implications listed above clear.
• A question list should show the current taxonomy node assignment.

5 Implementation

MJ 21 Aug 2013: The functionality mentioned under 1.1 (Already Funded) is already implemented and available in the v4.4.0 Pre-Alpha release.

5.1 Implementation for 4.4

Currently the options listed in the section "1.1 Already Funded" are implemented as described below. If you are interested in using the continuous testing mode have a look at the implementation details on the correspinding feature wiki page: Continuous Testing Mode

How to manage taxonomies with the question pool?

• Create a question pool object as usual an open the new subtab screen "Taxonomies" within the settings tab
• You can manage multiple taxonomies with different properties and purposes
• Taxonomies can be marked as useable for navigation
• An manual sorting of questions assigned to the taxonomy can be activated for the taxonomy



How to assign questions to the taxonomy within the question pool?

• Within the question editing screen there is a new form section for taxonomy assignments (only for questions in question pools)
• The question can be assigned to one or more taxonomies one or multiple times



How to order questions within a taxonomy maually?

• After creating a taxonomy and after assigning questions the questions are sortable per taxonomy node
• The sorting of questions within the taxonomy is managed within the taxonomy settings screen again
• Edit an existing taxonomy an open the "Assigned Items" screen



What are the benefits of having question pool taxonomies for the pool itself?

• Taxonomies can be shown in the question pool by activating the corresponding setting
• If shown, you can use the taxonomies to filter the question list within the question pool
• If shown, the taxonomies are provided as table filter except the taxonomy that is selected as navigation taxonomy

How can I manage filters for the participant's test result presentations based on taxonomy assignments?

• Test result filters based on taxonomy assignments can only be used with random tests with the current state
• A random test with at least one source pool definition, that includes a pool containing at least one taxonomy, is required
• After configuring the random question selection, any of the involved taxonomies can be selected on the scoring and results settings screen
"Settings" Tab > "Scoring and Results" > "Misc" Form Section



BH 09 Oct 2013: The functionality mentioned under 1.2.2 (Selection of questions in random test via taxonomies) is already implemented and available in the v4.4.0 Pre-Alpha release.

MJ 09 Oct 2013: The feature mentidones in 1.2.3 (Presentation of User Solutions According to Taxonomy) will be part of ILIAS v. 4.5.x and the optes-branch.

5.2 Implementation for 5.0

BH 22 Jan 2014: The feature mentioned under 1.2.3 (Presentation of User Solutions According to Taxonomy) has been implemented according to the discussion under 'Presentation of User Solutions According to Taxonomy'. It is now available in the optes branch and will be merged for 4.5 as soon as the trunk is open for 4.5 development.

5.3 Implementation for 5.1

Heyser, Björn [bheyser]: Export of question pools with taxonomies is supported with ILIAS 5.1.

×
Full Screen
Cancel