Show Advanced KnowledgeHide Advanced Knowledge

Introduction of Question Groups

This is a sub-entry of the feature request Competence-driven Question Groups and Test Parts

1 Initial Problem

At the moment, it is not possible to group questions logically together. There are many scenarios where grouped questions offer great opportunities especially when random tests ans self-study settings are concerned. An often asked form use case at our University is probing the learner's knowledge concerning a certain topic with different questions that build on one another. Another scenario is that instructors want to present a set of questions to the learner that address a single problem but fron different angles or with different levels of complexity.

Achieveing this in 5.2 without dirty patches and hacks is very painful. One way of doing this in tests is by using a fixed set of questions and a fixed order. When it comes to self-study scenarios and large amount of question groups taht are supposed to be presented to the learners as self-evalution bundles in randoms tests, this is not a solution. There is virtually no way of realizing the logical grouping in random tests. Even taxonomies do not provide the means to achieve grouped questions.

2 Conceptual Summary

2.1 Overview

• Instructors should be able to group questions "logically" in a test. That means that a set of questions should be "played" back as a unit or cluster.
• If a test a random test, the entire unit should be drawn an not single question.

The community discussion has shown that there are several aspects and lots of ideas around the concept of questions groups. Here is a list of feature wiki pages connected with this feature (we will deal with point 1 now):

1. this page: introduce question groups for tests and question pools
2. Display Multiple Questions from a Question Group on One Page - visual grouping auf questions / develop a page-based concept
3. Use Question Groups for Competence Estimations / Measurements - implications for the competence system and the results processing

2.2 General Features

1. question groups can be created and edited in both questions pools and tests
2. a question group constists of several test questions, it acts as a "meta-structure" / "bracket" for a set of test questions
3. logical assumptions:
   • only questions from the same question pool or test can be added to a question group
   • flat hierarchy: a question group cannot be part of another question group
   • questions that have been assigned to a group cannot be added as single questions (they will not be shown in the relevant selection lists)
4. a question group is selected and added to a test "en block", that means that all questions from this block are "inserted" in the test in which it is used / another pool
5. question groups are preserved under copy operations for tests and questions pools and can be copied / moved from one pool / test to another
6. basic metadata set of a question group
   • text: title
   • text: description
   • text: author
   • date: created / modified dates

2.3 Creation of question groups

Question Pools

the creation could be done in two different ways:

1. [preferred] extending the question management list of the pool with a "create question group" button and then adding question to the group from the set of existing question or select existing questions and the table action from the dropdown "Add to new question group"
2. create new subtab "Manage Question Groups", list question group there, enter editing of question group settings and assign questions from the pool

Mockups Creation in Question Pool

Test

creation of question groups in the test is can be done in two ways on the subtab  "List View" (tab "Questions"):

1. by using a new button (to be introduced) "Create Question Group" (same procedure as in question pool "Manage Question Groups" > "Create Question Group")
2. though a table list dropdown action "Add to new question group" (same procedure as in corresponding question pool feature)

• the pool / test shows an error message when a user tries to add a question to a group that has already been assigned to a group
• set of settings of a question group:
  • Title
  • Author
  • Description
  • Playback Mode {Inherit from Test / Pool, Inner Shuffle, Fixed Order, Fix by Foreward}

Mockups Creation in Test

2.4 Presentation and Question Group Page Content

• question groups are visually marked in the "question list" and test dashboard if activated
  • the group title is shown in line with the singular question names
  • the question  names of the questions within the group are slightly indented
• group title
  • the group should have a common title
  • setting of the question group: the group title can replace the individual question titles, which are then not shown during playback

Mockups Questions Groups Used in Test

Mockups Questions Groups: Playback and Results Page

2.6 Operations

operations for questions groups with in a question pool (this is the maximally complicated version of these operations - we might have to simplify through policy decisions)

• copying a question group:
  • tree selector shown to choose target question pool
  • the questions are copied and a new group is created which contains the same questions
  • settings / sorting are copied as well
• moving a question group to another question pool:
  • tree selector shown to choose target question pool
  1. new group is created in target pool / test with the same settings
  2. questions are moved to target and assigned to the new group
  3. old group is disbanded in the the original pool
  4. questions are deleted from original pool
• disbanding a question group:
  • users are asked on a confirmation screen, whether they want to disband the question group or not
  • group is deleted and the questions live on as standalone questions

2.7 Import / Export

• Import and Export should work in the normal ILIAS way

• QTI-compatibility is NOT neccessary
• when a question pool / test is exported and imported the questions groups are preserved
• references of questions are kept as references (with the new pool) after importing a pool
• a question group only contains questions with status "final". when a question's status gets changed, the lifecycle concept should consider to fire an event. This way the concept for question groups can consider to handle this event by removing the question from all related groups.

3 User Interface Modifications

3.1 List of Affected Views

Question Pool

• Tab "Questions": new button "Create question group, some changes to question list table and to filter
• new views for question group editing: Edit Page, Preview, Edit Settings, Edit Question List

Test

• Page View: Jump to question dowpdown with indentations, display of commont page content above / below content of question
• Question selection after hitting "Add form Pool" / "From another Test":
  • introduction of two separate tabs: "Single Questions" (existing view where wquestions are selcted from pools) and "Question Groups" (new view)
  • "Question  Groups": table with regular filters to select question group
• new views for question group editing: Edit Page, Preview, Edit Settings, Edit Question List
• playback of questions during test: displaying of common group page content above / below the question
• question list (if activated):
  • new separate line for questiongroup title
  • indentation for grouped question titles beneath group line
• Detailed results: new column "Question Group" in table of question results

3.2 User Interface Details

3.3 New User Interface Concepts

There are no newly introduced user interface elements involved in this concept. Many common elements like tables and question presentations are reused.

4 Technical Information

Like every additional functionality this featue of course will cause some additional load for the server, but since my refactorings did take care of bad developed question data loading constructs, I am not worrying about significant issues regarding performance.

In general we have no technical depency, but one small aspect is to be named. Since this concept aims to introduce an additional content page level that wraps the question and its own surrounding content page, there might be a technical depency in the content page service.

Rendering the question and its own content page first offers the opportunity to embedded the corresponding html snippet within another higher content page by using the question element that normaly intents to process a html snippet representing the question container only. This way it should work. Perhaps the corresponding page object property and its setter/getter could be renamed to match the more generic case of embedding html snippets, but this is not technically required.

For fixed or randomly sequenced test modes the playback of question groups can be of course implemented in a way that matches all requirements for the different scenarios that are described with this feaure request.

For the continuous testing mode (aka. endless/dynamic/liquid test) it might be also possible to support question groups from a conceptional point of view. There is at least one technical issue, question groups references questions that are not guaranteed a set of independent questions, since the duplication of questions for use in the test does not take place at any time.

I can imagine solutions to solve this, but since this leads in larger refactorings of the test player for this testing mode that also needs a lot of funding, i suggest to not support question groups with the CTM in the first implementation. Imho it makes more sense to wait until someone have a strong need for this, because we will have a more realistic understanding of what we are thinking about.

5 Contact

• Author of the Request: Glaubitz, Marko [mglaubitz], Universität Freiburg
• Maintainer: Heyser, Björn [bheyser]
• Implementation of the feature is done by: {The maintainer must add the name of the implementing developer.}

6 Funding

If you are interest in funding this feature, please add your name and institution to this list.

•  the project "kosmic - KompetenzOrientierte Selbstlernangebote für Mathematik, Interkulturalität und Chemie" Universität Freiburg is interested in funding parts of this
• the Uniklinik Göttingen is interested in funding parts of this

7 Discussion

09.05.2017, Glaubitz, Marko [mglaubitz]: I have integrated the result of the SIG EA VC meeting on 4th May 2017 into the body of the feature wiki article. Basic change was: the development shold be split up into at least three steps (see above). We will try to realise Step 1 for ILIAS 5.3.

Neumann, Fred [fneumann], May 10, 2017: We forgot an important feature yesterday. It should be possible to assign a question group to the taxonomies in the question pool. The assignment elements on the "edit" page can look like those on the form of a question. And the list of question groups should get a taxonomy filter and optional tree navigation like the list of questions in the pool. The selection rule for question groups in a random test should have taxonomy conditions like the selection rule for single questions. This will allow case-based tests with a random selection of cases from a taxonomy-structured pool and a fixed set of questions per case. Question groups should not fall behind the current random selection possibilities of single questions.

Heyser, Björn [bheyser] The maintainer strongly supports this feature request, since several higher scenarios are depending on this basic feature, that are strongly needed by the users. Question Groups offers the basic possibilities to realize multi question pages, key feature sequences, complex configurations with different behaviours for certain parts of the test like shuffling questions within a block that keeps a unit for itself, or to keep the static order for one certain question block.

Since it is a complex feature that curretnly depends on the fact that questions are duplicated within the consumer objects, so even questions added to a group multiple times are technically different questions, i suggest to left oit the support for CTM at this time. When there is a strong need we can clarify problematical aspects. At least a refactoring of the CTM is required that uses question duplicates that are created removed on demand to reach a compatibility level. Additionally required funding will be needed but more important the remaining time for implementations is strongly restricted. Additional capacities wont help, since it is to be integrated in one trunk version together with all other implementations finally.

Killing, Alexander [alex], 3 July 2017:

• "At the moment, ILIAS only offers competence measurment in the test based evaluating single questions in order to derive conclusions as to which competences have been acquired by the learner. In many cases, it is virtually impossible to measure a competence (level) on the basis of the results of a single, isolated questions": Is this really the case? I thought it is easily possible to assign multiple questions to a competence like in the survey feature.
• "visual grouping auf questions / develop a page-based concept (similar to the survey)" In surveys this is only visual. The survey has a page concept for multiple questions being presented on the same page. This has nothing to do and is completely separated from competence assignments. We should not mix these things in my opinion.
• "only questions from the same question pool can be added to a question group with that question pool": If groups are "like in the survey" they would not be defined in a pool, since it is a presentation question being tackled in the survey (or the test).
• "a single question can be part of multiple questions groups": this is not possible in surveys. As I understand this means, the same question can be asked twice in a test. If you use competence to question assignments this could lead to obscure calculations of competences in my opinion. Can the author assign these two instances of the same question question to different competences? Or is it one assignment that gets an implicit double weighting?
• How are the group / competence assignments edited/managed? Are there any screen designs for this?
• The status ("distributed", "outdate", ...) is a separate FR as far as I know, correct?
• Is copy/move to test part of the proposal? What happens now and what happens in the future, if tests support groups?
• What does the playback mode, if blocks are not supported in tests yet? (It should be possible to add groups to test, but not to edit them. Questions of groups in tests can not be deleted)
• The page editor allows to insert other components, also between questions. How should this work?
• How relates the playback mode of the groups to the test mode?

JourFixe, ILIAS [jourfixe], July 03, 2017: We discussed the feature request but would like to separate this suggestion into several feature request to reduce its complexity:

• Please describe how questions pools are created and managed in question pools
• Please describe how they are used in the test (can they be edited or not?)
• Please describe how they are presented to users
• Please remove the hint to surveys as this suggestion is very different from the handling of multiple questions on one page in the survey.
• Please remove additional concepts like "lifecycle" that are not a general aspect of the feature itself.

Glaubitz, Marko [mglaubitz], 17.07.2017: The feature page has been overhauled incorporating the JF feedback from 3rd July 2017 and the work of two additional VC meetings on 13./14.07.2017. We have tried to tackle all issues that the JF desired us to elaborate upon. We hope to make it to the JF agenda on 31.07.2017.

Main conceptual adaptations:

• questions can only be part of one question group
• question groups can be created, edited and managed in both tests and question pools for tests
• we have abandoned the concept of a status for question groups for the time being (procedure for standalone questions needs to be fixed first)
• shared content: question groups get header / footer content, i.e. a separate PME "page" to be displayed above / beneath every question of a group

We would like to postpone the discussion of a visual grouping of more than one question on the same page to 5.4 together with a separate discussion on the conceptual possibilities and limitations for using question groups together with the competences.

AT 2017-07-17:I would like to re-iterate: 

There is an option to specify how many questions are minimally required to write a competence record. See screenshot in my comment above.

1. Why is this not good enough? Please outline where this approach comes short. What is it exactly that you cannot do if a competence record will only be set after x questions are answered? 
2. What is speicifcally gained by this approach that justifies adding  this much complexity? Please outline what is the specific added value of question groups. 

Killing, Alexander [alex], 19 Jul 2017: Thanks for the additions. It would be really nice to have the screenshots in the page, like done for almost all other feature wiki pages.

• Page 5: Mixing groups and questions in the table puts limitations on column ordering and filter handling. How will the filter work exactly (e.g. if a group matches a title filter, will it's question that do not match be displayed)? In surveys it is not possible to sort any column, and I always thought this would be a sign that something is wrong with this approach.
• Page 10: On page 5 it seems, that questions and groups are managed on one screen. How will the "Create Question Group" workflow on this screen work? Where will the group be added? How will its position be managed?

Glaubitz, Marko [mglaubitz], 28.07.2017: Thanks to alexandra and Alex for the feedback. I have integrated the mockups into the feature page.
Regarding Alex's questions:

1. Filter for this screen: the name filter will match both standolone questions and question groups, but I gues I do not really understand what you meant by this question.
2. Again, I don't know what you mean exactly, but I'll try to clarify something reagrding the question pools. We have thought of two alternative ways different way of creating and managing question groups. The JF should decide for one. We prefer Option 1 (I have only included screnshots / mockups for option one into this feature page, option two only lives in the PDF file).

JourFixe, ILIAS [jourfixe], July 31, 2017: We highly appreciate this request and schedule it for 5.3. We would like to have the following changes:

• We should only use 'Question' and avoid 'standalone questions' or 'single questions'. So we have 'Questions' and 'Questions Groups'.
• Question groups should be activated in Administration. People that do not want to use question groups do not see related UI changes.
• List of questions: instead of one column with questions and questions groups with indented questions we want to have two columns - one for questions and one for questions groups in case the question is assigned to a question group. This would allow a better support for sorting and filtering.

Berggold, Christine [berggold], 15.12.2017: We highly appreciate this feature! We already suggested the related feature "Different test parts for randomized tests" almost 5 years ago. The possibility to show different question groups in only one test is very important for our online examinations. With question groups we could present different topics in one test, instead of using different tests for each topic. This means also easier test administraion and reporting, better navigation for the students...

Gerl , Stefanie [stefanie.gerl], 18.12.2017: Members of SIG EA discussed this feature again at their meeting in Cologne in November and supported it. For several assessment settings (key feature test, fix by foreward, inner shuffle, fixed order) this feature is indispensable for further development of the T&A. We need this feature for ILIAS 5.4.

Lauener, Hansjörg [lauener], 3.1.2018.

We (University of Bern) have two main issues with this feature:
1.    It adds a big new layer of complexity for the user.
2.    It adds an even bigger layer of complexity for the maintainer.

Especially the second point we regard as very critical since 1) the amount of open issues concerning the test is overwhelming. 2) From the maintainer we hear a lot of concern about the massive amount of problematic legacy code inside this component. However the other point is critical as well, since 3) the test as it is suffers from a huge feature bloat rendering it very hard to operate. We believe that those issues have to be tackled first before adding such a huge new layer of complexity on top of it.

Concerning the additional layers of complexity for the end user, here some examples:
- How does a test-admin decide, wheter he wants to use Question-Groups or not, what are the (dis)advantages?
- What is the difference if you use Question Groups in "Fixed Set of Questions", or in Random-tests?
- New functionalities to (un)define Question-Groups in the List-view
- New sorting-functionalities in the List-view.
- Add Cut/Copy/Delete functionalities in the Page-View (if introduced, then Question Groups should be available also in the Page-view!?)
- Underworlds are difficult for users to understand. Underworlds for each Question Group and individual settings for each Question-group (Playback, Modes).
- ...

Further we have some thoughts to share about the didactical purpose. In the initial problem-part of this FR, multiple didactic motivations are addressed.

a) A first didactic motivation (see part 1 of FR) says: "An often asked form use case at our University is probing the learner's knowledge concerning a certain topic with different questions that build on one another."  - AND - " Achieveing this in 5.2 without dirty patches and hacks is very painful. One way of doing this in tests is by using a fixed set of questions and a fixed order. When it comes to self-study scenarios and large amount of question groups that are supposed to be presented to the learners as self-evaluation bundles in random tests"

=> As far as we see, building different questions that build on one another AND create a  self-study scenario with a large amount of questions AND present them to learners as self-evaluation bundles in random tests is not an often used scenario. We think: It is a good way to create a fixed set of questions and order, when you want to build questions that build on one another.


b) A second didactic motivation says: " Another didactic motivation for the introduction of question groups are the so called "key feature tests" that are widely used in medical education and that allow to test procedural knowledge very well. The idea of a key feature test is basically to present a "chain" of questions to the learner in which questions build on the correct answers of previous questions. A popular example are tests in the area of medical diagnosis: the student has to decide in the first question what to do with a patient's story and how he/she interprets it. They answer the question and the question is then fixed, because the correct interpretation of the patient's story is given in the context of the next question."

=> There is a study from the university of Bern, where KF-scenarios are probed. They used the CAMPUS assessment software as an assessment tools, together with the medical faculty of Heidelberg  (http://www.medizinische-fakultaet-hd.uni-heidelberg.de/CAMPUS-Lehr-und-Pruefungssystem.110009.0.html). Two Key-Feature examples, made as a formative scenario with CAMPUS-software can be found here: http://www.medizinische-fakultaet-hd.uni-heidelberg.de/index.php?id=109993&L=de

Those two cases in CAMPUS-software show, that for KF-scenarios, it is not necessary to have a assessement software where different questions are grouped on ONE page. The CAMPUS-software shows each question on a separate page, similar to ILIAS.

But those cases show also, that in KF-scenarios often content-pages without a question are used.  So in our opinion, also an improved ILIAS Learning Module could be a possible solution for KF-scenarios. And the ILIAS Learning Module is already able to "group" several questions on one page.


c) Third the FR says, that there "are many scenarios where grouped questions offer great opportunities especially when random tests and self-study settings are concerned".

=> At our university, the test is widely used in formative settings, but random tests and self-study settings are not respectively very seldom asked. We collect every feature-request from our teachers, but Question-Groups are never asked. Of course, we could activate this feature in the administration (JF-decision July 31, 2017), and this would be maybe our solution, because we actually think that there are already too much test-settings. So such an administration settings should not be the first solution for us, we are interested to have features that are easy to use, and to have features we can offer to all of our users. So it would be good to see whether other community members would like this feature.

AT 2017-01-04: The Technical Board called for options to decouple large components in ILIAs, one of them being the test. The aim is to improve the reliability.
Could we defer the decision on this feature untill we have weighed the options to move forward with the decoupling?
Once we know about the way forward we will be much more informed about repercussions of this feature. 
Please consider this suggestion to defer the decision. 

2018-01-08, Glaubitz, Marko [mglaubitz]: 
Firstly, Hansjörg, thank you for your remarks, although I disagree with some points that you made. The perspective and experience of the Uni Bern seems to be different from ours and many other institutions. I can only mirror that our users and the users of a majority of the SIG EA representatives have been waiting for the possibility of grouping questions (and as a second step) displayig them on one page for some years now.
In addition, the SIG EA has discussed the issue of code quality / refeactoring vs. the introduction of new features at great length and we have decided that a certain list of features should be realised before the maintainer focuses on refactoring. Please have a look at the SIG EA's statement reagrding this issue
We have discussed the issue of complexity and improving the existing code at great length and have asked Björn to consider this in his offer for the feature. This means that a certain part of the costs will be availablefor the maintainer to improve code if need be.

However, I strongly reject Alexandra's proposal to defer the decision, since this would mean that the feature will not be available before 5.5 or later. This is not an option for us and many other institutions. If we can not tackle this feature for 5.4, we will have let it go for good. That would be a great pity in terms of lost funding and the work many people of were involved in developing the extensive concept for this feature.

Killing, Alexander [alex], 10 Jan 2018: I am also tempted to vote for postponing the decision, but not necessarily to the 5.5 release. In addition to Alexandras reason a main point for me is the combination of

• the past critisism of the T&A quality,
• the increased bug votings in the T&A (the number doubled in the last year from around 15 votings to more than 30, second place (per component) is currently the course management with 7 bug votings) and
• the fact that we should try to put higher quality standards on something that provides online exam functionality.

As I understand there is currently even funding for bug fixing available. I would like to see this being spent first in the next weeks and to get the number of bug votings going down significantly before making a decision on additional complexity.

JourFixe, ILIAS [jourfixe], 10 JAN 2018 : We had a long discussion about this feature request and its implecations on the test & assessment in general. We need to postpone a final decision about this feature request until we know the results and recommendations of the T&A revision (which will be available end of March at the latest).

8 Implementation

{The maintainer has to give a description of the final implementation and add screenshots if possible.}

Test Cases

Test cases completed at {date} by {user}

• 18615 : Fragengruppe in einem Fragenpool erstellen

Approval

Approved at {date} by {user}.