09.05.2017, Glaubitz, Marko [mglaubitz]: I have integrated the result of the SIG EA VC meeting on 4th May 2017 into the body of the feature wiki article. Basic change was: the development shold be split up into at least three steps (see above). We will try to realise Step 1 for ILIAS 5.3.

Neumann, Fred [fneumann], May 10, 2017: We forgot an important feature yesterday. It should be possible to assign a question group to the taxonomies in the question pool. The assignment elements on the "edit" page can look like those on the form of a question. And the list of question groups should get a taxonomy filter and optional tree navigation like the list of questions in the pool. The selection rule for question groups in a random test should have taxonomy conditions like the selection rule for single questions. This will allow case-based tests with a random selection of cases from a taxonomy-structured pool and a fixed set of questions per case. Question groups should not fall behind the current random selection possibilities of single questions.

Heyser, Björn [bheyser] The maintainer strongly supports this feature request, since several higher scenarios are depending on this basic feature, that are strongly needed by the users. Question Groups offers the basic possibilities to realize multi question pages, key feature sequences, complex configurations with different behaviours for certain parts of the test like shuffling questions within a block that keeps a unit for itself, or to keep the static order for one certain question block.

Since it is a complex feature that curretnly depends on the fact that questions are duplicated within the consumer objects, so even questions added to a group multiple times are technically different questions, i suggest to left oit the support for CTM at this time. When there is a strong need we can clarify problematical aspects. At least a refactoring of the CTM is required that uses question duplicates that are created removed on demand to reach a compatibility level. Additionally required funding will be needed but more important the remaining time for implementations is strongly restricted. Additional capacities wont help, since it is to be integrated in one trunk version together with all other implementations finally.

Killing, Alexander [alex], 3 July 2017:

• "At the moment, ILIAS only offers competence measurment in the test based evaluating single questions in order to derive conclusions as to which competences have been acquired by the learner. In many cases, it is virtually impossible to measure a competence (level) on the basis of the results of a single, isolated questions": Is this really the case? I thought it is easily possible to assign multiple questions to a competence like in the survey feature.
• "visual grouping auf questions / develop a page-based concept (similar to the survey)" In surveys this is only visual. The survey has a page concept for multiple questions being presented on the same page. This has nothing to do and is completely separated from competence assignments. We should not mix these things in my opinion.
• "only questions from the same question pool can be added to a question group with that question pool": If groups are "like in the survey" they would not be defined in a pool, since it is a presentation question being tackled in the survey (or the test).
• "a single question can be part of multiple questions groups": this is not possible in surveys. As I understand this means, the same question can be asked twice in a test. If you use competence to question assignments this could lead to obscure calculations of competences in my opinion. Can the author assign these two instances of the same question question to different competences? Or is it one assignment that gets an implicit double weighting?
• How are the group / competence assignments edited/managed? Are there any screen designs for this?
• The status ("distributed", "outdate", ...) is a separate FR as far as I know, correct?
• Is copy/move to test part of the proposal? What happens now and what happens in the future, if tests support groups?
• What does the playback mode, if blocks are not supported in tests yet? (It should be possible to add groups to test, but not to edit them. Questions of groups in tests can not be deleted)
• The page editor allows to insert other components, also between questions. How should this work?
• How relates the playback mode of the groups to the test mode?

JourFixe, ILIAS [jourfixe], July 03, 2017: We discussed the feature request but would like to separate this suggestion into several feature request to reduce its complexity:

• Please describe how questions pools are created and managed in question pools
• Please describe how they are used in the test (can they be edited or not?)
• Please describe how they are presented to users
• Please remove the hint to surveys as this suggestion is very different from the handling of multiple questions on one page in the survey.
• Please remove additional concepts like "lifecycle" that are not a general aspect of the feature itself.

Glaubitz, Marko [mglaubitz], 17.07.2017: The feature page has been overhauled incorporating the JF feedback from 3rd July 2017 and the work of two additional VC meetings on 13./14.07.2017. We have tried to tackle all issues that the JF desired us to elaborate upon. We hope to make it to the JF agenda on 31.07.2017.

Main conceptual adaptations:

• questions can only be part of one question group
• question groups can be created, edited and managed in both tests and question pools for tests
• we have abandoned the concept of a status for question groups for the time being (procedure for standalone questions needs to be fixed first)
• shared content: question groups get header / footer content, i.e. a separate PME "page" to be displayed above / beneath every question of a group

We would like to postpone the discussion of a visual grouping of more than one question on the same page to 5.4 together with a separate discussion on the conceptual possibilities and limitations for using question groups together with the competences.

AT 2017-07-17:I would like to re-iterate: 

There is an option to specify how many questions are minimally required to write a competence record. See screenshot in my comment above.

1. Why is this not good enough? Please outline where this approach comes short. What is it exactly that you cannot do if a competence record will only be set after x questions are answered? 
2. What is speicifcally gained by this approach that justifies adding  this much complexity? Please outline what is the specific added value of question groups. 

Killing, Alexander [alex], 19 Jul 2017: Thanks for the additions. It would be really nice to have the screenshots in the page, like done for almost all other feature wiki pages.

• Page 5: Mixing groups and questions in the table puts limitations on column ordering and filter handling. How will the filter work exactly (e.g. if a group matches a title filter, will it's question that do not match be displayed)? In surveys it is not possible to sort any column, and I always thought this would be a sign that something is wrong with this approach.
• Page 10: On page 5 it seems, that questions and groups are managed on one screen. How will the "Create Question Group" workflow on this screen work? Where will the group be added? How will its position be managed?

Glaubitz, Marko [mglaubitz], 28.07.2017: Thanks to alexandra and Alex for the feedback. I have integrated the mockups into the feature page.
Regarding Alex's questions:

1. Filter for this screen: the name filter will match both standolone questions and question groups, but I gues I do not really understand what you meant by this question.
2. Again, I don't know what you mean exactly, but I'll try to clarify something reagrding the question pools. We have thought of two alternative ways different way of creating and managing question groups. The JF should decide for one. We prefer Option 1 (I have only included screnshots / mockups for option one into this feature page, option two only lives in the PDF file).

JourFixe, ILIAS [jourfixe], July 31, 2017: We highly appreciate this request and schedule it for 5.3. We would like to have the following changes:

• We should only use 'Question' and avoid 'standalone questions' or 'single questions'. So we have 'Questions' and 'Questions Groups'.
• Question groups should be activated in Administration. People that do not want to use question groups do not see related UI changes.
• List of questions: instead of one column with questions and questions groups with indented questions we want to have two columns - one for questions and one for questions groups in case the question is assigned to a question group. This would allow a better support for sorting and filtering.

Berggold, Christine [berggold], 15.12.2017: We highly appreciate this feature! We already suggested the related feature "Different test parts for randomized tests" almost 5 years ago. The possibility to show different question groups in only one test is very important for our online examinations. With question groups we could present different topics in one test, instead of using different tests for each topic. This means also easier test administraion and reporting, better navigation for the students...

Gerl , Stefanie [stefanie.gerl], 18.12.2017: Members of SIG EA discussed this feature again at their meeting in Cologne in November and supported it. For several assessment settings (key feature test, fix by foreward, inner shuffle, fixed order) this feature is indispensable for further development of the T&A. We need this feature for ILIAS 5.4.

Lauener, Hansjörg [lauener], 3.1.2018.

We (University of Bern) have two main issues with this feature:
1.    It adds a big new layer of complexity for the user.
2.    It adds an even bigger layer of complexity for the maintainer.

Especially the second point we regard as very critical since 1) the amount of open issues concerning the test is overwhelming. 2) From the maintainer we hear a lot of concern about the massive amount of problematic legacy code inside this component. However the other point is critical as well, since 3) the test as it is suffers from a huge feature bloat rendering it very hard to operate. We believe that those issues have to be tackled first before adding such a huge new layer of complexity on top of it.

Concerning the additional layers of complexity for the end user, here some examples:
- How does a test-admin decide, wheter he wants to use Question-Groups or not, what are the (dis)advantages?
- What is the difference if you use Question Groups in "Fixed Set of Questions", or in Random-tests?
- New functionalities to (un)define Question-Groups in the List-view
- New sorting-functionalities in the List-view.
- Add Cut/Copy/Delete functionalities in the Page-View (if introduced, then Question Groups should be available also in the Page-view!?)
- Underworlds are difficult for users to understand. Underworlds for each Question Group and individual settings for each Question-group (Playback, Modes).
- ...

Further we have some thoughts to share about the didactical purpose. In the initial problem-part of this FR, multiple didactic motivations are addressed.

a) A first didactic motivation (see part 1 of FR) says: "An often asked form use case at our University is probing the learner's knowledge concerning a certain topic with different questions that build on one another."  - AND - " Achieveing this in 5.2 without dirty patches and hacks is very painful. One way of doing this in tests is by using a fixed set of questions and a fixed order. When it comes to self-study scenarios and large amount of question groups that are supposed to be presented to the learners as self-evaluation bundles in random tests"

=> As far as we see, building different questions that build on one another AND create a  self-study scenario with a large amount of questions AND present them to learners as self-evaluation bundles in random tests is not an often used scenario. We think: It is a good way to create a fixed set of questions and order, when you want to build questions that build on one another.


b) A second didactic motivation says: " Another didactic motivation for the introduction of question groups are the so called "key feature tests" that are widely used in medical education and that allow to test procedural knowledge very well. The idea of a key feature test is basically to present a "chain" of questions to the learner in which questions build on the correct answers of previous questions. A popular example are tests in the area of medical diagnosis: the student has to decide in the first question what to do with a patient's story and how he/she interprets it. They answer the question and the question is then fixed, because the correct interpretation of the patient's story is given in the context of the next question."

=> There is a study from the university of Bern, where KF-scenarios are probed. They used the CAMPUS assessment software as an assessment tools, together with the medical faculty of Heidelberg  (http://www.medizinische-fakultaet-hd.uni-heidelberg.de/CAMPUS-Lehr-und-Pruefungssystem.110009.0.html). Two Key-Feature examples, made as a formative scenario with CAMPUS-software can be found here: http://www.medizinische-fakultaet-hd.uni-heidelberg.de/index.php?id=109993&L=de

Those two cases in CAMPUS-software show, that for KF-scenarios, it is not necessary to have a assessement software where different questions are grouped on ONE page. The CAMPUS-software shows each question on a separate page, similar to ILIAS.

But those cases show also, that in KF-scenarios often content-pages without a question are used.  So in our opinion, also an improved ILIAS Learning Module could be a possible solution for KF-scenarios. And the ILIAS Learning Module is already able to "group" several questions on one page.


c) Third the FR says, that there "are many scenarios where grouped questions offer great opportunities especially when random tests and self-study settings are concerned".

=> At our university, the test is widely used in formative settings, but random tests and self-study settings are not respectively very seldom asked. We collect every feature-request from our teachers, but Question-Groups are never asked. Of course, we could activate this feature in the administration (JF-decision July 31, 2017), and this would be maybe our solution, because we actually think that there are already too much test-settings. So such an administration settings should not be the first solution for us, we are interested to have features that are easy to use, and to have features we can offer to all of our users. So it would be good to see whether other community members would like this feature.

AT 2017-01-04: The Technical Board called for options to decouple large components in ILIAs, one of them being the test. The aim is to improve the reliability.
Could we defer the decision on this feature untill we have weighed the options to move forward with the decoupling?
Once we know about the way forward we will be much more informed about repercussions of this feature. 
Please consider this suggestion to defer the decision. 

2018-01-08, Glaubitz, Marko [mglaubitz]: 
Firstly, Hansjörg, thank you for your remarks, although I disagree with some points that you made. The perspective and experience of the Uni Bern seems to be different from ours and many other institutions. I can only mirror that our users and the users of a majority of the SIG EA representatives have been waiting for the possibility of grouping questions (and as a second step) displayig them on one page for some years now.
In addition, the SIG EA has discussed the issue of code quality / refeactoring vs. the introduction of new features at great length and we have decided that a certain list of features should be realised before the maintainer focuses on refactoring. Please have a look at the SIG EA's statement reagrding this issue
We have discussed the issue of complexity and improving the existing code at great length and have asked Björn to consider this in his offer for the feature. This means that a certain part of the costs will be availablefor the maintainer to improve code if need be.

However, I strongly reject Alexandra's proposal to defer the decision, since this would mean that the feature will not be available before 5.5 or later. This is not an option for us and many other institutions. If we can not tackle this feature for 5.4, we will have let it go for good. That would be a great pity in terms of lost funding and the work many people of were involved in developing the extensive concept for this feature.

Killing, Alexander [alex], 10 Jan 2018: I am also tempted to vote for postponing the decision, but not necessarily to the 5.5 release. In addition to Alexandras reason a main point for me is the combination of

• the past critisism of the T&A quality,
• the increased bug votings in the T&A (the number doubled in the last year from around 15 votings to more than 30, second place (per component) is currently the course management with 7 bug votings) and
• the fact that we should try to put higher quality standards on something that provides online exam functionality.

As I understand there is currently even funding for bug fixing available. I would like to see this being spent first in the next weeks and to get the number of bug votings going down significantly before making a decision on additional complexity.

JourFixe, ILIAS [jourfixe], 10 JAN 2018 : We had a long discussion about this feature request and its implecations on the test & assessment in general. We need to postpone a final decision about this feature request until we know the results and recommendations of the T&A revision (which will be available end of March at the latest).