Show Advanced KnowledgeHide Advanced Knowledge

Introducing Assessment Question Service

This requests comprises also the following sub-projects: 

• Test Player as Independent Object
• Test Question Pool as Independent Object

1 Initial Problem

The component Test & Assessment needs to be divided into a question service and (at least) two clearly separated components using the question service to split up the large component Test & Assessment:

1. Test Object
2. Test Question Pool Object
3. Assessment Question Service

First thoughts in 2018 were, that the existing T&A and its code could be split up into three components. But this task wasn’t as practicable as imagined, so the Assessment Question Service is based on a completely new code and as a new service, that can be used by components like question pool and test players.
 
For now when we talk about splitting up the T&A, we talk about the originally idea of having separate components/services for question service, question pools, test players and maybe further components and services.

2 Conceptual Summary

2.1 What is a question?

In Assessment Question Service and also in the new T&A a question is the smallest component. A question or the question service contains e.g. following:

• Defined question type
• Title
• Author
• Description
• Lifecycle
• Question Text
• Working Time
• Answers
• Points
• Feedback
• Hints
• (Versions / History)

 
The different types consider further points like shuffling answers, answer images, etc.
It is important, that the new service contains at least all existing functions of existing T&A.

An open discussion is the usage of an editor for creating questions.
 
The question service itself can be used in different contexts, it focusses on specific questions and question types. The ASQ can be seen as an ILIAS-internal interface, that needs other components like e.g. Questionpools and Testplayers. It has no noticeable influence on users without these components.

2.2 Creation and Authoring of Questions

In the Assessment Question Service it is possible to create specific questions, until now 14 different question types are implemented. 

• Multiple Choice Questin (Single Answer)
• Multiple Choice Question (Multiple Answer)
• Multiple Choice Question (Kprim Answer)
• Error Text
• Hot Spot / Image Map Question
• Cloze Question (including long menu gap)
• Numeric Question
• Formula Question
• Test Subset Question
• Ordering Question (vertical & horizontal)
• Matching Question
• Essay Question
• File Upload Question
• Text Order Question

The implementation of new types is possible.

For every question type the ASQ saves assigned answer options, points for correct/incorrect answers, uploaded files, etc.

2.3 Feedback and Hints

As a part of a question the ASQ saves common solution feedback and hints for questions.

In the register "Preview" a question with the status "Completed" can be displayed with its specific components such as title, question, answers, etc.

2.4 Deleting Questions

The Assessment Question Service will enable administrators to delete questions from the database (e.g. reasons of copyright, privacy information, etc.).

Following concept was discussed on a workshop:

• Questions, which neither is used by a test nor a question pool, can be deleted irreversible.
• Question pools only can be deleted, if they are not used by any test player.
• Deleting sequences: 
  • test (players)
  • question pools
  • automatically, if a questions isn't used by any components (synchronous or asynchronous, e.g. cronjob)

2.5 The question service is not responsible for...

The Question Service currently does not include the following features which surround the question. These are especially the following.
But: We like to work with other components and other maintainers to guarantee, that the new T&A and all its components create a sustainable, stable and maintainable environment and future for tests, exams, etc.

Link Learning Material to a Question
A specific question can be used in many different contexts. The context (test player or pool) is responsible to assign the concrete learning material for this context to the question. Possibly the Question Services needs an interface to show/edit/share the Learning Material to a question in the Question Service.

Link Competences to a Question
A specific question can be used in many different contexts. The context (test player or pool) is responsible to assign the concrete competences.

Link Taxonomies to a Question
A specific question can be used in many different contexts. The context (test player or pool) is responsible to assign the concrete taxonomies. Currently the administration of the taxonomies is located in the question pool.

Embed a question in additional content
A specific question can be used in many different contexts. The context (test player, pool or learning module) is responsible to embed the question.

Usage of the Page Editor
The editing of the pages is not included and could be a part of the test player. We see the need of the Page Editor in different use cases, so we are open to discuss this point.

Results/Manual Scoring
The results in a test will not be saved in the Question Service, rather the test player should be responsible for the results and manual scoring.


For all those features we are open to discuss the special placement in a special component (Question Service, Pool, Test Player). 

3 Replacement of T&A

ASQ and new test players do not interfere with preexisting tests using the old T&A implementation, but T&A will be phased out slowly in the future considering the need of the ILIAS users.

3.1 Migration path to ASQ

Funding needed, see: Migration path for question types to ASQ

The migration from old question pools to new question pools needs new question pools, based on asq technology. Each question type needs an own path for migration.

4 User Interface Modifications

4.1 List of Affected Views

None.

4.2 User Interface Details

None.

4.3 New User Interface Concepts

No new user interface concepts necessary for implementing the assessment question service. A possible different user navigation concept could be discussed during the development of test player(s) and question pools.

5 Technical Information

The maintainer strongly supports this request but needs to say: this will be a new implementation that will probably bring up some technical issues, that cannot be seen at the moment. Nevertheless the mission to get a decoupled assessment question service needs to be completed.

This request also requires two further requests to be implemented:

• Test Question Pool as Independent Object
• Test Player as Independent Object

6 Privacy Information

We save for each change by question authoring, the change of the question and the executing user. We need this to display a question history. 

7 Security Implications

The assessment question service offers service endpoints. These will now have to be addressed by the test and question pool. Authorisation is in the responsibility of test pool(s), test player(s), learning modul(s).... The consuments of the question service.

8 Contact

• Author of the Request: & Studer, Martin [mstuder]Heyser, Björn [bheyser], Contents as of December 2019
• Maintainer: Studer, Martin [mstuder]
• Implementation of the feature is done by: & Studer, Martin [mstuder]

9 Funding

If you are interest in funding this feature, please add your name and institution to this list.
Crowdfunding: https://docu.ilias.de/goto_docu_dcl_3700_17_3226.html

• ILIAS-Verein
• Universität Bern
• DHBW
• FH Aachen
• Universität Freiburg
• PH Freiburg
• Universität zu Köln
• Universität Marburg
• Universität Hohenheim
• sr.solutions

10 Discussion

JourFixe, ILIAS [jourfixe], 12 NOV 2018 : We accept Björn's suggestion to split up the existing test and assessment component into three new components, incl. a new assessment question service for ILIAS 6.0. All questions about implementation and interfaces are tackled in pull requests as agreed with the Technical Board. The PM will change the related pages and suites in the feature wiki, Mantis and Test Rail accordingly. Please note that all existing test question plugins won't work any longer with this new service without according changes. New plugins made for this plugin slot won't work with versions before 6.0.

Neumann, Fred [fneumann], 31 SEP 2019
The proposed changes in the authoring interfaces look nearly all convinging and straight to me. I just have doubts with 7.6  - getting the ordering questions into one type. These subtypes are used differently. The horizontal ordering is mailny used for ordering words in a sentence which is at best entered in a text area. The vertical ordering may use images and has the option to nest terms (currently in an own authoring view.) It would make the authoring interface of common question type rather complex to support both equally.

Kunkel, Matthias [mkunkel] @ SIG EA, 13 NOV 2019 : The separation of the Test&Assessment has been postponed to ILIAS 7 because the project cannot be finished satisfactorily until 'Coding Completed' at NOV 29. Therefore, this feature won't make it into ILIAS 6 but is now suggested for ILIAS 7. Please add it to the JF agenda to schedule it again.

JourFixe, ILIAS [jourfixe], 16 DEC 2019: We re-schedule the current suggestion for ILIAS 7. In addition, we would like to get rid of TinyMCE in the question service and use only the new page editor. But before making a final decision we have to discuss the options with the maintainer of the IPE, Alexander Killing (who is not joining the JF today). Migration of questions created with the Tiny needs to be supported as well. Martin will invite to a 2h VC at beginning of January.

Studer, Martin [mstuder], 20.03.2020: We published a first version of the Assessment Question Service: https://github.com/studer-raimann/AssessmentQuestion

Kergomard, Stephan [skergomard], 07 DEC 2020: Sadly, I won't be able to join the JF today, sorry. Here a few thoughts:
First: There where a lot of changes made in the FR last Friday and I have a hard time really going through the whole text. I think this is really not ideal to discuss the topic today.
And then: As a representative of the HSLU, I see a few conditions, before I believe the ASQ can be merged:

• The points from the TB review should be implemented fully.
• As I don't see a good way to have the current and the upcoming Test & Assessment visible in the interface in parallel (Would we have two Questionpools? Could you mix questions from the current and the new Test & Assessments in Learning Modules and other potential "integrated players" or would you need two versions of each question to use them in parallel? If we need to have two versions: How do we make sure they don't drift apart? ...):
  • There needs to be a plan with a timetable for the process until the complete removal of the current Test & Assessment and the full activation of the new system.
  • There needs to be a deadline for the removal of the current Test & Assessment. If this deadline can not be met, the ASQ needs to be removed again.
  • If the new service is implemented in a core component, there needs to be a way back, until we are absolutely sure we can remove the current Test & Assessment. ILIAS needs to remain fully functional without activating the ASQ until the current Test & Assessment is completely removed.
  • We don't need to implement all functionality in the new Test & Assessment. But we need to be sure that we are fine with loosing the corresponding functionality, the moment we get rid of the current Test & Assessment. Not having all functionality implemented shouldn't be a condition for not removing the current Test & Assessment. Implemented Plugins shouldn't be a condition for not removing the ASQ if the deadline is not met.

Klees, Richard [rklees], 07 DEC 2020

Dear Martin,

this topic has been debated for a long time in various circles and I'm glad that this comes up on the JF today. I already said a lot about the question service in my capacity as a TB member, most of which is documented in our common feedback in the various PRs linked here. Still, I'm also a community member, and I also want to document my opinion regarding this request from that point of view.

First and foremost, I'm very thankful that you took over the responsibility for a question service when it became appearant that the T&A is too big for just one maintainer in 2018. The plan that the community, the maintainer and the TB have laid out for the T&A was to factor out a question service from the existing code to make the maintainance of the T&A easier by distributing the responsibility for the various features over more people. I truly understand that this was and is hard to accomplish, but a lot of problems I see with your request today originate in the deviation from that original plan.

I see that you and your company already invested a lot in the implementation of the new question service and that for various reasons it most probably will not be possible to stop or even delay the integration of your new question service further without causing substantial damage to our community and you. So I won't be in your way to integrate the new question service (as long as the technical change requests from the TB are implemented).

Still I want to point out my doubts and concerns regarding your current approach and hope that the JF, the PM and most notably you will find a way to address these concerns in a comprehensible and reliable manner. My most notable concerns are:

• I can hardly see how the reimplementation of the question service caters to our initial goal to make the T&A more maintainable. When we started, we had one huge T&A code base with one maintainer and some other people more or less familiar with the code base. It was complicated, but battle proven. When the new question service is merged, that code base still exists and will still be used for an indefinite time. On top, we get a second code base for the T&A which is currently not known in the community, is not battle proven and uses various new techniques currently not known in the ILIAS community. So currently from my POV the maintenance situation has stayed the same as best. It more likely will get worse, as we would have new code without a clear idea how to get rid of the old one.
• The T&A is not only complicated from the perspective of the code, but also from a feature perspective. I find it hard to see how we could make sure that use cases or expected behaviour is transfered to the new service appropriately. It might be ok to drop some of the behaviour, but how would we identify the behaviour we in fact have dropped? How would we decide which behaviour can be dropped? The situation is very similar to the code: The current T&A is complicated, but there are people in the community who deeply know how the current T&A works. We will get a second implementation that most possibly requires new knowledge. Thus, there will be more to know and more to test as well.
• Even if the question service is included in into the core of ILIAS, there is a tremendous amount of work to be done to replace the current T&A with new implementations based on the new question service. Even if we aim to replace a part of the current T&A only, e.g. removing the "question"-parts and replacing it with the new question service, there is a lot way to go. In this time we will have to maintain the old and the new T&A-stuff. How long will this process take? How will we handle new feature requests in the meantime? Bug reports and testing? Documentation? We will in fact have two competing systems from various perspectives and I fear that the community will show rips at that borderline or at least requires substantial effort to prevent these rips.
• The ressources to finish that effort someday are not currently allocated. I also cannot see a clear plan how we could get to the initial goal to improve the maintainability of the T&A, as initially planned. I totally understand that it is not possible to guarantee ressources or make a detailed plan for the complete undertaking. But, as outlined in the previous items, I see that we will move to an overall worse state when including the new question service. This might be ok temporarily, but the current T&A and the new question service should not exist side by side indefinitely. How can we make sure that this won't happen?

The T&A is a component of ILIAS that is very important to many. We probably will need to take risks here to move it forward, but we should at least understand and acknowledge these risks to have a chance to mitigate them. Please understand that I come from a place where I deeply care about the future of ILIAS. My doubts currently prevent that I can support your approach with a clean conscience, even if I disregard any technical dimensions. I hope that you will find a way to convince me or prove me wrong in the future.

Best regards!

Becker, Maximilian [mbecker] 07 DEC 2020
I see a few key challenges here and wonder if this is truly intended and would consider discussing thoroughly if this is acceptable:

• The solution as laid out does not lead to a separation of Test and QuestionPool. It does separate the questions from the QuestionPool. This does in fact lead to a situation where some work could be shared among separate maintainership-fiefdoms, but 
• the definition what belongs to a question and what doesn't, which happened a bit behind closed doors, leads to a requirement that the one question service with generic minimal question implementations demands for a number of new QuestionPool & Test couples, each addressing special needs in testing, which are then - again - most intimately tied together over considerations what the questions need to be, that the QuestionService defined off of their plate.
  From my observation, the feature set of the current questions is relatively stable, most of the variable are in test presentation, scoring and result presentation.
• Many of the service advantages, phrased as "Git for Questions", are defused over the questions being conceptually disconnected from the features in 2.3.
• Most of the migration path is left to yet unknown creators and maintainers of new Pool-Test-Combos. Future People.

JourFixe, ILIAS [jourfixe], 07 DEC 2020: We had a long discussion today about the suggested question service but still have some conceptual questions before making a final decision. It is not really clear what is the relation between question and question service and of what a question consists of. There are also objections in the community that taxonomies, page content and links to learning material should not be part of a question - or at least under the control of the ASQ (see suggestion in chap. 2.3). In addition, it would be very helpful if all information provided by Stephi, Martin and Adrian today in the JF would be found in the wiki page as well. This concerns also the questions posted to this request in the last days (see chap. 10).

There is a need to specify this relation with the community and stakeholders before making a final decision about the request as this determines the future steps to realise the planned separation of the existing T&A.

We kindly ask all stakeholders and interested developers to participate at the upcoming workshop(s) to help us pushing this large development activity.

Studer, Martin [mstuder]

Dear Kergomard, Stephan [skergomard]


First: There where a lot of changes made in the FR last Friday and I have a hard time really going through the whole text. I think this is really not ideal to discuss the topic today.

Correct, but I would like to note that the whole topic has already been discussed in detail - sind 2018-11-20 (https://github.com/ILIAS-eLearning/AQS/tree/review/docs/0_History) and is also well documented (https://github.com/ILIAS-eLearning/AQS/tree/review/docs). You have been involved in these discussions for the last year. So I hope that at least for yourself - regardless of this feature wiki page - it would have been possible to participate.


The points from the TB review should be implemented fully.

I completely agree with you! I hope you also noticed how much extra work we did because of the feedback.


As I don't see a good way to have the current and the upcoming Test & Assessment visible in the interface in parallel (Would we have two Questionpools? Could you mix questions from the current and the new Test & Assessments in Learning Modules and other potential "integrated players" or would you need two versions of each question to use them in parallel? If we need to have two versions: How do we make sure they don't drift apart? ...):

This is an important question! I think we should and have(!) to clarify this question in the context of the projects Test Question Pool as Independent Object and Test Player as Independent Object which have unfortunately come to a standstill. Here it will be necessary to evaluate which use cases exist and the amount of work one wants to and can put into them.


There needs to be a plan with a timetable for the process until the complete removal of the current Test & Assessment and the full activation of the new system.

I completely agree with you!


There needs to be a deadline for the removal of the current Test & Assessment. If this deadline can not be met, the ASQ needs to be removed again.
Which advantage do you see in this action. If the schedule cannot be kept. the current test and question pool is no longer maintainable. Are you not destroying the whole test & assessment with this action? i don't think i understand your statement. can you explain it in more detail?


If the new service is implemented in a core component, there needs to be a way back, until we are absolutely sure we can remove the current Test & Assessment. ILIAS needs to remain fully functional without activating the ASQ until the current Test & Assessment is completely removed.
Back to where? To the soon no longer maintainable Test & Assessment?


We don't need to implement all functionality in the new Test & Assessment. But we need to be sure that we are fine with loosing the corresponding functionality, the moment we get rid of the current Test & Assessment. Not having all functionality implemented shouldn't be a condition for not removing the current Test & Assessment. Implemented Plugins shouldn't be a condition for not removing the ASQ if the deadline is not met.

I fully agree with you that we will require a future Test & Assessment that meets the high standards that the current Test & Assessment meets. But this will not be possible in the future with the current Test & Assessment. I think we need a maintainable and expandable system. With the Assessment Question Service we have made big step in this direction. Please help us to look into the future.

,  & Studer, Martin [mstuder]


Dear Richard
Stephi, Adi and myself have discussed your feedback together. It was important to us that all three of us work on your feedback, as your statement is already the second statement by a technical board member and you have a very high standing as a person in the community. That's why we felt that a common answer is very important.


Thank you for your work as a member of the TB. We understand that the new Assessment Question Service was an important and time consuming part of your work in the last months as well. We thank you and the other members again for the useful discussion and the reviews of the code. Now we are glad to explain some of your remarks as a part of the ILIAS community. The ideas and comments from the community are very important for us.


I see that you and your company already invested a lot in the implementation of the new question service and that for various reasons it most probably will not be possible to stop or even delay the integration of your new question service further without causing substantial damage to our community and you. So I won’t be in your way to integrate the new question service (as long as the technical change requests from the TB are implemented).

The last weeks we are working on the implementation, that are requested from the TB. It took a while, but Adi and Martin are confident to finish this work in 2020. Thanks for preparing the way to the ILIAS trunk.


Still I want to point out my doubts and concerns regarding your current approach and hope that the JF, the PM and most notably you will find a way to address these concerns in a comprehensible and reliable manner. My most notable concerns are:

• I can hardly see how the reimplementation of the question service caters to our initial goal to make the T&A more maintainable. When we started, we had one huge T&A code base with one maintainer and some other people more or less familiar with the code base. It was complicated, but battle proven. When the new question service is merged, that code base still exists and will still be used for an indefinite time. On top, we get a second code base for the T&A which is currently not known in the community, is not battle proven and uses various new techniques currently not known in the ILIAS community. So currently from my POV the maintenance situation has stayed the same as best. It more likely will get worse, as we would have new code without a clear idea how to get rid of the old one.

We understand your doubts as to the new architecture of the question service and its code, but we are interested in working closely with the future maintainer and stakeholders of the new components. Our knowledge has to be shared, because that is the key point for the future. For a sustainable, maintainable T&A. Without support and maintainance from the community and future maintainer the splitting-up of T&A could not be reached.

• The T&A is not only complicated from the perspective of the code, but also from a feature perspective. I find it hard to see how we could make sure that use cases or expected behaviour is transfered to the new service appropriately. It might be ok to drop some of the behaviour, but how would we identify the behaviour we in fact have dropped? How would we decide which behaviour can be dropped? The situation is very similar to the code: The current T&A is complicated, but there are people in the community who deeply know how the current T&A works. We will get a second implementation that most possibly requires new knowledge. Thus, there will be more to know and more to test as well.

A good point. We still have a long way to go – and to test the new question service as well as the other future components. But: We should not shy away from this huge task. For new implementations (and features) we need new ways, if we could see, that old components of ILIAS aren’t sustainable.

• Even if the question service is included in into the core of ILIAS, there is a tremendous amount of work to be done to replace the current T&A with new implementations based on the new question service. Even if we aim to replace a part of the current T&A only, e.g. removing the “question”-parts and replacing it with the new question service, there is a lot way to go. In this time we will have to maintain the old and the new T&A-stuff. How long will this process take? How will we handle new feature requests in the meantime? Bug reports and testing? Documentation? We will in fact have two competing systems from various perspectives and I fear that the community will show rips at that borderline or at least requires substantial effort to prevent these rips.

As a maintainer of a single T&A component we could not give a sample solution and timetable. We need other maintainers, ideas and discussions.

• The ressources to finish that effort someday are not currently allocated. I also cannot see a clear plan how we could get to the initial goal to improve the maintainability of the T&A, as initially planned. I totally understand that it is not possible to guarantee ressources or make a detailed plan for the complete undertaking. But, as outlined in the previous items, I see that we will move to an overall worse state when including the new question service. This might be ok temporarily, but the current T&A and the new question service should not exist side by side indefinitely. How can we make sure that this won’t happen?

We don’t have the competence to answer this alone…

• The T&A is a component of ILIAS that is very important to many. We probably will need to take risks here to move it forward, but we should at least understand and acknowledge these risks to have a chance to mitigate them. Please understand that I come from a place where I deeply care about the future of ILIAS. My doubts currently prevent that I can support your approach with a clean conscience, even if I disregard any technical dimensions. I hope that you will find a way to convince me or prove me wrong in the future.

We try to support our LMS and its sustainability. The current T&A has reached its limits and we try to find a solution – together with you and the community.

Best regards!
Stephi, Adi & Martin

The project was successfully reviewed by the technical board.
After discussions with the TB, project Manager and SIG EA leader become clear, that the ASQ is not requested for ILIAS 8.
Further clarification would like to be done, before a clear decision could be made.
The ongoing discussions about the future of Test & Assessment, risks and profits for ILIAS has economic influence on our company - we already invested 1.5 man years in this project.
Therefore we will publish a question pool, test player with ASQ as a plugin. We are open for discussions. Those projects we will made with paying clients. And we will focus on theirs needs.

Glaubitz, Marko [mglaubitz] and Falkenstein, Rob [rob], 18 Aug 2021:

• the ASQ should be extended in order to incorporate external services that the current state of implementation of an "assessment question" in ILIAS offers
• Extending ASQ: Competence Assignment for Questions
• Extend ASQ: Page Content for Questions
• Extending ASQ: Taxonomy Assignment for Questions

11 Implementation

{The maintainer has to give a description of the final implementation and add screenshots if possible.}

Test Cases

Test cases completed at {date} by {user}

• {Test case number linked to Testrail} : {test case title}

Approval

Approved at {date} by {user}.