Feature Wiki
Tabs
Item Statistic in Pool
Page Overview
[Hide]1 Initial Problem
The item pool functions as container and is the primary location for activities regarding creating and editing items. To realize psychometric quality assurance, it is also the best place to aggregate statistical values originating from participant data in various tests.
Currently ILIAS lacks the functionality to provide any of those values in a convenient way. Only the percentage of correct answers is displayed on a separate page, accessible via an icon/button per row in the item table. Even this single value is neither sortable nor bound to specific versions of a question.
This request wants to add item statistics on a more visible scale to the item pool and to allow for their usage during test creation and item editing.
2 Conceptual Summary
Tab Questions in Pool
The tab in the Pool gets optional additional, optional, sortable columns:
- Discriminatory Index (ability of the question to distinct between high and low achieving participants, e.g. via point-biserial correlation between the score on the item and the test)
- Facility Index (average of reached points, displayed as percentage, interpretable as difficulty)
- Standard Deviation (of points)
There will be a range filter for each column on top of the table.
Questions will get an extra underworld for the distractor / atractor figures and charts (attractors = correct answers, distractors = wrong answers in choice-format items). Seeing e.g. a specific distractor chosen by the majority or even by mostly overall high achieving participants indicates potential for a missinterpretable answeroption.
Data sets for calculation
Data from consumers is NOT dynamically calculated: The data is either
- manually drawn by click on a dedicated "Push Item Statistics" or
- drawn by a cron-job from the consumer and pushed to the pool or
- received via the SOAP interface from another plattform (e.g. when using a setup with separate plattforms for item-keeping and test-execution)
Ex- and Import
If Question Pools are imported or exported, users can choose: ex / import with or without Items Statistics. Only weighted avarages will be taken along.
tab Quality of Question of a Question
Every Question gets a new tab Quality of Question it bears two sub-tabs:
- Item Statistik
- Versions
This sub-tab will feature the folloing information:
Version | Datum | Eingriff | Login | Lifecycle Status | Anzahl nutzender Objekte | Aktionen |
1 | 31.12.2016, 09:25 | Erstversion | fred.neumann | Draft | 0 | Preview |
2 | 01.01.2017 13:30 | Auswahl erweitert | fred.neumann | Draft | 3 | Preview |
There are Actions :
- Preview
- Link to Item Statistic of this Version (could be automatic filtered)
Sub-tab Item Statistic
There is a an Overview of all uses of the question and the datasets of the results. Dataset might come from a different plattform.
Data sets can be selected and there can be alternatively calculated weighted means.
Above the table Versions there is a filter.
Verwendet in Test | NIC | Datum von | Datum bis | # Antworten | Schwierigkeit |
| |
[ ] | Erstklausur | 01.01.2017 | 02.02.2017 | 50 | 0,5 | [Aktionen Þ] | |
[ ] | Andere Klausur | 01.01.2017 | 04.02.2017 | 40 | 0,5 | [Aktionen Þ] | |
[ ] | Nachholklausur | 01.02.2017 | 03.02.2017 | 50 | 0,6 | [Aktionen Þ] |
There are the following Actions:
- Details (all key figures for test and questions)
- delete data set
- Multi-Actions
3 User Interface Modifications
3.1 List of Affected Views
- Tab Questions in Pool
- New Quality of Question in Question
- sub-tab Item Statistic
- Versions
3.2 User Interface Details
Within the question pool the aggregated statistical figures will be shown in the questions table.
The statistics tab in tests offers a toolbar button that allows to push the item statistics to the corresponding pools.
![](./data/docu/mobs/mm_65261/push-btn.png?il_wac_token=1cad9a256e2895ebfb0f53ad28369196d7db8014&il_wac_ttl=5&il_wac_ts=1725756016)
Automatically pushing the item statistic is possible with an additional test setting that enables a new cronjob task.
![](./data/docu/mobs/mm_65262/push-set.png?il_wac_token=1cad9a256e2895ebfb0f53ad28369196d7db8014&il_wac_ttl=5&il_wac_ts=1725756016)
3.3 New User Interface Concepts
None.
4 Technical Information
none
5 Contact
- Author of the Request: Tödt, Alexandra [atoedt]
- Maintainer: Heyser, Björn [bheyser]
- Implementation of the feature is done by: {The maintainer must add the name of the implementing developer.}
6 Funding
If you are interest in funding this feature, please add your name and institution to this list.
- …
7 Discussion
Klees, Richard [rklees], 2018-12-17: It is very hard to understand what this is supposed to do and how the concept relates to that goal, given the very sparse "initial problem"-description here. As far as I understand, this feature wiki pages are a documentation as well as a ground for JF discussion and it will be well-nigh impossible to understand any of this feature based on this page. This request leaves a lot of questions open:
- What is the Discriminatory Index, why would I want to display it in a statistic about a test question. Even if I know what the Discriminatory Index is: How is it computed here?
- What is the Facility Index, why would I want to display it in a statistic about a test question. Google helps even less here than for the Discriminatory Index. How is it computed here?
- Standard Deviation of what?
- Why should we use 2*figure*participants/(sum of participants) as weighted mean?
- What are "the distractor attractor figures and charts"?
- How is "Import via SOAP" connected to the initial problem?
I'm sure this is all grounded on a well-devised concept with solid foundations on statistics. Please present this in a more comprehensible form before taking this to the JF.
JourFixe, ILIAS [jourfixe], 17 DEC 2018: We highly appreciate this suggestion and schedule the feature for 6.0 with the following changes:
- We would like to have columns for all indices in sub-tab Item Statistics (and not only Difficulty).
- All columns should be sortable as well.
- The used formulas to calculate the three mentioned indices need to be documented.
- Please implement an interface that allows also to connect an external and alternative service to calculate the indices, e.g. https://github.com/kyro46/ilIRTEvaluations.
Kunkel, Matthias [mkunkel] @ SIG EA, 13 NOV 2019 : The separation of the Test&Assessment has been postponed to ILIAS 7 because the project cannot be finished satisfactorily until 'Coding Completed' at NOV 29. Therefore, this feature won't make it into ILIAS 6 but is now suggested for ILIAS 7. Please add it to the JF agenda to schedule it again.
2020-07-29: It is planned to create the item statistics as a new register in the details of a question in the ASQ. In this tab there is a table with an overview in which tests and in which versions a question is used. Every column is sortable. The table has various filter-options.
There are action fields for the whole table and every individual row.
![](./data/docu/mobs/mm_78257/itemstatistics.png?il_wac_token=1cad9a256e2895ebfb0f53ad28369196d7db8014&il_wac_ttl=5&il_wac_ts=1725756016)
Strassner, Denis [dstrassner] 30 JUL 2020: This has not been agreed with the SIG EA, so I refuse to discuss this FR at the JF until this has been done and my other objections are fully resolved.
In detail:
- So far no approval from SIG EA
- You propose to change the localisation of the "item statistic" from pool to the AQS (assessment question service). But there is no AQS for ILIAS 7 at the moment and right now, i don't believe tha AQS ships with ILIAS 7.
- as long as the AQS is not in the core, the T&A should remain independently functional
- since the technical aspects of AQS are still subject to discussion with the TB, feature requests with AQS as a basis are rather supoptimal in my view.
Strassner, Denis [dstrassner] 03 SEP 2020:
- We (srag, Timon Amstutz, Fred Neumann, Denis Strassner, Stephan Winiker) discussed the approach to implement this feature today and agreed to postpone this feature request until it is clear if the Assessment Question service is implemented or not and how.
- We also noticed that the communication in this case was lacking and that future changes to already discussed features should be discussed before introducing them into the JF (e.g. in a Feature Workshop, with the SIG EA, lead SIG EA).
8 Implementation
{The maintainer has to give a description of the final implementation and add screenshots if possible.}
Test Cases
Test cases completed at {date} by {user}
- {Test case number linked to Testrail} : {test case title}
Approval
Approved at {date} by {user}.
Last edited: 3. Sep 2020, 09:57, Strassner, Denis [dstrassner]