Show Advanced KnowledgeHide Advanced Knowledge

Item Statistic in Pool

1 Initial Problem

The item pool functions as container and is the primary location for activities regarding creating and editing items. To realize psychometric quality assurance, it is also the best place to aggregate statistical values originating from participant data in various tests.
 
Currently ILIAS lacks the functionality to provide any of those values in a convenient way. Only the percentage of correct answers is displayed on a separate page, accessible via an icon/button per row in the item table. Even this single value is neither sortable nor bound to specific versions of a question.
 
This request wants to add item statistics on a more visible scale to the item pool and to allow for their usage during test creation and item editing.

2 Conceptual Summary

Tab Questions in Pool
The tab in the Pool gets optional additional, optional, sortable columns:

• Discriminatory Index (ability of the question to distinct between high and low achieving participants, e.g. via point-biserial correlation between the score on the item and the test)
• Facility Index (average of reached points, displayed as percentage, interpretable as difficulty)
• Standard Deviation (of points)

The figures will be in a weighted average. [(figure x participants) + (figure x participants)] / sum of participants. The weighting is necessary, because it shall be able to aggregate multiple tests with a differing number of participants. E.g. we can't just create the shallow average of the difficulty of an item for a 500 and a 10 participant test.

There will be a range filter for each column on top of the table. 
Questions will get an extra underworld for the distractor / atractor figures and charts  (attractors = correct answers, distractors = wrong answers in choice-format items). Seeing e.g. a specific distractor chosen by the majority or even by mostly overall high achieving participants indicates potential for a missinterpretable answeroption.

Data sets for calculation 
Data from consumers is NOT dynamically calculated: The data is either

• manually drawn by click on a dedicated "Push Item Statistics" or 
• drawn by a cron-job from the consumer and pushed to the pool or 
• received via the SOAP interface from another plattform (e.g. when using a setup with separate plattforms for item-keeping and test-execution)

The data set comprises frequency distributions, Discriminatory Index, Facility Index, Standard Deviation from the consuming tests. The cosuming test may repeatedly push data: The new data set will overwrite the existing one. Per test there can only be one data set. 

Ex- and Import

If Question Pools are imported or exported, users can choose: ex / import with or without Items Statistics. Only weighted avarages will be taken along. 

tab Quality of Question of a Question 
Every Question gets a new tab Quality of Question it bears two sub-tabs:

1. Item Statistik
2. Versions

sub-tab Versions
This sub-tab will feature the folloing information: 

Version	Datum	Eingriff	Login	Lifecycle Status	Anzahl nutzender Objekte	Aktionen
1	31.12.2016, 09:25	Erstversion	fred.neumann	Draft	0	Preview
2	01.01.2017 13:30	Auswahl erweitert	fred.neumann	Draft	3	Preview

There are Actions :

• Preview
• Link to Item Statistic of this Version (could be automatic filtered)

The table can be exported as CSV or Excel.

Sub-tab Item Statistic
There is a an Overview of all uses of the question and the datasets of the results. Dataset might come from a different plattform. 
Data sets can be selected and there can be alternatively calculated weighted means. 
Above the table Versions there is a filter. 

	Verwendet in Test	NIC	Datum von	Datum bis	# Antworten	Schwierigkeit
[ ]	Erstklausur		01.01.2017	02.02.2017	50	0,5	[Aktionen Þ]
[ ]	Andere Klausur		01.01.2017	04.02.2017	40	0,5	[Aktionen Þ]
[ ]	Nachholklausur		01.02.2017	03.02.2017	50	0,6	[Aktionen Þ]

There are the following Actions:

• Details (all key figures for test and questions)
• delete data set
• Multi-Actions

Export as CSV or Excel .

3 User Interface Modifications

3.1 List of Affected Views

• Tab Questions in Pool
• New Quality of Question in Question 
  • sub-tab Item Statistic 
  • Versions

3.2 User Interface Details

Within the question pool the aggregated statistical figures will be shown in the questions table.

The statistics tab in tests offers a toolbar button that allows to push the item statistics to the corresponding pools.

Automatically pushing the item statistic is possible with an additional test setting that enables a new cronjob task.

3.3 New User Interface Concepts

None.

4 Technical Information

none

5 Contact

• Author of the Request: Tödt, Alexandra [atoedt]
• Maintainer: Heyser, Björn [bheyser]
• Implementation of the feature is done by: {The maintainer must add the name of the implementing developer.}

6 Funding

If you are interest in funding this feature, please add your name and institution to this list.

• …

7 Discussion

Klees, Richard [rklees], 2018-12-17: It is very hard to understand what this is supposed to do and how the concept relates to that goal, given the very sparse "initial problem"-description here. As far as I understand, this feature wiki pages are a documentation as well as a ground for JF discussion and it will be well-nigh impossible to understand any of this feature based on this page. This request leaves a lot of questions open:

• What is the Discriminatory Index, why would I want to display it in a statistic about a test question. Even if I know what the Discriminatory Index is: How is it computed here?
• What is the Facility Index, why would I want to display it in a statistic about a test question. Google helps even less here than for the Discriminatory Index. How is it computed here?
• Standard Deviation of what?
• Why should we use 2*figure*participants/(sum of participants) as weighted mean?
• What are "the distractor attractor figures and charts"?
• How is "Import via SOAP" connected to the initial problem?

These questions only cover the first third of the conceptual summary, I'm sure there will be more regarding the following parts.

 I'm sure this is all grounded on a well-devised concept with solid foundations on statistics. Please present this in a more comprehensible form before taking this to the JF.

JourFixe, ILIAS [jourfixe], 17 DEC 2018: We highly appreciate this suggestion and schedule the feature for 6.0 with the following changes:

• We would like to have columns for all indices in sub-tab Item Statistics (and not only Difficulty).
• All columns should be sortable as well.
• The used formulas to calculate the three mentioned indices need to be documented.
• Please implement an interface that allows also to connect an external and alternative service to calculate the indices, e.g. https://github.com/kyro46/ilIRTEvaluations.

Kunkel, Matthias [mkunkel] @ SIG EA, 13 NOV 2019 : The separation of the Test&Assessment has been postponed to ILIAS 7 because the project cannot be finished satisfactorily until 'Coding Completed' at NOV 29. Therefore, this feature won't make it into ILIAS 6 but is now suggested for ILIAS 7. Please add it to the JF agenda to schedule it again.

2020-07-29: It is planned to create the item statistics as a new register in the details of a question in the ASQ. In this tab there is a table with an overview in which tests and in which versions a question is used. Every column is sortable. The table has various filter-options.
There are action fields for the whole table and every individual row.

8 Implementation

{The maintainer has to give a description of the final implementation and add screenshots if possible.}

Test Cases

Test cases completed at {date} by {user}

• {Test case number linked to Testrail} : {test case title}

Approval

Approved at {date} by {user}.