Internal consistency (Cronbachs alpha)
Indicates how well items in the test measure the same concept. Values range from 0 to 1.
If your test assesses highly related topics,  than Cronbach's alpha is an important key figure and higher values are desirable. If your tests assesses a broad range of different topics, then you must expect low and meaningless figures.

• 0 to 0,7 Arbitrary This test should not be the only source of input to assign a fair grade to a student. The quality of the test is too low, the grade assigned to a student will be kind of arbitrary. This arbitrariness will be balanced out if the result of this test contributes to the overall grade along with other assessments i.e. this test + presentation + oral exam combined would more be more likely to produce a fair grade. To improve this test compute the correlation of each test item with the total score test; delete items with low correlations add questions.
• 0,7 to 0,8 Relevant Feedback This test should not be used as the single source of input to assign a fair grade to a student. The quality of the test is good enough to inform students within a semester about their progress. It cannot be used for a high-stakes or low-stakes exams. To improve this test compute the correlation of each test item with the total score test; delete items with low correlations add questions
• 0,8 to 0,85 Good This test is well developed and can be used for low stakes exams that will assign fair grades to students even if the test is the only source of grading. It is not fit to be used in a high-stakes scenario.
• 0,85 to 0,9 Excellent This is a professionally developed standardized test that can be deployed for high - stakes exams: It is fit to be administered only once to a student and will produce a grade so sound that decisions about an entire career can be taken on the basis of this grade.
• 0,9 to 1,0 Redundant test questions Shorten you test, it contains questions that are mearuring identical things. These 'doubles' should be removed from the test.

Item Difficulty / Facility Index
The item difficulty is the proportion of persons who answer a particular item correctly to the total attempting the item.  The item difficulty index ranges from 0 to 100. The higher the value, the easier the question.
The value of desirable difficulty changes with the number of answer options a question has. This is not taken into account by the following classification.

• 0 to 20 Hard question, few students could answer that question correctly
• 20 to 80 Medium difficulty, many students could answer that question correctly
• 80 to 100 Easy question, almost all could answer that question correctly

• Below 0,1 Poor Delete questions from test do not bother to improve it it is beyond repair and not worth the effort
• 0,1 to 0,3 In need of revision Revise question for ambiguous wording
• Above 0,3 Good Keep question
• Not computable

Fred Neumann, March 30, 2015: This feature was discussed and appreciated at the SIG E-Assessment Metting on March 25.

BH 02 June 2015: The maintainer supports this request. Since we decided to rename the statistics tab to "Quality of Test", this feature request is a good step to supplement the existing statistical calculations made within this tab, so we get a more exact statement about the test's quality.

Neumann, Fred [fneumann], 24 June 2015: This reature request has a "sister" Plugin Slot for Test and Item Statistics. I propose to implement both together. The core statistics provide only the basic figures which we quickly could agree on at the SIG EA meeting. But one or more plugins are able to do advanced and experimental analyses. Those can be tested in pilots from 5.1 on and the proven ones are candidates for additional statistics in the core. I think it is important to provide a "playground" for gaining practical experience before further extensions of core statistics are discussed.

JourFixe, ILIAS [jourfixe], July 13, 2015: We generally appreciate the introduction of item statistics for test questions but we still see several open conceptual questions before being able to schedule the feature.

• We need a clear distinction between test statistic and test item statistic. Test item statistics focus on the use and the results of one question in several tests. But this requires a feasible solution for the lifecycle of a question (at the time being every question in a test is a destinct copy and could be changed completely which makes every statistic useless).
• Only a few number of question types like single choice, KPrim or multiple choice (restricted) can be supported.
• Clear use cases are needed to show which statistics can help to improve the quality of a question, e.g. distribution of distractors (and which statistics have no impact).
• We see the need of a workshop to create a concept and plan a possible implementation for version 5.2.

JourFixe, ILIAS [jourfixe], May 09, 2016: Postponed to next JF when Fred Neumann and Björn Heyser both can attempt the meeting. We ask Fred to contact Björn before this meeting and also to give us a report about first experiences with the existing plugin slot.

Neumann, Fred [fneumann], June 1, 2016:
We currently develop a plugin for ILIAS 5.1 and higher that realizes the functionality of the related feature request Plugin Slot for Test and Item Statistics. Please look at this feature request for details. It addresses most of the issues mentioned here by separating the presentation of test statistics from the actual calculation of statistical values (called “evaluations”):

• Evaluations are small classes that get filtered test data as input and produce statistical data as output. The produced data format allows different kind s of use: display on screen, in excel tables, diagrams etc.
• Our solution already shows test and question evaluations on different screens. Evaluations have no own GUI and can easily be re-used in future functionality, e.g. in the life-cycle of questions.
• Evaluations can have a description. Calculated values can be assigned with a comment and an alert status. This allows showing a detailed advice like Alexandra proposed.
• Core properties of an evaluation specify for which test type it can be applied, e.g. fixed question set, random question set, CTM. It can also be specified for which question types an evaluation is available (question evaluations that only analyze the points reached in a question can be calculated for every question type).
• Furthermore we can define in the ILIAS administration which evaluations are available to all users and which to admins only.
• It is already foreseen to filter the source data by test passes, e.g. first pass, last pass, best pass. The initial implementation will have a fixed filter to the scored pass.

Heyser, Björn [bheyser], 04. January 2019:

The Jour Fixe mentioned on 03. July 2015 the following issues that I want to address here.

• We need a clear distinction between test statistic and test item statistic. Test item statistics focus on the use and the results of one question in several tests. But this requires a feasible solution for the lifecycle of a question (at the time being every question in a test is a destinct copy and could be changed completely which makes every statistic useless).

• Only a few number of question types like single choice, KPrim or multiple choice (restricted) can be supported.

• Clear use cases are needed to show which statistics can help to improve the quality of a question, e.g. distribution of distractors (and which statistics have no impact).

• We see the need of a workshop to create a concept and plan a possible implementation for version 5.2.

• Will allow the statistics to be computed for more than one test pass?

• How is the statistics computed for the CTM-Mode that by definition never finishes? At what point in time will the statistics be presented if we never arrive at a finished test pass?

• How will missing values be handled?
  • Good practice would be to take out the data set with at least one missing value from the comutation altogether. This will spell trouble for the never ending CTM and can thus be ruled out.
  • Only to question-related statistics and only use the ansewers actually provided. 
  • Refuse the concept of missing values altogether: Set all not-answered questions (never shown and flipped over) to 0 and compute anyway (moodle solution) 
  • Alternative suggestion: never shown questions are set to 0 and flipped over questions are categorized s missing values.

Jobst, Christoph [cjobst], 07. January 2019:

"Will allow the statistics to be computed for more than one test pass?"

I'd vote for a selection of the pass to be analyzed/pushed, as is present in the current implementation of ExtendedTestStatistics (etstat).

Reasoning:

• The most likely case to be analyzed is cross-sectional data. Inherent to this type of data is the loss of interpretability when thrown into one pot with other timepoints, e.g. you can't combine the data from the fifth pass of participant A and the first pass of participant B because they might be uncomparable due to variing existing knowledge of the test. This differing knowledge was the reason to add another option for the pass selection in etstat 1.1.2: in addition to "best" and "last" it now features also explicitly "first", because the data generated after the first pass is flawed by preknowledge in achievement/performance tests.
• Longitudinal data may use multiple passes to compute the participants progression over time. This kind of analysis is easy to implement as sub-plugin of etstat.

• The ILIAS T&A-module is conceptualized for achievement/performance tests - not for medical/psychological assessment. Therefore we don't have to worry about assigning wrong psychological variables to participants when assuming things for unanswered or unseen questions in the testanalysis.
• If a participant can't answer a question, no points will be granted and a lower sumscore will be achieved. No examinator will base the grading of an examination only on the individually answered/seen questions, so it's only consequent to interpret missing values as wrong answers for the teststatistics as well. To keep this psychometrically sane, one of three conditions needs to be fulfilled:
  • The time for the test is generally sufficient to begin work (Inangriffnahme) on all items.
  • When using a tight timeframe the questions have to be presented in a random order, so every question has the same chance to remain unseen.
  • The next item can be selected by the participant in a negligible amount of time.
• It seems to me, those conditions are mostly met in usual szenarios within T&A. Specific use cases, where missing/not seen has to be handled differently might be added via sub-plugins for etstat when needed.