Source Code as a Type of Submission

1 Initial Problem

Repeatedly, teachers at our university would like to have automated evaluation of programming tasks. Unfortunately, the requirements of the lecturers often have highly individual parts. These are ( discipline-)specific script languages in different teaching contexts. These aspects make a permanent integration of individual programming tasks into the ILIAS Core impossible due to the small target groups and short half-lives.

At the same time many requirements of teachers have corresponding intersection in the area of organization and administration of programming tasks, many of these required components are already contained in the exercise object of ILIAS, the overview over several tasks, their participants and the specific performance of these, are all already contained.

This is where we would like to start and use the good foundation of the organisation and administration of exercises to combine them with the different aspects of individual programming tasks. It quickly became clear to us that the overhead of permanent integration and maintenance in the core of ILIAS is not in a good cost-benefit ratio. Not to mention that as a central e-learning center we have not enough resources and competencies at our disposal to maintain and support several automated programming task systems for different programming languages and requirements.  

1.1 User Stories

1.1.1 Teacher

Teacher Stefanie teaches script programming at the FH Dortmund. She wants to offer her students the opportunity to practice for the exam on their own. She creates a new assignment of the type "Source Code" in the exercise object, selects the programming language "Python" and then the task "control structures".

After the submission deadline Stefanie looks into the exercise unit under "Submits and grades" and recognizes by the status passed/not passed how her students completed the exercise, additional feedback to the automatic evaluation can be displayed in the text feedback field.

1.1.2 Student

Student Simon is interested in scripting languages and would like to prepare for the corresponding exam. He calls up Prof. Stefanie's exercise unit and edits the current exercise unit "Python - Control Structures", then submits his source code. Shortly after the submission Simon can see the result in the exercise.

2 Conceptual Summary

We want to provide lecturers with a standardized set of requirements that will enable them to implement a programming task service themselves, provide it with submissions from assignments, and subsequently query the results. Due to the clearly defined technical and organizational standard, this is also possible for different services, teachers and subjects. 

Each programming task service is integrated into ILIAS by a minimal plugin, which defines the name of the service e.g. 'Prof. Recker - Python 2.7', the URL of the endpoint and the corresponding API key. All programming task services have to support defined API functions, only so several different programming tasks can be connected, ILIAS only manages the results and the submitted source code, the analysis and evaluation of the tasks is completely up to the programming task service. Communication takes place via the same API functions regardless of the programming language or programming task used: 

• getAssignments
  • List of all existing programming tasks at this Endpoint
  • → assignmentID
  • → assignmentName
• addSubmission
  • Upload of a new student submission
  • → ILIAS-Returned-ID
  • → Source Code des Studierenden
  • → assignmentID
• getResult
  •  ask for submission result
  • → ILIAS-Returned-ID
  • ← passed/not passed
  • ← Message
  • ← Mark
• version
  • ← versionID
  • ← Status des Endpoint-Services

2.1 Usage

After creating a new assignment type "source code", the teacher can choose the new evaluation 'Prof. Recker - Python 2.7', the available submissions/programming tasks of the service are queried and displayed directly from the API endpoint (getAssignments). So it is a possibility to verify in the creation process of a exercise if this service is reachable and functional. In order to complete the exercise unit, the teacher now chooses a programming task and decides whether the students' submission should be evaluated directly or only after the deadline. (see Abb. 1)

If a student is submitted (see Abb. 2), the submitted source code is sent to the end point of the programming task service. A process controlled by CronJob regularly asks the individual programming task services whether results are now available for previously unvalued submissions. The student can view the result in the exercise unit. If automated feedback is possible, it appears under 'Evaluation of the tutor'.

The teacher can view the results of the automated review under 'Submissions and grades', using the existing fields of ILIAS, as a mandatory field 'passed/not passed' and optionally submit a grade and text for the feedback field. If the teacher needs a new automated evaluation of one or more submissions, she can trigger this by clicking on 'Re-evaluate submission' in the action menu. (see Abb. 4)

3 User Interface Modifications

3.1 List of Affected Views

Exercise > Add Assignment (new assignment type: "Source Code")
Exercise > Submissions and Grades (possibilty to re-evaluate using automated testing in the dropdown)

3.2 User Interface Details

New interface elements:

• Dropdown Menu: Evaluation (Auswertung)
  • See 2.1 Usage: Choose existing programming task service
• Dropdown Menu: Assignment (Aufgabe)
  • See 2.1 Usage: Choose available submissions/programming tasks of this service
• Checkbox: Direct analysis (Direkte Auswertung) [default unchecked]

• Type of Submission
  • added new Submission Type "Source Code"

New interface element:

• Actions Menu Entry: Reavaluate Submission (Aufgabe neu bewerten)

3.3 New User Interface Concepts

{If the proposal introduces any completely new user interface elements, you might consult UI Kitchen Sink in order to find the necessary information to propose new UI-Concepts. Note that any maintainer might gladly assist you with this.}

4 Technical Information

{The maintainer has to provide necessary technical information, e.g. dependencies on other ILIAS components, necessary modifications in general services/architecture, potential security or performance issues.}

5 Contact

• Author of the Request: {Please add your name.}
• Maintainer: {Please add your name before applying for an initial workshop or a Jour Fixe meeting.}
• Implementation of the feature is done by: {The maintainer must add the name of the implementing developer.}

6 Funding

If you are interest in funding this feature, please add your name and institution to this list.

• Fachhochschule Dortmund (Thomas Langkau)
• …

7 Discussion

Killing, Alexander [alex], 24 July 2019: This sounds like a proprietary approach for something that LTI tries to do. Or do I misunderstand this completely? Or something x-API-Proxy like may even be suitable.

8 Implementation

{The maintainer has to give a description of the final implementation and add screenshots if possible.}

Test Cases

Test cases completed at {date} by {user}

• {Test case number linked to Testrail} : {test case title}

Approval

Approved at {date} by {user}.