Show Advanced KnowledgeHide Advanced Knowledge

Guideline for Community Testing

1 Guideline

Test Cases are managed at http://testrail.ilias.de.
 

1.1 Organising Test Case

• Test Cases are created and organised in Test Suites. The Test Suite is labelled exactly the same way as the respective Mantis-category to insure that one will be able to push bugs.
• Every object or service is dealt with in a separate Test Suite.
• Within a Test Suite the list of Test Cases is structured by sections and subsections. For options with far-reaching impact on the system’s behaviour it is recommended to create separate sections. Example: separate sections for the different option of  “Registration Procedure” in groups.

1.2 Organising the Testing of the Test Case

• From every Test Suite a Test Run is derived: The Test Case Author creates a Test Run handing over the cases to the respective tester. Doing so two settings have to be set:
• In the dialogue of creating the Test Run all cases have to be assigned to the Tester responsible using the field “Assign to”.
• In the dialogue of creating the Test Run the setting “Include all test cases” must be selected. Only with this settings subsequent changes in the Test Suite will automatically be carried out in the Test Run as well.

1.3 Writing Test Cases

Each Test Case consists of

• Title
• Important, non-trivial preconditions: list of prerequisites that are to be fulfilled before the Test Case can ce carried out
• Indicate the role or permissions the test user must have
• References prerequisite Test Cases
• Required non-trivial settings /options of the object or service
• Whether certain data has to be already available or definitively absent.
• Steps: numbered list of actions
• The first step should always clearly state the starting point of the Case (object, tab)
• Steps state alternating actions of the Tester and the system: user does a, ILIAS does b, user does c, ILIAS does d
• Variants can be included underneath the steps to encourage testing alternate paths
• Expected Results:  outline ILIAS is supposed have done as a result of these steps
• If appropriate indicate where exactly and how can one can verify this result
• Indicate any specific settings that ILIAS should have made
• Indicate where the test user is taken to at arriving at the result
• If there is a relevant message provided by ILIAS, state that message.

In general you should provide as much information as necessary to put inexperienced users in the position to execute the test case.
 
The settings “Milestone“, “Priorities” and “Estimate” can be ignored. In the field “References” one can enter the link to the related Featurewiki page. The field “Release” should not be used.
 

1.4 Testing

• All testing for the trunk and recording results is carried out on the official ILIAS Evaluation platform.
• Every Testrail user is required to map his or her user to the respective Mantis-Account to ensure a direct line of communication. This is done in “My settings”, a drop-down of the login-name.
• If possible use the minimum supported version of Internet Explorer (9 for ILIAS 5.1), if this is not possible please use a current stable version of Internet Explorer, Chrome or Firefox
• Carry out all steps of the Test Case.
• Record the Test Result using the “Add Test Result” dialogue.
• Report all defects in Mantis. Use the Push-Functionality. Do NOT write a comment and assign the Test Result Report to a developer. These Reports will not be processed.
• As a tester, do not assume that the developer will set test cases to status "Re-Test". Please always check the status of the corresponding bug report.

1.5 Next Major Release

• Archive test plans / runs from former release
• Update test cases / test suites for new release
• Add test plan for new release
• Perform tests (until end of beta phase)

2 Status

• Effective from release 5.1.0
• Approved by Jour Fixe at: JourFixe-2015-03-02
• Implementation status: partly implemented

3 Discussion

Jour Fixe, March 03, 2015: The suggested guideline has been discussed, slightly modified and approved. It will already be used for testing 5.1.