Tester Guide

TestRail for Testers: Using Test Runs, Reporting Bugs and Saving Test Results

What are Test Runs?

While the creation and organisation of test cases always happens in the tab Test Suites & Cases, the test itself has to be executed in the tab Test Runs & Results.

Just like the test suites, test runs are organised according to the ILIAS modules. In most cases, a test run comtains all test cases that were created in the suite. Test runs are only set up by the test manager.

A test run looks very similar to the suite it is derived from. But whole the test suite only indicates how a module should be tested, the test run contains the test itself.

In this sense, the test suite as a kind of "test case library" is of interest to the author, while the test run matters to the tester. There are a few significant differences between the two:

  • Only in a test run, it is possible to save results. The results document whether a test case could be executed successfully.
  • Test cases in a test run can be assigned to the person who should execute them.
  • The test cases cannot be edited anymore within the test run.
  • After the testing period, all test runs get archived and cannot be changed anymore. In contrast, the test cases in the test suite remain editable and stay where they are. They remain in TestRail and can be re-used (if necessary with improvements or changes) for the next testing period.

When the test runs have been created for the test of a new ILIAS release, TestRail shows an overview for each run. It contains the number of passed, blocked, feedback, untested and failed test cases. In the test run, the current status of each test case is shown next to its name.

Overview of a Test Run (for the Administration module)

Executing Test Cases and Saving Results

In order to execute a test case, you have to open it by clicking on it. There are two ways to do this:

  1. By clicking on its title: The test case is then opened on a new page in your browser.
  2. By clicking on the small arrow glyph ">": The test case then gets opened in a new column next to the overview of the test run. This allows for faster navigation from one test case to the next.

Clicking on the small grey arrow will open the test case in the three-column view.

Once you have opened the test case, look for any preconditions you need to fulfill in order to execute the test case (for example concerning permissions of your test user or special settings). Next, execute all the steps and compare the actual result to the expected results listed in the test case.

Finally, save the result of the test case by clicking on the button Add Test Result. TestRail will then open a modal where you need to enter the following information:

The three-column view in TestRail

  • Status: "Untested", "Passed" and "Failed" are self-explanatory.
    • Retest: After fixing a bug that lead to the status "Failed", the maintainer will set the status of the test case to "Retest". This indicates that the test case should be executed again. If the original problem was fixed and no new issues turn up, you can then set the test result to "Passed".
    • Blocked: This status indicates that a test case cannot be executed. This can happen when a bug elsewhere prevents the tester from even starting the test case or when the test case itself contains an error.
      • If a bug in another test case is blocking execution, please add the respective Mantis ID under "Defects".
      • IMPORTANT: Whenever you set this status, please be sure to assign the test case to the appropriate person in order to fix it (for example the test case author). Only assign yourself if you can take care the issue yourself, e.g. unblock the test case after a related bug fix! If you are not sure, please assign the test case to the test manager!
    • Feedback (NEW!): This status is set by a maintainer to indicate that a tester needs to provide more information in order to get a bug fix. It’s a new status introduced with the test of ILIAS 9. Once you provided the missing information, be sure to set a new status accordingly (e.g. failed, passed or retest).
  • Comment: If you find a bug, any text entered into this field will get pushed to Mantis. Maintainers need detailed information on how to reproduce the bug.
  • Assign To: This field can be used in order to reassign the test case to another tester, the maintainer or the test manager.
  • Permalink: Add a permalink to the object on the test installation so that the maintainer can easily reproduce the issue you found.
    • Hint: Do NOT simply copy the link from your browser navigation! The permalink can be found in the footer of most pages in ILIAS. If no permalink is provided, open the Info tab of the object and copy the permalink from the footer.
    • Hint: Rename the respective object by putting the Test Case ID (T12345) or the Mantis ID (#34567) in its title. This makes it easier for you to keep track of your test objects and issues.
  • Defects: This field is used to list the IDs of bugs added to Mantis. Next to it, there are two links "Push" and "Add".
    • The link Push opens a new modal which allows you to send a bug to Mantis directly from TestRail. This is the fastest and easiest way to report a bug that occured during testing.
      • IMPORTANT: All bugs need to be pushed to or reported in Mantis! It is NOT sufficient to merely record a bug in TestRail without pushing it to Mantis. Only Mantis bugs are treated by our maintainers!
      • Please note: In order to push bugs to Mantis you need to enter your Mantis account data in the  TestRail settings.
    • The link Add leads directly to Mantis. You can there enter all the bug information manually. Generally, this is not required, as pushing is more convenient. A description of all the fields in a bug report can be found in the chapter How to Write a Bug Report.

The modal "Add Test Result"


Information Message
No comment has been posted yet.