Branch Results page presents high level information about flows
and runs of given branch.. It organizes flows in rows. The top row
contains the latest flow. Each flow row contains list of boxes. Each
box contains information about a run of given stage. The order of
boxes with runs is chronological according to their start time.
The color of run box indicates final status of the run:
- red: one or more jobs finished with an error
- yellow: no errors in jobs but some test results failed or there are issues discovered in the code
- green: all jobs finished without error and there is no failures in tests and no issues in the code
A run box contains:
- data about execution: total number of scheduled jobs, number of erred jobs, etc.
- summary of testing: pass ratio, total, regressions and fixes counts, etc
- issues summary: total number of discovered issues, number of new issues
This run has 1 successful job that reported tests results. There were run 100 test cases, where 16 did not passed, so pass ratio is 16%. There were detected 15 regressions and 13 fixes. There are failures so the box is yellow.
This run has 4 job with no errors. Some of them that reported tests results, the other ones reported issues. There were run 17 test cases, where all passed, so pass ratio is 100%. There were detected 15 issues in the code. There are issues so the box is yellow.
This run has 2 job with no errors. In the reported tests results, there are 100% passed test cases. No failures and no issues so the box is green.
This run has 4 job with 1 error. All tests reported are passed but there could be not reported tests due to error in job execution. There were detected 6 issues in the code. There is one erred job so the box is red.
A job that performs testing reports test results. The test results are
visible on a
Run Results page, in
Test Results tab. They are
presented in a table.
The table presents test case results in each row. They contain a details of the test case result:
- the result status
- a name of the test case
- job name and ID
- age, instability and relevancy
- execution environment: system, agent group and actual agent and config
In Kraken test case result beside status can contain values. Valuas are pairs: attribute and its value. They are also a result of a test case but hold numbers. They can hold e.g. performance data or anything similar to performance. These values are tracked over time. Kraken can draw charts of values and do more things with them.
It is possible to filter test results by various attributes:
No Changes; the change is determined between current result and previous result from previous run
Agewhere age is a number of previous runs with the same result as in current run; so e.g. age 0 means that in previous run test case had different result than it has now (so these are regressions and fixes); age 1 means that current run and previous run has the same result but the result before previous one is different
Newis a shortcut to set
Agebetween 0 and 0, ie. show changes that happend in last run (fixes + regressions)
Instabilitywhere instability is a count of result status changes in last 10 runs; if this is 0 then in the last 10 runs test case has the same result (it was pretty stable); if this is 10 then it means that in every run of the last 10 runs the status was changing comparing to previous one (it was very unstable)
Jobname of ID
Result Changes: show only regressions. Regression means
a change from non-passed status to passed status.
Result Status and
Result Changes: show only failures
that are regressions.
Age: show only results in age between 2 and 3.
Instability: show only results that have instability
greater than 7 i.e. very unstable test cases.
Test case results can be sorted. This can be done using several columns:
By default results are sorted by test case name.
One of the most important sorting order is by
Relevancy is estimated according to this rules:
- 0 is initial
- +1 for not passed
- +1 for failure
- +1 for instability <= 3
- +1 for age < 5
- +1 for regression
This way the most relevant results get the highest score. They need an attention.
Each test case has its history of results. In the table with test case results a link under test case name leads to a page dedicated to given test case. This page presents a table with historical results of the test case.
There is another tab called
Charts that shows these results on charts.
The left chart shows history of result status. The right one presents history of values in time. Error bars are shown if there were run multiple iterations and there were reported multiple values for given attribute.