Other CI Systems
Introduction
In this section, three CI/CD tools will be compared with Kraken CI. These tools are among the most popular systems:
The most significant difference between Kraken CI and these three systems, as well as other CI/CD systems, is that they lack test result analytics. All three systems barely touch upon the topic and are only able to report results acquired during a build.
Having only information about passes and failures, it is difficult to make a decision. It is not known if this failure has appeared before. Maybe it has been failing for several months, or perhaps it is sporadic and only fails from time to time. Alternatively, it could be a significant regression that requires urgent fixing.
This requires either using other third-party tools that can fill this gap or developing custom scripts that can analyze the results and generate comprehensive reports.
In contrast, Kraken CI focuses on this topic and excels in both execution and results analytics. It can show:
- fixes and regressions,
- age of test result,
- stability of a test case,
- importance of negative test result,
- history of a test case in a table and in a chart,
- total results chart ie. pass ratio over time,
- pass ratio chart for selected subset of tests (e.g. tests results obtain on particular operating system).
Furthermore, it allows users to comment on test results, such as referencing a submitted bug or identifying the root cause, among other things.
The details are presented in Test Results Basics and Test Results Tracking of Kraken CI docs.
But now let's discuss the differences between specific CI/CD systems.
Jenkins
Jenkins is an open-source on-premises automation server used for building, testing, and deploying software. It offers a broad range of plugins and integrations that support continuous integration and continuous delivery workflows. Jenkins is widely used in software development and is recognized for its flexibility and extensibility.
Bright Side
Jenkins has many powerful features, with one of the most prominent being its flexible and convenient method for defining pipelines using scripting code.
Another very positive aspect of Jenkins is its community. The community, besides providing lots of blog posts and tutorials about Jenkins, also develops many plugins that greatly improve Jenkins.
Dark Side
1. Plugin Overload and Compatibility Issues
Jenkins relies heavily on plugins for extended functionality. However, the sheer number of plugins available can make it challenging to select, manage, and maintain the right set of plugins. Plugin compatibility and stability issues can also arise, leading often to conflicts or performance problems.
Kraken CI has a different approach. All of its features are built-in, which ensures compatibility and thorough cross testing. Users are encouraged to contribute directly to the core of Kraken CI. The only extensions available to users are tools, which enable the integration of third-party tools with Kraken CI. This is similar to third-party actions in GitHub Actions.
2. Bad User Experience in UI
Jenkins' user interface (UI) is often criticized for being outdated and not user-friendly. There is a dichotomy between the classic UI and the Ocean Blue UI. Both show more or less the same things, but one is better in some areas, while the other is better in other areas. This leads to confusion.
Many UI pages in Jenkins lack usability. Some areas are exceptionally
bad, such as presenting logs. When the logs are very large, Jenkins
shows them very slowly, especially if timestamps are
enabled. Sometimes the UI freezes completely and cannot show
anything. There is also a page called Pipeline Steps that shows logs
for each step separately, but there are so many internal steps
presented that it is very hard to navigate.
Presenting logs in Kraken CI is impressive. They are divided by default into steps that the user has defined in a workflow, or they are displayed in a flat format. Users can enable internal logs to see details, and loading logs is paged, so it is not necessary to load the entire log at once. However, it is still possible to download the entire log in raw format. Logs can be filtered by multiple fields and viewed at the individual step level, job level, run, flow, branch, or project levels. ANSI codes are colored, and there are many more features. Check out these two blog posts that present these features:
Standing out Logs Handling and Further Logging Improvements.
There are more examples like that. For instance, navigation, browsing test results, and so on. In Kraken CI, all of these aspects were thoughtfully reevaluated and engineered with user experience (UX) in mind.
3. Reliance on Groovy Scripting
Jenkins pipelines, which are used to define CI/CD workflows, are written in Groovy scripting language. While this provides flexibility and extensibility, it requires from developers to learn and use Groovy, which can be an additional learning curve for teams not familiar with Groovy.
In the case of Kraken CI, Python is used as the language to define workflows. According to the TIOBE Index, Python is ranked number 1, making it the most popular programming language, while Groovy is ranked 44th.
GitHub Actions
GitHub Actions is a powerful and flexible automation platform that allows you to streamline and optimize your software development workflows. With GitHub Actions, you can automate tasks, build, test, and deploy your applications directly from your GitHub repositories.
Bright Side
The biggest advantage of GH Actions is that it is integrated with GitHub repositories, and it is free up to a certain usage level. In addition, it allows for the extension of defined workflows by using community-developed actions. The number of available third-party actions is quite substantial.
It is also possible to expand the computing power by connecting your own "self-hosted" machines to the pool of all machines. Therefore, workflows can run on both GitHub's machine cluster and on your own on-premises machines.
Aside from that, GitHub Actions can do more than just CI/CD. Workflows can be triggered for various project events, such as a new issue being submitted, a new comment being added to an issue, a repository being forked, a release being created, and more.
Dark Side
1. Poor, not Scalable UI/UX
The user interface is cluttered and does not scale well for larger projects. When a workflow reuses other workflows, the job names become excessively long and are frequently truncated in multiple areas within the UI. This makes it difficult to identify which jobs we are currently viewing. The problem is further compounded when nested workflows are utilized.
The UI also does not scale well for a larger number of jobs. If there are more than 30 jobs in a workflow, it becomes impossible to filter, sort, or list them in pages. Instead, all jobs are just presented to you in a flat list.
One more weakness is the logs panel. Initially, it appears to be sleek. Nonetheless, if there are numerous steps and the logs are quite extensive, containing thousands of lines, then navigation becomes difficult. While jobs are in progress, the UI attempts to predict what to display, causing it to jump from one location to another, making it difficult to choose the correct step. Additionally, paging through the logs is not feasible. They are all displayed at once and are progressively loaded while scrolling down. This is a pretty ineffective method of finding the appropriate location in the log if the log contains hundreds of pages.
Kraken's logging facilities in the UI were already described in the Jenkins section. In terms of UI scalability, this is one of Kraken's goals. It is designed from the ground up to handle multiple jobs, up to thousands or even tens of thousands. It provides means for filtering and sorting jobs, and presenting workflows, even those that are more complicated and involve multiple forks, in a more scalable way than GH Actions.
2. Reusability Is Quite Hard
Initially, it seems that reusability is well supported in GH Actions workflow using third-party actions and reusable workflows. However, upon closer inspection, it is not as seamless as it appears. Externalizing some parts of the workflow into reusable actions or workflows can be quite tedious. If there is a need to locally reuse a fragment of code, one would like to use the YAML feature called anchors and aliases. Unfortunately, GH Actions does not support this feature.
The situation becomes complicated if a user needs to maintain multiple repositories. Certain parts of workflows can be reused, but the framework of the workflow, including trigger definitions, global environment variables, and main jobs (which may reference reusable workflows), must be present in every repository. Ideally, one would like to fully externalize the entire workflow to a single repository and simply reference it in every other repository.
Kraken allows for defining a workflow for particular repository in several way. The workflow definition can be stored
- in Kraken server,
- in the same repository,
- in another repository.
In the first case, the entire workflow definition resides in the Kraken server's database. It is then possible to link the workflow definition to any Git repository - whether it is the repository that needs to be built and tested or any other repository. This makes it easy to experiment with a workflow for any repository without the need to make changes. Later on, it is easy to transfer that workflow to the relevant repository. Furthermore, the entire workflow definition can be extracted to a separate repository and reused in multiple other repositories.
3. Cloud and Security
GitHub Actions is a cloud-based CI/CD tool, which means that its reliability and availability are out of your control. If an outage occurs, the only thing left for you to do is wait until GitHub's team fixes their services. This may last from a few minutes to a few hours.
Although GitHub Actions provides security features like access controls, secrets management, and vulnerability scanning, it may not fulfill the specific security or compliance requirements of all organizations or industries. Organizations with stringent security or compliance needs must evaluate GitHub Actions' security features carefully and ensure that they align with their requirements.
GitLab CI
GitLab CI is a comprehensive DevOps platform that combines source code management, CI/CD pipelines, issue tracking, and more in a single integrated solution. It enables organizations to streamline their software development processes, promote collaboration, and achieve faster and more efficient delivery of high-quality software.
Bright Side
GitLab CI enables the definition of pipelines as code by using a YAML configuration file. This approach makes the CI/CD process version-controlled, easily replicable, and auditable. Infrastructure modifications, test scripts, and deployment steps can be defined alongside the code, promoting transparency and collaboration.
Furthermore, GitLab is offered as software that can be installed on-premises, as well as a service that can be used in the cloud. This way, it combines the software aspect of Jenkins with the cloud service aspect of GitHub Actions in one solution. The choice belongs to the user.
GitLab offers a wide range of features and integrations, with most of them only available in the Enterprise Edition. These features include code quality checks. Another strong aspect of GitLab is its reusability, as it is easy to reuse fragments of YAML definitions thanks to anchors in the language.
Dark Side
1. Too complex in advanced cases
The learning curve for GitLab CI/CD is rather steep, and some poor
design decisions have made configuring CI/CD pipelines in the
.gitlab-ci.yml file quite complex. The documentation is full of
notes that certain aspects are not obvious, discouraged, or may
behave unexpectedly. Furthermore, the issue tracker is filled with
problems related to excessively complicated features.
One of the problems is that a CI pipeline is triggered by all possible events. Then it is user role to explicitly exclude which jobs should be run or not for particular event. Sometimes it leads to situations that a pipeline is triggered multiple times. There is a dedicated section in the docs how to avoid such cases: https://docs.gitlab.com/ee/ci/jobs/job_control.html#avoid-duplicate-pipelines
All of these factors make developing more advanced workflows quite challenging.
In contrast, Kraken is more deterministic. Workflows are triggered by user-indicated events, and it is clear why a given workflow was started - this information is presented in the UI.