Skip to main content

Autoscale in Cloud

Kraken CI allows for creating executing machines dynamically in the cloud when they are needed. They can be either virtual machines or containers. When new jobs are triggered and there are no agents available for them, new machines with Kraken agents are spawned. The configuration of the way of spawning new machines is located in agent groups.

Currently, Kraken CI autoscaling is supporting:

  • AWS EC2 virtual machines

Global Cloud Settings#

First, global settings have to be set to allow access to a given cloud provider. In Web UI, on Kraken -> Settings page, in Cloud tab, there is a form for collecting credentials to cloud providers:

In the case of AWS, there are required Access Key and Secret Access Key.

Preparing Cloud Environment#

In case of AWS it is also required to assign proper permissions so that Kraken can create or destroy EC2 instances. List of all requires permissions looks as follows:

{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "VisualEditor0",            "Effect": "Allow",            "Action": [                "ec2:AuthorizeSecurityGroupIngress",                "ec2:CreateKeyPair",                "ec2:CreateSecurityGroup",                "ec2:CreateTags",                "ec2:DeleteKeyPair",                "ec2:DescribeInstances",                "ec2:DescribeInstanceStatus"                "ec2:DescribeInstanceTypeOfferings",                "ec2:DescribeRegions",                "ec2:DescribeSecurityGroups",                "ec2:DescribeVpcs",                "ec2:RunInstances",                "ec2:TerminateInstances",            ],            "Resource": "*"        }    ]}

Configuration in Agents Groups#

Having set credentials to cloud providers, it is possible now to configure aspects of spawning new machines. This can be done on Kraken -> Agents -> Groups page. When a particular group is selected, then its details will be presented on a separate tab. On this tab, there is a section Agents Deployment. So the deployment can be manual (default) or automated to particular cloud provider. In the case of Amazon Web Services, there are the following options that can be set:

The meaning of the settings is as follows.

In General section:

Region - AWS region where a machine will be spawned

Instances Limit - maximum number of spawned machines at a time

Default AMI Image - if AMI image is not provided in job definition that this one is used

In VM Options section:

Instance Type - EC2 instance type that should be used for spawned machines, the full list is available on AWS web page

Spot Instance - indicates where Spot instance should be used (AWS docs)

Monitoring - indicates if monitoring should be enabled (AWS docs)

CPU Credits Unlimited - this allows having more burst power, more in AWS docs

Init Script - a value of this text is passed as UserData to EC2 instance, generally, this is a script that is executing during the first boot of instance (AWS docs)

In Destruction section:

Destroy VM after N jobs - this indicates that the machine should be destroyed (terminated) after it is used in N Kraken jobs, default is 1, ie. machine is used once and then it is destroyed so each time a new fresh machine is used for each job; 0 means that this mechanism is disabled

Destroy VM after N minutes of idle time - this indicates that the machine should be destroyed (terminated) after it is not used in any Kraken jobs for a given number of minutes, default is 10 minutes, ie. a machine is destroyed if it sits idle for 10 minutes; 0 means that this mechanism is disabled

Usage#

Now when a job is assigned to an agents group with configured Agents Deployment then a new machine will be spawned for that job if there are no available agents.