Intermediate Representation Specification (Draft) Cloud Native Continuous Delivery (CNCD) Working Group

This Version
https://github.com/cncd/cncd
Previous Versions
https://github.com/cncd/cncd/tree/master
Editors
bradrydzewski
Raise Issues
https://github.com/cncd/cncd/issues

Abstract

This specification introduces an intermediate representation (IR) for defining continuous delivery pipelines and their container execution environments. A continuous delivery pipeline is an automated manifestation of your process for getting software from version control to release.

Introduction

This section is non-normative.

This specification introduces an intermediate representation (IR) for defining continuous delivery pipelines and their container execution environments. The intermediate representation should be machine writable, machine executable, and platform agnostic.

Frontends

This section is non-normative.

The intermediate representation is not intended to be written by humans. Instead higher-level file formats are compiled to the intermediate representation. These compilers are known as frontends. Example frontend compilers may include:

The Cloud Native Continuous Delivery working group is also authoring a specification for a YAML representation of a continuous delivery pipeline. This will include a reference implementation for compiling to the intermediate representation.

Backends

This section is non-normative.

The intermediate representation should be platform agnostic. This means it can be consumed by different container engines and orchestration platforms. These engines and platforms are known as backends. Example backends may include:

Conformance

As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC2119.

Structure

This section defines the structure of the intermediate representation and custom types used to define the pipeline configuration and execution environment.

The Config object

The Config is the top-level object used to represent the pipeline.

interface Config {
  version: string
  pipeline: Stage[]
  networks: Network[]
  volumes: Volume[]
}

The Stage object

The Stage object defines a group of steps.

interface Stage {
  name: string
  steps: Step[]
}

The name attribute

The name of the stage. This attribute is of type string and is required. The name should be globally unique and must match [a-zA-Z0-9_-].

The steps attribute

The steps attribute is required and must contain at least one step. If the stage contains multiple steps, each step is executed in parallel. The runtime agent should wait until all steps complete prior before it continues to the next stage in the pipeline.

The Step object

The Step object defines an individual container process in the pipeline.

interface Step {
  name: string
  alias: string
  image: string
  pull: boolean
  detached: boolean
  privileged: boolean
  working_dir: string
  environment: [string, string]
  entrypoint: string[]
  command: string[]
  devices: string[]
  extra_hosts: string[]
  dns: string[]
  dns_search: string[]
  shm_size: number
  tmpfs: string[]
  volumes: string[]
  networks: Conn[]
  auth_config: Auth
  on_failure: boolean
  on_success: boolean
}

The name attribute

The name of the container. This attribute is of type string and is required. This container name should be globally unique and must match [a-zA-Z0-9_-].

The alias attribute

The name of the container as defined by the user (i.e. user-friendly name). This attribute is of type string and is required. The container alias must match [a-zA-Z0-9_-]. The alias should be used to create network links, allowing inter-container communication using the alias as the hostname.

The image attribute

The fully qualified image to start the container from. This attribute is of type string and is required. The image name must also include the tag, where applicable.

image: redis:latest
image: library/redis:latest
image: index.docker.io/library/redis:latest

The pull attribute

Pull the latest version of the image from the remote image registry. This attribute is of type boolean and is optional. The default value is false.

The detached attribute

Start the container and send to the background, and proceed to the next step in the pipeline. This attribute is of type boolean and is optional. The default value is false.

The privileged attribute

Start the container with extended privileges. This attribute is of type boolean and is optional. The default value is false.

The working_dir attribute

Start the container in the specified working directory. This attribute is of type string and is optional. Note that the value must be the absolute path of the directory inside the container.

The environment attribute

Set environment variables in the container. This value is of type [string, string] and is optional.

{
  "GOPATH": "/go",
  "DOCKER_HOST": "unix:///var/run/docker.sock"
}

The entrypoint attribute

Override the default image entrypoint.

The command attribute

Override the default image command.

The devices attribute

Expose devices to a container. This value is of type string[] and is optional.

[
  "/dev/sdc:/dev/xvdc"
]

The dns attribute

Sets the IP addresses added as server lines to the container’s /etc/resolv.conf file. This value is of type string[] and is optional.

[
  "8.8.8.8",
  "9.9.9.9"
]

The dns_search attribute

Sets the domain names that are searched when a bare unqualified hostname is used by writing search lines to the container’s /etc/resolv.conf file. This value is of type string[] and is optional.

[
  "dc1.example.com",
  "dc2.example.com"
]

The extra_hosts attribute

Adds additional lines to the container’s /etc/hosts file. This value is of type string[] and is optional.

[
  "somehost:162.242.195.82",
  "otherhost:50.31.209.229"
]

The shm_size attribute

Sets the size of /dev/shm in bytes. This value is of type int64 and is optional.

{
  "shm_size": 64000000
}

The tmpfs attribute

Mount an empty temporary file system inside of the container. This value is of type string[] and is optional.

[
  "/run",
  "/tmp"
]

The volumes attribute

Mount paths or named volumes, optionally specifying a path on the host machine. This value is of type string[] and is optional.

[
  "default:/root",
  "/var/run/docker.sock:/var/run/docker.sock"
]

The networks attribute

Connects the container to zero or many networks. This attribute is of type Conn[] and is optional.

The auth_config section

Authentication credentials used to download a container image. This attribute is of type Auth and is optional.

The on_success attribute

Execute the step when the pipeline state is passing. This attribute is of type boolean and is required. If this value is missing the step may be ignored and may not execute.

The on_failure attribute

Execute the step even when the pipeline state is failing. This attribute is of type boolean and is optional. If this value is missing the the default value of false is assumed.

This attribute can be used in conjunction with on_success. If both attributes are true, the step will always execute regardless of the pipeline state. This may be useful for configuring a notification step that executes both on success and on failure (non-normative).

The Auth object

The Auth object defines authentication credentials used to download container images.

interface Auth {
  username: string
  password: string
}

The Conn object

The Conn object defines a container network connection. This information is used to connect a container to a network with optional support for inter-container communication using hostname aliases.

interface Conn {
  name: string
  aliases: string[]
}

The Network object

The Network object defines a network interface. Each pipeline can have zero or many networks and use these network to facilitate inter-container communication. Example use cases may include linking a service container (e.g. redis) to the build container for integration testing.

interface Network {
  name: string
  driver: string
  driver_opts: [string, string]
}

The name attribute

The name of the network. This attribute is of type string and is required. The name should be globally unique and must match [a-zA-Z0-9_-].

The driver attribute

The name of the network driver. This attribute is of type string and is required.

The driver_opts attribute

Additional network driver options in key value format.

The Volume object

The Volume object defines a container volume. Each pipeline can have zero or many volumes and use these volumes to to persist data and share state. Example use cases may include cloning a git repository to a volume so that subsequent containers can access the source.

interface Volume {
  name: string
  driver: string
  driver_opts: [string, string]
}

The name attribute

The name of the volume. This attribute is of type string and is required. The name should be globally unique and must match [a-zA-Z0-9_-].

The driver attribute

The name of the volume driver. This attribute is of type string and is required.

The driver_opts attribute

Additional volume driver options in key value format.

The State enum

The State enum defines the state of the pipeline. The default pipeline state is Success. If the pipeline encounters and exception or a steps returns a non-zero exit code, the pipeline state is set to Failure.

enum State {
  Success,
  Failure
}

Configuration

The intermediate representation is a JSON document that defines the pipeline execution environment and execution steps. The intermediate representation consists of a top-level object of type Config.

{
  "version": "1",
  "pipeline": [],
  "networks": [],
  "volumes": []
}

The version attribute

The version attribute specifies the version of the intermediate representation. This attribute is of type string and is optional. When empty the runtime should assume to the latest supported version of the specification.

The volumes section

The volumes section defines a list of volumes created at runtime for the pipeline. Individual steps are optionally configured to mount these volumes. Note that volumes are scoped to a single running pipeline, and may not be shared with other running pipelines.

Example volume configuration:

{
  "volumes": [
    {
      "name": "default",
      "driver": "local"
    }
  ]
}

Example step configured to mount the default volume:

{
  "pipeline": [
    {
      "name": "stage_1",
      "steps": [
        {
          "name": "step_1",
          "image": "golang:latest",
          "volumes": [
            "default:/go"
          ]
        }
      ]
    }
  ]
}

The networks section

The networks section defines a list of networks created at runtime for the pipeline. Individual steps are optionally configured to join these networks. Note that networks are scoped to a single running pipeline, and may not be shared with other running pipelines.

Example bridge network configuration:

{
  "networks": [
    {
      "name": "default",
      "driver": "bridge"
    }
  ]
}

Example step configured to connect to the default network:

{
  "pipeline": [
    {
      "name": "stage_0",
      "steps": [
        {
          "name": "step_0",
          "image": "redis:latest",
          "networks": [
            {
              "name": "default",
              "aliases": [ "redis "]
            }
          ]
        }
      ]
    }
  ]
}

Note the above example defines a network alias for the container. Containers on the same network can use the alias as a hostname to connect to the container.

The pipeline section

The pipeline section defines a list of stages that are executed sequentially. Each stage contains of a list of one or more steps.

Example pipeline with multiple stages:

{
  "pipeline": [
    {
      "name": "stage_1",
      "steps": []
    }
    {
      "name": "stage_2",
      "steps": []
    }
  ]
}

The steps section

The steps section defines a list of steps executed in parallel. The runtime must wait until all steps in the current stage are finished before moving to the next stage in the pipeline.

Example stage with multiple steps:

{
  "pipeline": [
    {
      "name": "stage_1",
      "steps": [
        {
          "name": "step_01",
          "image": "golang:latest",
          "entrypoint": [ "/bin/sh" ],
          "command": [ "-c", "set -e; go build; go test"],
          "on_success": true,
          "on_failure": false
        },
        {
          "name": "step_02",
          "image": "node:latest",
          "entrypoint": [ "/bin/sh" ],
          "command": [ "-c", "set -e; npm install; npm test"],
          "on_success": true,
          "on_failure": false
        }
      ]
    }
  ]
}

The step attributes

The step object defines an individual container process. The runtime starts the container process and waits for the container to exit. If the container exit code != 0 the pipeline is set to a Failure state.

Example step:

{
  "pipeline": [
    {
      "name": "stage_1",
      "steps": [
        {
          "name": "step_01",
          "image": "golang:latest",
          "entrypoint": [ "/bin/sh" ],
          "command": [ "-c", "go test"],
          "on_success": true,
          "on_failure": false
        }
      ]
    }
  ]
}

Example docker command used to run the step:

docker run --name "step_01" --entrypoint "/bin/sh" golang:latest "-c" "go test"

The detached attribute

The detached attribute instructs the runtime to start a container in the background without waiting for the container to exit. Subsequent stages and steps in the pipeline are executed while the container continues to run in the background.

Detached containers are run for the duration of the pipeline only. When the last stage in the pipeline completes all service containers are stopped and destroyed. Note that the exit code for detached containers is ignored and does not impact the overall pipeline state.

Example configuration starts a redis:latest service container in detached mode. The service container is available to subsequent steps in the pipeline using the redis hostname.

{
  "pipeline": [
    {
      "name": "stage_1",
      "steps": [
        {
          "name": "step_1",
          "image": "redis:latest",
          "networks": [
            {
              "name": "default",
              "aliases": [ "redis" ]
            }
          ],
          "detached": true,
          "on_success": true
        }
      ]
    },
    {
      "name": "stage_2",
      "steps": [
        {
          "name": "step_2",
          "image": "golang:latest",
          "networks": [
            {
              "name": "default",
              "aliases": []
            }
          ],
          "entrypoint": [ "/bin/sh" ],
          "command": [ "-c", "go test"],
          "on_success": true
        }
      ]
    }
  ],
  "networks": [
    {
      "name": "default",
      "driver": "bridge"
    }
  ]
}

States

The pipeline state can be one of the below values. The pipeline state is set to Failure if an exception is encountered or if a step returns an non-zero exit code.

enum State {
  Success,
  Failure
}

Please note that when the pipeline state is set to Failure the pipeline must continue execution. The state is evaluated prior to execution of each pipeline step to determine if it should be executed or skipped.

Example step executes when pipeline state != Failure

{
  "pipeline": [
    {
      "name": "stage_1",
      "steps": [
        {
          "name": "step_01",
          "image": "golang:latest",
          "entrypoint": [ "/bin/sh" ],
          "command": [ "-c", "go test"],
+         "on_success": true,
          "on_failure": false
        }
      ]
    }
  ]
}

Example step executes when pipeline state == Failure

{
  "pipeline": [
    {
      "name": "stage_1",
      "steps": [
        {
          "name": "step_01",
          "image": "golang:latest",
          "entrypoint": [ "/bin/sh" ],
          "command": [ "-c", "go test"],
          "on_success": false,
+         "on_failure": true
        }
      ]
    }
  ]
}

Security

This section is non-normative.

Backends should audit the use of privileged features and capabilities because hostile authors could otherwise use these settings to compromise the host machine.

Disk Space

This section is non-normative.

Backends should limit the total amount of space allowed for volume storage, because hostile authors could otherwise use this feature to exhaust the system’s available disk space.

Backends should also limit the total amount of space allowed for caching images (e.g. Docker images), and should regularly flush the cache to remove unused or stale images.

Examples

This section is non-normative.

This section provides samples of the intermediate representation to highlight various features of this specification.

Example Pipeline

This section is non-normative.

In the following example the intermediate representation defines two stages. The first stage clones the github project to a shared volume. The second stage executes the test suite for the project.

{
  "pipeline": [
    {
      "name": "clone_stage",
      "steps": [
        {
          "name": "git_clone_step",
          "image": "git:latest",
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "git clone git://github.com/foo/bar.git /go/src/github.com/foo/bar"
          ],
          "volumes": [
            "default:/go"
          ],
          "on_success": true,
          "on_failure": false
        }
      ],
    },
    {
      "name": "test_stage",
      "steps": [
        {
          "name": "go_test_step",
          "image": "golang:latest",
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "go test -v github.com/foo/bar"
          ],
          "volumes": [
            "default:/go"
          ],
          "on_success": true,
          "on_failure": false
        }
      ],
    }
  ],

  "volumes": [
    {
      "name": "default",
      "driver": "local"
    }
  ]
}

Example Pipeline with Services

This section is non-normative.

In the following example the intermediate representation defines a service container (redis) that runs in detached mode, which is non-blocking. Subsequent steps in the pipeline are able to access the service container using its alias hostname.

{
  "pipeline": [
    {
      "name": "clone_stage",
      "steps": [
        {
          "name": "clone",
          "image": "git:latest",
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "git clone git://github.com/foo/bar.git /go/src/github.com/foo/bar"
          ],
          "volumes": [
            "default:/go"
          ],
          "on_success": true,
          "on_failure": false
        }
      ],
    },
    {
      "name": "service_stage",
      "steps": [
        {
          "name": "redis_step",
          "alias": "redis",
          "image": "redis:latest",
          "detach": true,
          "networks": [
            {
              "name": "default",
              "aliases": [ "redis" ]
            }
          ],
          "volumes": [
            "default:/go"
          ],
          "on_success": true,
          "on_failure": false
        }
      ],
    },
    {
      "name": "test_stage",
      "steps": [
        {
          "name": "test_step",
          "image": "golang:latest",
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "go test -v github.com/foo/bar"
          ],
          "networks": [
            {
              "name": "default",
              "aliases": [ "redis" ]
            }
          ],
          "volumes": [
            "default:/go"
          ],
          "on_success": true,
          "on_failure": false
        }
      ],
    }
  ],

  "networks": [
    {
      "name": "default",
      "driver": "bridge"
    }
  ],

  "volumes": [
    {
      "name": "default",
      "driver": "local"
    }
  ]
}

Example Pipeline with Parallelism

This section is non-normative.

In the following example the intermediate representation defines two stages. The first stage clones the github project to a shared volume. The second stage builds and tests the frontend and backend in parallel.

{
  "pipeline": [
    {
      "name": "clone_stage",
      "steps": [
        {
          "name": "git_clone_step",
          "image": "git:latest",
          "working_dir": "/go/src/github.com/foo/bar",
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "git clone git://github.com/foo/bar.git /go/src/github.com/foo/bar"
          ],
          "volumes": [
            "default:/go"
          ],
          "on_success": true,
          "on_failure": false
        }
      ],
    },
    {
      "name": "test_stage",
      "steps": [
        {
          "name": "go_test_step",
          "image": "golang:latest",
          "working_dir": "/go/src/github.com/foo/bar",
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "go test -v; go build"
          ],
          "volumes": [
            "default:/go"
          ],
          "on_success": true,
          "on_failure": false
        },
        {
          "name": "node_test_step",
          "image": "node:latest",
          "working_dir": "/go/src/github.com/foo/bar",
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "npm install; npm test; npm run bundle"
          ],
          "volumes": [
            "default:/go"
          ],
          "on_success": true,
          "on_failure": false
        }
      ],
    }
  ],

  "volumes": [
    {
      "name": "default",
      "driver": "local"
    }
  ]
}

Example Travis Configuration

The goal of this specification is to provide a machine writable format to which higher-level configurations can compile. This is an example .travis.yml configuration:

language: node

node_js:
  - 6.1

install:
  - npm install
script:
  - npm test

Example .travis.yml compiled to the intermediate representation:

{
  "pipeline": [
    {
      "name": "clone_stage",
      "steps": [
        {
          "name": "clone_step",
          "image": "node:6.1",
          "working_dir": "/Users/travis/build",
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "git clone git://github.com/foo/bar.git /Users/travis/build"
          ],
          "volumes": [
            "default:/Users/travis/build"
          ],
          "on_success": true,
          "on_failure": false
        }
      ],
    },
    {
      "name": "install_stage",
      "steps": [
        {
          "name": "install_step",
          "image": "node:6.1",
          "working_dir": "/Users/travis/build",
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "npm install"
          ],
          "volumes": [
            "default:/Users/travis/build"
          ],
          "on_success": true,
          "on_failure": false
        }
      ],
    }
    {
      "name": "script_stage",
      "steps": [
        {
          "name": "script_step",
          "image": "node:6.1",
          "working_dir": "/Users/travis/build",
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "npm test"
          ],
          "volumes": [
            "default:/Users/travis/build"
          ],
          "on_success": true,
          "on_failure": false
        }
      ],
    }
  ],

  "volumes": [
    {
      "name": "default",
      "driver": "local"
    }
  ]
}

References

Normative References

JSON SPECIFICATION
A. Barth. HTTP State Management Mechanism. April 2011. https://tools.ietf.org/html/rfc6265

Informative References

DOCKER RUN
Docker Run Reference, Docker Inc.
DOCKER COMPOSE
Compose File Reference, Docker Inc.