Skip to main content

Hacking on Kargo

Kargo is implemented in Go. For maximum productivity in your text editor or IDE, it is recommended that you have installed the latest stable releases of Go and applicable editor/IDE extensions, however, this is not strictly required to be successful.

Running Tests​

In order to minimize the setup required to apply small changes and to reduce the incidence of tests passing locally, but failing during the continuous integration process due to environmental differences, we've made it easy to execute tests within a container that is maximally similar to those used in CI.

To take advantage of this, you only need make and Docker (or a Docker-compatible container-runtime).

To run all unit tests:

make hack-test-unit
info

If you wish to opt-out of executing the tests within a container (for performance reasons, perhaps), drop the hack- prefix from the target:

make test-unit

This will require Go to be installed locally.

Running Linters​

It is also possible to execute a variety of different linters that perform static code analysis, detect code hygiene issues, assert adherence to project standards, etc. As with unit tests, we've made it easy to execute linters within a container that is maximally similar to those used in CI.

To lint Go code only:

make hack-lint-go

To lint generated protobuf definitions only:

make hack-lint-proto

To lint Helm charts only:

make hack-lint-charts

To run all linters with one command:

make hack-lint
info

If you wish to opt-out of executing any or all linters within a container (for performance reasons, perhaps), drop the hack- prefix from the desired target.

This will require quite a variety of tools to be installed locally, so we do not recommend this if you can avoid it.

Executing Code Generation​

Anytime the contents of the api/ directory have been modified, a code generation process must be manually executed. As with tests and linters, this process is easy to execute within a container, which eliminates the need to install various tools or specific versions thereof:

make hack-codegen
info

If you wish to opt-out of executing code-generation within a container (for performance reasons, perhaps), drop the hack- prefix from the target:

make codegen

This will require quite a variety of tools to be installed locally, so we do not recommend this if you can avoid it.

Building the Image​

To build source into a Docker image that will be tagged as kargo:dev, execute the following:

make hack-build
info

There is seldom a need to do this, as the next section will cover a better option for rapidly building and deploying Kargo from source.

info

The Docker buildx machine required by the build process has to be created with the --driver-opt network=host option to allow it to access the (temporary) local image registry used for the base image.

If you encounter an error during the build process (e.g. failed to resolve source metadata for localhost:5001/kargo-base:latest-arm64 or granting entitlement network.host is not allowed by build daemon configuration), you may need to (re)create the machine using docker buildx create with this option set.

Iterating Quickly​

This section focuses on the best approaches for gaining rapid feedback on changes you make to Kargo's code base.

The fastest path to learning whether changes you have applied work as desired is to execute unit tests as described in previous sections. If the changes you are applying are complex, it can also be advantageous to exercise them, end-to-end, as a user would. Because Kargo is dependent on a Kubernetes cluster, this raises the question of how Kargo can not only be built from source, but also deployed to a live Kubernetes cluster efficiently enough to enable a tight feedback loop as you continue iterating on your changes.

The remainder of this section covers the approaches we recommend for enabling this.

  1. Launch or re-use an existing local Kubernetes cluster.

    Any of the following options are viable:

    If you are a Docker Desktop user, you can follow these instructions to enable its built-in Kubernetes support. If it's already enabled, you're ready to go.

    info

    A specific benefit of this option is that nothing special is required in terms of creating a local image registry connected to the cluster. Additionally, this approach requires no specific port-forwarding rules to be defined.

    info

    Although this is one of the fastest paths to a local Kubernetes cluster, be aware that Docker Desktop supports only a single Kubernetes cluster. If that cluster reaches a state you are dissatisfied with, resetting it will remove not just Kargo-related resources, but all your workloads and data.

  2. Optional: Configure and start a tunnel for the external webhooks server:

    If you are working on or testing the external webhooks server, you will benefit from configuring a tunnel from the outside world so that traffic originating from platforms like GitHub, Docker Hub, and others can reach the server. It is easy to accomplish this using ngrok. If you wish to do so, you must first install ngrok yourself.

    With ngrok installed, you can conveniently open a tunnel to localhost:30083 (where the next step will run the external webhooks server) using:

    make hack-ngrok

    Allow this process to run while you are working on or testing the external webhooks server. Interrupt it with ctrl + c when you are done.

    If you have a paid ngrok account that allows you to use a custom domain name for your tunnels, you can specify that domain name using the KARGO_EXTERNAL_WEBHOOKS_SERVER_HOSTNAME and (to properly set the protocol scheme) KARGO_EXTERNAL_WEBHOOKS_SERVER_TLS_TERMINATED_UPSTREAM environment variables before running the make target:

    export KARGO_EXTERNAL_WEBHOOKS_SERVER_HOSTNAME=my-tunnel.ngrok.io
    export KARGO_EXTERNAL_WEBHOOKS_SERVER_TLS_TERMINATED_UPSTREAM=true
    make hack-ngrok

    If the KARGO_EXTERNAL_WEBHOOKS_SERVER_HOSTNAME environment variable is undefined, the tunnel will utilize a dynamically-generated subdomain of ngrok.io.

  3. Build and deploy Kargo from source:

    Tilt is a convenient tool that builds container images from source and seamlessly deploys them to a local Kubernetes cluster. More importantly, it enables developers to rapidly rebuild and replace running components with the click of a button. You do not need to install it in advance.

    make hack-tilt-up
    info

    The hack-tilt-up target will ensure the installation of a suitable versions of tilt and helm into hack/bin/.

    When run for the first time on a new development cluster, make hack-tilt-up will install suitable versions of cert-manager, Argo CD, and Argo Rollouts into your cluster using helm.

    The Argo CD dashboard will be exposed at localhost:30080.

    The username and password are both admin.

    You may safely ignore any certificate warnings.

    Tilt will launch a web-based UI running at http://localhost:10350. Visit this in your web browser to view the build and deployment status of each Kargo component as well as the logs from each component.

    info

    Tilt is often configured to watch files and automatically rebuild and replace running components when their source code is changed. This is deliberately disabled for Kargo since the Docker image takes long enough to build that it’s better to conserve system resources by only rebuilding when you choose. The web UI makes it easy to identify components whose source has been altered. They can be rebuilt and replaced with a single click.

    info

    If you specified a custom domain name for a tunnel to the external webhooks server in the previous step by defining a value for the KARGO_EXTERNAL_WEBHOOKS_SERVER_HOSTNAME environment variable, you should export the same value for that environment variable before running make tilt up as well:

    export KARGO_EXTERNAL_WEBHOOKS_SERVER_HOSTNAME=my-tunnel.ngrok.io
    make hack-tilt-up

  4. If necessary, build the CLI from source:

    make hack-build-cli

    This will produce an executable at bin/kargo-<os>-<arch>.

    You can log in using:

    bin/kargo-<os>-<arch> login http://localhost:30081 \
      --admin \
      --password admin \
      --insecure-skip-tls-verify

  5. If necessary, access the Kargo UI at localhost:30082.

    The admin account password is admin.

    You may safely ignore any certificate warnings.

  6. When you are done with Tilt, interrupt the running make hack-tilt-up process with ctrl + c. Components will remain running in the cluster, but Tilt will no longer be in control. If Tilt is restarted later, it will retake control of the already-running components.

    If you wish to undeploy everything Tilt has deployed for you (except for prerequisites), use make hack-tilt-down.

  7. Clean up your local Kubernetes cluster.

    Docker Desktop supports only a single Kubernetes cluster. If you are comfortable deleting not just just Kargo-related resources, but all your workloads and data, the cluster can be reset from the Docker Desktop Dashboard.

    If, instead, you wish to preserve non-Kargo-related workloads and data, you will need to manually uninstall Kargo's prerequisites:

    make hack-uninstall-prereqs

Working with PostgreSQL​

Tilt runs the PostgreSQL bundled with the Helm chart, configured by the database.postgres values in hack/tilt/values.dev.yaml, and forwards 127.0.0.1:15432 to its port 5432. The database, username, and password are all kargo. Inside the cluster, the address is kargo-postgres.kargo.svc:5432.

Two kinds of data live in it. The management controller mirrors Projects, Stages, and Promotions from Kubernetes, which remains their source of truth. Project and Stage rows record identities and follow their objects, including out of existence. A Stage row also records the selectors by which the Stage governs Targets (NULL for a classic Stage), and carries the number last given to one of its PromotionRollouts, which the mirror never writes. Promotion rows record what each Promotion did, its Stage, Freight, Target if any, phase, and timing, along with the Freight of the whole collection it rendered, one row per origin (promotion_freight), and are never deleted: Kubernetes retains only a bounded number of recent Promotions per Stage, and the database keeps them all as a history. A Promotion deleted before it finished is recorded as Aborted as of its deletion, so no row claims a Promotion is running that no longer exists. Targets, on the other hand, exist only in the database: the API server serves them through /v1beta1/projects/{project}/targets, and Target is no longer a custom resource at all. The chart installs no CRD for it, kubectl does not see Targets, and the Go type keeps the shape of a Kubernetes resource without being one (no apiVersion or kind, no scheme registration), so that clients and the UI can treat it like one. Target custom resources from earlier versions are not migrated; a chart upgrade removes the CRD and them with it. A Target names its Project rather than referencing the mirrored row, so writing one never waits for the mirror. Deleting a Project does not yet remove its Targets.

Only control plane components open the database. The controller may run far from the control plane, on infrastructure where the database is not reachable, so it resolves the Targets a Stage governs and the Target a Promotion promotes to by calling the API server at API_SERVER_ADDRESS. It acts as the ServiceAccount of the control plane that its Kubernetes credential for the control plane resolves to. At startup it asks the control plane who it is, with a SelfSubjectReview, and refuses to start if the answer is not a ServiceAccount; from then on it requests short-lived tokens for that ServiceAccount through the Kubernetes TokenRequest API and sends them to the API server, which verifies each with a TokenReview and authorizes each request against the ServiceAccount's RBAC, which grants list and get on targets. client-go authenticates the identity check and the token requests however the credential says, so the projected ServiceAccount token of an in-cluster controller, a token in a shard's kubeconfig, a client certificate or an exec plugin all serve, and nothing configures the ServiceAccount. The ServiceAccount must be allowed to create its own token subresource: the chart grants that to the in-cluster controller, and a shard's operator grants it to the ServiceAccount of the control plane they issued the shard a credential for, bound to the kargo-controller ClusterRole. Each token lives for API_SERVER_TOKEN_LIFETIME (an hour by default) and is replaced once most of that has passed.

The chart points a controller installed beside the API server at the API server's Service and, when it generated the API server's certificate, hands the controller its CA (API_SERVER_CA_CERT_PATH). A controller installed elsewhere, such as a shard in another cluster, is given the address at which it reaches the API server through controller.apiServer.address. Each request is bounded by API_SERVER_TIMEOUT (30 seconds by default), so an API server that accepts a connection and then hangs cannot hold a reconcile open indefinitely.

Every component that uses the database finds it through the same settings (DATABASE_URL, or DATABASE_HOST and its companions), which Tilt sets from the chart. A component without them runs with the database features off: the API server answers Target requests with 501 Not Implemented. A component with them waits, for up to DATABASE_CONNECT_TIMEOUT, for the database to accept connections and for the schema to reach the version it expects, since the migrations usually run alongside it; it refuses to serve against an older schema. A controller without API_SERVER_ADDRESS (the chart leaves it unset when the API server is disabled) reports a Stage that selects Targets as Stalled instead of promoting to it, and a Promotion that names a Target fails.

Targets are created, replaced and deleted through POST, PUT and DELETE on the same path, with a Target resource as the body; the generic /v1beta1/resources endpoints refuse kind: Target, so kargo apply of a Target manifest fails with a pointer to those endpoints. A PUT that carries the metadata.resourceVersion or metadata.uid it read is refused with 409 Conflict if the Target changed in between. There is no watch on Targets: the database offers no change feed, so the UI polls the list every few seconds instead.

To open a psql session against it:

make db-shell

This runs psql inside the PostgreSQL pod, so nothing needs to be installed locally and the client always matches the server.

Migrations​

Migrations are embedded in the control plane binary and applied by its migrate subcommand, which waits for the database to accept a connection, takes a lock on the schema so concurrent runners serialize, and applies whatever is pending. The chart's migration Job runs it on every install and upgrade, and Tilt runs that same Job as the db-migrate resource, so the Job and its connection wiring are exercised in development too. Like the other workloads, it deploys once at startup. After that, a change under db/migrations/ recompiles the binary and rebuilds the image, and Tilt marks db-migrate as having pending changes, but applies them only when you trigger it, from the Tilt UI or with:

hack/bin/tilt trigger db-migrate

This keeps a migration you are still editing from being applied early. A failed migration is retried up to the Job's backoffLimit, and each attempt's logs appear in Tilt.

To run the same subcommand from your working tree, outside Tilt:

make db-migrate

To run the database integration tests, which create and drop their own databases on the development server:

make test-db

Both target DATABASE_URL, which defaults to the development database.

The subcommand only applies migrations. For inspecting, rolling back, or resetting, use the Goose tool directly after configuring your shell:

export GOOSE_DRIVER=postgres
export GOOSE_DBSTRING='postgres://kargo:kargo@127.0.0.1:15432/kargo?sslmode=disable'
export GOOSE_MIGRATION_DIR=db/migrations

go tool goose status

Migrations must be forward-only and compatible with the previous release's code: during a rollout, and after a rollback, pods built from the previous release run against the migrated schema. Add a column in one release and remove its predecessor in the next rather than renaming in place. Keep a Down section for local development, but do not rely on it in production.

Creating a migration​

Create the migration in place:

go tool goose -dir db/migrations create create_widgets sql

Replace the generated SQL with:

-- +goose Up
CREATE TABLE widgets (
    id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    name text NOT NULL
);

-- +goose Down
DROP TABLE widgets;

Trigger db-migrate, then inspect the result:

hack/bin/tilt trigger db-migrate
make db-shell
\d widgets
note

Goose tracks migrations by version and never re-applies one it has already run. If an applied migration turns out to be wrong, roll it back with go tool goose down before editing it. Once a migration has merged, add a new one instead of changing it.

Rollback and reset are explicit operations. With the environment variables above set, go tool goose down rolls back the latest migration, and go tool goose reset rolls back all migrations. Both may delete data, depending on the migrations' down statements. Run make db-migrate to reapply them.

PostgreSQL uses a persistent volume from the cluster's default StorageClass. Data survives pod replacement, stopping/restarting Tilt, and make hack-tilt-down: Tilt retains namespaces by default, and the StatefulSet retains its volume claim when deleted. Deleting the kargo namespace or the data-kargo-postgres-0 volume claim removes this persistence; the StorageClass's reclaim policy determines whether the backing volume is also deleted. Treat this database as disposable development data.

Tracing database operations​

Connection pools created with database.NewPool record an OpenTelemetry span for every query and batch, using the same tracing setup as the rest of the control plane. Spans are named after the sqlc query that produced them, for example UpsertProject, and carry the operation, the number of rows returned, and the server's SQLSTATE code when a query fails. For sqlc queries, which are always parameterized, the query text is recorded as well; parameters never are. Lock timeouts, deadlocks, and serialization failures each have their own code, so they can be told apart from other errors.

A span cannot distinguish a slow query from one that waited on a lock held by another session. The development values set log_lock_waits so the PostgreSQL instance logs every lock wait longer than one second, naming both the waiting and the holding session, and the pod's logs answer that question:

kubectl logs -n kargo kargo-postgres-0 | grep 'still waiting'

Every component's sessions carry an application_name, so the log lines and pg_stat_activity show which component held the lock.

Generating query code with sqlc​

Kargo uses sqlc to generate type-safe Go code from SQL. sqlc.yaml reads the schema from the Goose migrations in db/migrations/ and the queries in db/queries/, and writes pgx/v5 code to pkg/database/. While Tilt is running, the codegen-db resource regenerates this code whenever either directory changes, and the back end recompiles with the result. To regenerate manually:

make codegen-db

make codegen also runs this step. Commit the generated code together with the SQL that produced it.

Database integration tests​

Tests in pkg/database that need a real database are tagged integration and run against a disposable schema that they create and drop, so existing tables are left untouched. They require a database user allowed to create schemas and target DATABASE_URL, which defaults to the development database:

make test-db

Contributing to Documentation​

Contributors should ensure that their changes are accompanied by relevant documentation updates. This helps maintain the project's sustainability. Pull requests with corresponding documentation updates are more likely to be merged faster.

To make this process smoother, you can refer to Docusaurus for guidance on writing and maintaining docs effectively.

Previewing Doc Changes Locally​

After making your changes, preview the documentation locally to ensure everything renders correctly. You can either run it in a container or natively on your system.

To build and serve the docs inside a container:

make hack-serve-docs
info

If you wish to opt-out of executing code-generation within a container (for performance reasons, perhaps), drop the hack- prefix from the target to run the docs natively on your system:

make serve-docs

This will require quite a variety of tools to be installed locally, so we do not recommend this if you can avoid it.