Skip to main content

Managing Secrets

Kargo uses Kubernetes Secret resources to store repository credentials and other types of sensitive data, such as API keys for third-party services. The namespaces within which they are contained impact who or what can access them. Labels on those Secrets may also constrain how they are accessed.

It is crucial that operators managing Kargo instances understand how Secrets are organized and accessed.

Not what you were looking for?

If you're a Kargo user looking to learn more about managing credentials or any other kind of secret within your own Project, refer instead to the Managing Secrets section of the User's Guide.

Overview​

Operators managing a Kargo instance will find themselves concerned with secrets falling into one of two broad categories:

  • Shared secrets intended for read-only access by any or all Projects within the instance. These, in turn, can be classified as one of:

    • Repository credentials: Secrets specifically representing credentials for the three types of repositories supported by Kargo: Git repositories, container image repositories, and Helm chart repositories.

    • "Generic credentials": Any secrets that are not specifically repository credentials.

      info

      The misnomer "generic credentials" is used for historical reasons, but nothing limits these to storing only credentials. In actuality, they can be any sort of sensitive information. So although they are called "generic credentials," they are best thought of in more general terms as, simply, generic secrets.

  • System-level secrets used by Kargo itself and not intended to be accessed by Kargo Projects.

The remainder of this document will cover each of these in turn, explaining in detail what such secrets look like, where they are stored, who can access them, and how they are accessed.

Shared Secrets​

As the name implies, shared secrets are those intended to be accessible by all Projects within a Kargo instance. Their corresponding Secret resources belong in one, specific Kubernetes namespace referred to as the shared resources namespace.

Why shared resources?

Prior to Kargo v1.9.0, what is now the shared resources namespace was referred to as "global credentials namespaces" (plural). Three factors prompted the Kargo team to refine and rename the concept:

  • "Global" was prone to various misinterpretations.
  • "Namespaces" (plural) added unnecessary technical complications to the system.
  • "Credentials" was too specific. Not all secrets are credentials. And not all things to be shared across Projects are secrets. The more general term "resources" speaks to a broader purpose for the namespace.
Migration

If you are migrating from a Kargo version lesser than v1.9.0 to version v1.9.0 or greater, please consult the migration section at the bottom of this page.

Repository Credentials​

Kargo expects Secret resources representing repository credentials to be labeled in specific ways and to conform to a specific format. Such Secrets generally take the following form:

apiVersion: v1
kind: Secret
metadata:
  name: <name>
  namespace: <project namespace or shared resources namespace>
  labels:
    kargo.akuity.io/cred-type: <type>
data: # base64 encoded
  repoURL: <repo url>
  username: <username>
  password: <password>

The label key kargo.akuity.io/cred-type, together with its value, specify the type of the repository accessed with the credential:

  • git: Credentials for Git repositories
  • helm: Credentials for Helm chart repositories
  • image: Credentials for container image repositories

Secrets representing repository credentials MUST include the key repoURL in their data block. Its value may be either a full, exact URL OR a regular expression matching the URLs of multiple repositories for which the credentials are valid, in which case, the data block must also contain the key/value pair repoURLIsRegex: "true".

The remaining key/value pairs in such a Secret's data block are dependent upon exactly what kind of credential the Secret represents. Commonly, they may be:

  • username: The username to use when authenticating to the repository

  • password: A password or personal access token

    info

    If the value of the password key is a personal access token, the value of the username field is often inconsequential. You should consult your repository's documentation for more information.

Alternatively, for Git repositories only (and specifically ones that support SSH-style URLs of the form git@github.com:example/repo.git), the key sshPrivateKey in the Secret's data block may have as its value a PEM-encoded SSH private key.

Deprecated

Support for SSH URLs and SSH private keys is deprecated as of v1.10.0 and scheduled for removal in v1.13.0.

The proprietary APIs offered by the major Git hosting platforms (e.g. GitHub or GitLab) to enable actions such as opening or closing pull requests are invariably HTTP-based and therefore cannot use an SSH private key for authentication. This forces users who rely on SSH to maintain a second set of credentials (e.g. a personal access token) for API operations.

Using HTTPS URLs with a single token-based credential that works for both standard Git operations and provider API calls is simpler and more secure. Users should migrate to HTTPS URLs before v1.13.0.

Credential Shapes

Secret resources representing repository credentials come in a wide variety of other "shapes" (different keys in the data block) corresponding to various authentication mechanisms. These are covered in the Managing Secrets section of the User's Guide.

Using Repository Credentials​

A unique property of Secret resources representing repository credentials is that Projects do not (and cannot) reference them directly. Any time Kargo accesses a repository, it automatically attempts to locate suitable credentials, searching by repository type and URL.

tip

Because of the above, operators managing a Kargo instance can place repository credentials in the shared resources namespace, knowing that they can be used by all Projects without their values ever being exposed to users.

When Kargo needs repository credentials, it searches for Secrets in two specific namespaces, in the following order:

  1. Project namespace: Kargo searches the Project's own namespace first.

  2. Shared resources namespace: If no match is found in the Project's own namespace, Kargo searches the shared resources namespace.

Credential Matching Precedence

Within each namespace searched, Kargo considers credentials in this order:

  1. Exact repoURL matches (where repoURLIsRegex is "false" or unspecified)
  2. Pattern matches using regex (where repoURLIsRegex is "true")

Within each category, Secrets are considered in lexical order by name.

The credentials used by Kargo will be the first to match the repository type and URL.

Generic Credentials​

"Generic credentials" (a misnomer) are any secrets that are not specifically repository credentials.

Secret resources representing generic credentials MUST be labeled with kargo.akuity.io/cred-type: generic.

info

The misnomer "generic credentials" is used for historical reasons, but nothing limits these to storing only credentials. In actuality, they can be any sort of sensitive information. So although they are called "generic credentials," they are best thought of in more general terms as, simply, generic secrets.

Using Generic Credentials​

In contrast to repository credentials, Secret resources representing shared generic credentials can be accessed directly by name and their data blocks are not required to conform to any specific structure. This makes them suitable for storing any arbitrary secret data that Projects may depend upon. Projects can access such secrets within expressions used by their promotion processes by utilizing the sharedSecret() expression function.

caution

Always remember that any generic credential in the shared resources namespace can be accessed directly by all Projects, which means it is possible to learn their values.

Exercise due caution when deciding what secrets are suitable to be shared in this manner.

Replicating Shared Resources to Project Namespaces​

By default, shared secrets are accessed indirectly -- repository credentials are matched automatically by URL, and generic credentials are accessed through the sharedSecret() expression function. In both cases, the Secret resources themselves remain in the shared resources namespace and their values are never copied into Project namespaces.

In some situations, however, workloads running in a Project namespace may need direct access to a Secret or ConfigMap -- for example, as a volume mount or environment variable reference. For these cases, Kargo can automatically replicate resources from the shared resources namespace into every Project namespace.

Enabling Replication​

To opt a resource into replication, annotate it with kargo.akuity.io/replicate-to: "*":

apiVersion: v1
kind: Secret
metadata:
  name: shared-tls-cert
  namespace: kargo-shared-resources
  labels:
    kargo.akuity.io/cred-type: generic
  annotations:
    kargo.akuity.io/replicate-to: "*"
data:
  tls.crt: <base64-encoded cert>
  tls.key: <base64-encoded key>

ConfigMap resources work the same way:

apiVersion: v1
kind: ConfigMap
metadata:
  name: shared-config
  namespace: kargo-shared-resources
  annotations:
    kargo.akuity.io/replicate-to: "*"
data:
  settings.yaml: |
    key: value
info

Currently, the only supported value for the kargo.akuity.io/replicate-to annotation is "*", which replicates to all Project namespaces. Selective replication to specific Projects is not yet supported.

info

Only Secret resources labeled with kargo.akuity.io/cred-type (i.e. recognized as credentials) are eligible for replication. Arbitrary Secret resources without this label will not be replicated, even if annotated with kargo.akuity.io/replicate-to. All ConfigMap resources in the shared resources namespace are eligible.

How Replication Works​

When a resource in the shared resources namespace is annotated for replication, Kargo's management controller automatically creates a copy of it in every Project namespace. These copies are kept in sync: any change to the source resource is propagated to all replicas. If the source is deleted, all replicas are cleaned up.

Replicated resources are labeled with kargo.akuity.io/replicated-from (set to the name of the source resource) so they can be easily identified.

Immutability of Replicated Resources​

Replicated copies are immutable to end users. A validating webhook prevents any user from modifying or deleting them. Only Kargo's management controller is permitted to update or remove replicated resources.

To change the contents of a replicated resource, modify the source resource in the shared resources namespace. The change will propagate automatically to all replicas.

Configuring the Shared Resources Namespace​

The shared resources namespace, by default, is kargo-shared-resources. Operators may override this at the time of installation or upgrade by overriding the Kargo Helm chart's global.sharedResources.namespace setting.

System Secrets​

Various components of Kargo itself, at times, have the need to reference operator-defined secrets. The canonical example for this involves configuring cluster-scoped webhook receivers.

Cluster-scoped webhook receivers are defined as part of a ClusterConfig resource, which is, unsurprisingly, a cluster-scoped resource. (i.e. It does not belong to any namespace.) When such a configuration must reference a Secret, because Kubernetes has no cluster-scoped "ClusterSecret" resource type, the question is raised of exactly which namespace a Secret that is conceptually cluster-scoped should belong to.

The existence of the system resources namespace provides an answer to this conundrum.

An example ClusterConfig:

apiVersion: v1
kind: Secret
metadata:
  name: gh-wh-secret
  namespace: kargo-system-resources
  labels:
    kargo.akuity.io/cred-type: generic
data:
  secret: <base64-encoded secret>
---
apiVersion: kargo.akuity.io/v1alpha1
kind: ClusterConfig
metadata:
  # Note this resource is not namespaced
  name: cluster
spec:
  webhookReceivers: 
    - name: gh-wh-receiver
      github:
        # Referenced Secrets are implicitly known to be in
        # the system resources namespace (kargo-system-resources)
        secretRef:
          name: gh-wh-secret
info

Prior to Kargo v1.9.0, what is now the system resources namespace was referred to as the "cluster secrets namespace." Two factors prompted the Kargo team to refine and rename the concept:

  • "Cluster" was prone to various misinterpretations.
  • "Secrets" was too specific. The Kargo team does not anticipate that Secret resources will forever be the only type of namespaced resource that will be referenced by ClusterConfig as a workaround for a non-existent cluster-scoped analog.
Migration

If you are migrating from a Kargo version lesser than v1.9.0 to version v1.9.0 or greater, please consult the migration section at the bottom of this page.

Configuring the System Resources Namespace​

The system resources namespace, by default, is kargo-system-resources. Operators may override this at the time of installation or upgrade by overriding the Kargo Helm chart's global.systemResources.namespace setting.

Migrating from Kargo < 1.9.0​

Kargo v1.9.0 introduced terminology and configuration changes to better reflect the intended use of what are now the shared resources namespace and system resources namespace. These changes are summarized here.

Terminology Changes:

  • Global credentials namespaces (plural) → shared resources namespace (singular)
  • Cluster secrets namespace → system resources namespace

Chart Setting Changes:

  • controller.globalCredentials.namespaces → global.sharedResources.namespace

    • The old setting had no default value(s).

    • The new setting has a default value of kargo-shared-resources.

    • The move from the controller section of the chart's settings to the global section reflects that this configuration is used by more than one Kargo component.

  • global.clusterSecretsNamespace → global.systemResources.namespace

    • The old setting had a default value of kargo-cluster-secrets.

    • The new setting has a default value of kargo-system-resources.

Automatic Migration:

Kargo versions v1.9.0 through v1.11.x automatically and continuously performed a one-way sync of Secret resources from their old locations to their new locations, with a few exceptions:

  • If the old controller.globalCredentials.namespaces setting was empty (as it had no default value(s)), there were no Secret resources in need of migration to the namespace specified by the new global.sharedResources.namespace.

  • Due to the potential for name conflicts if Kargo were to attempt consolidating resources from multiple namespaces into a single namespace, a chart upgrade to v1.9.0 through v1.11.0 failed if the old controller.globalCredentials.namespaces setting specified multiple namespaces. In this case (believed to be an outlier), the operator needed to migrate affected resources manually.

  • If the value of the new global.sharedResources.namespace matched the value of the old controller.globalCredentials.namespaces[0] setting, no migration of shared Secret resources was necessary.

  • If the value of the new global.systemResources.namespace matched the value of the old global.clusterSecretsNamespace setting, no migration of system Secret resources was necessary.

As of Kargo v1.12.0, the automatic migration has been removed. Upgrading to v1.12.0 or later will fail if values are detected for either of the old settings.

Sync Behavior:

The automatic sync from old to new locations (in Kargo v1.9.0 through v1.11.x) worked as follows:

  • Unmodified secrets: If a Secret in the new location has not been modified since it was synced, updates from the old location will continue to propagate. Deleting from the old location will also delete from the new location.

  • Modified secrets: If you modify a Secret in the new location (via the Kargo UI or otherwise), those changes are protected. Updates from the old location will not overwrite your modifications. Similarly, deleting from the old location will not delete a modified secret from the new location.

  • Deleting from the new location: If you delete a Secret from the new location while it still exists in the old location, it will be recreated on the next sync. To permanently remove a secret, delete it from the old location.

  • Namespace cleanup: When the old namespace itself is deleted (bulk cleanup), all secrets in the new location are preserved, regardless of whether they were modified.

What this means, practically speaking:

  • New installations of Kargo need not be concerned with any of this.

  • If you are still running Kargo v1.9.0 through v1.11.x with either controller.globalCredentials.namespaces or global.clusterSecretsNamespace defined, your Secrets have already been automatically synced to their new locations (global.sharedResources.namespace and global.systemResources.namespace, respectively). Before upgrading to v1.12.0 or later, remove both settings from your values -- the chart will refuse to upgrade otherwise -- and, if you GitOps your Secrets, update your manifests to reference the new namespaces. You may then safely delete the old namespaces using kubectl.