Skip to main content

Managing Credentials

To orchestrate the promotion of Freight from Stage to Stage, Kargo will often require read/write permissions on private Git repositories and read-only permissions on private container image or Helm chart repositories.

This section focuses on an operator's role in providing Kargo projects with necessary credentials.

info

Not what you were looking for?

If you're user looking to learn more about managing credentials at the project level, refer instead to the Managing Credentials section of the User's Guide.

info

Whether you're installing Kargo using Helm or via Argo CD, the next two sections assume familiarity with procedures for configuring that installation.

Repository Credentials as Secret Resources​

Kargo expects repository credentials it uses to have been stored as specially labeled Kubernetes Secret resources containing specially-formatted data. These Secrets generally take the following form:

apiVersion: v1
kind: Secret
metadata:
  name: <name>
  namespace: <project namespace>
  labels:
    kargo.akuity.io/cred-type: <type>
stringData:
  repoURL: <repo url>
  username: <username>
  password: <password>

The names of Secret resources are inconsequential because Kargo matches credentials to repositories by repository type and URL. Secret names may therefore observe any naming convention preferred by the user.

The label key kargo.akuity.io/cred-type and its value, one of git, helm, image, or generic is important, as it designates the Secret as representing credentials for a Git, Helm chart, or container image repository, or something else, respectively.

Secrets labeled as git, image, or helm credentials must generally contain the following keys:

  • repoURL:

    • The full URL of the repository the credentials are for.

    OR

    • A regular expression matching the URLs of multiple repositories for which the credentials may be used, with the repoURLIsRegex key additionally set to true.

      info

      This is useful if, for example, your project accesses many GitHub repositories, all beginning with https://github.com/example-org, and can use the same token for accessing all of them.

  • Either:

    • username: The username to use when authenticating to the repository.

    • password: A password or personal access token.

      info

      If the value of the password key is a personal access token, the value of the username field is often inconsequential. You should consult your repository's documentation for more information.

    OR:

    • sshPrivateKey: A PEM-encoded SSH private key. Applicable only to Git repositories using SSH-style URLs -- for instance git@github.com:example/repo.git.

Global Credentials​

Credentials are generally managed at the project level by project admins, but in cases where one or more sets of credentials are needed widely across many or all Kargo projects, an operator may opt into designating one or more namespaces as containing "global" credentials, accessible to all projects. It is then the operator's responsibility to create and manage such credentials as well.

When Kargo searches for repository credentials, these additional namespaces are searched only after finding no matching credentials in the project's own namespace.

note

Precedence

When Kargo searches for repository credentials in a "global" namespace, it first iterates over all appropriately labeled Secrets without repoIsRegex set to true looking for a repoURL value matching the repository URL exactly.

Only if no exact match is found does it iterate over all appropriately labeled Secrets with repoIsRegex set to true looking for a regular expression matching the repository URL.

When searching for an exact match, and then again when searching for a pattern match, appropriately labeled Secrets are considered in lexical order by name.

info

Because Kargo matches credentials to repositories by repository type and URL, users do not need to be informed of the details (e.g. names) of any global credentials, except possibly that they exist.

Enabling Global Credentials​

To designate one or more namespaces as containing "global" credentials, list them under the Kargo Helm chart's controller.globalCredentials.namespaces option at installation time.

Operators must also manually ensure Kargo controllers receive read-only access to Secrets in the designated namespaces. For example, if kargo-global-creds is designated as a global credentials namespace, the following RoleBinding should be created within that namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: kargo-controller-read-secrets
    namespace: kargo-global-creds
subjects:
    - kind: ServiceAccount
      name: kargo-controller
      namespace: kargo
roleRef:
    kind: ClusterRole
    name: kargo-controller-read-secrets
    apiGroup: rbac.authorization.k8s.io
note

The kargo-controller-read-secrets ClusterRole is predefined by the Kargo Helm chart and grants read-only access to Secret resources.

info

By default, Kargo controllers lack cluster-wide permissions on Secret resources. Instead, the Kargo management controller dynamically expands and contracts controller access to Secrets on a namespace-by-namespace basis as new Projects are created and deleted.

It is because this process does not account for "global" credential namespaces that these bindings must be created manually by an operator.

warning

Setting the controller.serviceAccount.clusterWideSecretReadingEnabled option to true at installation will grant Kargo controllers cluster-wide read permission on Secret resources.

This is highly discouraged, especially in sharded environments where this permission would have the undesirable effect of granting remote Kargo controllers read permissions on all Secrets throughout the Kargo control plane's cluster -- including Secrets having nothing to do with Kargo.

Ambient Credentials​

This section provides guidance on configuring Kargo and various cloud platforms to support "ambient" credentials. Kargo users are presumed not to have sufficient access to those platform to configure these options themselves, so this section is intended for operators and cloud platform administrators.

Amazon Elastic Container Registry (ECR)​

Kargo can be configured to authenticate to ECR repositories using EKS Pod Identity or IAM Roles for Service Accounts (IRSA).

If Kargo locates no Secret resources matching a repository URL and is deployed within an EKS cluster, it will attempt to use EKS Pod Identity or IAM Roles for Service Accounts (IRSA) to authenticate. Leveraging either eliminates the need to store ECR credentials in a Secret resource.

Follow this overview to set up EKS Pod Identity in your EKS cluster or this one to set up IRSA. For either, you will assign an IAM role to the kargo-controller ServiceAccount within the namespace in which Kargo is (or will be) installed.

note

To use IRSA, you will additionally need to specify the ARN of the controller's IAM role as the value of the controller.serviceAccount.iamRole setting in Kargo's Helm chart at installation.

At this point, an IAM role will be associated with the Kargo controller, however, that controller acts on behalf of multiple Kargo projects, each of which may require access to different ECR repositories. To account for this, when Kargo attempts to access an ECR repository on behalf of a specific project, it will first attempt to assume an IAM role specific to that project. The name of the role it attempts to assume will always be of the form kargo-project-<project name>. It is this role that should be granted read-only access to applicable ECR repositories.

info

The name of the IAM role associated with each Kargo project is deliberately not configurable to prevent project admins from attempting to coerce Kargo into assuming arbitrary IAM roles.

caution

For optimal adherence to the principle of least permissions, the IAM role associated with the kargo-controller ServiceAccount should be limited only to the ability to assume project-specific IAM roles. Project-specific IAM roles should be limited only to read-only access to applicable ECR repositories.

info

If the Kargo controller is unable to assume a project-specific IAM role, it will fall back to using its own IAM role directly. For organizations without strict tenancy requirements, this can eliminate the need to manage a large number of project-specific IAM roles. While useful, this approach is not strictly recommended.

Tokens Kargo obtains for accessing any specific ECR repository on behalf of any specific Kargo project, are valid for 12 hours and cached for 10. A controller restart clears the cache.

Google Artifact Registry (GAR)​

Kargo can be configured to authenticate to Google Artifact Registry (GAR) repositories using Workload Identity Federation (WIF).

If Kargo locates no Secret resources matching a repository URL, and if Kargo is deployed within a Google Kubernetes Engine (GKE) cluster with WIF enabled, it will attempt to use it to authenticate. Leveraging this option eliminates the need to store credentials in a Secret resource. WIF can be enabled when creating a new cluster or can be added to an existing cluster.

note

Clusters managed by GKE Autopilot have WIF enabled automatically.

With WIF enabled, GCP Identity and Access Management (IAM) automatically understands a principal identifier of the following form to be a reference to the Kargo controller's Kubernetes Service Account (KSA):

principal://iam.googleapis.com/projects/<gcp project number>/locations/global/workloadIdentityPools/<gcp project name>.svc.id.goog/subject/ns/<kargo namespace>/sa/kargo-controller
note

There is no need to annotate the Kargo controller's KSA in any specific way to enable the above.

Because the Kargo controller acts on behalf of multiple Kargo projects, each of which may require access to different GAR repositories, when accessing a repository on behalf of a given project, it will attempt to impersonate a project-specific Google Service Account (GSA). The name of the GSA that the controller will attempt to impersonate will always be of the form kargo-project-<kargo project name>@<gcp project name>.iam.gserviceaccount.com.

info

The name of the GSA associated with each Kargo project is deliberately not configurable to prevent project admins from attempting to coerce Kargo into impersonating arbitrary GSAs.

To enabled this, each project-specific GSA must:

  • Have an IAM policy that permits the Kargo controller's KSA to impersonate the GSA by creating a token (roles/iam.serviceAccountTokenCreator).

  • Be granted read-only access (roles/artifactregistry.reader) to the specific GAR repositories with which it interacts.

caution

Following the principle of least privilege, the IAM principal associated with the Kargo controller's GSA should be granted no permissions beyond the ability to impersonate project-specific GSAs.

note

Beginning with Kargo v1.5.0, if maintaining a separate GSA for every Kargo project is deemed too onerous and strict adherence to the principle of least privilege is not a concern, permissions may be granted directly to the Kargo controller's KSA. In the event that a project-specific GSA does not exist or cannot be impersonated, Kargo will fall back on using the controller's KSA directly to access GAR repositories. While useful, this approach is not strictly recommended.

Tokens Kargo obtains for accessing any specific GAR repository on behalf of any specific Kargo project, are valid for 60 minutes and cached for 40. A controller restart clears the cache.

Azure Container Registry (ACR)​

Support for authentication to ACR repositories using workload identity is not yet implemented. Assuming/impersonating a project-specific principal in Azure is notably complex. So, while a future release is very likely to implement some form of support for ACR and workload identity, it is unlikely to match the capabilities Kargo provides for ECR or GAR.