Managing Credentials
To manage the progression of freight from stage to stage, Kargo will often require read/write permissions on private GitOps repositories and read-only permissions on private container image and/or Helm chart repositories.
This section presents an overview of how these credentials can be managed.
Credentials as Kubernetes Secret
Resources
Kargo formerly borrowed its general credential-management approach from Argo CD, but has since diverged.
Kargo expects any credentials it requires to have been stored as specially
labeled Kubernetes Secret
resources containing specially-formatted data. These
Secret
s take the following form:
apiVersion: v1
kind: Secret
metadata:
name: <name>
namespace: <project namespace>
labels:
kargo.akuity.io/cred-type: image
stringData:
repoURL: <repo url>
username: <username>
password: <password>
The name
of such a Secret
is inconsequential and may follow any convention
preferred by the user.
Kargo uses Kubernetes Namespace
s to mark project boundaries. Secret
s
representing credentials will typically exist in the same Namespace
as the
Stage
resources that will require them. There are exceptions to this, which
are covered in the next section.
The label key kargo.akuity.io/cred-type
and its value, one of git
, helm
,
or image
, is important, as it designates the Secret
as representing
credentials for a Git repository, a Helm chart repository, or a container image
repository, respectively.
The Secret
's data
field (set above using plaintext in the stringData
field), MUST contain the following keys:
-
repoURL
: The full URL of the repository the credentials are for. -
Either:
-
username
: The username to use when authenticating to the repository. -
password
: A password or personal access token.infoIf the value of the
password
key is a personal access token, the value of theusername
field may be inconsequential. You should consult your repository's documentation for more information.
OR:
sshPrivateKey
: A PEM-encoded SSH private key. Applicable to Git repositories only.
-
Optionally, the following keys may also be included:
repoURLIsRegex
: Set this totrue
if the value of therepoURL
key is a regular expression. Any other value of this key or the absence of this key is interpreted asfalse
.
When Kargo searches for repository credentials in a project Namespace
, it
first checks all appropriately labeled Secret
s for a repoURL
value
matching the repository URL exactly. Only if no Secret
is an exact match does
it check all appropriately labeled Secret
s for a repoURL
value containing a
regular expression matching the repository URL.
When searching for an exact match, and again when searching for a pattern match,
appropriately labeled Secret
s are considered in lexical order by name.
Only username/password (or personal access token) authentication is supported at this time. Others are likely to be added in the future.
Global Credentials
In cases where one or more sets of credentials are needed widely across all
Kargo projects, the administrator/operator installing Kargo may opt-in to
designating one or more Namespace
s as homes for "global" credentials using the
controller.globalCredentials.namespaces
setting in Kargo's Helm chart.
Refer to
the advanced section of the installation guide
for more details.
Any matching credentials (exact match or pattern match) found in a project's
own Namespace
take precedence over those found in any global credentials
Namespace
.
When Kargo searches for repository credentials in global credentials
Namespace
s, it first checks all appropriately labeled Secret
s for a
repoURL
value matching the repository URL exactly. Only if no Secret
is an
exact match does it check all appropriately labeled Secret
s for a
repoURL
value containing a regular expression matching the repository URL.
When searching for an exact match, and again when searching for a pattern match,
appropriately labeled Secret
s are considered in lexical order by name.
When Kargo is configured with multiple global credentials Namespace
s, they are
searched in lexical order by name. Only after no exact match and no pattern
match is found in one global credentials Namespace
does Kargo search the next.
It is important to understand the security implications of this feature. Any
credentials stored in a global credentials Namespace
will be available to
all Kargo projects.
Managing Credentials with the CLI
The Kargo CLI can be used to manage credentials in a project's Namespace.
The following example creates credentials for a Git repository:
kargo create credentials --project kargo-demo my-credentials \
--git --repo-url https://github.com/example/kargo-demo.git \
--username my-username --password my-my-personal-access-token
secret/my-credentials created
If you do not wish for your password or personal access token to be stored
in your shell history, you may wish to omit the --password
flag, in which
case the CLI will prompt you to enter the password interactively.
Credentials can be listed or viewed with kargo get credentials
:
kargo get credentials --project kargo-demo my-credentials
NAME TYPE REGEX REPO AGE
my-credentials git false https://github.com/example/kargo-demo.git 8m25s
If requesting output as YAML or JSON, passwords and other potentially sensitive information will be redacted.
kargo get credentials --project kargo-demo my-credentials -o yaml
apiVersion: v1
kind: Secret
metadata:
creationTimestamp: "2024-05-30T20:02:46Z"
labels:
kargo.akuity.io/cred-type: git
name: my-credentials
namespace: kargo-demo
resourceVersion: "17614"
uid: ca2660e4-867d-4709-b1a7-57fbb93fc6dc
stringData:
password: '*** REDACTED ***'
repoURL: https://github.com/example/kargo-demo.git
username: my-username
type: Opaque
Credentials can be updated using the kargo update credentials
command and
the flags corresponding to attributes of the credential that you wish to modify.
Other attributes of the credentials will remain unchanged.
The following example updates my-credentials
with a regular expression for the
repository URL:
kargo update credentials --project kargo-demo my-credentials \
--repo-url '^http://github.com/' --regex
secret/my-credentials updated
And credentials can, of course, be deleted with kargo delete credentials
:
kargo delete credentials --project kargo-demo my-credentials
secret/my-credentials deleted
While the CLI may be a fine way of managing project-level credentials whilst
getting to know Kargo, it is unquestionably more secure to use other means to
ensure the existence of these specially-formatted Secret
s in the appropriate
project Namespace
s.
Git Provider-Specific Authentication Options
This section provides Git provider-specific guidance on credential management.
GitHub
Personal Access Token
GitHub supports authentication using a
personal access token,
which can be used in place of a password. The corresponding username must be
the GitHub handle of the user who created the token. These can be stored in
the username
and password
fields of a Secret
resource as described
in the first section of this
document.
This method of authentication may be best when wishing to rigorously enforce the principle of least privilege, as personal access tokens can be scoped to specific permissions on specific repositories.
A drawback to this method, however, is that the token is owned by a specific GitHub user, and if that user should lose their own access to the repositories in question, Kargo will also lose access.
GitHub App Authentication
GitHub Apps can be used as an authentication method.
-
- In the GitHub App name field, specify a unique name.
- Set the Homepage URL to any URL you like.
- Under Webhook, de-select Active.
- Under Permissions → Repository permissions → Contents, select whether the App will require Read-only or Read and write permissions. The App will receive these permissions on all repositories into which it is installed.
- Under Where can this GitHub App be installed?, leave Only on this account selected.
- Click Create GitHub App.
- Take note of the App ID.
- Scroll to the bottom of the page and click Generate a private key. The resulting key will be downloaded immediately. Store it securely.
- On the left-hand side of the page, click Install App.
- Choose an account to install the App into by clicking Install.
- Select Only select repositories and choose the repositories you wish to grant the App access to. Remember that the App will receive the permissions you selected earlier on all of these repositories.
- Click Install.
- In your browser's address bar, take note of the numeric identifier at the end of the current page's URL. This is the Installation ID.
-
Create a
Secret
resource with the following structure:apiVersion: v1
kind: Secret
metadata:
name: <name>
namespace: <project namespace>
labels:
kargo.akuity.io/cred-type: git
stringData:
githubAppID: <app id>
githubAppPrivateKey: <PEM-encoded private key>
githubAppInstallationID: <installation id>
repoURL: <repo url>
repoURLIsRegex: <true if repoURL is a pattern matching multiple repositories>
Compared to personal access tokens, a benefit of authenticating with a GitHub App is that the App's permissions are not tied to a specific GitHub user.
It is all too easy to violate the principle of least privilege when authenticating using this method.
For convenience sake, it may be tempting to register a single GitHub App and select a broad set of repositories when installing that App into your organization. It may also be tempting to create a single set of global credentials such that all Kargo projects can use them to access their repositories, however, this will have the undesirable effect of granting all Kargo projects access to all of the repositories selected when the App was installed.
It is, instead, recommended to register a separate GitHub App for each Kargo project. When installing each App into your organization, only those repositories to which each Kargo project requires access should be selected.
GitHub organizations are limited to registering 100 GitHub Apps, however, so this approach may not be feasible for organizations with many Kargo projects.
A second way in which authentication using GitHub Apps may violate the principle of least privilege involves the fact that the same permissions are granted to the App on all repositories that are selected when it is installed.
If a Kargo project requires read-only access to one repository and read/write access to another, it is not possible to grant the App different permissions on the two. This may then lead to granting broader permissions than are strictly necessary.
Image Registry-Specific Authentication Options
While many container image registries support authentication using long-lived credentials, such as a username and password (or personal access token), some either require or offer more secure options.
This section provides registry-specific guidance on credential management and also covers options for gaining image repository access using workload identity on applicable platforms.
Amazon Elastic Container Registry (ECR)
The authentication options described in this section are applicable only to container image repositories whose URLs indicate they are hosted in ECR.
Long-Lived Credentials
Elastic Container Registries do not directly support long-lived credentials, however, an AWS access key ID and secret access key can be used to obtain an authorization token that is valid for 12 hours. Kargo can seamlessly obtain such a token and will cache it for a period of 10 hours.
To use this option, your Secret
should take the following form:
apiVersion: v1
kind: Secret
metadata:
name: <name>
namespace: <project namespace>
labels:
kargo.akuity.io/cred-type: image
stringData:
awsAccessKeyID: <access key id>
awsSecretAccessKey: <secret access key>
repoURL: <ecr url>
Following the principle of least privilege, the IAM user associated with the access key ID and secret access key should be limited only to read-only access to the required ECR repositories.
This method of authentication is a "lowest common denominator" approach that will work regardless of where Kargo is deployed. i.e. if running Kargo outside EKS, this method will still work.
If running Kargo within EKS, you may wish to either consider using EKS Pod Identity or IRSA instead.
EKS Pod Identity or IAM Roles for Service Accounts (IRSA)
If Kargo locates no Secret
resources matching a repository URL and is deployed
within an EKS cluster, it will attempt to use
EKS Pod Identity
or
IAM Roles for Service Accounts (IRSA)
to authenticate. Both of these rely upon some external setup. Leveraging either
eliminates the need to store ECR credentials in a Secret
resource.
Follow
this overview
to set up EKS Pod Identity in your EKS cluster or
this one
to set up IRSA. For either, you will assign an IAM role to the
kargo-controller
ServiceAccount
within the Namespace
to which Kargo is (or
will be) installed.
To use IRSA, you will additionally need to specify the
ARN of
the controller's IAM role as the value of the
controller.serviceAccount.iamRole
setting in Kargo's Helm chart. Refer to
the advanced section of the installation guide
for more details.
At this point, an IAM role will be associated with the Kargo controller,
however, that controller acts on behalf of multiple Kargo projects, each of
which may require access to different ECR repositories. To account for this,
when Kargo attempts to access an ECR repository on behalf of a specific project,
it will first attempt to
assume an IAM role
specific to that project. The name of the role it attempts to assume will
always be of the form kargo-project-<project name>
. It is this role that
should be granted read-only access to applicable ECR repositories.
The name of the IAM role associated with each Kargo project is deliberately not configurable to prevent project admins from attempting to coerce Kargo into assuming arbitrary IAM roles.
For optimal adherence to the principle of least permissions, the IAM role
associated with the kargo-controller
ServiceAccount
should be limited only
to the ability to assume project-specific IAM roles. Project-specific IAM roles
should be limited only to read-only access to applicable ECR repositories.
If the Kargo controller is unable to assume a project-specific IAM role, it will fall back to using its own IAM role directly. For organizations without strict tenancy requirements, this can eliminate the need to manage a large number of project-specific IAM roles. While useful, this approach is not strictly recommended.
Once Kargo is able to gain necessary permissions to access an ECR repository, it will follow a process similar to that described in the previous section to obtain a token that is valid for 12 hours and cached for 10.
Google Artifact Registry
The authentication options described in this section are applicable only to container image repositories whose URLs indicate they are hosted in Google Artifact Registry.
Google Container Registry (GCR) has been deprecated in favor of Google Artifact Registry. For authentication to repositories with legacy GCR URLs, the same options outlined here may be applied.
Long-Lived Credentials
Google Artifact Registry does directly support long-lived credentials
as described here.
The username _json_key_base64
and the base64-encoded service account key
may be stored in the username
and password
fields of a Secret
resource as
described in the first section of
this document. Kargo and Google both strongly discourage this method of
authentication however.
Google documentation recommends using a service account key to obtain an access token that is valid for 60 minutes. Compared to the discouraged method of using the service account key to authenticate to the registry directly, this process does not transmit the service account key over the wire. Kargo can seamlessly carry out this process and will cache the access token for a period of 40 minutes.
To use this option, your Secret
should take the following form:
apiVersion: v1
kind: Secret
metadata:
name: <name>
namespace: <project namespace>
labels:
kargo.akuity.io/cred-type: image
stringData:
gcpServiceAccountKey: <base64-encoded service account key>
repoURL: <artifact registry url>
Service account keys contain structured data, so it is important that the key be base64-encoded.
Following the principle of least privilege, the service account associated with the service account key should be limited only to read-only access to the required Google Artifact Registry repositories.
This method of authentication is a "lowest common denominator" approach that will work regardless of where Kargo is deployed. i.e. If running Kargo outside of GKE, this method will still work.
If running Kargo within GKE, you may wish to consider using Workload Identity Federation instead.
Workload Identity Federation
If Kargo locates no Secret
resources matching a repository URL, and if Kargo
is deployed within a GKE cluster, it will attempt to use
Workload Identity Federation
to authenticate, but this relies upon some external setup. Leveraging this
option eliminates the need to store credentials in a Secret
resource.
First, follow these directions to provision a new GKE cluster with Workload Identity Federation enabled or these directions to enable Workload Identity Federation on an existing GKE cluster.
At this point, the kargo-controller
ServiceAccount
within the Namespace
to
which Kargo is (or will be) installed will be associated with an IAM principal
identifier, which takes the following form:
principal://iam.googleapis.com/projects/<gcp project number>/locations/global/workloadIdentityPools/<gcp project name>.svc.id.goog/subject/ns/<kargo namespace>/sa/kargo-controller
Although associated with this one principal, the Kargo controller acts on
behalf of multiple Kargo projects, each of which may require access to
different Google Artifact Registry repositories. To account for this, when
Kargo attempts to access a Google Artifact Registry repository on behalf of a
specific project, it will first attempt to
impersonate a Google service account
specific to that project. The name of the service account it attempts to
impersonate will always be of the form
kargo-project-<kargo project name>@<gcp project name>.iam.gserviceaccount.com
.
It is this service account that should be granted read-only access to applicable
Google Artifact Registry repositories.
The name of the Google service account associated with each Kargo project is deliberately not configurable to prevent Kargo project admins from attempting to coerce Kargo into impersonating arbitrary Google service accounts.
Once Kargo is able to impersonate the appropriate Google service account for a given project, it will follow a process similar to that described in the previous section to obtain a token that is valid for 60 minutes and cached for 40.
Following the principle of least privilege, the IAM principal associated with
the kargo-controller
ServiceAccount
should be limited only to the ability to
impersonate project-specific Google service accounts. Project-specific Google
service accounts should be limited only to read-only access to the applicable
Google Artifact Registry repositories.
Azure Container Registry (ACR)
Azure Container Registry directly supports long-lived credentials.
It is possible to
create tokens with repository-scoped permissions,
with or without an expiration date. These tokens can be stored in the
username
and password
fields of a Secret
resource as described
in the first section of this
document.
Support for authentication to ACR repositories using workload identity, on par with Kargo's support for ECR and Google Artifact Registry, is likely to be included in a future release of Kargo.