Using external secrets in CI

DETAILS: Tier: Free, Premium, Ultimate Offering: GitLab.com, Self-managed, GitLab Dedicated

Secrets represent sensitive information your CI job needs to complete work. This sensitive information can be items like API tokens, database credentials, or private keys. Secrets are sourced from your secrets provider.

Unlike CI/CD variables, which are always presented to a job, secrets must be explicitly required by a job. Read GitLab CI/CD pipeline configuration reference for more information about the syntax.

GitLab provides support for the following secret management providers:

1. Vault by HashiCorp
2. Google Cloud Secret Manager
3. Azure Key Vault

GitLab has selected Vault by HashiCorp as the first supported provider, and KV-V2 as the first supported secrets engine.

Use ID tokens to authenticate with Vault. The Authenticating and Reading Secrets With HashiCorp Vault tutorial has more details about authenticating with ID tokens.

You must configure your Vault server before you can use Vault secrets in a CI job.

The flow for using GitLab with HashiCorp Vault is summarized by this diagram:

1. Configure your vault and secrets.
2. Generate your JWT and provide it to your CI job.
3. Runner contacts HashiCorp Vault and authenticates using the JWT.
4. HashiCorp Vault verifies the JWT.
5. HashiCorp Vault checks the bounded claims and attaches policies.
6. HashiCorp Vault returns the token.
7. Runner reads secrets from the HashiCorp Vault.

NOTE: Read the Authenticating and Reading Secrets With HashiCorp Vault tutorial for a version of this feature. It's available to all subscription levels, supports writing secrets to and deleting secrets from Vault, and supports multiple secrets engines.

Vault Secrets Engines

• generic option introduced in GitLab Runner 16.11.

The Vault Secrets Engines supported by GitLab Runner are:

Configure your Vault server

To configure your Vault server:

1. Ensure your Vault server is running on version 1.2.0 or later.

2. Enable the authentication method by running these commands. They provide your Vault server the OIDC Discovery URL for your GitLab instance, so Vault can fetch the public signing key and verify the JSON Web Token (JWT) when authenticating:

   $ vault auth enable jwt
   
   $ vault write auth/jwt/config \
     oidc_discovery_url="https://gitlab.example.com" \
     bound_issuer="gitlab.example.com"

3. Configure policies on your Vault server to grant or forbid access to certain paths and operations. This example grants read access to the set of secrets required by your production environment:

   vault policy write myproject-production - <<EOF
   # Read-only permission on 'ops/data/production/*' path
   
   path "ops/data/production/*" {
     capabilities = [ "read" ]
   }
   EOF

4. Configure roles on your Vault server, restricting roles to a project or namespace, as described in Configure Vault server roles on this page.

5. Create the following CI/CD variables to provide details about your Vault server:

   • VAULT_SERVER_URL - The URL of your Vault server, such as https://vault.example.com:8200. Required.
   • VAULT_AUTH_ROLE - Optional. The role to use when attempting to authenticate. If no role is specified, Vault uses the default role specified when the authentication method was configured.
   • VAULT_AUTH_PATH - Optional. The path where the authentication method is mounted, default is jwt.
   • VAULT_NAMESPACE - Optional. The Vault Enterprise namespace to use for reading secrets and authentication. If no namespace is specified, Vault uses the root ("/") namespace. The setting is ignored by Vault Open Source.

   NOTE: Support for providing these values in the user interface is tracked in this issue.

Use Vault secrets in a CI job

DETAILS: Tier: Premium, Ultimate Offering: GitLab.com, Self-managed, GitLab Dedicated

After configuring your Vault server, you can use the secrets stored in Vault by defining them with the vault keyword:

job_using_vault:
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.com
  secrets:
    DATABASE_PASSWORD:
      vault: production/db/password@ops  # translates to secret `ops/data/production/db`, field `password`
      token: $VAULT_ID_TOKEN

In this example:

• production/db - The secret.
• password The field.
• ops - The path where the secrets engine is mounted.

After GitLab fetches the secret from Vault, the value is saved in a temporary file. The path to this file is stored in a CI/CD variable named DATABASE_PASSWORD, similar to variables of type file.

To overwrite the default behavior, set the file option explicitly:

secrets:
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.com
  DATABASE_PASSWORD:
    vault: production/db/password@ops
    file: false
    token: $VAULT_ID_TOKEN

In this example, the secret value is put directly in the DATABASE_PASSWORD variable instead of pointing to a file that holds it.

Use a different secrets engine

The kv-v2 secrets engine is used by default. To use a different engine, add an engine section under vault in the configuration.

For example, to set the secret engine and path for Artifactory:

job_using_vault:
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.com
  secrets:
    JFROG_TOKEN:
      vault:
        engine:
          name: generic
          path: artifactory
        path: production/jfrog
        field: access_token
      file: false

In this example, the secret value is obtained from artifactory/production/jfrog with a field of access_token. The generic secrets engine can be used for kv-v1, AWS, Artifactory and other similar vault secret engines.

Configure Vault server roles

When a CI job attempts to authenticate, it specifies a role. You can use roles to group different policies together. If authentication is successful, these policies are attached to the resulting Vault token.

Bound claims are predefined values that are matched to the JWT claims. With bounded claims, you can restrict access to specific GitLab users, specific projects, or even jobs running for specific Git references. You can have as many bounded claims you need, but they must all match for authentication to be successful.

Combining bounded claims with GitLab features like user roles and protected branches, you can tailor these rules to fit your specific use case. In this example, authentication is allowed only for jobs running for protected tags with names matching the pattern used for production releases:

$ vault write auth/jwt/role/myproject-production - <<EOF
{
  "role_type": "jwt",
  "policies": ["myproject-production"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_email",
  "bound_claims_type": "glob",
  "bound_claims": {
    "project_id": "42",
    "ref_protected": "true",
    "ref_type": "tag",
    "ref": "auto-deploy-*"
  }
}
EOF

WARNING: Always restrict your roles to a project or namespace by using one of the provided claims like project_id or namespace_id. Without these restrictions, any JWT generated by this GitLab instance may be allowed to authenticate using this role.

For a full list of ID token JWT claims, read the How It Works section of the Authenticating and Reading Secrets With HashiCorp Vault tutorial.

You can also specify some attributes for the resulting Vault tokens, such as time-to-live, IP address range, and number of uses. The full list of options is available in Vault's documentation on creating roles for the JSON web token method.

Using a self-signed Vault server

When the Vault server is using a self-signed certificate, you see the following error in the job logs:

ERROR: Job failed (system failure): resolving secrets: initializing Vault service: preparing authenticated client: checking Vault server health: Get https://vault.example.com:8000/v1/sys/health?drsecondarycode=299&performancestandbycode=299&sealedcode=299&standbycode=299&uninitcode=299: x509: certificate signed by unknown authority

You have two options to solve this error:

• Add the self-signed certificate to the GitLab Runner server's CA store. If you deployed GitLab Runner using the Helm chart, you have to create your own GitLab Runner image.
• Use the VAULT_CACERT environment variable to configure GitLab Runner to trust the certificate:
  • If you are using systemd to manage GitLab Runner, see how to add an environment variable for GitLab Runner.
  • If you deployed GitLab Runner using the Helm chart:

    1. Provide a custom certificate for accessing GitLab, and make sure to add the certificate for the Vault server instead of the certificate for GitLab. If your GitLab instance is also using a self-signed certificate, you should be able to add both in the same Secret.

    2. Add the following lines in your values.yaml file:

       ## Replace both the <SECRET_NAME> and the <VAULT_CERTIFICATE>
       ## with the actual values you used to create the secret
       
       certsSecretName: <SECRET_NAME>
       
       envVars:
         - name: VAULT_CACERT
           value: "/home/gitlab-runner/.gitlab-runner/certs/<VAULT_CERTIFICATE>"

Troubleshooting

resolving secrets: secret not found: MY_SECRET error

When GitLab is unable to find the secret in the vault, you might receive this error:

ERROR: Job failed (system failure): resolving secrets: secret not found: MY_SECRET

Check that the vault value is correctly configured in the CI/CD job.

You can use the kv command with the Vault CLI to check if the secret is retrievable to help determine the syntax for the vault value in your CI/CD configuration. For example, to retrieve the secret:

$ vault kv get -field=password -namespace=admin -mount=ops "production/db"
this-is-a-password