Using Git submodules with GitLab CI/CD

DETAILS: Tier: Free, Premium, Ultimate Offering: GitLab.com, Self-managed, GitLab Dedicated

Use Git submodules to keep a Git repository as a subdirectory of another Git repository. You can clone another repository into your project and keep your commits separate.

Configure the `.gitmodules` file

When you use Git submodules, your project should have a file named .gitmodules. You have multiple options to configure it to work in a GitLab CI/CD job.

Using absolute URLs

Introduced in GitLab Runner 15.11.

For example, your generated .gitmodules configuration might look like the following if:

Your project is located at https://gitlab.com/secret-group/my-project.
Your project depends on https://gitlab.com/group/project, which you want to include as a submodule.
You check out your sources with an SSH address like git@gitlab.com:secret-group/my-project.git.

[submodule "project"]
  path = project
  url = git@gitlab.com:secret-group/project.git

In this case, use the GIT_SUBMODULE_FORCE_HTTPS variable to instruct GitLab Runner to convert the URL to HTTPS before it clones the submodules.

Alternatively, if you also use HTTPS locally, you can configure an HTTPS URL:

[submodule "project"]
  path = project
  url = https://gitlab.com/secret-group/project.git

You do not need to configure additional variables in this case, but you need to use a personal access token to clone it locally.

Using relative URLs

WARNING: If you use relative URLs, submodules may resolve incorrectly in forking workflows. Use absolute URLs instead if you expect your project to have forks.

When your submodule is on the same GitLab server, you can also use relative URLs in your .gitmodules file:

[submodule "project"]
  path = project
  url = ../../project.git

The above configuration instructs Git to automatically deduce the URL to use when cloning sources. You can clone with HTTPS in all your CI/CD jobs, and you can continue to use SSH to clone locally.

For submodules not located on the same GitLab server, always use the full URL:

[submodule "project-x"]
  path = project-x
  url = https://gitserver.com/group/project-x.git

Use Git submodules in CI/CD jobs

To make submodules work correctly in CI/CD jobs:

You can set the GIT_SUBMODULE_STRATEGY variable to either normal or recursive to tell the runner to fetch your submodules before the job:
```
variables:
  GIT_SUBMODULE_STRATEGY: recursive
```
For submodules located on the same GitLab server and configured with a Git or SSH URL, make sure you set the GIT_SUBMODULE_FORCE_HTTPS variable.
Use GIT_SUBMODULE_DEPTH to configure the cloning depth of submodules independently of the GIT_DEPTH variable:
```
variables:
  GIT_SUBMODULE_DEPTH: 1
```
You can filter or exclude specific submodules to control which submodules are synchronized using GIT_SUBMODULE_PATHS.
```
variables:
  GIT_SUBMODULE_PATHS: submoduleA submoduleB
```
You can provide additional flags to control advanced checkout behavior using GIT_SUBMODULE_UPDATE_FLAGS.
```
variables:
  GIT_SUBMODULE_STRATEGY: recursive
  GIT_SUBMODULE_UPDATE_FLAGS: --jobs 4
```

If you use the CI_JOB_TOKEN to clone a submodule in a pipeline job, the user executing the job must be assigned to a role that has permission to trigger a pipeline in the upstream submodule project. Additionally, CI/CD job token access must be properly configured in the upstream submodule project.

Troubleshooting

Can't find the `.gitmodules` file

The .gitmodules file might be hard to find because it is usually a hidden file. You can check documentation for your specific OS to learn how to find and display hidden files.

If there is no .gitmodules file, it's possible the submodule settings are in a git config file.

`fatal: run_command returned non-zero status` error

This error can happen in a job when working with submodules and the GIT_STRATEGY is set to fetch.

Setting the GIT_STRATEGY to clone should resolve the issue.