GitLab CI/CD instance configuration

DETAILS: Tier: Free, Premium, Ultimate Offering: Self-managed

GitLab administrators can manage the GitLab CI/CD configuration for their instance.

Disable GitLab CI/CD in new projects

GitLab CI/CD is enabled by default in all new projects on an instance. You can set CI/CD to be disabled by default in new projects by modifying the settings in:

gitlab.yml for self-compiled installations.
gitlab.rb for Linux package installations.

Existing projects that already had CI/CD enabled are unchanged. Also, this setting only changes the project default, so project owners can still enable CI/CD in the project settings.

For self-compiled installations:

Open gitlab.yml with your editor and set builds to false:

## Default project features settings
default_projects_features:
  issues: true
  merge_requests: true
  wiki: true
  snippets: false
  builds: false

Save the gitlab.yml file.
Restart GitLab:
```
sudo service gitlab restart
```

For Linux package installations:

Edit /etc/gitlab/gitlab.rb and add this line:

gitlab_rails['gitlab_default_projects_features_builds'] = false

Save the /etc/gitlab/gitlab.rb file.
Reconfigure GitLab:
```
sudo gitlab-ctl reconfigure
```

Set the `needs` job limit

DETAILS: Tier: Free, Premium, Ultimate Offering: Self-managed

The maximum number of jobs that can be defined in needs defaults to 50.

A GitLab administrator with access to the GitLab Rails console can choose a custom limit. For example, to set the limit to 100:

Plan.default.actual_limits.update!(ci_needs_size_limit: 100)

To disable needs dependencies, set the limit to 0. Pipelines with jobs configured to use needs then return the error job can only need 0 others.

Change maximum scheduled pipeline frequency

Scheduled pipelines can be configured with any cron value, but they do not always run exactly when scheduled. An internal process, called the pipeline schedule worker, queues all the scheduled pipelines, but does not run continuously. The worker runs on its own schedule, and scheduled pipelines that are ready to start are only queued the next time the worker runs. Scheduled pipelines can't run more frequently than the worker.

The default frequency of the pipeline schedule worker is 3-59/10 * * * * (every ten minutes, starting with 0:03, 0:13, 0:23, and so on). The default frequency for GitLab.com is listed in the GitLab.com settings.

To change the frequency of the pipeline schedule worker:

Edit the gitlab_rails['pipeline_schedule_worker_cron'] value in your instance's gitlab.rb file.
Reconfigure GitLab for the changes to take effect.

For example, to set the maximum frequency of pipelines to twice a day, set pipeline_schedule_worker_cron to a cron value of 0 */12 * * * (00:00 and 12:00 every day).

Disaster recovery

You can disable some important but computationally expensive parts of the application to relieve stress on the database during ongoing downtime.

Disable fair scheduling on instance runners

When clearing a large backlog of jobs, you can temporarily enable the ci_queueing_disaster_recovery_disable_fair_scheduling feature flag. This flag disables fair scheduling on instance runners, which reduces system resource usage on the jobs/request endpoint.

When enabled, jobs are processed in the order they were put in the system, instead of balanced across many projects.

Disable compute quota enforcement

To disable the enforcement of compute quotas on instance runners, you can temporarily enable the ci_queueing_disaster_recovery_disable_quota feature flag. This flag reduces system resource usage on the jobs/request endpoint.

When enabled, jobs created in the last hour can run in projects which are out of quota. Earlier jobs are already canceled by a periodic background worker (StuckCiJobsWorker).

CI/CD troubleshooting Rails console commands

The following commands are run in the Rails console.

WARNING: Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case.

Cancel stuck pending pipelines

project = Project.find_by_full_path('<project_path>')
Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count
Ci::Pipeline.where(project_id: project.id).where(status: 'pending').each {|p| p.cancel if p.stuck?}
Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count

Try merge request integration

project = Project.find_by_full_path('<project_path>')
mr = project.merge_requests.find_by(iid: <merge_request_iid>)
mr.project.try(:ci_integration)

Validate the `.gitlab-ci.yml` file

project = Project.find_by_full_path('<project_path>')
content = p.ci_config_for(project.repository.root_ref_sha)
Gitlab::Ci::Lint.new(project: project,  current_user: User.first).validate(content)

Disable AutoDevOps on Existing Projects

Project.all.each do |p|
  p.auto_devops_attributes={"enabled"=>"0"}
  p.save
end

Obtain runners registration token

WARNING: The ability to pass a runner registration token, and support for certain configuration arguments, was deprecated in GitLab 15.6 and is planned for removal in GitLab 18.0. Runner authentication tokens should be used instead. For more information, see Migrating to the new runner registration workflow.

Prerequisites:

Runner registration tokens must be enabled in the Admin area.

Gitlab::CurrentSettings.current_application_settings.runners_registration_token

Seed runners registration token

appSetting = Gitlab::CurrentSettings.current_application_settings
appSetting.set_runners_registration_token('<new-runners-registration-token>')
appSetting.save!

Run pipeline schedules manually

You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible.

# schedule_id can be obtained from Edit Pipeline Schedule page
schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>)

# Select the user that you want to run the schedule for
user = User.find_by_username('<username>')

# Run the schedule
ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule)