GitLab + GitLab Runner (Pitfalls and Good Practices)

Recently we started using GitLab including the CI/CD features here at KeyIdentity. During the first weeks we stumbled upon some issues, which can make it a bit hard to get it up and running. Our Setup includes a GitLab Server installed via Omnibus (https://docs.gitlab.com/omnibus/README.html) and some GitLab-runners within docker. In addition, we are using a docker registry which requires authentication for pull and push. In this article we wanted to share how, we solved problems, when setting up the CI/CD system of GitLab.

1) Using own CA or a self-signed certificate on GitLab and trying to connect a GitLab-runner to it

You may see an error like this, when trying to register the runner:
x509: certificate signed by unknown authority
Fix it by adding the “–tls-ca-file” parameter to the gitlab-runner register command.
Example:

gitlab-runner register -n \
--tls-ca-file=/path/to/my/ca.pem \
--url=https://gitlab.domain.com \
--registration-token= \
--name=my-runner-1
...

Source: https://docs.gitlab.com/runner/configuration/tls-self-signed.html

2) “Cannot connect to the Docker daemon…”

Your runner ist not able to connect to the docker-deamon to spawn additional docker container for your tests or builds. The error you may run into looks like this:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? 
Fix it by adding the privileged flag to the configuration or add to corresponding parameter to the register command:

Register command:

gitlab-runner register -n \
 --docker-privileged
...

Config file at /etc/gitlab-runner/config.toml

 [[runners]]
 name = "my-runner-1"
 url = "https://gitlab.domain.com/"
 token = "yourToken"
 ...
 [runners.docker]
 privileged = true
 ....

Explanations and Manual:
https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#use-docker-in-docker-executor

3) Using an internal docker registry with SSL and own CA

It is very likely that the error looks quite similar to the one mentioned at 1).
ERROR: Preparation failed: Error response from daemon: Get https://docker-registry.domain.com/v1/_ping: x509: certificate signed by unknown authority

This error can have two different causes:

1) Your docker host is missing your CA file.
If so, include it by using “update-ca-certificates” on Debian / Ubuntu -based Linux systems (https://wiki.ubuntuusers.de/CA/#CA-Zertifikate-verwalten)
In short:
Place your ca file with .crt extension (important!) into “/usr/local/share/ca-certificate”

 run "sudo update-ca-certificates"

2) When using docker-in-docker (dind), you have probably a part like this in your gitlab-ci.yml

...
 image: docker:latest
 services:
 - docker:dind
 ...

More details on this can be found here:
https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#use-docker-in-docker-executor

In this case the error is arised from the “docker:dind” container.
The only resolution i found for this, was adding the CA-file into the “docker:dind” container, resulting in a new, self-build image:
Your own dind-Dockerfile may look like this:

FROM docker:dind

COPY /root-ca/ca.crt /usr/local/share/ca-certificates/ca.crt

RUN update-ca-certificates

After building and pushing to your internal docker registry, you can use your own “docker:dind” in your .gitlab-ci.yml

...
image: docker:latest
services:
  - docker-registry.domain.com:dind
...

This problem is known and discussed at GitLab.
https://gitlab.com/gitlab-org/gitlab-runner/issues/1350

4) Using an internal docker registry with Basic-Auth

Trying to pull from the internal docker-registry will fail, if it is read-protected and the gitlab-runner has no credentials.
This can be resolved by adding credentials to your gitlab-runner config.toml.
Create a TOKEN, based in username and password.

echo -n 'user:password' | base64

Add your docker-registry URL and the TOKEN this code snippet and add it to your config. Restart the service after adding the line 😉

[[runners]]
 ...
 environment = ["DOCKER_AUTH_CONFIG={ \"auths\": { \"docker-registry.domain.com:port\": { \"auth\": \"TOKEN\" } } }"]
[runners.docker]
 ....

Source:
https://gitlab.com/gitlab-org/gitlab-ce/issues/27036#note_21980104

Further Reading:

There are two very helpful websites, which help a lot, when getting started:
1) The manual for the .gitlab-ci.yml, which can be found here
https://docs.gitlab.com/ee/ci/yaml/README.html

2) The list of available environment variables during a job run
https://docs.gitlab.com/ce/ci/variables/README.html

Happy CI/CD!

Feel free to share the newsShare on Facebook
Facebook
Share on Google+
Google+
Tweet about this on Twitter
Twitter
Share on LinkedIn
Linkedin

Published by

Andreas Bauer

DevOps Engineer