GitLab CI/CD workflow ¶

GitLab CI/CD is a well-known and popular mature solution for DevOps. From my perspective, GitLab CI free version is relatively limited. For instance, most interfaces to display CI results are rudimentary or paid.

• CI/CD Documentation ⛪
• Code Coverage Documentation ✍️
• Code Quality Documentation 🧼
• Unit Tests Documentation 🧪
• Security Tools Documentation 🛡️ (static analysis...)
• GitLab Templates ✨

We assume you're familiar with GitLab, and already have a project created with Maintainer+ access on it.

To enable CI/CD, go to Settings > General > Visibility [...] > Repository and enable CI/CD. Don't forget to save your changes.

Terminology

• Jobs ⚒️: a task such as "build project" or "run tests". It's basically a set of commands to accomplish a goal.

• Stages 🗃️: jobs are sorted in stages (ex: Build, Test, Deploy). We commonly have the order: Build > Test > Deploy.

• Pipeline 🚀: an execution of the stages, on the code associated with a specific commit/branch. The pipeline will fail if a job fails.

The pipeline status is visible next to each commit: (passed).

A runner is an agent running the jobs, e.g., executing the commands.

• Project runner 🏠: available for a single project
• Group runner 🏘️: available for any project in a group
• Shared runner 🍾: available for any project

👉 Only shared runners are available when using GitLab.com.

Runners are associated with an executor. It's the environment used to execute commands, such as a shell or a docker.

GitLab runners are managed by a service called gitlab-runner. You can find instructions here on how to install it.

Some commands you might use:

$ sudo gitlab-runner -h
$ # you may use a new user
$ sudo gitlab-runner install --user=xxx --working-directory=yyy
$ sudo gitlab-runner start
$ sudo gitlab-runner list
$ sudo gitlab-runner verify --delete # delete "dead" runners

$ # use --tls-ca-file=xxx.crt if you've CA problems
$ sudo gitlab-runner register --url URL --registration-token XXX
Enter a description
Enter tags (cannot be edited, can be used to assign jobs)
Enter a maintenance note
Enter an executor (docker, shell, ...)
$ sudo gitlab-runner verify
$ sudo gitlab-runner restart

Once created, the runner can be configured, to some extent, by editing /etc/gitlab-runner/config.toml. For instance, you could:

• use a custom helper image (docker executor, helper_image = "")
• change the default docker image (docker executor, image = "")
• add docker volumes (docker executor, volumes = [])
• ...

Don't forget to restart after any change:

$ sudo gitlab-runner restart

It's a YAML file. See the reference. When using the online editor, in CI/CD > Editor (you can select the file and the branch):

• you know if the file is valid or not 🚀
• you can visualize the pipeline 🔎
• you can see the merged YAMl 💵 (useful if you use templates)
• ...

Stages ¶

The first step is usually to define the stages:

stages:
  - build
  - test
  - deploy

Variables ¶

You can declare variables globally or inside a job.

variables:
  VAR_NAME: "some value"
  VAR_NAME_2: "$VAR_NAME/2"   # can use variables
  # GIT_STRATEGY: none # empty clone
  # see also: $CI_PROJECT_NAME (repository name)
  #           $CI_COMMIT_REF_NAME (commit name)

job:
  stage: build
  script: # usages (quoted + ${}, or unquoted)
    - echo "$VAR_NAME ${VAR_NAME_2}" $VAR_NAME

➡️ See also: external secrets and predefined variables.

Tags ¶

If you add a tag to a job, then only runners with this tag can run it.

job:
  tags:
    - xxx

Image ¶

You can declare the image to use globally or inside a job.

image: xxx:5000/docker_img

Default ¶

You can use the default keyword to set default properties.

default:
  tag:
    - xxx

Script ¶

You can use before_script, script, and after_script to write down the commands executed by the runner.

  script:
    - pwd          # a simple command call
    - "pwd"        # some complex command must be quoted

    - exit 0       # job success
    - exit 1       # job failure
    
    - xxx || true  # allow one command to fail

    # some "sed" that may be useful
    - sed -i 's/xxx/yyy'$(echo $XXX | sed 's/\//\\\//g')'\/yyy/g' file
    - "line=$(($(grep -n "xxx" file | cut -d: -f1) - 1))"
    - sed -i $line' i xxx'
    - sed -i $line' i \\txxx'

⚠️ Environment is reset between two jobs. See also: artifacts.

⚠️ Commands are not exactly behaving the same as in your terminal.

# tested on a runner with a docker executor
   - source xxx.sh              # export XXX="..."
   - echo $XXX                  # XXX is empty
   - source xxx.sh && echo $XXX # XXX is not empty

Allowed repositories can be cloned from a pipeline.

    - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@example.com/XXX

allow_failure ¶

You can allow a job to fail without failing the pipeline. In such a scenario, an orange icon with an exclamation mark will be shown.

job:
  allow_failure: true

when ¶

You can use when to execute jobs conditionally.

job:
  when: on_failure   # a stage failed in the previous stage
  when: always       # even if the pipeline fails

rules ¶

You can use rules to conditionally add/remove a job.

job:
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: always

only ¶

You can use only to execute jobs only on predefined branches.

job:
  only:
    - master

Artifacts ¶

Artifacts can be used to pass files to following jobs or to export results for users to see or download them.

job-a:
  artifacts:
    paths:
      - folder/

# inherit folder from "job-a"
job-b:
  dependencies:
    - job-a

For instance, for code quality or unit tests which are both free:

  artifacts:
    when: always
    reports:
      # shown in the pipeline "tests" tab
      junit: "*_tests.xml"
      # shown in the pipeline "code quality" tab
      codequality: "report.json"
    expire_in: 3 days

It's possible to extract some logic into a reusable YAML template. The aforementioned template can be included using various ways. One is:

include:
  # include https://example.com/path/to/project
  - project: 'path/to/project'
    file: 'xxx.yml'
    ref: 'main'

Once loaded, you can override jobs and variables from the template.

Assuming the template has a variable VAR_NAME, declaring a variable with the same name in the including file will override its value.

Assuming the template has a job some_job, then we may further tune it in the including file (check the merged YAML view to see what happens).

some_job:       # job declared in the template
  before_script: # we can add new properties
    - xxx
  script:        # we can override a property's value
    - yyy    

GitLab offers 3 badges: pipeline status, coverage status, and release status. It's possible to use external APIs for public projects. Otherwise, a common solution is to build badges during the CI pipeline.

🧨 A major downside is that badges are NOT shown when a pipeline fails. A possible solution (not tested) is to save badges somewhere instead of using artifacts (ex: make CI commit to a branch).

👉 You'll need to pull the image xxx. In the code sample, it's assumed to be available at xxx:5000/anybadge, but you can change it.

To load a badge, go to Settins > General > Badges. Assuming the job is called generate_badges and the badge is badge.svg, the URL is: https://example.com/%{project_path}/-/jobs/artifacts/%{commit_sha}/raw/badge.svg?job=generate_badges

generate_badges:
  script:
    # output is a sort of: [ hello ][  world  ]
    #              colors: [ black ][  red  ]
    - badge_label="hello"
    - badge_text="world"
    - badge_color="red"
    - docker run --rm -v $(PWD):/src xxx:5000/anybadge anybadge --value="$badge_text" --file=badge.svg --label="$badge_label" -c=$badge_color -o
  artifacts:
    paths:
      - badge.svg

If needed, you can use test and &&/|| to write conditionals:

    - res=$(exit 0)
    - badge_color=$(test $res -eq 0 && echo "green" || echo "red")

Enumeration ¶

• Custom installations are often at gitlab.example.com
• Navigate to /help to expose the version (logged)
• Username fuzzing with username wordlists

$ ffuf -u 'URL/FUZZ' -w wordlist -mc 200,301

Foothold ¶

• Look for public repositories (/explore)
• Look if you can log in using Windows
• Look if you can create accounts (/users/sign_up). We often need to use the company email which may block us.
• Look if the newly created account need admin verification. If they don't, we may be able to access internal repositories (/explore).

Exploitation ¶

• May contains token, ssh keys, credentials, passwords

Well-known CVEs ¶

Gitlab 13.10.2 RCE:

$ sudo apt install djvulibre-bin
$ searchsploit -p 49951
$ python3 49951.py -u xxx -p xxx -t URL -c 'command'

• how to trigger the pipeline
• Logs
• Webhook

cache:
  paths:
    - /xxx/