Aaryamann Challani

Introduction

CI/CD pipelining for Kubernetes is hard 🥱 and messy at times. There are a couple good CI/CD options out there like Argo CD and Jenkins but they require quite a deal of setting up, especially if you are not familiar with them.

For a team migrating their Monolith to Kubernetes but still want to continue using github actions, this would be a good option. ✅

This is also a good option for small teams(<10) ✅

---

Requirements

• A Github Account 😉
• Actions enabled on the repository
• Experience with YAML
• A Running GKE Cluster
• Service account with the Kubernetes Engine Admin IAM policy

---

Initial Setup

1. Get the service account key with the given IAM policy or ask your sys admin to save it in the secrets section of the repo with the name GKE_CREDENTIALS

2. Move all your kubernetes manifests into a single directory of choice.

3. Create a workflow.yml in your .github/workflows directory

---

The Main Course

Copy the following into the workflow.yml

name: k8s GitOps
on:
  push:
    branches:
      - "development"
    paths:
      - "k8s/**"

What this does is tells the workflow to trigger only when - There is a push to the development branch - There is a change in the k8s/ directory(which is where the manifests live)

Get Changed Manifests

In this step we will perform a simple search to find which manifests have been changed. Note that "changed" means either - modified, created, deleted, renamed

1. Checkout the repo

   get_changed_manifests:
     name: Get changed k8s manifests
     runs-on: ubuntu-latest
     steps:
       - name: checkout
         uses: actions/checkout@v2
           with:
           fetch-depth: 0
   

2. Get the list of changed files

   - name: Get changed files
     id: getfile
     run: |
       echo "::set-output name=files::$(git diff --name-only ${{ github.event.before }}..${{ github.event.after }} | grep 'k8s/' | xargs)"
   

Using git diff we are able to get the list of changed files from before the push and after the push

3. Parse the list into a matrix(more on this later)

   - name: Parse changed files into matrix
     id: matrix
     run: |
       arr=(`echo ${{ steps.getfile.outputs.files }}`);
       empty_array=()
       json_array() {
         echo -n '['
         while [ $# -gt 0 ]; do
           x=${1//\\/\\\\}
           echo -n \"${x//\"/\\\"}\"
           [ $# -gt 1 ] && echo -n ', '
           shift
         done
         echo ']'
       }
       output_arr=$(json_array ${arr[@]})
       output_empty=$(json_array ${empty_array[@]})
       output=$(echo "{\"manifests\": $output_arr}")
       empty=$(echo "{\"manifests\": $output_empty")
       echo "::set-output name=manifests::$output"
       echo "::set-output name=empty_matrix::$empty"
     outputs:
       matrix: ${{ steps.matrix.outputs.manifests }}
       empty_matrix: ${{ steps.matrix.outputs.empty_matrix }}
   

Quite a bit isn't it? What we're doing here is parsing the bash array of changed manifests into a json array. We can then use this json array as a matrix strategy for the next step, i.e to apply these manifests.

You probably noticed we've also initialized an empty matrix as well. This is used to check whether no files have been changed

Verify Changed Manifests

In this step we will be performing a simple sanity check on the structures produced by the previous step

check_matrix:
  name: Validate Generated Matrix
  needs: get_changed_manifests
  runs-on: ubuntu-latest
  steps:
    - name: Test with jq
      id: check
      run: |
        matrix='${{ needs.get_changed_manifests.outputs.matrix }}'
        empty_matrix='${{ needs.get_changed_manifests.outputs.empty_matrix }}'
        echo $matrix
        echo $matrix | jq .
        if [[ "$matrix" == "$empty_matrix" ]]; then
          echo "::set-output name=is_empty::true"
        else
          echo "::set-output name=is_empty::false"
        fi
  outputs:
    is_empty: ${{ steps.check.outputs.is_empty }}

Notice that one of the outputs is the is_empty variable. We will use this as a conditional for the next step

Apply Changed Manifests

In this step we will parse the produced matrix into a strategy and apply/delete the resources as required. Note that ideally we would use gke-deploy instead of the regular kubectl if it was a package.

1. Check if the changed manifests matrix is empty

   if: ${{ needs.check_matrix.outputs.is_empty == 'false' }}
   

If it is false the workflow cleanly exits

2. Set Environment Variables

   env:
     cluster_name: development-cluster
     location: zone-of-cluster
   

3. Set the needs and the strategy

   - name: Apply changed k8s manifests
     runs-on: ubuntu-latest
     needs:
       - get_changed_manifests
       - check_matrix
     strategy:
       matrix: ${{ fromJSON(needs.get_changed_manifests.outputs.matrix) }}
   

fromJson() was launched in April, 2020 to make the dev-ex simpler for handling complex workflows. We are using it to parse the json array generated from the first job.

Putting the previous 2 jobs in the needs field ensures that it is pipelined correctly

4. Setup Deployment Scaffold

   steps:
     - name: Checkout
       uses: actions/checkout@v2
       with:
         fetch-depth: 0
     - name: Get GKE Credentials
       uses: google-github-actions/get-gke-credentials@main
       with:
         cluster_name: ${{ env.cluster_name }}
         location: ${{ env.location }}
         credentials: ${{ secrets.GKE_CREDENTIALS }}
   

5. Apply the manifests

   - name: Apply Manifests
     id: apply
     run: |
       if [[ -f ${{ matrix.manifests }} ]]; then
         kubectl apply -f ${{ matrix.manifests }}
       else
         git show ${{ github.event.before }}:${{ matrix.manifests }} | kubectl delete --ignore-not-found=true -f -
       fi
   

If the file exists, we apply it. If it has been deleted, we get the commit it existed at and delete it from the cluster.

---

That's It! 🚀 A fully custom ci/cd setup for small teams.

Note that you may want to use something more robust if you have hundreds of resources needing to be deployed. This works just fine though! 🎉

---

Final Notes

Setting this up took quite a bit of time since there were quite a few struggles with the json array bit, but I'm glad it worked out!

Future steps:

• Figure out how to integrate gke-deploy into the last step of the workflow
• Have a workflow that validates manifests in the PR

---

Thanks for reading! 🎉