At DreamQuark, we develop products, to help financial institutions use transparent artificial intelligence without complexity. Hence, security is a key issue for our customers since sensitive data might move along DreamQuark’s services. That is why, on the one hand our teams are always seeking for new ways to improve the security level of our products and fulfil as much as possible our client’s requirements. On the other hand, we are a small DevOps team and we need to improve our quality of service without spending to much time and by automating as many things as possible.
Software development best practices can prevent new vulnerabilities to appear. But when adding new functionalities or new libraries, we might have to deal with the appearance of new vulnerabilities. Then, the questions we have to answer are: these new vulnerabilities are acceptable with respect to the security standards imposed? Are they relevant with respect to what we produce?
Since a few years DreamQuark embarks its services thanks to docker images, and perform its cloud deployments with Kubernetes/Helm.
Trivy is an open sourced security scanner which allow to detect vulnerabilities on both code and os dependencies of an image, and generate a report in a tunable format. For these reasons, we decided to use this tool for our security scans, and in particular to build the action and the orb.
These continuous integration jobs allow to highlight the new vulnerabilities and the one that have been removed from a Docker image after code edition by publishing a markdown report as a comment of a pull request. In particular, it allows us to see if added dependencies contain vulnerabilities or, when performing pull requests to fix former vulnerabilities, to see if they have really been removed.
You will be able to find all the related code on the repo dreamquark-ai/ci-security-report.
Example of Result
You will be able to find a working example on this pull request. Basically, we pull a python image and build a Dockerfile which removes a vulnerable package and add new ones with other vulnerabilities. Here is an example of report published in the pull request’s comments, using either the GitHub action or the orb:
Example of Security Report in the Pull Request
In the coming part, we will go deeper into this example and see how we have generated this report using the Github action and the CircleCI orb.
Integrate the Security Report to your CI
In order to properly integrate the security report to your workflow, a GitHub PAT is required in the environment variable with a full control of private repositories when calling the
dreamquark-ai/ci-security-report action in the last step.
Rights required for the GitHub PAT to provide the action/orb
This PAT will allow the action to comment your PR with the differential markdown report.
Then do not forget to add this PAT to your secrets or to your context depending on if you are using the GitHub Action or the CircleCI orb, respectively.
Get Security Reports with the GitHub Action
In order to use this action, we need to run some preliminary steps in order to make the
First, it is an usual thing, we need to checkout in order to let the workflow access the repository:
- uses: actions/checkout@master
Then we need into the workflow the base image used as reference. For our example, we used a
- run: docker pull python:3.8-buster
Then we can call the GitHub action with the appropriate arguments:
Example of Use of dreamquark-ai/ci-security-report Action
Notice that we also provide an env variable valued with the GitHub PAT (named
GITHUB_PAT), which is the expected name from the bash scripts.
At the end, the following workflow definition allows us to publish a security report in any pull request’s comment onto the repository
Example of Use of the GitHub Action
As you can notice our workflow is logically triggered on pull request since the final aim of the action is to post a report as a pull request comment.
Get the Security Reports with the CircleCI Orb
To call the CircleCI orb, we use the same steps than for the GitHub Action. Hence you can call the
dreamquark-ai/ci-security-report orb using the following
Example of use of the CircleCI orb
security-report-context must at least contain a variable named
GITHUB_PAT with the PAT previously generated.
You can notice that we use the
default executor which is a custom Docker image
quay.io/dreamquark/security-report:latest containing all what we need to run the Trivy reports, generate the markdown report and comment the PR with it (Docker, Trivy and the scripts).
Development of the Orb and Action
At Dreamquark, we often use CircleCI and GitHub actions for our integration tests. However, to easily integrate the implementation of these security reports to our existing CI pipelines without overloading the existing CircleCI pipelines, we initially wanted to only implement a GitHub action.
However, we faced an issue with some of our heaviest images. Indeed, when the image size increases up to approximately 4.5 GB, we were getting a 137 exit code error at the run of the Trivy security scan, meaning that the allocated resources are not enough to run the process (probably an Out Of Memory error).
Since we can not tune (yet?) the allocated resources by the virtual machine running the GitHub Action, we had to move some of the security report to CircleCI where we can choose among several sets of capacity resources.
For both the orb and the action, three different steps have been set:
- Create a security report for the the base image in a json format.
- Create a security report for the new image on which we want to find the new vulnerabilities and the ones that have been removed.
- Run a bash script which will look at the differences between the two security reports that have been generated, create a markdown report and publish it as a comment of the related pull request. We will come back on these scripts a bit later.
Reusable GitHub Action Creation
The GitHub action takes back the three previously described steps. But before these ones, another step is added to set up the environment. It creates the folder where the reports and some temporary files will be located, and it also installs the latest version Trivy.
Reusable CircleCI Orb Creation
The orb only takes back the three steps in which we generate the Trivy reports. Circle CI allowing you to use your own Docker images, we set up directly the environment through the image
quay.io/dreamquark/security-report:latest which basically contains Docker, Trivy and the scripts allowing to generate the markdown security report and to create a pull request comment with it.
If you want more details about the code, you can refer to the README of the repository.
Contribution and Contact
Do you want to reach a better CI/CD/CS ?
Feel free to contribute either by opening an issue or a pull request on GitHub !