Once the solution is designed and the code is written, it's "just" about deploying, and then we're done, right? We often use CI/CD solutions to build and deploy, run tests, and much more.
If someone can compromise pipelines, the build agent building the solution, or the connection to the resources we deploy to, we will have major problems.
Even though it's challenging to cover everything in a few short articles, we still try to provide insights into the issues that delivery teams should address.
1 - CI/CD
CI/CD is the overarching framework for automating builds, verification, and deployment. A good pipeline provides traceability, quality assurance, and controlled rollout across environments.
Continuous Integration and Continuous Delivery (CI/CD) are practices for automating the flow from code change to running solution. Pipelines provide a standardized process for building, verification, and rollout, and reduce the risk of manual errors.
The point of CI/CD is not just speed but control. Each run should be traceable, repeatable, and have clear stopping points before anything goes to production.
Principles for CI/CD
Safe CI/CD practice is built on a few simple foundational principles:
Least privilege: The pipeline should only have the access it actually needs.
Traceability: You should be able to see what was run, by whom, and against what code.
Automated rules: Quality and security requirements should be in the pipeline, not in people’s heads.
Controlled flow: Production should have its own control points with tests and approvals.
Environment separation: Build, test, and production should not share identity and access.
Security controls in the pipeline
A pipeline should have at minimum:
Secret scanning: Find hardcoded keys, tokens, and other sensitive information.
Dependency scanning: Find known vulnerabilities in third-party libraries.
Static code analysis (SAST): Find common weaknesses early.
Automated tests: Run unit, integration, and relevant security tests.
Audit logging: Keep logs and metadata for audit trails.
Critical findings should stop deployment, not just give a warning.
Branching and promotion model
Branching is closely connected to CI/CD, but the details belong in the Git article.
In CI/CD context, the main points are:
Code to production should go via pull request and protected main branch
Deployment to production should require passing controls and approval
Secrets and credentials in the pipeline
Secrets used in pipelines must be handled with particular care:
Never in code: Keys and tokens should not be in the repository.
Use secure storage: Use dedicated secret storage in your platform.
Rotate regularly: Change keys and tokens at fixed intervals.
Avoid leakage in logs: Ensure secrets don’t end up in logs.
Third-party components in the pipeline
Pipeline configuration is part of the supply chain. Third-party components in the pipeline (actions, orbs, templates, build images) run with pipeline access and must be treated as code you’re responsible for.
This means a compromised or malicious third-party component can:
Exfiltrate secrets and credentials available to the run
Manipulate artifacts being built without it being visible in your code
Change behavior when a tag is updated to point to new code without your approval
Measures:
Pin to commit hash, not tag: A tag can be moved without you noticing.
Limit access for actions: Give a component only the permissions it actually needs.
Review third-party actions: Review the source code of actions you use, especially those handling secrets or deploying to production.
Use internal or official actions: Prefer actions from the same organization or from verified actors with good track records.
Monitor changes: Use Dependabot or similar to notify you when actions are updated, so updates are explicit and tracked.
Using AI in pipelines: Be sure you understand the risks and potential challenges before considering using AI-based actions in your pipeline.
For AI solutions, the same pipeline principles should apply to code, models, datasets, and configuration. Deployment should stop if verification fails.
Details about safe building of artifacts (including signing) are described in Building, and safe deployment to environments in Deployment.
Practical adoption in teams
CI/CD implementation should be done gradually:
Start with basic building and testing, add security controls gradually.
Document pipeline configuration as code so it can be reviewed and versioned.
Give the team time to learn the tool and practice before enforcing stricter requirements.
Follow up on findings in practice. Alerts without follow-up have little value.
Building is about producing a reproducible artifact that can be verified and trusted before deployment. Build environment, dependencies, and signing are critical control points.
Building is the step where source code becomes a deployable artifact. The goal is for the artifact to be reproducible, verifiable, and traceable back to a specific commit and pipeline run.
This article focuses on the building process itself. Overall pipeline management is described in CI/CD, and deployment to environments is described in Deployment.
Build environments and build agents
Build agents typically come in two main forms:
Cloud provider-managed agents
Self-hosted agents (in the cloud or on-premise)
Cloud provider-managed agents often provide lower operating costs and simpler maintenance. Self-hosted agents give you more control but full responsibility for hardening, patching, and access management.
Regardless of which model you choose, the build environment is a high-risk point. If it’s compromised, all artifacts produced there become unreliable.
Requirements for secure building
A robust build setup should have at least:
Reproducibility: Same input should produce the same artifact. Avoid hidden dependencies and undocumented manual steps.
Pinned dependencies: Lock versions explicitly and avoid “latest” in builds.
Isolation: Separate build jobs between projects with different risk levels.
Control over build tools: Compilers, base images, and plugins should be versioned and approved.
Traceability: The artifact must be traceable back to the commit, pipeline run, and build configuration.
Yes, signing naturally belongs in the build phase. Signing should happen right after the artifact is produced and verified.
Practical controls:
Sign artifacts: Sign containers, binaries, and packages with a controlled key.
Verify signature on deployment: Deployment should stop if the signature is missing or invalid.
Generate SBOM: Create an inventory of what’s in the artifact for traceability and vulnerability management.
Build provenance attestation: Document what built the artifact, where it came from, and under what conditions.
Building for AI systems
For AI solutions, building encompasses more than just application code. Trained models, supporting files for text processing, data representations, and runtime containers are all artifacts that need controlled handling throughout the process.
Version model artifacts: Treat model versions the same way as code versions.
Sign model and runtime artifacts: Use the same integrity requirements for models as for other artifacts.
Document dataset and model versions: Clearly describe what’s included in each release.
Avoid dynamic downloads in production: Don’t download model files outside of controlled release workflows.
Deployment is about controlled movement of a verified artifact to runtime environments. The goal is safe rollout, fast rollback, and predictable operations.
Deployment starts after the artifact has been built and verified. The main principle is that the same artifact is promoted across environments without rebuilding in between.
This article focuses on rollout to environments. Overall pipeline management is covered in CI/CD, and details about safe artifact production in Building.
Principles for safe deployment
To reduce operational and security risk, the deployment process should be built on:
Promotion of the same artifact: Don’t rebuild for each environment.
Verification before rollout: Deployment should stop if required checks don’t pass.
Automated and idempotent rollout: Same deployment command should produce the same result.
Traceability: Each deployment should be traceable to artifact, commit, and approval.
What a deployment process should cover
A mature deployment process should cover at minimum:
Environment promotion (dev-test-prod): Clear gates between environments.
Approval and change control: Requirements for approval before production.
Release strategy: Support for blue/green, canary, or rolling deployment where appropriate.
Rollback: Defined and tested procedure for fast reversal.
Smoke tests after deployment: Verify that the solution actually works in the target environment.
Observability: Monitoring and alerting that catch errors early after rollout.
Access management and environment security
Deployment access must be treated as a privileged operation:
Least privilege for deployment identities: Give only access needed for deployment.
Separate roles: Separate roles for development and production deployment where possible.
Limit production deployment triggers: Restrict who can start deployment to production.
Traceability on deployments: Log who deployed what, when, and to which environment.
Deployment of AI systems
For AI solutions, the deployment process must also handle model behavior and data-driven risk:
Model rollout with control: Use gradual rollout (for example canary) to reduce impact if model behavior is wrong.
Clear version linkage: Your running solution should be traceable to the exact model version, code version, and configuration.
Verification in target environment: Run relevant quality and security checks in the target environment after deployment, not just during the build phase.
Operational monitoring: Watch model quality, response time, and signs of data or output drift.
Fast rollback: Have clear procedure to switch back to previous model version if issues occur.
Penetration testing, often referred to as pentesting, is the art of testing a system to find weak points that can be exploited and the risk these weaknesses pose to the owner of the solution.
Security testing and pentesting have many similarities, but while approaches like DAST primarily focus on web applications and more automated tests, a pentest is more comprehensive and typically also includes underlying infrastructure and networks. In some cases, it may also have a physical element where pentesters will attempt to gain access to premises to uncover weaknesses in physical security or routines.
A penetration test will always have an agreed scope that regulates what the pentesters can do, when they can do it, and which resources and services they can test.
Why Pentest?
It is not possible to prove that a solution is secure, only that it is not vulnerable to certain attacks. If delivering a solution that has strict security requirements or operates within an agreement that dictates it, a pentest is a useful tool to ensure that the solution and its surrounding environment are secure.
After the testing is completed, a report will usually be delivered that describes what was tested and how, as well as an assessment of all findings. In some cases, findings may be described as vulnerabilities, but these do not necessarily need to be addressed due to other mitigating measures or because the risk or consequence is low.
What is Required to Conduct a Pentest?
First and foremost, you need one or more pentesters. This is not something you do on your own after watching a few videos on YouTube! A pentest requires expertise in several areas, as some attacks depend on exploiting multiple vulnerabilities that are not particularly serious on their own.
As a development team, you must ensure that the environment to be tested is properly identified so that everyone understands where the testing is taking place. The scope of the test must be defined - remember that it must be possible to distinguish an actual attack from a pentest if both occur simultaneously: If you see signs of an attack on an environment that is not part of the test and you have segregated your environments, you should take action!
As part of the planning, it is important to check with the customer what routines they have for pentesting. In many cases, they will have a Security Operations Center (SOC) and/or a Network Operations Center (NOC) that continuously monitors the infrastructure. These must be part of the planning to avoid misunderstandings or problems when the test begins.
In some cases, it is desirable to conduct a pentest without notifying anyone, as you want to see if such a test is detected - remember that a pentest is, in practice, an attack.
When to Conduct a Pentest, and What to Do While It Is Ongoing?
In a perfect world, you should conduct a pentest with every major change, but this is not feasible except for a few actors with special requirements. Each customer will have different requirements and expectations, so it is important to establish guidelines for this before planning to conduct the test.
If the test is announced in advance, it is a great opportunity to monitor logs and other monitoring tools to see if you notice anything unusual. If you can correlate this information with the tests reported afterward, you have a good opportunity to create automatic alerting routines that detect deviations from the norm.
What to Do After a Pentest?
When the team receives the report after a completed test, it is important to review it with the product owner. Always remember that security is never the responsibility of individuals alone - it is the delivery manager’s responsibility to ensure that security measures are implemented, but it is the team’s collective responsibility to ensure that what is built meets the set requirements.
Identified findings must be classified and added to the backlog. Then, the findings must be assessed against the importance of addressing them; some findings can wait, while others must be addressed as quickly as possible. This will vary from delivery to delivery and finding to finding.
Remember
You should never conduct a pentest yourself unless you know very well what you are doing. It is not allowed to run tools used in connection with pentesting on Bouvet machines or in Bouvet’s network without this being cleared with Internal IT & Security in advance.
