Password Deleted from Git but Still Present: Major CI/CD Security Mistakes

To compromise an application, attackers do not always need a clever code flaw. A token left in configuration, an outdated library inside a container, or an overly permissive cloud role is often enough. Builds succeed, tests pass, and infrastructure applies dangerous settings without raising alarms. Automation simply follows the instructions it receives.

Password deleted from Git but history remains

Secrets rarely appear with comments like "production key, do not publish." Developers more often insert real values into example configuration files, save them in .env for local runs, or leave them in test scenarios. Private SSH keys, database connection strings, package manager authentication files, and cloud client configurations belong to the same category. Even private repositories require checks because any reader with history access can retrieve the secret.

Deleting a file in a new commit does not erase earlier versions. Adding a path to .gitignore does not stop tracking of files already under Git control. After a leak, the old value must first be revoked and replaced, followed by checks of usage logs and any distributed copies. History rewriting does not revoke the secret from existing clones.

The first check should run before commit. Gitleaks and similar secret scanners integrate with local Git hooks. The scan must target staged changes in the index rather than only the working directory, because partial staging can create differences between the two locations.

A Bash hook using current Gitleaks can be placed in .githooks/pre-commit. The --staged flag checks the index while --redact hides discovered values in output.

#!/usr/bin/env bash
set -euo pipefail
exec gitleaks git --pre-commit --staged --redact .

Local hooks can be disabled or forgotten, so CI pipelines must repeat the scan on incoming commits. Server-side push protection adds another layer when supported by the platform.

Environment variables do not automatically protect tokens

Moving a password from source code into an environment variable removes one copy from Git. The next step is determining who supplies the value, which tasks receive it, and where it might be logged. Debug output, verbose command logs, or diagnostic dumps can still expose the secret. Masking in CI helps but does not guarantee concealment of every transformed string.

A practical approach starts with separation of duties. A test task usually does not need the production database password. An image publishing task needs access only to a specific container registry. Production workloads require their own credentials with limited operations. A single shared token across testing, build, and deployment turns every task into a potential entry point for the entire chain.

Real values should come from a secret manager or the platform's protected mechanism, issued only to the required task. When cloud and CI support OIDC federation, the pipeline can exchange a verified identity for short-lived credentials. Trust is limited to a specific project, branch, or environment, and the issued role receives only necessary permissions.

In Kubernetes, values stored in the data field of a Secret object are Base64 encoded. Encoding is reversible and does not protect the secret from anyone reading the YAML file. Git should store either a reference to an external secret or an encrypted representation with separate key management.

Secrets can survive removal from containers

A Dockerfile can package content that was carefully removed from the repository. A local .env file remains on disk, the COPY . . instruction transfers the entire build context, and .gitignore rules do not apply to Docker. A .dockerignore file is required to exclude local secrets, the .git directory, and unnecessary build files.

A common attempt to fix the mistake looks convincing until image layers are considered:

COPY .env /app/.env
RUN ./build.sh
RUN rm /app/.env

The file is absent from the final filesystem, yet its content remains in a previous layer available to anyone who receives the image. Secrets must never be written into a layer with the intention of later removal. Multi-stage builds do not automatically solve the problem because intermediate results and cache can persist separately.

BuildKit supports secret mounts for values needed only during build. The secret is temporarily available to the specified RUN instruction and is not included in the layer. An npm build stage can receive a configuration file for a private package registry without embedding it.

Minimal images must also receive updates

Dependency checks for the application do not necessarily cover system packages in the base image. Outdated libraries, interpreters, and utilities may exist outside the project manifest. The final image must be scanned, including operating system packages and application dependencies. The container shares the host kernel, so a clean image report says nothing about host updates.

Minimization begins by asking what the process actually needs at runtime. Compilers, headers, and build tools stay in the build stage. The final stage receives only the application, required libraries, and service data such as trusted certificates. Distroless images reduce the set of common utilities but require application compatibility and a separate diagnostic approach.

Base images should come from a supported trusted source and be pinned by digest. A tag such as latest may point to a different image without changes to the Dockerfile. Digest pinning allows reproduction of the chosen version but also locks in its known vulnerabilities. Regular updates, rebuilds, testing, and rescanning are therefore required.

Images can be scanned with Trivy or Grype. These tools differ in capabilities, so vulnerability search, secret detection, and configuration checks are not a single automatic operation. The exact artifact that will be deployed must be scanned. If an image is rebuilt after scanning, the previous report no longer validates the new build.

Terraform reliably creates incorrect permissions

Infrastructure as code makes settings repeatable. An error in a shared module also becomes repeatable. Broad cloud roles, public storage access, or open administrative ports can propagate across multiple environments with a single template change. A successful terraform validate confirms syntactic correctness but does not prove that granted access matches the actual task.

Access rights should describe the specific actions the application performs. If a service reads objects from one bucket and prefix, it does not need full object storage management or policy modification. In AWS this can be expressed as s3:GetObject on the required objects. Additional operations such as listing are added only when the application actually performs them.

Network rules require the same context. A source of 0.0.0.0/0 in an allow rule covers every address. For a public HTTPS service the rule may be expected, but administrative access or databases require different solutions. Real public reachability also depends on routes, external addresses, load balancers, and service settings.

Both original .tf files and the computed plan should be checked. Checkov can analyze Terraform configuration and the JSON representation of the plan. The plan itself may contain secrets, so it must not be published in open merge request comments or public CI artifacts.

Sensitive hides output but not necessarily state content

Terraform maintains a state file that links described resources to real infrastructure. Passwords and other sensitive values can enter state along with resource attributes. The sensitive marker removes the value from normal output but does not exclude it from state or saved plans. Modern Terraform versions support ephemeral values and write-only arguments that help avoid storing secrets in state under supported scenarios.

Kubernetes: applications do not need cluster-wide access

In Kubernetes, process rights inside the container and service account rights in the API are separate layers. Running without root does not correct an excessive ClusterRoleBinding. A narrow RBAC role does not compensate for a privileged container with access to sensitive host directories. Both layers must be reviewed.

A reasonable starting point for Linux applications includes running as non-root, disabling privilege escalation, dropping unnecessary capabilities, and applying a seccomp profile. The root filesystem should be read-only when possible. A sample securityContext looks like this:

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL
  seccompProfile:
    type: RuntimeDefault

RBAC should prefer specific actions on required resources and limit scope to a namespace when possible. Applications that do not call the Kubernetes API should not automatically mount the service account token.

Where to place checks in the pipeline

A single large scan at the end of the release provides feedback too late. Secrets may already be on the server, installation scripts may have executed, and broad cloud roles may already be active. Checks should be distributed across stages where the team can still stop the corresponding action.

• Pre-commit: staged changes for secrets — hook blocks accidental addition of recognized passwords or tokens.
• Merge request: incoming commits, Dockerfile, Terraform, and Kubernetes manifests — team fixes findings before merge.
• Post-build: specific image for known vulnerabilities and secrets — result tied to digest of released artifact.
• Pre-deployment: Terraform plan and final manifests after Helm or Kustomize — policy validates actual parameters for the target environment.
• Admission to cluster: created and modified resources — forbidden settings blocked regardless of local checks.
• Post-release: running images, permissions, network access, and drift from templates — new vulnerabilities and manual changes do not remain unnoticed.

Work on the process should be validated with several safe test changes on a test project. An example secret string must trigger the secret search rule, a forbidden parameter must stop manifest checking, and an unwanted network connection must be refused on the test cluster. Rights to modify rules and exceptions must also be verified.