Millions of developers worldwide use git to manage software projects, but improper use can leave your systems vulnerable to attack.

Developers use the services from Companies like GitHub, GitLab and Atlassian - which have each created project management tools on top of the open-source git protocol - to iterate on software projects, keep track of changes, perform security scans, automate deployments and more. These companies are worth billions of dollars, at least partly because they have a combined user base of over 100 million registered users creating value with their products.

With this scale of use, security ramifications would be catastrophic if there is a problem with the underlying technology - or how it is used.

Hardcoded Secrets

In the infancy of a project, agile principles to iterate quickly and get stakeholder feedback early has a crucial benefit of ensuring development teams don't waste their time building the wrong product. However, with this in mind, sometimes teams misinterpret the idea of iterating quickly with the idea of cutting (security) corners and develop systems with hardcoded secrets embedded in the codebase.

For example, a developer team might add AWS access tokens into a project so they can save and retrieve files from S3 rather than setting up environment variables or other more secure mechanisms.

Cutting corners is not new, and companies are often eager to meet deadlines, so it is unfortunately common to see developers committing passwords, API keys, SSH keys and other secrets directly into code repositories, often with the good intention of cleaning things up later.

Notwithstanding the fact the cleaning up stage is often forgotten, the interim period can allow various stakeholders - not necessarily only developers - to have access to these hardcoded secrets and the opportunity for nefarious activity. It should be obvious that API keys to payment systems like PayPal and Stripe should be locked down as much as possible and only be accessible by key personnel and administrators - not in the codebase for every tester, business analyst and developer to exfiltrate.

Instead, organisations should ensure secrets are correctly handled from the outset of a project and injected into the runtime of a service with environment variables or with an integration with a system like AWS Secrets Manager.

Git Memory

The agile evangelist in you may be thinking that so long as you remember to remove those secrets from the code in the next iteration, everything will be okay again, and the team will be adhering to principles of continuous improvement. If so, you would be correct - but only if you update the project correctly and only if you consider those initial secrets to have already been compromised.

Most importantly, best security practices would recommend that any secrets that have been hardcoded into a repository - especially public repositories - be immediately revoked and rotated. There are many automated crawlers on the internet constantly searching for S3 access keys, CI/CD API tokens and many other secret values, so it's better to be paranoid about security and assume someone else has already copied your secret values.

In terms of removing the hardcoded secrets from the codebase, you will need to use a tool like the BFG Repo-Cleaner tool to correctly remove the secret from the latest version and the history of all tags and branches.

The reason for this precautionary measure to rewrite history is best demonstrated with the following screenshots on GitHub, showing hundreds of thousands of commits where developers were not aware that git remembers your secrets in the project history.

The following image shows a search for "removed password" as the git commit message across all public repositories (no longer possible on GitHub):

One of the first results shows the password and connection details to a MySQL database, presumably with a project relating to burgers, somehow:

The results showed dozens of public systems - databases, cameras, storage devices, etc. - and their access credentials in plain view. Of course, it's possible the credentials were no longer valid, but we would imagine a good percentage would have still worked if we had wanted to try them out.

Ironically, the git commit to remove the passwords with the commit messages of "removed password" actually highlighted the security flaw they were trying to overcome!

Importantly, this idea that git remembers your secrets applies to all git repositories and not just the public repositories shown in the screenshots. So if you have ever had any secrets hardcoded in your projects at your organisation, they will still exist in the git history, which is easy to look back through, and they might be worth cleaning up even now.

Best Practices Recap

1. If you have ever pushed code containing secrets to a git repository, you should consider those secrets compromised, and you should go about rotating them immediately.
2. It would also be good to clean up your repository history and remove all sensitive information with the simple BFG Repo-Cleaner tool. Alternatively, you can choose the advanced git filter-branch methods; both are described fully in GitHub's article on removing sensitive data from a repository. This will ensure your codebase removes all traces of any secrets that were ever there.

Finally, keep reading security tips to secure your organisation and your code.