General

A Guide to Keeping Secrets out of Git Repositories

Published May 31, 2022

Updated Feb 13, 2023

6 min read

This article was written over 18 months ago and may contain information that is out of date. Some content may be relevant but please refer to the relevant official documentation or available resources for the latest information.

If you’ve been a developer for a while, then you hopefully know it is wise to keep secret information such as passwords and encryption keys outside of source control. If you didn’t know that, then surprise! Now you know.

Sometimes slip-ups do happen and a password ends up in a default config file or a new config file was not added to “.gitignore” and that same someone ran “git add .” and didn’t even notice it got committed. There should be protections in place no matter how diligent your programmers are since nobody is infallible, and the peace of mind is well worth it.

How can software know something is a secret?

It can’t know for sure, but it can make an educated guess. Secrets typically fit certain known patterns, or have higher entropy than other strings in your code and configuration files. A good scanner should check for strings that fit these patterns throughout your entire repository’s history, and raise anything suspicious to you.

Checking for Secrets in CI and CD

When it comes to automatically checking for secrets in your code, you have quite an array of options. To keep this article brief, I am just going to cover a few tools, and which tools you use may depend on your repository host.

GitHub

If you’re using GitHub for your project and your repository is either public or you use GitHub Enterprise Cloud, then GitHub will automatically scan the code you upload for secrets. GitHub’s solution is special because they have partnered with several different companies to allow for automatic revocation of secrets pushed to the repo. See the following excerpt from GitHub’s secret scanner documentation:

When you make a repository public, or push changes to a public repository, GitHub always scans the code for secrets that match partner patterns. If secret scanning detects a potential secret, we notify the service provider who issued the secret. The service provider validates the string and then decides whether they should revoke the secret, issue a new secret, or contact you directly. Their action will depend on the associated risks to you or them.

Well, that’s nice now isn’t it? If you’re curious if the services you use are partnered with GitHub so that their secrets can be scanned for, you can view the full list here. Just keep in mind that this functionality is only available for public repositories and private repositories using GitHub Enterprise Cloud with a “GitHub Advanced Security” license.

GitLab

GitLab, like GitHub, has secret detection as well. GitLab uses Gitleaks for their secret detection. This is a well documented tool whose source code is freely available. The capabilities of secret detection in GitLab does vary based on your tier, though.

You will have to use GitLab Ultimate to view detected secrets in the pipeline, and merge request sections for example. You can still use the scanner in free and premium versions, but it isn’t nearly as integrated as it is in the ultimate version.

Gitleaks

We mentioned that GitLab uses Gitleaks, but you aren’t just limited to using it with GitLab! Since Gitleaks is open source, that means you can use it with other providers such as GitHub, and even run it locally on your own system. It is also very easy to set up either as a CI job, or if you need to run it locally.

Scanning for Secrets using a CI Job

For GitHub you can simply use this action made by the author of Gitleaks. In this case Gitleaks is helpful if you’re using private repositories on GitHub without GitHub Enterprise Cloud. It is fully configurable with the action as it allows you to specify a custom .gitleaks.toml file. This is optional of course, and the default might work fine for you.

Checking for Secrets in a Pre-Commit Hook

There are a couple of ways to accomplish setting up the hook. A pre-commit script is available on the Gitleaks GitHub that will run Gitleaks on your staged files before you commit. Your commit will be stopped if any secrets are detected. This script can simply be copied into your .git/hooks/ directory. It does require that Gitleaks is installed and in your $PATH, however.

The other method involves using the pre-commit utility. It will assist with installing Gitleaks automatically for any developers that clone the repository and it can also assist with installing the hooks for the first time as well. Using the pre-commit tool might make more sense if you want to ensure other linters and checkers run, and you don’t want to have developers juggle installing everything themselves.

A Good Code Review Process Goes a Long Way

Although automated tooling for identifying secrets in code works well, it’s still good to keep an eye out for them when reviewing code. Automated scanning tools, as I mentioned earlier in the article, work great and you should definitely use them. However, they aren’t perfect. These tools look for sets of patterns and strings with high entropy, but not all secrets fit these criteria.

Knowing that even with the best scanning tools it’s still possible secrets could sneak through, it’s easier to understand why it is important to also have a good code review process to catch these issues. Also remember that committing secrets isn’t the only thing you should be worried about. You should have others review your work to help mitigate the chances that your changes could introduce new security vulnerabilities in the code as well!

Oh no, there’s already a secret in my Git history!

If you already have secrets in your repository, and they’re pushed to a main branch, not all hope is lost. Before we get into methods of removing the secrets, I have a massive disclaimer I should get out of the way. The only way to truly remove secrets from your repository is to rewrite your git history. This is a destructive operation, and will require developers to re-pull branches and cherry-pick changes from their local branches if applicable.

Did you read the disclaimer? Good, we can discuss methods then. What method we do depends on how and when the secret made it into the repository.

The Secret is in a Single Branch

If a secret made it into a feature branch by mistake, then you could simply initiate an interactive rebase to remove it and force push that branch to the remote. It should be noted that this is only effective if no other branches are based off of your branch, and if it isn’t tagged.

Let’s say, for example, you push your code to the remote and a CI job identifies a secret after your push. At this point there would be nobody else using your branch and it shouldn’t be tagged, so this is the perfect opportunity to just rewrite the commit that triggered the CI failure. If your commit hash is let’s say 09fac8dbfd27bd9b4d23a00eb648aa751789536d, then these are the first command you would have to execute to begin cleaning up your branch’s history:

$ git rebase --interactive 09fac8dbfd27bd9b4d23a00eb648aa751789536d^

Note the caret at the end of the SHA1 commit hash. This is vitally important to include as we need to rebase to the commit prior to the commit introducing the secret. The gist is that we’re going to return back to that point in time, and prevent the secret from ever being added in the first place.

Git will now open your default command-line text editor and ask you how you want to execute your rebase. Find the line referencing the problematic commit and replace pick with edit on that line. If you save the file and quit you then should now find yourself at the commit where the secret was introduced. From here you can remove the secret, stage the affected files, and then execute the following commands:

$ git commit --all --amend --no-edit

And if that’s successful:

$ git rebase --continue

Then after that the history should be successfully rewritten to not include the secret locally. If you pushed your branch already, then you’ll need to get these changes pushed to the remote as well. You can do this with a force push using the -f flag like so:

$ git push origin <your_branch_name> -f

Now you should be set! At this point, you can get to fixing up your code to pull the secret some other way that doesn’t include config files or strings hard-coded inside of your codebase.

The Secrets are in Main Already…

If your secrets are present in many branches, like tagged versions or your main branch, then things get a little more complicated. There’s more than one way to handle this situation, but I am going to cover only one. Just revoke the secret.

In this scenario, you should revoke the secret and issue a new one. If you do this, then it doesn’t matter that the old secret is still in the repository history because it will be entirely useless! How this is done of course depends on what service the secret was issued from.

There is also the added benefit in that anyone that has cloned your repo with the secret will also be unable to use it. Simply rewriting history doesn’t matter if an adversary has already downloaded it before you deleted it.

It’s important to note that this isn’t always great if, for example, the secret is used in multiple projects. You will need to ensure your revoked secret is replaced everywhere before you actually revoke it or else you may experience downtime.

Conclusion

With scanning tools becoming more accessible, there are fewer and fewer reasons to not use them. Secret scanning is especially important for public repositories, but it is also useful for private repositories where a compromised developer account can access secrets and wreak havoc.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

Jamie Kuppens
I’m a software engineer with an interest in web development and some more esoteric things like emulator development.
@Reshurum @Reshurum

Git Reflog: A Guide to Recovering Lost Commits

Losing data can be very frustrating. Sometimes data is lost because of hardware dying, but other times it’s done by mistake. Thankfully, Git has tools that can assist with the latter case at least. In this article, I will demonstrate how one can use the git-reflog tool to recover lost code and commits. What is Reflog? Whenever you add data to your local Git repository or perform destructive operations, Git keeps track of all these using reference logs, also known as reflogs. These log entries contain a SHA-1 hash of the commit associated with it and any references, or refs for short. Refs themselves are branch names, tags, and symbolic refs like HEAD, which is always pointing to the ref or commit id that’s currently checked out. These reflogs can prove very useful in assisting with data recovery against a Git repository if some code is lost in a destructive operation. Reflog records contain data such as the SHA-1 hash that HEAD was pointing to when an operation was performed, and a description of the operation that was performed as well. Here is an example of what a reflog might look like: ` The first part 956eb2f is the commit hash of the currently checked out commit when this entry was added to the reflog. If a ref currently exists in the repo that points to the commit id, such as the branch-prefix/v2-1-4 branch in this case, then those refs will be printed alongside the commit id in the reflog entry. It should be noted that the actual refs themselves are not always stored in the entry, but are instead inferred by Git from the commit id in the entry when dumping the reflog. This means that if we were to remove the branch named branch-prefix/v2-1-4, it would no longer appear in the reflog entry here. There’s also a HEAD part as well. This just tells us that HEAD is currently pointing to the commit id in the entry. If we were to navigate to a different branch such as main, then the HEAD -> section would disappear from that specific entry. The HEAD@{n} section is just an index that specifies where HEAD was n moves ago. In this example, it is zero, which means that is where HEAD currently is. Finally what follows is a text description of the operation that was performed. In this case, it was just a commit. Descriptions for supported operations include but are not limited to commit, pull, checkout, reset, rebase, and squash. Basic Usage Running git reflog with no other arguments or git reflog show will give you a list of records that show when the tips of branches and other references in the repository have been updated. It will also be in the order that the operations were done. The output for a fresh repository with an initial commit will look something like this. ` Now let’s create a new branch called feature with git switch -c feature and then commit some changes. Doing this will add a couple of entries to the reflog. One for the checkout of the branch, and one for committing some changes. ` This log will continue to grow as we perform more operations that write data to git. A Rebase Gone Wrong Let’s do something slightly more complex. We’re going to make some changes to main and then rebase our feature branch on top of it. This is the current history once a few more commits are added. ` And this is what main looks like: ` After doing a git rebase main while checked into the feature branch, let’s say some merge conflicts got resolved incorrectly and some code was accidentally lost. A Git log after doing such a rebase might look something like this. ` Fun fact: if the contents of a commit are not used after a rebase between the tip of the branch and the merge base, Git will discard those commits from the active branch after the rebase is concluded. In this example, I entirely discarded the contents of two commits “by mistake”, and this resulted in Git discarding them from the current branch. Alright. So we lost some code from some commits, and in this case, even the commits themselves. So how do we get them back as they’re in neither the main branch nor the feature branch? Reflog to the Rescue Although our commits are inaccessible on all of our branches, Git did not actually delete them. If we look at the output of git reflog, we will see the following entries detailing all of the changes we’ve made to the repository up till this point: ` This can look like a bit much. But we can see that the latest commit on our feature branch before the rebase reads 138afbf HEAD@{6}: commit: here's some more. The SHA1 associated with this entry is still being stored in Git and we can get back to it by using git-reset. In this case, we can run git reset --hard 138afbf. However, git reset --hard ORIG_HEAD also works. The ORIG_HEAD in the latter command is a special variable that indicates the last place of the HEAD since the last drastic operation, which includes but is not limited to: merging and rebasing. So if we run either of those commands, we’ll get output saying HEAD is now at 138afbf here's some more and our git log for the feature branch should look like the following. ` Any code that was accidentally removed should now be accessible once again! Now the rebase can be attempted again. Reflog Pruning and Garbage Collection One thing to keep in mind is that the reflog is not permanent. It is subject to garbage collection by Git on occasion. In reality, this isn’t a big deal since most uses of reflog will be against records that were created recently. By default, reflog records are set to expire after 90 days. The duration of this can be controlled via the gc.reflogExpire key in your git config. Once reflog records are expired, they then become eligible for removal by git-gc. git gc can be invoked manually, but it usually isn’t. git pull, git merge, git rebase and git commit are all examples of commands that will trigger git gc to run behind the scenes. I will abstain from going into detail about git gc as that would be deserving of its own article, but it’s important to know about in the context of git reflog as it does have an effect on it. Conclusion git reflog is a very helpful tool that allows one to recover lost code and commits when used in conjunction with git reset. We learned how to use git reflog to view changes made to a repository since we’ve cloned it, and to undo a bad rebase to recover some lost commits....

Nov 14, 2022

5 mins

Git

"How do I undo my most recent commit?" - Mastering the git reset command

"How do I undo my most recent commit?" - Mastering the git reset command Git is a powerful version control system, but even experienced developers sometimes make mistakes. Whether you've committed changes to the wrong branch, made a typo in your commit message, or simply want to undo recent changes, the git reset command is your go-to solution. In this post, we'll explore how to use git reset to manage your commit history effectively. The Basics of git reset At its core, git reset moves the HEAD and current branch pointer to a specified commit, effectively "resetting" your working directory to a previous state. The basic syntax is: ` Where specifies how to handle changes in the working directory and staging area, and is the commit you want to reset to. Soft vs. Hard Reset: Understanding the Difference Git reset has two main modes: --soft and --hard. Let's explore each: Soft Reset (--soft) A soft reset moves your HEAD to the specified commit but keeps your changes staged. This is useful when you want to undo commits but keep the changes for recommitting. Example: ` This command undoes the last commit, keeping the changes staged. You can now make additional changes or recommit with a new message. Hard Reset (--hard) A hard reset is more drastic. It moves your HEAD to the specified commit and discards all changes after that commit. Example: ` This command undoes the last commit and discards all changes. Be cautious with this command, as it can lead to data loss! Specifying a Commit Hash Instead of using relative references like HEAD~1, you can specify an exact commit hash: ` This resets to the commit with hash 1a2b3c4, keeping changes staged. Using HEAD~n Notation The HEAD~n notation allows you to specify how many commits back you want to reset. For example: ` This resets to three commits before the current HEAD, discarding all changes. Reverting Pushed Changes If you've already pushed commits you want to undo, you should use git revert instead of git reset to avoid rewriting public history. However, if you must use git reset on a shared branch, you'll need to force push: ` Be extremely cautious with force pushing, as it can cause issues for other developers. Generic Usage of git reset git reset can also be used to unstage changes: ` This removes file.txt from the staging area without changing its contents. Conclusion The git reset command is a powerful tool for managing your Git history. Whether you're undoing recent commits, unstaging changes, or completely resetting your working directory, understanding the different modes of git reset can help you navigate tricky situations in your Git workflow. Remember to use git reset with caution, especially when working with shared repositories. When in doubt, create a backup branch before performing any reset operations....

Oct 18, 2024

2 mins

Git

How to Implement Soft Delete with Prisma using Partial Indexes

Most APIs and applications that allow you to manage content have some form of deletion functionality. Sometimes you may want to make something recoverable after it’s been deleted, or perform the actual deletion later, and soft deletion is a way to accomplish this. Entities in your database may also have unique identifiers such as slugs associated with them as well, and when you delete things you typically expect these identifiers to become available for use again. In the case of soft deletes though your deleted entities will still be in the database! If you’re using Postgres or SQLite you can get the best of both worlds by using partial unique indexes, which will in the case of soft deletes would only add slugs on undeleted records to the index; however if you’re connecting a partial index compatible database with Prisma then everything isn’t rainbows and sunshine when it comes to partial indexes. A Simple Soft Delete Implementation Before getting into the issue with partial unique indexes, let’s implement soft delete in Prisma using middleware. The following example is adapted from the Prisma documentation regarding soft deletes, though with some slight alterations for convenience. ` The notable change here is this version could be extended to work with multiple models fairly easily by modifying the MODELS_SUPPORTING_SOFT_DELETE constant, and we also handle the throwable variants of the find actions as well. The latter change requires we create a custom type named ExpandedPrismaAction with additional actions as the imported Prisma types don’t have these actions for some reason even though they are encountered in actual usage. Anything specified in the model's constant is assumed to have a field named deletedAt that is nullable. In this example specifically it’s a timestamp field since it’s nice to know when something has been deleted, but it could also be a boolean if you so choose. With that done, soft deletion is transparently implemented, at least insofar as select queries go. If you plan on blindly running update and delete queries as well, then you will need to add middleware for those actions as well. The Problem So as stated before we plan on using unique slugs with our model, in this case slugs for posts, and to free slugs for usage later we need to exclude them from the unique index without deleting the post records outright. Unfortunately, at the time of writing this article Prisma doesn’t allow you to define partial indexes in your schema when defining unique indexes using @@unique inside of your schema. Thankfully, there is a way to workaround this issue. The Solution To workaround this issue you can manage your unique indexes outside of the Prisma schema, though it does come with its own pitfalls that I will cover later. This can be done by creating partial indexes in a migration file instead of defining them in the schema file. It should be mentioned that if you do this then you should ensure a conflicting index is not defined in the schema, or else Prisma will try to overwrite the other index. First let’s create a basic schema with the following contents defining posts: ` In the above schema I’ve left a comment that mentions where a partial unique index definition would go if it was supported. The problem with keeping the definition solely in the migration is that it makes it difficult to discover, and also doesn’t explain the reason why it’s there. Now that we have a Post model with a deletedAt column, create an empty migration and do the following: ` Then the contents of the newly created migration file will be as follows: ` With that all initialized and the middleware setup, and router handler defined, everything should work! Check out the full example on GitHub as I don’t show the full implementation of the example API in this article. The route handlers don’t look any different than you would expect as the middleware handles all of the heavy lifting for us. It is worth noting that DELETE handlers do look different with soft-delete as we’re not intercepting delete calls to Prisma. This is how the implementation of the deletion looks for this example: ` Caveats So there are some caveats to implementing partial delete in this way. The first being that the schema doesn’t reflect the state of the database 1:1. Thankfully, Prisma doesn’t care about untracked indexes existing, but knowing the index exists requires looking in the migration files instead of the schema. This is worked around by leaving a comment, but it still isn’t great. Another downside is that upsert calls do not work with the Prisma client. Making upserts with Prisma requires a unique key, but since Prisma doesn’t track this unique key the typings needed to call upsert do not exist. This can be worked around by making raw queries without the query builder. Summary I hope this article has helped you ease the integration of soft deletes into your application. You can find a full example on → GitHub ← that includes all of the above code snippets, a buildable project, and instructions on how to get it all working....

Feb 2, 2024

4 mins

Prisma

How Nuxt Studio is Redefining Developer & User Experience with Baptiste Leproux & Ferdinand Coumau

In this episode of the Modern Web Podcast, Danny Thompson sits down with Ferdinand Como and Baptiste Leproux from Nuxt Labs to uncover the story behind Nuxt Studio, a tool that's transforming how developers and non-technical users interact with Nuxt applications. Ferdinand and Baptiste share how Nuxt Studio bridges the gap between developer customization and user-friendly content management. Built to empower agencies, freelancers, and their clients, Nuxt Studio combines powerful features like live previews, Vue component integration, and schema-driven forms to make managing content seamless. The conversation also explores the broader mission of Nuxt Labs—building sustainable open-source tools that enhance developer experience and meet real-world needs. With insights into the future of Nuxt Studio and its potential to scale for larger organizations, this episode is a must-listen for anyone passionate about innovation in web development. Chapters 1. Introduction and Setting the Stage (00:00:00) 2. The Vision Behind Nuxt Studio (00:03:10) 3. Nuxt Studio’s Core Features (00:08:45) 4. Challenges in Building Nuxt Studio (00:16:20) 5. Target Audience and Use Cases (00:22:35) 6. Sustainability in Open Source (00:29:00) 7. The Future of Nuxt Studio (00:35:10) 8. Nuxt Studio’s Role in the Nuxt Ecosystem (00:42:30) 9. Closing Thoughts and What’s Next (00:48:00) 10. Sponsor Shoutout and Wrap-Up (00:53:20) Follow Baptiste Leproux and Ferdinand Coumau Baptiste Twitter: https://x.com/_larbish Ferdinand Twitter: https://x.com/CoumauFerdinand Baptiste Linkedin: https://www.linkedin.com/in/baptiste-leproux-618842b0/ Ferdinand Linkedin: https://www.linkedin.com/in/ferdinand-coumau-nuxt/ Sponsored by This Dot: thisdot.co...

Nov 20, 2024

2 mins

VueNuxtJS

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

A Guide to Keeping Secrets out of Git Repositories

How can software know something is a secret?

Checking for Secrets in CI and CD

GitHub

GitLab

Gitleaks

Scanning for Secrets using a CI Job

Checking for Secrets in a Pre-Commit Hook

A Good Code Review Process Goes a Long Way

Oh no, there’s already a secret in my Git history!

The Secret is in a Single Branch

The Secrets are in Main Already…

Conclusion

Jamie Kuppens

You might also like

Git Reflog: A Guide to Recovering Lost Commits

"How do I undo my most recent commit?" - Mastering the git reset command

How to Implement Soft Delete with Prisma using Partial Indexes

How Nuxt Studio is Redefining Developer & User Experience with Baptiste Leproux & Ferdinand Coumau

Let's innovate together!

You might also like

Git Reflog: A Guide to Recovering Lost Commits

"How do I undo my most recent commit?" - Mastering the git reset command

How to Implement Soft Delete with Prisma using Partial Indexes

How Nuxt Studio is Redefining Developer & User Experience with Baptiste Leproux & Ferdinand Coumau