Repository navigation
Support for shallow repositories #3058
Description
git can write a file .git/shallow to indicate that the history is cut off at particular commits. We currently do not read this at all and thus libgit2 will regularly fail to work on these repositories with error messages about failing to find objects.
The typicall way such a repository is created is by the use of git clone --depth N. We do not support this option to clone either, as we do not support the depth negotiation in the protocol.
In order to provide support we would have to make sure to check against the list in .git/shallow whether we're at the end of the history we should expect to be at, and then not try to walk further back or ignore errors when looking up parents.
There are a couple of issues which ask questions about this, I will close them and redirect them here to have a cleaner position from which to start.
Activity
arthurschreiber commentedon Apr 18, 2015
Maybe this is super naive, but couldn't support for this be added by simply automatically hiding all commits that are in this list when creating a new
revwalkinstance?carlosmn commentedon Apr 19, 2015
Something like that might indeed be enough for the git-log use-case, but we still need to figure out how to make that work with merge-base.
The network layer also needs to know this because we need to make the server aware that we're on a shallow repo so it doesn't try to re-use objects we don't have, and push needs to make sure we don't attempt to perform an impossible upload.
briansoule commentedon Apr 28, 2015
+1 FWIW
aral commentedon Jun 24, 2016
Just ran into #1430, which led me here. What’s the current status on this? Has anyone managed to get shallow repositories to work with fetches? Thanks!
Re: use case: Using git for differential updates to data in iOS app. A shallow clone results in a ~50MB app; a full clone in ~150MB app (and, of course, growing).
ethomson commentedon Jun 25, 2016
@aral No, there has been no update here - this is not currently supported, nor is it a high priority feature for us to implement, I'm afraid.
aral commentedon Jun 25, 2016
@ethomson Understood. Thanks.
Remove git2go
Qix- commentedon Feb 2, 2018
🤷♂️ bump and +1 here.
getify commentedon Feb 4, 2018
My use case: a huge github repo (more huge in terms of history than in terms of file sizes), but I only need to append to it with a single commit. To avoid the big (and growing) working directory, I just do a bare clone, and then use the API (via nodegit, actually) to create and commit the file, and push.
But the repository also has a long history (and growing, by a few thousand commits per day). So over time, all those commits I'm doing just chew up space with all that local git history. Every 20,000 commits or so, I'd like to just clear out the repository and re-clone it shallow (depth=1), because I only need the very last commit to add a new one.
Of course, I can do the clean/re-clone via a cron job, but the problem is that coordinating between a cron job and any running node process (to ensure I don't clobber in progress commit work) is much uglier/hacky than I'd like it to be. :(
pks-t commentedon Feb 8, 2018
Just to clarify: there really is no need to provide usecases for this. It is definitly a thing we want included in libgit2, the only problem is that we lack the time to do so. Unfortunately, we are already fully loaded maintaining the current state of affairs with libgit2 with the little spare time we have for it. Because of this, it's quite hard to find time to even review merge requests implementing bigger features such as this one here.
I'm really feeling sorry for it, but that's unfortunately how it is right now.
Move from nodegit to git cmd
painhardcore commentedon Apr 25, 2019
Maybe this year we can make it happen?
Qix- commentedon May 11, 2019
Hi. I'm back a year later. Kind of strange this is considered low priority. Shallow clones are what keep tooling quick.
painhardcore commentedon May 12, 2019
@pks-t can we make this as high priority?
dannyzhan commentedon Aug 15, 2019
I think shallow clone support is a very important feature too.
Not sure why it's put as low priority.
RiadGahlouz commentedon Oct 1, 2019
Please make this a high priority since it may bring huge optimizations for certain use-cases (which are fairly common when developing external tools)
normanr commentedon Oct 15, 2019
Instead of requiring supporting shallow repos for all operations, how about supporting them incrementally, i.e. get walks working, then merge, then network, etc, etc?
Requiring support (this is probably just my perception) for everything-at-once is hard for contributors to help out with.
Thoughts?
normanr commentedon Oct 15, 2019
It looks like #5254 may be bringing most(?) of the required changes to make this work?
pks-t commentedon Oct 17, 2019
Indeed, it does. And in fact, it does exactly what you're proposing with supporting shallow clones incrementally with the revwalks stuff first. Networking code has been implemented by @tiennou already, but will be reviewed separately after the first part has been merged.
QiBaobin commentedon May 13, 2020
My use case is that I can't push in a shallow cloned repo.
painhardcore commentedon Jun 27, 2020
Soon we can celebrate 5 years anniversary of this issue. Have we can make something happened :)
Auto merge of #10079 - dtolnay-contrib:fetch, r=ehuss
omniproc commentedon Oct 10, 2022
adding myself to the list of ppl. interested in this feature here. I see a lot of depending work merged already. does anyone track what's left to do until this can be implemented?
lrm29 commentedon Oct 10, 2022
This pull request (which itself builds on others) has an implementation that allows shallow cloning as well as unshallowing: #6396
tiagobento commentedon Feb 17, 2023
Is there news about this work somewhere else? I'm watching this issue and the PR (#6396) closely, but seems like this has been paused for a while. Is there something blocking this work to move forward?
I don't want to sound pushy or anything, I'm just very interested in this feature and unfortunately I don't have the knowledge to help..
Qix- commentedon May 9, 2023
Shallow cloning was just landed into
main! Go get it while it's hot!