Simplify object storage directory traversals - #6223

This PR simplifies the functions used by the Git LFS to traverse its internal temporary and object storage directories, as there is no need for these functions to check for Git submodules or .git directories.

This PR will be most easily reviewed on a commit-by-commit basis.

Background

In PR #1616 we added a FastWalkGitRepo() function to our tools package for the purpose of improving the performance of the git lfs track command. Prior to this, the git lfs track command used the Walk() function of the filepath package from the Go standard library to traverse the contents of the current Git working tree and locate all .gitattributes files.

Using the FastWalkGitRepo() function made searches for .gitattributes files in large repositories faster for three reasons. First, the functions called by the FastWalkGitRepo() function did not sort the entries in each directory, unlike the filepath package's Walk() function. Second, our functions ignored all .git directories and did not traverse into them. And third, our functions also ignored all directories entries which matched the patterns found in any .gitignore files.

Later, in PRs #1870 and #2689, we expanded the number of callers of the FastWalkGitRepo() function. In particular, in PR #2689 we began to make use of the function in our git lfs prune command to locate all the object files in our local storage directories, which by default are located within the .git/lfs/objects directory. As well, with this PR we began to use the FastWalkGitRepo() function during the final phase of all Git LFS commands where we find and delete any stale temporary files from the directory where we store such files, which by default is the .git/lfs/tmp directory.

PR #2689 introduced our fs package, whose Filesystem structure's EachObject() and cleanupTmp() methods both called the FastWalkGitRepo() function, passing the paths to the local object storage directory and the local temporary file storage directory, respectively.

We then added a FastWalkGitRepoAll() function to our fs package in PR #3190. This function operated in a similar fashion as the FastWalkGitRepo() function but did not read .gitignore files and so also did not skip directory entries matching any patterns found in those files.

Next, in PR #3686, we updated the internal implementation of the FastWalkGitRepo() and FastWalkGitRepoAll() functions to avoid entering submodules when traversing a Git working tree. To make this change, we added a check to the fastWalker structure's Walk() method so it would return immediately when processing a directory if the directory contained an entry named .git, unless the directory was the root of the working tree.

Then in commit 83d7f76 of PR #3823 we first implemented the NewLsFiles() function in our git package, which executes a git ls-files command and returns the list of files it outputs. As well, we updated the findAttributeFiles() function of our git package and the fixFileWriteFlags() function in our locking package to both call the NewLsFiles() function instead of calling either the FastWalkGitRepo() or FastWalkGitRepoAll() functions.

Since these were the only instances where we actually used the FastWalkGitRepo() or FastWalkGitRepoAll() functions to traverse a Git working tree, in the same commit of PR #3823 we removed the FastWalkGitRepo() function and renamed the FastWalkGitRepoAll() function to FastWalkDir(). We also simplified some of the internal functions called by the FastWalkDir() function because they no longer needed to detect or parse .gitignore files, or skip directory entries matching the patterns from those files.

Although the two remaining use cases for the FastWalkDir() function also did not need the function to detect and skip submodules or directories named .git, the logic to do so was retained in the internal functions of the tools package.

Specifically, the fastWalkWithExcludeFiles() function, which is called by the FastWalkDir() function, establishes two file path filters with the patterns .git and **/.git, which are then passed down to the Walk() method of the fastWalker structure. That method checks whether the current directory entry's name matches either of the filter patterns, and if it does returns immediately without processing the entry any further. In addition, the Walk() method still also performs the check added in PR #3686 to try to avoid traversing into Git submodules.

Current State

As noted above, neither of the two remaining callers of the FastWalkDir() function require these checks for .git directory entries and for submodules, because they only need the function to traverse directory hierarchies which the Git LFS client has created, and which are specifically not Git working trees.

The cleanupTmp() method of the Filesystem structure in our fs package uses the FastWalkDir() function to find stale files within the local temporary file storage directory. The EachObject() method of the same structure, meanwhile, uses the FastWalkDir() function to invoke a callback function for each object file in the local object storage directory.

By default, these local storage directories are the .git/lfs/tmp and .git/lfs/objects directories. If the "lfs.storage" configuration option is set to a relative path, then these directories will be located somewhere within the ".git" directory, while if the option is set to an absolute path, our local storage directories will be located under that arbitrary location.

In all these cases the Git LFS client creates and manages these local storage directories, so we can expect them not to contain .git directories or submodules. This is true even when a user has configured the lfs.storage option with an absolute path, since the client is still responsible for creating and managing tmp and objects directories within that arbitrary location.

Simplifications

We therefore now simplify the FastWalkDir() function and the internal functions it invokes by removing the unnecessary checks for submodules and for directory entries named .git.

First, we update the fastWalkWithExcludeFiles() function so that does not initialize any file path filters, and we rename the function to fastWalkDir().

Next, we remove the Walk() method's excludePaths parameter, and alter the method so that it no longer skips directory entries based on whether their names match the patterns in that parameter's file path filters.

We also eliminate the check in the Walk() method for directories containing .git directory entries, as this check's only purpose was to skip Git submodules within a working tree.

As well, we revise the code comments relating to all of these functions and methods to reflect their new names and their simplified behaviours.

To ensure that the variables and parameters in these functions accurately represent the functions' current purpose and implementation, we replace the term rootDir with the simple termdir because the FastWalkDir() function does not specifically start from the root of a Git working tree as its predecessor function FastWalkGitRepo() always did (at least, as that function always did when it was first introduced in PR #1616).

For the same reason, we rename the isRoot parameter of the fastWalker structure's Walk() method to the more generic isInitialDir, since any directory may be used as the initial starting point for a directory traversal by that method.

We also change the name of the method's workDir parameter to simply dir because the method is no longer used to only traverse Git working trees, and we likewise rename the parentWorkDir and childWorkDir variables to just parentDir and childDir.

Testing

The cleans only temp files and directories older than an hour test in our t/t-tempfile.sh shell test script verifies the behaviour of cleanupTmp() function, which employs the FastWalkDir() function, while the TestFastWalkBasic() test function in our Go test suite directly exercises the fastWalkDir() internal function. Both tests thus provide some assurance that our changes in this commit have not introduced any unexpected regressions.