Update tests to dynamically match internal program limits - #6285

In commit 01e28e1 of PR git-lfs#5617 we
added a check to the "smudge with invalid pointer" test in our
"t/t-smudge.sh" shell script which exercises our "git lfs smudge"
filter command with invalid pointer data longer than the size of
the internal memory buffer used by the Spool() function in our
"tools" package.  Under these conditions, the Spool() function
should create a temporary file into which it can spool the
additional non-pointer data.

However, if we change the size of this internal buffer in the future,
we will need to remember to adjust the "smudge with invalid pointer"
test, or else its check of the "smudge" filter's use of a temporary
file may silently become ineffective.

In fact, we already have a test in our "t/t-clean.sh" shell script
which no longer creates test data of the appropriate size because
its initial version was written at a time when the size of the
read buffer used to decode Git LFS pointers was only 512 bytes,
and the test was never revised after the buffer size was increased
to 1024 bytes.  (We will document the history of this test's
development in a subsequent commit in this PR, at which time we
will also update the test.)

In order to help avoid these types of issues with our shell tests,
in a previous commit in this PR we revised our "lfs" package so
that the constant which sets the size of our pointer read buffer
is now exported with the name BlobSizeCutoff.

We now make a similar update to our "tools" package by revising the
name of the constant which sets the size of the buffer used by the
Spool() function from "memoryBufferLimit" to MemoryBufferLimit,
which allows the constant to be exported outside of the package.

In a subsequent commit in this PR we will then be able to add a
Go helper program to our test suite which outputs the value of this
constant.  This will allow us to also adjust the "smudge with
invalid pointer" test so it no longer creates data of a size
set by a hard-coded value but instead scales the length of the
data to the current value of the MemoryBufferLimit constant.

Note that we also correct one code comment where we inadvertently
used mismatched delimiters around a literal term.

In PR git-lfs#6011 we will add a test to our "t/t-filter-process.sh" shell
script with the intent that it should validate the revisions made in
the PR to our "clean" Git filter commands.  Specifically, the new
test creates the conditions described in this comment:

  git-lfs#4974 (comment)

To reproduce these conditions, the test first needs to create a file
larger than the size of the buffer we use to detect and decode
Git LFS pointers.

We also have a number of other tests in our shell suite which depend
on knowledge of the size of the read buffer, which at present is 1024
bytes and is set by the "blobSizeCutoff" constant in our "lfs" package.

At least one of these existing tests, the "clean pseudo pointer with
extra data" test in our "t/t-clean.sh" script, no longer creates test
data of the appropriate size because its initial version was written
at a time when the size of the read buffer used to decode Git LFS
pointers was only 512 bytes, and the test has never been appropriately
revised.

In order to help avoid these types of issues with our shell tests,
in a previous commit in this PR we revised our "lfs" package so that
the constant which sets the size of our pointer read buffer is now
exported with the name BlobSizeCutoff.

We can now introduce a new Go helper program named "lfstest-getlimit"
which when run with the command-line option --max-pointer-size will
output the value of the BlobSizeCutoff constant.

Our helper utility also supports three other possible command-line
options.  When the --max-bufio-scan-token-size option is provided,
the program will return the value of the MaxScanTokenSize constant
from the "bufio" package in the Go standard library.  We will be
able to make use of this value in the "prune doesn't hang on long
lines in diff" and "prune doesn't hang on long lines in stash diff"
tests in our "t/t-prune.sh" script, as they are intended to exercise
our "git lfs prune" command with files larger than the maximum token
size supported by the "bufio" package's Scanner interface.

Alternatively, when our new helper utility is run with the option
--max-pktline-len, it will output the maximum length of a Git packet
line, as specified by the MaxPacketLength constant from our
"git-lfs/pktline" Go package.  We will be able to make use of this
value in the "filter process: hash-object --stdin --path does not
hang" test in our "t/t-filter-process.sh" script, as it is intended
to exercise our "clean" filter with a file larger than the maximum
length of a single Git packet line.

Lastly, when the --max-spool-mem-buffer-size option is provided to
our new helper program, it will output the value of the
MemoryBufferLimit constant from our "tools" package.  We will be
able to make use of this value in the "smudge with invalid pointer"
test in our "t/t-smudge.sh" script, as it is designed to exercise
our "smudge" filter with input data larger than the size of the
internal memory buffer used to spool data by the Spool() function
in our "tools" package.

Since we first introduced the calc_oid() helper function to our
shell test suite in commit 54601c6
of PR git-lfs#676, we have used the "printf" shell built-in command to pass
the function's first parameter to a program which calculates the
SHA-256 hash of its input.  (Note that the specific program used
for this purpose depends on the operating system.)

However, because the calc_oid() function assigns its first parameter
to be the "format" argument of the "printf" built-in command, if the
parameter were to contain any backslash characters or format
specifications like "%d", the hash value calculated from the
argument's contents might not match the value we expect.

In a subsequent commit in this PR we intend to revise several of
our shell tests to make use of the calc_oid() function with
dynamically-generated data of various sizes.  This data should
not contain any special formatting characters like "\" or "%".

Nevertheless, we can guard against the possibility that such
characters might be inadvertently introduced in the future by using
"%s" as the "format" argument of the "printf" built-in command, and
then passing the calc_oid() function's first parameter as the second
argument of the "printf" command.  This guarantees that the
function's parameter will be interpolated in place of the "%s"
format specification in a literal, unmodified form.

We then update four of our shell tests that depend on the legacy
behaviour of the calc_oid() function to convert the character
sequence "\n" into an LF (linefeed) character.  In each of these
tests, we can simply use our calc_oid_file() helper function
to determine the SHA-256 hash of a given file's contents, instead
of passing a string which represents the contents of the file to
the calc_oid() function.

In a prior commit in this PR we introduced a new Go test helper
program named "lfstest-getlimit" which can return the value of
either the size of the read buffer used to decode Git LFS pointers
or the maximum length of a Git packet line.

We can now update a number of tests in our shell test suite to make
use of this helper utility.  Many of these tests previously relied on
hard-coded constants to create test data of various sizes to validate
the Git LFS client's ability to handle filter input data from Git
which fills or does not fill the client's pointer read buffer.

We rewrite one test in particular, the "clean pseudo pointer with
extra data" test in our "t/t-clean.sh" script, because its initial
version was written at a time when the size of the read buffer used
to decode Git LFS pointers was only 512 bytes, and the test has never
been appropriately revised.

The original version of this test was a Go test function named
TestCleanPointerWithWhitespaceAndExtra() that was added in commit
e09e5e1 of PR git-lfs#271.  The Go test
function was first rewritten as part of a shell test in commit
8b54aee of PR git-lfs#336, and then
converted into a separate shell test in commit
a251b2e of PR git-lfs#447.

Later, in commit f58db7f of PR git-lfs#684,
the size of the buffer used by the DecodeFrom() function in the "lfs"
package was changed from 512 bytes to the value specified by the
"blobSizeCutoff" constant.  At that point in time the constant was
already set to its current value of 1024 bytes.  However, the
"clean pseudo pointer with extra data" test was never updated to
reflect the larger size of the pointer read buffer.

We therefore now rewrite this test to retrieve the size of the
pointer read buffer using our new "lfstest-getlimit" utility, and
then create an invalid pointer larger than the size of the buffer.
Note that the test is intended in part to check that the client
ignores the fact that the initial portion of the input data appears
to be a valid Git LFS pointer.  The exact content of this initial
portion is not significant, however, so we change the value of the
"size" field in the invalid pointer from 1024 to 9999, which more
clearly indicates that this value is arbitrary and not related
to the size of the client's read buffer.

What is important for the soundness of the test, though, is that the
total length of the invalid pointer data exceeds the buffer's size.
To align with the original design of the test, we insert a large
number of linefeed characters after the initial portion of the test
data, which requires us to work around the fact that the shell will
strip trailing whitespace from a command substitution.  To avoid our
linefeed characters being removed from the value we assign to the
"fill" shell variable, we append the string "EOF", and then remove
this suffix when interpolating the variable's value into the final
version of the test data that we pass to the "git lfs clean" command.

To maintain consistency between our tests, we also update the
preceding "clean pseudo pointer" test in the "t/t-clean.sh" script to
change the arbitrary value of the "size" field in its invalid pointer
data from 1024 to 9999.  This test's invalid pointer data is not
intended to exceed the size of the client's read buffer, so we do
not need to invoke our new "lfstest-getlimit" helper utility.  The
data only needs to begin with an apparently valid set of pointer fields,
but as with the following test, the exact value of the "size" field is
not significant, which is more clearly indicated by the value 9999.

Both of these tests previously used hard-coded SHA-256 hash values
to check the output of the "git lfs clean" command, which should
simply process the invalid pointer data as a regular file and not
a Git LFS pointer.  Since we have changed the input data in both
tests, we need to recalculate the expected SHA-256 hashes.  While
we could continue to use hard-coded hash values, we instead make use
of our test helper functions to now calculate these hash values
dynamically.

We then update a number of other tests in several scripts to make
use of our new "lfstest-getlimit" helper utility.  All of these
tests previously used hard-coded values to create data of a specific
length in order to demonstrate that the Git LFS client does not
exhibit undesired behaviour when its input data fills or does not
fill a given internal buffer.

Note, though, that in the "malformed pointers" test we still have
one instance where we create a file of a specific size using a
hard-coded constant value.  In this instance, the test checks that
the Git LFS client can process invalid pointer data from Git which
exceeds the default capacity of a pipe, which is normally 64 kB,
at least on Linux and macOS.

We should be able to dynamically determine the capacity of a pipe
on Linux using the fcntl(2) system call and the F_GETPIPE_SZ operation,
and on Windows using the GetNamedPipeInfo() system call and the
"lpInBufferSize" and "lpOutBufferSize" parameters:

  https://man7.org/linux/man-pages/man2/F_GETPIPE_SZ.2const.html
  https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-getnamedpipeinfo#parameters

However, on macOS we would apparently have to experimentally feed
data into a pipe to try to determine the pipe's capacity.  This is
beyond the scope of the current PR, though, so for the present
time we leave this one instance of a hard-coded buffer size in
our shell tests.  We add a comment which explains the intent of this
check, however, as a reminder to revisit this issue in the future.

To help clarify the behaviour of the many tests where we simply need
to generate a file of a specific length, we now adopt the use of the
head(1) command to read a specific number of zero bytes from the
"/dev/zero" pseudo-device instead of using the dd(1) command as we
have done previously.  The result is the same in all cases, but the
"dd" command requires several somewhat obscure options, while the
"head" command only requires a single byte-count argument.

In addition to these changes to the tests, we also add or update
the comments in the tests.  In each case, we try to more fully
document the purpose of the test, with links to the original issues
and PRs for which the test was developed.

Finally, we revise the names of two of the tests in our
"t/t-filter-process.sh" script.  We rename the "filter process: add a
file with 1024 bytes" test to "filter process: non-pointer file of
maximum pointer size", since the size of the file created by the test
is now variable rather than fixed, and the intent of the test is
to check that the Git LFS client properly handles filter input which
happens to be exactly the same size as that of the pointer read
buffer.

In a previous commit in this PR we updated two tests in our
"t/t-filter-process.sh" shell test script to make use of the new
"lfstest-getlimit" helper utility we added in an earlier commit in
this PR.  In addition to updating the tests and adding comments to
them, we renamed one of the tests and revised both to make use of
the head(1) command instead of the dd(1) command to generate files
of specific sizes.

We now take the opportunity to further refine these tests in several
ways.  For example, because we added comments to each of the tests
with links to the original issues and PRs for which the tests were
developed, we can adjust the name the "filter process: non-pointer
file of maximum pointer size" test uses for its repository to reflect
the name of the test itself rather than a GitHub issue ID.

We also try to use more consistent variable names in the tests,
such as "contents_oid" rather than "oid", and in the "filter process:
hash-object --stdin --path does not hang" test in particular we try
to make use of the existing "contents" variable to output data
to the "git hash-object" command rather than repeat the same
bare text string in multiple instances.  As well, in one instance
in this test we can use the length of the "contents" variable instead
of a hard-coded constant value, and we add quotation marks around a
number of the test's command substitutions, following our current
idiom for shell tests.

Finally, in the last check of the "filter process: hash-object
--stdin --path does not hang" test we update the argument of the
"git hash-object" command's --path option from "third.dat" to
"large.dat" so that it corresponds with the name of the file whose
contents are piped to the "git hash-object" command.  This change
has no direct effect on the command's behaviour, since the Git LFS
filter entry in the ".gitattributes" file has a "*.dat" pattern and
so Git will invoke the "git lfs filter-process" command for any file
path matching that pattern.  However, our change helps to clarify the
purpose of the last check in the test and to distinguish it from
the preceding check where we deliberately use the name of a file
which does not exist as the argument of the --path option.