When our go.mod file was introduced in commit
114e85c2002091eb415040923d872f8e4a4bc636 in PR #3208, the module
path chosen did not include a trailing /v2 component. However,
the Go modules specification now advises that module paths must
have a "major version suffix" which matches the release version.
We therefore add a /v2 suffix to our module path and all its
instances in import paths.
See also https://golang.org/ref/mod#major-version-suffixes for
details regarding the Go module system's major version suffix rule.
When we invoke the file helper to download files from a remote system,
we use the repository's temporary directory in order to make sure we can
quickly and easily rename files into the object store without the need
for a copy, which might be necessary if we used the system temporary
directory. In many cases, these files we generate are actually hard
links to the remote repository, which means we can cheaply and easily
copy files at maximum speed. Note, however, that these files don't look
like a normal object ID; instead, they have a name generated by Go's
temporary file code.
However, our current code causes a problem if, during the pull, a file
is checked out and Git causes it to be uselessly cleaned, which happens
in some cases. That's because our temporary file cleanup code will
remove all files in the temporary directory that don't look like normal
object ID components, and as a result, the clean operation can remove
files that are in use by the fetch.
Instead of immediately purging anything that looks like it's not a valid
object ID, let's wait an hour before pruning any temporary file unless
it's an object that we've already downloaded and verified is in our
object store.
We also need to do one more thing here, which is that we need to ignore
any file that lives in a directory younger than an hour old and adjust
our file helper to create a new directory for its temporary files. This
is important because if we create a hard link to an object in a remote
repository, it may be older than one hour, and we don't want to prune
those if we're not done yet.
Note that we use a sync.Map to make this as efficient as possible and
avoid the need for many additional stat calls. Ideally, we won't
actually see an increase in stat calls at all.
Now that we have a function which does path canonicalization correctly
on a variety of systems, let's use it in favor of filepath.EvalSymlinks.
We use this function to handle any path we know doesn't come from
outside Git LFS, since we know we'll never need to handle Cygwin paths
in these cases and it's more efficient not to invoke a subprocess if
it's not necessary.
The Go test suite adds remotes in the form of temporary bare Git
repositories, which lack a ".git" directory. These are then configured
with their local paths as the value of the "remote.origin.url" key,
which the EndpointFinder returns, but with "/.git" appended by the
EndpointFromLocalPath() function.
When the standalone transfer adapter attempts to chdir(2) to one of
these file paths in order to run the "git rev-parse --git-dir" command
when setting up a new handler, it obviously fails.
We can address this by simply removing any trailing "/.git" path
segment in the gitDirAtPath() function in the standalone adapter.
The "git rev-parse --git-dir" command will succeed in either case,
whether in a bare repository or a regular one.
This should permit any users who have Git LFS configurations with
custom adapters that depend on the EndpointFinder's current logic
to continue to work, while allowing our Go test suite to succeed
with its bare remote repositories.
One commonly requested feature for Git LFS is support for local files.
Currently, we tell users that they must use a standalone transfer
agent, which is true, but nobody has provided one yet. Since writing a
simple transfer agent is not very difficult, let's provide one
ourselves.
Introduce a basic standalone transfer agent, git lfs standalone-file,
that handles uploads and downloads. Add a default configuration required
for it to work, while still allowing users to override this
configuration if they have a preferred implementation that is more
featureful. We provide this as a transfer agent instead of built-in
because it avoids the complexity of adding a different code path to the
main codebase, but also serves as a demonstration of how to write a
standalone transfer agent for others who might want to do so, much
like Git demonstrates remote helpers using its HTTP helper.
