In 2.46, Git will support multistage authentication from credential
helpers. This will allow Git and Git LFS to implement functionality
like NTLM and Kerberos in the credential helper, which lets this
functionality to work even if Git LFS doesn't support it natively.
This requires two separate pieces of data. First, it involves a
`state[]` field, which each credential helper can add to keep track of
state. Second, it allows the usage of a boolean `continue` field, which
indicates that the response is multistage and this is not the final
stage.
In order to make this work, we adjust a few things. First, we advertise
the `state` capability. Additionally, we save and pass back the
`state[]` fields that the credential helper may send to us. We also
don't change the authentication scheme if the helper told us that this
was a multistage response. Finally, we add a check to avoid a
credential helper getting stuck in an infinite loop if it keeps handing
back the same credentials.
Now that our test credential helper and test server are capable of
supporting the new authtype and credential fields, let's pass the
appropriate capability to the helper. We use the fields if they exist,
and pass them back to the `approve` call to the helper on success, along
with the capability.
If these fields don't both exist, we fall back to the username and
password fields. This allows us to do thing such as query different
accounts with the username if one is specified in the URL, but get the
credentials back in a format that contains a non-Basic response.
Since we now support more than just Basic and Negotiate, let's make sure
we iterate through all of the supported authentication schemes, not just
those two. This allows the credential helper to support multiple
schemes.
The io/ioutil package has been deprecated as of Go 1.16 [1]. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.
[1]: https://golang.org/doc/go1.16#ioutil
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Right now, we call `GitDir` to find the Git directory for our repository
every time we look up an endpoint. However, we _might_ already have
this data in our Git configuration handler, so let's use that if
possible so we don't need to spawn a process every time. If it's not,
then cache the data so that we spawn at most one process.
When we're locking or unlocking multiple times, we may end up looking up
the endpoint a few times. This is very cheap if we don't need to invoke
Git. However, as written right now, we call Git every time we validate
a remote, which is wasteful.
To make this more efficient, let's compute the remote list once and
cache it to avoid spawning a process. Then, we can use the cached value
every time we need to validate a remote.
Some people have very slow SSH round trips and the extra time to attempt
a failed git-lfs-transfer operation is bothersome. Let's add an option
to allow users to use either the pure SSH protocol or the older hybrid
protocol based on the URL in question to let them choose the right value
if they know what it should be.
When we compute the contents of an Endpoint struct, we don't preserve
the URL from which we computed the value in the first place. For
example, if we use an SSH URL, we don't preserve the SSH URL anywhere
for further use. Let's do so, since we'll use it in a future commit.
Note that for anything but an SSH URL, this is the same value as the URL
field.
In some cases, we might end up with an entry that's missing either a
username or password, and in such a case, we wouldn't want to panic,
like we do now. Solve this by using the function `FirstEntryForKey` to
look up the value and return an empty string if none is present.
When we want to handle a local path, we must rewrite it internally into
a `file:///` URL, because we dispatch our standalone transfer agent
based on the URL scheme. However, once we've looked up an appropriate
URL and formatted it as the scheme prefers, if it contains a trailing
slash, we ignore the entire lookup and replace the URL with the raw one
given, but without the trailing slash.
This behaviour seems a bit bizarre, and it causes us to take a local
path with a trailing slash and treat it not as the `file:///` URL it
needs to be, but as a raw local path, which gets refused. Let's avoid
this problem by looking at the rewritten URL, and modifying that instead
if we find its trailing slash to be offensive.
Git recently added a new field to the credential helper, `wwwauth[]`,
which may be repeated and includes all of the `WWW-Authenticate` headers
so that the credential helper may choose an appropriate set of
credentials and extract any sort of necessary data from the field (such
as challenge).
In Git LFS, we also want to do this with the `LFS-Authenticate` headers
as well, since those are often used for the same purpose, so include
both these headers in that field when passing them to `git credential
fill`.
Note that `git credential fill` only honours this value and passes it to
the credential helper in Git 2.41 and newer (including the latest
`HEAD`). However, just to be safe, let's add an undocumented and
experimental option (`credential.*.skipwwwauth`) that users can use to
control this, which we can remove in a few releases if it turns out it's
not needed. Similarly, skip our new tests if we have an older version
of Git where this doesn't work, since they'll otherwise fail.
In the upcoming changes to the credential helper protocol, we'll be able
to specify multiple values for the same key. In order to make this
work, let's make the credential structure able to handle multiple
values.
Note that if there are multiple values, we'll only use the first one for
most keys. That simplifies the logic and allows us to gracefully handle
future extensions to the protocol.
Following on from the changes in PR #4781, we can make
additional message strings translatable using the
tr.Tr.Get() method.
Because this method performs printf(3)-style format string
parsing and interpolation, we can simplify some of the
surrounding calls, e.g., from fmt.Errorf() to errors.New(),
and from fmt.Fprintf() to fmt.Fprintln(). This ensures
that if either the translated text or any interpolated
arguments happen to contain character sequences that would
be interpreted as Go format specifiers (e.g., "%s" or "%d"),
these will not result in warnings such as "%!s(MISSING)"
in the output text.
Note also that we try to remove newlines from the message
strings were possible and change the surrounding calls to
append them instead, e.g., with fmt.Fprintln().
In general, we can make the translation task somewhat
easier for translators, and less prone to error, if we
remove non-language-specific newlines from the text strings
to be translated.
To do this we can in some cases simply drop newlines (e.g.,
in panic messages), or break multi-line text strings into
several separate messages where that makes sense. For the
most part we simply append the necessary trailing or
intermediate newlines using the formatting tools available,
especially those provided in the commands/commands.go file.
Note that those tools (i.e., Print(), Exit(), etc.) always
append a newline, but some messages are intended to have
two final newlines, such as those from the "git lfs status"
command.
We also add a note to translators regarding some specific
spacing required by a message output by the "git lfs dedup"
command.
We convert several log messages to use the uppercase acronym
"HTTP" instead of the lowercase variant.
We also edit one message in lfshttp/client.go to remove an
idiosyncratic prefix of the full source code file name, and
edit the message slightly for greater clarity.
We rephrase some error message strings which are used when
wrapping other errors so they are clearer and more consistent
with other messages. Note that these strings will be
prepended to the wrapper errors' messages.
In the lfs/diff_index_scanner.go file in particular we
rephrase the additional message to include a full Git
command ("git diff-index"), which is similar to how errors
are reported in the git package.
The Negotiate authentication scheme can support multiple different types
of authentication. The two most popular are NTLM and Kerberos. When we
supported NTLM, we'd first try Kerberos, and if it failed, fall back to
NTLM.
However, we no longer support NTLM, but some people still have server
configuration that uses NTLM via Negotiate. For these people,
authentication may be broken. Let's fall back to Basic in such a case
by keeping track of which authentication mechanisms we've tried,
keeping only the supported mechanisms if we got a response, and
stripping out failing mechanisms, falling back to Basic.
To help with servers that support both Negotiate and Basic, we
specifically consider SPNEGO (Negotiate) errors as authentication
errors. This is because if the server supports Kerberos but the client
does not have a ticket, then we'll get an error trying to read the
client's tickets, which will manifest in this way.
Since we're about to do a v3.0.0 release, let's bump the version to v3.
Make this change automatically with the following command to avoid any
missed items:
git grep -l github.com/git-lfs/git-lfs/v2 | \
xargs sed -i -e 's!github.com/git-lfs/git-lfs/v2!github.com/git-lfs/git-lfs/v3!g'
When our go.mod file was introduced in commit
114e85c2002091eb415040923d872f8e4a4bc636 in PR #3208, the module
path chosen did not include a trailing /v2 component. However,
the Go modules specification now advises that module paths must
have a "major version suffix" which matches the release version.
We therefore add a /v2 suffix to our module path and all its
instances in import paths.
See also https://golang.org/ref/mod#major-version-suffixes for
details regarding the Go module system's major version suffix rule.
When we process an SSH URL, we intentionally strip off the slash at the
beginning of the URL. While that was convenient for
git-lfs-authenticate, it also prevents us from handling an absolute path
in git-lfs-transfer, since the path will have its leading slash stripped
off and will therefore be relative.
Instead, let's adopt Git's behavior, which is to not remove the leading
slash. This is an incompatible change, but we're about to do a major
release, so it's a good time to make it. This will affect both
git-lfs-transfer and git-lfs-authenticate commands, but at least GitHub
already supports the proper syntax.
Note that since we process the non-URL form of SSH remotes by converting
them to a URL and then parsing, let's strip off the leading slash when
we process that form, since there we do have the ability to distinguish
between absolute and relative paths.
Update the lfs-ssh-echo binary to handle this new format.
There are places in the testsuite that invoke the manifest code with no
operation provided (that is, an empty string). In such a case, we
attempt to spawn a git-lfs-transfer process with no operation, which of
course fails.
Instead of wastefully doing so, let's just avoid doing this in the first
place by skipping a spawn attempt when no operation is provided.
The lfsapi client is used to perform operations for SSH authentication
already, so we know we'll have one wherever SSH operations might be
done. Let's move the instantiation to the client so that we can reuse
it both in the transfer queue code and in the locking code as well.
We'll want to access the context that was originally passed to us later
on in a future commit, so let's preserve it so we can extract it from
our API client.
In the future, we'll want to call into the SSH code from multiple
packages, so let's move it out of the lfshttp package into its own
package to avoid package import loops. While we're at it, rename the
function names to remove the "ssh" prefix, since it's implied by the
fact that they're in a package called "ssh".
Move the tests to their own package to prevent an import loop and expose
the private functions so we can test them there.
Right now, all of the SSH metadata for an endpoint is in the Endpoint
struct, but in the future we'd like to move the SSH code to its own
package. At that point, we'll want to avoid a dependency on the
Endpoint struct, so let's move the SSH metadata out into its own struct,
which we'll include in Endpoint.
While we're at it, let's adjust most of the SSH code to use this new
struct instead so we can easily move it in the future.
Currently, we warn about aliases (e.g., url.*.insteadOf) when there's
already an alias for the same URL; that is, when the insteadOf entry
matches an existing one. However, in some environments, multiple
aliases are defined, but all to the same root URL (the one in the key).
In such a case, don't warn, since these aliases are already identical.
Do continue to warn if the root URL is different at all, since these are
different aliases for the same URL, and add a test for both of these
cases.
Our NTLM support has been known to be broken in various situations for a
while, specifically on Windows. The core team is unable to troubleshoot
these problems, and nobody has stepped up to maintain the NTLM support.
In addition, NTLM uses cryptography and security techniques that are
known to be insecure, such as the algorithms DES, MD4, and MD5, as well
as simple, unsalted hashes of passwords.
Since we now support Kerberos, most users should be able to replace
their use of NTLM with Kerberos instead. Users have reported this
working on Windows and it is known to work well on at least Debian as
well. Drop support for NTLM and remove it from the codebase.
When generating a local path endpoint, we now test whether the
path corresponds to a bare repo lacking a .git directory, and if so,
we ensure the endpoint path has any trailing .git segment trimmed.
Conversely, we ensure the endpoint path has a final .git segment if
a .git directory is found.
The test suite is updated to check the four relevant cases.
Add support for Kerberos authentication using SPNEGO (Negotiate).
Because NTLM is also supported by GSSAPI and can also use Negotiate, try
Kerberos (which is far more secure) first, and only then fall back to
NTLM. Similar to NTLM, no credentials are required, because the user
has a credential storage mechanism that provides them automatically.
In the future, we'll adjust the transport based on the access type, so
cache clients based on both the host name and the access type. If it's
not clear what type of access we're using, default to NoneAccess, since
this will be correct for most types.
In the future, we're going to need to access the access-related types
in the lfshttp package. To avoid an import loop, move Access and
AccessMode into the creds package. Add constructors and accessors since
the members are private.
Currently, we only support local remotes using file URLs, but not local
paths. This is a highly requested feature, however, so implement
support for local paths as remotes.
First, fix the handling of local paths we already have: the handling of
absolute Unix-style paths. Instead of checking for the remote to
contain a file URL to determine whether to append "info/lfs" to the URL,
look to see if the URL we're using is a file URL, which will catch local
paths which will have been rewritten as such by this point. The
"info/lfs" part of the URL is not handled by the transfer adapter, so we
don't want to add it on.
If we get something that looks like it's not a URL, check if it's a
file on the file system, and if so, don't attempt to interpret it as an
SSH URL. This fixes the confusion with Windows paths, which resemble
SSH-style locations with a single-letter alias as the host name.
Finally, when turning a local path into a file URL, turn the path into
an absolute one if possible and rewrite it using slashes, which is
required for file URLs. Add several tests for the various cases: one
for Unix-style paths, one for native paths, and one for relative paths.
In a future commit, we'll want the HTTP client to take a URL so we can
extract the scheme. Additionally, in some cases, specifically when a
user forces the use of HTTP/2 when it's not available, we'll need to
return an error, so return an additional error value.
Go 1.12.8 introduces a security fix for parsing URLs that contain a
colon followed by an invalid port number. Since our SSH remotes can
contain just such a colon, our hack to make these into URLs no longer
works.
Fix this by replacing the first colon in these "URLs" with a slash,
which is a path delimiter, which makes them parsable by newer versions
of Go. Update the name of the function since it now does more than its
previous name implies.
One commonly requested feature for Git LFS is support for local files.
Currently, we tell users that they must use a standalone transfer
agent, which is true, but nobody has provided one yet. Since writing a
simple transfer agent is not very difficult, let's provide one
ourselves.
Introduce a basic standalone transfer agent, git lfs standalone-file,
that handles uploads and downloads. Add a default configuration required
for it to work, while still allowing users to override this
configuration if they have a preferred implementation that is more
featureful. We provide this as a transfer agent instead of built-in
because it avoids the complexity of adding a different code path to the
main codebase, but also serves as a demonstration of how to write a
standalone transfer agent for others who might want to do so, much
like Git demonstrates remote helpers using its HTTP helper.
