The base64(1) binary provided with macOS does not accept a bare input
file name, unlike any of the GNU coreutils implementations of base64(1)
or either the older Fourmilab or newer bintrans(1) BSD implementations.
Instead, it appears to be a version unique to OS X which requires an -i
option before a file name to be used for input. See, for reference:
https://www.unix.com/man-page/osx/1/base64/https://man7.org/linux/man-pages/man1/base64.1.htmlhttps://www.fourmilab.ch/webtools/base64/https://man.freebsd.org/cgi/man.cgi?query=base64&manpath=FreeBSD+15.0-CURRENT
With the Xcode developer tools for macOS installed, along with Go, Git,
etc., our test suite mostly succeeds, but several tests fail because
they expect to call base64(1) and pass the bare /dev/urandom file name.
We can resolve this inconvenience for developers by adjusting those
tests to use the technique we already use in the t/t-migrate-*.sh
test suites where we redirect /dev/urandom into base64 as its
standard input. This allows the test suites to function on both
macOS as well as Linux systems which use one of the other base64(1)
implementations.
The io/ioutil package has been deprecated as of Go 1.16 [1]. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.
[1]: https://golang.org/doc/go1.16#ioutil
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
In commit 10c4ffc6b888eee8f2134a7009a0db1bc393e17b we added the "does
not look in current directory for git with credential helper" test in
order to validate changes made to remediate the issues reported in
CVE-2021-21237, and revised it as part of our response to CVE-2022-24826
in commit 11092ef2b17eaa67f3363edc68b59331b979f7ee. This test checks
that Git LFS will not run a potentially malicious Git executable found
in the working tree of a repository.
As noted in commit 7e8d3ba84ad8ee5314cb74a32b80622b919163fe, in order
to avoid invoking this executable while setting up the test conditions
we generally remove it from the working directory as soon as possible,
but we have to at least leave it there while running "git add git.exe"
in order to add it to the current Git index. Therefore we explicitly set
the PATH environment variable before running this command to include a
minimal set of necessary directories, such as the ones for the real Git
and Git LFS executables.
As of Go 1.19 we now also need to specify the GODEBUG environment variable
with a value of "execerrdot=0" in order to avoid occasional failures of
the "git add git.exe" command on Windows. These failures occur due to a
specific set of conditions. First, if the last-modified time of the
.git/index file is within a second of that of z.dat, the "git add" command
will refresh the Git index (assuming Git was compiled with USE_NSEC=0, as
appears to be the case for Git for Windows), and Git LFS will be invoked
to "clean" the z.dat file again. For more details, see the "racy Git"
documentation and the is_racy_stat() function in Git:
https://git-scm.com/docs/racy-git/ena54a84b333/read-cache.c (L356)
If Git LFS is rerun by the "git add git.exe" command, then when Git LFS
executes it looks for Git, and until we revise Git LFS to rely on Go's
os/exec package to not execute programs found in the current working
directory the os/exec package will detect our malicious Git program in the
current working directory and report an error. This occurs when Git LFS
first initializes a new exec.Cmd structure, even though Git LFS would then
locate the true Git executable from our custom PATH and reset the Path
member of the Cmd structure before trying to execute the program.
See, for reference, the Go blog article on the changes in Go 1.19 and
the Windows version of the lookPath() function in the os/exec package:
https://go.dev/blog/path-securitydcbe772469/src/os/exec/lp_windows.go (L163)dcbe772469/src/os/exec/exec.go (L24-L90)
Since we explicitly test Git LFS's avoidance of programs in the current
working directory using the "git-lfs pull" command later in our test,
we just want the "git add git.exe" command to succeed, so for the time being
we disable Go's new security checks for this command only using the
execerrdot=0 setting in GODEBUG. We will revisit this when we address the
larger issue of re-adopting Go's own logic for locating executable programs
safely and without security issues.
As we have introduced a --patch option to our script/changelog script
in this PR, we note its use in the appropriate step in the how-to
documentation of our Git LFS release process.
We also expand our description of several steps with additional notes,
and fix one typo.
When we prepare a patch release of Git LFS, we cherry-pick specific merged
changes from the "main" branch onto a "release-next" branch using the
"git cherry-pick" command with the -m1 option, which flattens all the
commits unique to the merge on the "main" branch into a single non-merge
commit on the release branch.
As a result, when we run our script/changelog script to generate a
summary of the changes for the patch release, the "git rev-list" command
it executes returns nothing because it uses the --merges option, and
the flattened commits on the release branch are not actual merge commits.
We therefore add a --patch option (as suggested by bk2204) to our script
which runs "git rev-list" without the --merges option.
We also add the -s option to the "git show" command the script runs
so as to avoid cluttering the output with the patch diff for each
commit it presents for categorization by the user. When run for a
non-patch Git LFS release, this is not important because the commits
found by "git rev-list" are all merges and generally have no blob
changes directly associated with them, so the "git show" output is
relatively succinct. But with the commits on the release branch for
a Git LFS patch release, which are not merges, the "git show" output
without the -s option is potentially quite verbose.
Right now, we call `GitDir` to find the Git directory for our repository
every time we look up an endpoint. However, we _might_ already have
this data in our Git configuration handler, so let's use that if
possible so we don't need to spawn a process every time. If it's not,
then cache the data so that we spawn at most one process.
When we're locking or unlocking multiple times, we may end up looking up
the endpoint a few times. This is very cheap if we don't need to invoke
Git. However, as written right now, we call Git every time we validate
a remote, which is wasteful.
To make this more efficient, let's compute the remote list once and
cache it to avoid spawning a process. Then, we can use the cached value
every time we need to validate a remote.
Right now, every time we call `ValidateRemote`, we look up the list of
remotes by invoking `git remote`. It would be nice if we could cache
the list of remotes so we could validate the remote multiple times
without the expense of calling Git multiple times, so let's add a
function, `ValidateRemoteFromList`, which can take a cached list and
perform the same computation.
If we're locking or unlocking multiple files, we don't need to look up
the top-level directory for the repository repeatedly. We've almost
certainly done so when we've looked up the configuration, so let's save
that information in the fields we already have and use that to compute
the absolute path.
When we lock or unlock multiple files, we compute the same data for each
path, resulting in possibly calling multiple different commands. The
repository on which we're working is not going to change while the
command is running (and if it does get moved or removed, then dying is
fine), so let's cache this data to try to avoid the expense of calling
`git rev-parse` command multiple times.
Make some changes here to use the existing values in the config object
instead of calling the `git` package ourselves. That will compute both
of these values once instead of recomputing them again and again, and
also moves the error handling out of this codepath. In addition, move
these calls up to the top of their respective functions so that we can
cache these values before further calls, eliminating more useless `git
rev-parse` invocations.
Some people have very slow SSH round trips and the extra time to attempt
a failed git-lfs-transfer operation is bothersome. Let's add an option
to allow users to use either the pure SSH protocol or the older hybrid
protocol based on the URL in question to let them choose the right value
if they know what it should be.
When we compute the contents of an Endpoint struct, we don't preserve
the URL from which we computed the value in the first place. For
example, if we use an SSH URL, we don't preserve the SSH URL anywhere
for further use. Let's do so, since we'll use it in a future commit.
Note that for anything but an SSH URL, this is the same value as the URL
field.
In `--dry-run` mode, we're documented to not modify the disk, which
means users will not expect us to be modifying `.gitattributes`. Fix
this by setting the `--no-modify-attrs` flag implicitly if `--dry-run`
is set.
We now need a team ID to notarize binaries, so let's pass one in from
the environment and down from the Makefile into new script. Use the new
script instead of `gon` because the latter uses the old `altool`, which
will no longer work as of November 1, 2023.
We'll want to notarize macOS binaries using the new `notarytool` command
rather than `gon` because the latter uses `altool` under the hood, which
will be disabled as of November 1, 2023. Create a script that runs the
proper command without echoing the arguments so that we don't include the
password in the log, even if it is filtered.
We should use `(*regexp.Regexp).MatchString` instead of
`(*regexp.Regexp).Match([]byte(...))` to avoid unnecessary `[]byte`
conversions and reduce allocations.
Example benchmark:
func BenchmarkMatch(b *testing.B) {
for i := 0; i < b.N; i++ {
if match := oidRE.Match([]byte("2a3b6578c2de07a35b2f2d9bf1869d437aed2e5b8f1c96a0df9d9be3c05a4a8a")); !match {
b.Fail()
}
}
}
func BenchmarkMatchString(b *testing.B) {
for i := 0; i < b.N; i++ {
if match := oidRE.MatchString("2a3b6578c2de07a35b2f2d9bf1869d437aed2e5b8f1c96a0df9d9be3c05a4a8a"); !match {
b.Fail()
}
}
}
goos: linux
goarch: amd64
pkg: github.com/git-lfs/git-lfs/v3/lfs
cpu: AMD Ryzen 7 PRO 4750U with Radeon Graphics
BenchmarkMatch-16 1044811 1202 ns/op 64 B/op 1 allocs/op
BenchmarkMatchString-16 2201485 560.9 ns/op 0 B/op 0 allocs/op
PASS
ok github.com/git-lfs/git-lfs/v3/lfs 3.882s
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
The project home page for Git LFS was moved to https://git-lfs.com/
from https://git-lfs.github.com/ last year, so we now update our
Debian and RPM package metadata to refer to the new home page URL.
Since commit git-lfs/build-dockers@d4c2fe6f79
the centos_script.bsh script in that repository has not built the
"repository configuration" or "repo" RPM packages, which were intended,
it seems, to contain extra packages and GPG signatures for Git LFS.
We can therefore remove the stale RPM spec and source files which
pertained only to this extra RPM build, and also some lines in the
rpm/build_rpms.bsh which created an empty stub GPG key file for this
build.
