git-lfs/test
Taylor Blau 5346b02747 commands/command_migrate.go: introduce '--fixup' flag on 'import'
A common invocation of the 'git lfs migrate import' command is with
'--include' and/or '--exclude' flag(s), which specify wildmatch
pattern(s) for which paths to migrate and/or not migrate.

This is useful for retroactively importing a set of files into Git LFS's
care, or fixing up a file that should have been tracked by Git LFS but
was accidentally committed as a large object instead.

In the later case, it is often the reality that a user will run 'git lfs
migrate --import' with an '--include' path that they believe will gather
the file (and the file alone). This approach is brittle because it
requires the user to infer not only the applicable pattern but the
meaning of that pattern. It also requires the user to run more than one
migration when fixing multiple types of files.

The .gitattributes file(s) contained within a repository provide an
authoritative source on what file(s) are considered by Git to be tracked
in Git LFS. We can use this information to infer the correct patterns to
``fix up'' a broken repository.

In the simplest case, if a repository's .gitattributes file contains the
following:

    *.txt filter=lfs merge=lfs diff=lfs -text

But a .txt file matched by that pattern is not parse-able as an LFS
pointer, it will appear as unable to checkout.

Running 'git lfs migrate import --fixup --everything' will correctly
traverse history and find the affected .txt file, read it, create an
object file for it, and store it as an LFS pointer in history.

Thus, a user can run one command which will recognize arbitrarily
complex problems where a file should be tracked by Git LFS, but isn't.

Later, this feature could be combined with the new 'git lfs migrate
export' functionality to also clean files _out_ of Git LFS to object
files when they are not supposed to be tracked as Git LFS objects.
2018-07-06 14:42:48 -05:00
..
cmd Use unique repo name for tests that require credentials. 2018-03-06 19:03:07 -05:00
git-lfs-test-server-api tq, commands: send refspec on batch upload calls 2018-01-05 11:12:57 -07:00
README.md glide: remove dependency 'github.com/kr/text' 2017-08-10 17:17:03 -06:00
test-askpass.sh only use askpass if credential.helper is not configured 2017-10-02 11:57:07 -06:00
test-batch-error-handling.sh commands/track: quote pattern in output of track command 2017-03-24 11:13:11 -06:00
test-batch-retries.sh tq,test: trace shorthand transfer error when retrying 2017-04-26 11:24:03 -06:00
test-batch-transfer.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-batch-unknown-oids.sh tq,test/cmd: raise an error on unknown OIDs 2016-12-12 15:33:15 -07:00
test-checkout.sh test/test-checkout.sh: remove logging grep site 2018-06-11 12:28:14 -05:00
test-chunked-transfer-encoding.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-clean.sh lfs: clean contents larger than 1024 bytes over stdin 2017-08-08 21:07:40 -06:00
test-clone-deprecated.sh test: suppress output from popd 2017-08-29 10:06:38 -04:00
test-clone.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-commit-delete-push.sh always use 'git rev-parse' to figure out git and working tree dirs 2016-02-01 11:20:32 -07:00
test-config.sh localstorage: exit stage left 2017-10-25 13:23:12 -06:00
test-credentials-no-prompt.sh test: fix bad-askpass test 2017-10-27 10:28:48 -06:00
test-credentials.sh *: use patched version of bgetnry/go-netrc 2018-05-25 12:42:02 -07:00
test-custom-transfers.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-duplicate-oids.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-env.sh test/test-env: skip comparing GIT_EXEC_PATH 2018-05-21 13:53:30 -07:00
test-expired.sh test: test credential changes 2017-10-27 09:57:08 -06:00
test-ext.sh LFS Extensions: use lfs.extension instead of lfs-ext 2015-08-01 21:13:37 -04:00
test-extra-header.sh test/test-extra-header.sh: run git config with --local 2018-05-18 15:28:15 -07:00
test-fetch-include.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-fetch-paths.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-fetch-recent.sh commands/track: quote pattern in output of track command 2017-03-24 11:13:11 -06:00
test-fetch-refspec.sh lfs,git: send refspec in batch api calls from smudge filter 2018-01-05 15:01:29 -07:00
test-fetch.sh add failing test for 'lfs.fetchinclude' usage 2018-01-30 14:05:26 +01:00
test-filter-branch.sh test: ensure filter-process works with git filter-branch 2016-12-26 13:28:07 -07:00
test-filter-process.sh commands/filter_process: implement 'delay' capability 2017-08-15 16:51:38 -06:00
test-fsck.sh rewrite fsck command to use gitscanner api 2016-11-29 13:56:47 -07:00
test-happy-path.sh Merge pull request #2811 from git-lfs/new-progress-design 2018-01-17 16:19:51 -07:00
test-install-custom-hooks-path-unsupported.sh Add post-commit hook to make newly added lockable files read-only 2017-01-02 14:36:28 +00:00
test-install-custom-hooks-path.sh Add post-merge hook to make files read-only again if not locked 2017-01-02 16:56:47 +00:00
test-install.sh lfs/attribute.go: remove default value from upgradeables 2018-05-04 09:09:03 -07:00
test-lock.sh commands/command_lock: resolve symbolic links in cwd 2018-05-04 15:28:16 -07:00
test-locks.sh test: fix branch assertions in lock tests 2018-01-05 14:25:50 -07:00
test-logs.sh fs: create logdir 2017-10-24 16:21:15 -06:00
test-ls-files.sh test/test-ls-files.sh: skip on CircleCI 2018-07-03 12:11:15 -05:00
test-malformed-pointers.sh test: remove outdated assertion skip 2017-08-24 18:05:12 -04:00
test-mergetool.sh test: ensure that git-mergetool(1) works with large files 2018-03-23 12:13:13 -04:00
test-migrate-export.sh commands: address export PR feedback 2018-07-03 11:13:25 -07:00
test-migrate-fixtures.sh commands/command_migrate.go: introduce '--fixup' flag on 'import' 2018-07-06 14:42:48 -05:00
test-migrate-fixup.sh commands/command_migrate.go: introduce '--fixup' flag on 'import' 2018-07-06 14:42:48 -05:00
test-migrate-import-no-rewrite.sh test: shorten repo/folder names 2018-05-31 11:26:58 -07:00
test-migrate-import.sh git/git.go: pass --multiple to git-fetch(1) when appropriate 2018-06-13 14:07:19 -05:00
test-migrate-info.sh commands/migrate: support '^'-prefix refspec in arguments 2017-12-13 11:13:11 -08:00
test-object-authenticated.sh Remove system credential helper specific work-arounds in tests 2016-09-28 21:30:05 +02:00
test-pointer.sh fix pointer error messages 2016-08-18 09:39:06 -06:00
test-post-checkout.sh test: fix 'refute_file_writable' typo 2017-08-17 18:09:10 -06:00
test-post-commit.sh test: add an integration test for git-lfs-post-commit 2017-08-28 17:58:31 -04:00
test-post-merge.sh test: fix 'refute_file_writable' typo 2017-08-17 18:09:10 -06:00
test-pre-push.sh commands/uploader: set lfs.allowincompletepush default to false 2018-07-05 23:20:02 +02:00
test-progress-meter.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-progress.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-prune-worktree.sh commands: drop use of *Spinner in git-lfs-prune(1) 2017-11-28 20:08:48 -08:00
test-prune.sh commands: drop use of *Spinner in git-lfs-prune(1) 2017-11-28 20:08:48 -08:00
test-pull.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-push-bad-dns.sh lfs/tq: retry transfers before accepting new ones 2016-12-12 09:24:10 -07:00
test-push-failures-local.sh commands/uploader: set lfs.allowincompletepush default to false 2018-07-05 23:20:02 +02:00
test-push-failures-remote.sh test: rename test-push-failures to test-push-failures-remote 2018-07-05 21:17:06 +02:00
test-push-file-with-branch-name.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-push.sh test: refactor test-push-* and move tests to test-push-failures-local 2018-07-05 21:17:06 +02:00
test-reference-clone.sh commands/filter_process: implement 'delay' capability 2017-08-15 16:51:38 -06:00
test-resume-http-range.sh commands/track: quote pattern in output of track command 2017-03-24 11:13:11 -06:00
test-resume-tus.sh commands/track: quote pattern in output of track command 2017-03-24 11:13:11 -06:00
test-smudge.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-ssh.sh sanitize ssh options parsed from ssh:// url 2017-05-19 09:25:22 -06:00
test-status.sh commands/status: make paths relative to the working directory 2018-06-21 15:38:17 -05:00
test-submodule-lfsconfig.sh test lfsconfig behavior with git submodule update --init --remote 2016-01-29 13:47:21 -07:00
test-submodule.sh test lfsconfig behavior with git submodule update --init --remote 2016-01-29 13:47:21 -07:00
test-track-attrs.sh test: skip test "track (--no-modify-attrs)" on Git <2.1.0 2017-06-26 09:34:25 -06:00
test-track-wildcards.sh commands/track: quote pattern in output of track command 2017-03-24 11:13:11 -06:00
test-track.sh test/test-track: examine system gitattributes 2018-06-18 19:43:03 -05:00
test-uninstall.sh commands/uninstall: do not log about global hooks with --local 2018-04-18 15:21:59 -07:00
test-unlock.sh test: fix branch assertions in lock tests 2018-01-05 14:25:50 -07:00
test-untrack.sh commands/{un,}track: local-prefix agnostic comparison 2018-04-05 10:51:26 -07:00
test-unusual-filenames.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
test-update.sh lfs/setup: tweak whitespace around hook contents 2017-07-11 16:10:30 -06:00
test-verify.sh test/test-verify: ensure that Authorization header is sent 2017-03-24 15:30:35 -06:00
test-version.sh commands: make --version a synonym for 'version' 2018-04-12 11:56:48 -07:00
test-worktree.sh test/test-worktree.sh: ignore GIT_EXEC_PATH in tests 2018-05-21 14:19:53 -07:00
test-zero-len-file.sh tq: new progress meter formatting 2018-01-05 18:43:31 -08:00
testenv.sh test: use $PATH_SEPARATOR 2017-08-08 14:20:02 -06:00
testhelpers.sh test: add refute_pointer test helper 2018-07-03 11:13:25 -07:00
testlib.sh testlib: clear GIT_CONFIG and XDG_CONFIG_HOME 2017-11-07 15:45:00 -05:00
testutils.go lfsapi: add *git.Configuration to Client 2017-10-25 15:38:13 -06:00

Git LFS Tests

Git LFS uses two form of tests: unit tests for the internals written in Go, and integration tests that run git and git-lfs in a real shell environment. You can run them separately:

$ script/test        # Tests the Go packages.
$ script/integration # Tests the commands in shell scripts.

CI servers should always run both:

$ script/cibuild

Internal Package Tests

The internal tests use Go's builtin testing package.

You can run individual tests by passing arguments to script/test:

# test a specific Go package
$ script/test lfs

# pass other `go test` arguments
$ script/test lfs -run TestSuccessStatus -v
github.com/rubyist/tracerx
github.com/git-lfs/git-lfs/git
github.com/technoweenie/assert
=== RUN TestSuccessStatus
--- PASS: TestSuccessStatus (0.00 seconds)
PASS
ok  	_/Users/rick/git-lfs/git-lfs/lfs	0.011s

Integration Tests

Git LFS integration tests are shell scripts that test the git-lfs command from the shell. Each test file can be run individually, or in parallel through script/integration. Some tests will change the pwd, so it's important that they run in separate OS processes.

$ test/test-happy-path.sh
compile git-lfs for test/test-happy-path.sh
LFSTEST_URL=/Users/rick/git-lfs/git-lfs/test/remote/url LFSTEST_DIR=/Users/rick/git-lfs/git-lfs/test/remote lfstest-gitserver
test: happy path ...                                               OK
  1. The integration tests should not rely on global system or git config.
  2. The tests should be cross platform (Linux, Mac, Windows).
  3. Tests should bootstrap an isolated, clean environment. See the Test Suite section.
  4. Successful test runs should have minimal output.
  5. Failing test runs should dump enough information to diagnose the bug. This includes stdout, stderr, any log files, and even the OS environment.

There are a few environment variables that you can set to change the test suite behavior:

  • GIT_LFS_TEST_DIR=path - This sets the directory that is used as the current working directory of the tests. By default, this will be in your temp dir. It's recommended that this is set to a directory outside of any Git repository.
  • GIT_LFS_TEST_MAXPROCS=N - This tells script/integration how many tests to run in parallel. Default: 4.
  • KEEPTRASH=1 - This will leave the local repository data in a tmp directory and the remote repository data in test/remote.
  • SKIPCOMPILE=1 - This skips the Git LFS compilation step. Speeds up the tests when you're running the same test script multiple times without changing any Go code.

Also ensure that your noproxy environment variable contains 127.0.0.1 host, to allow git commands to reach the local Git server lfstest-gitserver.

Test Suite

The testenv.sh script includes some global variables used in tests. This should be automatically included in every test/test-*.sh script and script/integration.

testhelpers.sh defines some shell functions. Most are only used in the test scripts themselves. script/integration uses the setup() and shutdown() functions.

testlib.sh is a fork of a lightweight shell testing lib that is used internally at GitHub. Only the test/test-*.sh scripts should include this.

Tests live in this ./test directory, and must have a unique name like: test-{name}.sh. All tests should start with a basic template. See test/test-happy-path.sh for an example.

#!/usr/bin/env bash

. "test/testlib.sh"

begin_test "template"
(
  set -e

  echo "your tests go here"
)
end_test

The set -e command will bail on the test at the first command that returns a non zero exit status. Use simple shell commands like grep as assertions.

The test suite has standard setup and shutdown functions that should be run only once, before/after running the tests. The setup and shutdown functions are run by script/integration and also by individual test scripts when they are executed directly. Setup does the following:

  • Resets temporary test directories.
  • Compiles git-lfs with the latest code changes.
  • Compiles Go files in test/cmd to bin, and adds them the PATH.
  • Spins up a test Git and Git LFS server so the entire push/pull flow can be exercised.
  • Sets up a git credential helper that always returns a set username and password.

The test Git server writes a test/remote/url file when it's complete. This file is how individual test scripts detect if script/integration is being run. You can fake this by manually spinning up the Git server using the lfstest-gitserver line that is output after Git LFS is compiled.

By default, temporary directories in tmp and the test/remote directory are cleared after test runs. Send the "KEEPTRASH" if you want to keep these files around for debugging failed tests.