2015-03-19 19:30:55 +00:00
|
|
|
# Git LFS Specification
|
2014-06-02 14:03:37 +00:00
|
|
|
|
2015-03-19 19:30:55 +00:00
|
|
|
This is a general guide for Git LFS clients. Typically it should be
|
|
|
|
implemented by a command line `git-lfs` tool, but the details may be useful
|
2014-06-02 14:03:37 +00:00
|
|
|
for other tools.
|
|
|
|
|
|
|
|
## The Pointer
|
|
|
|
|
2015-03-19 19:30:55 +00:00
|
|
|
The core Git LFS idea is that instead of writing large blobs to a Git repository,
|
2014-06-02 14:03:37 +00:00
|
|
|
only a pointer file is written.
|
|
|
|
|
2015-04-19 18:42:41 +00:00
|
|
|
* Pointer files are text files which MUST contain only UTF-8 characters.
|
|
|
|
* Each line MUST be of the format `{key} {value}\n` (trailing unix newline).
|
|
|
|
* Only a single space character between `{key}` and `{value}`.
|
2015-06-07 21:48:08 +00:00
|
|
|
* Keys MUST only use the characters `[a-z] [0-9] . -`.
|
2015-04-19 18:42:41 +00:00
|
|
|
* The first key is _always_ `version`.
|
|
|
|
* Lines of key/value pairs MUST be sorted alphabetically in ascending order
|
|
|
|
(with the exception of `version`, which is always first).
|
|
|
|
* Values MUST NOT contain return or newline characters.
|
Fix spec: tree entry for pointer file preserves exec bit
In practice, Git LFS preserves the executable status of files when they are
cleaned-to/smudged-from pointer files - in fact the
[git attributes clean/smudge filter mechanism](http://git-scm.com/docs/gitattributes#__code_filter_code)
doesn't actually provide any support for changing the executable status of
file, they just work on the file's contents with standard input & output.
This is good behaviour, because users don't want their big *.exe files to
stop being executable when they start using Git LFS, and the executable
status is not preserved anywhere else (eg. rightly, it is not stored within
the pointer file).
As an example, here are two *.exe files that have been stored with Git LFS,
which has preserved their exec bit in the file tree object. Note that
`bar.exe` has it's exec bit _set_ (`100755`), contrary to the spec:
```
$ git cat-file -p HEAD^{tree}
100644 blob ce8828c1134a1f9feb8177e9e2b988d94a94d951 .gitattributes
100644 blob 9242c60b11d49b42e829070feef1c5b7cf7a6159 foo.exe
100755 blob 25637885004ac19bd1e0404e2bb0d2247de37805 bar.exe
$ git cat-file -p 25637885004ac19bd1e0404e2bb0d2247de37805
version https://git-lfs.github.com/spec/v1
oid sha256:364ea686b8d77dbf49d1c1ce28023e818954574310b68c61433fbb02596744e2
size 51200
```
I think the spec just needs correcting here. The wording was introduced
with https://github.com/github/git-lfs/pull/246, tweaked with
https://github.com/github/git-lfs/commit/4bdecf, but I couldn't find any
justification in those for the statement that the executable bit should
always be cleared.
2015-08-31 16:54:42 +00:00
|
|
|
* Pointer files MUST be stored in Git with their executable bit matching that
|
|
|
|
of the replaced file.
|
2015-04-19 18:42:41 +00:00
|
|
|
|
|
|
|
The required keys are:
|
|
|
|
|
|
|
|
* `version` is a URL that identifies the pointer file spec. Parsers MUST use
|
|
|
|
simple string comparison on the version, without any URL parsing or
|
|
|
|
normalization. It is case sensitive, and %-encoding is discouraged.
|
|
|
|
* `oid` tracks the unique object id for the file, prefixed by its hashing
|
2015-04-22 15:51:27 +00:00
|
|
|
method: `{hash-method}:{hash}`. Currently, only `sha256` is supported.
|
2015-04-19 18:42:41 +00:00
|
|
|
* `size` is in bytes.
|
|
|
|
|
|
|
|
Example of a v1 text pointer:
|
|
|
|
|
2014-06-02 14:03:37 +00:00
|
|
|
```
|
2015-03-19 19:30:55 +00:00
|
|
|
version https://git-lfs.github.com/spec/v1
|
2014-08-14 17:15:22 +00:00
|
|
|
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
|
|
|
|
size 12345
|
2014-06-02 14:03:37 +00:00
|
|
|
(ending \n)
|
|
|
|
```
|
|
|
|
|
2015-06-27 00:59:40 +00:00
|
|
|
Blobs created with the pre-release version of the tool generated files with
|
2015-04-22 15:51:38 +00:00
|
|
|
a different version URL. Git LFS can read these files, but writes them using
|
|
|
|
the version URL above.
|
|
|
|
|
|
|
|
```
|
|
|
|
version https://hawser.github.com/spec/v1
|
|
|
|
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
|
|
|
|
size 12345
|
|
|
|
(ending \n)
|
|
|
|
```
|
|
|
|
|
2015-04-19 18:42:41 +00:00
|
|
|
For testing compliance of any tool generating its own pointer files, the
|
|
|
|
reference is this official Git LFS tool:
|
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
**NOTE:** exact pointer command behavior TBD!
|
2014-07-24 21:45:41 +00:00
|
|
|
|
2015-04-19 18:42:41 +00:00
|
|
|
* Tools that parse and regenerate pointer files MUST preserve keys that they
|
|
|
|
don't know or care about.
|
2015-04-22 16:13:40 +00:00
|
|
|
* Run the `pointer` command to generate a pointer file for the given local
|
|
|
|
file:
|
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
```
|
|
|
|
$ git lfs pointer --file=path/to/file
|
|
|
|
Git LFS pointer for path/to/file:
|
2015-04-22 16:16:41 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
version https://git-lfs.github.com/spec/v1
|
|
|
|
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
|
|
|
|
size 12345
|
|
|
|
```
|
2015-04-22 16:13:40 +00:00
|
|
|
|
|
|
|
* Run `pointer` to compare the blob OID of a pointer file built by Git LFS with
|
|
|
|
a pointer built by another tool.
|
|
|
|
|
|
|
|
* Write the other implementation's pointer to "other/pointer/file":
|
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
```
|
|
|
|
$ git lfs pointer --file=path/to/file --pointer=other/pointer/file
|
|
|
|
Git LFS pointer for path/to/file:
|
2015-04-22 16:13:40 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
version https://git-lfs.github.com/spec/v1
|
|
|
|
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
|
|
|
|
size 12345
|
2015-04-22 16:13:40 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
Blob OID: 60c8d8ab2adcf57a391163a7eeb0cdb8bf348e44
|
2015-04-22 16:13:40 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
Pointer from other/pointer/file
|
|
|
|
version https://git-lfs.github.com/spec/v1
|
|
|
|
oid sha256 4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
|
|
|
|
size 12345
|
2015-04-22 16:13:40 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
Blob OID: 08e593eeaa1b6032e971684825b4b60517e0638d
|
2015-04-22 16:13:40 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
Pointers do not match
|
|
|
|
```
|
2015-04-22 16:13:40 +00:00
|
|
|
|
|
|
|
* It can also read STDIN to get the other implementation's pointer:
|
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
```
|
|
|
|
$ cat other/pointer/file | git lfs pointer --file=path/to/file --stdin
|
|
|
|
Git LFS pointer for path/to/file:
|
2015-04-22 16:13:40 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
version https://git-lfs.github.com/spec/v1
|
|
|
|
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
|
|
|
|
size 12345
|
2015-04-22 16:13:40 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
Blob OID: 60c8d8ab2adcf57a391163a7eeb0cdb8bf348e44
|
2015-04-22 16:13:40 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
Pointer from STDIN
|
|
|
|
version https://git-lfs.github.com/spec/v1
|
|
|
|
oid sha256 4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
|
|
|
|
size 12345
|
2015-04-22 16:13:40 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
Blob OID: 08e593eeaa1b6032e971684825b4b60517e0638d
|
2015-04-22 16:13:40 +00:00
|
|
|
|
2015-06-07 21:48:08 +00:00
|
|
|
Pointers do not match
|
|
|
|
```
|
2014-07-21 19:41:45 +00:00
|
|
|
|
2014-06-02 14:03:37 +00:00
|
|
|
## Intercepting Git
|
|
|
|
|
2015-03-19 19:30:55 +00:00
|
|
|
Git LFS uses the `clean` and `smudge` filters to decide which files use it. The
|
2015-11-16 20:31:26 +00:00
|
|
|
global filters can be set up with `git lfs install`:
|
2014-06-02 14:03:37 +00:00
|
|
|
|
|
|
|
```
|
2015-11-18 18:27:00 +00:00
|
|
|
$ git lfs install
|
2014-06-02 14:03:37 +00:00
|
|
|
```
|
|
|
|
|
2015-04-11 20:14:57 +00:00
|
|
|
These filters ensure that large files aren't written into the repository proper,
|
|
|
|
instead being stored locally at `.git/lfs/objects/{OID-PATH}` (where `{OID-PATH}`
|
|
|
|
is a sharded filepath of the form `OID[0:2]/OID[2:4]/OID`), synchronized with
|
2015-06-27 00:59:40 +00:00
|
|
|
the Git LFS server as necessary. Here is a sample path to a file:
|
2015-04-19 18:22:15 +00:00
|
|
|
|
|
|
|
.git/lfs/objects/4d/7a/4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
|
2015-04-11 20:14:57 +00:00
|
|
|
|
2014-06-02 14:03:37 +00:00
|
|
|
The `clean` filter runs as files are added to repositories. Git sends the
|
|
|
|
content of the file being added as STDIN, and expects the content to write
|
|
|
|
to Git as STDOUT.
|
|
|
|
|
|
|
|
* Stream binary content from STDIN to a temp file, while calculating its SHA-256
|
|
|
|
signature.
|
2015-04-19 18:22:15 +00:00
|
|
|
* Atomically move the temp file to `.git/lfs/objects/{OID-PATH}` if it does not
|
|
|
|
exist, and the sha-256 signature of the contents matches the given OID.
|
2014-06-02 14:03:37 +00:00
|
|
|
* Delete the temp file.
|
|
|
|
* Write the pointer file to STDOUT.
|
|
|
|
|
|
|
|
Note that the `clean` filter does not push the file to the server. Use the
|
2015-05-15 13:00:23 +00:00
|
|
|
`git push` command to do that (lfs files are pushed before commits in a pre-push hook).
|
2014-06-02 14:03:37 +00:00
|
|
|
|
|
|
|
The `smudge` filter runs as files are being checked out from the Git repository
|
|
|
|
to the working directory. Git sends the content of the Git blob as STDIN, and
|
|
|
|
expects the content to write to the working directory as STDOUT.
|
|
|
|
|
|
|
|
* Read 100 bytes.
|
|
|
|
* If the content is ASCII and matches the pointer file format:
|
2015-04-10 13:25:47 +00:00
|
|
|
* Look for the file in `.git/lfs/objects/{OID-PATH}`.
|
2014-06-02 14:03:37 +00:00
|
|
|
* If it's not there, download it from the server.
|
2015-06-27 00:59:40 +00:00
|
|
|
* Write its contents to STDOUT
|
2014-06-02 14:03:37 +00:00
|
|
|
* Otherwise, simply pass the STDIN out through STDOUT.
|
|
|
|
|
2015-06-27 00:59:40 +00:00
|
|
|
The `.gitattributes` file controls when the filters run. Here's a sample file that
|
2015-03-22 19:02:33 +00:00
|
|
|
runs all mp3 and zip files through Git LFS:
|
2014-06-02 14:03:37 +00:00
|
|
|
|
|
|
|
```
|
|
|
|
$ cat .gitattributes
|
2015-07-09 16:01:50 +00:00
|
|
|
*.mp3 filter=lfs -text
|
|
|
|
*.zip filter=lfs -text
|
2014-06-02 14:03:37 +00:00
|
|
|
```
|
2014-06-24 21:19:25 +00:00
|
|
|
|
2015-04-20 17:11:03 +00:00
|
|
|
Use the `git lfs track` command to view and add to `.gitattributes`.
|