17 KiB
Multipart HTTP transfer mode proposal
This is a proposal for a new transfer mode, designed to support multi-part HTTP uploads. This is a protocol extension to
Git LFS, defining a new transfer mode to be implemented by Git LFS clients and servers in addition to the current basic
transfer mode.
This proposal is based on the experimental multipart-basic
transfer mode originally
implemented by datopian/giftless.
Reasoning
Many storage vendors and cloud vendors today offer an API to upload files in "parts" or "chunks", using multiple HTTP requests, allowing improved stability and performance. This is especially handy when files are multiple gigabytes in size, and a failure during the upload of a file would require re-uploading it, which could be extremely time consuming.
The purpose of the multipart
transfer mode is to allow Git LFS servers and client facilitate direct-to-storage
uploads for backends supporting multipart or chunked uploads.
As the APIs offered by storage vendors differ greatly, multipart
transfer mode will offer abstraction over most
of these complexities in hope of supporting as many storage vendors as possible.
Terminology
Throughout this document, the following terms are in use:
- LFS Server - The HTTP server to which the LFS
batch
request is sent - Client or LFS Client - a client using the Git LFS protocol to push large files to storage via an LFS server
- Storage Backend - The HTTP server handling actual storage; This may or may not be the same server as the LFS server, and for the purpose of this document, typically it is not. A typical implementation of this protocol would have the Storage Backend be a cloud storage service such as Amazon S3 or Google Cloud Storage.
Design Goals
- Abstract vendor specific API and flow into a generic protocol
- Remain as close as possible to the
basic
transfer API - Work at least with the multi-part APIs of Amazon S3, Google Cloud Storage and Azure Blob Storage,
- Define how uploads can be resumed by re-doing parts and not-redoing parts that were uploaded successfully (this may be vendor specific and not always supported)
- Do not require any state to be maintained in the server side
High Level Protocol Specs
- The name of the transfer is
multipart
- Batch requests are the same as
basic
requests except that{"transfers": ["multipart", "basic"]}
is the expected transfers value. Clients MUST retainbasic
as the fallback transfer mode to ensure compatibility with servers not implementing this extension. {"operation": "download"}
replies work exactly likebasic
download request with no change{"operation": "upload"}
replies will break the upload into severalactions
:parts
(optional), a list of zero or more part upload actionsverify
(optional), an action to verify the file is in storage, similar tobasic
upload verify actionsabort
(optional), an action to abort the upload and clean up all unfinished chunks and state
- Just like
basic
transfers, if the file fully exists and is committed to storage, noactions
will be provided in the reply and the upload can simply be skipped - If a
verify
action is provided, calling it is required and not optional. In some cases, this endpoint may be used to finalize the upload. - An empty or missing list of
parts
with averify
action may mean all parts have been uploaded butverify
still needs to be called by the client. - Authentication and authorization behave just like with the
basic
protocol.
Action Objects
Each one of the parts
and the verify
and abort
actions contain instructions for the client on how to send a
request performing a the particular action. These are similar to basic
transfer adapter actions
but may include
some common, as well as action specific additional parameters.
All actions allow href
, header
and expires_in
parameters just like basic
transfer actions.
parts
actions
Each parts
action should include the pos
and size
attributes, in addition to the attributes specified
above:
pos
indicate the position in bytes within the file in which the part should begin. If not specified,0
(that is the beginning of the file) is assumed.size
is the size of the part in bytes. Ifsize
is omitted, default to read until the end of file.- If both
pos
andsize
are omitted, the action is expected to be a single-part upload of the entire file
In addition, parts
actions may include the following parameters:
method
, withPUT
as the default method if none is specified. This allows customizing the HTTP method used when uploading object parts.want_digest
to specify an expected HTTPDigest
header, as described below.
verify
action
The verify
action is similar to basic
transfer mode verify
, with the following additional parameters:
params
- an object with additional parameters to send to the server when sending theverify
request. These parameters are to be sent to the server exactly as provided, as the value of theparams
JSON attribute.
abort
action
The abort
action may include the method
attribute as specified for parts
actions above.
Batch Request / Response Examples
Upload Batch Request
The following is a ~10mb file upload request:
{
"transfers": ["multipart", "basic"],
"operation": "upload",
"objects": [
{
"oid": "20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e",
"size": 10000000
}
]
}
Upload Batch Response
The following is a response for the same request, given an imaginary storage backend:
{
"transfer": "multipart",
"objects": [
{
"oid": "20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e",
"size": 10000000,
"actions": {
"parts": [
{
"href": "https://storage.cloud.example/storage/upload/20492a4d0d84?part=0",
"header": {
"Authorization": "Bearer someauthorizationtokenwillbesethere"
},
"pos": 0,
"size": 2500000,
"expires_in": 86400
},
{
"href": "https://storage.cloud.example/storage/upload/20492a4d0d84?part=1",
"header": {
"Authorization": "Bearer someauthorizationtokenwillbesethere"
},
"pos": 2500000,
"size": 2500000,
"expires_in": 86400
},
{
"href": "https://storage.cloud.example/storage/upload/20492a4d0d84?part=2",
"header": {
"Authorization": "Bearer someauthorizationtokenwillbesethere"
},
"pos": 5000000,
"size": 2500000,
"expires_in": 86400
},
{
"href": "https://storage.cloud.example/storage/upload/20492a4d0d84?part=3",
"header": {
"Authorization": "Bearer someauthorizationtokenwillbesethere"
},
"pos": 7500000,
"expires_in": 86400
}
],
"verify": {
"href": "https://lfs.mycompany.example/myorg/myrepo/multipart/verify",
"authenticated": true,
"header": {
"Authorization": "Basic 123abc123abc123abc123abc123="
},
"expires_in": 86400,
"params": {
"uploadId": "20492a4d0d84",
"partIds": [0, 1, 2, 3]
}
},
"abort": {
"href": "https://storage.cloud.example/storage/upload/20492a4d0d84",
"authenticated": true,
"header": {
"Authorization": "Basic 123abc123abc123abc123abc123="
},
"method": "DELETE",
"expires_in": 86400
}
}
}
]
}
verify
request example
Given the batch
response above, after all parts have been uploaded the client should send the following verify
request to https://lfs.mycompany.example/myorg/myrepo/multipart/verify
:
POST /myorg/myrepo/multipart/verify
Host: lfs.mycompany.example
Authorization: Basic 123abc123abc123abc123abc123=
Content-type: application/json
{
"oid": "20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e",
"size": 10000000,
"params": {
"uploadId": "20492a4d0d84",
"partIds": [0, 1, 2, 3]
}
}
Assuming that all parts were uploaded successfully, the server should respond with a 200 OK
response.
abort
request example
Given the batch
response above, the client may choose to cancel the upload by sending the following
abort
request to https://storage.cloud.example/storage/upload/20492a4d0d84
:
> DELETE /storage/upload/20492a4d0d84
> Host: storage.cloud.example
> Content-length: 0
Uploaded Part Digest
Some storage backends will support, or even require, uploading clients to send a digest of the uploaded part when uploading the part. This is a useful capability even if not required, as it allows backends to validate each part separately as it is uploaded.
To support this, parts
request objects may include a want_digest
value, which is expected to be a list of digest
algorithms in the same format of the Want-Digest
HTTP header specified by RFC-3230.
Any cryptographically secure digest algorithm registered with IANA
via the process outlined in RFC-3230 may be specified in want_digest
. Algorithms
considered cryptographically insecure, including MD5
and SHA-1
, should not be accepted. Namely, contentMD5
is
not an accepted value of want_digest
.
If one or more digest algorithms with non-zero q-value is specified in want_digest
, clients should select a favored
supported algorithm, calculate the part digest using that algorithm, and send it when uploading the part using the Digest
HTTP
header as specified by RFC-3230 section 4.3.1.
While clients may include the part digest calculated using more than one algorithm, this is typically not required and should be avoided.
Note that if want_digest
is specified but the client cannot support any of the requested algorithms, the client may
still choose to continue uploading parts without sending a Digest
header. However, the storage server may choose
to reject the request in such cases.
Uploaded Part Digest Example
Examples of a batch response with want_digest
in the reply
With SHA-512 as a preferred algorithm, and SHA-256 as a less preferred option if SHA-512 is not possible:
{
"actions": {
"parts": [
{
"href": "https://storage.cloud.example/storage/upload/20492a4d0d84?part=3",
"header": {
"Authorization": "Bearer someauthorizationtokenwillbesethere"
},
"pos": 7500001,
"want_digest": "sha-512;q=1.0, sha-256;q=0.5"
}
]
}
}
Example of part upload request send to the storage server
Following on the want_digest
value specified in the last example, the client should now send the following headers
to the storage server when uploading the part, assuming SHA-512
is supported:
HTTP/1.1 PUT /storage/upload/20492a4d0d84?part=3
Authorization: Bearer someauthorizationtokenwillbesethere
Digest: SHA-512=thvDyvhfIqlvFe+A9MYgxAfm1q5thvDyvhfIqlvFe+A9MYgxAfm1q5=
Expected HTTP Responses
For each one of the parts
, as well as abort
and verify
requests sent by the client, the following responses are
to be expected:
-
Any response with a
20x
status code is to be considered by clients as successful. This ambiguity is by design, to support variances between vendors (which may use200
or201
to indicate a successful upload, for example). -
Any other response is to be considered as an error, and it is up to the client to decide whether the request should be retried or not. Implementors are encouraged to follow standard HTTP error status code guidelines.
batch
replies for partially uploaded content
When content was already partially uploaded, the server is expected to return a normal reply but omit request and parts
which do not need to be repeated. If the entire file has been uploaded, it is expected that no actions
value will be
returned, in which case clients should simply skip the upload.
However, if parts of the file were successfully uploaded while others weren't, it is expected that a normal reply would
be returned, but with less parts
to send.
verify
HTTP 409 errors
An HTTP 409
error on verify
requests typically indicates that the file could not be fully committed or verified.
In this case, clients should follow the following process to try and recover from the error:
- retry the
batch
request to see if any parts of the file were not uploaded yet. If there are stillparts
to upload (i.e.parts
is not empty), proceed to upload them and re-doverify
- If
parts
is empty, it is possible that the file exists in storage but is corrupt / has wrong size. In this case it is recommended to issue anabort
and re-attempt the same upload again - It is recommended to take special note of the number of retries, to avoid infinite recovery attempt loops
Additional Considerations
Chunk sizing
It is up to the LFS server to decide the size of each file chunk.
Action lifetime considerations
As multipart uploads tend to require much more time than simple uploads, it is recommended to allow for longer expires_in
values than one would consider for basic
uploads. It is possible that the process of uploading a single object in multiple
parts may take several hours from batch
to verify
.
Falling back to basic
transfer for small files
Using multipart upload APIs has some complexity and speed overhead. For this reason, if a client specifies support for
both multipart
and basic
transfer modes in a batch request, and the object(s) uploaded are small enough to fit in a
single part upload, servers may choose to respond with a basic
transfer mode even if multipart
is supported:
For example a small (2mb) upload batch request:
{
"transfers": ["multipart", "basic"],
"operation": "upload",
"objects": [
{
"oid": "13aea96040f2133033d103008d5d96cfe98b3361f7202d77bea97b2424a7a6cd",
"size": 2000000
}
]
}
May be responded with:
{
"transfer": "basic",
"objects": [
...
]
}
Even if the server does support multipart
, as basic
can be preferable in this case.
Implementation Notes
Hiding initialization / commit complexities from clients
While part
requests are typically quite similar between vendors, the specifics of multipart upload initialization and
commit procedures are very specific to vendors. For this reason, in many cases, it will be up to the LFS server to
take care of initialization and commit code. This is fine, as long as actual uploaded data is sent directly to the
storage backend.
For example, in the case of Amazon S3:
- All requests need to have an "upload ID" token which is obtained in an initial request
- When finalizing the upload, a special "commit" request need to be sent, listing all uploaded part IDs.
These are very hard to abstract in a way that would allow clients to send them directly to the server. In addition, as we do not want to maintain any state in the server, there is a need to make two requests when finalizing the upload: one to fetch a list of uploaded chunks, and another to send this list to the S3 finalization endpoint.
For this reason, it is expected that any initialization actions will be handled by the Git LFS server during the
batch
request handling. In most cases, the verify
action will also be responsible for any finalization / commit actions.
The params
attribute of the verify
action is designed specifically to transfer some vendor-specific "state" between
initialization and finalization of the upload process.
Implementing Complex abort
actions
Some storage backends will accept a simple DELETE
or POST
request to a URL, with no request body, in order to abort
the upload. In such cases, abort
may refer directly to the storage backend. However, in cases where aborting the upload
requires more complex logic or some payload in the request body, abort
actions should point to an endpoint of the LFS
server, and it should be up to the LFS server to abort the upload and clean up any partially uploaded parts.
As abort
requests do not have a body, any parameters required by the LFS server in order to complete the request should
be passed as part of the URL in the href
parameter.
It should be noted that clients will not always be able to abort
partial uploads cleanly. Implementors are expected to
ensure proper cleanup of partially uploaded files via other means, such as a periodical cron job that locates uncommitted
uploaded parts and deletes them.