git-lfs/tq
brian m. carlson 9c46a38281
ssh: support concurrent transfers using the pure SSH protocol
When using the pure SSH-based protocol, we can get much higher speeds by
multiplexing multiple connections on the same SSH connection.  If we're
using OpenSSH, let's enable the ControlMaster option unless
lfs.ssh.automultiplex is set to false, and multiplex these shell
operations over one connection.

We prefer XDG_RUNTIME_DIR because it's guaranteed to be private and we
can share many connections over one socket, but if that's not set, let's
default to creating a new temporary directory for the socket.  On
Windows, where the native SSH client doesn't support ControlMaster,
we should fall back to using multiple connections since we use
ControlMaster=auto.

Note that the option exists because users may already be using SSH
multiplexing and we would want to provide a way for them to disable
this, in addition to the case where users have an old or broken OpenSSH
which cannot support this option.

We pass the connection object into each worker and adjust our transfer
code to pass it into each function we invoke.  We also make sure to
properly terminate each connection at the end by reducing our connection
count to 0, which closes the extra (i.e., all) connections.

Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-07-20 19:15:59 +00:00
..
schemas tq,lfsapi/ssh: convert int64 to int 2017-04-05 14:46:45 -06:00
adapterbase.go Fix error strings to follow Go guidelines 2019-10-22 17:33:49 +03:00
api_test.go lfsapi: extract new lfshttp package 2018-09-11 14:51:29 -07:00
api.go tq: turn the batch client into an interface 2021-07-20 18:39:04 +00:00
basic_download.go tq: implement a pure SSH-based protocol for transfers 2021-07-20 19:15:40 +00:00
basic_upload.go fix upload retry 'file already closed' issue' 2020-02-26 11:13:54 +08:00
custom_test.go lfsapi: extract new lfshttp package 2018-09-11 14:51:29 -07:00
custom.go Fix error strings to follow Go guidelines 2019-10-22 17:33:49 +03:00
errors_test.go tq/errors: implement MalformedObjectError 2017-03-24 14:36:59 -06:00
errors.go tq/errors: implement MalformedObjectError 2017-03-24 14:36:59 -06:00
manifest_test.go Remove NTLM support 2021-02-02 16:41:41 +00:00
manifest.go ssh: support concurrent transfers using the pure SSH protocol 2021-07-20 19:15:59 +00:00
meter.go Use proper repo permissions when creating directories 2018-12-13 17:51:02 +00:00
ssh.go ssh: support concurrent transfers using the pure SSH protocol 2021-07-20 19:15:59 +00:00
transfer_queue_test.go tq: add exponential backoff for retries 2020-04-15 14:10:16 -07:00
transfer_queue.go ssh: support concurrent transfers using the pure SSH protocol 2021-07-20 19:15:59 +00:00
transfer_test.go lfsapi: extract new lfshttp package 2018-09-11 14:51:29 -07:00
transfer.go tq: implement a pure SSH-based protocol for transfers 2021-07-20 19:15:40 +00:00
tus_upload.go Fix error strings to follow Go guidelines 2019-10-22 17:33:49 +03:00
verify_test.go tq: use correct access 2018-09-28 14:16:55 -07:00
verify.go tq: ensure we pass the correct Accept header in verify requests 2019-05-22 21:01:54 +00:00