2021-02-03 22:27:01 +00:00
|
|
|
package ssh
|
|
|
|
|
|
|
|
import (
|
|
|
|
"fmt"
|
ssh: support concurrent transfers using the pure SSH protocol
When using the pure SSH-based protocol, we can get much higher speeds by
multiplexing multiple connections on the same SSH connection. If we're
using OpenSSH, let's enable the ControlMaster option unless
lfs.ssh.automultiplex is set to false, and multiplex these shell
operations over one connection.
We prefer XDG_RUNTIME_DIR because it's guaranteed to be private and we
can share many connections over one socket, but if that's not set, let's
default to creating a new temporary directory for the socket. On
Windows, where the native SSH client doesn't support ControlMaster,
we should fall back to using multiple connections since we use
ControlMaster=auto.
Note that the option exists because users may already be using SSH
multiplexing and we would want to provide a way for them to disable
this, in addition to the case where users have an old or broken OpenSSH
which cannot support this option.
We pass the connection object into each worker and adjust our transfer
code to pass it into each function we invoke. We also make sure to
properly terminate each connection at the end by reducing our connection
count to 0, which closes the extra (i.e., all) connections.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-04-06 14:08:06 +00:00
|
|
|
"io/ioutil"
|
|
|
|
"os"
|
2021-02-03 22:27:01 +00:00
|
|
|
"path/filepath"
|
|
|
|
"regexp"
|
|
|
|
"strings"
|
ssh: support concurrent transfers using the pure SSH protocol
When using the pure SSH-based protocol, we can get much higher speeds by
multiplexing multiple connections on the same SSH connection. If we're
using OpenSSH, let's enable the ControlMaster option unless
lfs.ssh.automultiplex is set to false, and multiplex these shell
operations over one connection.
We prefer XDG_RUNTIME_DIR because it's guaranteed to be private and we
can share many connections over one socket, but if that's not set, let's
default to creating a new temporary directory for the socket. On
Windows, where the native SSH client doesn't support ControlMaster,
we should fall back to using multiple connections since we use
ControlMaster=auto.
Note that the option exists because users may already be using SSH
multiplexing and we would want to provide a way for them to disable
this, in addition to the case where users have an old or broken OpenSSH
which cannot support this option.
We pass the connection object into each worker and adjust our transfer
code to pass it into each function we invoke. We also make sure to
properly terminate each connection at the end by reducing our connection
count to 0, which closes the extra (i.e., all) connections.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-04-06 14:08:06 +00:00
|
|
|
"syscall"
|
2021-02-03 22:27:01 +00:00
|
|
|
|
2021-08-10 06:18:38 +00:00
|
|
|
"github.com/git-lfs/git-lfs/v2/config"
|
|
|
|
"github.com/git-lfs/git-lfs/v2/subprocess"
|
|
|
|
"github.com/git-lfs/git-lfs/v2/tools"
|
2021-02-03 22:27:01 +00:00
|
|
|
"github.com/rubyist/tracerx"
|
|
|
|
)
|
|
|
|
|
ssh: implement Git SSH variant selection
We'd implemented Git's autodetection of SSH variants in the past, but we
hadn't implemented the newer explicit SSH selection. Since we're about
to pass some additional options to our ssh binaries, let's implement the
proper variant using GIT_SSH_VARIANT and ssh.variant.
Roughly, the algorithm is this: GIT_SSH_VARIANT overrides ssh.variant,
and if neither is set, autodetection occurs. A user can specify either
"ssh" (OpenSSH), "putty" (or its synonym "plink"), "tortoiseplink", or
"simple", or, if they'd like the autodetection behavior, "auto". If the
value is not any of these, then it is interpreted as "ssh".
Remove the special-casing of certain command names in favor of the
variant type.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-03-30 17:05:22 +00:00
|
|
|
type sshVariant string
|
|
|
|
|
|
|
|
const (
|
|
|
|
variantSSH = sshVariant("ssh")
|
|
|
|
variantSimple = sshVariant("simple")
|
|
|
|
variantPutty = sshVariant("putty")
|
|
|
|
variantTortoise = sshVariant("tortoiseplink")
|
|
|
|
)
|
|
|
|
|
2021-02-03 22:27:01 +00:00
|
|
|
type SSHMetadata struct {
|
|
|
|
UserAndHost string
|
|
|
|
Port string
|
|
|
|
Path string
|
|
|
|
}
|
|
|
|
|
|
|
|
func FormatArgs(cmd string, args []string, needShell bool) (string, []string) {
|
|
|
|
if !needShell {
|
|
|
|
return cmd, args
|
|
|
|
}
|
|
|
|
|
|
|
|
return subprocess.FormatForShellQuotedArgs(cmd, args)
|
|
|
|
}
|
|
|
|
|
ssh: support concurrent transfers using the pure SSH protocol
When using the pure SSH-based protocol, we can get much higher speeds by
multiplexing multiple connections on the same SSH connection. If we're
using OpenSSH, let's enable the ControlMaster option unless
lfs.ssh.automultiplex is set to false, and multiplex these shell
operations over one connection.
We prefer XDG_RUNTIME_DIR because it's guaranteed to be private and we
can share many connections over one socket, but if that's not set, let's
default to creating a new temporary directory for the socket. On
Windows, where the native SSH client doesn't support ControlMaster,
we should fall back to using multiple connections since we use
ControlMaster=auto.
Note that the option exists because users may already be using SSH
multiplexing and we would want to provide a way for them to disable
this, in addition to the case where users have an old or broken OpenSSH
which cannot support this option.
We pass the connection object into each worker and adjust our transfer
code to pass it into each function we invoke. We also make sure to
properly terminate each connection at the end by reducing our connection
count to 0, which closes the extra (i.e., all) connections.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-04-06 14:08:06 +00:00
|
|
|
func GetLFSExeAndArgs(osEnv config.Environment, gitEnv config.Environment, meta *SSHMetadata, command, operation string, multiplexDesired bool) (string, []string) {
|
|
|
|
exe, args, needShell := GetExeAndArgs(osEnv, gitEnv, meta, multiplexDesired)
|
2021-03-17 18:09:17 +00:00
|
|
|
args = append(args, fmt.Sprintf("%s %s %s", command, meta.Path, operation))
|
2021-02-03 22:27:01 +00:00
|
|
|
exe, args = FormatArgs(exe, args, needShell)
|
|
|
|
tracerx.Printf("run_command: %s %s", exe, strings.Join(args, " "))
|
|
|
|
return exe, args
|
|
|
|
}
|
|
|
|
|
|
|
|
// Parse command, and if it looks like a valid command, return the ssh binary
|
|
|
|
// name, the command to run, and whether we need a shell. If not, return
|
|
|
|
// existing as the ssh binary name.
|
|
|
|
func parseShellCommand(command string, existing string) (ssh string, cmd string, needShell bool) {
|
|
|
|
ssh = existing
|
|
|
|
if cmdArgs := tools.QuotedFields(command); len(cmdArgs) > 0 {
|
|
|
|
needShell = true
|
|
|
|
ssh = cmdArgs[0]
|
|
|
|
cmd = command
|
|
|
|
}
|
|
|
|
return
|
|
|
|
}
|
|
|
|
|
ssh: implement Git SSH variant selection
We'd implemented Git's autodetection of SSH variants in the past, but we
hadn't implemented the newer explicit SSH selection. Since we're about
to pass some additional options to our ssh binaries, let's implement the
proper variant using GIT_SSH_VARIANT and ssh.variant.
Roughly, the algorithm is this: GIT_SSH_VARIANT overrides ssh.variant,
and if neither is set, autodetection occurs. A user can specify either
"ssh" (OpenSSH), "putty" (or its synonym "plink"), "tortoiseplink", or
"simple", or, if they'd like the autodetection behavior, "auto". If the
value is not any of these, then it is interpreted as "ssh".
Remove the special-casing of certain command names in favor of the
variant type.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-03-30 17:05:22 +00:00
|
|
|
func findVariant(variant string) (bool, sshVariant) {
|
|
|
|
switch variant {
|
|
|
|
case "ssh", "simple", "putty", "tortoiseplink":
|
|
|
|
return false, sshVariant(variant)
|
|
|
|
case "plink":
|
|
|
|
return false, variantPutty
|
|
|
|
case "auto":
|
|
|
|
return true, ""
|
|
|
|
default:
|
|
|
|
return false, variantSSH
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func autodetectVariant(osEnv config.Environment, gitEnv config.Environment, basessh string) sshVariant {
|
|
|
|
if basessh != defaultSSHCmd {
|
|
|
|
// Strip extension for easier comparison
|
|
|
|
if ext := filepath.Ext(basessh); len(ext) > 0 {
|
|
|
|
basessh = basessh[:len(basessh)-len(ext)]
|
|
|
|
}
|
|
|
|
if strings.EqualFold(basessh, "plink") {
|
|
|
|
return variantPutty
|
|
|
|
}
|
|
|
|
if strings.EqualFold(basessh, "tortoiseplink") {
|
|
|
|
return variantTortoise
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return "ssh"
|
|
|
|
}
|
|
|
|
|
|
|
|
func getVariant(osEnv config.Environment, gitEnv config.Environment, basessh string) sshVariant {
|
|
|
|
variant, ok := osEnv.Get("GIT_SSH_VARIANT")
|
|
|
|
if !ok {
|
|
|
|
variant, ok = gitEnv.Get("ssh.variant")
|
|
|
|
}
|
|
|
|
autodetect, val := findVariant(variant)
|
|
|
|
if ok && !autodetect {
|
|
|
|
return val
|
|
|
|
}
|
|
|
|
return autodetectVariant(osEnv, gitEnv, basessh)
|
|
|
|
}
|
|
|
|
|
ssh: support concurrent transfers using the pure SSH protocol
When using the pure SSH-based protocol, we can get much higher speeds by
multiplexing multiple connections on the same SSH connection. If we're
using OpenSSH, let's enable the ControlMaster option unless
lfs.ssh.automultiplex is set to false, and multiplex these shell
operations over one connection.
We prefer XDG_RUNTIME_DIR because it's guaranteed to be private and we
can share many connections over one socket, but if that's not set, let's
default to creating a new temporary directory for the socket. On
Windows, where the native SSH client doesn't support ControlMaster,
we should fall back to using multiple connections since we use
ControlMaster=auto.
Note that the option exists because users may already be using SSH
multiplexing and we would want to provide a way for them to disable
this, in addition to the case where users have an old or broken OpenSSH
which cannot support this option.
We pass the connection object into each worker and adjust our transfer
code to pass it into each function we invoke. We also make sure to
properly terminate each connection at the end by reducing our connection
count to 0, which closes the extra (i.e., all) connections.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-04-06 14:08:06 +00:00
|
|
|
// findRuntimeDir returns a path to the runtime directory if one exists and is
|
|
|
|
// guaranteed to be private.
|
|
|
|
func findRuntimeDir(osEnv config.Environment) string {
|
|
|
|
if dir, ok := osEnv.Get("XDG_RUNTIME_DIR"); ok {
|
|
|
|
return dir
|
|
|
|
}
|
|
|
|
return ""
|
|
|
|
}
|
|
|
|
|
|
|
|
func getControlDir(osEnv config.Environment) (string, error) {
|
|
|
|
dir := findRuntimeDir(osEnv)
|
|
|
|
if dir == "" {
|
|
|
|
return ioutil.TempDir("", "sock-*")
|
|
|
|
}
|
|
|
|
dir = filepath.Join(dir, "git-lfs")
|
|
|
|
err := os.Mkdir(dir, 0700)
|
|
|
|
if err != nil {
|
|
|
|
// Ideally we would use errors.Is here to check against
|
|
|
|
// os.ErrExist, but that's not available on Go 1.11.
|
|
|
|
perr, ok := err.(*os.PathError)
|
|
|
|
if !ok || perr.Err != syscall.EEXIST {
|
|
|
|
return ioutil.TempDir("", "sock-*")
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return dir, nil
|
|
|
|
}
|
|
|
|
|
2021-02-03 22:27:01 +00:00
|
|
|
// Return the executable name for ssh on this machine and the base args
|
|
|
|
// Base args includes port settings, user/host, everything pre the command to execute
|
ssh: support concurrent transfers using the pure SSH protocol
When using the pure SSH-based protocol, we can get much higher speeds by
multiplexing multiple connections on the same SSH connection. If we're
using OpenSSH, let's enable the ControlMaster option unless
lfs.ssh.automultiplex is set to false, and multiplex these shell
operations over one connection.
We prefer XDG_RUNTIME_DIR because it's guaranteed to be private and we
can share many connections over one socket, but if that's not set, let's
default to creating a new temporary directory for the socket. On
Windows, where the native SSH client doesn't support ControlMaster,
we should fall back to using multiple connections since we use
ControlMaster=auto.
Note that the option exists because users may already be using SSH
multiplexing and we would want to provide a way for them to disable
this, in addition to the case where users have an old or broken OpenSSH
which cannot support this option.
We pass the connection object into each worker and adjust our transfer
code to pass it into each function we invoke. We also make sure to
properly terminate each connection at the end by reducing our connection
count to 0, which closes the extra (i.e., all) connections.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-04-06 14:08:06 +00:00
|
|
|
func GetExeAndArgs(osEnv config.Environment, gitEnv config.Environment, meta *SSHMetadata, multiplexDesired bool) (exe string, baseargs []string, needShell bool) {
|
2021-02-03 22:27:01 +00:00
|
|
|
var cmd string
|
|
|
|
|
|
|
|
ssh, _ := osEnv.Get("GIT_SSH")
|
|
|
|
sshCmd, _ := osEnv.Get("GIT_SSH_COMMAND")
|
|
|
|
ssh, cmd, needShell = parseShellCommand(sshCmd, ssh)
|
|
|
|
|
|
|
|
if ssh == "" {
|
|
|
|
sshCmd, _ := gitEnv.Get("core.sshcommand")
|
|
|
|
ssh, cmd, needShell = parseShellCommand(sshCmd, defaultSSHCmd)
|
|
|
|
}
|
|
|
|
|
|
|
|
if cmd == "" {
|
|
|
|
cmd = ssh
|
|
|
|
}
|
|
|
|
|
|
|
|
basessh := filepath.Base(ssh)
|
ssh: implement Git SSH variant selection
We'd implemented Git's autodetection of SSH variants in the past, but we
hadn't implemented the newer explicit SSH selection. Since we're about
to pass some additional options to our ssh binaries, let's implement the
proper variant using GIT_SSH_VARIANT and ssh.variant.
Roughly, the algorithm is this: GIT_SSH_VARIANT overrides ssh.variant,
and if neither is set, autodetection occurs. A user can specify either
"ssh" (OpenSSH), "putty" (or its synonym "plink"), "tortoiseplink", or
"simple", or, if they'd like the autodetection behavior, "auto". If the
value is not any of these, then it is interpreted as "ssh".
Remove the special-casing of certain command names in favor of the
variant type.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-03-30 17:05:22 +00:00
|
|
|
variant := getVariant(osEnv, gitEnv, basessh)
|
2021-02-03 22:27:01 +00:00
|
|
|
|
|
|
|
args := make([]string, 0, 7)
|
|
|
|
|
ssh: implement Git SSH variant selection
We'd implemented Git's autodetection of SSH variants in the past, but we
hadn't implemented the newer explicit SSH selection. Since we're about
to pass some additional options to our ssh binaries, let's implement the
proper variant using GIT_SSH_VARIANT and ssh.variant.
Roughly, the algorithm is this: GIT_SSH_VARIANT overrides ssh.variant,
and if neither is set, autodetection occurs. A user can specify either
"ssh" (OpenSSH), "putty" (or its synonym "plink"), "tortoiseplink", or
"simple", or, if they'd like the autodetection behavior, "auto". If the
value is not any of these, then it is interpreted as "ssh".
Remove the special-casing of certain command names in favor of the
variant type.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-03-30 17:05:22 +00:00
|
|
|
if variant == variantTortoise {
|
2021-02-03 22:27:01 +00:00
|
|
|
// TortoisePlink requires the -batch argument to behave like ssh/plink
|
|
|
|
args = append(args, "-batch")
|
|
|
|
}
|
|
|
|
|
ssh: support concurrent transfers using the pure SSH protocol
When using the pure SSH-based protocol, we can get much higher speeds by
multiplexing multiple connections on the same SSH connection. If we're
using OpenSSH, let's enable the ControlMaster option unless
lfs.ssh.automultiplex is set to false, and multiplex these shell
operations over one connection.
We prefer XDG_RUNTIME_DIR because it's guaranteed to be private and we
can share many connections over one socket, but if that's not set, let's
default to creating a new temporary directory for the socket. On
Windows, where the native SSH client doesn't support ControlMaster,
we should fall back to using multiple connections since we use
ControlMaster=auto.
Note that the option exists because users may already be using SSH
multiplexing and we would want to provide a way for them to disable
this, in addition to the case where users have an old or broken OpenSSH
which cannot support this option.
We pass the connection object into each worker and adjust our transfer
code to pass it into each function we invoke. We also make sure to
properly terminate each connection at the end by reducing our connection
count to 0, which closes the extra (i.e., all) connections.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-04-06 14:08:06 +00:00
|
|
|
multiplexEnabled := gitEnv.Bool("lfs.ssh.automultiplex", true)
|
|
|
|
if variant == variantSSH && multiplexDesired && multiplexEnabled {
|
|
|
|
controlPath, err := getControlDir(osEnv)
|
|
|
|
if err != nil {
|
|
|
|
controlPath = filepath.Join(controlPath, "sock-%C")
|
|
|
|
args = append(args, "-oControlMaster=auto", fmt.Sprintf("-oControlPath=%s", controlPath))
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2021-02-03 22:27:01 +00:00
|
|
|
if len(meta.Port) > 0 {
|
ssh: implement Git SSH variant selection
We'd implemented Git's autodetection of SSH variants in the past, but we
hadn't implemented the newer explicit SSH selection. Since we're about
to pass some additional options to our ssh binaries, let's implement the
proper variant using GIT_SSH_VARIANT and ssh.variant.
Roughly, the algorithm is this: GIT_SSH_VARIANT overrides ssh.variant,
and if neither is set, autodetection occurs. A user can specify either
"ssh" (OpenSSH), "putty" (or its synonym "plink"), "tortoiseplink", or
"simple", or, if they'd like the autodetection behavior, "auto". If the
value is not any of these, then it is interpreted as "ssh".
Remove the special-casing of certain command names in favor of the
variant type.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-03-30 17:05:22 +00:00
|
|
|
if variant == variantPutty || variant == variantTortoise {
|
2021-02-03 22:27:01 +00:00
|
|
|
args = append(args, "-P")
|
|
|
|
} else {
|
|
|
|
args = append(args, "-p")
|
|
|
|
}
|
|
|
|
args = append(args, meta.Port)
|
|
|
|
}
|
|
|
|
|
ssh: implement Git SSH variant selection
We'd implemented Git's autodetection of SSH variants in the past, but we
hadn't implemented the newer explicit SSH selection. Since we're about
to pass some additional options to our ssh binaries, let's implement the
proper variant using GIT_SSH_VARIANT and ssh.variant.
Roughly, the algorithm is this: GIT_SSH_VARIANT overrides ssh.variant,
and if neither is set, autodetection occurs. A user can specify either
"ssh" (OpenSSH), "putty" (or its synonym "plink"), "tortoiseplink", or
"simple", or, if they'd like the autodetection behavior, "auto". If the
value is not any of these, then it is interpreted as "ssh".
Remove the special-casing of certain command names in favor of the
variant type.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-03-30 17:05:22 +00:00
|
|
|
if variant == variantSSH {
|
2021-02-03 22:27:01 +00:00
|
|
|
// inserts a separator between cli -options and host/cmd commands
|
|
|
|
// example: $ ssh -p 12345 -- user@host.com git-lfs-authenticate ...
|
ssh: implement Git SSH variant selection
We'd implemented Git's autodetection of SSH variants in the past, but we
hadn't implemented the newer explicit SSH selection. Since we're about
to pass some additional options to our ssh binaries, let's implement the
proper variant using GIT_SSH_VARIANT and ssh.variant.
Roughly, the algorithm is this: GIT_SSH_VARIANT overrides ssh.variant,
and if neither is set, autodetection occurs. A user can specify either
"ssh" (OpenSSH), "putty" (or its synonym "plink"), "tortoiseplink", or
"simple", or, if they'd like the autodetection behavior, "auto". If the
value is not any of these, then it is interpreted as "ssh".
Remove the special-casing of certain command names in favor of the
variant type.
Co-authored-by: Chris Darroch <chrisd8088@github.com>
2021-03-30 17:05:22 +00:00
|
|
|
args = append(args, "--", meta.UserAndHost)
|
2021-02-03 22:27:01 +00:00
|
|
|
} else {
|
|
|
|
// no prefix supported, strip leading - off host to prevent cmd like:
|
|
|
|
// $ git config lfs.url ssh://-proxycmd=whatever
|
|
|
|
// $ plink -P 12345 -proxycmd=foo git-lfs-authenticate ...
|
|
|
|
//
|
|
|
|
// Instead, it'll attempt this, and eventually return an error
|
|
|
|
// $ plink -P 12345 proxycmd=foo git-lfs-authenticate ...
|
|
|
|
args = append(args, sshOptPrefixRE.ReplaceAllString(meta.UserAndHost, ""))
|
|
|
|
}
|
|
|
|
|
|
|
|
return cmd, args, needShell
|
|
|
|
}
|
|
|
|
|
|
|
|
const defaultSSHCmd = "ssh"
|
|
|
|
|
|
|
|
var (
|
|
|
|
sshOptPrefixRE = regexp.MustCompile(`\A\-+`)
|
|
|
|
)
|