[midnightveil]

Previously every remote command e.g. mq.sh sem dumpall would SSH to tftp.keg.cse.unsw.edu.au multiple times. Even with ControlMastering, this still needs to open lots of login shells, which for us tends to be very slow. Instead, if we re-run the entire mq.sh command on the remote host, then we only need to SSH once.

We don't do this for run, however, because it needs to copy files back and forth. It could definitely be done, however, it is more effort to do so. Hopefully, this would reduce the load on our servers and make 'run' less intensive.

[midnightveil]

For me this makes mq.sh sem dumpall almost instant, even locally, whereas before it could be 10-20s+. It used to check every lock separately. This matches closer to what happens when I do ssh tftp.keg "mq.sh sem dumpall", which I've been resorting to doing.

[lsf37]

Overall I think this is a good idea, but I'm unsure how this behaves with Ctrl-C behaviour. At least on sem -wait I do use that from time to time, and it's unclear to me if the cleanup still works the same as before or if this will leave a stale lockfile.

Maybe it's just sem -wait, I don't think there is anything else that is blocking/waiting.

[midnightveil]

Ctrl+C .... should pass through to ssh. But you're right that if it just interrupts the connection without doing cleanup...

I could explicitly blocklist -wait?

[midnightveil]

Right, it only does it with -tt:

julia@eta-carinae ~> ssh tftp.keg "bash /tmp/tmp.EvtFAHUbr5/test.sh"
^C
julia@eta-carinae ~ [255]> ssh -tt tftp.keg "bash /tmp/tmp.EvtFAHUbr5/test.sh"
^Chiiiiiiiiiiiiiiii
hiiiiiiiiiiiiiiii
Shared connection to tftp.keg.cse.unsw.edu.au closed.
julia@eta-carinae ~ [SIGINT]> ssh -tt tftp.keg "cat /tmp/tmp.EvtFAHUbr5/test.sh"
hello() {
  echo hiiiiiiiiiiiiiiii
}

trap hello EXIT INT

sleep 100

[Indanz]

For me this makes mq.sh sem dumpall almost instant, even locally, whereas before it could be 10-20s+.

I thought it took so long because I'm on the other side of the world...

[midnightveil]

Every single board it prints require 3 or 4 new ssh connections. Or, more accurately, since ControlMaster is set up, 3 or 4 new logins in linux (try emulating this with the login command). For some reason this is especially slow when the server is under high load (where load = lots of these at once).

I thought this was the wtmpdb ("w" tmp database) command because it did a bunch of sqlite3 db lock acquires (spin lick with exponential back off) but apparently that's disabled now

[Indanz]

One thing to be aware of is that the remote mq.sh script can be different than the local one, so this behaviour is not always wanted. So I think we may need an option to disable this for such cases or something.

Alternative is to send over the script + any input files via scp before execution or something. We can pipe one file as input, but no more.