That's really helpful. Thank you! MOSH might work, I'll have to play around with it.
Could you go into more detail about the tmux functions? If it's a way to write everything to files instead of a STDOUT in a predictable way, that would be great, since each packet could be a (compressed) shell script that explicitly includes which data to send back, if any.