It is possible the remote machine is freezing at the command line level and ssh isnt really effected or involved in any way BTW
Only time I've seen that happen is on my laptop when it goes to sleep. Since packets are dropped while asleep when it wakes back up any ssh connections tend to be frozen if it was any appreciable amount of time.
@EpicKitty I'll try to set up keepalive to minute and see if it changes. Manual says it doesn't even send keepalive by default O_o.