when there is a delay

when there is a delay between the remote and the local computer, is there a way of knowing whether the delay is due to one or the other computers.  I work on Macs and sometimes I get the proverbial spinning beachball and I can't tell whether it's my local or remote computer that needs to be restarted to eliminate the delay.