Date: July 29, 2026
Duration: Approximately 4 hours (13:50–17:50 UTC / 9:50 AM–1:50 PM EDT)
What happened
From 13:50 to 17:50 UTC on July 29, the Jump Desktop API experienced elevated latency and intermittent request failures, including several intervals of complete unavailability. Requests that normally complete in a fraction of a second took far longer or failed.
During this time, users were unable to initiate new connections to machines, access the Jump Desktop Teams dashboard, or log in to their Jump Desktop accounts. Existing connections that had been established before the incident were not affected.
Service was fully restored by 17:50 UTC (1:50 PM EDT).
Why it happened
Two failures occurred during the incident, and the first helped set off the second.
The trigger. Routine maintenance rolled through our server fleet and restarted all application instances within a short period. Those restarts disconnected the computers connected to our service, and they all attempted to reconnect at roughly the same time. This mass reconnection, known as a thundering herd, produced a surge of database activity far above normal levels. At the same moment, each restarting instance was performing a database consistency check at startup, and that check takes locks on core database tables. Under normal conditions both behaviors are harmless: instances start at different times, each check completes in moments, and reconnections trickle in gradually. But with many instances starting concurrently, the startup checks queued behind one another, the reconnection surge queued behind the locks, and API latency and errors rose sharply.
The amplification loop. Our health-checking automation then made the problem self-sustaining. Instances delayed by the lock contention were judged unhealthy and replaced, and every replacement re-ran the same locking startup check, adding to the contention that had stalled its predecessor. The incident required a specific combination of conditions: restarts occurring in close succession across the fleet, a lock-taking startup check, and automated replacement of slow-starting instances. Our safeguards did not account for these mechanisms reinforcing one another, which is why the underlying defect had never surfaced before.
The second failure. While database processing was constrained, internal messages accumulated faster than they could be processed. The growing backlog drove up memory usage in the component that handles message delivery, and the added memory pressure impaired the machine hosting that component. Under normal circumstances an impaired machine is detected and replaced automatically. In this case the machine failed in a way that did not trip the health signals our automatic replacement systems rely on, so it was never flagged for replacement, and our engineers identified and replaced it manually. The impaired machine meant messages stopped being delivered, and because our services need message delivery to start up, the affected services could not fully recover until it was replaced. Replacing the machine restored message delivery, instances could start and stay running again, the wave of restarts subsided, and with it the database contention. Service returned to normal.
We have identified and confirmed the causes of both failures. The failure modes were reproduced end to end in a test environment and independently corroborated by our infrastructure provider’s analysis.
What we’re doing about it
Our post-incident review identified 21 corrective action items, 7 of which are top priority. They target every stage of the incident, including:
- Removing the lock-taking consistency check from instance startup, so simultaneous restarts can no longer contend for the database
- Changing how routine maintenance is performed to enable a broader, staggered restart of the fleet.
- Teaching our health checks to distinguish “still starting up” from “stuck,” so automation cannot create a restart loop
- Extending our automatic machine replacement systems to also catch the kind of silent failure seen in this incident
- Adding redundancy to the components this incident showed to be single points of failure
Work on these items is already underway.
For customers who need cloud-independent connectivity
Several customers have asked about connecting without any dependency on our cloud infrastructure. Jump Desktop supports this today through Cloudless Fluid, which establishes direct connections between your devices and does not rely on our cloud services for connection setup, access control, or relaying. Connections made this way would not have been affected by this incident. To learn more:
Note that Cloudless Fluid requires a Jump Desktop for Teams Enterprise plan and direct network reachability between the client and the remote machine. Our support team is happy to help you evaluate whether it fits your environment.
Using Cloudless Fluid during a cloud service outage
Cloudless Fluid requires an initial sign-in to establish the user’s entitlement. Once a user has successfully signed in and used Cloudless Fluid, the client retains that entitlement locally. If our cloud services later become unavailable, the user can continue making Cloudless Fluid connections without signing in again, provided they have not manually signed out of the client and discarded the locally stored token.
This means a previously configured Cloudless Fluid client can continue connecting during a cloud service outage, even if the client cannot reach Jump Desktop’s cloud services at that time.
We sincerely apologize for the disruption and the impact it had on you. We are committed to completing every one of these improvements.
The Jump Desktop Team
Comments
0 comments
Article is closed for comments.