Workspace databases ride out dropped connections

Now and then a request from an app or workflow to a workspace database was lost on the way. A network connection that had sat idle was dropped without notice, and the next request on it failed. Reads were tried again, but a write was never sent twice, so an app showed that it couldn't reach its database and a workflow step could fail. A request that vanished entirely waited out the full 30 seconds before giving up. Now a request the connection lost is sent again within seconds on a new connection, and a write lands exactly once however many times it is sent.

How it works

• Every request carries an id. A query or batch keeps one id across all its attempts. The database saves a write's result under that id together with the write itself, so a resent write is answered with its first result instead of running again.
• A silent request gets a second copy after 2 seconds. If a request has no reply within 2 seconds, we send a copy on a new connection while the first keeps going, and use whichever answers first. A slow query is never cut off, and a write still lands once.
• Refusals stay refusals. A write the database refused, like a duplicate key, gets the same refusal when it is resent, even if it would succeed by then.

This builds on how our outages wait, not fail.