Just enough long polling
Firebase launched a long time ago in a different Internet, one where Websockets were a relatively new feature for web browsers and support for them couldn't always be assured. Because of this, Firebase also includes support for a connection type called "Long Polling." This is where you issue a series of normal HTTP requests in a series which approximates the ability to send messages back and forth between teh server and client.
In the modern Internet, Websockets are very well supported in nearly all network conditions. However, when making sure that Lark worked with existing Firebase client SDKs, we quickly discovered that there were many situations where Firebase's JS SDK would swap over to Long Polling mode, and once that was activated, it actually wouldn't exit it (even between page reloads!) until a successful Long Polling session could be established, and a "switch connection type" message could be exchanged. This led us down a path of needing to build our own support for Long Polling...although really just enough to fake it.
How a client gets stuck
The Firebase JS SDK supports two transports, Websocket and long polling, and it picks between them in a way that turns out to be a lot stickier than you'd expect.
When the SDK opens a Websocket it writes a flag into localStorage called previous_websocket_failure, and it does this before it has even attempted the connection. The comment in the source reads "Assume failure until proven otherwise," which is a perfectly sensible way to write that code. The flag then gets cleared once the Websocket connection is considered healthy (defined as two responses inside of thirty seconds).
So if a Websocket gets interrupted at the wrong moment, usually due to a flaky overall network connection rather than a browser functionality issue, the flag is left sitting in localStorage. And since localStorage sticks around, so does the flag, right through the next page load.
On the next connection the SDK checks that flag, sees that Websocket previously failed, and puts long polling at the front of the transport list with Websocket sitting behind it as an upgrade. If your server doesn't speak long polling, the client never gets that working connection, never tries the upgrade, and just sits there polling an endpoint that doesn't exist.
The easier alternative
There's an alternative way out of this and we did consider going that route instead.
If your database URL starts with wss:// instead of https://, the Firebase JS SDK sets an internal webSocketOnly flag and drops long polling from the transport list altogether.
The trouble is that it's a change every existing app would have to make, and a big part of the point of Lark is that you change one line in your Firebase config and everything else keeps working.
There's also a whole group of users that wouldn't have helped anyway. The SDK's check for whether Websocket previously failed also comes back true when localStorage isn't available in the first place, on the reasonable theory that if it can't remember what happened, it should assume the worst. That covers Safari private browsing along with a fair number of embedded webviews, and those clients begin on long polling every single time with no failure required at all.
What's the minimum viable build?
Our goal was to implement just enough long polling so that we could support clients in this state without actually using it as a first-class transport.
"Just enough" turned out to be more than we'd initially assumed.
Firebase's long polling is older than a lot of the web platform and it rather shows. Responses come back as JSONP rather than JSON, so the server writes out a little script that calls pRTLPCB(packetNum, [messages]), which is what you had to do to get data across origins through a <script> tag back when this was designed. Sessions get established with a start=t request that hands back a session ID and a password, and every poll after that has to present both of them. Messages carry sequential packet numbers so the client can order them and ask to resume from a particular point.
Data heading the other direction is equally onerous. A poll is a GET request, so anything the client wants to send has to fit in the query string, which means it arrives base64 encoded and chopped across numbered parameters (seg0, ts0, d0, then seg1, ts1, d1, and onward) that the server collects and reassembles in order before it has anything resembling a message.
After that come the lifecycle details, like the disconnect frame and a "recently closed" set so that a poll arriving against a session we've just torn down receives a graceful close instead of an error (send an error and the SDK will simply keep trying).
None of it is difficult, exactly. It's just that you have to build most of the protocol before the client will trust you enough to let you tell it to stop using that protocol.
Fake upgrade then reset
Once a client is happily long polling it will eventually try to upgrade. It opens a Websocket with ?s=<sessionId> pointing at its long poll session, and it treats that new connection as a candidate rather than a replacement, because it wants to watch the connection prove that it's healthy before committing anything to it.
The clean way to handle that is a real session migration, where the server recognises the session ID, swaps the transport underneath, and the client never notices anything happened. We wrote that, and it's still sitting in the codebase, turned off, because it needs the long poll session and the Websocket to land on the same edge node and we didn't want to require supporting sticky sessions.
So we do something rather more of a shortcut instead, which we call "fake upgrade then reset," also known as "fake it til you make it."
When a client turns up with a session ID we don't know about, we accept the Websocket and immediately send it two pongs. Two, as noted earlier, is the number the SDK needs before it flags a connection as healthy, and clears previous_websocket_failure. The client, now satisfied that Websocket works after all, sends its switch acknowledgement. At which point we send it a reset and hang up, forcing it to reconnect from scratch.
That reconnect goes straight to Websocket, and we're back in business.
It is a slightly ridiculous bit of code. We implemented an entire transport in order to convince clients not to use it, and the way we accomplish that is to briefly lie and then hang up. The result is that a Firebase JS SDK app that hits a rough patch on a train journey quietly recovers on its own, which beats leaving a user stranded indefinitely unable to connect.
And that's just enough long polling!