As hinted by an earlier post, one of my latest work projects was a building a WebRTC based video streaming system. This system features a Websocket backend server handling the client sessions and signaling, written in Python with Gevent. My co-worker was performing some testing and everything seemed to be working just fine. Except one point he encountered a situation, where the server insisted another client was already connected, when it clearly wasn’t. Even netstat said that the TCP connection was ‘established’.
Some Websocket client connections were arbitrarily staying open even if the browser was closed! I had just added some hooks to the server to better detect disconnected / timed-out clients and a good guess was that I had messed something up. Nope. Debugged the thing for hours but couldn’t find single bug.
That is, until I tried toggling network connections. Being a system targeted for mobile devices, one facet of testing is to check how well it works with different network connections. If the network connection was turned of while the client was connected, the server couldn’t detect that until after about ten to fifteen minutes, even though it was sending keep-alive packets every 20 seconds. Strange indeed.
But maybe it wasn’t, maybe the write call was blocking the specific greenlet? That is an easy thing to test, just dump some stack traces. But nope again. How about if I run the server via strace and try to spot the writes there? It took bit of an effort, but the strace output revealed that the write calls were performed just fine! This is starting to be most troubling…
But then a revelation; write returned EPIPE. After quite a bit of research I had finally found the reason for this behavior: TCP timeouts. Turning off the network connection really did what it did. It turned off the connection without giving the protocol stack time to say its goodbyes. The server then though the client just had a really bad connection and tried resending the queued data with an exponential delay back off per TCP spec. My math didn’t quite match, but in an old thread the total timeout was calculated to be 924.6 seconds with the default TCP parameters. This was quite close to the actual timeout observed with the server.
* * *
I sighed and changed the keep-alive protocol so that timely replies were required instead of just relying on failed writes. Now it works beautifully.
Tl;dr: TCP was just really hard trying to resend data after it detected packet loss, only giving up when about fifteen whole minutes had passed.
 |
| That strange character is actually FE 80 00, rest of the address not rendering as a string. |
Having worked on this document for several(maybe even a dozen) hours over the past two years, I feel it’s finally time to let the world see how I have big plans. Big plans that will probably never be fulfilled. Actually I don’t expect to get everything done, but rather see it as something to work towards to.
There is also some videos on my YouTube channel.
* * *
For those unfamiliar with my projects: USG is the newest incarnation of my long-running series of space themed 2D games.
Had a short discussion about my game project today. Short story shorter, I didn’t have any material to show what is was like years ago.
Now I do. It was surprisingly simple to get most of the versions to run like they used to. The only real challenge was eliminating the dependency for an asset server; at that time one of my interests was streaming patching and bigfiles as in Guild Wars. Of course I had to implement my own version to a ‘real’ game. After some asset hunting and code patching the games successfully loaded local files and I could finally record the videos in question.
Except one. I had some serious issues with font rendering in one of the versions(pygame either segfaults or freezes without error) and in the end had to replace the fonts with placeholder squares; no idea why in broke and no real reason to fix them.
Been silent as usual, but now I have SVG-based polygon map(Inkscape as preferred editor) and collision detection based on movement vectors(even high-speed objects collide to map).
Collision detection can be run in pure python(only ~120 colls/s on 108 polygon map…), or optionally accelerated with numpy(~10k colls/s) or external c++ app(~140k colls/s(that is almost 4.5M vector-vector pairs per second!) when compiled with vc++ 2010).
All values on single i7 930 thread(my new system(frostfire), with HD5850, 6GB RAM and 24” full-hd display).
The Peli is feeling just fine and has once again been re-written, this time Python(what would you except) and Pyglet(aka OpenGL). Now it is tenth or eleventh rewrite, depending on counting.
Features include
