frequency of messages sent by server to same client

[FlashHero]

I'm currently converting one of our action games to Union and today I noticed something disturbing. Somehow the movement of the characters was jerky at random times. After some digging I traced the problem back to two movement messages being sent in quick succession of eachother.

It seems Union server throttles the messages being sent? After several test I have concluded that messages sent to the same client within a 200ms window are pooled together and sent in "bursts" every 200ms. One can see how this would be problematic in an action game since 200ms can be too long for a movement update.

Is this throttling intentional and is there any way around it? If not is it something that will be available in the future?

Thank you in advance.

[FlashHero]

Not entirely sure if the problem was caused by Union or by the server running Union I wanted to make sure. So I built a very small java socket listener to see if the same 200ms pooling would occur. And yes indeed it did. After some more research I found that this behaviour is caused by a feature of TCP/IP: Nagle's algorithm (http://tools.ietf.org/html/rfc896).

The use of this bandwidth saving mechanism can be disabled through setTcpNoDelay(true) of the Socket object. I have verified that after disabling this the messages are delivered instantly, without delay.

Would it be possible to disable this for incoming connections on Union server? I can imagine other users would also find this helpful for high action games that require quick responses.

[colin]

it is definitely not our experience that messages are sent in 200ms batches. i just wrote a test client to connect to union server running on amazon ec2 and was able to consistently send 25 round-trip messages per second. here's the code (actionscript 3 + reactor), modified to connect to tryunion (which is a bit less responsive due to hardware and network configuration):

Code: Select all

package {
  import flash.display.Sprite;
  import flash.text.TextField;
  import flash.utils.setInterval;
 
  import net.user1.reactor.Client;
  import net.user1.reactor.Reactor;
  import net.user1.reactor.ReactorEvent;
 
  public class UnionTest1 extends Sprite {
    protected var reactor:Reactor;
    protected var lastTestReceived:uint;
    protected var output:TextField;
   
    public function UnionTest1 () {
      reactor = new Reactor();
      reactor.addEventListener(ReactorEvent.READY, readyListener);
      reactor.connect("tryunion.com", 80);
     
      output = new TextField();
      output.width = 400;
      output.height = 300;
      output.background = true;
      output.border = true;
     
      addChild(output);
    }
   
    public function readyListener (e:ReactorEvent):void {
      reactor.getMessageManager().addMessageListener("TEST", testMessageListener);
      reactor.self().sendMessage("TEST");
    }
   
    public function testMessageListener (fromClient:Client):void {
      var now:uint = new Date().time;
      output.appendText("Time since last TEST received: " + (now-lastTestReceived) + "\n");
      output.scrollV = output.maxScrollV;
      lastTestReceived = now;
      reactor.self().sendMessage("TEST");
    }
  }
}

here's a sample of my ec2 output:

Code: Select all

Time since last TEST received: 34
Time since last TEST received: 33
Time since last TEST received: 17
Time since last TEST received: 32
Time since last TEST received: 20
Time since last TEST received: 18
Time since last TEST received: 18
Time since last TEST received: 22
Time since last TEST received: 20
Time since last TEST received: 18
Time since last TEST received: 19
Time since last TEST received: 21
Time since last TEST received: 9
Time since last TEST received: 30
Time since last TEST received: 18
Time since last TEST received: 20
Time since last TEST received: 18
Time since last TEST received: 14
Time since last TEST received: 24
Time since last TEST received: 18
Time since last TEST received: 20
Time since last TEST received: 21
Time since last TEST received: 17
Time since last TEST received: 19
Time since last TEST received: 20
Time since last TEST received: 17
Time since last TEST received: 19
Time since last TEST received: 25
Time since last TEST received: 16
Time since last TEST received: 36
Time since last TEST received: 67
Time since last TEST received: 33
Time since last TEST received: 50
Time since last TEST received: 33
Time since last TEST received: 51
Time since last TEST received: 33
Time since last TEST received: 51
Time since last TEST received: 33
Time since last TEST received: 49
Time since last TEST received: 17
Time since last TEST received: 32
Time since last TEST received: 14
Time since last TEST received: 35
Time since last TEST received: 50
Time since last TEST received: 33
Time since last TEST received: 49
Time since last TEST received: 18
Time since last TEST received: 17


there is one possible explanation for your observation: you might be connected over http. union uses http as a transport mechanism when a direct socket connection cannot be established due to firewall restrictions. by default, when http is the current transport, reactor and orbiter both batch messages into 200ms groups. that 200ms delay is, however, configurable. for details, see:

http://unionplatform.com/docs/reactor/a ... SendDelay()

if you are 100% certain you are connected over WebSocket or XMLSocket, and have explicit steps and a test case to reproduce the issue you are seeing, please post the code/steps.

thanks,
colin

[FlashHero]

I'm absolute certain I'm connected using an xmlsocket. I ran your test and did not observe the 200ms batching however. I think this is due to the nature of how Nagle's algorithm works, it probably only kicks in when packets are sent in quick succession of eachother. If tcp packets are sent to same client within say 20ms (I do not know the exact number on this) it kicks in and sending the packet is delayed until 200ms have passed. In high action games with multiple clients sending their moves to eachother client this situation can arise quickly. Assume there are 4 players in a game and they send their movement whenever it changes (jump,up, down, etc). The chance that one of these clients will be issued multiple messages by the server at some point is pretty high (for example if both client A and B jump at the same time).

I have created some sample code that simulates this situation and which provides the observed 200ms wait.

Code: Select all

package test;

import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

import net.user1.union.api.Client;
import net.user1.union.api.Module;
import net.user1.union.api.Server;
import net.user1.union.core.context.ModuleContext;
import net.user1.union.core.event.ServerEvent;

public class TestServerModule implements Module
{
   private ScheduledThreadPoolExecutor stpe = new ScheduledThreadPoolExecutor(1);
   private Server server;
   
   @Override
   public boolean init(ModuleContext ctx)
   {
      server = ctx.getServer();
      server.addEventListener(ServerEvent.CLIENT_ADDED, this, "onClientAdded");
      
      return true;
   }
   
   public void onClientAdded(ServerEvent e)
   {
      stpe.schedule(new SendTestTask(e.getClient(),1), 2000, TimeUnit.MILLISECONDS);
      stpe.schedule(new SendTestTask(e.getClient(),2), 3000, TimeUnit.MILLISECONDS);
   }
   
   private class SendTestTask implements Runnable
   {
      private Client cl;
      private int numMsgs;
      
      public SendTestTask(Client c, int n)
      {
         numMsgs = n;
         cl = c;
      }
      
      public void run()
      {
         for(int i = 0; i < numMsgs; i++)
         {
            cl.sendMessage("TEST");
         }
      }
   }
   
   @Override
   public void shutdown()
   {
      stpe.shutdown();
      stpe = null;
      server.removeEventListener(ServerEvent.CLIENT_ADDED, this, "onClientAdded");
      server = null;
   }
}

This server module will send each connecting client three messages: one after 2 seconds and no 2 and no 3 after 3 seconds. As you will observe the third message will have a 200ms delay as compared to the second one, while the server has sent them at the same time.

Here is your client, slightly modified to measure this:

Code: Select all

    package {
      import flash.display.Sprite;
      import flash.text.TextField;
      import flash.utils.setInterval;
     
      import net.user1.reactor.Client;
      import net.user1.reactor.Reactor;
      import net.user1.reactor.ReactorEvent;
     
      public class UnionTest1 extends Sprite {
        protected var reactor:Reactor;
        protected var lastTestReceived:uint;
        protected var output:TextField;
       
        public function UnionTest1 () {
          reactor = new Reactor();
          reactor.addEventListener(ReactorEvent.READY, readyListener);
          reactor.connect("tryunion.com", 80);
         
          output = new TextField();
          output.width = 400;
          output.height = 300;
          output.background = true;
          output.border = true;
         
          addChild(output);
        }
       
        public function readyListener (e:ReactorEvent):void {
          reactor.getMessageManager().addMessageListener("TEST", testMessageListener);
        }
       
        public function testMessageListener (fromClient:Client):void {
          var now:uint = new Date().time;
          output.appendText("Time since last TEST received: " + (now-lastTestReceived) + "\n");
          output.scrollV = output.maxScrollV;
          lastTestReceived = now;
        }
      }
    }


[colin]

unsolving until we test your example. thanks for the sample code.

[derek]

We actually had TCP_NODELAY=true for the old IO but it is false by default in the underlying library we use for NIO.

I've created a ticket to make the use of delay configurable: http://factory.user1.net/issues/show/673

In the meantime you can find staging builds of 2.1.0 posted at:

http://unionplatform.com/staging/releases/union/latest/

At this early stage these builds are essentially Union 2.0.1 with TCP_NODELAY hardcoded to true for testing.

Please let us know if you observe this solving the issue in your environment.

Thanks!

Derek