Clustering Tutorial: Survey

We're going to write a survey application and because we expect many users we are going to write it to run in a cluster in MASTER-SLAVE mode. This means that when we create our survey room on a node it will be created as a master room. All other nodes will automatically create a survey room running as slaves.

Application Design

As with any usual server side application we need to specify which module messages the client will send to the room and which messages the room will send to the client.

Room Messages to Client

QUESTION - sent with 1 argument that represents the text of the question

Client Module Messages to Room

RESPONSE - sent with 1 argument, response, that can be either "yes" or "no"

We are also building the application to run in a cluster. Our master room will act as the brain and control when and which question is asked. A good design philosophy is to limit the amount of communication between nodes. So while we could send a remote event from each slave to the master as every client answers instead we'll gather the information on the slaves and only send the master aggregates.

Our survey application will use the following remote events:

Master to Slave Remote Events

ASK_QUESTION - dispatched when slaves should ask a question to their clients
END_QUESTION - dispatched when the question response time is complete and slaves should send their aggregate response totals to the master

Slave to Master Remote Events

SURVEY_RESULTS - dispatched to report aggregate response totals to the master

Create the Remote Event Objects

Each event is dispatched with an object that must implement net.user1.union.core.event.RemoteEvent. The object, and any non-transient objects it contains, must implement either Serializable or Externalizable so they can be transported across the cluster.

Following is the source for our AskQuestionEvent remote event object.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

package net.user1.union.example.survey;

import java.io.Externalizable;
import java.io.IOException;
import java.io.ObjectInput;
import java.io.ObjectOutput;

import net.user1.union.core.event.BaseEvent;
import net.user1.union.core.event.RemoteEvent;

/**
* The AskQuestionEvent is dispatched by the master when a new survey question is asked.
*/
public class AskQuestionEvent extends BaseEvent implements RemoteEvent, Externalizable {
private String m_question;

public AskQuestionEvent(String question) {
m_question = question;
}

public String getQuestion() {
return m_question;
}

public void readExternal(ObjectInput in)
throws IOException, ClassNotFoundException {
m_question = in.readUTF();
}

public void writeExternal(ObjectOutput out)
throws IOException {
out.writeUTF(m_question);
}
}

Create the Master Room Class

Our master room module is the brains of the survey application. It controls the behaviour of the application across the cluster. It also takes the aggregates of all responses across the cluster and outputs the total to System.out.

When creating our room we won't actually pass this class as the module class since we want the room to use a different class if it is a master versus a slave. At the end we'll create a factory class to handle this for us.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140

package net.user1.union.example.survey;

import net.user1.union.api.Module;
import net.user1.union.core.context.ModuleContext;
import net.user1.union.core.event.RemoteEvent;
import net.user1.union.core.event.RemoteRoomEvent;
import net.user1.union.core.event.RoomEvent;

/**
* The class that is created when the local room is the MASTER room.
*/
public class SurveyMasterModule implements Module, Runnable {
private ModuleContext m_ctx;

// the game thread
private Thread m_gameThread;

// the questions to ask
private String[] m_questions = {
"Are you a programmer?",
"Can you ice skate?",
"Have you ever been to Australia?"
};

// results
private int m_yes;
private int m_no;

/**
* Invoked by the server when the room is being created and the module should initialize
* itself
*/
public boolean init(ModuleContext ctx) {
m_ctx = ctx;

m_gameThread = new Thread(this);
m_gameThread.start();

// listen for when a client has sent a response (it is sent with a module message)
m_ctx.getRoom().addEventListener(RoomEvent.MODULE_MESSAGE, this, "onModuleMessage");

// listen for when a new slave room as been added to the cluster so that we can
// initialize that slave with the current state of the survey
m_ctx.getRoom().addRemoteEventListener(RemoteRoomEvent.ADD_SLAVE_ROOM, this,
"onAddSlaveRoom");

// listen for when a slave room has sent us the results from clients connected to that
// node
m_ctx.getRoom().addRemoteEventListener("SURVEY_RESULTS", this, "onSurveyResults");

return true;
}

public void run() {
int currentQuestion = 0;

while (m_gameThread != null) {
// ask a question by dispatching a remote event containing the question
// the event will automatically be sent to all nodes on the cluster
// and dispatched by the slave instances of this room
m_ctx.getRoom().dispatchRemoteEvent("ASK_QUESTION",
new AskQuestionEvent(m_questions[currentQuestion]));

// and send to clients connected to our server
// broadcast the question to the clients connected to this server
m_ctx.getRoom().sendMessage("QUESTION", m_questions[currentQuestion]);

// advance to the next question
currentQuestion = (currentQuestion+1) % m_questions.length;

// wait 10 seconds for people to answer
try {
Thread.sleep(10000L);
} catch (InterruptedException e) {
e.printStackTrace();
}

// then end the question
m_ctx.getRoom().dispatchRemoteEvent("END_QUESTION", new EndQuestionEvent());

// wait 5 seconds for results to come in
try {
Thread.sleep(5000L);
} catch (InterruptedException e) {
e.printStackTrace();
}

// output results to System.out and reset results
System.out.println("Yes: " + m_yes + " No: " + m_no);
synchronized (this) {
m_yes = 0;
m_no = 0;
}
}
}

/**
* This method is invoked when a module message has been sent by a client to the room. We
* use it to get responses from clients that are connected to this server. Responses
* collected by clients connected to other servers will be collected by the remote event
* "SURVEY_RESULTS".
*/
public void onModuleMessage(RoomEvent evt) {
if ("RESPONSE".equals(evt.getMessage().getMessageName())) {
// add the response to our totals
synchronized (this) {
if ("yes".equals(evt.getMessage().getArg("response"))) {
m_yes++;
} else if ("no".equals(evt.getMessage().getArg("response"))) {
m_no++;
}
}
}
}

/**
* This method is invoked when a slave room has dispatched its results to the master room.
*/
public void onSurveyResults(RemoteEvent evt) {
// get the event Object and increment yes and no
SurveyResultsEvent event = (SurveyResultsEvent)evt;
synchronized (this) {
m_yes += event.getYes();
m_no += event.getNo();
}
}

/**
* Invoked by the server when the room is being removed
*/
public void shutdown() {
// clean up event listeners
m_ctx.getRoom().removeEventListener(RoomEvent.MODULE_MESSAGE, this, "onModuleMessage");
m_ctx.getRoom().removeRemoteEventListener(RemoteRoomEvent.ADD_SLAVE_ROOM, this,
"onAddSlaveRoom");
m_ctx.getRoom().removeRemoteEventListener("SURVEY_RESULTS", this, "onSurveyResults");

m_gameThread = null;
}
}

Create the Slave Room Class

While the master room serves as the brain for the cluster wide application we still want to have our slave rooms process as much local client information as possible without having to resort to communicating with or passing it on to the master since that involves sending traffic through the cluster. In this simple case we can save a lot of network traffic by only sending the totals in an aggregate rather than passing through each client response to the master.

package net.user1.union.example.survey;

import net.user1.union.api.Module;
import net.user1.union.core.context.ModuleContext;
import net.user1.union.core.event.RemoteEvent;
import net.user1.union.core.event.RoomEvent;

/**
* The class that is created when the local room is a SLAVE room.
*/
public class SurveySlaveModule implements Module {
private ModuleContext m_ctx;

// results
private int m_yes;
private int m_no;

public boolean init(ModuleContext ctx) {
m_ctx = ctx;

// listen for when a client has sent a response (it is sent with a module message)
m_ctx.getRoom().addEventListener(RoomEvent.MODULE_MESSAGE, this, "onModuleMessage");

// listen for when the master room is asking a question
m_ctx.getRoom().addRemoteEventListener("ASK_QUESTION", this, "onAskQuestion");

// listen for when the master room has finished asking a question
m_ctx.getRoom().addRemoteEventListener("END_QUESTION", this, "onEndQuestion");

return true;
}

/**
* This method is invoked when a module message has been sent by a client to the room. We
* use it to get responses from clients that are connected to this server. Responses
* collected by clients connected to other servers will be collected by the remote event
* "SURVEY_RESULTS".
*/
public void onModuleMessage(RoomEvent evt) {
if ("RESPONSE".equals(evt.getMessage().getMessageName())) {
// add the response to our totals
synchronized (this) {
if ("yes".equals(evt.getMessage().getArg("response"))) {
m_yes++;
} else if ("no".equals(evt.getMessage().getArg("response"))) {
m_no++;
}
}
}
}

/**
* This method is invoked when the master room is asking a question.
*/
public void onAskQuestion(RemoteEvent evt) {
// a custom event so we have to cast it
AskQuestionEvent event = (AskQuestionEvent)evt;

// broadcast the question to the clients connected to this server
m_ctx.getRoom().sendMessage("QUESTION", event.getQuestion());
}

/**
* This method is invoked when the master room has finished asking a question.
*/
public void onEndQuestion(RemoteEvent evt) {
synchronized (this) {
// send the results to the master room and reset results
m_ctx.getRoom().dispatchRemoteEvent("SURVEY_RESULTS",
new SurveyResultsEvent(m_yes, m_no));

m_yes = 0;
m_no = 0;
}
}

public void shutdown() {
// clean up events
m_ctx.getRoom().removeEventListener(RoomEvent.MODULE_MESSAGE, this, "onModuleMessage");
m_ctx.getRoom().removeRemoteEventListener("ASK_QUESTION", this, "onAskQuestion");
m_ctx.getRoom().removeRemoteEventListener("END_QUESTION", this, "onEndQuestion");
}
}

Create a Room Module to Load the Master or Slave Code.

For this application we want our module in the slave rooms to run a different set of code than the module in the master room. So our room module acts as a simple factory and queries the room to determine if it is the master room or a slave and then loads the appropriate survey application code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

package net.user1.union.example.survey;

import net.user1.union.api.Module;
import net.user1.union.cluster.ClusterRole;
import net.user1.union.core.context.ModuleContext;

/**
* Module code to add the survey application to a room. This class is designed to work in a
* scaled environment and will initialize a master or slave depending on its role within the
* cluster. One node in the cluster will have an instance of the room with a ClusterRole of
* MASTER (the node where the room was initially created). All other nodes in the cluster will
* have an instance of the room with a ClusterRole of SLAVE.
*
* The survey application asks clients a "yes/no" question and then tallies the results of the
* responses.
*/
public class SurveyRoomModule implements Module {
private Module m_surveyModule;

public boolean init(ModuleContext ctx) {
// if it is a slave module then run the slave module code otherwise it is either being
// deployed as a master or in a non-clustered environment and so we want the master code
// to run
if (ctx.getRoom().getClusterRole() == ClusterRole.SLAVE) {
m_surveyModule = new SurveySlaveModule();
} else {
m_surveyModule = new SurveyMasterModule();
}

return m_surveyModule.init(ctx);
}

public void shutdown() {
m_surveyModule.shutdown();
}
}

Creating the Room

Now we can create our room just as we would in an non-clustered environment.

By configuring union.xml

Add the following entry under the <rooms> element. The room will be created when the server starts.

1
2
3
4
5
6
7
8
9
10
11

<room>
<id>SurveyTutorial</id>
<attributes>
<attribute name="_DIE_ON_EMPTY">false</attribute>
</attributes>
<modules>
<module>
<source type="class">net.user1.union.example.survey.SurveyRoomModule</source>
</module>
</modules>
</room>

Programmatically on the Server

Java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

// Create the room def
RoomDef roomDef = new RoomDef();

// Set the ID of the room
roomDef.setRoomID("SurveyTutorial");

// Set the room attribute to not die on empty
AttributeDef attrDef = new AttributeDef();
attrDef.setName(LocalRoom.ATTR_DIE_ON_EMPTY);
attrDef.setValue(Boolean.FALSE);
attrDef.setFlags(Attribute.FLAG_SHARED);
roomDef.addAttribute(attrDef);

// Add the room module
ModuleDef modDef = new ModuleDef();
modDef.setType(ModuleDef.CLASS);
modDef.setSource("net.user1.union.example.survey.SurveyRoomModule");
roomDef.addModule(modDef);

// Use the room context that was passed to your module to get a reference to the server and create our room
Room room = roomContext.getServer().createRoom(roomDef);