RE-ENTER SAS
Voyez le cloud autrement

Endeca Proactive Load Balancing

by Vina Rakotondrainibe | Oracle Commerce Expert and Cloud Deployment Specialist
Paris area,

Introduction

In the previous article, we talked about Endeca MDEX Load Balancing with HAProxy. In this article, I will go further and explain how to efficiently manage the backend servers using HAProxy in order not to loose requests during a baseline update.

As a reminder, a baseline update is the process where Endeca republishes the entire index to its Dgraphs (e.g. MDEXes). During this process, the index file is generated for each MDEX and the Dgraph process is stopped during the index switch. If you let the load balancer continue to send requests to the Dgraph processes, the end user might receive an HTTP 503 code which is not really good for your site reputation.

Fortunately, you can instruct HAProxy to remove a backend server temporarily. By doing this proactive load balancing, the index switch can be performed without being noticed on your e-commerce site.

Preparing HAProxy

The configuration file example from the previous blog entry has to be modified to simplify our dgraph administration task. We called the MDEX backend servers mdex1 and mdex2. Let's call them with the same name as what is in the Endeca XML control files (i.e. LiveDgraphCluster.xml):

backend livemdexes
server DgraphA1 <MDEX 1 IP>:15000 check inter 2000 fall 2 rise 1 cookie DgraphA1 server DgraphB1 <MDEX 2 IP>:15000 check inter 2000 fall 2 rise 1 cookie DgraphB1

We also need to add a file socket to be able to talk to HAProxy. This socket will be accessible to root only.

socket /var/lib/haproxy/haproxysock level admin

To remove or add a backend server dynamically, HAProxy supports direct calls to the socket. To do so, you need to install socat (example on Red Hat):

yum install socat

Now you can use socat to send messages to the HAProxy socket we created above:

# to disable DgrapA1
echo "disable server livemdexes/DgraphA1" | socat stdio /var/lib/haproxy/haproxysock
 
# to re-enable DgrapA1
echo "enable server livemdexes/DgraphA1" | socat stdio /var/lib/haproxy/haproxysock

You would need to write your own shell script to include these statements and do extra checks on HAProxy topology, state, user rights etc... before performing these actions. For the remainder of this article, let's say you wrote a script called haproxy.sh to administer your HAProxy process. You will be able to call this script remotely from your Endeca server through SSH:

ssh -tt  "sudo /opt/scripts/haproxy.sh enable livemdexes DgraphA1"

Configuring the Dgraphs

Once the HAProxy administration shell script is ready, you need to add the remote calls to the Dgraph's pre-shutdown and post-startup actions. To do so, you first need to declare the new scripts. Create a new file called DgraphScripts.xml under the config/scripts folder of your Endeca application. Here is an example of the content of this XML file:

 <script id="AdminLBServer">
   <bean-shell-script>
   <![CDATA[
      
     //public void adminBalancer( String mEacHost, String mLbHost, String mBackendName, String mServerName, String mAction) {
       adminLB = new ShellUtility("CRS", mEacHost, 8888, false);
       shell_id = "AdminLBServer";
       host = "ITLHost"; // name of server you want to run command
        
       // --- Uncomment the line below after you have the relevant scripts for your load balancer
       command = "ssh  -tt " + mLbHost + " \"sudo /opt/scripts/haproxy.sh " + mAction + " " + mBackendName + " " + mServerName + " \""; 
       log.info("*******************************************************************************************");
       log.info("Launching command: " + command);
       log.info("*******************************************************************************************");
       adminLB.init(shell_id, host, command, "/");
       adminLB.run();
     //}
   ]]>
   </bean-shell-script>
 </script>
 
 <!-- the script to remove the dgraph from the available servers in HAProxy -->
 <script id="StopLBServer">
   <bean-shell-script>
   <![CDATA[
     id = invokingObject.getElementId();
     dGraphHost = invokingObject.getHost().getHostName();
     port = invokingObject.getPort();
     log.info("Removing dgraph with id " + id + " (host: " + dGraphHost + ", port: " + port + ") from load balancer cluster.");

mEacHost = "your EAC hostname Here"; mLbHost = "Your HAProxy hostname Here";
mBackendName = "LiveDgraphCluster"; mServerName = invokingObject.getElementId(); mAction = "remove";

AdminLBServer.run();
]]> </bean-shell-script> </script> <!-- the script to add the dgraph to the available servers in HAProxy --> <script id="StartLBServer"> <bean-shell-script> <![CDATA[ id = invokingObject.getElementId(); dGraphHost = invokingObject.getHost().getHostName(); port = invokingObject.getPort(); log.info("Adding dgraph with id " + id + " (host: " + dGraphHost + ", port: " + port + ") to load balancer cluster.");
mEacHost = "your EAC hostname Here"; mLbHost = "Your HAProxy hostname Here";
mBackendName = "LiveDgraphCluster"; mServerName = invokingObject.getElementId(); mAction = "add";

AdminLBServer.run();
LiveDgraphCluster.applyConfigSnapshot(id); ]]> </bean-shell-script> </script>

Add the above XML definition file to your application definition file (i.e. in AppConfig.xml ):

<spr:import resource="InitialSetup.xml" />
<spr:import resource="DgraphDefaults.xml" />
<!-- vrakoton: specific stop and start scripts for dgraphs -->
<spr:import resource="DgraphScripts.xml" />
<spr:import resource="AuthoringDgraphCluster.xml" />
<spr:import resource="LiveDgraphCluster.xml" />
<spr:import resource="LiveAppServerCluster.xml" />
<spr:import resource="UsageCollectionConfig.xml" />
<spr:import resource="WorkbenchConfig.xml" />
<spr:import resource="ReportGeneration.xml" />
<spr:import resource="DataIngest.xml" />
<spr:import resource="AssemblerConfig.xml" /> 

Now you have to modify the Live Dgraph definition to call the HAProxy Admin script (in the LiveDgraphCluster.xml file):

<dgraph id="DgraphA1" host-id="LiveMDEXHostA" port="15000"
  post-startup-script="StartLBServer" pre-shutdown-script="StopLBServer">
  ...
<dgraph id="DgraphB1" host-id="LiveMDEXHostB" port="15000"
  post-startup-script="StartLBServer" pre-shutdown-script="StopLBServer">

Update your Endeca configuration by running an EAC command in the control folder of your application:

 [endeca@eac.local.com ~] cd /opt/endeca/Apps/CRS/control
[endeca@eac.local.com ~] ./runcommand.sh --update-definition

Now everytime you run a baseline update, the host running the switched Dgraph will be removed from the load balancer and put back in the backend server list when the index switch is over. You can try to load test your MDEXes while performing a baseline, you should not loose a single user request.

Conclusion

Proactively enabling/disabling your load balancer workers prevents search engine down time from the end user point of view. It will also enable you to choose less agressive parameters on the fall/rise parameter of HAProxy. But above all, this will prevent hanging requests to burden your site availability.
Top