Feature

Welcome to the first issue of BEA WebLogic Developer's Journal! This article is the first of a three-part series geared around the clustering capabilities of BEA WebLogic Server (WLS) 6.1 and aimed at introductory and advanced audiences. This article will talk about the importance of clustering and the high-level clustering capabilities of WLS, provide an in-depth analysis of HttpSession clustering and persistence, discuss basic configuration and trouble shooting, and provide an example that ties together everything discussed in this article. The second article will provide an in-depth analysis of replica-aware stubs, their impact on a clustered system, and how they are used with EJBs, JMS objects, and DataSource objects. The last article will discuss clustering best practices, including the single-tier, coupled model; multi-tiered, coupled model; and the multi-tier, decoupled model.

The Motivation Behind Clustering
A single server, despite the best engineering in the world, can only do so much. There are many things that a single server cannot reliably survive: hardware failure, network failure, or too many requests. This aspect of enterprise development is just a fact of life. How do developers deal with these issues? Clustering. Clustering is a broad term applied to any system that gets multiple servers, components, or resources to act in a coordinated way and to provide a single, simple view of the services they offer.

An e-business system probably makes use of a cluster to maintain levels of throughput for requests, provide back-up systems in case of unforeseen failures in nodes, or to provide additional processing power to the system. Any system that supports clustering can be characterized by the Reliability, Availability, and Serviceability (RAS) properties:

• Reliability: An indication of whether or not the system can predictably handle requests in a consistent amount of time. If the number of requests, transactions, or concurrent users increases, the reliability of the system is high if the system can maintain the same quality of service for requests that was available when there were few requests, transactions, or concurrent users. The reliability of a system is low if response time or quality of service decreases as the load on the system increases. A two-node system has higher reliability than a one-node system.
• Availability: An indication of the amount of time that the system makes its services available to clients. Highly-available systems have very few periods (if any) that the services cannot be accessed by a client. Systems with many periods where the services are not available have low availability. A two-node system has higher availability than a one-node system.
• Serviceability: An indication of the effort involved with maintaining, monitoring, configuring, and operating a system. The serviceability of a system is high if very little effort is involved with monitoring the system and keeping it functioning. The serviceability of a system is low if the effort involved with operating the system is high, cost prohibitive, or tedious. A two-node system has lower serviceability than a one-node system.

There is no such thing as a perfect design for a clustered system. Anything that is done to a system to increase reliability and availability decreases the serviceability of that system. The antithesis holds true as well: anything done to increase the serviceability of a system decreases the availability and reliability of that system. The guiding light for clustering architectures is a trade-off analysis: the right design is the one that meets your requirements for each of the RAS properties.

WLS Clustering Capability Overview
There are a variety of things that WLS can do for a system to increase reliability or availability. Generally, many of the options that are available to select from increase either the reliability or the availability of the system, but not always both. Generally, WLS supports technologies that make load balancing of requests feasible or facilitate a high availability switch over of a current request to another server that can successfully process the request. This section discusses the major requirements and capabilities of WLS clusters.

System Requirements for Clustering
There are certain physical requirements and guidelines that must be followed when planning a WLS cluster. These include:

• A cluster can include one or more servers. Both administration servers and managed servers can participate in a cluster, but it would not be wise to have an administration server participate in a cluster that is performing the bulk of the processing for your system. Currently, the administration capabilities are not clusterable, but will be in the future. Many people believe that this is a single point of failure of the system, but it is not. If the administration server crashes, any managed servers that have been started will continue to execute. Allow administration servers to administrate and allow managed servers to perform your business logic.
• A cluster can be run on one or more computers. Since a single computer can execute multiple WebLogic Server instances, an entire cluster of multiple servers can be run on a single machine. A computer with many processors and lots of RAM could potentially support a cluster with many nodes. Your physical node design should distribute servers that will participate in a cluster appropriately across one or more machines that you may have available. Placing different nodes on different machines allows your system to resist hardware failures. Additionally, when you have a scenario where you are replicating your HttpSession or stateful session EJBs (discussed below), a WLS cluster will attempt to replicate the backup object onto a server located on a different machine.
• WebLogic Server clusters can only operate on a LAN. WLS clusters use IP multicast in certain situations to communicate with one another. IP multicast traffic is very unreliable over a WAN and would disrupt cluster communication. IP multicast traffic isn't 100% reliable in a LAN environment, but it is much more so than a WAN. This requirement also implies that all servers in the cluster must be reachable by IP multicast.
• WebLogic Server servers that participate in a cluster must have static IP addresses allocated to them. You cannot have a machine use dynamic IP addresses with a WLS cluster.
• All servers in a cluster must be running the same version of WLS. Since different versions of WLS will have different serialVersionUID numbers associated with each class (due to a recompilation), inter-server communication in a cluster would be fraught with ClassCastExceptions when objects are transferred between servers running different versions.

WLS Clustering Capabilities
There are many places, locations, and activities that a WLS cluster performs to provide developers with a reliable and available system. The most common misconception about WLS clusters is that many developers believe that the cluster itself (the servers in the cluster) performs load balancing or request re-direction. This NEVER occurs in a WLS cluster. Once a request has made it to a server, that server (and only that server) will handle the request or propagate an error condition. This means that all load balancing occurs by business logic that is hosted in front of the cluster (or something that communicates to the cluster). Additionally, this means that failover logic exists outside of the cluster but the cluster replicates any data that needs to be replicated to make sure that a failover request is successful.

The high level clustering capabilities of WebLogic Server include:

• Unified Naming Servers
• Cluster Heartbeats
• Cluster-Aware Stubs
• Replicated Data Objects (HttpSession and stateful session EJBs)

Unified Naming Servers
WLS implements a decentralized, unified naming server approach. Each server in the cluster has its own naming server and all of the naming servers advertise the same services to all clients even if some of the services advertised are located on other servers in the cluster. Each naming server is aware of the services that are hosted on it and the services that are hosted on another server. IP multicast messages are used for inter-server communication to announce new, updated, or removed services from one server to the others in the cluster. For example, when you create a new TxDataSource to a connection pool and deploy the TxDataSource to a single server in the cluster, the server that hosts the TxDataSource will send a copy of the TxDataSource object representing the connection pool to all of the other servers in the cluster using IP multicast. The TxDataSource is hard-wired (pinned) to the server that has its connection pool so any client that requests the TxDataSource will be able to download it from any server. Once downloaded to the remote client, the TxDataSource will communicate with the server that contains the connection pool even if the client downloaded the TxDataSource from a different server.

A WLS naming server will track all of the services located on all servers. In the situations where the same service (such as an EJB) is available on more than one server, the naming server will track the objects that represent the same service for all of the servers. Following the connection pool example described above, if the TxDataSource is deployed to the whole cluster and the same connection pool that the TxDataSource manages is available on each server in the cluster, each naming server will have a TxDataSource object that was sent to it from each of the other servers in the cluster. If you have a ten-server cluster, each naming server will have ten TxDataSource objects, each one representing a connection pool on a server.

When a client requests a service from a naming server, the server will always attempt to send the object representing the service on the current server to the client. Following our example, the naming server will send the TxDataSource that maps to the connection pool on the current server before it sends a TxDataSource representing the same connection pool hosted on another server. If the service requested is not currently available on the current server, then the naming server will traverse its linked list of objects representing resources on other servers and send one of those instead. As services are deployed and undeployed, a WLS server will announce the changes over IP multicast. When a naming server receives an undeploy message for a service, it merely has to remove the object representing that service for that particular server from its naming tree. When a service is deployed or redeployed, a naming server merely has to add the object representing the service for that server to its naming tree.

Cluster Heartbeats
Each server in the cluster sends out a regular heartbeat every ten seconds over IP multicast. This "I'm alive" heartbeat is to allow the other servers in the cluster to be aware that a particular server is still present. All servers in a cluster monitor and watch for heartbeats from all other servers in the cluster. If a heartbeat from another server in the cluster is missed three times (30 seconds), then the server tracking the heartbeats assumes that the other server is dead and removes all of the objects representing services located on that server. In a situation where a network failure has occurred and none of the nodes can communicate over IP multicast, all of the servers will miss the heartbeats from all of the other servers. After 30 seconds, each server will assume all of the other servers are dead and remove any objects from those servers that represent services hosted elsewhere. Essentially, each server would be operating in a standalone fashion, but still waiting for heartbeats to reappear. In the situation where two servers are able to re-connect, they will re-synchronize their objects for their naming trees.

Cluster-Aware Stubs
JMS ConnectionFactory objects, EJBs, home stubs, and JDBC TxDataSource objects are all factory objects that are hosted in a naming tree. Clients download these objects and then invoke a method to obtain a remote reference to the resource they intend to use. This is a Connection object for JMS, a remote stub object for an EJB, and a Connection object for JDBC.

In WLS, the factory objects that are downloaded are called cluster-aware stubs. Since the same service may be available on multiple servers in the cluster, the factory object downloaded from JNDI includes load balancing and failover logic for locating an appropriate reference to the resource. JMS ConnectionFactory objects, EJBs, home stubs, and JDBC TxDataSource objects are all cluster-aware stubs that can return a reference to a resource that is located on any server in the cluster. Cluster-aware stubs communicate with the cluster to maintain an active list of servers that host the service they are a factory for. When a client uses a cluster-aware stub to obtain a reference to a resource, the cluster-aware stub can load balance that request to different servers in the cluster. If you have a connection pool deployed to all of the servers in the cluster, a client that calls getConnection() on a TxDataSource multiple times will receive a Connection object from a different server upon each invocation. This behavior only applies to Java clients that are not colocated within the same VM as the server itself. In a situation where the requesting client IS colocated in the same VM as the server, it always makes sense to return the reference to the resource hosted locally. Any attempt to load balance a request that can be handled locally is an unnecessary burden to the system. Additionally, since a cluster-aware stub is constantly updated with the correct list of servers that are hosting the service, the cluster-aware stub will never make a request to a server that isn't hosting the service.

Cluster-aware stubs and EJBs are an interesting topic because:

1. EJBs have two stubs (home and remote stubs)
2. EJBs have three behavior types (stateless, stateful, and persistent) that can facilitate or limit the load balancing or failover capability of a cluster-aware stub.

Cluster-aware stubs and EJBs will be discussed in more detail in the next article in this series.

Replicated Data Objects (HttpSession and Stateful Session EJBs)
The J2EE specification defines two types of server-side data objects for short-term storage of an application's state: HttpSession objects and stateful session EJBs. These objects provide a means of sharing data between requests without explicitly requiring the passing of data as method parameters and return values. These objects serve to simplify intra-application communication, not to provide robust storage of transactional data - that is the role played by entity EJBs or direct database interactions via JDBC.

Since HttpSession objects and stateful session EJBs are not designed to survive server crashes, WebLogic Server provides data object replication to help resist failures in enterprise systems. For HttpSession objects, there are several options: JDBC replication, cookie persistence, and in-memory replication.

JDBC replication stores session content in a relational database table. This process offers two advantages:. First, persistent storage ensures survival of the session data even if the entire cluster were to become inoperable. Second, storage in a central repository allows any server in a cluster to be able to handle any session. Despite this, JDBC replication has poor performance since every update to a session instance requires synchronization with the database.

Cookie persistence involves serialization of the session object into a cookie that is passed between the Web browser and server with each HTTP request. Cookie persistence allows any server access to any session while also providing fast in-memory access to the session's contents. However, there are some drawbacks:

• Only String attributes can be stored; all others will be ignored when the cookie is generated.
• The cookie must be written to the header data before the response is committed. The HTTP response cannot be flushed programmatically or based on buffer overflow until the cookie has been written.
• There is limited security since the session data is sent in clear text.
• The browser may impose a maximum size of the cookie and, therefore, the amount of stored data.

The most popular method for HttpSession failover is in-memory replication, which can also be applied to stateful session EJBs. With this technique, there are at most two servers in the cluster that have a copy of a data object in memory. This method offers high efficiency with quick access to the data object's contents while conserving runtime memory across the servers in a cluster.

In-memory replication requires routing all requests for a given HttpSession to the server that has that session. This can be done with cookies or URL rewriting. When a clustered server receives an HTTP request unaffiliated with an HttpSession, the server creates a new HttpSession object with a unique ID, registers itself as the primary server, and selects a backup server from those remaining in the cluster. Point-to-point communication between the primary and backup servers is established at this time, to allow for the synchronization of the HttpSession object. The IP addresses and ports of the primary and backup servers are encoded as part of the session ID for the HttpSession.

Load Balancing and HttpSession Fail Over
The specifics of how subsequent HTTP requests are handled are based on the load-balancing mechanism selected for initial access to the cluster. WLS offers several options: the HttpClusterServlet, Web server plug-ins, and support for load-balancing hardware.

The HttpClusterServlet, available with the standard WebLogic Server installation, runs in a non-clustered instance of WLS (i.e., a proxy server) and contains the logic necessary to parse the routing information in the WLS-generated session ID. HTTP requests not containing a session ID are load balanced by a round-robin algorithm. HTTP requests that contain a session ID are routed to the server containing the primary HttpSession instance (see Figure 1). When the server hosting the primary instance becomes unavailable, the backup server senses it through the loss of point-to-point communication, promotes itself to primary, and selects a new backup from the remaining clustered servers. Upon the next request, the HttpClusterServlet detects that the original primary server is unavailable and reroutes the request to the specified backup server. The response contains the session ID with revised routing data. The ease of configuring the HttpClusterServlet makes it a valuable tool during the development process.

With Web server plug-ins, developers can capitalize on the investment already made on Netscape, Microsoft, and Apache Web server licenses to serve static Web content while delegating requests for dynamic page generation, via servlets and JSPs, to the WebLogic Server. The plug-ins, available as shared or dynamic libraries, provide the same round-robin load balancing and HttpSession routing mechanisms as the HttpClusterServlet. Additionally, the Netscape and Apache plug-ins provide routing based on URL pattern matching.

Load-balancing hardware provides built-in firewall capabilities and sophisticated load-balancing algorithms. To consistently route traffic to a server hosting a primary instance, most vendors will either insert their own cookie in the client session (active cookie persistence) or interpret part of an existing cookie as a single numerical differentiator (passive cookie persistence). WebLogic Server in-cluster routing mitigates load-balancing hardware's disregard for the backup server (see Figure 2). When a request is received on a clustered server that has no knowledge of a given session, the routing data stored in the session ID is queried, the server hosting the back instance is contacted, and the receiving server becomes the host to the new primary instance. In other words, the server hosting the backup instance becomes a factory for creating new primary instances, rather than being promoted itself.

Setting Up a Simple Cluster
It is time to see how easy it is to set up a cluster. This last section will set up a cluster of three servers for handling business components whose external access is controlled by a fourth server acting as a proxy with the HttpClusterServlet. The business content will consist of a servlet, which will display the session ID associated with each request that illustrates the in-memory replication mechanism described above.

The entire domain for this exercise is available for download the Web site (www.sys-con.com/weblogic). The downloaded domain files will have the cluster, business components, and proxy configured for some default IP addresses and ports. You can setup your cluster by following the steps listed here or by installing the domain download and re-mapping IP addresses to match those on your machine.

Basic Cluster Configuration with the Administration Console
First, using the Administration Console, we must create our server and cluster profiles and tie the former to the latter.

To create the server profiles, right click on the Servers folder and select "Configure a New Server.... In the Configuration->General pane, fill in and submit the Name, Listen Port, and Listen Address for each server. Remember that the clustered servers must share the same Listen Port and have unique Listen Address settings. Figure 3 has a screen shot of what our screen looks like when filled out. Please note that, while the Listen Address could be either an IP address or host name, the use of the latter is necessary to run this example. To map IP addresses to host names on your individual computer, add entries to C:\WINNT\system32\drivers\etc\hosts.

To create the cluster profile, right click on the Clusters folder and select "Configure a New Cluster...". In the Configuration->General pane, fill in and submit the Name and Cluster Address for the cluster. Set and apply the Multicast Address in the Configuration-> Multicast pane. Finally, add the clustered servers to the Chosen column in the Configuration-> Servers pane. Figures 4, 5, and 6 are screen shots of what our cluster configuration looks like when filled out.

All of the configuration data used for the business servers, proxy server, and cluster is summarized in Table 1.

Setting Up the HTTPClusterServlet
It is time to configure the front-end load-balancing mechanism. This example will use the HttpClusterServlet running in its own instance of WLS. In the web.xml file of the proxy server's default Web application, XML elements are added to:

• register the HttpClusterServlet with its fully qualified class name;
• initialize the parameter defaultServers with the list of servers participating in the cluster;
• determine which URLs the servlet will intercept.

Listing 1 shows the web.xml configuration we used. The listing may be found at www.weblogicdevelopersjournal.com.

Defining Business Content and a Fail Over Mechanism
It is now time to build and deploy the SimpleServlet, which displays the current contents of a session ID in two parts: the unique identifier (by default, 52 bytes in length), and the routing data. (Listing 2).

In the web.xml file of SimpleWebApp, the clustered Web application, XML elements are added to register SimpleServlet with its fully qualified class name and provide a single URL mapping element to allow invocation of the servlet through the alias simple. Listing 2 contains the source code for the SimpleServlet servlet. In the weblogic.xml file of the application, XML elements are added to set the session-related property PersistentStoreType to replicated. This is the only parameter required to enable in-memory replication. Listing 3 lists the configuration files for the SimpleWebApp.

The last step for configuring the test application is to deploy it to the cluster. Deployment to a cluster can be accomplished in the Administration Server:

1. Select the SimpleWebApp node from the navigation pane.
2. Select the Targets->Clusters pane.
3. Move the appropriate cluster name (in our case, PresentationTierCluster) from the Available list to the Chosen list and click Apply.

Starting the cluster and interpreting results
To start the cluster:

1. Start the administration server with the startWebLogic.cmd script.

Start each of the clustered servers with the startManagedWebLogic.cmd script, with two command-line parameters. The first parameter is the name of the managed server (i.e., ServerA, ServerB, or ServerC). The second parameter is the URL of the administration server (in our case, http://127.0.0.1:7001). In order to determine if each server successfully joined the cluster, you can first check the standard output or server log for the text "Starting Cluster Service?". For more detailed monitoring information, click on the PresentationTierCluster node in the navigation pane, tab over to Monitoring, and click on the "Monitor server participation in this cluster" hyperlink. The displayed table provides valuable information on the number of packets sent/received as well the role of each server with respect to in-memory replication.

If the table does not show three servers receiving updates, try the following troubleshooting measures:

• Check your start script command line parameters for typos.
• Verify that there are no physical network problems with the PING option of the Java application weblogic.Admin.
• Verify that multicast communications are working by using the Java application utils.MulticastTest.
• If the multicast utility fails, confirm that the selected multicast address falls within the valid range (224.0.0.1 - 239.255.255.255) and that no other application on the network is using that address.

If none of these measures shed any light on the encountered problems, consult Appendix A, "Troubleshooting Common Problems" of the document "Using WebLogic Server Clusters" for further guidance.

2. Start the proxy server with the startManagedWebLogic.cmd script, with the following parameters: ProxyServer and http://127.0.0.1:7001).

To test the sample application,

1. Start a web browser and disable cookies.
2. Go to: http://192.168.1.103:9001/SimpleWebApp/simple. If everything is working properly, the session ID and routing data will be displayed in your browser. Embedded within the routing data are aliases for the servers hosting the primary and back HttpSession instances.
3. Reload the servlet multiple times. With each reload, both the session ID and routing data change. Since the browser is not sending the cookie along with each HTTP request, the WLS servlet container always treats the user as a first-time visitor to the site. Note that the use of the round-robin load-balancing algorithm is evident in the server aliases displayed in the routing data.
4. Now, enable cookies in your browser and reload the servlet several times. With each reload, the session ID and routing data stay the same. Since cookies are being sent with each HTTP request, the proxy server is able to use the routing data to direct the traffic to the server hosting the primary HttpSession instance.
5. To demonstrate failover, bring down the clustered server whose alias appears first in the displayed routing data and reload the servlet. Observe that the session ID, which represents that application state, did not change, but the routing data has been modified. The server alias has moved forward, representing a promotion of the backup server to primary, and a new second alias is specified, representing the selection of a new backup server. Bringing down a second server will result in the last remaining clustered server being promoted to primary with the text "None" being inserted into the routing data since there are no candidates for backup.
6. As a final test, restart the dead servers and reload the servlet. The routing data will change to reflect the selection of a new backup server demonstrating the dynamic nature of cluster membership.

Conclusion
Whew! Granted, that's a lot to swallow, but WLS clustering is straightforward. Its flexibility comes from the number of integration touch points that it offers to architects of e-business systems. Future articles will discuss the implications of EJBs in a cluster and clustering best practices, so keep practicing and reading!

Add Your Feedback

In order to post a comment you need to be registered and logged in.

Register | Log in

Please wait while we process your request...