Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1

Ashwin Gupta (gupta.ashwin@in.ibm.com) Technical Sales Specialist IBM Rahul Gupta (rahul.gupta@us.ibm.com) Advisory IT Specialist IBM Venkatesh Murugan (n.venkatesh@in.ibm.com) Technical Services Professional IBM This article describes multi-instance brokers, a feature introduced in WebSphere Message Broker V7 that helps you configure high-availability environments. The feature is based on multi-instance queue managers in WebSphere MQ V7.0.1, which provide multiple instances of a queue manager residing on separate physical servers to provide a highly available configuration. In WebSphere MQ, the multi-instance queue manager's configuration is hosted on shared network storage, so that if the active queue manager fails, another instance automatically assumes the stopped queue manager's configuration and becomes active. Similarly, in Message Broker V7 and V8, a broker's configuration is hosted on network storage to enable the high-availability function. 03 November 2010

Introduction
This article describes the new capability in IBM WebSphere Message Broker (hereafter called Message Broker) for handling failover and providing high availability (HA). It will help you create and configure multi-instance queue managers and brokers. This article also provides step-bystep procedures to develop and test the Message Broker HA environment. The target audience for this article includes senior IT architects and IT specialists who design SOA solutions and need to implement HA ESB solutions. To understand Message Broker multi-instance broker feature (hereafter called MI broker), you must first understand how the multi-instance queue manager (hereafter called MI queue manager) works. This article presents a conceptual overview of MI queue manager instances and the tasks
Copyright IBM Corporation 2010 Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Trademarks Page 1 of 23

developerWorks

ibm.com/developerWorks/

for administering them, followed by a demonstration of how you can switch over between multiple instances. After that, the article describes the concepts of MI brokers, including their setup and how to configure and test them for high availability.

Multi-instance queue managers

The term MI queue manager refers to the combination of active and standby instances of the queue manager that share the queue manager data and logs. MI queue managers protect applications against the failure of queue manager processes by having one instance of the queue manager active on one server, and another instance on standby on another server, ready to take over automatically should the active instance fail. Replicating queue manager instances is an effective way to improve the availability of queue manager processes. Examples in this article were run on a system using WebSphere MQ and Message Broker, with three servers running Red Hat Enterprise Linux 5.0. Figure 1 shows an overview of the MI queue manager and MI broker components on these three servers:

Figure 1. Overview of MI queue manager and MI broker components

The three servers are: wmbmi1.in.ibm.com Hosts the primary active instance of the MI queue manager IBMESBQM and the primary active instance of the MI broker IBMESBBRK wmbmi2.in.ibm.com Hosts the duplicate standby instance of the MI queue manager IBMESBQM and the duplicate standby instance of the MI broker IBMESBBRK wmbmi3.in.ibm.com Hosts the shared network file system /mqha through NFS V4 Though it's not mandatory to run the standby instance and host the shared file system on separate servers, it is advisable to prevent the queue manager from failing due to a hardware problem. In this example, WebSphere MQ is installed on two servers: wmbmi1.in.ibm.com for the primary MI queue manager, and wmbmi2.in.ibm.com for the standby MI queue manager. As shown in Figure
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 2 of 23

ibm.com/developerWorks/

developerWorks

1, the active instance of IBMESBQM is running on wmbmi1.in.ibm.com; the other instance, running as standby on wmbmi2.in.ibm.com, is doing no active processing and is ready to take over from the active instance if it fails. The tasks to administer these MI queue managers are described below.

Configuring the networked file system

An MI queue manager uses a networked file system to manage queue manager instances. The queue manager automates failover using a combination of file system locks and shared queue manager data and logs. Before V7.0.1, WebSphere MQ did not support networked storage accessed as a shared file system. If queue manager data was placed on shared network storage, you had to ensure that this data was not accessed by another instance of the queue manager running at the same time. The situation is improved in WebSphere MQ V7.0.1: queue managers can acquire read/write locks to provide access to their data; at any given time, only one instance of a queue manager is allowed to lock data, eliminating dirty reads. Prior to configuring NFS, you need to ensure that the user ID (uid) and group ID (gid) of the user are the same for all the servers where MI queue manager instances reside -- wmbmi1.in.ibm.com, wmbmi2.in.ibm.com, and wmbmi3.in.ibm.com, in our example. This also applies to the NFS server. On Linux and UNIX systems, the uid used to run the WebSphere MQ and WebSphere Message Broker commands must be a member of both the mqbrkrs group and the mqm group, and the UNIX gids of the mqbrkrs and mqm groups must be the same on all the systems. Additionally, you should make sure that the uid and its accompanying gid for this uid are the same on all the systems, and is the same uid that created the first instance of the queue manager. Change uids and gids in Linux and UNIX environments with caution, because doing so affects the permission levels of files on the system. Changing a uid or gid causes the ownership of all the files previously owned by that user or group to change to the actual integer representing the file's previous owner. The system administrator must ensure that the ownership of the affected files and directories are manually restored. In this example, the user mqm administers the MI queue manager and MI brokers. Use the commands in Listing 1 to match the uid and gid for the mqm user in /etc/passwd, then reboot the system if you need to change the values:

Listing 1. Matching the uid and gid of the mqm user on all member servers
[mqm@wmbmi1.in.ibm.com]$ cat /etc/passwd | grep mqm mqm:x:500:501:MQSeries User:/home/mqm:/bin/bash [mqm@wmbmi2.in.ibm.com]$ cat /etc/passwd | grep mqm mqm:x:500:501:MQSeries User:/home/mqm:/bin/bash [mqm@wmbmi3.in.ibm.com]$ cat /etc/passwd | grep mqm mqm:x:500:501:MQSeries User:/home/mqm:/bin/bash

Additionally, you also need to set the uid and gid of the mqm group to be identical on all the systems. Create log and data directories in a common, shared folder named /mqha. Make sure that the mqha directory is owned by the user and group mqm, and that the access permissions are set to rwx for both user and group. The commands in Listing 2, executed as root on wmbmi3.in.ibm.com, will achieve this:
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 3 of 23

developerWorks

ibm.com/developerWorks/

Listing 2. Creating and setting ownership for directories under the shared folder /mqha
[root@wmbmi3.in.ibm.com]$ [root@wmbmi3.in.ibm.com]$ [root@wmbmi3.in.ibm.com]$ [root@wmbmi3.in.ibm.com]$ [root@wmbmi3.in.ibm.com]$ mkdir mkdir mkdir chown chmod -p -p -p -R -R /mqha/WMQ/IBMESBQM/data /mqha/WMQ/IBMESBQM/logs /mqha/WMB/IBMESBBRK mqm:mqm /mqha ug+rwx /mqha

Next, you need to configure the NFS server on wmbmi3.in.ibm.com and then start it on the same machine. Add the excerpt in Listing 3 to the /etc/exports file, which should be executed as root on wmbmi3.in.ibm.com:

Listing 3. Configuring the NFS server

/mqha *(rw,fsid=0,no_wdelay,sync)

Start the NFS server by executing the command in Listing 4 as root on wmbmi3.in.ibm.com:

Listing 4. Starting the NFS server

/etc/init.d/nfs start

If the NFS server is already running, refresh it using the command in Listing 5:

Listing 5. Refreshing the NFS server

exportfs -ra

Now the shared directory /mqha on wmbmi3.in.ibm.com should be mounted on both wmbmi1.in.ibm.com and wmbmi2.in.ibm.com. Use the command in Listing 6 to check whether the shared /mqha directory is visible from both instance hosting servers:

Listing 6. Checking to make sure /mqha is mounted

showmount e wmbmi3.in.ibm.com

If the output of the command in Listing 6 is negative, you will need to mount the shared folder from the two instance hosting servers. Execute the command in Listing 7 as root on both of the servers to mount the exported file system:

Listing 7. Mounting the exported file system

mount -t nfs4 -o hard,intr wmbmi3.in.ibm.com:/ /mqha

Not all network file system locking protocols are robust, and a file system might be configured for performance rather than data integrity. You must run the amqmfsck command to test whether your network file system will properly control access to queue manager data and logs: Run amqmfsck without any options on each system to check basic locking. Run amqmfsck on both WebSphere MQ systems simultaneously, using the -c option, to test writing to the directory concurrently.
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 4 of 23

ibm.com/developerWorks/

developerWorks

Run amqmfsck on both WebSphere MQ systems simultaneously, using the -w option, to test waiting for and releasing a lock on the directory concurrently. To work reliably with WebSphere MQ, a shared file system must provide: Data write integrity Guaranteed exclusive access to files The release of locks upon failure If the file system does not provide these features, the queue manager data and logs may become corrupted when the shared file system is used in an MI queue manager configuration. Currently, NFS V4 provides all the above mentioned facilities, so it is the file system in use in this example.

Creating a multi-instance queue manager

Start by creating the MI queue manager on the first server, wmbmi1.in.ibm.com. Log on as the user mqm and issue the command in Listing 8:

Listing 8. Creating a queue manager

crtmqm -md /mqha/WMQ/IBMESBQM/data -ld /mqha/WMQ/IBMESBQM/logs IBMESBQM

Once the queue manager is created, display the properties of this queue manager using the command in Listing 9:

Listing 9. Displaying the properties of the queue manager

[mqm@wmbmi1.in.ibm.com]$ dspmqinf -o command IBMESBQM addmqinf -s QueueManager -v Name=IBMESBQM -v Directory=IBMESBQM -v Prefix=/var/mqm v DataPath=/mqha/WMQ/IBMESBQM/data/IBMESBQM

Copy the output from the dspmqinf command and paste it on the command line on wmbmi2.in.ibm.com from the console of user mqm, as shown in Listing 10:

Listing 10. Configuring wmbmi2.in.ibm.com

[mqm@wmbmi2.in.ibm.com]$ addmqinf -s QueueManager -v Name=IBMESBQM -v Directory=IBMESBQM -v Prefix=/var/mqm -v DataPath=/mqha/WMQ/IBMESBQM/data/IBMESBQM WebSphere MQ configuration information added.

Now display the queue managers on both servers using the dspmq command on each. The results should look like Listing 11:

Listing 11. Displaying the queue managers on both servers

[mqm@wmbmi1.in.ibm.com]$ dspmq QMNAME(IBMESBQM) [mqm@wmbmi2.in.ibm.com]$ dspmq QMNAME(IBMESBQM) STATUS(Ended immediately) STATUS(Ended immediately)

The MI queue manager IBMESBQM has now been created both on the server wmbmi1.in.ibm.com and on the server wmbmi2.in.ibm.com.
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 5 of 23

developerWorks

ibm.com/developerWorks/

Starting and stopping a multi-instance queue manager

Starting a queue manager in multi-instance mode is as simple as starting a normal queue manager. To start the queue manager instances in either order, issue the strmqm command with the -x parameter, as shown in Listing 12:

Listing 12. Starting the queue manager in multi-instance mode

[mqm@wmbmi1.in.ibm.com]$ strmqm -x IBMESBQM WebSphere MQ queue manager 'IBMESBQM' starting. 5 log records accessed on queue manager 'IBMESBQM' during the log replay phase. Log replay for queue manager 'IBMESBQM' complete. Transaction manager state recovered for queue manager 'IBMESBQM'. WebSphere MQ queue manager 'IBMESBQM' started. [mqm@wmbmi2.in.ibm.com]$ strmqm -x IBMESBQM WebSphere MQ queue manager 'IBMESBQM' starting. A standby instance of queue manager 'IBMESBQM' has been started. The active instance is running elsewhere.

The queue manager is first started on wmbmi1.in.ibm.com and later on wmbmi2.in.ibm.com. Hence, IBMESBQM starts as an active instance on wmbmi1, processing all requests, and as a standby instance on wmbmi2, ready to take over if the active instance fails. The active instance is granted exclusive access to the queue manager data and logs, while the standby waits to be granted such access; if the standby is granted exclusive access, it becomes the active instance. You can also always start a queue manager configured with multiple instances on different servers as a single-instance queue manager. Should the queue manager be started using the strmqm command (without the -x option), the instances of the queue manager configured on other machines are prevented from starting as standby instances. If an attempt is made to start another instance, the user would receive the response that the queue manager instance is not permitted to run as a standby. Only two queue manager instances can run at the same time: an active instance and a standby. Should the two instances be started at the same time, WebSphere MQ has no control over which instance becomes the active instance; this is determined by the NFS server. The first instance to acquire exclusive access to the queue manager data becomes the active instance. Now that the queue manager has been started, you can create and start the queue manager listener for applications to connect the active queue manager IBMESBQM, as shown in Listing 13. All administration commands in Listing 13 are issued on the active queue manager:

Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1

Page 6 of 23

ibm.com/developerWorks/

developerWorks

Listing 13. Creating and starting the queue manager listener

[mqm@wmbmi1.in.ibm.com]$ runmqsc IBMESBQM 5724-H72 (C) Copyright IBM Corp. 1994, 2009. Starting MQSC for queue manager IBMESBQM. define listener(IBMESBLISTENER) trptype(tcp) port(1414) control(qmgr) 1 : define listener(IBMESBLISTENER) trptype(tcp) port(1414) control(qmgr) AMQ8626: WebSphere MQ listener created. START LISTENER(IBMESBLISTENER) 2 : START LISTENER(IBMESBLISTENER) AMQ8021: Request to start WebSphere MQ Listener accepted. ALL RIGHTS RESERVED.

There are two ways to stop the MI queue managers you've started. The first is to switch the active and standby instances by using the endmqm command with the -s option. If you issue the endmqm -s IBMESBQM command on the active instance, you will manually switch control to the standby instance. The endmqm -s command shuts down the active instance without shutting down the standby. The exclusive access lock on the queue manager data and logs is released, and the standby queue manager takes over. The second is to stop both the active and the standby instances of the queue manager. If you issue the command endmqm IBMESBQM on the active instance of the queue manager -- without, as you will note, the -s option -- you will stop both the active and the standby instances. If you are running multiple instances of the queue manager and decide that you don't need to do so, then you can choose to stop only the standby instance by issuing the endmqm command with the -x option. The standby instance will no longer serve as standby, but the active instance will keep running.

Deleting a multi-instance queue manager

To delete multiple instances of a queue manager, you will first need to delete its references from other standby queue manager servers. Run the rmvmqinf command on all servers that have a definition of the queue manager, as shown in Listing 14:

Listing 14. Deleting references to a queue manager

rmvmqinf IBMESBQM

Next, you must delete the queue manager on all active queue manager servers. Run the dltmqm command, as shown in Listing 15, to delete a queue manager that has instances defined on other servers.

Listing 15. Deleting references to a queue manager

dltmqm IBMESBQM

Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1

Page 7 of 23

developerWorks

ibm.com/developerWorks/

You need not run this command on the same server where you created the queue manager; you can run this command on any server where that queue manager is defined. You could also delete MI queue managers just as well by executing the dltmqm IBMESBQM command on both servers.

Multi-instance queue manager runtime file system organization

Figure 2 shows a hierarchical representation of the MI queue manager /mqha file system. Queue manger-specific objects and active logs are stored in this location:

Figure 2. Hierarchical representation of MI queue manager file system

Managing a multi-instance queue manager

WebSphere MQ Explorer cannot run locally to more than one instance of an MI queue manager. You need to configure WebSphere MQ Explorer to manage MI queue managers remotely. Select Queue Managers => Add Remote Queue Manager menu item to add connections to an MI queue manager, as shown in Figure 3:

Figure 3. WebSphere MQ Explorer Navigator

The Add Queue Manager wizard will appear. Provide the queue manager name and select the appropriate connection method, as shown in Figure 4:

Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1

Page 8 of 23

ibm.com/developerWorks/

developerWorks

Figure 4. Add Queue Manager wizard

Click Next once you've made your selections. Next, you will need to provide connection details for the active and standby queue managers, and instruct WebSphere MQ Explorer to auto-reconnect to the active queue manager in case of a failover. Figure 5 demonstrates how to enter this information properly:

Figure 5. Setting up an MI queue manager on the Add Queue Manager wizard

If no further client configuration is required, click Finish. WebSphere MQ Explorer will now connect the active queue manager instance, as illustrated in Figure 6. Remember, in this sample configuration, the active queue manager resides on the server wmbmi1.in.ibm.com:

Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1

Page 9 of 23

developerWorks

ibm.com/developerWorks/

Figure 6. Connected active queue manager in MQ Explorer Navigator

The next logical step is to verify the auto-reconnect functionality by performing a controlled failover of the IBMESBQM queue manager.

Failover between multi-instance queue managers

You should perform a failover test of the MI queue manager IBMESBQM in a controlled environment. (Later in this article, you'll see a more rigorous test for the results of a network or power failure when an MI broker has been implemented as well.) Controlled failover can be achieved by stopping the active queue manager and seeing if the standby queue manager acquires a lock on the /mqha file system and becomes the active queue manager. Before you issue the command to stop the active instance of the MI queue manager, it is important to identify the server on which the active instance is running. You can do so using the dspmq command with the -x flag. In Listing 16, you can see that, in our sample setup, the queue manager IBMESBQM is running as the active queue manager on wmbmi1.in.ibm.com and as standby on wmbmi2.in.ibm.com:

Listing 16. Identifying the location of the active instance

[mqm@wmbmi1.in.ibm.com$ dspmq -x QMNAME(IBMESBQM) INSTANCE(wmbmi1.in.ibm.com) MODE(Active) INSTANCE(wmbmi2.in.ibm.com) MODE(Standby) [mqm@wmbmi2.in.ibm.com]$ dspmq -x QMNAME(IBMESBQM) INSTANCE(wmbmi1.in.ibm.com) MODE(Active) INSTANCE(wmbmi2.in.ibm.com) MODE(Standby) STATUS(Running as standby) STATUS(Running)

Now that you know where it's running, you can stop the active queue manager on wmbmi1.in.ibm.com with the endmqm -s IBMESBQM command, as shown in Listing 17:
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 10 of 23

ibm.com/developerWorks/

developerWorks

Listing 17. Stopping the active queue manager

[mqm@wmbmi1.in.ibm.com]$ endmqm -s IBMESBQM Quiesce request accepted. The queue manager will stop when all outstanding work is complete, permitting switchover to a standby instance. [mqm@wmbmi1.in.ibm.com]$ dspmq -x QMNAME(IBMESBQM) INSTANCE(wmbmi2.in.ibm.com) MODE(Active) [mqm@wmbmi2.in.ibm.com]$ dspmq -x QMNAME(IBMESBQM) INSTANCE(wmbmi2.in.ibm.com) MODE(Active) STATUS(Running) STATUS(Running elsewhere)

In Listing 17 above, you can see that the queue manager IBMESBQM stopped on wmbmi1.in.ibm.com and is now running as an active queue manager on wmbmi2.in.ibm.com. Thus, the test shows that IBMESBQM failed over from the first server to the second. You can also determine this fact from WebSphere MQ Explorer, which auto-reconnected to the active instance of the queue manager now running on wmbmi2.in.ibm.com, as shown in Figure 7:

Figure 7. Auto-reconnect after failover

Now there is only one instance of the queue manager running, and there would not be any failover if problems arise. Hence, to restore the standby queue manager, you should execute the strmqm -x IBMESBQM command on wmbmi1.in.ibm.com. This will ensure that the queue manager IBMESBQM runs actively on wmbmi2.in.ibm.com and passively on wmbmi1.in.ibm.com. To test that the failover occurs properly from wmbmi2.in.ibm.com to wmbmi1.in.ibm.com, and to restore the configuration to its original arrangement, just repeat the steps in this section.

Multi-instance brokers
The MI broker feature of Message Broker works with WebSphere MQ in one of two ways. Each broker instance is embedded into a WebSphere MQ service so that, when the queue manager switches over to the standby system, the broker is automatically started on the standby node. This feature is available in Message Broker V7.0.0.1 or later.
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 11 of 23

developerWorks

ibm.com/developerWorks/

You can configure MI brokers with a -d option on the mqsicreatebroker and mqsiaddbrokerinstance commands, where the value of the option is defined if you want to create the WebSphere MQ service or undefined if you want to remove it. When you first create the broker instance with this option, the queue manager has already been started and then the WebSphere MQ service is created, so the broker will not be started by the service until the queue manager has been stopped and restarted. This is true only when you are creating the broker. Thereafter, the WebSphere MQ service has control. Alternatively, the standby broker can run continuously in a semi-initialized state, waiting for the associated standby queue manager and shared broker configuration to become available. In this section, you will create the MI broker IBMESBBRK on the sample system using the MI queue manager IBMESBQM that you set up previously. To do this, you'll need to use the same user (mqm), and will make this user a member of the mqbrkrs group. Ensure that the gid of mqbrkrs remains same across all the servers. Here are the tasks to administer the MI broker:

Creating a multi-instance broker

There are three sets of actions that must be performed to configure Message Broker in multiinstance mode: 1. Create a shared work path directory on an NFS server 2. Create a WebSphere MQ MI queue manager 3. Create an MI broker In the previous section, you configured a shared work path and set up an MI queue manager, so you can move on to third step. First, you need to verify the server on which the active queue manage is running; this can be achieved using the dspmq -x command. In our setup, the active queue manager is on wmbmi1.in.ibm.com, so you should create the active broker instance on that server. The command used to create a broker is mqsicreatebroker, and its implementation and output for the sample system are shown in Listing 18:

Listing 18. Creating a broker

[mqm@wmbmi1.in.ibm.com]$ mqsicreatebroker IBMESBBRK -q IBMESBQM -e /mqha/WMB/IBMESBBRK AMQ8110: WebSphere MQ queue manager already exists. The setmqaut command completed successfully. The setmqaut command completed successfully. BIP8071I: Successful command completion.

Once the active instance of this broker is created, create the standby instance on wmbmi2.in.ibm.com using the mqsiaddbrokerinstance command:

Listing 19. Creating the standby instance of the broker

mqsiaddbrokerinstance IBMESBBRK -e /mqha/WMB/IBMESBBRK

The mqsiaddbrokerinstance command is used to create an MI broker on a UNIX/Linux or Windows server where Message Broker has been installed. The mqsiaddbrokerinstance
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 12 of 23

ibm.com/developerWorks/

developerWorks

command is also used to add a broker instance to any additional server on which you require multi-instance support. The command expects that the user has first created a multi-instanceenabled broker on an initial server using the mqsicreatebroker command. The syntax for this command varies by operating system: Windows:mqsiaddbrokerinstance <brokerName> -e <sharedWorkPath> -w <Workpath> -i UNIX/Linux:mqsiaddbrokerinstance <brokerName> -e <sharedWorkPath> -w <Workpath> Here, the -w option is optional in both cases. It indicates the directory in which working files specific to this broker instance should be stored locally on the server where the broker instance is going to run. On Linux and UNIX systems, the uid used to run this command must be a member of both the mqbrkrs and the mqm groups. Additionally, you need to make the uid and gid of mqbrkrs the same on all systems. Listing 20 shows the output of the mqsiaddbrokerinstance command on wmbmi2.in.ibm.com:
<serviceUserid> -a <servicePassword>

Listing 20. Output of mqsiaddbrokerinstance

[mqm@wmbmi2.in.ibm.com]$ mqsiaddbrokerinstance IBMESBBRK -e /mqha/WMB/IBMESBBRK AMQ7272: WebSphere MQ configuration information already exists. BIP8071I: Successful command completion.

Starting and stopping a multi-instance broker

Now that duplicate instances of the broker IBMESBBRK have been created on both servers, the next step is to start those instances. One will run as the active instance and the other as the standby instance, based on the state of the instances of the corresponding queue manager IBMESBQM. Start the brokers by issuing the mqsistart command on both servers, as shown in Listing 21:

Listing 21. Starting the broker instances

[mqm@wmbmi1.in.ibm.com]$ mqsistart IBMESBBRK BIP8096I: Successful command initiation. Check the system log to ensure that the component started without problems and that it continues to run without problems. [mqm@wmbmi2.in.ibm.com]$ mqsistart IBMESBBRK BIP8236I: A standby instance of Broker IBMESBBRK has been started against the standby queue manager IBMESBQM. The active broker instance and active queue manager instance are running elsewhere. The multi-instance broker has started in standby mode. The broker will not become active until the current active broker instance and active queue manager instance end, or are stopped. No action required.

The broker IBMESBBRK starts as the active broker on wmbmi1.in.ibm.com, and starts in standby mode on wmbmi2.in.ibm.com. Broker processes running as active and standby instances are described below, but first, you need to create an execution group on the active broker on wmbmi1.in.ibm.com by executing the mqsicreateexecutiongroup IBMESBBRK -e IBMESBEG command:
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 13 of 23

developerWorks

ibm.com/developerWorks/

Listing 22. Creating an execution group

[mqm@wmbmi1.in.ibm.com]$ mqsicreateexecutiongroup IBMESBBRK -e IBMESBEG BIP1124I: Creating execution group 'IBMESBEG' on broker 'IBMESBBRK'... BIP1117I: The execution group was created successfully. The broker has initialized the execution group. BIP8071I: Successful command completion. [mqm@wmbmi1.in.ibm.com]$ ps -eaf | grep I IBMESBEG mqm 364 32622 17 20:00 ? 00:00:07 DataFlowEngine IBMESBBRK 9a38eebe-2701-0000-0080-ad888bea8d47 IBMESBEG

Once the execution group has been created, it starts automatically. You can verify this by using the ps command and doing a grep for the string IBMESBEG (which is the execution group name). Now that the broker and DataFlowEngine are up on the active instance, you can examine the processes running on the active and standby instances. On the active node, you will find the following broker processes running:
bipservice bipbroker biphttplistener One DataFlowEngine

process for each execution group created on each broker.

On the standby node, you will find following broker processes running: bipservice (in standby mode) bipbroker (in standby mode) You can see the details for the sample systems in Listing 23:

Listing 23. Processes running on active and standby nodes

[mqm@wmbmi1.in.ibm.com]$ ps -eaf | grep IBMESBBRK mqm mqm mqm mqm 364 32622 1 20:00 ? 00:00:07 DataFlowEngine IBMESBBRK 9a38eebe-2701-0000-0080-ad888bea8d47 IBMESBEG 32617 1 0 19:56 ? 00:00:00 bipservice IBMESBBRK 32622 32617 0 19:56 ? 00:00:02 bipbroker IBMESBBRK 32665 32622 0 19:56 ? 00:00:01 biphttplistener IBMESBBRK

[mqm@wmbmi2.in.ibm.com]$ ps -eaf | grep IBMESBBRK mqm mqm 31916 31921 1 31916 0 20:04 ? 0 20:04 ? 00:00:00 bipservice IBMESBBRK 00:00:00 bipbroker IBMESBBRK

It is important to know how to stop an MI broker. Using the mqsistop command with the broker name will end an MI broker or a broker instance. No switchover will happen upon execution of this command; it will just bring down the running broker. The command must be executed on both servers separately to bring down the active and standby instances of the broker.

Deleting a multi-instance broker

Though you won't delete the MI brokers on your simple system at this point, it is important to know how to delete these MI components. The order of deletion of an MI broker and its associated
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 14 of 23

ibm.com/developerWorks/

developerWorks

instances is important, and you need to use the correct command for each step in the process. Here is an overview or how to delete MI brokers, using the sample setup: 1. Stop the MI broker IBMESBBRK on the active node. 2. Stop the MI broker IBMESBBRK on the standby node. 3. Remove the broker instance IBMESBBRK on the standby node using the mqsiremovebrokerinstance IBMESBBRK command. 4. Remove the broker IBMESBBRK on the active node using the mqsideletebroker IBMESBBRK command.

Multi-instance broker runtime file system organization

Figure 8 shows a hierarchical representation of the MI broker's /mqha file system. The broker's components and registry-specific data are stored in this location:

Figure 8. Hierarchical representation of an MI broker file system

Backing up and restoring a multi-instance broker

No special techniques are needed to back up and restore an MI broker. You would perform these tasks in a standard way, using the commands in Listing 24 on the active instance:

Listing 24. Backing up and restoring a broker

mqsibackupbroker IBMESBBRK -d /backup/WMB mqsirestorebroker IBMESBBRK -d /backup/WMB -a <archive file name>

The mqisbackupbroker command detects an MI broker and backs up the registry and configuration from the shared work path of the broker. Similarly, mqsirestorebroker restores the registry and configuration to the shared work path of the broker. Execute the backup command with the active instance of the broker. While using the restore command, stop all brokers with which the broker you're restoring shares archive data (both active and standby instances). If you run this command when a relevant broker is active the results may be unpredictable.

Failover switching between multi-instance brokers

Before starting with this section, you should have clear understanding what exactly failover means in the context of MI brokers. When a queue manager goes down for any reason, this will cause the associated broker to fail; Message Broker will then switch to the standby instance. At that time,
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 15 of 23

developerWorks

ibm.com/developerWorks/

the previously active instance of the broker sets itself to standby mode, and the standby instance becomes active. The MI broker relies on the status of the MI queue manager. Consider two failover scenarios: a controlled switchover and an uncontrolled switchover due to a power or network failure.

Controlled switchover of a multi-instance broker

Begin the controlled switchover test by displaying the state of the MI broker on wmbmi1.in.ibm.com, to confirm that it's active. You can do this by issuing the command in Listing 25, and you should see output similar to what's shown in the listing:

Listing 25. Displaying the state of IBMESBBRK on wmbmi1.in.ibm.com

[mqm@wmbmi1 .in.ibm.com]$ mqsilist BIP1295I: Broker 'IBMESBBRK' is a multi-instance broker running in active mode on multi-instance queue manager 'IBMESBQM'. BIP8071I: Successful command completion

Next, do the same for the queue manager, as shown in Listing 26:

Listing 26. Displaying the state of IBMESBQM on wmbmi1.in.ibm.com

[mqm@wmbmi1 ~]$ dspmq m IBMESBQM x QMNAME(IBMESBQM) STATUS(Running) INSTANCE(wmbmi1.in.ibm.com) MODE(Active) INSTANCE(wmbmi2.in.ibm.com) MODE(Standby)

Now switch to the standby server, wmbmi2.in.ibm.com, and execute the same commands. You should see the results shown in Listings 27 and 28, indicating that the queue manager and broker instances there are the standby instances.

Listing 27. Displaying the state of IBMESBBRK on wmbmi2.in.ibm.com

[mqm@wmbmi2 .in.ibm.com]$ mqsilist BIP1294I: Broker 'IBMESBBRK' is a multi-instance broker running in standby mode on multi-instance queue manager 'IBMESBQM'. More information will be available when the broker instance is active. BIP8071I: Successful command completion.

Listing 28. Displaying the state of IBMESBQM on wmbmi2.in.ibm.com

[mqm@wmbmi2.in.ibm.com]$ dspmq m IBMESBQM x QMNAME(IBMESBQM) STATUS(Running as standby) INSTANCE(wmbmi2.in.ibm.com) MODE(Standby) INSTANCE(wmbmi1.in.ibm.com) MODE(Active)

Next, back on wmbmi1.in.ibm.com, stop the active instance of IBMESBQM, as shown in Listing 29:

Listing 29. Stopping the active instance of IBMESBQM

[mqm@wmbmi1 ~]$ endmqm -s IBMESBQM Quiesce request accepted. The queue manager will stop when all outstanding work is complete, permitting switchover to a standby instance.

Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1

Page 16 of 23

ibm.com/developerWorks/

developerWorks

Now display its state, as shown in Listing 30. The output should indicate that what had been the standby instance has become the active instance:

Listing 30. Confirming that IBMESBQM has stopped on wmbmi1.in.ibm.com

[mqm@wmbmi1.in.ibm.com]$ dspmq -m IBMESBQM x QMNAME(IBMESBQM) STATUS(Running elsewhere) INSTANCE(wmbmi2.in.ibm.com) MODE(Active)

Use the command in Listing 31 to list the state of the broker IBMESBBRK. You will see that it too has stopped running on wmbmi1.in.ibm.com:

Listing 31. Confirming that IBMESBBRK has stopped on wmbmi1.in.ibm.com

[mqm@wmbmi1.in.ibm.com]$ mqsilist BIP1292I: The multi-instance Broker 'IBMESBBRK' on multi-instance queue manager 'IBMESBQM' has stopped. BIP8071I: Successful command completion.

Now switch back to wmbmi2.in.ibm.com again. Issue the command in Listing 32 to confirm that the instance of IBMESBQM there has become active:

Listing 32. Confirming that IBMESBQM has become active on wmbmi2.in.ibm.com

[mqm@wmbmi2.in.ibm.com]$ dspmq -m IBMESBQM x QMNAME(IBMESBQM) STATUS(Running) INSTANCE(wmbmi2.in.ibm.com) MODE(Active)

Finally, use the command in Listing 33 to confirm that IBMESBBRK on wmbmi2.in.ibm.com is active as well:

Listing 33. Confirming that IBMESBBRK is active on wmbmi2.in.ibm.com

[mqm@wmbmi2 ~]$ mqsilist BIP1295I: Broker 'IBMESBBRK' is a multi-instance broker running in active mode on multi-instance queue manager 'IBMESBQM'. BIP8071I: Successful command completion.

With IBMESBQM and IBMESBBRK active on wmbmi2.in.ibm.com, the switchover from one server to the other is now complete.

Uncontrolled failover due to power or network failure

In the previous section, you saw how processing could fail over from one server to another in a controlled environment. But what happens when one server's processing is abruptly terminated, or when it is unexpectedly disconnected from the network? Message Broker's high availability features can handle such a scenario. To perform this failover test, create a real-time scenario with
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 17 of 23

developerWorks

ibm.com/developerWorks/

message flow processing in the execution group IBMESBEG on the broker IBMESBBRK. Create and deploy a very simple message flow (shown in Figure 9) having following nodes: MQInput Node (IBMESB_IN) Compute Node (Copy Message) MQOutput Node (IBMESB_OUT)

Figure 9. Message flow

The input and output queues (IBMESBQM_IN and IBMESBQM_OUT) have their persistence property set to persistent, which means that all messages arriving on these queues are made persistent. This will avoid any message loss during failover. In this section, you will see that the client connection also survives through the failover. This client would put messages on the input queue IBMESBQM_IN of the queue manager IBMESBQM. is a ready-to-use sample program that comes with WebSphere MQ; you can use it to check out the connection status of the message flow with the queue manager. It keeps putting messages on the input queue and informs you about the status of the put. You can use this sample program to check the connection status as well. Listing 34 shows the results when the program is invoked well before wmbmi1.in.ibm.com goes down due to a power or network failure. The program keeps running until wmbmi2.in.ibm.com takes over from the failed server:
amqsphac

Listing 34. amqsphac output illustrating failover in action

[mqm@wmbmi2.in.ibm.com]$ amqsphac IBMESBQM_IN IBMESBQM Sample AMQSPHAC start target queue is IBMESBQM_IN message <Message 1> // Client application is well connected here // and successfully putting message to Input Queue message <Message 2> message <Message 3> message <Message 4> message <Message 5> message <Message 6> message <Message 7> message <Message 8> message <Message 9> message <Message 10> message <Message 11> message <Message 12> message <Message 13> message <Message 14> message <Message 15> // wmbmi1 server has gone down here and the connection of client is lost 20:41:53 : EVENT : Connection Reconnecting (Delay: 168ms) 20:41:54 : EVENT : Connection Reconnecting (Delay: 1220ms) 20:41:55 : EVENT : Connection Reconnecting (Delay: 2324ms) 20:41:57 : EVENT : Connection Reconnecting (Delay: 4494ms) 20:42:02 : EVENT : Connection Reconnecting (Delay: 8731ms)

Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1

Page 18 of 23

ibm.com/developerWorks/

developerWorks

20:42:10 : EVENT : Connection Reconnecting (Delay: 19732ms) // wmbmi2 server has gained the control here and has become active. 20:42:31 : EVENT : Connection Reconnected // Client Connection to Queue Manager is re-established message <Message 16> // Client application is successfully putting the messages again message <Message 17> message <Message 18> message <Message 19> message <Message 20>

We got the results in Listing 34 from our own network failure test when we disconnected a network cable from the wmbmi1.in.ibm.com server. The standby queue manager IBMESBQM on wmbmi2.in.ibm.com became the active queue manager, and IBMESBBRK became the active broker, in less than two minutes. The client was then automatically reconnected to the active queue manager. The amqsphac client produced 20 messages, all of which were processed and put in the IBMESB_OUT queue by the copy message flow. Here is the output:

Listing 35. Output demonstrating successful failover

[mqm@wmbmi2.in.ibm.com]$ dspmq m IBMESBQM QMNAME(IBMESBQM) INSTANCE(wmbmi2.in.ibm.com) MODE(Active) -x STATUS(Running)

[mqm@wmbmi2.in.ibm.com]$ echo "DISPLAY QLOCAL(IBMESBQM_OUT) CURDEPTH" | runmqsc IBMESBQM 5724-H72 (C) Copyright IBM Corp. 1994, 2009. ALL RIGHTS RESERVED. Starting MQSC for queue manager IBMESBQM. 1 : DISPLAY QLOCAL(IBMESBQM_OUT) CURDEPTH AMQ8409: Display Queue details. QUEUE(IBMESBQM_OUT) TYPE(QLOCAL) CURDEPTH(20) One MQSC command read. No commands have a syntax error. All valid MQSC commands were processed.

A limitation with Message Broker Explorer

There is currently a limitation when using WebSphere Message Broker Explorer to examine the standby instance of an MI broker. Although you can list and view all instances of an MI queue manager on WebSphere MQ Explorer, and switch the Explorer connection to the standby queue manager in case of a failover, you can't list or view the standby instance of an MI broker on the WebSphere Message Broker Explorer in a failover situation.

Conclusion
This article outlined the concepts and basic architecture of MI queue mangers and MI brokers, and explained and how to create, configure, manage, and test these MI components for high availability. The systems described here have included one broker or queue manager instance that is active and one standby. This represents an active-passive architecture for designing highly available messaging solutions. There is another type of HA architecture called an active-active scenario, in which you use multiple sets of MI queue managers and brokers overlayed on a WebSphere MQ cluster. Part 2 of this article will describe the other HA scenario in detail.
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 19 of 23

developerWorks

ibm.com/developerWorks/

Resources
WebSphere MQ resources Testing and support statement for WebSphere MQ multi-instance queue managers, This IBM Technote describes support for WebSphere MQ multi-instance queue managers on various operating systems, including IBM i5/OS and Solaris. Backing up a multi-instance queue manager This IBM Technote provides detailed steps for backing up a multi-instance queue manager. Network File System (NFS) V4 Protocol This document from the Network Working Group specifies an Internet standards track protocol for the Internet community. WebSphere MQ V7.5 information center A single Web portal to all WebSphere MQ documentation, with conceptual, task, and reference information on installing, configuring, and using your WebSphere MQ environment. WebSphere Message Broker resources WebSphere Message Broker developer resources page Technical resources to help you use WebSphere Message Broker for connectivity, universal data transformation, and enterprise-level integration of disparate services, applications, and platforms to power your SOA. WebSphere Message Broker product page Product descriptions, product news, training information, support information, and more. WebSphere Message Broker V8 information center A single Web portal to all WebSphere Message Broker V8 documentation, with conceptual, task, and reference information on installing, configuring, and using your WebSphere Message Broker environment. What's new in WebSphere Message Broker V8 WebSphere Message Broker V8 provides universal connectivity with its ability to route and transform messages from anywhere to anywhere. Through its simple programming model and a powerful operational management interface, it makes complex application integration solutions much easier to develop, deploy, and maintain. This article describes the major enhancements in V8. Download free trial version of WebSphere Message Broker V8 WebSphere Message Broker V8 is an ESB built for universal connectivity and transformation in heterogeneous IT environments. It distributes information and data generated by business events in real time to people, applications, and devices throughout your extended enterprise and beyond. WebSphere Message Broker documentation library WebSphere Message Broker specifications and manuals. WebSphere Message Broker forum Get answers to your technical questions and share your expertise with other WebSphere Message Broker users. WebSphere Message Broker support page
Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1 Page 20 of 23

ibm.com/developerWorks/

developerWorks

A searchable database of support problems and their solutions, plus downloads, fixes, problem tracking, and more. WebSphere resources developerWorks WebSphere developer resources Technical information and resources for developers who use WebSphere products. developerWorks WebSphere provides product downloads, how-to information, support resources, and a free technical library of more than 2000 technical articles, tutorials, best practices, IBM Redbooks, and online product manuals. developerWorks WebSphere application integration developer resources How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application integration and business integration solutions. Most popular WebSphere trial downloads No-charge trial downloads for key WebSphere products. WebSphere forums Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users. WebSphere on-demand demos Download and watch these self-running demos, and learn how WebSphere products and technologies can help your company respond to the rapidly changing and increasingly complex business environment. WebSphere-related articles on developerWorks Over 3000 edited and categorized articles on WebSphere and related technologies by top practitioners and consultants inside and outside IBM. Search for what you need. developerWorks WebSphere weekly newsletter The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing. WebSphere-related books from IBM Press Convenient online ordering through Barnes & Noble. WebSphere-related events Conferences, trade shows, Webcasts, and other events around the world of interest to WebSphere developers. developerWorks resources Trial downloads for IBM software products No-charge trial downloads for selected IBM DB2, Lotus, Rational, Tivoli, and WebSphere products. developerWorks business process management developer resources BPM how-to articles, downloads, tutorials, education, product info, and other resources to help you model, assemble, deploy, and manage business processes. developerWorks blogs Join a conversation with developerWorks users and authors, and IBM editors and developers. developerWorks tech briefings

Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1

Page 21 of 23

developerWorks

ibm.com/developerWorks/

Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most challenging software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide. developerWorks podcasts Listen to interesting and offbeat interviews and discussions with software innovators. developerWorks on Twitter Check out recent Twitter messages and URLs. IBM Education Assistant A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.

Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1

Page 22 of 23

ibm.com/developerWorks/

developerWorks

About the authors

Ashwin Gupta Ashwin Gupta has been a Staff Software Engineer with IBM India for five years. His expertise spans the IBM middleware product portfolio, including WebSphere Message Broker, WebSphere MQ, WebSphere MQ FTE, WebSphere eXtreme Scale, and WebSphere Transformation Extender. He currently works as a Technical Sales Specialist with the Worldwide WebSphere Business Partners Team, working on Business Partner Technical Enablement for various WebSphere products. You can contact Ashwin at gupta.ashwin@in.ibm.com.

Rahul Gupta Rahul Gupta is an Advisory IT Specialist with IBM Global Technology Services. He is a Certified SOA Architect with eight years experience in IBM messaging technologies, and currently, he works as a middleware consultant for various clients in North America. His core experience is in lab testing, performance tuning, and development for WebSphere Message Broker and WebSphere MQ. He has been a speaker on messaging-related topics at various WebSphere conferences, and has been recognized as an inventor by the IBM Innovation Community. He has also authored IBM Redbooks and developerWorks articles on messaging and application infrastructure. You can contact Rahul at rahul.gupta@us.ibm.com.

Venkatesh Murugan Venkatesh Nainar Murugan is a Technical Services Professional and has worked as a WebSphere MQ and WebSphere Message Broker Specialist with IBM India for 1 1/2 years. He currently works on the administration and support of WebSphere MQ and WebSphere Message Broker on various platforms. His previous experience includes designing, developing, and testing these products. You can contact Venkatesh at n.venkatesh@in.ibm.com. Copyright IBM Corporation 2010 (www.ibm.com/legal/copytrade.shtml) Trademarks (www.ibm.com/developerworks/ibm/trademarks/)

Configuring and administering multi-instance brokers for high availability in IBM WebSphere Message Broker - Part 1

Page 23 of 23