MQ Queue Plug-In

Introduction

The MQ Queue plug-in is part of a suite of plug-ins designed to monitor an IBM WebSphere MQ server. MQ is a middleware product which allows independent client programs to communicate by passing messages to each other via a Message Queue. These queues provided added benefits by adding reliability of communications, and storing and persisting messages allowing client programs to operate asynchronously.

The MQ Queue plug-in monitors all configured queues for a particular Queue Manager (an MQ system service which holds a set of queues). Using the Geneos framework administrators can set alerts for events such as a queue approaching its maximum number of messages, or to ensure that the expected number of producers and consumers are connected to the queue. Additionally, database logging of monitored metrics allows these figured to be used later in calculating trends and aid in capacity planning.

MQ Architecture

mq_queue_image2

Websphere MQ servers are based around queues of messages, each Message containing some arbitrary application data. Messages are contained within Queues, which are in turn managed by a Queue Manager. An MQ server may contain several queue managers.

Client applications communicate with each other by sending messages via a queue. To do so they connect to a queue manager - typically using TCP/IP - and send or receive messages accordingly.

Communication between components are handled using pre-configured links called Channels. Channels are mono-directional, so a full-duplex link between two components requires two channels.

A queue manager also has an associated Command Server, which allows a remote client to execute commands. The MQ Queue plug-in connects as a client to the configured queue manager, and obtains monitoring data by issuing status query commands using the command server.

Views

View

MQ Queue produces a single dataview displaying the configured queues for the queue manager it is connected to, and the selected attributes for each view. By default these attributes are the same as those reported by a DISPLAY QSTATUS command. An example of this is shown below.

mq_queue_image3

The legends below list all columns which can be configured for the plug-in. For a more in-depth discussion of columns, please see the columns configuration section.

Headline Legend

Name	Description
mqServer	Displays the connection details used to connect to the MQ server.
mqChannelTable	Displays the MQ channel table file used to connect to the MQ server.
queueManager	The name of the queue manager the plug-in is connecting to.
connectionStatus	The current connection status (`UP` or `DOWN`).
samplingStatus	The current status of the plug-in.

Table Legend

Name	Description
name	Name of the queue being monitored.
queueType	Type of queue, one of `Alias`, `Cluster`, `Local`, `Remote`, `Model`.
queueDefinition	Indicates how the queue was defined, one of `Predefined`, `Permanent`, `Shared`, `Temporary.`
description	A user-configured description of the queue.
maxQueueDepth	Maximum queue depth (number of messages).
currentQueueDepth	Current number of messages on the queue.
percentQueueDepth	Current queue depth as a percentage of the maximum.
maxQueueCapacity	Maximum queue capacity shows the maximum amount of data a queue can contain, calculated using `maxQueueDepth` and maximum message size.
creationDate	Date the queue was created, in the form `yyyy‑mm‑dd`.
creationTime	Time the queue was created, in the form `hh.mm.ss`.
creationDateTime	A concatenation of the `creationDate` and `creationTime` columns showing when the queue was created, in the form `yyyy‑mm‑dd hh.mm.ss`.
alterationDate	Date the queue definition was last altered, in the form `yyyy‑mm‑dd`.
alterationTime	Time the queue definition was last altered, in the form `hh.mm.ss`.
alterationDateTime	A concatenation of the `alterationDate` and `alterationTime` columns showing when the queue definition was last altered, in the form `yyyy‑mm‑dd hh.mm.ss`.
remoteQueueManager	Name of the remote queue manager hosting the remote queue.
remoteQueue	The local queue name on the remote queue manager, for a remote queue.
transmitQueueName	Name of the transmission queue, for a remote queue.
inhibitPut	Whether put operations are allowed for the queue - values are `Allowed` or `Inhibited`.
inhibitGet	Whether get operations are allowed for the queue - values are `Allowed` or `Inhibited`.
defaultPersistence	The default message persistence for this queue - values are `Persistent` or `Not Persistent`.
msgEnqCount	Number of enqueued messages put on the queue but not committed, at the time of a queue statistics reset.
msgDeqCount	Number of dequeued messages got from the queue but not committed, at the time of a queue statistics reset.
highQueueDepth	Maximum number of messages on a queue, at the time of a queue statistics reset.
timeSinceReset	Time in seconds since the queue statistics were last reset.
oldestMsgAge	Age of the oldest message on the queue in seconds.
openInputCount	Number of input processes which are using the queue (an input process GETs messages from the queue).
openOutputCount	Number of output processes which are using the queue (an output process PUTs messages on the queue).
lastGetDate	Date a message was last taken from the queue, in the form `yyyy‑mm‑dd`.
lastGetTime	Time a message was last taken from the queue, in the form `hh.mm.ss`.
lastGetDateTime	A concatenation of the `lastGetDate` and `lastGetTime` columns showing when a message was last taken from the queue, in the form `yyyy‑mm‑dd hh.mm.ss`.
lastPutDate	Date a message was last put on the queue, in the form `yyyy‑mm‑dd`.
lastPutTime	Time a message was last put on the queue, in the form `hh.mm.ss`.
lastPutDateTime	A concatenation of the `lastPutDate` and `lastPutTime` columns showing when a message was last put on the queue, in the form `yyyy‑mm‑dd hh.mm.ss`.
queueTimeIndicator	Indicator of the amount of time a message stayed on the queue. Two comma-separated values are displayed, showing the message time in microseconds averaged over a short and longer periods of time.
stagnantMsgCount	Number of stagnant messages on the queue. A message is considered stagnant if it is older than a user-defined time period.
averageMsgAge	The average message age (in seconds) of all messages on the queue at the time of sampling.
newestMsgAge	The age of the newest message on the queue in seconds.
processName	Name of the process servicing the queue.
clusterName	Name of the cluster to which the queue belongs.
triggerData	Additional data appended to the trigger message when a trigger event occurs.
triggerControl	Controls whether trigger messages are created to start an application to service the queue. Values are `On` or `Off`.
triggerDepth	The number of messages of priority `triggerMessagePriority` that must be on the queue before a trigger message is written, when `triggerType` is Depth.
triggerMessagePriority	Message priority threshold for trigger message creation. Values are positive integers, with 0 being the lowest priority.
triggerType	Controls the conditions under which a trigger message is created. Values are `None`, `First`, `Every` and `Depth`.
defaultBind	The default binding used when a client connects to a queue, and the queue is part of a cluster. Values are On `Open` or `Not Fixed`.
usage	Indicates the queue usage, either `Normal` or `Transmission`.

Prerequisites

Before running the MQ Queue plug-in, you must first install the IBM WebSphere MQ Client v7.1 (or greater) software on the machine which the Netprobe will run on. This client comes as part of the MQ software bundle and must be installed correctly - simply copying across the DLLs or shared object files will typically result in a 2012 (MQ environment) error.

On Windows, the mqic.dll library needs to be available to the Netprobe. This is usually done by ensuring that the directory in which this library resides is configured on the PATH environment variable available to the Netprobe.

On Solaris/Linux, the libmqic.so library needs to be available to the Netprobe. This is usually done by ensuring that the directory in which this library resides is configured on the LD_LIBRARY_PATH environment variable available to the Netprobe.

On AIX, the libmqic.a library needs to be available to the Netprobe. This is usually done by ensuring that the directory in which this library resides is configured on the LD_LIBRARY_PATH or LIBPATH environment variable available to the Netprobe.

In order to gather status commands, the MQ command server must be running. Additionally, the Netprobe user must be granted the following permissions to operate, if it is not a member of the mqm administration group:

+connect+dsp permissions on the Queue Manager object
+put+inq permissions on the SYSTEM.ADMIN.COMMAND.QUEUE. If using the send queue parameter, these permissions should be set on the named queue alias instead.
+get+dsp permissions on the SYSTEM.DEFAULT.MODEL.QUEUE. If using the receive queue parameter, the permissions +get +dsp+put+inq should be set on the named queue instead.

In addition certain columns will need additional permissions to be granted on each queue to be monitored as below. The table in the column setting section describes which columns are of which type.

Inquire Queue Status columns require no extra permissions.
Inquire Queue columns require +dsp permissions on each queue.
Reset Queue Statistics columns require +dsp+chg permissions on each queue.
Browse Queue columns require +browse permissions on each queue.

It is also advisable to enable queue monitoring to gather status data, either for the queue manager as a whole or the individual queues which will be monitored. This can either be done from the WebSphere MQ Explorer, or by issuing the following commands in runmqsc:

alterqmgr monq(medium) to enable monitoring for all queues on the queue manager with monq(qmgr) set, or
alterqlocal(queueName)monq(medium) to enable monitoring for a specific queue.

Note: If you are using a very short sample interval for your plug-in, it is advised to increase the rate of data collection in MQ from medium to high.

Plug-in Configuration

The following parameters can be configured for this plug-in:

Connection Details

The connection section describes how the MQ Queue plug-in will connect to the MQ server. There are two ways to configure a connection:

Configure an inline connection using the MQSERVER variable.
Configure a connection using a channel table.

If multiple different configuration settings and environment variables exist, MQ Queue will choose a configuration using the following order:

Using the file specified in the mqChannelTable setting.
Using the file specified by the MQCHLLIB and MQCHLTAB environment variables. In this case, variables configured in the managed entity section will be used in preference to those configured in the Netprobe start script.
Using the connection details specified in the mqServer setting.
Using the connection details specified in the MQSERVER environment variable. In this case if the variable is configured in the managed entity section it will be used in preference to a value configured in the Netprobe start script.

If no connection details are supplied, an error message will be produced in the Netprobe log and in the samplingStatus headline of the sampler dataview.

This sub-section contains details relating to SSL-secured connections to an MQ server.

Note: If using SSL connections, it is recommended that all MQ plug-ins running on that Netprobe use the same connection details. The reason for this is that the MQ API only supports a single SSL connection instance per process, and so if using multiple different SSL configurations only the first plug-in will connect successfully.

Mandatory: No

connection > ssl > keyRepository

This setting configures the location of the SSL key repository, which is used when establishing an SSL connection to an MQ server. The key repository contains SSL certificates, and should be created and managed using the IBM key manager utility.

To use this setting specify the full path of the key repository file. It is recommended to exclude the file extension (.kdb) for this value, so that it matches the syntax of the MQSSLKEYR environment variable.

The value of this setting is only required when using SSL connections. These can be specified in two ways:

By configuring the mqServer setting (or MQSERVER environment variable) and also configuring the cipherSpec setting.
By using a channel table (mqChannelTable setting or MQCHLLIB and MQCHLTAB environment variables) which specify an SSL connection.

If specified, this setting will override the value in the MQSSLKEYR environment variable. For more details on this variable, please consult this IBM documentation page.

Mandatory: No

connection > ssl > cipherSpec

This setting is used when the connection is configured using the mqServer setting or the MQSERVER environment variable. If configured, this setting specifies the SSL encryption which will be used to secure the connection, subject to SSL certificate checks.

The value of this setting can either be typed in directly, or selected from the following list of options:

DES_SHA_EXPORT
DES_SHA_EXPORT1024
FIPS_WITH_3DES_EDE_CBC_SHA
FIPS_WITH_DES_CBC_SHA
NULL_MD5
NULL_SHA
RC2_MD5_EXPORT
RC4_56_SHA_EXPORT1024
RC4_MD5_EXPORT
RC4_MD5_US
RC4_SHA_US
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_DES_CBC_SHA
TRIPLE_DES_SHA_US

For more information on cipher specs with MQ, please consult this IBM documentation article.

Mandatory: No

connection > mqSender

This setting allows the user to specify the name of the queue to send commands on, for execution by the command server. If not specified, the plug-in will send requests for monitoring data (commands) using the default name SYSTEM.ADMIN.COMMAND.QUEUE.

In practise this setting need not be defined unless the MQ server has been configured with a non-default command queue name.

Mandatory: No

Default: SYSTEM.ADMIN.COMMAND.QUEUE

connection > mqReceiver

This setting allows the user to specify the name of a (permanent) queue to receive command results (monitoring data) from, following execution by the command server. If not specified the plug-in will use the default behaviour of the IBM MQAI library, which is to create a new dynamic queue for the reply (based on the SYSTEM.DEFAULT.MODEL.QUEUE) which is automatically deleted after use.

Users may wish to configure this setting if they are having problems with command results being placed on the dead letter queue. These "dead" messages contain stale monitoring data and so are safe to delete, but their creation can be avoided by use of a permanent reply queue. Use of a permanent reply queue can avoid also the (small) cost of creation/deletion of a dynamic reply queue.

When using a permanent queue only a single queue per MQ server instance is required (although you may use additional queues as required). The Netprobe user will require a minimum of +get+dsp +put+inq permissions on the queue to operate correctly. The queue can safely be shared by multiple MQ plug-in instances (either running on the same or different Netprobes) as each MQ plug-in will obtain its own monitoring data from the queue using a correlation id. The depth of the permanent queue should remain at (or close to) 0. Any messages on the queue older than a few minutes may be safely deleted.

Mandatory: No

connection > securityExitConfig > securityExit

This setting allows the user to specify a client side security exit to use when authenticating with the MQ server. See the IBM documentation for the SecurityExit field on the MQCD structure for more information.

Note: This configuration will only work with MQServer. If a channel table is used in the configuration then the settings here will be ignored.

Mandatory: No

connection > securityExitConfig > remoteUserIdentifier

This setting allows the user to specify the Remote User Identifier that will be sent to the client side security exit and the server when authenticating. Only the first 12 characters will be used, see the IBM documentation for the RemoteUserIdentifier field on the MQCD structure for more information.

Mandatory: No

connection > securityExitConfig > remotePassword

This setting allows the user to specify the Remote Password that will be sent to the client side security exit and the server when authenticating. Only the first 12 characters will be used, see the IBM documentation for the RemotePassword field on the MQCD structure for more information.

Mandatory: No

connection > securityConnectionAuthenticationConfig > username

This setting allows the user to specify a username in order to connect to the MQ server which supports Security Connection Authentication. The username provided does not need to be part of the mqm group. But keep in mind that the user running the Netprobe should have been given the default MQ user permissions. This includes having the user added to the mqm group, authorised to connect to the Queue manager, channel, and queue. As the netprobe user would be the one used for MQ authorisation checks. But if ADOPTCTX attribute is set to YES then the Connection Authentication username provided would be used instead throughout the connection. See the IBM documentation on MQCSP password protection and on Connection Authentication: Configuration for more information.

Note: The Security Connection Authentication is applicable only to IBM MQ Version 8.0. If the version is lower than 8.0 then this configuration will have no bearing.

Mandatory: No

connection > securityConnectionAuthenticationConfig > password

This setting allows the user to specify the password for the given username that will connect to the MQ server which supports Security Connection Authentication. See the IBM documentation on MQCSP password protection and on Connection Authentication: Configuration for more information.

Mandatory: No

Queues

By default MQ Queue will monitor all queues on the configured queue manager. The queues section allows users to configure a set of queues they are interested in, thereby filtering the results returned and also reducing the processing that the plug-in must perform.

queueManager

This setting specifies the name of the queue manager the plug-in is to connect to. If a channel table is used, then this name should match the one specified for the connection.

Mandatory: Yes

queues

The column setting configures a single column which will be displayed in the resulting dataview. For a description of what each column shows, please see the view section table legend.

Columns will be added to the dataview in the order in which they are configured. If a column has already been added, then a duplicate entry will be ignored. Columns may also be ignored if the disableQueueResetAttributes or disableQueueBrowseAttributes settings have been enabled.

Column data comes from various MQ attributes obtained by executing commands. These commands will only be run if the column that requires this data has been configured. The commands which may be run include:

Inquire queue - obtains queue configuration values
Inquire queue status - obtains information about the current status of the queue.
Reset queue statistics - resets queue statistics, giving count values since the last reset.
Browse queue messages - browses messages on a queue to calculate statistics.

If using columns taking data from queue status attributes, you should enable queue monitoring as described in the prerequisites section of this document otherwise no values will be returned.

Note: Resetting queue statistics affects all applications using reset statistics on that queue. In particular, monitoring the same queue from two different instances of MQ Queue (perhaps running on different gateways) will produce unexpected results since the values returned will not be over the sample interval, but from the last reset time.

The following table shows which command is run to obtain data for each column. Some columns can obtain the same data from multiple commands - in this case MQ Queue will choose which command to run, in order to minimise the total number of commands run.

Column	Queue	Queue Status	Reset Queue	Browse
queueType	X
queueDefinition	X
description	X
maxQueueDepth	X
currentQueueDepth	X	X
percentQueueDepth	X
maxQueueCapacity	X
creationDate	X
creationTime	X
creationDateTime	X
alterationDate	X
alterationTime	X
alterationDateTime	X
remoteQueueManager	X
remoteQueue	X
transmitQueueName	X
inhibitPut	X
inhibitGet	X
defaultPersistence	X
msgEnqCount			X
msgDeqCount			X
highQueueDepth			X
timeSinceReset			X
oldestMsgAge		X
openInputCount	X	X
openOutputCount	X	X
lastGetDate		X
lastGetTime		X
lastGetDateTime		X
lastPutDate		X
lastPutTime		X
lastPutDateTime		X
queueTimeIndicator		X
stagnantMsgCount				X
averageMsgAge				X
newestMsgAge				X
processName	X
clusterName	X
triggerData	X
triggerControl	X
triggerDepth	X
triggerMessagePriority	X
triggerType	X
defaultBind	X
usage	X

Further descriptions of the attributes used can be found at the following IBM documentation pages:

disableQueueResetAttributes

A "reset queue statistics" command is issued for each monitored queue if any of the following columns has been configured:

highQueueDepth
msgDeqCount
msgEnqCount
timeSinceReset

If this setting is enabled (set to true) then the MQ Queue plug-in will not issue a "reset queue statistics" command. Also if any of the above columns have been configured, they will not be displayed (since they will not be populated with data).

Mandatory: No

Default: false

disableQueueBrowseAttributes

In order to calculate message statistics such as the average message age, it is necessary to browse (the headers of) all messages in a queue. Queue browsing occurs if any of the following columns have been configured:

averageMsgAge
newestMsgAge
stagnantMsgCount

For queues with a large number of messages this can be a time consuming process. If this setting is enabled (set to true) then the MQ Queue plug-in will not browse queue messages. Also if any of the above columns have been configured, they will not be displayed (since they will not be populated with data).

Mandatory: No

Default: false

stagnantMessageAge

This setting sets a threshold (in seconds) over which a message will be considered stagnant, and will then be counted in the stagnantMsgCount column. For example if the value of this setting is 60, then any messages on the queue over a minute old will be considered stagnant.

A value of 0 disables this threshold, meaning no messages are considered stagnant (and so the stagnantMsgCount column will always show 0). Since 0 is the default value for this setting, the user should always configure this setting if they intend to use the stagnantMsgCount column.

Mandatory: No

Default: 0

Debug

The debug section contains settings used for debugging purposes. You may be asked to configure these by ITRS support if you are having issues with the MQ Queue plug-in, in order to diagnose the problem.

debug

The debug section contains debugging settings for the MQ Queue plug-in. Typically users should not need to configure any setting in this section.

Mandatory: No

debug > sampleTimeout

The sample timeout controls the number of seconds MQ Queue will spend during a sample, before it gives up and displays any data obtained so far. This setting typically only comes into effect if the MQ server is slow to respond to queries, which may happen during times of heavy load.

Mandatory: No

Default: Half the Netprobe heartbeat interval (approx 35 seconds), with a minimum of 20s.

debug > disableBrowseTimeAdjust

When using columns that obtain data by browsing the messages on a queue, the message headers give the time the message was placed on the queue in server time. Calculations for the newest or average message age then compare this time against the local time of the Netprobe machine, introducing a possible source of error if the times are not synced between machines.

To compensate for this, the MQ Queue plug-in also requests the oldest message age using the queue statistics command. This value is then compared with the oldest message age computed using the local time, and any offset is used to adjust the resulting values accordingly.

This adjustment can be disabled by configuring this setting to true.

Mandatory: No

Default: false

debug > browseWarnThreshold

The browse warn threshold is used to warn when a message browsing operation may take a long time due to a large number of messages on the queue. Enable this setting to adjust how many messages must be present on a queue before a warning is produced. A value of 0 disables all warnings.

Mandatory: No

Default: 10,000

This section describes the mechanism to prevent MQ connections after a certain number of consecutive failures.

By default, MQ plugins will continually attempt connections to the MQ server, even after connections have continually failed. To prevent this, connectionFailureThreshold and connectionFailureStopTime can be configured to manage MQ connection backoff. These configurations are located under the Probe configuration's advanced tab (refer to Gateway2 Reference Guide). Setting these configurations will put into place a mechanism which will prevent MQ connections from being attempted after a given number of failures.

The connectionFailureThreshold is the maximum number of failures that can be allowed before initiating an MQ connection backoff. Once a backoff has been started, the netprobe will try to reconnect only after the indicated connectionFailureStopTime in seconds has been reached. If reconnection is successful, MQ connections will work normally. If reconnection is unsuccessful, backoff for the indicated time in connectionFailureStopTime will be once again put into effect.

These configurations are shared across several MQ plugins using the same connection in the same probe. This means that the count for consecutive failures will be counted against each unique MQ connection and not in each sampler.

XML Examples

Minimal setup

This example shows the bare minimum configuration for MQ Queue. This example assumes that the connection details are configured as environment variables. As no queues and columns have been specified, all queues on the queue manager will be monitored for their status statistics.

<sampler name="minimal">
        <plugin>
                <mq-queue>
                        <queueManager><data>QMname</data></queueManager>
                </mq-queue>
        </plugin>
</sampler>

SSL connection using MQSERVER setting

The following example shows how to configure MQ Queue to connect using an SSL connection in conjunction with the MQSERVER variable. This is done by specifying an SSL cipher for encryption, and the location of the SSL certificate key repository.

The example also shows a configuration for queues, selecting all queues starting TRD.PRD1 and the specific queue SYSTEM.DEAD.LETTER.QUEUE.

<sampler name="sslMQSERVER">
        <plugin>
                <mq-queue>
                        <connection>
                                <mqServer><data>CHAN/TCP/host(port)</data></mqServer>
                                <ssl>
                                        <keyRepository><data>/var/mqm/ssl/key</data></keyRepository>
                                        <cipherSpec><data>TRIPLE_DES_SHA_US</data></cipherSpec>
                                </ssl>
                        </connection>
                        <queueManager><data>QMname</data></queueManager>
                        <queues>
                                <queue>
                                        <matches><data>SYSTEM.DEAD.LETTER.QUEUE</data></matches>
                                </queue>
                                <queue>
                                        <startsWith><data>TRD.PRD1</data></startsWith>
                                </queue>
                        </queues>
                </mq-queue>
        </plugin>
</sampler>

Channel table file connection

This example shows how to configure MQ Queue to use a channel table to connect to the MQ server. If the channel table configuration requires SSL, then you will also need to configure the location of the key repository. Additionally, any security exit configuration in the sampler will be ignored.

This example also shows the use of the columns setting to select a set of columns for display.

<sampler name="channelTableFile">
        <plugin>
                <mq-queue>
                        <connection>
                                <mqChannelTable><data>/var/mqm/AMQCLCHL.TAB</data></mqChannelTable>
                        </connection>
                        <queueManager><data>QMname</data></queueManager>
                        <columns>
                                <column>queueType</column>
                                <column>queueDefinition</column>
                                <column>maxQueueDepth</column>
                                <column>currentQueueDepth</column>
                                <column>percentQueueDepth</column>
                                <column>openInputCount</column>
                                <column>openOutputCount</column>
                                <column>remoteQueueManager</column>
                                <column>remoteQueue</column>
                                <column>transmitQueueName</column>
                        </columns>
                </mq-queue>
        </plugin>
</sampler>