Autonomous Lost Message Notifier

November 2, 2017 Roger Lacroix

All too often, I and the world of MQAdmins hear architects, programmers, support personal, etc. say:

  • MQ has lost my message.
  • Where is my message?
  • What happened to the reply message for the request message?

If you need to track or audit messages and/or message flows then your company needs to invest in a commercial grade MQ solution. Many years ago, I put together a list of available MQ solutions for message tracking here. Of course, I would recommend everyone have a look at MQ Auditor (a Capitalware product).

Over the years, I have built many light-weight message tracking programs for various customers. These Autonomous Lost Message Notifier programs use features of IBM MQ called:

  • Confirmation of Arrival (COA)
  • Confirmation of Delivery (COD)

COA means that the message has reached its target queue. COD means that the message has been retrieved by a receiving application.

Types of COA & COD report options:

  • MQRO_COA & MQRO_COD report options means the ‘CO’ message is put to the ‘Reply To Queue’ and contains no message data just the MQMD.
  • MQRO_COA_WITH_DATA & MQRO_COD_WITH_DATA report options means the ‘CO’ message is put to the ‘Reply To Queue’ and contains the first 100 bytes of the message data along with the MQMD.
  • MQRO_COA_WITH_FULL_DATA & MQRO_COD_WITH_FULL_DATA report options means the ‘CO’ message is put to the ‘Reply To Queue’ and contains the entire message data along with the MQMD.

Note: I will only be using MQRO_COA & MQRO_COD report options as I don’t care about the message data being returned to the application.

Most architects and programmers skip over the section in the MQ Knowledge Center that talks about COA and COD report options of a message. These are the same people that when there is a production problem say “MQ has lost my message”, etc. and have no clue how to determine what happened, when it happened or what the REAL reason is for the production problem except to say “MQ has lost my message”.

Here is a picture of a standard request/reply message flow:

  • The ‘Requesting Client’ application puts a request message (blue) into the ‘REQUEST.Q’ queue to be processed by the ‘Backend Server’ application.
  • The ‘Backend Server’ application retrieves the request message (blue) from the ‘REQUEST.Q’ queue, performs some business logic then sends a reply message (green) to the queue and queue manager specified in the request message’s MQMD replyToQueueName and replyToQueueManagerName fields.
  • The ‘Requesting Client’ application gets the reply message (green) from its temporary dynamic queue and performs whatever business functions it needs to.

Think of the message flow between the 2 applications as 2 sides to the same coin. Generally speaking, the developers and/or support personal are responsible for 1 side on the coin. Hence, originally I was going to write this blog item from the perspective of the ‘Requesting Client’ application and then let people use the information/concept for their own ‘Backend Server’ application. It occurred to me that since I’m writing about 1 side of a coin then 50% of the people would be happy and 50% would not be happy. So, I decided to write about both sides of the coin including fully functioning sample programs for the ‘Requesting Client’ and ‘Backend Server’ applications.

So, lets begin: what will you need? IBM MQ of course, a database somewhere and some programming knowledge.

  • Find a database you can use – it can be your corporate one (i.e. Oracle, DB2, SQL Server, etc.) or a simple one like SQLite, Derby, H2, etc.
  • You can read about MQMD’s report field here

Items to Note # 1: For COA & COD message generation, the queue manager will copy the application message’s MQMD MessageId to the COA & COD message’s CorrelationId field. This is VERY important to understand and remember.

Items to Note # 2: If the messages stay within the same queue manager then life is easy. If not, then you need to know that the COD message is returned using the original message’s UserId. So you need to plan to have the correct security permission or the COD message will land in the DLQ (Dead Letter Queue).

Items to Note # 3: Please remember that IBM MQ Best Practices says to always allow the queue manager to generate a unique MessageId for each message. I’ve seen far too many samples where people write there own MessageId generator and at some point it breaks or never worked and the application is always using the same MessageId for every message it sends. Don’t do this – let the queue manager generate a unique MessageId for each message sent.

Now lets look again at the picture with the COA and COD messages enabled for both applications.

  • The request message and associated COA and COD messages are highlighted in blue. The COA & COD messages will be automatically generated by the queue manager and flow back to the RQClient01 application,
  • The reply message and associated COA and COD messages are highlighted in green. The COA & COD messages will be automatically generated by the queue manager and flow back to the BEServer01 application,
  • I have also drawn a separate database for each application rather than a corporate database since that is how my sample applications are coded.

If you compare the 2nd picture with the 1st picture, the message flow is exactly the same. We have just enabled COA and COD for the request and reply messages that are put to their respective queues.

By now, I’m sure you are asking where is the “Autonomous” part of the blog. I’m getting to it. Basically, by adding/setting the message’s MQMD report field to enable COA and COD messages, that is where the “Autonomous” part comes into play because we are getting MQ to track itself. Hence, “Autonomous” is the part of MQ being self aware of what has happened to each message.

For this blog item, I decided to use Java and SQLite. I used the “sqlite-jdbc” driver from Taro L. Saito. The JDBC driver works extremely well and he keeps the code in sync with the SQLite code base. Now I could have written it in C or C# with SQLite but since Java is compile once, run everywhere, I decided to go with Java.

You can download the sample code for this blog item from HERE. The download file includes:

  • RQClient01.class – Requesting Client compiled class
  • RQClient01.bat and RQClient01.sh – Windows & Unix/Linux script to set environment variables and run the program
  • BEServer01.class – Backend Server compiled class
  • BEServer01 .bat and BEServer01.sh – Windows & Unix/Linux script to set environment variables and run the program
  • PutQuit.class – PutQuit compiled class
  • PutQuit.bat and PutQuit.sh – Windows & Unix/Linux script to set environment variables and run the program
  • “sqlite-jdbc” driver – libs/sqlite-jdbc-3.20.0.jar – the driver that handles everything related to SQLite
  • Requesting Client source code – src/RQClient01.java
  • Backend Server source code – src/BEServer01.java
  • PutQuit source code – src/PutQuit.java

Note: You will need to have MQ installed or use the MQ Redistributable Client to use the samples.

For tools to view an SQLite database, you can download the Command Line Shell for SQLite from SQLite or you can download one of the many GUI tools for SQLite. For several years, I’ve used DB Browser for SQLite. It is fast, light-weight and easy to use.

 

Requesting Client (RQClient01.java)


Note: If your application has a transaction id or reference number that is used as an identifier for a business process/transaction then it should be included in the database, to help speed up problem resolution. Note: I am creating a simple ‘Requesting Client’ application so there is no transaction id or reference number to be included but normally this is something that you will have.

The 1st thing that ‘RQClient01.java’ program does is create a database table, if one does not already exist.

    Fields of the table:

  • MessageId is the request message’s MQMD messageId returned after the MQPUT
  • BeforePut_TS is the current time before the MQPUT
  • Put_TS is the request message’s MQMD putDateTime (returned after the MQPUT)
  • COA_TS is the COA message’s MQMD putDateTime
  • COD_TS is the COD message’s MQMD putDateTime
  • Reply_TS is the reply message’s MQMD putDateTime
  • Received_TS is the current time after the MQGET

Note: ‘RQClient01.java’ will do the following 10 times then exit: send a request message and get the reply, COA & COD messages.

Here’s the code from ‘RQClient01.java’ to enable COA & COD messages:

requestMsg.messageType = CMQC.MQMT_REQUEST;
requestMsg.replyToQueueManagerName = qMgrName;
requestMsg.replyToQueueName = replyQName;
requestMsg.report = CMQC.MQRO_COA | CMQC.MQRO_COD;

To retrieve the reply, COA & COD messages, the code has a for-loop to be performed up to 3 times. To determine which message is which, the code checks the incoming message’s MQMD ‘messageType’ and ‘feedback’ fields.

if ( (replyMsg.messageType == CMQC.MQMT_REPORT) &&
     (replyMsg.feedback == CMQC.MQFB_COA) )
{
   updateTable(dbConn, msgIdAsStr, "COA_TS", replyMsg.putDateTime.getTime());
   RQClient01.logger("Received COA");
}
else if ( (replyMsg.messageType == CMQC.MQMT_REPORT) &&
          (replyMsg.feedback == CMQC.MQFB_COD) )
{
   updateTable(dbConn, msgIdAsStr, "COD_TS", replyMsg.putDateTime.getTime());
   RQClient01.logger("Received COD");
}
else
{
   updateTable(dbConn, msgIdAsStr, "Received_TS", new Date());
   updateTable(dbConn, msgIdAsStr, "Reply_TS", replyMsg.putDateTime.getTime());

   byte[] b = new byte[replyMsg.getMessageLength()];
   replyMsg.readFully(b);
   RQClient01.logger("Received reply message: \"" + new String(b) + "\"");
}

 

Items to Note # 4: To simulate a potential lost message, ‘BEServer01.java’ has code to NOT send a reply message for the 4th request message received (every time the user starts BEServer01 program).

Running the sample programs:

Start the BEServer01 program first, then start the RQClient01 program and here is the screen-shot of the RQClient01 data in the table (via DB Browser for SQLite):

We can see that application RQClient01 never received the reply message. The ‘Reply_TS’ and ‘Received_TS’ fields are empty for message # 4. Note: MQ did NOT lose the message, BEServer01 application did not send it!!!

So, how can we make this an ‘Autonomous Lost Message Notifier’ from RQClient01 application point of view? Use a corporate database and request that the DBA add a scheduler item that fires hourly or every 30 minutes (or whatever you want) and if the following SQL generates a ResultSet then an alert, email, text message, etc. be generated because there is an issue with the message flow.

SELECT * FROM MQSLA
         WHERE COA_TS IS NULL OR
               COD_TS IS NULL OR
               Reply_TS IS NULL OR
               Received_TS IS NULL

 

Backend Server (BEServer01.java)


As mentioned earlier, if your application has a transaction id or reference number that is used as an identifier for a business process/transaction then it should be included in the database, to help speed up problem resolution. Note: I am creating a simple ‘Backend Server’ application so there is no transaction id or reference number to be included but normally this is something that you will have.

The 1st thing that ‘BEServer01.java’ program does is create a database table, if one does not already exist.

    Fields of the table:

  • MessageId is the request message’s MQMD messageId
  • Received_TS is the current time after the MQGET
  • Request_TS is the reply message’s MQMD putDateTime
  • Reply_TS is the reply message’s MQMD putDateTime returned after the MQPUT
  • ReplyMessageId is the reply message’s MQMD messageId returned after the MQPUT
  • COA_TS is the COA message’s MQMD putDateTime
  • COD_TS is the COD message’s MQMD putDateTime

‘BEServer01.java’ program is really 2 components in one: (1) the main loop and (2) a separate thread.

  • The main loop will wait up to 60 seconds for a message and if none arrives, the program will terminate. If a message arrives, it sleeps for 100-300 milliseconds, to simulate doing something then it sends a reply message (it does not send a reply message for the request 4th message – fake an issue).
  • A separate thread will continuously loop processing the COA & COD messages and updating the database

Note: Normally, a ‘Backend Server’ application does not have a need for its own reply queue but to receive COA & COD messages, the BEServer01 program opens a temporary dynamic queue for this purpose.

Code from main loop to get a request message:

MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.options = CMQC.MQGMO_FAIL_IF_QUIESCING + CMQC.MQGMO_WAIT;
gmo.waitInterval = 60 * 1000; // wait up to 60 seconds
requestMsg = new MQMessage();
requestMsg.messageId = CMQC.MQMI_NONE;
requestMsg.correlationId = CMQC.MQCI_NONE;

Note: You can shutdown the ‘BEServer01.java’ program early by putting a ‘Quit’ message on the ‘REQUEST.Q’ queue.

if (requestMsg.feedback == CMQC.MQFB_QUIT)
{
   more = false;
   BEServer01.logger("Quit message received.");
   break;
}

Code for the reply message with COA & COD report options:

String replyText = "This is reply message # "+mCount++;
MQMessage replyMsg = new MQMessage();
replyMsg.messageId = CMQC.MQMI_NONE;
replyMsg.correlationId = messageId;
replyMsg.messageType = CMQC.MQMT_REPLY;
replyMsg.replyToQueueManagerName = qMgrName;
replyMsg.replyToQueueName = coQName;
replyMsg.report = CMQC.MQRO_COA | CMQC.MQRO_COD;
replyMsg.writeString(replyText);
qMgr.put(replyQ, replyQMgr, replyMsg);

The thread code to handle the COA & COD messages:

if ( (coMsg.messageType == CMQC.MQMT_REPORT) &&
     (coMsg.feedback == CMQC.MQFB_COA) )
{
   BEServer01.logger("Received COA");

   /**
    * This thread could get ahead of the mothership in updating
    * the DB record.  So put in a small delay.
    */
   try
   {
      Thread.sleep(75);
   }
   catch(Exception ee)
   {}

   // Update record in table
   updateTableForReply(dbConn, replyMsgIdAsStr, "COA_TS", coMsg.putDateTime.getTime());
}
else if ( (coMsg.messageType == CMQC.MQMT_REPORT) &&
          (coMsg.feedback == CMQC.MQFB_COD) )
{
   BEServer01.logger("Received COD");

   // Update record in table
   updateTableForReply(dbConn, replyMsgIdAsStr, "COD_TS", coMsg.putDateTime.getTime());
}
else
{
   BEServer01.logger("Unknown message type: " + coMsg.messageType);
}

Note: There is potentially a timing issue in that the thread could try and update the database before the main loop updates the table with the reply message’s MessageId. Hence, that is why the thread sleeps for 75 milliseconds.

By now I’m sure you are wondering: “why does the database table for BEServer01 have a MessageId and a ReplyMessageId field?”. It is very simple but people get confused by what is happening. Here’s a high-level example:

  • If the request message’s MessageId had a value of ‘XYZ001’ then when the BEServer01 program issues the MQPUT, the reply message’s MessageId will have a unique value generated by the queue manager (i.e. ‘ABC001’) and the code sets the CorrelationId to ‘XYZ001’ (the request message’s MessageId). So, the reply message will reach the reply queue and RQClien01 program will be able to process it.

  • When the queue manager generates the COA & COD messages, it will copy the MessageId of the reply message (i.e. ‘ABC001′) to the COA & COD messages’ CorrelationId field. Hence, the reply message’s CorrelationId is is not used by the queue manager when generating the COA or COD messages. Therefore, that is why the code needs to update the table with the reply message’s MessageId so that it can cross-match the incoming (request) message’s MessageId value with the outgoing (reply) message’s MessageId. Hopefully, that makes sense.

Running the sample programs:

As previously mentioned, start the BEServer01 program first, then start the RQClient01 program and here is the screen-shot of the BEServer01 data in the table (via DB Browser for SQLite):

We can see that application BEServer01 never sent the 4th message!! ‘Reply_TS’ is blank.

So, how can we make this an ‘Autonomous Lost Message Notifier’ from BEServer01 application point of view? Use a corporate database and request that the DBA add a scheduler item that fires hourly or every 30 minutes (or whatever you want) and if the following SQL generates a ResultSet then an alert, email, text message, etc. be generated because there is an issue with the message flow.

SELECT * FROM MQSLA
         WHERE Reply_TS IS NULL OR
               ReplyMessageId IS NULL OR
               COA_TS IS NULL OR
               COD_TS IS NULL

 

Archiving


In this scenario, I have always recommended and implemented a ‘Today MQ SLA’ table and an ‘Archive MQ SLA’ table. The ‘Today MQ SLA’ table only contains records for the current day and every day a few minutes after midnight, all of the previous day’s records are moved from the ‘Today MQ SLA’ table to the ‘Archive MQ SLA’ table.

Some companies purge the ‘Archive MQ SLA’ table for records older than 30 days, others do it yearly and some have never purged the ‘Archive MQ SLA’ table. It is your choice on how long the data is useful.

 

Conclusion


I know this was a long blog item but hopefully you have seen how simple and useful the concept of having MQ track its own messages using COA & COD report options and how it can become an ‘Autonomous Lost Message Notifier’ with a little help from a database.

Regards,
Roger Lacroix
Capitalware Inc.

This entry was posted in Database, IBM i (OS/400), IBM MQ, Java, Linux, macOS (Mac OS X), Programming, Unix, Windows.

Comments are closed.