Tuning Java Programs for Optimum MQ Get API Calls Performance

You should really read this blog posting first.

The results of some of the test scenarios below, simply do not make any sense. I would like to think there is some sort of logic behind it but it is probably lack of testing by IBM that introduced some bugs in the code or should I say, ‘unexpected results’.

I’m going to repeat some of the information from the Tuning JMS blog posting, so that it will makes sense to people without having to jump back and forth between blog postings.

For each test run, I used the same 100 non-persistent MQRF2 (aka JMS) messages:

  • The first 25 messages are random sized messages between 735 bytes and 200KB
  • The next 25 messages are 8439 bytes in size
  • The next 24 messages are 735 bytes in size
  • The next 15 messages are 8439 bytes in size
  • The next 11 messages are 12298 bytes in size
  • A message consumer will not always receive the same sized message, hence, the first 25 messages are of a random size. The reason the latter 75 messages are in groups of same sized messages is because of something Steve Webb said in his post (IBM Developer question/answer) called MAXMSGL questions in regards to MQ JMS application. Near the bottom he said:

    If a number of smaller messages are then consumed, for example 10 messages at 5KB following on from the large message, the MQ-JMS classes will downsize the buffer to 5KB, and so on.

    I was curious when/what auto-resizing was going on. Hence, that is why I structured the last 75 messages the way I did.

    Now to the layout of MQ Auditor audit files. An MQ Auditor audit file is a simple CSV (Comma Separated Values) file, so you can load it into your spreadsheet program for easy viewing. Each API call is on a separate line in the audit file.
    i.e.
    {Date}{Time}{API Call}{PID}{TID}{CC}{RC}{UserId}{HConn}{MQ Structures}

    where

  • PID is Process Id
  • TID is Thread Id
  • CC is completion code
  • RC is reason code
  • HConn is the connection handle
  • Note: For MQGet API calls, after the MQ Structures are 2 important fields: BufferLength and DataLength. After the BufferLength and DataLength fields, there will be the internal headers (i.e. MQRFH2, MQIMS, etc.), if present, and the message data.

  • The BufferLength field is the length of the buffer used on the MQGet API call.
  • The DataLength field is the length of the message data returned in the buffer that was used on the MQGet API call.
  • Here is a sample MQGet API line from an MQ Auditor audit file with the BufferLength and DataLength fields highlighted.

    2019/08/02 16:06:54.320703, MQXF_GET , A, PID=12456, TID=1, CC=0, RC=0, UserId=tester, HConn=20971526, HObj=2, GMO_Options=MQGMO_NO_WAIT+MQGMO_BROWSE_NEXT+MQGMO_ACCEPT_TRUNCATED_MSG, GMO_WaitInterval=0, GMO_ResolvedQName=TEST.Q1, GMO_MatchOptions=MQMO_NONE, MD_PutDate=2019/08/03, MD_PutTime=00:19:08.00, MD_MsgId=414D51204D5141312020202020202020134CCD4020000B01, MD_Format=MQHRF2, MD_MsgType=MQMT_DATAGRAM, MD_Persistence=MQPER_NOT_PERSISTENT, BufferLength=256000, DataLength=725, RFH2_CCSID=437, RFH2_Encoding=273, RFH2_Format=MQSTR, RFH2_Folder1=jms_text , RFH2_Folder2=THis is a value for j1this is another value, RFH2_Folder3=This is a value for USR property U1 , MsgDataAsHex=496E204D515365726965732C20696E746572636F6D6D756E69,

    To set the baseline, I am going to be using the MQ sample program called amqsbcg.exe. I updated the BUFFERLENGTH define (line # 70) to be 256000 (from 65536) then compiled and linked the program. The reason I updated the define is because I don’t want to see reason code of 2079 (MQRC_TRUNCATED_MSG_ACCEPTED) for messages larger than 65536. It may confuse some people.

    Test #1:

  • Load the 100 MQRFH2 messages into a queue
  • Run amqsbcg in bindings mode against the same queue
  • Here is the MQ Auditor audit file. You can see that there are exactly 100 successful MQGets and 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE). This is exactly what is to be expected. If you scroll to the right of any MQGET line, you will see that in every case the size of the buffer given to MQ (BufferLength field) is 256000 bytes.

    For the MQ for Java program, I will be using one of my samples called MQTest12L.java. You can download the source code from here. The structure of the Java program is very similar to amqsbcg. It loops getting all messages until the queue is empty (it does not wait for more messages).

    I’m willing to bet that no one at IBM could correctly answer the following 2 questions for all 6 ‘MQ for Java’ test scenarios below!!
    (1) How many MQGets will be performed?
    (2) What will be the MQGET buffer length?

    I’m sure everyone reading this will say ‘that’s easy’ but I guarantee that after you read the 6 test scenarios, you will say WTF!!!

    Note: I’ll summarize the answers to the 2 questions above for the 8 ‘MQ for Java’ test scenarios at the end of the blog, so that you can re-iterate WTF!

    Test #2:

  • Load the 100 MQRFH2 messages into a queue
  • Run MQTest12L in bindings mode against the same queue
  • java MQTest12L -m MQWT2 -q TEST.Q1

    MQ Get Message Options set to the following:

    MQGetMessageOptions gmo = new MQGetMessageOptions();
    gmo.options = CMQC.MQGMO_NO_WAIT + CMQC.MQGMO_FAIL_IF_QUIESCING;

    The loop for getting messages is (the abridged version):

    while(getMore)
    {
       receiveMsg = new MQMessage();
       try
       {
          queue.get(receiveMsg, gmo);
       }
       catch (MQException e)
       {
          if ( (e.completionCode == CMQC.MQCC_FAILED) &&
               (e.reasonCode == CMQC.MQRC_NO_MSG_AVAILABLE) )
          {
             getMore = false;
          }
       }
    }

    Here is the MQ Auditor audit file. You can see that there are a total of 111 MQGets:

  • 100 successful MQGets
  • 10 unsuccessful MQGet with RC of 2080 (MQRC_TRUNCATED_MSG_FAILED)
  • 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE)
  • This means that MQTest12L performed 10% more MQGets API calls than amqsbcg to accomplish the same thing. So, lets analyze why there were 10 unsuccessful MQGet with RC of 2080.

  • For the 1st MQGet API call, the internal JMQI routine allocated a 4096 byte buffer and the size of the message was 725 bytes
  • For the 2nd MQGet API call, it failed with RC of 2080 because the internal JMQI routine was using a 4096 byte buffer but the message length was 4602 bytes. So, the internal JMQI routine allocated a new 4602 byte buffer and retrieve the message on the 3rd MQGet API call.
  • For the 4th, 10th, 12th, 14th, 21st and 31st MQGet API calls, they all failed with RC of 2080. For each occurrence, the internal JMQI routine allocates a new buffer of the size of the message that it failed to get.
  • On the 44th MQGet API call, for some unknown and very strange reason, the internal JMQI routine allocated a new buffer 4096 byte buffer. If you look closely, you will see that 44th MQGet is the 11th MQGet call with a smaller message size (8439 bytes) than what the internal JMQI routine had allocated for a buffer length (202601 bytes). Hence, the 44th MQGet API call failed with RC of 2080, because the internal JMQI routine was using a 4096 byte buffer but the message length was 8439 bytes. So, the internal JMQI routine allocated a new 8439 byte buffer and retrieve the message on the 44th MQGet API call.
  • On the 70th MQGet API call, for some unknown and very strange reason, the internal JMQI routine allocated a new buffer 4096 byte buffer. If you look closely, you will see that 70th MQGet is the 38th MQGet call with a smaller message size (725 bytes) than what the internal JMQI routine had allocated for a buffer length (8439 bytes). This is such strange behaviour.
  • For the 83rd MQGet API call, failed with RC of 2080 because the internal JMQI routine was using a 4096 byte buffer but the message length was 8439 bytes. So, the internal JMQI routine allocated a new 8439 byte buffer and retrieve the message on the 84th MQGet API call.
  • For the 99th MQGet API call, failed with RC of 2080 because the internal JMQI routine was using an 8439 byte buffer but the message length was 12286 bytes. So, the internal JMQI routine allocated a new 12286 byte buffer and retrieve the message on the 100th MQGet API call.
  • I have no idea what the internal JMQI routine is doing but it is very, very strange.

    Now, lets do the very same test again with MQTest12L program but add the com.ibm.mq.jmqi.defaultMaxMsgSize JVM environment variable.

    Test #3:

  • Load the 100 MQRFH2 messages into a queue
  • Run MQTest12L in bindings mode with com.ibm.mq.jmqi.defaultMaxMsgSize set to 260000 against the same queue
  • java -Dcom.ibm.mq.jmqi.defaultMaxMsgSize=260000 MQTest12L -m MQWT2 -q TEST.Q1

    Here is the MQ Auditor audit file. You can see that there are exactly 100 successful MQGets and 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE). If you scroll to the right of any MQGET line, you will see that in every case the size of the buffer given to MQ (BufferLength field) is 260000 bytes.

    Test #4:

  • Load the 100 MQRFH2 messages into a queue
  • Set a message size of 254,000 on the MQQueue’s get method
  • Run MQTest12L in bindings mode against the same queue
  • java MQTest12L -m MQWT2 -q TEST.Q1

    MQQueue’s get method:

    queue.get(receiveMsg, gmo, 254000);

    The results are EXACTLY the same as test scenario #2. It makes no sense. The code is explicitly setting the value for the get method. Here is the MQ Auditor audit file. You can see that there are a total of 111 MQGets:

  • 100 successful MQGets
  • 10 unsuccessful MQGet with RC of 2080 (MQRC_TRUNCATED_MSG_FAILED)
  • 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE)
  • Test scenario #5 will be exactly the same as test scenario # 4 but add the com.ibm.mq.jmqi.defaultMaxMsgSize JVM environment variable.

    Test #5:

  • Load the 100 MQRFH2 messages into a queue
  • Set a message size of 254,000 on the MQQueue’s get method
  • Run MQTest12L in bindings mode with com.ibm.mq.jmqi.defaultMaxMsgSize set to 260000 against the same queue
  • java -Dcom.ibm.mq.jmqi.defaultMaxMsgSize=260000 MQTest12L -m MQWT2 -q TEST.Q1

    Here is the MQ Auditor audit file. You can see that there are exactly 100 successful MQGets and 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE). If you scroll to the right of any MQGET line, you will see that in every case the size of the buffer given to MQ (BufferLength field) is 254000 bytes.

    This time the internal JMQI routine used the value set on the get call. Why? Why did it use it here and not for test scenario # 4?

    Now to “bake your noodle” (a Matrix reference), lets add MQGMO_ACCEPT_TRUNCATED_MSG to the MQ Get Message Options. Seriously, how can adding 1 option change the number of MQGETs and the buffer length ‘if’ the buffer length is larger than the message size!

    Test #6:

  • Load the 100 MQRFH2 messages into a queue
  • Set a message size of 254,000 on the MQQueue’s get method
  • Add MQGMO_ACCEPT_TRUNCATED_MSG to the MQGMO Options
  • Run MQTest12L in bindings mode against the same queue
  • java MQTest12L -m MQWT2 -q TEST.Q1

    MQ Get Message Options set to the following:

    MQGetMessageOptions gmo = new MQGetMessageOptions();
    gmo.options = CMQC.MQGMO_NO_WAIT + CMQC.MQGMO_FAIL_IF_QUIESCING + CMQC.MQGMO_ACCEPT_TRUNCATED_MSG;

    MQQueue’s get method:

    queue.get(receiveMsg, gmo, 254000);

    For some unexplained reason the buffer length was set to 254000 for all MQGets which makes absolutely no sense. Why was the internal JMQI routine now honoring the setting of the MQGet size to 254000 but in test scenario #4 it was not?!?! The ONLY difference between test scenario #4 and #6 was that MQGMO option of MQGMO_ACCEPT_TRUNCATED_MSG was added.

    Here is the MQ Auditor audit file. You can see that there are a total of 111 MQGets:

  • 100 successful MQGets
  • 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE)
  • Test scenario #7 will be exactly the same as test scenario # 6 but add the com.ibm.mq.jmqi.defaultMaxMsgSize JVM environment variable.

    Test #7:

  • Load the 100 MQRFH2 messages into a queue
  • Set a message size of 254,000 on the MQQueue’s get method
  • Add MQGMO_ACCEPT_TRUNCATED_MSG to the MQGMO Options
  • Run MQTest12L in bindings mode with com.ibm.mq.jmqi.defaultMaxMsgSize set to 260000 against the same queue
  • java -Dcom.ibm.mq.jmqi.defaultMaxMsgSize=260000 MQTest12L -m MQWT2 -q TEST.Q1

    Here is the MQ Auditor audit file. You can see that there are exactly 100 successful MQGets and 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE). If you scroll to the right of any MQGET line, you will see that in every case the size of the buffer given to MQ (BufferLength field) is 254000 bytes.

    This time the internal JMQI routine used the value set on the get call. Why? Why did it use it here and not for test scenario # 4? It is some truly weird stuff going on in the internal JMQI routine because it is not logical.

    Now time to turn UP the heat in the oven and REALLY bake that noodle!!! I’m going to redo Test scenarios #3 & #5 but lower the value of com.ibm.mq.jmqi.defaultMaxMsgSize to 55000.

    Test #8:

  • Load the 100 MQRFH2 messages into a queue
  • Run MQTest12L in bindings mode with com.ibm.mq.jmqi.defaultMaxMsgSize set to 55000 against the same queue
  • java -Dcom.ibm.mq.jmqi.defaultMaxMsgSize=55000 MQTest12L -m MQWT2 -q TEST.Q1

    Here is the MQ Auditor audit file. You can see that there are a total of 104 MQGets:

  • 100 successful MQGets
  • 3 unsuccessful MQGet with RC of 2080 (MQRC_TRUNCATED_MSG_FAILED)
  • 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE)
  • So, lets analyze why there were 3 unsuccessful MQGet with RC of 2080.

  • For the 1st MQGet API call, the internal JMQI routine allocated a 55000 byte buffer and the size of the message was 725 bytes
  • For the 10th MQGet API call, it failed with RC of 2080 because the internal JMQI routine was using a 55000 byte buffer but the message length was 109471 bytes. So, the internal JMQI routine allocated a new 109471 byte buffer and retrieve the message on the 11th MQGet API call.
  • For the 17th MQGet API call, it failed with RC of 2080 because the internal JMQI routine was using a 109471 byte buffer but the message length was 163730 bytes. So, the internal JMQI routine allocated a new 163730 byte buffer and retrieve the message on the 18th MQGet API call.
  • For the 27th MQGet API call, it failed with RC of 2080 because the internal JMQI routine was using a 163730 byte buffer but the message length was 202601 bytes. So, the internal JMQI routine allocated a new 202601 byte buffer and retrieve the message on the 18th MQGet API call.
  • Now, lets do the very same test again as Test scenario #8 but explicitly set the MQQueue’s get buffer length to be 110000.

    Test #9:

  • Load the 100 MQRFH2 messages into a queue
  • Set a message size of 110000 on the MQQueue’s get method
  • Run MQTest12L in bindings mode with com.ibm.mq.jmqi.defaultMaxMsgSize set to 55000 against the same queue
  • java -Dcom.ibm.mq.jmqi.defaultMaxMsgSize=55000 MQTest12L -m MQWT2 -q TEST.Q1

    Here is the MQ Auditor audit file. You can see that there are a total of 104 MQGets:

  • 15 successful MQGets
  • 2 unsuccessful MQGet with RC of 2080 (MQRC_TRUNCATED_MSG_FAILED)
  • So, lets analyze why there were 2 unsuccessful MQGet with RC of 2080.

  • For the 1st MQGet API call, the internal JMQI routine allocated a 55000 byte buffer and the size of the message was 725 bytes
  • For the 10th MQGet API call, it failed with RC of 2080 because the internal JMQI routine was using a 55000 byte buffer but the message length was 109471 bytes. So, the internal JMQI routine allocated a new 109471 byte buffer and retrieve the message on the 11th MQGet API call.
  • For the 17th MQGet API call, it failed with RC of 2080 because the internal JMQI routine was using a 109471 byte buffer but the message length was 163730 bytes. The internal JMQI routine does not allocate a new buffer because 163730 is above the specified MQQueue’s get specified buffer length of 110000.

    Here’s the summary of the test scenarios that I promised:

    Test Scenario com.ibm.mq.jmqi. defaultMaxMsgSize Application Buffer Length Specified on Get Used MQGMO-ACCEPT-TRUNCATED-MSG Number of MQGets Actual MQGet Buffer Length used
    #1 amqsbcg 256000 No 101 256000
    #2 MQTest12L No 111 Initially 4096, adjusts up & down
    #3 MQTest12L 260000 No 101 260000
    #4 MQTest12L 254000 No 111 Initially 4096, adjusts up & down
    #5 MQTest12L 260000 254000 No 101 254000
    #6 MQTest12L 254000 Yes 101 254000
    #7 MQTest12L 260000 254000 Yes 101 254000
    #8 MQTest12L 55000 No 104 Initially 55000, adjusts up & down – never lower than 55000
    #9 MQTest12L 55000 110000 No 15 Initially 55000, adjusts up & down – never lower than 55000 and never above 110000

    There are so many things that are illogical.
    (1) Test scenario #4 should have been using a Buffer Length of 254000 that was specified on the get.
    (2) Why does adding the MQGMO option of MQGMO_ACCEPT_TRUNCATED_MSG change how the internal JMQI routine handle the gets? Test scenario #4 vs #6

    Final summary (maybe):

  • First thing you need to do is add the com.ibm.mq.jmqi.defaultMaxMsgSize JVM environment variable to the run-time setup of your MQ Java program because it will keep the internal JMQI routine doing the right thing.
  • Second, com.ibm.mq.jmqi.defaultMaxMsgSize JVM environment variable is ONLY used IF the application does not explicit set a buffer length on the MQQueue’s get method OR if com.ibm.mq.jmqi.defaultMaxMsgSize JVM environment variable is lower than the MQQueue’s get buffer length.
  • Third, pick a reasonable value for the JVM environment variable.
  • Fourth, the internal JMQI routine uses the JVM environment variable for BOTH JMS and non-JMS Java applications even though it is not documented anywhere!!
  • Fifth, maybe people should be implementing some sort of auditing/tracking system like MQ Auditor to understand what is going on under the covers of MQ because I most certainly have learned a few things over the last several days!!!
  • Update: Currently, the JVM environment variable ‘com.ibm.mq.jmqi.defaultMaxMsgSize’ is undocumented. Hence, the usage may change in a future release.

    Regards,
    Roger Lacroix
    Capitalware Inc.

    IBM i (OS/400), IBM MQ, Java, Linux, macOS (Mac OS X), Programming, Unix, Windows, z/OS Comments Off on Tuning Java Programs for Optimum MQ Get API Calls Performance

    Tuning JMS Programs for Optimum MQ Get API Calls Performance

    On Wednesday, I decided to do some testing of some simple (1) MQ for JMS, (2) MQ for Java and (3) procedure language (i.e. amqsbcg) programs to show that using the JVM environment variable of com.ibm.mq.jmqi.defaultMaxMsgSize with an appropriate size could make a JMS application 10 to 15% faster because there would not be any double MQ Get API calls.

    My original goal was to use MQ Auditor show everyone what goes on behind the scene starting with an MQ for JMS program then MQ for Java program and amqsbcg because these 2 do not have an issue with double MQ Get API calls. And then a run of the MQ for JMS program setting the JVM environment variable of com.ibm.mq.jmqi.defaultMaxMsgSize to show that the double gets no longer happen.

    What should have taken me an hour to do turned into a shit-show with me saying over and over again WTF every time I ran one of my Java programs. I’ve now spend 2 full days on this and I’m pretty sure I have it straight in my head, the question will be, can I actually explain it without confusing the crap out of everyone!!!

    So, I’m not going to be showing you the MQ Auditor audit files from a MQ for Java program in this blog posting. I will do a complete write of it in the next blog posting because it is the only way I can think of not to confuse everyone. Hence, this blog posting will be about MQ for JMS program and a procedure language (i.e. amqsbcg).

    For each test run, I used the same 100 non-persistent MQRF2 (aka JMS) messages:

    • The first 25 messages are random sized messages between 735 bytes and 200KB
    • The next 25 messages are 8439 bytes in size
    • The next 24 messages are 735 bytes in size
    • The next 15 messages are 8439 bytes in size
    • The next 11 messages are 12298 bytes in size

    A message consumer will not always receive the same sized message, hence, the first 25 messages are of a random size. The reason the latter 75 messages are in groups of same sized messages is because of something Steve Webb said in his post (IBM Developer question/answer) called MAXMSGL questions in regards to MQ JMS application. Near the bottom he said:

    If a number of smaller messages are then consumed, for example 10 messages at 5KB following on from the large message, the MQ-JMS classes will downsize the buffer to 5KB, and so on.

    I was curious when/what auto-resizing was going on. Hence, that is why I structured the last 75 messages the way I did.

    Now to the layout of MQ Auditor audit files. An MQ Auditor audit file is a simple CSV (Comma Separated Values) file, so you can load it into your spreadsheet program for easy viewing. Each API call is on a separate line in the audit file.
    i.e.
    {Date}{Time}{API Call}{PID}{TID}{CC}{RC}{UserId}{HConn}{MQ Structures}

      where

    • PID is Process Id
    • TID is Thread Id
    • CC is completion code
    • RC is reason code
    • HConn is the connection handle

    Note: For MQGet API calls, after the MQ Structures are 2 important fields: BufferLength and DataLength. After the BufferLength and DataLength fields, there will be the internal headers (i.e. MQRFH2, MQIMS, etc.), if present, and the message data.

    • The BufferLength field is the length of the buffer used on the MQGet API call.
    • The DataLength field is the length of the message data returned in the buffer that was used on the MQGet API call.

    Here is a sample MQGet API line from an MQ Auditor audit file with the BufferLength and DataLength fields highlighted.

    2019/08/02 16:06:54.320703, MQXF_GET , A, PID=12456, TID=1, CC=0, RC=0, UserId=tester, HConn=20971526, HObj=2, GMO_Options=MQGMO_NO_WAIT+MQGMO_BROWSE_NEXT+MQGMO_ACCEPT_TRUNCATED_MSG, GMO_WaitInterval=0, GMO_ResolvedQName=TEST.Q1, GMO_MatchOptions=MQMO_NONE, MD_PutDate=2019/08/03, MD_PutTime=00:19:08.00, MD_MsgId=414D51204D5141312020202020202020134CCD4020000B01, MD_Format=MQHRF2, MD_MsgType=MQMT_DATAGRAM, MD_Persistence=MQPER_NOT_PERSISTENT, BufferLength=256000, DataLength=725, RFH2_CCSID=437, RFH2_Encoding=273, RFH2_Format=MQSTR, RFH2_Folder1=jms_text , RFH2_Folder2=THis is a value for j1this is another value, RFH2_Folder3=This is a value for USR property U1 , MsgDataAsHex=496E204D515365726965732C20696E746572636F6D6D756E69,

    To set the baseline, I am going to be using the MQ sample program called amqsbcg.exe. I updated the BUFFERLENGTH define (line # 70) to be 256000 (from 65536) then compiled and linked the program. The reason I updated the define is because I don’t want to see reason code of 2079 (MQRC_TRUNCATED_MSG_ACCEPTED) for messages larger than 65536. It may confuse some people.

    #define    BUFFERLENGTH  256000  /* Max length of message accepted */
      Test #1:

    • Load the 100 MQRFH2 messages into a queue
    • Run amqsbcg in bindings mode against the same queue

    Here is the MQ Auditor audit file. You can see that there are exactly 100 successful MQGets and 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE). This is exactly what is to be expected. If you scroll to the right of any MQGET line, you will see that in every case the size of the buffer given to MQ (BufferLength field) is 256000 bytes.

    For the MQ for JMS program, I will be using one of my samples called MQTestJMS12L.java. You can download the source code from here. The structure of the JMS program is very similar to amqsbcg. It loops getting all messages until the queue is empty (it does not wait for more messages).

      Test #2:

    • Load the 100 MQRFH2 messages into a queue
    • Run MQTestJMS12L in bindings mode against the same queue
    java MQTestJMS12L -m MQWT2 -q TEST.Q1

    The loop for getting messages is:

    receiver = session.createReceiver(myQ);
    while (more)
    {
       msg = receiver.receiveNoWait();
       if (msg != null)
          MQTestJMS12L.logger("Incoming Msg>>>"+msg);
       else
          more = false;
    }

    Here is the MQ Auditor audit file. You can see that there are a total of 111 MQGets:

    • 100 successful MQGets
    • 10 unsuccessful MQGet with RC of 2080 (MQRC_TRUNCATED_MSG_FAILED)
    • 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE)

    This means that MQTestJMS12L performed 10% more MQGets API calls than amqsbcg to accomplish the same thing. So, lets analyze why there were 10 unsuccessful MQGet with RC of 2080.

    • For the 1st MQGet API call, the internal JMQI routine allocated a 4096 byte buffer and the size of the message was 725 bytes
    • For the 2nd MQGet API call, it failed with RC of 2080 because the internal JMQI routine was using a 4096 byte buffer but the message length was 4602 bytes. So, the internal JMQI routine allocated a new 4602 byte buffer and retrieve the message on the 3rd MQGet API call.
    • For the 4th, 10th, 12th, 14th, 21st and 31st MQGet API calls, they all failed with RC of 2080. For each occurrence, the internal JMQI routine allocates a new buffer of the size of the message that it failed to get.
    • On the 44th MQGet API call, for some unknown and very strange reason, the internal JMQI routine allocated a new buffer 4096 byte buffer. If you look closely, you will see that 44th MQGet is the 11th MQGet call with a smaller message size (8439 bytes) than what the internal JMQI routine had allocated for a buffer size (202601 bytes). Hence, the 44th MQGet API call failed with RC of 2080, because the internal JMQI routine was using a 4096 byte buffer but the message length was 8439 bytes. So, the internal JMQI routine allocated a new 8439 byte buffer and retrieve the message on the 44th MQGet API call.
    • On the 70th MQGet API call, for some unknown and very strange reason, the internal JMQI routine allocated a new buffer 4096 byte buffer. If you look closely, you will see that 70th MQGet is the 38th MQGet call with a smaller message size (725 bytes) than what the internal JMQI routine had allocated for a buffer size (8439 bytes). This is such strange behaviour.
    • For the 83rd MQGet API call, failed with RC of 2080 because the internal JMQI routine was using a 4096 byte buffer but the message length was 8439 bytes. So, the internal JMQI routine allocated a new 8439 byte buffer and retrieve the message on the 84th MQGet API call.
    • For the 99th MQGet API call, failed with RC of 2080 because the internal JMQI routine was using an 8439 byte buffer but the message length was 12286 bytes. So, the internal JMQI routine allocated a new 12286 byte buffer and retrieve the message on the 100th MQGet API call.

    I have no idea what the internal JMQI routine is doing but it is very, very strange.

    Now, lets do the very same test again with MQTestJMS12L program but add the com.ibm.mq.jmqi.defaultMaxMsgSize JVM environment variable.

      Test #3:

    • Load the 100 MQRFH2 messages into a queue
    • Run MQTestJMS12L in bindings mode with com.ibm.mq.jmqi.defaultMaxMsgSize set to 250000 against the same queue
    java -Dcom.ibm.mq.jmqi.defaultMaxMsgSize=250000 MQTestJMS12L -m MQWT2 -q TEST.Q1

    Here is the MQ Auditor audit file. You can see that there are exactly 100 successful MQGets and 1 unsuccessful MQGet with RC of 2033 (MQRC_NO_MSG_AVAILABLE). This is exactly what is to be expected. If you scroll to the right of any MQGET line, you will see that in every case the size of the buffer given to MQ (BufferLength field) is 250000 bytes.

    There you go. If you don’t want the internal JMQI routine to cause excessive MQGet API calls, then you should use the com.ibm.mq.jmqi.defaultMaxMsgSize JVM environment variable and set it to an appropriate value to eliminate unnecessary MQGet API calls.

    Finally, my 100 test messages are just a simple sampling of messages and as you can see, the MQTestJMS12L program without using the com.ibm.mq.jmqi.defaultMaxMsgSize JVM environment variable caused an extra 10% calls for MQGet. I can easily see that number climbing to 15%-18% under a normal business load. Something to think about!!

    Update: Currently, the JVM environment variable ‘com.ibm.mq.jmqi.defaultMaxMsgSize’ is undocumented. Hence, the usage may change in a future release.

    Regards,
    Roger Lacroix
    Capitalware Inc.

    IBM i (OS/400), IBM MQ, Java, JMS, Linux, macOS (Mac OS X), Programming, Unix, Windows, z/OS Comments Off on Tuning JMS Programs for Optimum MQ Get API Calls Performance

    IBM MQ Fix Pack 9.1.0.3 Released

    IBM has just released Fix Pack 9.1.0.3 for IBM MQ V9.1 LTS:
    https://www.ibm.com/support/docview.wss?uid=ibm10961534

    Regards,
    Roger Lacroix
    Capitalware Inc.

    Fix Packs for MQ, IBM i (OS/400), Linux, Unix, Windows Comments Off on IBM MQ Fix Pack 9.1.0.3 Released

    MQ JMS Double Get API Calls

    Tuning applications is always a good thing. 🙂

    Colin Paice has reminded me of the MQ JMS default value for an MQGET buffer size is 4KB in his blog posting here. IBM’s default value of 4KB is far too low to be useful. Having such a low value causes excessive number of MQ JMS Get API calls being issued by MQ JMS applications. Most developers/programming have no clue that this happens in JMS when issuing an MQ Get API call.

    By MQ JMS Get API call, I’m talking about the QueueReceiver’s receive method.
    i.e.

    QueueReceiver qr = session.createReceiver(q);
    Message msg = qr.receive(10000);

    Normally, I would post my comments on Colin’s blog but he doesn’t approve my comments, so I figured I would write something here. Note: Colin is not the only one, IBM Developer blog author’s sometimes don’t approve my comments either. What can you do!

    The default maximum message length in IBM MQ is 4194304 (4MB). Unless your MQAdmin has changed the queue’s maximum message length then it will be 4MB.

    I have a lot of customers who send messages with XML data. One message could be 50KB whereas the next message could be 240KB (or larger). Trying to pick the correct maximum message length is like trying to catch a falling knife. It is just something you shouldn’t do. It is far, far better to simply use whatever the queue’s maximum message length is (which is typically 4194304).

    So, here is how you should invoke/start your MQ JMS application:
    i.e.

    java -Dcom.ibm.mq.jmqi.defaultMaxMsgSize=4194304 testpgm

    Now if you are worried about JVM running out of memory or transmiting an excessive amount data over the network that won’t happen. If you set a 4MB buffer for an MQGET, MQ will only transmit the actual size of the message (i.e. 20KB or 1.5MB). And the message object created in Java will only be the size of the actual message.

    Finally, for those users who have changed a queue’s maximum message length from 4MB to a larger value (maximum is 100MB), then I suggest you set your JVM parameter to be whatever your new maximum message length is, otherwise every JMS get call will actually result in 2 get calls. Because the first call will return a reason code of 2080 (MQRC_TRUNCATED_MSG_FAILED). Hence, the second get call will use a new larger buffer.

    Update: Currently, the JVM environment variable ‘com.ibm.mq.jmqi.defaultMaxMsgSize’ is undocumented. Hence, the usage may change in a future release.

    Regards,
    Roger Lacroix
    Capitalware Inc.

    IBM i (OS/400), IBM MQ, Java, JMS, Linux, macOS (Mac OS X), Programming, Unix, Windows, z/OS Comments Off on MQ JMS Double Get API Calls

    SQLite v3.29.0 Released

    D. Richard Hipp has just released SQLite v3.29.0.
    http://www.sqlite.org/news.html

    SQLite is a software library that implements a self-contained, serverless, zero-configuration, transactional SQL database engine. SQLite is the most widely deployed SQL database engine in the world. The source code for SQLite is in the public domain.

    Regards,
    Roger Lacroix
    Capitalware Inc.

    C, Database, IBM i (OS/400), Linux, macOS (Mac OS X), Open Source, Programming, Unix, Windows, z/OS Comments Off on SQLite v3.29.0 Released

    IBM MQ V9.1.3 Announced

    IBM has announced IBM MQ V9.1.3 for Multiplatforms:
    https://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/4/897/ENUS219-254/index.html
    Planned availability for IBM MQ V9.1.3 is July 11, 2019 for Electronic software delivery.

    IBM has announced IBM MQ V9.1.3 for z/OS:
    https://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/7/897/ENUS219-257/index.html
    Planned availability for IBM MQ V9.1.3 for z/OS is August 2, 2019 for Electronic software delivery.

    IBM has announced IBM MQ V9.1.3 for z/OS VUE:
    https://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/5/897/ENUS219-255/index.html
    Planned availability for IBM MQ V9.1.3 for z/OS VUE is August 2, 2019 for Electronic software delivery.

    IBM MQ (aka WebSphere MQ) homepage
    https://www.ibm.com/products/mq

    Regards,
    Roger Lacroix
    Capitalware Inc.

    Fix Packs for MQ, IBM MQ, IBM MQ Appliance, Linux, Unix, Windows, z/OS Comments Off on IBM MQ V9.1.3 Announced

    C# .NET MQ Code to Subscribe to a Topic

    The other day I answered a question on StackOverflow about subscribing to a topic using C# .NET code. I figured I should also post the code here for everyone to read & use.

    You can download the source code from here.

    using System;
    using System.Collections;
    using System.Collections.Generic;
    using System.Text;
    using System.Threading;
    using IBM.WMQ;
    
    /// <summary> Program Name
    /// MQTest82
    ///
    /// Description
    /// This C# class will connect to a remote queue manager
    /// and get messages from a topic using a managed .NET environment.
    ///
    /// Sample Command Line Parameters
    /// -h 127.0.0.1 -p 1414 -c TEST.CHL -m MQA1 -t abc/xyz -u tester -x mypwd
    /// </summary>
    /// <author>  Roger Lacroix
    /// </author>
    namespace MQTest82
    {
       public class MQTest82
       {
          private Hashtable inParms = null;
          private Hashtable qMgrProp = null;
          private System.String qManager;
          private System.String topicString;
    
          /*
          * The constructor
          */
          public MQTest82()
              : base()
          {
          }
    
          /// <summary> Make sure the required parameters are present.</summary>
          /// <returns> true/false
          /// </returns>
          private bool allParamsPresent()
          {
             bool b = inParms.ContainsKey("-h") && inParms.ContainsKey("-p") &&
                      inParms.ContainsKey("-c") && inParms.ContainsKey("-m") &&
                      inParms.ContainsKey("-t");
             if (b)
             {
                try
                {
                   System.Int32.Parse((System.String)inParms["-p"]);
                }
                catch (System.FormatException e)
                {
                   b = false;
                }
             }
    
             return b;
          }
    
          /// <summary> Extract the command-line parameters and initialize the MQ variables.</summary>
          /// <param name="args">
          /// </param>
          /// <throws>  IllegalArgumentException </throws>
          private void init(System.String[] args)
          {
             inParms = System.Collections.Hashtable.Synchronized(new System.Collections.Hashtable(14));
             if (args.Length > 0 && (args.Length % 2) == 0)
             {
                for (int i = 0; i < args.Length; i += 2)
                {
                   inParms[args[i]] = args[i + 1];
                }
             }
             else
             {
                throw new System.ArgumentException();
             }
    
             if (allParamsPresent())
             {
                qManager = ((System.String)inParms["-m"]);
                topicString = ((System.String)inParms["-t"]);
    
                qMgrProp = new Hashtable();
                qMgrProp.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_MANAGED);
    
                qMgrProp.Add(MQC.HOST_NAME_PROPERTY, ((System.String)inParms["-h"]));
                qMgrProp.Add(MQC.CHANNEL_PROPERTY, ((System.String)inParms["-c"]));
    
                try
                {
                   qMgrProp.Add(MQC.PORT_PROPERTY, System.Int32.Parse((System.String)inParms["-p"]));
                }
                catch (System.FormatException e)
                {
                   qMgrProp.Add(MQC.PORT_PROPERTY, 1414);
                }
    
                if (inParms.ContainsKey("-u"))
                   qMgrProp.Add(MQC.USER_ID_PROPERTY, ((System.String)inParms["-u"]));
    
                if (inParms.ContainsKey("-x"))
                   qMgrProp.Add(MQC.PASSWORD_PROPERTY, ((System.String)inParms["-x"]));
    
                logger("Parameters:");
                logger("  QMgrName ='" + qManager + "'");
                logger("  Topic String ='" + topicString + "'");
    
                logger("Connection values:");
                foreach (DictionaryEntry de in qMgrProp)
                {
                   logger("  " + de.Key + " = '" + de.Value + "'");
                }
             }
             else
             {
                throw new System.ArgumentException();
             }
          }
    
          /// <summary> Connect, open topic, get messages, close topic and disconnect. </summary>
          ///
          private void testReceive()
          {
             MQQueueManager qMgr = null;
             MQTopic inTopic = null;
             int openOptionsForGet = MQC.MQSO_CREATE | MQC.MQSO_FAIL_IF_QUIESCING | MQC.MQSO_MANAGED | MQC.MQSO_NON_DURABLE;
    
             try
             {
                qMgr = new MQQueueManager(qManager, qMgrProp);
                logger("successfully connected to " + qManager);
    
                inTopic = qMgr.AccessTopic(topicString, null, MQC.MQTOPIC_OPEN_AS_SUBSCRIPTION, openOptionsForGet);
                logger("successfully opened " + topicString);
    
                testLoop(inTopic);
             }
             catch (MQException mqex)
             {
                logger("CC=" + mqex.CompletionCode + " : RC=" + mqex.ReasonCode);
             }
             catch (System.IO.IOException ioex)
             {
                logger("ioex=" + ioex);
             }
             finally
             {
                try
                {
                   if (inTopic != null)
                      inTopic.Close();
                   logger("closed: " + topicString);
                }
                catch (MQException mqex)
                {
                   logger("CC=" + mqex.CompletionCode + " : RC=" + mqex.ReasonCode);
                }
    
                try
                {
                   if (qMgr != null)
                      qMgr.Disconnect();
                   logger("disconnected from " + qManager);
                }
                catch (MQException mqex)
                {
                   logger("CC=" + mqex.CompletionCode + " : RC=" + mqex.ReasonCode);
                }
             }
          }
    
          private void testLoop(MQTopic inTopic)
          {
             bool flag = true;
             MQGetMessageOptions gmo = new MQGetMessageOptions();
             gmo.Options |= MQC.MQGMO_NO_WAIT | MQC.MQGMO_FAIL_IF_QUIESCING;
             MQMessage msg = null;
    
             while (flag)
             {
                try
                {
                   msg = new MQMessage();
                   inTopic.Get(msg, gmo);
                   if (msg.Feedback == MQC.MQFB_QUIT)
                   {
                      flag = false;
                      logger("received quit message - exiting loop");
                   }
                   else
                      logger("Message Data: " + msg.ReadString(msg.MessageLength));
                }
                catch (MQException mqex)
                {
                   logger("CC=" + mqex.CompletionCode + " : RC=" + mqex.ReasonCode);
                   if (mqex.Reason == MQC.MQRC_NO_MSG_AVAILABLE)
                   {
                      // no meesage - life is good - loop again
                      logger("sleeping");
                      Thread.Sleep(60*1000);  // sleep for 60 seconds
                   }
                   else
                   {
                      flag = false;  // severe error - time to exit
                   }
                }
                catch (System.IO.IOException ioex)
                {
                   logger("ioex=" + ioex);
                }
             }
          }
    
          private void logger(String data)
          {
             DateTime myDateTime = DateTime.Now;
             System.Console.Out.WriteLine(myDateTime.ToString("yyyy/MM/dd HH:mm:ss.fff") + " " + this.GetType().Name + ": " + data);
          }
    
          /// <summary> main line</summary>
          /// <param name="args">
          /// </param>
          //        [STAThread]
          public static void Main(System.String[] args)
          {
             MQTest82 write = new MQTest82();
    
             try
             {
                write.init(args);
                write.testReceive();
             }
             catch (System.ArgumentException e)
             {
                System.Console.Out.WriteLine("Usage: MQTest82 -h host -p port -c channel -m QueueManagerName -t topicString [-u userID] [-x passwd]");
                System.Environment.Exit(1);
             }
             catch (MQException e)
             {
                System.Console.Out.WriteLine(e);
                System.Environment.Exit(1);
             }
    
             System.Environment.Exit(0);
          }
       }
    }

    Regards,
    Roger Lacroix
    Capitalware Inc.

    .NET, C#, IBM MQ, Linux, Open Source, Programming, Windows Comments Off on C# .NET MQ Code to Subscribe to a Topic

    Capitalware Products Minimum MQ Level Changed

    The minimum IBM MQ level for the following Capitalware products (distributed platforms) is now IBM MQ v7.1 (or higher):

  • MQ Auditor
  • MQ Authenticate User Security Exit
  • MQ Message Encryption
  • MQ Message Replication
  • MQ Standard Security Exit
  • Regards,
    Roger Lacroix
    Capitalware Inc.

    Capitalware, IBM i (OS/400), IBM MQ, Linux, MQ Auditor, MQ Authenticate User Security Exit, MQ Enterprise Security Suite, MQ Message Encryption, MQ Message Replication, MQ Standard Security Exit, Unix, Windows Comments Off on Capitalware Products Minimum MQ Level Changed

    Capitalware Products 2019 Release Train

    Here is a summary of all the recent releases that Capitalware Inc. has published:

      Updated ‘License as Free’ products:

    • MQ Channel Auto Creation Manager v1.0.5
    • MQ Channel Auto Creation Manager for z/OS v1.0.5
    • MQ Set UserID v1.0.4
    • MQ Set UserID for z/OS v1.0.4
    • Client-side Security Exit for Depository Trust Clearing Corporation v1.0.4
    • Client-side Security Exit for Depository Trust Clearing Corporation for z/OS v1.0.4

    Regards,
    Roger Lacroix
    Capitalware Inc.

    Capitalware, IBM i (OS/400), IBM MQ, Linux, MQ Auditor, MQ Authenticate User Security Exit, MQ Channel Connection Inspector, MQ Channel Encryption, MQ Channel Throttler, MQ Enterprise Security Suite, MQ Message Encryption, MQ Message Replication, MQ Standard Security Exit, Unix, Windows, z/OS Comments Off on Capitalware Products 2019 Release Train

    New: MQ Standard Security Exit v2.5.0

    Capitalware Inc. would like to announce the official release of MQ Standard Security Exit v2.5.0. This is a FREE upgrade for ALL licensed users of MQ Standard Security Exit. MQ Standard Security Exit is a solution that allows a MQAdmin to control and restrict who is accessing an IBM MQ resource.

    For more information about MQ Standard Security Exit go to:
    https://www.capitalware.com/mqssx_overview.html

      Changes for MQ Standard Security Exit v2.5.0:

    • Tuned the code that is called on entry
    • Tuned the logging code

    Regards,
    Roger Lacroix
    Capitalware Inc.

    Capitalware, IBM i (OS/400), IBM MQ, Linux, MQ Standard Security Exit, Unix, Windows Comments Off on New: MQ Standard Security Exit v2.5.0