objectName.close() method is a single line of code, if it is missed our job will close.
We are using a java class to open an MQ connection, get, put, SWIFT messages onto the MQ. This class was thoroughly tested and debugged. We have promoted this class for User acceptance testing (UAT), it performed well and got the sign off from the Users for Implementation.
The very first business day after the implementation this class was working fine, suddenly when a spike occurs, we have nearly four thousand messages on the Queue, we are getting MQ Exception (Exception number 2017), the description of the exception is the number of open handles exceeded its limit. We restarted the service after reading 256 messages again we are getting the number of handles exceeded error.
We debugged the code, wherever we are opening the queue we are closing it, but unable to find where the exception is from, this problem never occurred during UAT when we did Volume testing with more that ten thousand messages.
Finally we find out that we have missed localqueue.close() method on an escape route. The escape route i meant is the exception handle block (catch block of the try-catch), we are reading the messages from the Queue when some exception condition occurs, but forget to close the local queue after reading from the queue.
Since this exception never occurred during our UAT there was a potential bug unnoticed. But during production when the exception condition occurs the openned local queue handles are not closed, we missed out localqueue.close().
It took nearly ten days to simulate this condition on development environment and fix the error. This has a major impact on business as well.
We are using a java class to open an MQ connection, get, put, SWIFT messages onto the MQ. This class was thoroughly tested and debugged. We have promoted this class for User acceptance testing (UAT), it performed well and got the sign off from the Users for Implementation.
The very first business day after the implementation this class was working fine, suddenly when a spike occurs, we have nearly four thousand messages on the Queue, we are getting MQ Exception (Exception number 2017), the description of the exception is the number of open handles exceeded its limit. We restarted the service after reading 256 messages again we are getting the number of handles exceeded error.
We debugged the code, wherever we are opening the queue we are closing it, but unable to find where the exception is from, this problem never occurred during UAT when we did Volume testing with more that ten thousand messages.
Finally we find out that we have missed localqueue.close() method on an escape route. The escape route i meant is the exception handle block (catch block of the try-catch), we are reading the messages from the Queue when some exception condition occurs, but forget to close the local queue after reading from the queue.
Since this exception never occurred during our UAT there was a potential bug unnoticed. But during production when the exception condition occurs the openned local queue handles are not closed, we missed out localqueue.close().
It took nearly ten days to simulate this condition on development environment and fix the error. This has a major impact on business as well.
Lessons learned:
We are closing the connection, recordsets, handles after their use, but we should never forget to close them on the escape route.
We are closing the connection, recordsets, handles after their use, but we should never forget to close them on the escape route.
1 comment:
After reading the this i understand the impact of close.This is quite pragmatic and important for Techies.
The way you explain is the down to earth manner.
I will be regular reader of your blog.
Post a Comment