Page tree
Skip to end of metadata
Go to start of metadata

Problem

USAS or USPS app is hung or unresponsive to web browser.  Several ITC's have reported occasional hanging of the application.  This often manifests as being unable to reach the login page when first connecting to the app.  Restarting the app clears the problem.

At the time of this writing, the exact cause is unknown, but it is believed to be related to thread exhaustion of the container request pool.  

If you experience this problem, you can help the SSDT identify the root cause of the problem by performing the following steps.

Troubleshooting

  • From the docker project directory that contains the troubled container, do:

    /ssdt/scripts/console.sh usasapp (or uspsapp)
  • You should see a welcome banner like: ":: SSDT Application Console Boot :: (v) followed by a "#>" prompt. 

  • At the "#>" prompt type these three commands: 

    thread ls 
    thread ls | thread dump
    metrics
    exit
  • After exiting, you will be returned to the docker host.  There will be a console.log file in the current directory.  Send it to the SSDT with: 

    > /ssdt/scripts/send.sh console.log
  • Send an email or open a CSM ticket to the SSDT describing the nature of the hang and mention that you sent a console log.

Explanation

The USxS apps contain an instance of the CRaSH (Common Reusable SHell) and expose a Telnet port  (port 2000) on the running container.  This allows administrators to connect to the shell using a Telnet client and issue commands that interact directly with the JVM. 

The console.sh script is a helper which determines the IP address of the conainer running the requested service and connects to the CRaSH port. It also captures all console output to a file which can be sent to the SSDT.

The "thread ls" command lists all the Java threads and their states. The "thread ls | thread dump" command dumps the current stack trace for all threads.  The SSDT can use this information to see which thread(s) are blocked and the line of code being executed.

 

There is no content with the specified labels