CON37-J. Ensure that tasks executing in a thread pool do not fail silently

Tasks that run for extended periods of time should provide a notification mechanism to alert upper layers when they terminate abnormally. Failure to do this does not cause any resource leaks because the threads in the pool are still recycled, however, it makes failure diagnosis extremely difficult.

The best way to handle exceptions at a global level is to use an exception handler. The handler can perform diagnostic actions, clean-up and shutdown the Java Virtual Machine (JVM) or simply log the details of the failure. This guideline may be violated if the code for all runnable and callable tasks has been audited to ensure that no exceptional conditions are possible. Nonetheless, it is usually a good practice to install a task specific or global exception handler to initiate recovery.

Noncompliant Code Example

This noncompliant code example consists of class PoolService that encapsulates a thread pool and a runnable class, Task. The run() method of the task can throw runtime exceptions such as NullPointerException.

final class PoolService {
  private final ExecutorService pool = Executors.newFixedThreadPool(10);
		  
  public void doSomething() {	   
    pool.execute(new Task());
  }	
}

final class Task implements Runnable {
  @Override public void run() {
    // ...
    throw new NullPointerException();
    // ...	
  }
}

The task does not notify upper layers when it terminates unexpectedly as a result of the runtime exception. Moreover, it does not use any recovery mechanism.

Compliant Solution (Custom exception reporter)

This compliant solution refactors the task so that it catches Throwable and forwards it to a custom exception reporter (see EXC01-J. Use a class dedicated to reporting exceptions for details on class MyExceptionReporter).

// ...

final class Task implements Runnable {
  @Override public void run() {
    try {
      // ...
      throw new NullPointerException();
      // ...
    } catch(Throwable t) {
      // Execute any recovery code
      MyExceptionReporter.report(t);	 
    } finally {
      // Perform clean-up actions
    }   	
  }
}

A finally block can be used to perform clean-up actions. If the task cannot be refactored, it may be wrapped within a Runnable or Callable that catches Throwable, and submitted to the thread pool.

Task specific recovery or clean-up actions can also be performed by overriding the class java.util.concurrent.ThreadPoolExecutor's afterExecute() hook. This hook is called when a task completes successfully by executing all statements in its run() method, or halts because of an exception (A java.lang.Error might not be captured on specific implementations, see Bug ID 6450211 [[SDN 08]]). When using this approach, substitute the executor service with a custom ThreadPoolExecutor that overrides the afterExecute() hook as shown below:

class CustomThreadPoolExecutor extends ThreadPoolExecutor {
  // ... Constructor ...

  @Override
  public void afterExecute(Runnable r, Throwable t) {
    super.afterExecute(r, t);
    // Perform task-specific recovery and clean-up
  }

  @Override
  public void terminated() {
     super.terminated();
     // Perform final clean-up
  }
}

Similarly, the terminated() hook is called after all the tasks have finished executing, and the Executor has terminated cleanly. This hook can be overridden to release resources acquired by the thread pool over its lifetime, much like a finally block.

Compliant Solution (Uncaught exception handler)

This compliant solution sets an uncaught exception handler on behalf of the thread pool. During its construction, an argument of type ThreadFactory is passed to the thread pool. The factory is responsible for creating new threads and setting the uncaught exception handler on their behalf. The class Task remains the same as the noncompliant code example.

final class PoolService {
  private static final ThreadFactory factory = new
    ExceptionThreadFactory(new MyExceptionHandler());
  private static final ExecutorService pool =
    Executors.newFixedThreadPool(10, factory);

  public void doSomething() {	   
    pool.execute(new Task()); // Task is a runnable class	      	    	      
  }
 
  public static class ExceptionThreadFactory implements ThreadFactory  {			
    private static final ThreadFactory defaultFactory =
      Executors.defaultThreadFactory();
    private final Thread.UncaughtExceptionHandler handler;

    public ExceptionThreadFactory(Thread.UncaughtExceptionHandler handler) {
      this.handler = handler;
    }

    @Override public Thread newThread(Runnable run) {
      Thread thread = defaultFactory.newThread(run);
      thread.setUncaughtExceptionHandler(handler);
      return thread;
    }
  }
	  
  public static class MyExceptionHandler extends ExceptionReporter 
    implements Thread.UncaughtExceptionHandler {

    private final Logger logger = Logger.getLogger("com.organization.Log");
    // ... 

    @Override public void uncaughtException(Thread thread, Throwable t) {
      logger.log(Level.SEVERE, "Thread exited with exception: " + thread.getName(), t);			      
    }
  }	  
}

Note that the uncaught exception handler is not called if the method ExecutorService.submit() is invoked. This is because the thrown exception is considered to be part of the return status and is consequently, re-thrown by Future.get(), wrapped in an ExecutionException [[Goetz 06]].

Risk Assessment

Failing to provide a mechanism to report that tasks in a thread pool failed as a result of an exceptional condition, can make it harder to find the source of the issue.

Automated Detection

TODO

Related Vulnerabilities

Search for vulnerabilities resulting from the violation of this rule on the CERT website.

References

[[API 06]] interfaces ExecutorService, ThreadFactory and class Thread
[[Goetz 06]] Chapter 7.3: Handling abnormal thread termination

CON12-J. Avoid deadlock by requesting and releasing locks in the same order      11. Concurrency (CON)      VOID CON14-J. Ensure atomicity of 64-bit operations