SER12-J. Avoid memory and resource leaks during serialization

Serialization can prevent garbage collection and consequently induce memory leaks. Every time an object is written to a stream, a reference (or handle) to the object is retained by a table maintained by ObjectOutputStream. If the same object (regardless of its contents) is written out to the same stream again, it is replaced with a reference to the originally cached object (recall that ObjectOutputStream maintains a live reference to an object after it is written for the first time). The garbage collector cannot reclaim the memory associated with new objects as it cannot collect live references.

Noncompliant Code Example

This noncompliant code example writes a sequence of array objects to a stream. The array object arr is created afresh within the loop and filled uniformly with the value of the loop counter. This may result in an OutOfMemoryError because the stream is kept open while new objects are being written to it.

class MemoryLeak {
  public static void main(String[] args) throws IOException {
    ObjectOutputStream out = new ObjectOutputStream(
        new BufferedOutputStream(new FileOutputStream("ser.dat")));
    for (int i = 0; i < 1024; i++) {
      byte[] arr = new byte[100 * 1024];
      Arrays.fill(arr, (byte) i);
      out.writeObject(arr);
    }
    out.close();
  }
  }

Compliant Solution

Ideally, the stream should be closed as soon as the work is accomplished. This compliant solution adopts an alternative approach by resetting the stream after every write so that the internal cache no longer maintains live references, allowing the garbage collector to resume.

class NoMemoryLeak {
  public static void main(String[] args) throws IOException {
    ObjectOutputStream out = new ObjectOutputStream(
      new BufferedOutputStream(new FileOutputStream("ser.dat")));
    for (int i = 0; i < 1024; i++) {
      byte[] arr = new byte[100 * 1024];
      Arrays.fill(arr, (byte) i);
      out.writeObject(arr);
      out.reset(); // Reset the stream
    }
    out.close();
  }
}

Risk Assessment

Memory and resource leaks during serialization can corrupt the state of the object or crash the JVM.

Automated Detection

TODO

Related Vulnerabilities

Search for vulnerabilities resulting from the violation of this rule on the CERT website.

Bibliography

[[API 2006]]
[[Sun 2006]] "Serialization specification"
[[Harold 2006]] 13.4. Performance

SER11-J. Do not invoke overridable methods from the readObject method      18. Serialization (SER)      SER13-J. Prevent overwriting of Externalizable Objects