SER10-J. Avoid memory and resource leaks during serialization

Serialization of object on an output stream can extend the lifetime of objects, because the output stream keeps a reference to each serialized object, in its work at recognizing duplicate objects. Thus serializing a large set of objects prevents the objects from being garbage-collected.

While this behavior is desirable for data that potentially contains arbitrary object graphs, it can lead to memory exhaustion when serializing data that both lacks references to other objects being serialized and also can be allocated in part or in full after serialization has begun. One such example is serializing a data stream from an external sensor. In such cases, programs must take additional action to avoid memory exhaustion. That is, programs reading in independent serialized data must reset the object cache between reads to prevent memory exhaustion.

This rule is a specific instance of the more general rule MSC07-J. Do not assume infinite heap space.

Noncompliant Code Example

This noncompliant code example reads and serializes data from an external sensor. Each invocation of the readSensorData() method returns a newly created SensorData instance, each containing a megabyte of data. SensorData instances are pure data streams, containing data and arrays but lacking references to other SensorData objects.

As already described, the ObjectOutputStream maintains a cache of previously written objects. Consequently, all SensorData objects remain alive until the cache itself becomes garbage. This can result in an OutOfMemoryError because the stream remains open while new objects are being written to it.

class SensorData implements Serializable {
  // 1MB of data per instance!
  ... 
  public static SensorData readSensorData() {...}
  public static boolean isAvailable() {...}
}

class SerializeSensorData {
  public static void main(String[] args) throws IOException {
    ObjectOutputStream out = null;
    try {
      out = new ObjectOutputStream(
          new BufferedOutputStream(new FileOutputStream("ser.dat")));
      while (SensorData.isAvailable()) {
        // note that each SensorData object is 1MB in size
        SensorData sd = SensorData.readSensorData();
        out.writeObject(sd);
      }
    } finally {
      out.close();
    }
  }
}

Compliant Solution

This compliant solution takes advantage of the known properties of the sensor data by resetting the output stream after each write. The reset clears the output stream's internal object cache; consequently, the cache no longer maintains references to previously written SensorData objects. The garbage collector can collect SensorData instances that are no longer needed.

class SerializeSensorData {
  public static void main(String[] args) throws IOException {
    ObjectOutputStream out = null;
    try {
      out = new ObjectOutputStream(
          new BufferedOutputStream(new FileOutputStream("ser.dat")));
      while (SensorData.isAvailable()) {
        // note that each SensorData object is 1MB in size
        SensorData sd = SensorData.readSensorData();
        out.writeObject(sd);
        out.reset(); // reset the stream
      }
    } finally {
      out.close();
    }
  }
}

Risk Assessment

Memory and resource leaks during serialization can result in a resource exhaustion attack or crash the JVM.

Related Guidelines

Bibliography

SER09-J. Do not invoke overridable methods from the readObject method      13. Serialization (SER)