Page History

...

A char value, therefore, represents BMP code points, including the surrogate code points, or code units of the UTF-16 encoding. An int value represents all Unicode code points, including supplementary code points. The lower (least significant) 21 bits of int are used to represent Unicode code points and the upper (most significant) 11 bits must be zero.   Similar to UTF-8 (see STR00-J. Don't form strings containing partial characters from variable-width encodings), UTF-16 is a variable-width encoding.  It  uses a single 16-bit code unit to encode the most common 63K characters, and a pair of 16-bit code units to encode the 1M less commonly used characters in Unicode.   Because UTF-16 code point ranges for high and low surrogates, as well as for single units are all completely disjoint there are no false matches, the location of the character boundary can be directly determined from each code unit value, and a dropped surrogate will corrupt only a single character. Similar to UTF-8 and other variable-width encodings, programmers must be careful when reading UTF-16 data as a series of bytes to not form strings containing partial Unicode code points (that is, a high surrogate value without a corresponding low surrogate).  Because the UTF-16 representation is also used in char arrays and in the String and StringBuffer classes, care must also be taken when manipulating string data in Java. This typically means . In particular, do not write code that assumes that a value of the primitive type char (or a Character object) fully represents a Unicode code point. Conformance with this requirement typically requires using methods that accept a Unicode code point as an int value and avoiding methods that accept a Unicode code unit as a char value as these latter methods cannot support supplementary characters.

Noncompliant Code Example (Read)

This noncompliant code example tries to read up to 1024 bytes from a socket and build a String from this data. It does this by reading the bytes in a while loop, as recommended by rule FIO10-J. Ensure the array is filled when using read() to fill an array. If it ever detects that the socket has more than 1024 bytes available, it throws an exception. This prevents untrusted input from potentially exhausting the program's memory.

public final int MAX_SIZE = 1024;

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  byte[] data = new byte[MAX_SIZE+1];
  int offset = 0;
  int bytesRead = 0;
  String str = new String();
  while ((bytesRead = in.read(data, offset, data.length - offset)) != -1) {
    offset += bytesRead;
    str += new String(data, offset, data.length - offset, "UTF-16");
    if (offset >= data.length) {
      throw new IOException("Too much input");
    }
  }
  in.close();
  return str;
}

This code fails to account for the interaction between variable-width character encodings and the boundaries between the loop iterations. If the last byte read from the data stream in one read() operation is a high surrogate value the corresponding low surrogate is not encountered until the subsequent read. However, variable-width encoding is resolved during construction of the new String within the loop. Consequently, the supplementary characters can be interpreted incorrectly.

Compliant Solution (Read)

This compliant solution defers creation of the string until all the data is available.

public final int MAX_SIZE = 1024;

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  byte[] data = new byte[MAX_SIZE+1];
  int offset = 0;
  int bytesRead = 0;
  while ((bytesRead = in.read(data, offset, data.length - offset)) != -1) {
    offset += bytesRead;
    if (offset >= data.length) {
      throw new IOException("Too much input");
    }
  }
  String str = new String(data, "UTF-16");
  in.close();
  return str;
}

This code avoids splitting supplementary characters across buffers by deferring construction of the result string until the data has been read in full.

Compliant Solution (Reader)

This compliant solution uses a Reader rather than an InputStream. The Reader class converts bytes into characters on the fly, so it avoids the hazard of splitting multibyte characters. This routine aborts if the socket provides more than 1024 characters rather than 1024 bytes.

public final int MAX_SIZE = 1024;

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  Reader r = new InputStreamReader(in, "UTF-16");
  char[] data = new char[MAX_SIZE+1];
  int offset = 0;
  int charsRead = 0;
  String str = new String(data);
  while ((charsRead = r.read(data, offset, data.length - offset)) != -1) {
    offset += charsRead;
    str += new String(data, offset, data.length - offset);
    if (offset >= data.length) {
      throw new IOException("Too much input");
    }
  }
  in.close();
  return str;
}

...

Noncompliant Code Example

This noncompliant code example attempts to trim leading letters from string. 

...

They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter('\uD840') returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.

Compliant Solution

...

This noncompliant code example corrects the problem with supplementary characters by using the integer form of Character.isLetter() method that accepts a Unicode code point as an int argument. Java library methods that accept an int value support all Unicode characters, including supplementary characters.  

Code Block
bgColor #ccccff
public static String trim(String string) { int ch; int i; for (i = 0; i < string.length(); i += Character.charCount(ch)) { ch = string.codePointAt(i); if (!Character.isLetter(ch)) { break; } } return string.substring(i); }

Risk Assessment

Forming strings consisting of partial characters can result in unexpected behavior.

Bibliography

...