Page History

The char data type is based on the original Unicode specification, which defined characters as fixed-width 16-bit entities. The Unicode Standard has since been changed to allow for characters whose representation requires more than 16 bits. The range of legal Unicode code points is now U+0000 to U+10FFFF, known as Unicode scalar value. (Refer to the definition of the U+n notation in the Unicode Standard. ) The set of characters from U+0000 to U+FFFF is sometimes referred to as the Basic Multilingual Plane called the basic multilingual plane (BMP). Characters , and characters whose code points are greater than U+FFFF are called supplementary characters. Such characters are generally rare, but some are used, for example, as part of Chinese and Japanese personal names. To support supplementary characters without changing the char primitive data type and causing incompatibility with previous Java programs, supplementary characters are defined by a pair of Unicode code units called surrogates. According to the Java API [API 2014] class Character documentation (Unicode Character Representations):

The Java platform uses the UTF-16 representation in char arrays and in the String and StringBuffer classes. In this representation, supplementary characters are represented as a pair of char values, the first from the high-surrogates range, (\uD800-\uDBFF), the second from the low-surrogates range (\uDC00-\uDFFF).

A char value, therefore, represents

...

BMP

...

code points, including the surrogate code points, or code units of the UTF-16 encoding. An int value represents all Unicode code points, including supplementary code points. The lower (least significant) 21 bits of int are used to represent Unicode code points, and the upper (most significant) 11 bits must be zero.

...

Character information in Java SE 8 is based on the Unicode Standard, version 6.2.0 [Unicode 2012].

However, Java programs must often work with string data represented in various character sets.  Java 7 introduced the StandardCharsets Class that specifies character sets that are guaranteed to be available on every implementation of the Java platform including ISO Latin Alphabet No. 1, Seven-bit ASCII, UTF 8, and UTF 16.

The Java language assumes that every character in a string occupies 16 bits (a Java char). Unfortunately, neither the Java byte nor Java char data types can represent all possible Unicode characters. Many strings are stored or communicated using encodings such as UTF-8 that support characters with varying sizes.

While Java strings are stored as an array of characters and can be represented as an array of bytes, a single character in the string might be represented by two or more consecutive elements of type byte or of type char. Splitting a char or byte array risks splitting a multibyte character.

Ignoring the possibility of supplementary characters and multibyte characters may allow the formation of incorrect strings.

Multibyte Characters

Multibyte encodings are used for character sets that require more than one byte to uniquely identify each constituent character. For example, the Japanese encoding Shift-JIS (shown below) supports multibyte encoding where the maximum character length is two bytes (one leading and one trailing byte).

The trailing byte ranges overlap the range of both the single-byte and lead-byte characters. When a multibyte character is separated across a buffer boundary, it can be interpreted differently than if it were not separated across the buffer boundary; this difference arises because of the ambiguity of its composing bytes [Phillips 2005].

Supplementary Characters

The char data type is based on the original Unicode specification, which defined characters as fixed-width 16-bit entities. The Unicode Standard has since been changed to allow for characters whose representation requires more than 16 bits. The range of legal code points is now U+0000 to U+10FFFF, known as Unicode scalar value.Characters whose code points are greater than U+FFFF are called supplementary characters. Such characters are generally rare, but some are used, for example, as part of Chinese and Japanese personal names. To support supplementary characters without changing the char primitive data type and causing incompatibility with previous Java programs, supplementary characters are defined by a pair of code point values that are called surrogates. According to the Java API [API 2014] class Character documentation (Unicode Character Representations):

...

Similar to UTF-8 (see STR00-J. Don't form strings containing partial characters from variable-width encodings), UTF-16 is a variable-width encoding. Because the UTF-16 representation is also used in char arrays and in the String and StringBuffer classes, care must be taken when manipulating string data in Java. In

...

particular, do not write code that assumes that a value of the primitive type char (or a Character object) fully represents a Unicode code point. Conformance with this requirement typically requires using methods that accept a Unicode code point as an int value and avoiding methods that accept a Unicode code unit as a char value because these latter methods cannot support supplementary characters.

Noncompliant Code Example

Noncompliant Code Example (Read)

This noncompliant code example tries to read up to 1024 bytes from a socket and build a String from this data. It does this by reading the bytes in a while loop, as recommended by rule FIO10-J. Ensure the array is filled when using read() to fill an array. If it ever detects that the socket has more than 1024 bytes available, it throws an exception. This prevents untrusted input from potentially exhausting the program's memory.

public final int MAX_SIZE = 1024;

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  byte[] data = new byte[MAX_SIZE+1];
  int offset = 0;
  int bytesRead = 0;
  String str = new String();
  while ((bytesRead = in.read(data, offset, data.length - offset)) != -1) {
    offset += bytesRead;
    str += new String(data, offset, data.length - offset, "UTF-8");
    if (offset >= data.length) {
      throw new IOException("Too much input");
    }
  }
  in.close();
  return str;
}

This code fails to account for the interaction between characters represented with a multibyte encoding and the boundaries between the loop iterations. If the last byte read from the data stream in one read() operation is the leading byte of a multibyte character, the trailing bytes are not encountered until the next iteration of the while loop. However, multibyte encoding is resolved during construction of the new String within the loop. Consequently, the multibyte encoding can be interpreted incorrectly.

Compliant Solution (Read)

This compliant solution defers creation of the string until all the data is available.

public final int MAX_SIZE = 1024;

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  byte[] data = new byte[MAX_SIZE+1];
  int offset = 0;
  int bytesRead = 0;
  while ((bytesRead = in.read(data, offset, data.length - offset)) != -1) {
    offset += bytesRead;
    if (offset >= data.length) {
      throw new IOException("Too much input");
    }
  }
  String str = new String(data, "UTF-8");
  in.close();
  return str;
}

This code avoids splitting multi-byte encoded characters across buffers by deferring construction of the result string until the data has been read in full.

Compliant Solution (Reader)

This compliant solution uses a Reader rather than an InputStream. The Reader class converts bytes into characters on the fly, so it avoids the hazard of splitting multibyte characters. This routine aborts if the socket provides more than 1024 characters rather than 1024 bytes.

public final int MAX_SIZE = 1024;

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  Reader r = new InputStreamReader(in, "UTF-8");
  char[] data = new char[MAX_SIZE+1];
  int offset = 0;
  int charsRead = 0;
  String str = new String(data);
  while ((charsRead = r.read(data, offset, data.length - offset)) != -1) {
    offset += charsRead;
    str += new String(data, offset, data.length - offset);
    if (offset >= data.length) {
      throw new IOException("Too much input");
    }
  }
  in.close();
  return str;
}

...

This noncompliant code example attempts to trim leading letters from string. :

Code Block
bgColor #FFcccc
public static String trim(String string) { char ch; int i; for (i = 0; i < string.length(); i += 1) { ch = string.charAt(i); if (!Character.isLetter(ch)) { break; } } return string.substring(i); }

...

Unfortunately, the trim() method may fail because it is using the character form of the Character.isLetter() method.   Methods that accept only accept a char value cannot support supplementary characters. According to the Java API [API 2014] class Character documentation: 

They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter('\uD840') returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.

Compliant Solution

...

This noncompliant code example compliant solution corrects the problem with supplementary characters by using the integer form of the Character.isLetter() method that accepts a Unicode code point as an int argument. Java library methods that accept an int value support all Unicode characters, including supplementary characters.  

Code Block
bgColor #ccccff
public static String trim(String string) { int ch; int i; for (i = 0; i < string.length(); i += Character.charCount(ch)) { ch = string.codePointAt(i); if (!Character.isLetter(ch)) { break; } } return string.substring(i); }

Risk Assessment

Forming strings consisting of partial characters can result in unexpected behavior.

Automated Detection

Bibliography

[API 2014]

Classes Character and BreakIterator

 

[Java Tutorials

2008

]	Character Boundaries
[Seacord 2015]	Image Added STR01-J. Do not assume that a Java char fully represents a Unicode code point LiveLesson

...

Image Modified      Image Modified       Image Modified