Page History

Many old programs assumed that every character in a string occupied 8 bits, eg a Java byte. The Java language assumed that every character in a string occupies 16 bytes, eg a Java char. Unfortunately, the Java byte was not sufficient to hold all possible characters, and neither is a Java char. Many strings are stored on disk and in memory using an encoding such as UTF-8 that allows characters to have varying sizes.Consequently, while Java strings are stored as arrays of chars, and can be represented as an array of bytes, a single character in the string might be represented by two or more consecutive bytes or chars. Splitting a char or byte array runs the risk of splitting two chars or bytes that make up a multibyte character. Security vulnerabilities may arise when an application expects input in a form that an attacker is capable of bypassing. This can happen when an application disregards supplementary characters, multibyte characters, or when it fails to use combining characters appropriately. Combining characters are characters that modify other characters. Refer to the Combining Diacritical Marks chart for more details on combining characters.

Consequently, programs must take multibyte characters into account when manipulating string arrays of bytes or chars.

Multibyte Characters

Multibyte encodings such as UTF-8 are used for character sets that require more than one byte to uniquely identify each constituent character. For example, the Japanese encoding Shift-JIS (shown below), supports multibyte encoding wherein the maximum character length is two bytes (one leading and one trailing byte).

The trailing byte ranges overlap the range of both the single byte and lead byte characters. When a multibyte character is separated across a buffer boundary, it can be interpreted differently than if it were not separated across the buffer boundary; this difference arises due to the ambiguity of its composing bytes [Phillips 2005].

Supplementary Characters

According to the Java API \[[API 2006|AA. Bibliography#API 06]\], class {{Character}} documentation (Unicode Character Representations)

...

data type is based on the original Unicode specification, which defined characters as fixed-width 16-bit entities. The Unicode

...

Standard has since been changed to allow for characters whose representation requires more than 16 bits. The range of

...

Unicode code

...

points is now U+0000 to U+10FFFF. The set of characters from U+0000 to U+FFFF is called the basic multilingual plane (BMP), and characters whose code points are greater than U+FFFF are called supplementary characters. Such characters are generally rare, but some are used, for example, as part of Chinese and Japanese personal names. To support supplementary characters without changing the char primitive data type and causing incompatibility with previous Java programs, supplementary characters are defined by a pair of Unicode code units called surrogates. According to the Java API [API 2014] class Character documentation (Unicode Character Representations):

The Java , known as Unicode scalar value. The Java 2 platform uses the UTF-16 representation in char arrays and in the String and StringBuffer classes. In this representation, supplementary characters are represented as a pair of char values, the first from the high-surrogates range, (\uD800-\uDBFF), the second from the low-surrogates range (\uDC00-\uDFFF).

A char value, therefore, represents BMP code points, including the surrogate code points, or code units of the UTF-16 encoding. An int value represents all Unicode code points, including supplementary code points. The lower (least significant) 21 bits of int are used to represent Unicode code points, and the upper (most significant) 11 bits must be zero.

...

Similar to UTF-8 (see STR00-J. Don't form strings containing partial characters from variable-width encodings), UTF-16 is a variable-width encoding. Because the UTF-16 representation is also used in char arrays and in the String and StringBuffer classes, care must be taken when manipulating string data in Java. In particular, do not write code that assumes that a value of the primitive type char (or a Character object) fully represents a Unicode code point. Conformance with this requirement typically requires using methods that accept a Unicode code point as an int value and avoiding methods that accept a Unicode code unit as a char value because these latter methods cannot support supplementary characters.

Noncompliant Code Example

This noncompliant code example attempts to trim leading letters from string:

public static String trim

• The methods that only accept a char value cannot support supplementary characters. They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter('\uD840') returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.
• The methods that accept an int value support all Unicode characters, including supplementary characters. For example, Character.isLetter(0x2F81A) returns true because the code point value represents a letter (a CJK ideograph).

Noncompliant Code Example (byte)

This noncompliant code example attempts to read bytes from a FileInputStream, in chunks of 1024, and to return them as a String.

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  String str = "";
  byte[] data = new byte[1024];
  while (in.read(data) > -1) {
    str += new String(data, "UTF-8");
  }
  in.close();
  return str;
}

This code fails to consider the interaction between characters represented with a multi-byte encoding and the boundaries between the loop iterations. If the 1024th byte read from the data stream in one read() operation is the leading byte of a multibyte character, the trailing bytes will not be encountered until the next iteration of the while loop. However, multi-byte encoding is resolved during construction of the new String within the loop. Consequently, the multibyte encoding will be interpreted incorrectly.

Compliant Solution (byte)

This compliant solution reads all the desired bytes into its buffer, and does not create a string until all the data is available. It does assume that the string to be read is 4096 bytes or less.

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  int offset = 0;
  int bytesRead = 0;
  byte[] data = new byte[4096];
  while (true) { 
    bytesRead += in.read(data, offset, data.length - offset);
    if (bytesRead == -1 || offset >= data.length)
      break;
    offset += bytesRead;
  }
  in.close();
  String str = new String(data, "UTF-8");
  return str;
}

This code avoids splitting multibyte encoded characters across buffers by deferring construction of the result string until the data have been read in full. It also facilitates portability across systems that use different default character encodings by specifying an explicit character encoding for the String constructor.

The size of the data byte buffer depends on the maximum number of bytes required to write an encoded character. For example, UTF-8 encoded data requires four bytes to represent any character above U+FFFF. Because Java uses the UTF-16 character encoding to represent char data, such sequences are split into two separate char values of two bytes each. Consequently, the buffer size should be four times the size of a typical byte sequence.

Compliant Solution (byte, readFully())

The no-argument and one-argument readFully() methods of the DataInputStream class guarantee that they either will read all of the requested data or will throw an exception. These methods throw EOFException if they detect the end of input before the required number of bytes have been read; they throw IOException if some other input/output error occurs. This compliant solution also specifies an explicit character encoding to the String constructor. This compliant solution also assumes a maximum string size of 4096 bytes.

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  byte[] data = new byte[4096];
  DataInputStream din = new DataInputStream(in);
  din.readFully(data);
  in.close();
  String str = new String(data, "UTF-8");
  return str;
}

Noncompliant Code Example (char)

This noncompliant code example attempts to trim leading letters from the {{string}}. It fails to accomplish this task because {{Character.isLetter()}} lacks support for supplementary and combining characters \[[Hornig 2007|AA. Bibliography#Hornig 07]\].

// Fails for supplementary or combining characters
public static String trim_bad1(String string) {
  char ch;
  int i;
  for (int i = 0; i < string.length(); i += 1) {
    ch = string.charAt(i);
    if (!Character.isLetter(ch)) {
      break;
    }
  }
  return string.substring(i);
}

Noncompliant Code Example (char)

This noncompliant code example attempts to fix the problem by using the {{String.codePointAt()}} method, which accepts an {{int}} argument. This works for supplementary characters but fails for combining characters \[[Hornig 2007|AA. Bibliography#Hornig 07]\].

// Fails for combining characters
public static String trim_bad2(String string) {
  int ch;
  for (int i = 0; i < string.length(); i += Character.charCount(ch)) {
    int ch = string.codePointAt(i);
    if (!Character.isLetter(ch)) {
      break;
    }
  }
  
  return string.substring(i);
}

Compliant Solution (char)

This compliant solution works both for supplementary and for combining characters \[[Hornig 2007|AA. Bibliography#Hornig 07]\]. According to the Java API \[[API 2006|AA. Bibliography#API 06]\], class {{java.text.BreakIterator}} documentation

The BreakIterator class implements methods for finding the location of boundaries in text. Instances of BreakIterator maintain a current position and scan over text returning the index of characters where boundaries occur.

Unfortunately, the trim() method may fail because it is using the character form of the Character.isLetter() method. Methods that accept only a char value cannot support supplementary characters. According to the Java API [API 2014] class Character documentation:

They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter('\uD840') returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.

Compliant Solution

This compliant solution corrects the problem with supplementary characters by using the integer form of the Character.isLetter() method that accepts a Unicode code point as an int argument. Java library methods that accept an int value support all Unicode characters, including supplementary characters.  

...

public static String trim_good(String string) {
  BreakIterator iter = BreakIterator.getCharacterInstance()int ch;
   iter.setText(string)int i;
  for (int i = iter.first()0; i !=< BreakIteratorstring.DONElength(); i += iterCharacter.nextcharCount(ch)) {
    int ch = string.codePointAt(i);
    if (!Character.isLetter(ch)) {
      break;
    }    
  }
  
  if (i == BreakIterator.DONE) { // Reached first or last text boundary
    return ""; // The input was either blank or had only (leading) letters
  } else {
    return string.substring(i);
  }
}

...

Risk Assessment

Failure to correctly account for supplementary and combining characters can lead to Forming strings consisting of partial characters can result in unexpected behavior.

Related Guidelines

Search for vulnerabilities resulting from the violation of this rule on the CERT website.

Bibliography

Automated Detection

Bibliography

[API 2014]

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="7dc501fc-1d20-47c6-b23c-5f76bd68847c"><ac:plain-text-body><![CDATA[

[[API 2006

AA. Bibliography#API 06]]

Classes Character and BreakIterator

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="0a1efa1f-c27c-4bde-b941-8eb277ebfdd8"><ac:plain-text-body><![CDATA[

[[Hornig 2007

AA. Bibliography#Hornig 07]]

Problem areas: Characters

]]></ac:plain-text-body></ac:structured-macro>

[Java Tutorials]	Character Boundaries
[Seacord 2015]	Image Added STR01-J. Do not assume that a Java char fully represents a Unicode code point LiveLesson

...

Image Added Image Added Image AddedIDS08-J. Do not pass untrusted, unsanitized data to the Runtime.exec() method      Image Removed      IDS14-J. Perform lossless conversion of String data between differing character encodings