Page History

According to the Java API \[[API 06|AA. Java References#API 06]\], class {{Character}} documentation (Unicode Character Representations):

The char data type

...

is based on the original Unicode specification, which defined characters as fixed-width 16-bit entities. The Unicode

...

Standard has since been changed to allow for characters whose representation requires more than 16 bits. The range of

...

Unicode code

...

points is now U+0000 to U+10FFFF. The set of characters from U+0000 to U+FFFF is called the basic multilingual plane (BMP), and characters whose code points are greater than U+FFFF are called supplementary characters. Such characters are generally rare, but some are used, for example, as part of Chinese and Japanese personal names. To support supplementary characters without changing the char primitive data type and causing incompatibility with previous Java programs, supplementary characters are defined by a pair of Unicode code units called surrogates. According to the Java API [API 2014] class Character documentation (Unicode Character Representations):

The Java , known as Unicode scalar value. The Java 2 platform uses the UTF-16 representation in char arrays and in the String and StringBuffer classes. In this representation, supplementary characters are represented as a pair of char values, the first from the high-surrogates range, (\uD800-\uDBFF), the second from the low-surrogates range (\uDC00-\uDFFF).

A char value, therefore, represents BMP code points, including the surrogate code points, or code units of the UTF-16 encoding. An int value represents all Unicode code points, including supplementary code points. The lower (least significant) 21 bits of int are used to represent Unicode code points, and the upper (most significant) 11 bits must be zero.

...

Similar to UTF-8 (see STR00-J. Don't form strings containing partial characters from variable-width encodings), UTF-16 is a variable-width encoding. Because the UTF-16 representation is also used in char arrays and in the String and StringBuffer classes, care must be taken when manipulating string data in Java. In particular, do not write code that assumes that a value of the primitive type char (or a Character object) fully represents a Unicode code point. Conformance with this requirement typically requires using methods that accept a Unicode code point as an int value and avoiding methods that accept a Unicode code unit as a char value because these latter methods cannot support supplementary characters.

Noncompliant Code Example

This noncompliant code example attempts to trim leading letters from string:

• The methods that only accept a char value cannot support supplementary characters. They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter('\uD840') returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.
• The methods that accept an int value support all Unicode characters, including supplementary characters. For example, Character.isLetter(0x2F81A) returns true because the code point value represents a letter (a CJK ideograph).

Security vulnerabilities may arise if an application expects input in a form that an adversary is capable of bypassing. This can happen when an application disregards supplementary characters or when it does not use combining characters appropriately. Combining characters are characters that modify other characters. Refer to the Combining Diacritical Marks chart for more details on combining characters.

Noncompliant Code Example

This noncompliant code example attempts to trim leading characters from the {{string}}. It fails to accomplish this task because {{Character.isLetter()}} does not work for supplementary and combining characters. \[[Hornig 07|AA. Java References#Hornig 07]\] (sic)

// Fails for supplementary or combining characters
public static String trim_bad1(String string) {
  char ch;
  int i;
  for (int i = 0; i < string.length(); i += 1) {
    ch = string.charAt(i);
    if (!Character.isLetter(ch)) {
      break;
    }
  }
  return string.substring(i);
}

Noncompliant Code Example

This noncompliant code example ameliorates the problem by using the {{String.codePointAt()}} method which accepts an {{int}} argument. This works for supplementary characters but not for combining characters. \[[Hornig 07|AA. Java References#Hornig 07]\] (sic)

// Fails for combining characters
public static String trim_bad2(String string) {
  int ch;
  for (int i = 0; i < string.length(); i += Character.charCount(ch)) {
    int ch = string.codePointAt(i);
    if (!Character.isLetter(ch))
      break;
    }
  return string.substring(i);
}

Compliant Solution

This compliant solution works for both supplementary and combining characters \[[Hornig 07|AA. Java References#Hornig 07]\] (sic). According to the Java API \[[API 06|AA. Java References#API 06]\], class {{java.text.BreakIterator}} documentation:

The BreakIterator class implements methods for finding the location of boundaries in text. Instances of BreakIterator maintain a current position and scan over text returning the index of characters where boundaries occur.

Unfortunately, the trim() method may fail because it is using the character form of the Character.isLetter() method. Methods that accept only a char value cannot support supplementary characters. According to the Java API [API 2014] class Character documentation:

They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter('\uD840') returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.

Compliant Solution

This compliant solution corrects the problem with supplementary characters by using the integer form of the Character.isLetter() method that accepts a Unicode code point as an int argument. Java library methods that accept an int value support all Unicode characters, including supplementary characters.  

...

public static String trim_good(String string) {
  BreakIterator iter = BreakIterator.getCharacterInstance()int ch;
  iter.setText(string) int i;
  for (int i = iter.first()0; i !=< BreakIteratorstring.DONElength(); i += iterCharacter.nextcharCount(ch)) {
    int ch = string.codePointAt(i);
    if (!Character.isLetter(ch)) {
      break;
    }
    
    if (i == BreakIterator.DONE) { // first or last text boundary has been reached
      return "";
    } else {
      return string.substring(i);
    }
  }
  return string;
}

...

Risk Assessment

Failure to account for supplementary and combining characters can lead to Forming strings consisting of partial characters can result in unexpected behavior.

Automated Detection

...

TODO

Related Vulnerabilities

Search for vulnerabilities resulting from the violation of this rule on the CERT website.

References

\[[API 06|AA. Java References#API 06]\] Classes {{Character}} and {{BreakIterator}}
\[[Hornig 07|AA. Java References#Hornig 07]\] Problem areas: Characters

Tool	Version	Checker	Description
The Checker Framework	Include Page The Checker Framework_V The Checker Framework_V	Tainting Checker	Trust and security errors (see Chapter 8)
Parasoft Jtest	Include Page Parasoft_V Parasoft_V	CERT.STR01.NCUCP	Do not assume that a Java char fully represents a Unicode code point

Bibliography

[API 2014]	Classes Character and BreakIterator
[Java Tutorials]	Character Boundaries
[Seacord 2015]	Image Added STR01-J. Do not assume that a Java char fully represents a Unicode code point LiveLesson

...

Image Added Image Added Image AddedIDS12-J. Prevent against code injection      10. Input Validation and Data Sanitization (IDS)      IDS14-J. Perform loss less conversion of String to given encoding and back