Page History

...

Wiki MarkupThe trailing byte ranges overlap the range of both the single-byte and lead-byte characters. When a multibyte character is separated across a buffer boundary, it can be interpreted differently than if it were not separated across the buffer boundary; this difference arises because of the ambiguity of its composing bytes \ [[Phillips 2005|AA. References#Phillips 05]\].

Supplementary Characters

Wiki MarkupAccording to the Java API \[ [API 2006|AA. References#API 06]\] class {{Character}} documentation (Unicode Character Representations):

The char data type (and consequently the value that a Character object encapsulates) are based on the original Unicode specification, which defined characters as fixed-width 16-bit entities. The Unicode standard has since been changed to allow for characters whose representation requires more than 16 bits. The range of legal code points is now \u0000 to \u10FFFF, known as Unicode scalar value.

The Java 2 platform uses the UTF-16 representation in char arrays and in the String and StringBuffer classes. In this representation, supplementary characters are represented as a pair of char values, the first from the high-surrogates range, (\uD800-\uDBFF), the second from the low-surrogates range (\uDC00-\uDFFF).

An int value represents all Unicode code points, including supplementary code points. The lower (least significant) 21 bits of int are used to represent Unicode code points, and the upper (most significant) 11 bits must be zero. Unless otherwise specified, the behavior with respect to supplementary characters and surrogate char values is as follows:

• The methods that only accept a char value cannot support supplementary characters. They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter('\uD840') returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.
• The methods that accept an int value support all Unicode characters, including supplementary characters. For example, Character.isLetter(0x2F81A) returns true because the code point value represents a letter (a CJK ideograph).

...

Noncompliant Code Example (Substring)

Wiki MarkupThis noncompliant code example attempts to trim leading letters from the {{string}}. It fails to accomplish this task because {{Character.isLetter()}} lacks support for supplementary and combining characters \ [[Hornig 2007|AA. References#Hornig 07]\].

// Fails for supplementary or combining characters
public static String trim_bad1(String string) {
  char ch;
  int i;
  for (i = 0; i < string.length(); i += 1) {
    ch = string.charAt(i);
    if (!Character.isLetter(ch)) {
      break;
    }
  }
  return string.substring(i);
}

Noncompliant Code Example (Substring)

Wiki MarkupThis noncompliant code example attempts to correct the problem by using the {{String.codePointAt()}} method, which accepts an {{int}} argument. This works for supplementary characters but fails for combining characters \[ [Hornig 2007|AA. References#Hornig 07]\].

// Fails for combining characters
public static String trim_bad2(String string) {
  int ch;
  int i;
  for (i = 0; i < string.length(); i += Character.charCount(ch)) {
    ch = string.codePointAt(i);
    if (!Character.isLetter(ch)) {
      break;
    }
  } 
  return string.substring(i);
}

Compliant Solution (Substring)

...

This compliant solution works both for supplementary and for combining characters \ [[Hornig 2007|AA. References#Hornig 07]\]. According to the Java API \ [[API 2006|AA. References#API 06] \] class {{java.text.BreakIterator}} documentation:

The BreakIterator class implements methods for finding the location of boundaries in text. Instances of BreakIterator maintain a current position and scan over text returning the index of characters where boundaries occur.

...

Rule	Severity	Likelihood	Remediation Cost	Priority	Level
IDS10-J	low	unlikely	medium	P2	L3

Bibliography

...

IDS09-J. Do not use locale-dependent methods on locale-dependent data without specifying the appropriate locale            IDS11-J. Eliminate noncharacter code points before validation