In some versions prior to Unicode 5.2, conformance clause C7 allowed the deletion of noncharacter code points. For example, conformance clause C7 from Unicode 5.1 states: \[[Unicode 2007|AA. Bibliography#Unicode 2007]\]
{quote}
C7. When a process purports not to modify the interpretation of a valid coded character sequence, it shall make no change to that coded character sequence other than the possible replacement of character sequences by their canonical-equivalent sequences or the deletion of noncharacter code points.
{quote}
According to the Unicode Technical Report #36, Unicode Security Considerations \[[Davis 2008b|AA. Bibliography#Davis 08b]\], Section 3.5, "Deletion of Noncharacters"
{quote}
Whenever a character is invisibly deleted (instead of replaced), such as in this older version of C7, it may cause a security problem. The issue is the following: A gateway might be checking for a sensitive sequence of characters, say "delete". If what is passed in is "deXlete", where X is a noncharacter, the gateway lets it through: the sequence "deXlete" may be in and of itself harmless. However, suppose that later on, past the gateway, an internal process invisibly deletes the X. In that case, the sensitive sequence of characters is formed, and can lead to a security breach.
{quote}
Because character-level modifications of a string can nullify substring-level checks, it is important to perform the character-level modifications before substring-level checks.
h2. Noncompliant Code Example
This noncompliant code example accepts only valid ASCII characters and deletes any non conforming characters. It also checks for the existence of a {{<script>}} tag.
Input validation is being performed before the character checks. As such, this code also violates [IDS01-J. Normalize strings before validating them]. Consequently, an attacker can disguise a {{<script>}} tag and fool the filter.
{code:bgColor=#FFcccc}
String s = "<scr" + "\uFEFF" + "ipt>"; // "\uFEFF" is a non-character code point
s = Normalizer.normalize(s, Form.NFKC);
// Input validation
Pattern pattern = Pattern.compile("<script>");
Matcher matcher = pattern.matcher(s);
if(matcher.find()) {
System.out.println("Found black listed tag");
} else {
// ...
}
s = s.replaceAll("^\\p{ASCII}]", ""); // Deletes all non-valid characters
// s now contains "<script>"
{code}
h2. Compliant Solution
This compliant solution replaces the unknown or unrepresentable character with unicode sequence {{\uFFFD}} which is reserved to denote this condition. It also does this replacement before doing any other sanitization, in particular, checking for {{<script>}}. This ensures that malicious input cannot bypass filters.
{mc}
Strange things are happening with the regex below. Our bot inserts a link to the same rec within the code regex.
{mc}
{code:bgColor=#ccccff}
String s = "<scr" + "\uFEFF" + "ipt>";
s = Normalizer.normalize(s, Form.NFKC);
s = s.replaceAll("^\\p{ASCII}]", "\uFFFD"); // Replaces all non-valid characters with unicode U+FFFD
Pattern pattern = Pattern.compile("<script>");
Matcher matcher = pattern.matcher(s);
if(matcher.find()) {
System.out.println("Found black listed tag");
} else {
// ...
}
{code}
"{{U+FFFD}} is usually unproblematic, because it is designed expressly for this kind of purpose. That is, because it doesn't have syntactic meaning in programming languages or structured data, it will typically just cause a failure in parsing. Where the output character set is not Unicode, though, this character may not be available" \[[Davis 2008b|AA. Bibliography#Davis 08b]\].
h2. Risk Assessment
Deleting non-character code points can allow malicious input to bypass validation checks.
|| Rule || Severity || Likelihood || Remediation Cost || Priority || Level ||
| IDS11-J | high | probable | medium | {color:red}{*}P12{*}{color} | {color:red}{*}L1{*}{color} |
h2. Related Guidelines
Search for vulnerabilities resulting from the violation of this rule on the [CERT website|https://www.kb.cert.org/vulnotes/bymetric?searchview&query=FIELD+KEYWORDS+contains+MSC42-J].
| \[[MITRE 2009|AA. Bibliography#MITRE 09]\] | [CWE ID 182|http://cwe.mitre.org/data/definitions/182.html] "Collapse of Data Into Unsafe Value" |
h2. Bibliography
|\[[API 2006|AA. Bibliography#API 06]\] | |
|\[[Davis 2008b|AA. Bibliography#Davis 08b]\] | 3.5 Deletion of Noncharacters |
|\[[Weber 2009|AA. Bibliography#Weber 09]\]| Handling the Unexpected: Character-deletion |
|\[[Unicode 2007|AA. Bibliography#Unicode 2007]\]| The Unicode Consortium. The Unicode Standard, Version 5.1.0, defined by: The Unicode Standard, Version 5.0 (Boston, MA, Addison-Wesley, 2007. ISBN 0-321-48091-0), as amended by Unicode 5.1.0 (http://www.unicode.org/versions/Unicode5.1.0/). |
| \[[Unicode 2011|AA. Bibliography#Unicode 2011]\] | The Unicode Consortium. The Unicode Standard, Version 6.0.0, (Mountain View, CA: The Unicode Consortium, 2011. ISBN 978-1-936213-01-6) http://www.unicode.org/versions/Unicode6.0.0/ |
----
[!The CERT Oracle Secure Coding Standard for Java^button_arrow_left.png!|IDS01IDS10-J. Normalize strings before validating them Do not assume every character in a string is the same size] [!The CERT Oracle Secure Coding Standard for Java^button_arrow_up.png!|00. Input Validation and Data Sanitization (IDS)] [!The CERT Oracle Secure Coding Standard for Java^button_arrow_right.png!|IDS04IDS12-J. Do not log unsanitized user input Perform lossless conversion of String data between differing character encodings]
| 