Page History

import java.text.Normalizer;
import java.text.Normalizer.Form;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
 
public class TagFilter {
  public static String filterString(String str) {
    String 

According to the Unicode Technical Report #36, Unicode Security Considerations \[[Davis 2008b|AA. Bibliography#Davis 08b]\], Section 3.5, "Deletion of Noncharacters" 

{quote}
Conformance clause C7 reads

    C7. When a process purports not to modify the interpretation of a valid coded character sequence, it shall make no change to that coded character sequence other than the possible replacement of character sequences by their canonical-equivalent sequences or the deletion of noncharacter code points.

Although the last phrase permits the deletion of noncharacter code points, for security reasons, they only should be removed with caution.

Whenever a character is invisibly deleted (instead of replaced), it may cause a security problem. The issue is the following: A gateway might be checking for a sensitive sequence of characters, say "delete". If what is passed in is "deXlete", where X is a noncharacter, the gateway lets it through: the sequence "deXlete" may be in and of itself harmless. But suppose that later on, past the gateway, an internal process invisibly deletes the X. In that case, the sensitive sequence of characters is formed, and can lead to a security breach.
{quote}


h2. Noncompliant Code Example

This noncompliant code example accepts only valid ASCII characters and deletes any non conforming characters. Input validation is being performed before this step. Consequently, a malicious {{<script>}} tag would bypass the filter, despite being black-listed.   

{code:bgColor=#FFcccc}
String s = "<scr" + "\uFEFF" + "ipt>"; // "\uFEFF" is a non-character code point
s = Normalizer.normalize(sstr, Form.NFKC);

    // InputValidate validationinput
    Pattern pattern = Pattern.compile("<script>");
    Matcher matcher = pattern.matcher(s);
    if (matcher.find()) {
   System.out.println   throw new IllegalArgumentException("Found black listed tagInvalid input");
    }

 else {
  // ... 
}

 Deletes noncharacter code points
    s = s.replaceAll("^[\\p{ASCIICn}]", "");
    return s;
  }

  public static void main(String[] args) {
    // Deletes all non-valid characters
// s now contains "<script>"		
{code}


h2. Compliant Solution

This compliant solution replaces the unknown or unrepresentable character with unicode sequence {{\uFFFD}} which is reserved to denote this condition. This ensures that malicious input cannot bypass filters.  

{mc}
Strange things are happening with the regex below. Our bot inserts a link to the same rec within the code regex.
{mc}

{code:bgColor=#ccccff}
String s = "<scr" + "\uFEFF" + "ipt>";
s = Normalizer.normalize(s, Form.NFKC);
 "\uFDEF" is a noncharacter code point
    String maliciousInput = "<scr" + "\uFDEF" + "ipt>";
    String sb = filterString(maliciousInput);
    // sb = "<script>"
  }
}

import java.text.Normalizer;
import java.text.Normalizer.Form;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class TagFilter {
 
  public static String filterString(String str) {
    String s = Normalizer.normalize(str, Form.NFKC);

    // Replaces all noncharacter code points with Unicode U+FFFD
    s = s.replaceAll("[\\p{Cn}]", "\uFFFD");

    // Validate input
    Pattern pattern = Pattern.compile("<script>");
    Matcher matcher = pattern.matcher(s);
    if (matcher.find()) {
  System.out.println("Found black listed tag");
} else {
  // ... 
}
s = s.replaceAll("^\\p{ASCII}]", "\uFFFD"); // Replaces all non-valid characters with unicode U+FFFD
{code}

"{{U+FFFD}} is usually unproblematic, because it is designed expressly for this kind of purpose. That is, because it doesn't have syntactic meaning in programming languages or structured data, it will typically just cause a failure in parsing. Where the output character set is not Unicode, though, this character may not be available" \[[Davis 2008b|AA. Bibliography#Davis 08b]\].


h2. Risk Assessment

Deleting non-character code points can allow malicious input to bypass validation checks.

|| Guideline || Severity || Likelihood || Remediation Cost || Priority || Level ||
| IDS03-J | high | probable | medium | {color:red}{*}P12{*}{color} | {color:red}{*}L1{*}{color} |


h3. Related Vulnerabilities

Search for vulnerabilities resulting from the violation of this guideline on the [CERT website|https://www.kb.cert.org/vulnotes/bymetric?searchview&amp;query=FIELD+KEYWORDS+contains+MSC42-J].


h2. Bibliography

\[[API 2006|AA. Bibliography#API 06]\] 
\[[Davis 2008b|AA. Bibliography#Davis 08b]\] 3.5 Deletion of Noncharacters
\[[MITRE 2009|AA. Bibliography#MITRE 09]\] [CWE ID|http://cwe.mitre.org/data/definitions/182.html] "Collapse of Data Into Unsafe Value"
\[[Weber 2009|AA. Bibliography#Weber 09]\] Handling the Unexpected: Character-deletion

----
[!The CERT Oracle Secure Coding Standard for Java^button_arrow_left.png!|IDS02-J. Normalize strings before validating them]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[!The CERT Oracle Secure Coding Standard for Java^button_arrow_up.png!|13. Input Validation and Data Sanitization (IDS)]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[!The CERT Oracle Secure Coding Standard for Java^button_arrow_right.png!|IDS04-J. Properly encode or escape output]

    throw new IllegalArgumentException("Invalid input");
    }
    return s;
  }
  public static void main(String[] args) {
    // "\uFDEF" is a non-character code point
    String maliciousInput = "<scr" + "\uFDEF" + "ipt>";
    String s = filterString(maliciousInput);
    // s = <scr?ipt>
  }