Performing conversions of String objects between different character encodings or to byte arrays may result in loss of data.
According to the Java API [API 2006], String.getBytes(Charset) method documentation:
This method always replaces malformed-input and unmappable-character sequences with this charset's default replacement byte array.
When a String must be converted to bytes, for example, for writing to a file, and the string might contain sequences of unmappable characters, proper character encoding must be performed.
Converting a byte array to a string has similar consequences if the byte array does not indicate a valid encoded string. Attempts to read raw binary data as if it were character-encoded data often fail because some of the bytes fall outside the default or specified encoding scheme and for that reason fail to denote valid characters. For example, converting a cryptographic key containing nonrepresentable bytes to character-encoded data for transmission may result in an error.
Noncompliant Code Example
This noncompliant code example attempts to convert the byte array representing a BigInteger into a String. Because some of the bytes do not denote valid characters, the resulting String representation loses information. Converting the String back to a BigInteger produces a different value.
BigInteger x = new BigInteger("530500452766");
// convert x to a String
byte[] byteArray = x.toByteArray();
String s = new String(byteArray);
// convert s back to a BigInteger
byteArray = s.getBytes();
x = new BigInteger(byteArray);
When this program was run on a Linux platform where the default character encoding is US-ASCII, the string s got the value {?J??, because some of the characters were unprintable. When converted back to a BigInteger, x got the value 149830058370101340468658109.
Compliant Solution
This compliant solution first produces a String representation of the BigInteger object and then converts the String object to a byte array. This process is reversed on input. Because the textual representation in the String object was generated by the BigInteger class, it contains valid characters.
BigInteger x = new BigInteger("530500452766");
String s = x.toString(); // valid character data
try {
byte[] byteArray = s.getBytes("UTF8");
// ns prints as "530500452766"
String ns = new String(byteArray, "UTF8");
// construct the original BigInteger
BigInteger x1 = new BigInteger(ns);
} catch (UnsupportedEncodingException ex) {
// handle error
}
Do not try to convert the String object to a byte array to obtain the original BigInteger. Character encoded data may yield a byte array that, when converted to a BigInteger, results in a completely different value.
Noncompliant Code Example
This noncompliant code example corrupts the data when string contains characters that are not representable in the specified charset.
// Corrupts data on errorspublic static byte[] toCodePage(String charset, String string) throws UnsupportedEncodingException { return string.getBytes(charset);}// Fails to detect corrupt datapublic static String fromCodePage(String charset, byte[] bytes) throws UnsupportedEncodingException { return new String(bytes, charset);} |
Compliant Solution
The java.nio.charset.CharsetEncoder class can transform a sequence of 16-bit Unicode characters into a sequence of bytes in a specific charset, while the java.nio.charset.CharacterDecoder class can reverse the procedure [API 2006]. Also see rule FIO11-J. Do not attempt to read raw binary data as character data for more information.
This compliant solution uses the CharsetEncoder and CharsetDecoder classes to handle encoding conversions.
public static byte[] toCodePage(String charset, String string) throws IOException { Charset cs = Charset.forName(charset); CharsetEncoder coder = cs.newEncoder(); ByteBuffer bytebuf = coder.encode(CharBuffer.wrap(string)); byte[] bytes = new byte[bytebuf.limit()]; bytebuf.get(bytes); return bytes;}public static String fromCodePage(String charset,byte[] bytes) throws CharacterCodingException { Charset cs = Charset.forName(charset); CharsetDecoder coder = cs.newDecoder(); CharBuffer charbuf = coder.decode(ByteBuffer.wrap(bytes)); return charbuf.toString();} |
Noncompliant Code Example
This noncompliant code example attempts to append a string to a text file in the specified encoding. This is erroneous because the String may contain unrepresentable characters.
// Corrupts data on errorspublic static void toFile(String charset, String filename, String string) throws IOException { FileOutputStream stream = new FileOutputStream(filename, true); OutputStreamWriter writer = new OutputStreamWriter(stream, charset); writer.write(string, 0, string.length()); writer.close();} |
Compliant Solution
This compliant solution uses the CharsetEncoder class to perform the required function.
public static void toFile(String filename, String string, String charset) throws IOException { Charset cs = Charset.forName(charset); CharsetEncoder coder = cs.newEncoder(); FileOutputStream stream = new FileOutputStream(filename, true); OutputStreamWriter writer = new OutputStreamWriter(stream, coder); writer.write(string, 0, string.length()); writer.close();} |
Use the FileInputStream and InputStreamReader objects to read back the data from the file. InputStreamReader accepts a optional CharsetDecoder argument, which must be the same as that previously used for writing to the file.
Exceptions
FIO11-EX0: Binary data that is expected to be a valid string may be read and converted to a string. How to perform this operation securely is explained in rule IDS13-J. Use compatible encodings on both sides of file or network IO. Also see rule IDS10-J. Don't form strings containing partial characters.
Risk Assessment
Attempting to read a byte array containing binary data as if it were character data can produce erroneous results.
Rule | Severity | Likelihood | Remediation Cost | Priority | Level |
|---|---|---|---|---|---|
FIO11-J | low | unlikely | medium | P2 | L3 |
Related Guidelines
Bibliography