ERR02-A. Avoid in-band error indicators

Avoid in-band error indicators while designing interfaces. This practice is commonly used by C library functions but is not recommended. One example from the C standard of a troublesome in-band error indicator is EOF (see FIO34-C. Use int to capture the return value of character IO functions and FIO35-C. Use feof() and ferror() to detect end-of-file and file errors when sizeof(int) == sizeof(char)). Another problematic use of in-band error indicators from the C standard involving the size_t and time_t types is described by MSC31-C. Ensure that return values are compared against the proper type.

Non-Compliant Code Example

This specific non-compliant code example is from the Linux Kernel Mailing List archive site at http://lkml.org/ although similar examples are common.

int i;
ssize_t count = 0;

for (i = 0; i < 9; ++i)
  count += sprintf(buf + count, "%02x ", ((u8 *)&slreg_num)[i]);
count += sprintf(buf + count, "\n");

The sprintf() function returns the number of characters written in the array, not counting the terminating null character. This number is frequently added to an existing counter to keep track of the location of the index into the array. However, the call to sprintf() can (and will) return -1 on error conditions such as an encoding error. If this happens on the first call (which is likely), the count variable, already at zero, is decremented. If this index is subsequently used, it will result in an out-of-bounds read or write.

Compliant Solution

This compliant solution shows the redesigned API for sprintf() from the CERT managed string library [[Burch 06]].

errno_t sprintf_m(string_m buf, const string_m fmt, int *count, ...);

The sprintf_m() API separates out the return status of the function from information about the number of characters written. In this case, *count is set to the number of characters written in buf while the return value indicates the return status. Returning the status as the return value of the function increases the likelihood that a programmer will check the return status of the function.

One can thus amend the previous code example thus:

int i;
ssize_t count = 0;
errno_t err;

for (i = 0; i < 9; ++i) {
  if ((err = sprintf_m( buf + count, "%02x ", &count,
                        ((u8 *)&slreg_num)[i])) != 0) {
    /* handle print error */
  }
}
if ((err = sprintf_m( buf + count, "%02x ", &count,
                      ((u8 *)&slreg_num)[i]) ) != 0) {
  /* handle print error */
}

Exceptions

ERR02-EX1: Null pointers are another example of an in-band error indicator. Use of the null pointers is not quite as bad because it is supported for by the language. According to C99 Section 6.3.2.3, "Pointers":

If a null pointer constant is converted to a pointer type, the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function.

Risk Analysis

The risk of using in-band error indicators is difficult to quantify, and is consequently given as low. However, if the use of in-band error indicators results in programmers failing or incorrectly checking status code, the consequences can be more severe.

Related Vulnerabilities

Search for vulnerabilities resulting from the violation of this rule on the CERT website.

References

[[Burch 06]]
[[ISO/IEC 9899-1999]] Section 6.2.4, "Storage durations of objects," and Section 7.20.3, "Memory management functions"
[[ISO/IEC PDTR 24772]] "NZN Returning error status"

ERR01-A. Use ferror() rather than errno to check for FILE stream errors      13. Error Handling (ERR)       ERR30-C. Set errno to zero before calling a function, and use it only after the function returns a value indicating failure