ERR00-A. Detect errors by checking return values

According to the Question 20.4 of C-FAQ

In general, you should detect errors by checking return values, and use errno only to distinguish among the various causes of an error, such as "File not found" or "Permission denied". (Typically, you use perror or strerror to print these discriminating error messages.) It's only necessary to detect errors with errno when a function does not have a unique, unambiguous, out-of-band error return (i.e. because all of its possible return values are valid; one example is atoi). In these cases (and in these cases only; check the documentation to be sure whether a function allows this), you can detect errors by setting errno to 0, calling the function, then testing errno. (Setting errno to 0 first is important, as no library function ever does that for you.)

To make error messages useful, they should include all relevant information. Besides the strerror text derived from errno, it may also be appropriate to print the name of the program, the operation which failed (preferably in terms which will be meaningful to the user), the name of the file for which the operation failed, and, if some input file (script or source file) is being read, the name and current line number of that file.

Non-Compliant Code Example (Memory Management)

This example, taken from [[MEM32-C. Detect and handle critical memory allocation errors]] demonstrates why checking the return value of memory allocation routines is critical. The buffer input_string is copied into dynamically allocated memory referenced by str. However, the result of malloc() is not checked before str is referenced. Consequently, if malloc() fails, the program will abnormally terminate.

/* ... */
size_t size = strlen(input_string);
if (size == SIZE_MAX) {
  /* Handle Error */
}
str = malloc(size+1);
strcpy(str, input_string);
/* ... */
free(str);

Compliant Solution (Memory Management)

Upon failure, the malloc() function returns NULL. Failing to detect and properly handle this error condition appropriately can lead to abnormal and abrupt program termination.

/* ... */
size_t size = strlen(input_string);
if (size == SIZE_MAX) {
  /* Handle Error */
}
str = malloc(size+1);
if (str == NULL) {
  /* Handle Allocation Error */
}
strcpy(str, input_string);
/* ... */
free(str);

Non-Compliant Code Example (File Operations)

In this example, fopen() is used to open a file for reading. If fopen() is unable to open the file it returns a NULL pointer. Failing to detect and properly handle this error condition appropriately can lead to abnormal and abrupt program termination.

FILE *fptr = fopen("MyFile.txt","r");

Compliant Solution (File Operations)

To correct this example, the return value of fopen() should be checked for NULL.

FILE *fptr = fopen("MyFile.txt","r");
if (fptr == NULL) {
   /* Handle error condition */
}

This example also applies to rule [[FIO32-C. Detect and handle file operation errors]].

Risk Analysis

Failing to detect error condition can result in unexpected program behavior, and possibly abnormal program termination resulting in a denial-of-service condition.

Search for vulnerabilities resulting from the violation of this rule on the CERT website.

References

[[Horton 90]] Section 11 p. 168, Section 14 p. 254
[[ISO/IEC 9899-1999]] Sections 7.1.4, 7.9.10.4, and 7.11.6.2
[[Koenig 89]] Section 5.4 p. 73
[[Summit 05]] C-FAQ Question 20.4