Incorrect Check of Function Return Value

Description

Incorrect Check of Function Return Value is a vulnerability that occurs when a product incorrectly validates the return value from a function, preventing it from detecting errors or exceptional conditions. Many critical functions in programming indicate success or failure through their return values, such as memory allocation functions returning NULL on failure, file operations returning error codes, or network functions returning specific status indicators. When these return values are checked using incorrect logic, wrong comparison operators, or inappropriate value ranges, the program fails to detect and handle error conditions properly. This can lead to the program continuing execution in an invalid state, potentially causing crashes, data corruption, security bypasses, or exploitable vulnerabilities.

Risk

Incorrect return value checking poses significant risks because it allows programs to continue operating under erroneous assumptions. Memory allocation failures that go undetected lead to NULL pointer dereferences and crashes. Security function failures may allow unauthorized operations to proceed. Resource acquisition failures can cause data corruption or loss. The risk is amplified because these errors often only manifest under specific conditions such as low memory, network failures, or disk full situations - conditions that may not occur during testing but are common in production. Attackers can deliberately trigger these conditions to exploit the resulting undefined behavior, potentially gaining elevated privileges, bypassing security checks, or causing denial of service.

Solution

Understand the exact return value semantics for every function call and implement appropriate validation logic. For functions returning pointers, check for NULL rather than comparing against zero or negative values. For functions returning error codes, verify the specific success and failure values documented for that function. Use compiler warnings and static analysis tools that detect incorrect return value handling. Implement wrapper functions that enforce consistent error checking patterns. Consider using languages or libraries that make error handling more explicit through exceptions or result types. Document expected return values and error handling requirements in code comments and development guidelines.

Common Consequences

Impact	Details
Availability, Integrity	Scope: Availability, Integrity An incorrect return value check can place the system in an unexpected or unstable state. This may lead to crashes, data corruption, or behaviors that compromise system availability and data integrity.

Example Code

Vulnerable Code (C)

The following code demonstrates incorrect return value checking patterns:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>

// Vulnerable: Incorrect check for malloc() return value
void process_data_vulnerable(size_t size) {
    char *buffer = malloc(size);

    // WRONG: malloc returns NULL (0) on failure, not negative
    if (buffer < 0) {  // This check is always false for pointers!
        printf("Allocation failed\n");
        return;
    }

    // If malloc failed, buffer is NULL but we proceed anyway
    strcpy(buffer, "data");  // NULL pointer dereference = crash
    free(buffer);
}

// Vulnerable: Ignoring return value entirely
void write_file_vulnerable(const char *filename, const char *data) {
    FILE *fp = fopen(filename, "w");

    // No check if fopen succeeded
    fprintf(fp, "%s", data);  // Crash if fp is NULL

    // No check if fwrite succeeded
    fwrite(data, strlen(data), 1, fp);

    fclose(fp);  // Crash if fp is NULL
}

// Vulnerable: Wrong comparison for read() return value
ssize_t read_data_vulnerable(int fd, char *buffer, size_t size) {
    ssize_t bytes_read = read(fd, buffer, size);

    // WRONG: Only checks for -1, ignores 0 (EOF) and partial reads
    if (bytes_read == -1) {
        return -1;
    }

    // Assumes buffer has 'size' bytes, but may have fewer or zero
    buffer[size] = '\0';  // May write beyond actual data
    return bytes_read;
}

// Vulnerable: Incorrect snprintf check
void format_string_vulnerable(char *dest, size_t dest_size, const char *fmt, ...) {
    va_list args;
    va_start(args, fmt);

    int result = vsnprintf(dest, dest_size, fmt, args);

    // WRONG: snprintf returns chars that would have been written
    // If result >= dest_size, output was truncated
    if (result < 0) {
        // Only checks for encoding error, not truncation
    }

    va_end(args);
    // May proceed with truncated data without knowing
}

Fixed Code (C)

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>

// Fixed: Correct NULL check for malloc()
int process_data_secure(size_t size) {
    if (size == 0) {
        return -1;  // Invalid size
    }

    char *buffer = malloc(size);

    // CORRECT: Check for NULL pointer
    if (buffer == NULL) {
        perror("Memory allocation failed");
        return -1;
    }

    // Safe to use buffer
    strncpy(buffer, "data", size - 1);
    buffer[size - 1] = '\0';

    // ... use buffer ...

    free(buffer);
    return 0;
}

// Fixed: Comprehensive file operation error checking
int write_file_secure(const char *filename, const char *data) {
    if (filename == NULL || data == NULL) {
        return -1;
    }

    FILE *fp = fopen(filename, "w");

    // Check if file opened successfully
    if (fp == NULL) {
        perror("Failed to open file");
        return -1;
    }

    size_t data_len = strlen(data);
    size_t written = fwrite(data, 1, data_len, fp);

    // Check if write was complete
    if (written != data_len) {
        if (ferror(fp)) {
            perror("Write error");
        }
        fclose(fp);
        return -1;
    }

    // Check fclose return value (catches delayed write errors)
    if (fclose(fp) != 0) {
        perror("Failed to close file");
        return -1;
    }

    return 0;
}

// Fixed: Complete read() error handling
ssize_t read_data_secure(int fd, char *buffer, size_t size) {
    if (buffer == NULL || size == 0) {
        return -1;
    }

    size_t total_read = 0;

    while (total_read < size) {
        ssize_t bytes_read = read(fd, buffer + total_read, size - total_read);

        if (bytes_read < 0) {
            if (errno == EINTR) {
                continue;  // Interrupted, retry
            }
            perror("Read error");
            return -1;
        }

        if (bytes_read == 0) {
            // EOF reached
            break;
        }

        total_read += bytes_read;
    }

    // Null-terminate only the data actually read
    if (total_read < size) {
        buffer[total_read] = '\0';
    }

    return total_read;
}

// Fixed: Complete snprintf error handling
int format_string_secure(char *dest, size_t dest_size, const char *fmt, ...) {
    if (dest == NULL || dest_size == 0 || fmt == NULL) {
        return -1;
    }

    va_list args;
    va_start(args, fmt);

    int result = vsnprintf(dest, dest_size, fmt, args);

    va_end(args);

    if (result < 0) {
        // Encoding error
        dest[0] = '\0';
        return -1;
    }

    if ((size_t)result >= dest_size) {
        // Output was truncated
        // Depending on requirements: return error or just warn
        return -2;  // Indicate truncation
    }

    return result;  // Success - return chars written
}

The fixes implement proper NULL checks for pointer returns, verify operation success/failure correctly, handle partial operations, and check for truncation in formatting functions.

---

Exploited in the Wild

Squid Web Proxy Vulnerability (Squid, 2023)

CVE-2023-49286 demonstrated how incorrect return value checking in the Squid web caching proxy led to a chain of vulnerabilities. A function failed to properly validate its return value, which subsequently triggered an unreachable assertion (CWE-617), resulting in a denial of service condition. The vulnerability affected multiple Squid versions and required urgent patching.

• https://nvd.nist.gov/vuln/detail/CVE-2023-49286

OpenSSL Memory Allocation Failures (OpenSSL, Multiple CVEs)

Multiple OpenSSL vulnerabilities have resulted from incorrect handling of memory allocation failures. When malloc() returned NULL under memory pressure, the code continued execution with null pointers, leading to crashes and potential information disclosure. These patterns led to significant security improvements in OpenSSL's error handling.

• https://www.openssl.org/news/vulnerabilities.html

Linux Kernel NULL Pointer Dereferences (Linux Kernel, Ongoing)

The Linux kernel has experienced numerous vulnerabilities from incorrect return value checking, particularly around memory allocation. Attackers trigger low-memory conditions to cause allocation failures that result in NULL pointer dereferences, which on some systems can be exploited for privilege escalation when the NULL page is mappable.

• https://lwn.net/Articles/342330/

---

Tools to Test/Exploit

• Clang Static Analyzer — Detects incorrect return value checking patterns including unchecked malloc returns and improper error code handling.

• Coverity — Commercial static analysis tool with specific checkers for return value validation errors.

• cppcheck — Open-source static analyzer that identifies common return value checking mistakes in C/C++ code.

---

CVE Examples

• CVE-2023-49286 — Squid web proxy incorrect return value check led to assertion failure and denial of service.

• CVE-2019-16707 — Hunspell library failed to check return values from memory allocation, causing crashes.

• CVE-2014-1491 — Firefox/NSS incorrect return value check allowed invalid SSL certificate acceptance.

---

References

1. MITRE Corporation. "CWE-253: Incorrect Check of Function Return Value." Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/253.html

2. CERT C Coding Standard. "ERR33-C. Detect and handle standard library errors." https://wiki.sei.cmu.edu/confluence/display/c/ERR33-C.+Detect+and+handle+standard+library+errors

3. Microsoft. "Error Handling in C." https://docs.microsoft.com/en-us/cpp/c-runtime-library/error-handling-crt