Beautify Read Stream "Per Buffer Data" APIs
Core Problem
PostgreSQL's read stream infrastructure (introduced to support prefetching and efficient sequential/random I/O patterns) exposes a "per buffer data" mechanism that allows callers to attach metadata to each buffer request flowing through the stream. The current API requires callers to work with raw void * pointers, manually cast them, and use memcpy or direct pointer dereference to store/retrieve typed values. This pattern is error-prone, obscures intent, and reduces type safety across multiple call sites (vacuum, bitmap heap scan, etc.).
The specific API deficiencies are:
- Lack of type safety: Callers must cast
void *to their actual type, with no compile-time checking that the size or type matches what was allocated. - Verbose boilerplate: Every call site repeats the same cast-and-dereference pattern, making the code harder to read and maintain.
- Inconsistent usage patterns: Different callers handle the per-buffer-data differently, making it harder for developers to understand the intended idiom.
Proposed Solution
The patch series introduces helper macros/inline functions to wrap the raw pointer manipulation:
Patch 0001: read_stream_get_buffer_and_pointer()
Introduces a typed accessor that combines getting the next buffer from the stream with obtaining a properly-typed pointer to the per-buffer data. This eliminates the separate void-pointer retrieval and cast pattern at each call site. The BitmapHeapScanNextBlock() transition is included here for clarity.
Patch 0002: read_stream_put_value() and related helpers
Introduces a macro to store a value into the per-buffer-data slot from within the callback function. The macro uses memcpy with sizeof(value) to copy the value into the per-buffer-data pointer:
#define read_stream_put_value(stream, per_buffer_data, value) \
memcpy((per_buffer_data), &(value), sizeof(value))
The Rvalue Limitation
A notable technical issue exists with read_stream_put_value(): because it takes the address of value via &(value), it cannot accept rvalues (literal constants like false or 0). The & operator requires an lvalue in C.
The workaround discussed is to use C99 compound literals:
read_stream_put_value(stream, per_buffer_data, (bool){false});
This creates a temporary object with automatic storage duration whose address can be taken, but this idiom is unusual in PostgreSQL's codebase. The issue remains unresolved in the latest patches — callers must still pass a variable rather than a literal constant.
Alternative solutions could include:
- A
_Generic-based approach (C11, not available in PG's C99 baseline) - Separate macros for value types (
read_stream_put_bool, etc.) - Accepting the compound literal syntax as the canonical usage
Architectural Context
The read stream infrastructure is a relatively recent addition to PostgreSQL's storage layer, designed to enable kernel readahead hints and asynchronous I/O patterns for sequential scans, vacuum, and bitmap heap scans. The per-buffer-data mechanism is critical for communicating state from the prefetch callback (which decides what to read next) to the consumption site (which processes the buffer). For example, in vacuum, the callback stores whether a block was skipped due to visibility map status, and the consumer uses this to decide processing logic.
Improving this API matters because:
- More callers are expected as read_stream adoption grows across the codebase
- Type safety prevents subtle bugs where size mismatches or incorrect casts silently corrupt data
- Readability is important for an API that bridges asynchronous callback and consumption contexts
Release Timing Considerations
Melanie Plageman raises an important strategic point: since this is a developer-facing API improvement (not user-visible functionality), it can be committed early in PostgreSQL 20 development without losing value. Committing to PostgreSQL 19 would create an API that might need to change again in 20 if the rvalue limitation is addressed differently, which would be disruptive for external users of the read stream API (extensions, custom table AMs, etc.).