Incremental Update: Concrete Benchmarks, WIP Patches, and Protocol Design Discussion

Key Developments

1. Nathan posted concrete benchmark results at 1M scale comparing Jelte's CreateOneShotCachedPlan patch against HEAD's PQnfn() path:

   • HEAD (PQnfn): 0.319 seconds
   • With Jelte's patch (PQexecParams + OneShotCachedPlan): 0.444 seconds (~39% slower than HEAD)
   • Without Jelte's patch (plain PQexecParams): 0.457 seconds (~43% slower than HEAD)

   The OneShotCachedPlan optimization provides only a marginal improvement (~3% faster than plain PQexecParams), far from closing the gap with the fast-path protocol. The FunctionCall message remains significantly faster for high-frequency LO operations.

2. Nathan posted a WIP patch set with three patches:

   • 0001: Switches frontend LO interface to use prepared statements
   • 0002: Removes PQfn() entirely
   • 0003: Alternative that switches LO interface to PQexecParams() instead of prepared statements

3. Nathan posted a second WIP patch introducing a statement deallocation notification mechanism — a new protocol message that notifies clients when prepared statements are deallocated. This directly addresses the DISCARD ALL / DEALLOCATE ALL fragility problem identified in the prior analysis.

4. The protocol notification work has been spun off to a separate thread (https://postgr.es/m/ahm_4eOKkkKJ3Gds%40nathan), indicating it's being treated as its own feature rather than a dependency of PQfn() removal.

Agreed Short-Term Plan

Nathan and Jacob converged on a pragmatic approach:

• Keep PQnfn() internally for now — no rush to eliminate it
• Retire PQfn() (the public symbol) in v20
• Eventually replace PQnfn() with PQexecParams() once older servers without the new protocol support have aged out
• The fallback path for older servers would use PQexecParams() (slower but functional)

Protocol Design Debate

Jacob pushed back on Nathan's notification-based protocol approach on several grounds:

• Protocol extension vs. version bump: Jacob argues that a single new message type should be advertised via a protocol extension mechanism, not a minor protocol version bump.
• Skepticism about genericizing: Jacob expressed doubt about making the notification mechanism extensible without concrete use cases beyond this one.
• Architectural concern: Jacob is "down on the notify-the-client-that-X-happened method" as a long-term solution, preferring architectural separation (contexts, multiplexed streams) over reactive notifications.
• Alternative vision: Jacob floated the idea of middleware-aware "contexts" where libpq could mark its own server-side allocations as separate from application-context operations, so DISCARD wouldn't affect them.

Alternative Approaches Considered and Rejected

Nathan enumerated ideas he'd considered and rejected:

• "Pinning" statements: Problematic for connection poolers (they need to be able to clean up all state)
• "Built-in" statements for libpq: Too libpq-centric for what's a general client problem
• Guessing based on queries: Not viable
• Handling failures from missing/wrong prepared statements: Not viable (Jacob agreed, noting idempotent retry isn't worth the complexity)

Future of PQfn(): Retiring PostgreSQL's Fast-Path Function Call Interface

Core Problem

PQfn() is PostgreSQL's "fast path" interface in libpq, which allows clients to invoke server-side functions directly via the FunctionCall protocol message (PqMsg_FunctionCall) without going through the SQL parser. This interface has been considered problematic for over two decades:

1. Marked "somewhat obsolete" in 2003 (commit efc3a25bb0) — the interface was already seen as unnecessary given that SQL function calls provide the same capability with more safety.
2. Marked "unsafe" in 2026 (commit bd48114937) — the interface was formally designated as unsafe, and a new internal-only PQnfn() function was created to serve the one remaining internal consumer: the frontend Large Object (LO) interface.

The key architectural issue is that PQfn() bypasses the SQL layer entirely, communicating via a legacy binary protocol that lacks the safety guarantees of the extended query protocol. It's an exported symbol from libpq, meaning it's part of the ABI contract, but it has essentially zero external adoption — all found references are merely language binding wrappers that expose it mechanically without actually using it in application code.

Proposed Solutions

1. Retire PQfn() in v20

Nathan Bossart proposes keeping the symbol (ABI compatibility) but having it unconditionally return an error. This is the standard PostgreSQL approach for deprecating exported library functions without breaking the dynamic linker contract.

2. Frontend LO Interface Alternatives

The more interesting architectural question is what to do with the Large Object frontend code, which is the sole real consumer of the fast-path mechanism (now via internal PQnfn()):

• Option A: Keep PQnfn() as internal function — simplest approach, preserves current performance characteristics. This is the short-term winner.
• Option B: Use prepared statements with binary params/results — follows the project's own advice to users but has two serious problems:
  • Name collisions: libpq-managed prepared statement names could collide with user-created statements.
  • State fragility: DISCARD ALL or DEALLOCATE ALL would silently break libpq's internal prepared statements.
• Option C: Use PQexecParams() — simple but measured at ~41% slowdown for LO operations (tested with 10K LO create+unlink), making it unacceptable for performance-sensitive workloads.

3. Long-term Protocol Improvements

Two forward-looking ideas emerged:

• Protocol notification for deallocated statements: Teaching the server to inform clients when prepared statements are invalidated would solve the DISCARD/DEALLOCATE fragility problem not just for libpq but for all client libraries (JDBC has the same problem, as documented in pgjdbc).
• OneShotCachedPlan for unnamed statements: Jelte Fennema-Nio mentions a WIP patch (AI-generated, unvetted) that uses CreateOneShotCachedPlan for unnamed prepared statements to bring extended protocol performance closer to simple protocol. This could reduce or eliminate the 41% penalty of the PQexecParams() approach.

Key Technical Insights

The Prepared Statement Layering Problem

The most architecturally significant insight in this thread is Jacob Champion's observation that libpq's inability to safely use prepared statements internally is a general layering violation problem that affects all libpq clients. The server's prepared statement namespace is flat and shared between the application and any library code using the same connection. There's no mechanism for libraries to "own" private prepared statements without risk of collision or unexpected deallocation.

Nathan's proposed mitigation — using a documented prefix like libpq_internal_ — is pragmatic but imperfect. It handles accidental collisions but not the DISCARD/DEALLOCATE problem.

Performance Gap Between Protocol Paths

The 41% slowdown when switching from FunctionCall to simple query protocol for LO operations reveals a meaningful performance difference between the protocol paths. The FunctionCall message skips parsing, planning, and the full executor pipeline. For high-frequency, simple operations like LO manipulation, this overhead is substantial. Jelte's suggestion to test with CreateOneShotCachedPlan is well-targeted — it would eliminate the plan caching overhead for one-shot queries while still using the extended protocol.

Documentation Strategy

The thread reveals a minor disagreement on documentation approach: Nathan leans toward leaving a stub noting the removal, while Jacob suggests wiping all mentions. Given the Debian codesearch evidence showing only mechanical wrappers (no actual application usage), complete removal seems defensible but a brief deprecation note provides better archaeology for the inevitable future developer who encounters PQfn references in legacy binding code.

Architectural Context

The PQfn() retirement is part of a broader trend in PostgreSQL of consolidating on the extended query protocol as the single recommended client-server communication path. The FunctionCall message type is a relic from the pre-SQL-function era when direct function invocation was the only way to call server-side code. Its removal simplifies the protocol surface area and reduces attack surface (the "unsafe" designation likely relates to type safety and injection concerns in the binary parameter passing).