autovacuum launcher crash: assert in pgstat_count_io_op (IOOP_EXTEND on pg_database's VM)

First seen: 2026-05-31 04:36:45+00:00 · Messages: 5 · Participants: 3

Latest Update

2026-06-04 · claude-opus-4-6

Incremental Analysis: Root Cause Confirmed, Fix Strategy Agreed

Summary of Progress

The thread has moved from initial bug report to full root cause identification, a proposed patch, expert review, and consensus on a fix strategy — all within 48 hours.

Root Cause Confirmed (Two Interacting Commits)

Ewan (kdbase.hack) identified the precise two-commit interaction causing the bug:

  1. 4f7ecca84dd — Added unconditional (extending) visibilitymap_pin() in the on-access prune path, meaning any scan that triggers pruning may now extend the VM fork.
  2. 378a216187a — Made INSERT set pd_prune_xid, so on-access pruning now fires on insert-mostly catalogs like pg_database.

The race condition: the autovacuum launcher scans pg_database via get_database_list() with a catalog scan. On a full, prunable page, heap_page_prune_opt() calls visibilitymap_pin() which extends the VM fork. The launcher backend type is not permitted to do IOOP_EXTEND in pgstat_tracks_io_op(), triggering the assertion. The window is narrow because any regular backend or autovacuum worker scanning pg_database first would create the fork harmlessly.

Proposed Patch (Ewan's approach)

Ewan submitted a patch that conditionally avoids extending the VM fork: for non-read-only scans, it uses visibilitymap_get_status() (which only pins existing pages without extending) instead of visibilitymap_pin() (which extends). The rationale: if the VM doesn't cover the page yet, there's no corruption to detect, and the fork creation can be deferred to the next VACUUM.

Melanie's Authoritative Response and Fix Strategy

Melanie Plageman (author of the affected code) provided a detailed response:

  1. For PG19 (backport fix): Simply relax pgstat_tracks_io_op() to allow IOOP_EXTEND for B_AUTOVAC_LAUNCHER. The specific change removes the autovacuum launcher from the list of backend types that are blocked from extending:

    -   if ((bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
-        bktype == B_CHECKPOINTER) && io_op == IOOP_EXTEND)
+   if ((bktype == B_BG_WRITER || bktype == B_CHECKPOINTER) &&
+       io_op == IOOP_EXTEND)

  2. For PG20 (proper fix): Add a flags argument to table_beginscan_catalog() to properly communicate scan intent, combined with the pgstat_tracks_io_op() change.

  3. Melanie's hesitation on Ewan's patch: She acknowledged it's not wrong but expressed discomfort that it breaks the invariant that the VM page is always pinned and passed to heap_page_prune_and_freeze(). The logic relies on implicit knowledge that visibilitymap_pin() extends while visibilitymap_get_status() doesn't — making the code harder to reason about.

  4. Design context: SO_HINT_REL_READ_ONLY is only a performance hint, not a guarantee. Melanie noted they briefly discussed excluding catalog scans in the original thread but didn't pursue it.

Consensus Reached

Ewan agreed to drop his patch in favor of Melanie's simpler approach: relax the assertion for PG19, do the proper architectural fix in PG20.

History (1 prior analysis)
2026-06-01 · claude-opus-4-6

Autovacuum Launcher Crash: Assert in pgstat_count_io_op (IOOP_EXTEND on pg_database's VM)

Technical Problem

This thread reports a crash in the autovacuum launcher triggered by an assertion failure in pgstat_count_io_op when performing an IOOP_EXTEND operation on the visibility map (VM) of pg_database. The crash was discovered during stress testing on master (commit e2b35735b00) with assertions enabled, using a workload involving rapid creation and dropping of databases in a tight loop.

Root Cause Analysis

The core issue lies at the intersection of several PostgreSQL subsystems:

  1. Visibility Map Extension: When autovacuum processes pg_database (a shared catalog), it may need to extend the visibility map. The VM is a fork of the relation that tracks which pages are known to contain only tuples visible to all transactions.

  2. I/O Statistics Tracking: The pgstat_count_io_op function tracks I/O operations for the statistics subsystem. It contains assertions that validate the combination of I/O operation type, backend type, and I/O object being operated on.

  3. The Assertion Failure: The autovacuum launcher (as distinct from autovacuum workers) is apparently performing or triggering an I/O operation (IOOP_EXTEND on a relation fork) that the statistics subsystem does not expect from that particular backend type. The pgstat I/O statistics infrastructure has a matrix of valid combinations of (BackendType, IOObject, IOContext, IOOp), and the autovacuum launcher extending a VM file likely falls outside the expected valid combinations.

Architectural Significance

This bug exposes a subtle design tension:

  • Autovacuum launcher vs. worker distinction: The launcher is primarily responsible for scheduling and spawning workers, but it also performs lightweight vacuuming of shared catalogs (like pg_database) directly. This dual role means the launcher can perform I/O operations that the statistics infrastructure may not have accounted for.

  • Shared catalog vacuuming: pg_database is a shared catalog, and vacuuming it in the launcher process is a long-standing optimization to avoid spawning a full worker for a typically small table. However, if the table's VM needs extension (e.g., after many databases are created and dropped causing pg_database to grow), this triggers a code path that wasn't properly validated in the IO stats assertion matrix.

  • Statistics subsystem completeness: The pgstat I/O tracking introduced relatively recently (PG16+) aims to comprehensively track all I/O operations by backend type. This bug suggests the matrix of valid (backend_type, io_object, io_context, io_op) tuples is incomplete for the autovacuum launcher's catalog maintenance path.

Proposed Solutions

Given this is a single-message thread (initial bug report), no formal patches have been proposed yet. However, the likely fixes would include:

  1. Expand the IO stats valid combination matrix: Add IOOP_EXTEND as a valid operation for the autovacuum launcher backend type when operating on relation forks (specifically VM/FSM forks of shared catalogs).

  2. Separate the launcher's vacuum path: Potentially refactor so that shared catalog vacuuming is delegated to a worker rather than performed in the launcher, though this would be a larger architectural change and may not be desirable for performance reasons.

  3. Relax the assertion: Make the assertion less strict for edge cases, though this is the least desirable fix as it reduces the diagnostic value of the assertion framework.

Key Technical Context

  • The visibility map extension during vacuum is triggered when visibilitymap_set() needs to mark a page as all-visible but the VM doesn't yet have enough pages to cover the heap's current size.
  • The stress test of create/drop database in a tight loop causes pg_database to grow (new tuples for each database), and after drops, vacuum marks pages all-visible, potentially needing VM extension.
  • This is an assertion-only failure — production builds without assertions would not crash but would have incorrect I/O statistics tracking (silent data corruption in stats).