[antiguru]

Adds CompactionMemory, a ScenarioBig that measures environmentd's peak memory and compaction time while compacting table batches written by COPY INTO ... FROM S3. The data has low-cardinality text columns, which parquet dictionary-encodes, so the decoded arrow is much larger than the encoded parts the compaction memory budget counts.

The report's memory figures are taken after compaction finishes, so the scenario prints environmentd's cgroup memory.peak and compaction metrics as a COMPACTION-MEMORY log line. Run it with --other-tag=current-source to compare parameter sets on one build.

🤖 Generated with Claude Code

[def-]

Only a nit about the comment

[def-]

Doesn't every scenario measure envd memory and time?

[antiguru]

Yes, the first sentence said nothing specific to this scenario. The docstring now opens with the workload and the peak it reports, and says why the standard memory figure misses that peak.

Posted by Claude Code.

[def-]

QA LLM Review

1. MEDIUM -- The idle check counts applied chunks as completed requests

misc/python/materialize/feature_benchmark/scenarios/benchmark_main.py:544

The idle check can report zero while incremental compaction is still processing additional chunks. This lets the benchmark finish successfully before the measured work completes, understating its duration and potentially missing its memory peak or carrying compaction work into the next iteration.

Details

started increments once per request (src/persist-client/src/internal/compact.rs:327), but the incremental loop calls apply for each chunk (src/persist-client/src/internal/compact.rs:425), incrementing applied or noop each time (src/persist-client/src/internal/compact.rs:535). Incremental compaction is enabled by default. With requested=started=applied=1, this helper returns zero even while the second chunk runs. Five polls do not fix that accounting error. Extra applications from earlier iterations also mask newly started requests because these counters are cumulative.

The queue calculation separately subtracts skipped, although skipped requests return before incrementing requested (src/persist-client/src/internal/compact.rs:294), hiding queued work too. Use a completion signal or gauge that tracks whole requests through their actual completion, and exclude skipped from the queue balance.