[Bug]: 0.9 table optimizing status metrics stuck at IDLE after AMORO-3747 #4434

What happened?

After upgrading AMS from 0.8.x to 0.9.x, table optimizing status duration / in-status gauges are stuck at IDLE.

The Tables page correctly shows tables in major / minor / pending (for example t_entity_data major for 44 minutes). But Prometheus still reports:

amoro_table_optimizing_status_idle_duration_mills{...} 1.56e9   # ~18 days == AMS uptime
amoro_table_optimizing_status_executing_duration_mills{...} 0
amoro_table_optimizing_status_pending_duration_mills{...} 0
amoro_table_optimizing_status_in_executing{...} 0

So executing_duration cannot be used to alert on “table is optimizing”. In 0.8.x these gauges followed the real table status.

Expected

When table status is MINOR/MAJOR/FULL, executing_duration_mills / in_executing should be non-zero. When PENDING, pending_duration_mills / in_pending should be non-zero. Idle duration should reset when leaving IDLE.

Affects Versions

0.9.0-incubating / 0.9.x (works on 0.8.x)

What table formats are you seeing the problem on?

Iceberg

What engines are you seeing the problem on?

AMS

How to reproduce

1. Run AMS 0.9, register Iceberg tables, start optimizers.
2. Wait until the Tables page shows a table in minor / major / pending.
3. Scrape AMS metrics:

curl -s localhost:9090/metrics | grep table_optimizing_status

4. Observe idle_duration_mills ≈ process uptime and executing_duration_mills=0 even for tables that are clearly optimizing.

Relevant log output

Scraped while the UI showed `t_entity_data` major ~44m, `social_account` minor ~18m, `event_record_*` minor ~6m:


amoro_table_optimizing_status_idle_duration_mills{catalog="saas_cluster8",database="cdp_entity",table="t_entity_data",group="k8s-amoro",} 1.568392645E9
amoro_table_optimizing_status_executing_duration_mills{catalog="saas_cluster8",database="cdp_entity",table="t_entity_data",group="k8s-amoro",} 0.0
amoro_table_optimizing_status_idle_duration_mills{catalog="saas_cluster8",database="beacon_olap",table="social_account",group="k8s-amoro",} 1.568392638E9
amoro_table_optimizing_status_executing_duration_mills{catalog="saas_cluster8",database="beacon_olap",table="social_account",group="k8s-amoro",} 0.0

Anything else

This is a regression from AMORO-3747 / #3748 (new TableRuntime storage). Dashboard still reads table_runtime.status_code, so the UI is correct.

0.8 updated metrics on every status change:

private void updateOptimizingStatus(OptimizingStatus status) {
  this.optimizingStatus = status;
  this.currentStatusStartTime = System.currentTimeMillis();
  this.optimizingMetrics.statusChanged(status, currentStatusStartTime);
}

#3748 removed updateOptimizingStatus. TableOptimizingMetrics.statusChanged(...) still exists, but nothing in the tree calls it anymore. The gauge stays at the constructor default OptimizingStatus.IDLE and statusSetTimestamp = now at AMS start.

Suggested fix: after store().updateStatusCode(...).commit(), call

optimizingMetrics.statusChanged(getOptimizingStatus(), /* status_code_update_time */);

Also initialize from persisted status_code + status_code_update_time when the runtime is created, so a restart does not show “idle since boot” while the table is still MINOR.

This is separate from process rows stuck RUNNING after self-optimizing.enabled=false (dashboard process list vs table header). Here the table header is already major/minor; only Prometheus gauges are stale.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct