G
YouTube TikTok Games
Skip to content

P0 IEC-101: serial diagnostics, FC11 response correlation and safe dual-link reliability - #136

Closed
masarray wants to merge 14 commits into
masterfrom
fix/iec101-serial-startup-slow-frame-20261110
Closed

masarray wants to merge 14 commits into
masterfrom
fix/iec101-serial-startup-slow-frame-20261110

Conversation

@masarray

@masarray masarray commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

2026-11-10 P0 SICAM no-RX / FC11 correlation checkpoint — e5b6634

The owner captured two different field behaviors with SICAM A8000:

  1. An addressed FC11 status secondary after the master's GI/Class 2 (wrong service for pending FC3/FC11) despite FC9/FC11 status verification working; IEC_Test can independently see a valid FC9 no-data response and application values.
  2. On a later attempt, only repeated primary FC9 status TX with no completed RX frame, even with 3749-ms response timeout. Screenshot showed ARIEC COM4 while the prior successful IEC_Test screenshot showed COM5 — verify physical identity, not assume common adapter/port. The raw FC9 request has no COT; changing COT size cannot repair a no-response status transaction.

New code:

  • ITransportExchangeObserver is an optional boundary between IEC101 protocol exchange and serial diagnostics only: captures FC, accepted vs rejected exchange, decoded candidate count.
  • SerialByteTransport now collects actual driver RX byte totals and a bounded first-two/power-of-two no-frame/uncorrelated exchange trail, including CTS/DSR/DCD and TX bytes queued. IEC60870-SERIAL-NO-FRAME-EVIDENCE differentiates actual zero driver RX bytes vs data received but no complete FT1.2 candidate. IEC60870-SERIAL-RX-NOT-CORRELATED captures frames seen but unaccepted. No RTS/DTR auto-toggle, station reset, blind retry, or speculative profile switching.
  • IEC101-RX-OUT-OF-CONTEXT-FC11 identifies the exact observed mismatch: FC11 after FC3/FC10/FC11 must not prove GI/Class service or advance FCB, but this alone is not evidence the SICAM RTU violates IEC.
  • IEC101-NO-COMPLETE-RX-FRAMES summarizes zero complete RX candidates during a TX-only run and includes actionable safe field checks.
  • Deterministic regression: zero-RX fake transport sends FC9, never GI, captures observer's exchange evidence and finding; validator tests reject FC11 for Class 1/2/GI but accept addressed FC11 for FC9.

Reconciliation / RC independence:

  • Code branch remains based on merged public v3.6.8-rc.2 master checkpoint 4128f04e. Diff only changes IEC101 engine, tests and field FAT documentation; behind master = 0 at this checkpoint, no overlap with Frame Trace, installer/release notes.
  • Exact-HEAD Windows CI SUCCESS #42780242962: Release build, format, protocol smoke, full xUnit suite, x64 GUI candidate, IEC101 independent field toolkit, 1400-frame WPF acceptance. ARIEC60870-Passive-Monitor-P0-GUI-win-x64 is downloadable as a test artifact, not a newly published public RC.
  • Dependency review SUCCESS; repository hygiene SUCCESS; CodeQL SUCCESS at exact HEAD.
  • Keep this PR Draft pending isolated real SICAM/IEC_Test test with identical COM mapping, serial flow-control lines, and full RX evidence. Physical field behavior is not fixed/proven yet. Do not merge this physical-serial behavior into a published RC based on simulator and CI alone.

Latest reconciled checkpoint — 2026-11-10 · 2665226

Current scope: P0 IEC-101 standards-first serial reliability; unbalanced single-link + dual-link independent RTU busy handling, safe command ownership and conservative failover.

New single-link changes in this checkpoint:

  • Physical controls/clock adjustment are queued with the state and generation of the addressed, verified secondary link. The FIRST uncertain reply or DFC=1 invalidates the old command generation; an unverified or >30-second queued mutating request is rejected with IEC101-COMMAND-STALE-LINK rather than sent after recovery. Class 2/FC9 diagnostics continue; no blind operator control retry.
  • If initial clock synchronization receives DFC=1, only the UNSENT startup GI is deferred until FC9 status verifies DFC=0. Previously startup GI was silently lost; the clock adjustment that may already have reached the RTU is never automatically repeated. An ambiguously transmitted GI is also not blindly replayed.
  • Regression tests: pre-start single-link mutating control must never transmit after good FC9; DFC busy during clock sync must produce one clock sync and one deferred GI after status-only recovery.

Reconciliation: Merge commit 266522621 has parents 8ffc9c33 (P0 fixes atop old branch) and 4128f04e (current master with the public v3.6.8-rc.2, release validation documents and Frame Trace fixes). Compared with master: behind=0, 16 P0 IEC-101 engine/test/FAT files differ, no collisions with release/GUI work. No changes made to master, RC tag or published assets.

Exact-head verification: Windows CI #42770212161 SUCCESS: Release build, formatting, protocol smoke, all xUnit regression suites, Windows x64 GUI/test toolkit artifacts and 1400-frame real-WPF acceptance. Dependency review SUCCESS; repository hygiene SUCCESS; CodeQL SUCCESS (exact HEAD).

Evidence confidence: Owner reports simulator operation OK, but no documented physical SICAM 1200-bps/RTS/CTS, plug/unplug COM, or real dual-link/command FAT. Keep PR Draft, do not merge into released branch on simulator/CI proof alone. Compare against IEC_Test on an isolated, authorized physical test link; no production RTU resets or unapproved control commands.


Field regression

Owner reports Serial write timeout in History and cases where GI is TX but no RTU RX, while IEC_Test can communicate on the same RTU. A successful local COM open / TX call does not prove an electrical send or valid IEC101 secondary response.

Audited root causes and corrections

  • Single-link IEC101 ignored configured StartupDelayMs; dual-link probed before the second COM had settled. Both now wait the configured startup interval before first FT1.2 TX, cancellably.
  • Active FT1.2 reader truncated slow frames at the initial response timeout. It now allows one bounded baud-aware completion deadline after a physical 10 or 68 start byte, separate fixed/variable budgets; no-byte silence remains bounded.
  • Local write timeout previously used max(2000ms, ResponseTimeoutMs) even for up to 262-byte FT1.2 at 1200 baud, where full 8E2 wire time is up to ~2620ms. The driver write limit now uses a bounded baud-aware budget independent of remote response time (3620ms at 1200 8E2). This is not a blind retry and does not bypass hardware flow control.
  • Serial write-timeout diagnostic now preserves attempted TX size, elapsed write time, write limit, configured handshake/RTS/DTR and adapter CTS/DSR/DCD + queued TX bytes when available. Only actual live error evidence can distinguish CTS gating from USB/driver effects; CTS=0 is not causal proof with Handshake=None.
  • Protocol validator bug: E5 was accepted as an FC9 Request Link Status response and could mark the link verified / enable GI without actual secondary status evidence. Both single and dual unbalanced Masters now reject IEC101-RX-UNEXPECTED-E5-STATUS during FC9; contextual E5 after other requests preserved. Existing checksum, PRM, LA and FCB safety remain.
  • New and updated regression tests exercise fake COM startup, slow 1200-bps variable response, silence, truncated fixed response, E5 single/dual no false GI, and write budget. No production device settings or auto-transmission/resets changed.

Verification

  • Base master: 225d777a9fe0396718192d3ab880b88fb7c679d6, no open competing PRs at audit.
  • Prior intermediate HEAD 38ac3851 passed full Windows CI after fixing two old fake-COM fixture assumptions. Previous head 415f28bb passed complete Windows CI, xUnit, smoke, CodeQL, dependency and hygiene. The 065b6f47 intermediate CI exposed 9 pre-existing fake-RTU Table 10 violations (its build/format/smoke passed); current HEAD e0f74ab3 passed all Windows CI stages and all xUnit, including GUI packaging and WPF acceptance; CodeQL pending at this update.
  • Physical SICAM/IEC_Test A/B FAT not performed; leave this as draft until exact-head CI and isolated hardware tests are reviewed.
  • Do not merge just to green an RC. Production connections/wiring unchanged.

Hardware discrimination steps

Compare IEC_Test and ARIEC profiles (COM exclusive ownership; baud, 8E1/8E2, stop/parity, flow/RTS/DTR, 1/2-byte LA, CA/COT/IOA, unbalanced role). Reproduce with one isolated link and collect IEC60870-SERIAL-OPEN-PROFILE, IEC60870-TRANSPORT-WRITE-TIMEOUT and all IEC101-RX-* rows plus full TX/RX HEX. Distinguish driver TX completion, verified PRM=0 FC11 link status and actual GI data/ACTCON/ACTTERM. No RTS/CTS or cable changes on a production SCADA link without approval.

IEC-101 Table 10 context-aware validation — commit 065b6f47

  • IEC 60870-5-101 Amendment 2 / Table 10 request-response matrix added to the same active single/A-B validator: confirmed FC0/1/3/(configured legacy 7) -> FC0/FC1, FC8/9 status -> FC11, class FC10/11 -> FC8/9, contextual E5 only for confirm/no-data.
  • Prevents false link health from addressed but out-of-context FC0/8/9 during FC9, or FC11 during a command/class poll. Busy FC11/DFC=1 and FC14/15 link errors are retained as rejected evidence, not successful health proof. Raw A2 prohibited from active IEC101 positive/negative transaction acknowledgment.
  • Added positive/negative source-backed vectors and fixed a fake-SICAM fixture that previously replied FC11 status to every request. The generic FT1.2 parser and passive raw preservation remain intact.
  • IEC101 FC7 remains a pre-existing opt-in compatibility path, not a universal Table 10-conforming operation; follow-up policy/profile gating requires independent verification. No active balanced mode, no NUC compliance claim, no hardware FAT.
  • Exact-head Windows run: https://github.com/masarray/ARIEC60870/actions/runs/42760589144 (check final status before accepting).

Simulator/dual-link test architecture correction — commit e0f74ab3

  • CI of the stricter pending-service validator exposed incorrect built-in simulated RTU FC9 status responses (FC0 instead of FC11) and three negative-GI/sequence/CA-learning fake transports using FC9 no-data for FC9 link status. Strict validation was not relaxed.
  • Fixed simulator FC9->FC11 and dual fake RTUs. Negative TI100 GI COT is now queued after FC3 link-layer FC0 ACK and delivered via actual FC10 Class 1 FC8 user data; the production dual GI lifecycle reads that real negative COT from its Class 1 drain before bounded group fallback, rather than requiring a prohibited direct FC8 response to FC3.
  • Current full Windows CI: https://github.com/masarray/ARIEC60870/actions/runs/42761318559 — full CI success (Release build, format, protocol smoke, all xUnit, Windows x64 GUI/toolkit packages, 1400-frame WPF acceptance). Dependency review and hygiene success. CodeQL tracked separately; physical SICAM A/B FAT still required.
  • No RTS/CTS/profile autodetection changes, hardware FAT, master merge or RC republication. This is standard-aware response validation, not proof all RTUs are supported.

P0 follow-up: standards-based remote DFC busy recovery (2026-11-10)

Exact reviewed HEAD: b6cda620db6bf23a6c01a1ccb68afa23444079e3

Protocol architecture / safety

  • A checksum-valid, addressed FC11 with DFC=1 during FC9 is recognized as observed RTU busy (IEC101-RX-STATUS-BUSY), not remote silence, positive link readiness or FCB progress. This follows IEC 60870-5-101 unbalanced flow-control procedures; it is independent of host COM RTS/CTS.
  • One per-channel Iec101UnbalancedBusyGate centralizes validated DFC state and a bounded status-only FC9 recheck interval (BusyBackoffMs, clamp 100..30000 ms). E5 has no DFC and cannot clear the gate. On a valid addressed DFC=0 reply, Class 1/2 polling and queued commands may resume.
  • Both single and redundant active channels avoid extra GI, commands, class drain, group fallback and post-GI sweep while busy. Standby status supervision remains independent and bounded. A DFC=1 standby cannot be selected as healthy recovery/failover evidence; busy alone does not cause failover.
  • First link-ready proof now requires an addressed, integrity-valid secondary (PRM=0, DFC=0), not only E5 after an optional startup FC0/FC7 reset. This prevents bypassing FC9 and unverified startup GI.
  • Never autonomously modify serial handshake, automatically reset a production RTU or blindly replay an uncertain control command after DFC.

Deterministic regression evidence

Remaining non-negotiable gate: Isolated SICAM A8000/known-good IEC_Test A/B FAT including DFC=1 timing, 1200bps timing, RTS/CTS and repeated COM reopen/USB disruption, using real TX/RX traces. CI does not demonstrate physical IEC-101 interoperability. Keep PR Draft, do not merge or change the public RC until field validation/release reconciliation.

P0 follow-up: safe dual-link command ownership and fresh standby validation — 2026-11-10

Exact HEAD: a278070a15e830658989fcd33542d558b7f09dd5

  • The dual-link scheduler first proves active health; if the active RTU is unverified or has timed out, only status FC9 probes and separately supervised standby rescue are permitted. FC3/command/GI/Class 1/2 are not dispatched while active readiness is missing. Busy DFC=1 remains governed by the earlier bounded status-only recovery gate.
  • A queued mutating request (Single/Double/Regulating/Setpoint or ClockSync) records its verified active owner at enqueue time. It is rejected with IEC101-DUAL-COMMAND-STALE-OWNER if queued without verified ownership, after owner A↔B changes, or after 30s. No automatic replay of a stale physical control onto the new active path. Read and GI remain non-mutating but still require validated active readiness.
  • Standby rescue requires both the failure threshold and a recent, failure-free secondary response. The failover controller may perform one bounded FC9 pre-promotion proof if the previously healthy standby has stale evidence; rejecting a promotion no longer marks the application image stale.
  • New deterministic fake-serial regressions: unverified active plus queued physical command, healthy standby takeover without command replay, and standby that answers once then goes silent. Existing IEC 101/103/104 tests remain.
  • Exact-head full Windows CI SUCCESS: https://github.com/masarray/ARIEC60870/actions/runs/42767364569 — Release build, format, protocol smoke, all xUnit, x64 GUI, independent 101 field toolkit, WPF viewport 1400-frame acceptance.
  • Dependency review SUCCESS: https://github.com/masarray/ARIEC60870/actions/runs/42767364998 ; repository hygiene SUCCESS: https://github.com/masarray/ARIEC60870/actions/runs/42767364573 ; CodeQL: https://github.com/masarray/ARIEC60870/actions/runs/42767364561 .
  • master advanced independently with Frame Trace/GUI work; there are no overlapping files in the current diff. Do not overwrite those changes in the RC.
  • Not yet field-qualified: physical SICAM A8000/IEC_Test A/B test, USB COM unplug/reopen and live control safety FAT. Keep PR Draft and do not republish the public RC on CI evidence alone.

@masarray masarray changed the title P0 IEC-101 COM startup settling and 1200-bps FT1.2 reception deadline P0 IEC-101: COM write timeout diagnostics, FC9 link proof, 1200-bps reliability Nov 10, 2026
@masarray masarray changed the title P0 IEC-101: COM write timeout diagnostics, FC9 link proof, 1200-bps reliability P0 IEC-101: serial reliability, remote DFC recovery and addressed link proof Nov 10, 2026
@masarray masarray changed the title P0 IEC-101: serial reliability, remote DFC recovery and addressed link proof P0 IEC-101: serial reliability, DFC handling and safe dual-link command/failover Nov 10, 2026
@masarray masarray changed the title P0 IEC-101: serial reliability, DFC handling and safe dual-link command/failover P0 IEC-101: serial reliability, safe command recovery and RC .2 reconciliation Nov 10, 2026

Copy link
Copy Markdown
Owner Author

Canonical IEC-101 P0 reliability execution plan (2026-11-11)

The owner has approved a detailed shared transaction engine + serial resilience + interoperability implementation strategy. Tracking and proposed architecture are now committed for review in documentation PR #144, with the milestone issue #143, implementation plan and 43-case verification matrix.

Coordination for this existing draft PR #136: preserve its tested FC-specific response validation, DFC remote-busy gate, baud-aware deadlines, safe command ownership and A/B proof. First reconcile it with latest master (at plan review, 3 commits behind). Before treating IEC-101 as software complete, add red cases for FC1 NACK, delayed old FC11, partial 0x68/E5 recovery, FCB-safe read retry, no ambiguous control retransmission and serial RX/COM lifecycle, then unify single/A-B behavior under one per-link transaction authority.

Release boundary: public v3.6.8-rc.3 does not contain draft #136; simulator/CI success is not a real SICAM FAT. Do not merge a physical control or serial recovery claim merely to advance the public RC. All actual progress should be reported by IEC101-Rxx requirement ID, fixture case ID, exact Git SHA and explicit software vs physical proof.

This is coordination/documentation only, not a claim that the listed runtime fixes are done.

masarray added a commit that referenced this pull request Nov 10, 2026
M1 IEC101-R01–R04: migrate single and dual A/B to one admitted exchange, canonical secondary correlation, FCB completion rule, no automatic mutating replay and immutable TX per exchange. M2 IEC101-R05: persistent RX framer per COM epoch for single and dual. Add negative, busy, NACK, stale epoch and dual-parity fixtures. Refs #143; preserves draft #136.

Copy link
Copy Markdown
Owner Author

Reconciliation handoff — 2026-11-11

This draft #136 is superseded by integrated draft PR #145. Verified Git ancestry: comparing #136 head e5b6634ac2bd92312f633c7a779e8da62f5bc61b to #145 head 1a2eb88cb753da202e85faa72cf5ca0997a53614 returns 0 behind, proving the full #136 commit lineage is contained in #145, rather than being abandoned. #145 adds shared per-link transaction/FCB processing, COM physical-error recovery, independent A/B ownership and its 980/980 xUnit / 19/19 smoke software verification. Current master also includes active IEC-101 FC0 reset-on-connect by default via merged #163; #145 has reconciled it.

Closing this obsolete draft PR only reduces duplicate/unsafe merge risk from the old base. No code is deleted, and no P0 reliability completeness / production FAT is asserted. Follow-up work and independent physical 1200-bps SOE, late FC11, GI/SBO and serial unplug proof remain with #145. The SICAM A8000 startup connectivity symptom itself is user-confirmed resolved by configuring FC0. Issue #143 has been closed by the repo owner; do not confuse that closure with full vendor interoperability proof.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant