G
YouTube TikTok Games
Skip to content

chore(tools): classify build and test-helper diffs and warn on missing doc sync - #1625

Draft
VLukiantsev wants to merge 2 commits into
mainfrom
chore/diff-size-and-doc-sync-checks
Draft

VLukiantsev wants to merge 2 commits into
mainfrom
chore/diff-size-and-doc-sync-checks

Conversation

@VLukiantsev

Copy link
Copy Markdown
Collaborator

Summary

tools/analyze_diff_size.py now counts Go-style test-helper packages as tests. It also reports Bazel build files as their own build_lines metric, so neither inflates the production-code split threshold anymore. The tool also adds a non-blocking doc-sync warning, and the agent rules now list the specific docs to check when behavior changes.

Motivation & Context

  • Sizing accuracy: The tool counted helper packages like internal/prototest/ and */experimenttest/ as production code, along with BUILD.bazel and *.bzl edits. That pushed focused PRs toward the 400-line production threshold and caused split-verification audits that weren't needed.
  • Documentation drift: Root AGENTS.md already has a Documentation Alignment principle, but recent PRs still changed behavior and APIs without updating flow docs or READMEs. Following the rule-authoring control ladder, the strongest cheap control is a tool-level check (rung 3). It is backed by named doc locations in the existing Documentation Alignment bullet (an amendment, not a new rule file) and cross-linked from the pre-PR and review checklists.

Key Changes

  • tools/analyze_diff_size.py:
    • Classifies <name>test/ directory segments as tests. A small exclusion list keeps English words such as latest/, attest/ and contest/ in prod.
    • Adds a build category (BUILD, BUILD.bazel, *.bzl), reported as metrics.build_lines and left out of both prod and total code. BUILD matches case-sensitively, so lowercase build scripts still count as prod.
    • Adds a warnings list. It fires when production lines under api/, cmd/, deploy/, internal/ or pkg/ change and the diff has zero documentation lines. It is advisory and does not affect should_trigger_split_verifier.
    • Splits the analysis into tally_lines, exceeded_threshold_reasons and doc_sync_warnings to keep functions small.
  • tools/analyze_diff_size_test.py: covers the helper-dir positives and false positives, build classification, exclusion of build files from totals, and the warning's edge cases.
  • .agents/rules/core_engineering.md: the Documentation Alignment bullet now lists docs/flows/*.md, binary READMEs, component and profile READMEs, and docs/migrations/*.md, and points to the new warning. The sizing list also documents build definitions.
  • .agents/skills/xmc_feature_development/SKILL.md and .agents/rules/pr_review_guidelines.md: the existing pre-commit verification step and review checklist item now point to that bullet and to the warning.
  • .agents/skills/xmc_pr_split_verification/SKILL.md and .agents/rules/github_authoring_and_prs.md: the sizing tables gain the test-helper and build-definition categories.

Assumptions

  • Build definitions are excluded from total code as well as prod — they are mostly mechanical dependency wiring, not review effort — if reviewers want them in total code, it is a one-line change in analyze_diff_size.
  • [a-z0-9]+test/ with an English-word exclusion list is precise enough — git ls-files on main matches only real helper packages (prototest, securitytest, helmtest, healthprobetest, statusdetectortest, crreadertest) — a future production directory ending in test would be under-counted until added to the exclusion list.
  • Any production line under the listed roots triggers the doc-sync warning — it is advisory and authors can resolve it with a PR-body note — pure refactors also trigger it, which is noise but doesn't block anything.
  • License-header subtraction is deferred to a follow-up — it was explicitly out of scope — new files' license headers still count toward prod lines.
  • No trajectory A/B benchmark was run — diff classification and the warning are deterministic, unit-tested tool behavior, and the prose edits only amend existing bullets to reference that tool output, while testing them would need agents implementing full behavior changes end to end — if the prose cross-links don't change agent behavior, the tool warning still surfaces the gap.

Verification & Testing

  • Commands Executed: bazel test //tools:analyze_diff_size_test, ./.githooks/pre-commit (Ruff, mypy, agent doc path check, run_affected_tests.py), ./tools/analyze_diff_size.py --base-ref origin/main...
  • Coverage & Test Status: //tools:analyze_diff_size_test and //tools:run_affected_tests_test pass, and all pre-commit checks pass. Every new branch (helper dir, false-positive exclusion, build category, each warning edge case) has a dedicated test.

Related Issues / Bugs

N/A

TAG=agy
CONV=cfb1c6d7-b7c5-44e1-88a9-635a7588d712

…g doc sync

analyze_diff_size.py counted Bazel BUILD/.bzl files and Go-style
test-helper packages (prototest/, experimenttest/) as production code,
inflating prod_code_lines toward the 400-line split threshold.

- Classify `<name>test/` directories as tests, excluding English words
  such as latest/ and attest/.
- Add a `build` category (BUILD, BUILD.bazel, *.bzl), reported as
  build_lines and excluded from prod and total code.
- Emit a non-blocking `warnings` entry when production code under api/,
  cmd/, deploy/, internal/, or pkg/ changes with zero doc lines.
- List the concrete doc locations under Documentation Alignment and
  reference the warning from the feature-development pre-PR step and the
  PR review checklist.

TAG=agy
CONV=cfb1c6d7-b7c5-44e1-88a9-635a7588d712

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates agent rules, skills, and the analyze_diff_size.py tool to support a new 'Build Definitions' category (excluding Bazel build files from code totals) and introduces a 'doc-sync' warning when production code changes without accompanying documentation updates. The review feedback correctly identifies a critical runtime error in tools/analyze_diff_size.py due to the use of the unsupported inline flag subtraction syntax (?-i:...) in Python's standard re module. To resolve this, the reviewer suggests refactoring _ORDERED_CATEGORIES and classify_file to handle category-specific regex flags, ensuring case-sensitive matching for build files while keeping other categories case-insensitive.

Comment thread tools/analyze_diff_size.py Resolved
Comment thread tools/analyze_diff_size.py Resolved
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates agent rules, skills, and the analyze_diff_size.py tool to exclude build definitions (such as BUILD and *.bzl files) from code totals, categorize Go-style test-helper packages as test code, and introduce advisory warnings for production changes that lack corresponding documentation updates. The review feedback correctly identifies a critical runtime error in tools/analyze_diff_size.py caused by using the unsupported (?-i:...) inline regex modifier in Python's standard re module. The reviewer provided excellent suggestions to resolve this crash and optimize performance by pre-compiling the regular expressions at the module level with appropriate case-sensitivity flags.

Comment thread tools/analyze_diff_size.py Resolved
Comment thread tools/analyze_diff_size.py Resolved
Compile the classification patterns once at import time and compile the
Bazel build patterns case-sensitively instead of relying on an inline
`(?-i:...)` group. Behavior is unchanged; the existing build and
lowercase-build-script tests cover it.

TAG=agy
CONV=cfb1c6d7-b7c5-44e1-88a9-635a7588d712

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant