G
YouTube TikTok Games
Skip to content

Widespread perf regressions due to RPO layout #102763

New issue
New issue
Closed
Closed
Widespread perf regressions due to RPO layout#102763
Assignees
amanasifkhalid
Labels
area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
Milestone
10.0.0

Description

@EgorBo
EgorBo
opened on May 28, 2024

Looks like #102343 has a lot more regressions than improvements. cc @amanasifkhalid

Regressions:

Improvements:

NOTE: use Test report links, it looks like "all time history" and the images are a bit out of date.

Activity

  1. ghost added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on May 28, 2024
  2. dotnet-policy-service
    added
    untriagedNew issue has not been triaged by the area owner
    on May 28, 2024
  3. EgorBo
    assigned
    amanasifkhalid
    on May 28, 2024
  4. EgorBo
    added
    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
    and removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on May 28, 2024
  5. EgorBo
    added this to the 9.0.0 milestone on May 28, 2024

  6. dotnet-policy-service commented on May 28, 2024

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on May 28, 2024 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
    See info in area-owners.md if you want to be subscribed.

  7. dotnet-policy-service
    removed
    untriagedNew issue has not been triaged by the area owner
    on May 28, 2024

  8. amanasifkhalid commented on May 28, 2024

    @amanasifkhalid
    amanasifkhalid
    on May 28, 2024
    Contributor

    @EgorBo Thanks for opening this. I haven't spent too much time looking at the regressions yet, but based on the larger ones I've looked at, I think we need a more general/powerful implementation of #102461. Take GuardedDevirtualization.TwoClassVirtual.Call for instance, which regressed by over 60% on Windows x64 (GuardedDevirtualization.ThreeClassVirtual.Call regressed for similar reasons). Before reordering, the blocks look like this:

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight    IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1      18 [000..00E)-> BB07(0.00595),BB02(0.994)   ( cond )                     i IBC idxlen
BB02 [0009]  1       BB01                  0.99   18 [00E..???)-> BB03(1)                 (always)                     IBC internal idxlen
BB03 [0001]  2       BB02,BB06           167.07 3007 [00E..00E)-> BB05(0.07),BB04(0.93)   ( cond )                     i IBC loophead idxlen bwd bwd-target
BB04 [0005]  1       BB03                155.37 2797 [???..???)-> BB06(1)                 (always)                     i IBC internal idxlen nullcheck bwd
BB05 [0006]  1       BB03                 11.69  211 [???..???)-> BB06(1)                 (always)                     i IBC internal hascall gcsafe idxlen bwd
BB06 [0004]  2       BB04,BB05           167.07 3007 [00E..024)-> BB03(0.994),BB07(0.00595)   ( cond )                     i IBC idxlen bwd
BB07 [0010]  2       BB01,BB06             1.00   18 [024..026)                           (return)                     i IBC
BB09 [0011]  0                             0         [???..???)                           (throw )                     i rare keep internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    The interesting ones are BB03-BB06, which comprise a loop. The hot path is BB03->BB04->BB06->BB03; moving BB05 out-of-line to make the hot path branchless (except for the backward jump) would be ideal. The old layout does this:

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight    IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1      18 [000..00E)-> BB07(0.00595),BB02(0.994)   ( cond )                     i IBC idxlen
BB02 [0009]  1       BB01                  0.99   18 [00E..???)-> BB03(1)                 (always)                     IBC internal idxlen
BB03 [0001]  2       BB02,BB06           167.07 3007 [00E..00E)-> BB05(0.07),BB04(0.93)   ( cond )                     i IBC loophead idxlen bwd bwd-target
BB04 [0005]  1       BB03                155.37 2797 [???..???)-> BB06(1)                 (always)                     i IBC internal idxlen nullcheck bwd
BB06 [0004]  2       BB04,BB05           167.07 3007 [00E..024)-> BB03(0.994),BB07(0.00595)   ( cond )                     i IBC idxlen bwd
BB07 [0010]  2       BB01,BB06             1.00   18 [024..026)                           (return)                     i IBC
BB05 [0006]  1       BB03                 11.69  211 [???..???)-> BB06(1)                 (always)                     i IBC internal hascall gcsafe idxlen bwd
BB09 [0011]  0                             0         [???..???)                           (throw )                     i rare keep internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    In the new layout, fgMoveBackwardJumpsToSuccessors only considers moving backward unconditional jumps, so BB05 remains in the way of the hot path:

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight    IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1      18 [000..00E)-> BB07(0.00595),BB02(0.994)   ( cond )                     i IBC idxlen
BB02 [0009]  1       BB01                  0.99   18 [00E..???)-> BB03(1)                 (always)                     IBC internal idxlen
BB03 [0001]  2       BB02,BB06           167.07 3007 [00E..00E)-> BB05(0.07),BB04(0.93)   ( cond )                     i IBC loophead idxlen bwd bwd-target
BB04 [0005]  1       BB03                155.37 2797 [???..???)-> BB06(1)                 (always)                     i IBC internal idxlen nullcheck bwd
BB05 [0006]  1       BB03                 11.69  211 [???..???)-> BB06(1)                 (always)                     i IBC internal hascall gcsafe idxlen bwd
BB06 [0004]  2       BB04,BB05           167.07 3007 [00E..024)-> BB03(0.994),BB07(0.00595)   ( cond )                     i IBC idxlen bwd
BB07 [0010]  2       BB01,BB06             1.00   18 [024..026)                           (return)                     i IBC
BB09 [0011]  0                             0         [???..???)                           (throw )                     i rare keep internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    I think modifying fgMoveBackwardJumpsToSuccessors to consider forward jumps would be a good start to chipping away at these regressions. However, moving BB06 up to BB04 would break up the former's fallthrough into BB07. That's ok, since that's part of the slow path, but we probably want to update the cost check in fgMoveBackwardJumpsToSuccessors to also check if moving the target block will break up fallthrough into its next block, and whether the edge weights deem it's worth it.

    Thoughts on this? cc @AndyAyersMS for visibility. Thanks!

  9. AndyAyersMS commented on May 28, 2024

    @AndyAyersMS
    AndyAyersMS
    on May 28, 2024
    Member

    I would suggest we list all the regressions in rank order and then investigate at least the top 20 or so. We also should see what the arm64 data looks like.

    Based on the above and the the other fixup actions, it feels like we are getting to the point where we may actually want to create the explicit cost/benefit model to assess incremental improvements. We have a layout configuration A, and some proposed alternate B. Which is better?

    If you want to try to build this out, we should talk. The general model is something like the following:

    • identify 3 cut points in the linear chain of blocks (creating 4 segments overall). First segment must contain at least the entry block; last one can be empty.
    • swap the middle two segments; if that is lower cost, make that the new configuration
    • repeat until you can't find any more improvements or give up (drive to local minima)
    • (or just limit yourself to one pass or something to keep the TP under control)
    • (the someday refinement: occasionally allow reorderings that drive the cost up, to try and find a global minimum)

    Since there are just 3 cut points, if you have an initial cost model it's not too hard to compute the delta to the new cost model, you (mostly) just need to recost the segment-ending blocks (if the cost model is block-size sensitive then perhaps a bit more, but I don't think we have good size estimates yet?)

    For the time being the "identify" step can be finding a single misplaced block and a new proposed location, and then seeing if the cost improves.

  10. amanasifkhalid commented on May 29, 2024

    @amanasifkhalid
    amanasifkhalid
    on May 29, 2024
    Contributor

    I would suggest we list all the regressions in rank order and then investigate at least the top 20 or so.

    Sorry about the delay in getting back to this; the script we typically use to collate regression data doesn't seem to work since the perflab test report URLs changed, so I hacked up my own script. Here are the top 20 regressions on Windows x64:

    Name Base Test Ratio
    System.Collections.Sort.Array_Comparison 9.19 μs 17.66 μs 1.92
    System.Collections.Sort.Array_ComparerClass 9.85 μs 17.36 μs 1.76
    GuardedDevirtualization.ThreeClassVirtual.Call 3.82 ns 6.56 ns 1.72
    System.Linq.Tests.Perf_Enumerable.SelectToList 125.92 ns 209.71 ns 1.67
    System.Collections.Sort.LinqQuery 17.06 μs 28.35 μs 1.66
    GuardedDevirtualization.TwoClassVirtual.Call 1.75 ns 2.86 ns 1.63
    LinqBenchmarks.Count00ForX 91.54 ms 147.83 ms 1.61
    System.Collections.Sort.LinqOrderByExtension 17.45 μs 28.14 μs 1.61
    System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8 53.46 ns 85.59 ns 1.6
    System.Collections.Sort.LinqOrderByExtension 20.34 μs 32.52 μs 1.6
    System.Memory.Span.Reverse 32.25 ns 50.97 ns 1.58
    System.Tests.Perf_Random.NextBytes_unseeded 170.90 ns 244.84 ns 1.43
    System.Tests.Perf_Random.NextBytes_span_unseeded 171.98 ns 244.27 ns 1.42
    System.Memory.Span.Reverse 3.21 ns 4.54 ns 1.41
    System.Collections.Sort.LinqOrderByExtension 16.82 μs 23.40 μs 1.39
    System.Text.Perf_Ascii.ToLower_Chars 27.93 ns 38.72 ns 1.39
    System.Collections.Perf_Frozen.ToFrozenDictionary 32.28 μs 44.49 μs 1.38
    Benchstone.BenchI.CSieve.Test 4.25 ms 5.87 ms 1.38
    System.Text.Perf_Ascii.Equals_Bytes 4.55 ns 6.23 ns 1.37
    System.Text.Perf_Ascii.ToUpper_Chars 27.64 ns 37.66 ns 1.36

    And on Linux x64:

    Name Base Test Ratio
    System.IO.Hashing.Tests.Crc32_AppendPerf.Append 335.26 ns 1.07 μs 3.2
    GuardedDevirtualization.ThreeClassVirtual.Call 1.07 ns 2.90 ns 2.72
    GuardedDevirtualization.ThreeClassVirtual.Call 1.01 ns 2.73 ns 2.71
    GuardedDevirtualization.ThreeClassVirtual.Call 1.07 ns 2.91 ns 2.71
    GuardedDevirtualization.ThreeClassVirtual.Call 1.01 ns 2.71 ns 2.69
    GuardedDevirtualization.ThreeClassVirtual.Call 1.01 ns 2.71 ns 2.69
    GuardedDevirtualization.ThreeClassVirtual.Call 1.07 ns 2.87 ns 2.68
    System.IO.Hashing.Tests.Crc32_AppendPerf.Append 8.21 ns 21.65 ns 2.64
    System.IO.Hashing.Tests.Crc32_AppendPerf.Append 15.58 ns 38.16 ns 2.45
    GuardedDevirtualization.TwoClassVirtual.Call 1.46 ns 3.22 ns 2.21
    System.Tests.Perf_String.TrimEnd 1.69 ns 3.18 ns 1.88
    GuardedDevirtualization.TwoClassVirtual.Call 1.23 ns 2.30 ns 1.86
    Benchstone.BenchF.NewtR.Test 55.69 ms 102.90 ms 1.85
    LinqBenchmarks.Count00ForX 58.01 ms 103.57 ms 1.79
    System.Collections.Sort.Array_ComparerClass 8.31 μs 14.71 μs 1.77
    System.Collections.Sort.Array_Comparison 7.32 μs 12.29 μs 1.68
    System.Tests.Perf_String.TrimStart_CharArr 2.52 ns 4.12 ns 1.64
    System.Collections.Sort.Array_ComparerClass 7.15 μs 11.61 μs 1.62
    System.Collections.IterateFor.IList 287.08 ns 454.48 ns 1.58
    System.Text.Perf_Ascii.ToUpper_Bytes 11.49 ns 17.77 ns 1.55

    I'll start digging into the overlapping benchmarks (this explains GuardedDevirtualization.TwoClassVirtual.Call and GuardedDevirtualization.ThreeClassVirtual.Call already, and I suspect the theme of hot paths being polluted by a relatively cold (but not "cold" cold) jump will continue to come up).

  11. amanasifkhalid commented on May 29, 2024

    @amanasifkhalid
    amanasifkhalid
    on May 29, 2024
    Contributor

    I'm seeing the same theme of interleaving hot and colder blocks throughout these regressions. The RPO-based layout on its own doesn't seem to leverage edge weights heavily enough, and while fgMoveBackwardJumpsToSuccessors does consider edge weights, its backward jump limitation is blocking lots of potentially profitable moves. System.Memory.Span<Int32>.Reverse illustrates this succinctly. Here's what the new layout does:

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight   IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1    100 [000..00A)-> BB05(0.2),BB02(0.8)     ( cond )                     i IBC
BB02 [0001]  1       BB01                  0.80  80 [00A..00B)-> BB04(0.2),BB03(0.8)     ( cond )                     i IBC hascall
BB03 [0018]  1       BB02                  0.64  64 [00A..00B)-> BB04(1)                 (always)                     i IBC hascall gcsafe
BB04 [0019]  2       BB02,BB03             0.80  80 [00A..00B)-> BB06(1)                 (always)                     i IBC hascall gcsafe
BB05 [0003]  1       BB01                  0.20  20 [???..???)-> BB06(1)                 (always)                     i IBC internal hascall
BB06 [0002]  2       BB04,BB05             1    100 [01D..01E)                           (return)                     i IBC tail-succ
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    Notice how BB06 can be reached from BB04 and BB05, and the former is hotter, but doesn't fall into BB06. If fgMoveBackwardJumpsToSuccessors considered forward jumps, it would move BB06 up. For a more involved example, let's look at System.Text.Perf_Ascii.Equals_Bytes. Here's the new layout:

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight   IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1    100 [000..001)-> BB03(0.99),BB02(0.00994)  ( cond )                     i IBC
BB03 [0004]  1       BB01                  0.99  99 [000..001)-> BB04(1)                 (always)                     i IBC idxlen nullcheck
BB02 [0003]  1       BB01                  0.01   1 [000..001)-> BB04(1)                 (always)                     i IBC
BB04 [0005]  2       BB02,BB03             1    100 [000..001)-> BB06(0.99),BB05(0.00994)  ( cond )                     i IBC
BB06 [0009]  1       BB04                  0.99  99 [000..001)-> BB07(1)                 (always)                     i IBC idxlen nullcheck
BB05 [0008]  1       BB04                  0.01   1 [000..001)-> BB07(1)                 (always)                     i IBC
BB07 [0010]  2       BB05,BB06             1    100 [000..000)-> BB09(0),BB08(1)         ( cond )                     i IBC
BB08 [0012]  1       BB07                  1    100 [000..000)-> BB10(1)                 (always)                     i IBC internal hascall gcsafe
BB10 [0014]  2       BB08,BB09             1    100 [01C..01C)                           (return)                     i IBC
BB09 [0013]  1       BB07                  0      0 [000..000)-> BB10(1)                 (always)                     i IBC rare internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    If we considered moving forward jumps, we'd have the following decision tree:

    • BB03->BB04 is hotter than BB02->BB04, so move BB04 up.
    • BB04->BB06 is hot, and BB04 is the only pred of BB06, but BB02 is now between them, so move BB06 up; we're basically bubbling BB02 down the block list. Now we have this:
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight   IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1    100 [000..001)-> BB03(0.99),BB02(0.00994)  ( cond )                     i IBC
BB03 [0004]  1       BB01                  0.99  99 [000..001)-> BB04(1)                 (always)                     i IBC idxlen nullcheck
BB04 [0005]  2       BB02,BB03             1    100 [000..001)-> BB06(0.99),BB05(0.00994)  ( cond )                     i IBC
BB06 [0009]  1       BB04                  0.99  99 [000..001)-> BB07(1)                 (always)                     i IBC idxlen nullcheck
BB02 [0003]  1       BB01                  0.01   1 [000..001)-> BB04(1)                 (always)                     i IBC
BB05 [0008]  1       BB04                  0.01   1 [000..001)-> BB07(1)                 (always)                     i IBC
BB07 [0010]  2       BB05,BB06             1    100 [000..000)-> BB09(0),BB08(1)         ( cond )                     i IBC
BB08 [0012]  1       BB07                  1    100 [000..000)-> BB10(1)                 (always)                     i IBC internal hascall gcsafe
BB10 [0014]  2       BB08,BB09             1    100 [01C..01C)                           (return)                     i IBC
BB09 [0013]  1       BB07                  0      0 [000..000)-> BB10(1)                 (always)                     i IBC rare internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
    • BB06->BB07 is hotter than BB05->BB07, so move BB07 up. We've bubbled BB02 and BB05 down.
    • BB07->BB08 is hot, and BB07 is the only pred of BB08, so move BB08 up.
    • BB08->BB10is hot, andBB10's other pred doesn't fall into it anyway, so move BB10` up. Here's what we have:
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight   IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1    100 [000..001)-> BB03(0.99),BB02(0.00994)  ( cond )                     i IBC
BB03 [0004]  1       BB01                  0.99  99 [000..001)-> BB04(1)                 (always)                     i IBC idxlen nullcheck
BB04 [0005]  2       BB02,BB03             1    100 [000..001)-> BB06(0.99),BB05(0.00994)  ( cond )                     i IBC
BB06 [0009]  1       BB04                  0.99  99 [000..001)-> BB07(1)                 (always)                     i IBC idxlen nullcheck
BB07 [0010]  2       BB05,BB06             1    100 [000..000)-> BB09(0),BB08(1)         ( cond )                     i IBC
BB08 [0012]  1       BB07                  1    100 [000..000)-> BB10(1)                 (always)                     i IBC internal hascall gcsafe
BB10 [0014]  2       BB08,BB09             1    100 [01C..01C)                           (return)                     i IBC
BB02 [0003]  1       BB01                  0.01   1 [000..001)-> BB04(1)                 (always)                     i IBC
BB05 [0008]  1       BB04                  0.01   1 [000..001)-> BB07(1)                 (always)                     i IBC
BB09 [0013]  1       BB07                  0      0 [000..000)-> BB10(1)                 (always)                     i IBC rare internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
    • BB10 has no successors.
    • BB02 is a backward jump, but BB03->BB04 is hotter, so don't move anything.
    • Ditto BB05.
    • BB09 is rarely-run, so we don't care about it.

    For a few other short examples, just to build my case, here's the new layout of System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8:

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight   IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1    100 [000..001)-> BB03(0.99),BB02(0.00994)  ( cond )                     i IBC
BB03 [0004]  1       BB01                  0.50  50 [000..001)-> BB04(1)                 (always)                     i IBC idxlen nullcheck
BB02 [0003]  1       BB01                  0.00   0 [000..001)-> BB04(1)                 (always)                     i IBC
BB04 [0005]  2       BB02,BB03             1    100 [000..000)-> BB06(1),BB05(0)         ( cond )                     i IBC
BB06 [0009]  1       BB04                  0.50  50 [000..000)-> BB07(1)                 (always)                     i IBC internal idxlen nullcheck
BB07 [0013]  2       BB05,BB06             1    100 [027..027)                           (return)                     i IBC hascall gcsafe
BB05 [0008]  1       BB04                  0      0 [000..000)-> BB07(1)                 (always)                     i IBC rare internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    And for System.String.TrimWhiteSpaceHelper from System.Tests.Perf_String.TrimEnd (in particular, we could consider creating fallthrough for BB29->BB23->BB27->BB28):

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight    IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1    2261 [000..010)-> BB12(0.092),BB02(0.908) ( cond )                     i IBC idxlen
BB02 [0001]  1       BB01                  0.91 2053 [010..014)-> BB11(0),BB03(1)         ( cond )                     i IBC idxlen
BB03 [0246]  2       BB02,BB09             1.82 4120 [014..015)-> BB07(0.52),BB08(0.48)   ( cond )                     i IBC idxlen bwd
BB07 [0041]  1       BB03                  0.95 2142 [014..015)-> BB14(0.498),BB09(0.502) ( cond )                     i IBC bwd
BB08 [0055]  1       BB03                  1.57 3560 [014..022)-> BB14(0.498),BB09(0.502) ( cond )                     i IBC bwd
BB09 [0003]  2       BB07,BB08             1.82 4120 [022..02F)-> BB03(1),BB10(0)         ( cond )                     i IBC hascall idxlen bwd
BB14 [0011]  2       BB07,BB08             0.91 2053 [???..???)-> BB12(1)                 (always)                     i IBC internal hascall
BB12 [0005]  3       BB01,BB11,BB14        1.00 2261 [02F..034)-> BB23(0.000442),BB13(1)  ( cond )                     i IBC
BB13 [0006]  1       BB12                  1.00 2260 [034..03F)-> BB21(1)                 (always)                     i IBC idxlen
BB21 [0009]  2       BB13,BB20             1.01 2282 [051..055)-> BB15(1),BB22(0)         ( cond )                     i IBC bwd bwd-src
BB15 [0007]  1       BB21                  1.01 2282 [03F..040)-> BB18(0.52),BB19(0.48)   ( cond )                     i IBC loophead idxlen bwd bwd-target
BB18 [0169]  1       BB15                  0.52 1187 [03F..040)-> BB29(0.99),BB20(0.00964)  ( cond )                     i IBC bwd
BB19 [0183]  1       BB15                  0.87 1972 [03F..04D)-> BB29(0.99),BB20(0.00964)  ( cond )                     i IBC bwd
BB29 [0013]  2       BB18,BB19             1.00 2260 [???..???)-> BB23(1)                 (always)                     i IBC internal hascall
BB20 [0008]  2       BB18,BB19             0.01   22 [04D..051)-> BB21(1)                 (always)                     i IBC hascall bwd
BB23 [0010]  3       BB12,BB22,BB29        1.00 2261 [055..056)-> BB27(0.884),BB24(0.116) ( cond )                     i IBC hascall idxlen
BB27 [0243]  1       BB23                  0.88 1999 [055..056)-> BB28(1)                 (always)                     i IBC
BB24 [0240]  1       BB23                  0.12  262 [055..056)-> BB26(0.0461),BB25(0.954)  ( cond )                     i IBC
BB25 [0241]  1       BB24                  0.11  250 [055..056)-> BB28(1)                 (always)                     i IBC hascall gcsafe
BB26 [0242]  1       BB24                  0.01   12 [055..056)-> BB28(1)                 (always)                     i IBC
BB28 [0244]  3       BB25,BB26,BB27        1.00 2261 [055..05E)                           (return)                     i IBC
BB10 [0247]  1       BB09                  0       0 [???..???)-> BB11(1)                 (always)                     IBC rare internal
BB11 [0012]  2       BB02,BB10             0       0 [???..???)-> BB12(1)                 (always)                     i IBC rare internal hascall
BB22 [0014]  1       BB21                  0       0 [???..???)-> BB23(1)                 (always)                     i IBC rare internal hascall
BB30 [0248]  0                             0         [???..???)                           (throw )                     i rare keep internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    And for System.Tests.Perf_Random.NextBytes_unseeded (an interesting case in that the exceptional path is the hot path based on edge weights, even though the throw block has a weight of 0 due to RyuJIT's flowgraph semantics -- I guess if our post-RPO changes suggest a "rarely-run" block should be moved up because it isn't that rare, maybe we should unmark it as not rare?):

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight   IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1    100 [000..003)-> BB03(0.2),BB02(0.8)     ( cond )                     i IBC
BB03 [0002]  1       BB01                  1    100 [00A..017)                           (return)                     i IBC jmp hascall hist
BB02 [0001]  1       BB01                  0      0 [003..00A)                           (throw )                     i IBC rare hascall gcsafe
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    Based on the above and the the other fixup actions, it feels like we are getting to the point where we may actually want to create the explicit cost/benefit model to assess incremental improvements.

    I'd be interested in pursuing this, though based on the above, I think it might be worthwhile trying to implement a more general post-RPO pass that does the moves I described above, since it seems simple enough.

  12. AndyAyersMS commented on May 29, 2024

    @AndyAyersMS
    AndyAyersMS
    on May 29, 2024
    Member

    I am surprised some of these are so large. Do we have any results for AMD HW? All the reports above are for Intel.

  13. AndyAyersMS commented on May 29, 2024

    @AndyAyersMS
    AndyAyersMS
    on May 29, 2024
    Member

    ADX data for the three class virtual test. Strongly suggests that this may be JCC errata kicking in.

    image

    Of course this is a two-way street, some of the improvements may be like this too.

  14. AndyAyersMS commented on May 29, 2024

    @AndyAyersMS
    AndyAyersMS
    on May 29, 2024
    Member

    Randomly looking at improvements, it seems most are also intel only. But not all. Here's Benchstone.BenchI.BubbleSort2.Test

    image

    @DrewScoggins is looking into whether there are unfiled AMD64 reports out there. From what I can tell the newer AMD64 hardware is mostly indifferent.

  15. 18 remaining items

  16. amanasifkhalid commented on Jun 14, 2024

    @amanasifkhalid
    amanasifkhalid
    on Jun 14, 2024
    Contributor

    GuardedDevirtualization.TwoClassInterface:Call has a similar layout shape, and the updated fgMoveHotBlocks seems to handle it better:

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight    IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0009]  1                             0.60   10 [???..???)-> BB16(0.00595),BB02(0.994)   ( cond )                     IBC internal idxlen
BB02 [0010]  1       BB01                  0.59   10 [00E..???)-> BB10(0.01),BB05(0.99)   ( cond )                     IBC internal
BB05 [0012]  1       BB02                  0.59   10 [???..???)-> BB06(1)                 (always)                     IBC internal idxlen
BB06 [0001]  2       BB05,BB09            98.41 1654 [00E..00E)-> BB08(0),BB07(1)         ( cond )                     i IBC loophead idxlen bwd bwd-target
BB07 [0006]  1       BB06                 98.41 1654 [???..???)-> BB09(1)                 (always)                     i IBC internal idxlen nullcheck bwd
BB09 [0005]  2       BB07,BB08            98.41 1654 [00E..024)-> BB06(0.994),BB16(0.00595)   ( cond )                     i IBC idxlen bwd
BB16 [0021]  3       BB01,BB09,BB14        0.60   10 [024..026)                           (return)                     i IBC
BB10 [0013]  1       BB02                  0.01    0 [???..???)-> BB11(1)                 (always)                     IBC internal idxlen
BB11 [0014]  2       BB10,BB14             0.99   17 [00E..00E)-> BB13(0),BB12(1)         ( cond )                     i IBC loophead idxlen bwd bwd-target
BB12 [0015]  1       BB11                  0.99   17 [???..???)-> BB14(1)                 (always)                     i IBC internal idxlen nullcheck bwd
BB14 [0017]  2       BB12,BB13             0.99   17 [00E..024)-> BB11(0.994),BB16(0.00595)   ( cond )                     i IBC idxlen bwd
BB08 [0007]  1       BB06                  0       0 [???..???)-> BB09(1)                 (always)                     i IBC rare internal hascall gcsafe idxlen bwd
BB13 [0016]  1       BB11                  0       0 [???..???)-> BB14(1)                 (always)                     i IBC rare internal hascall gcsafe idxlen bwd
BB19 [0022]  0                             0         [???..???)                           (throw )                     i rare keep internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    All the other regressed GuardedDevirtualization benchmarks (TwoClassVirtual, ThreeClassVirtual, ThreeClassInterface) have the same shape that can now fall out of the hot loop into the return block. Those feature heavily in the top regressions on x64 and arm64, and the new layout looks optimal, so I'd be surprised if these regressions aren't resolved.

  17. amanasifkhalid
    added a commit that references this issue on Jun 15, 2024

    JIT: Consider block weights instead of BBF_RUN_RARELY flag in fgMoveC…

    5b96528

  18. amanasifkhalid commented on Jun 15, 2024

    @amanasifkhalid
    amanasifkhalid
    on Jun 15, 2024
    Contributor

    LinqBenchmarks:Count00ForX highlights a case where our loop misrotation fix makes the wrong decision. Here's the initial RPO layout:

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight    IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0011]  1                             1      98 [???..???)-> BB06(1)                 (always)                     i IBC internal
BB06 [0005]  4       BB01,BB03,BB04,BB05 100    9778 [02B..02C)-> BB14(0.00016),BB07(1)   ( cond )                     i IBC bwd bwd-src osr-entry
BB07 [0020]  1       BB06                 99.98 9776 [02B..02C)-> BB10(0.161),BB08(0.839) ( cond )                     i IBC bwd
BB08 [0021]  1       BB07                 83.93 8207 [02B..02C)-> BB11(1)                 (always)                     i IBC idxlen bwd
BB10 [0026]  1       BB07                 16.07 1571 [02B..02C)-> BB11(1)                 (always)                     i IBC bwd
BB11 [0023]  2       BB08,BB10           100    9778 [02B..034)-> BB04(0.983),BB12(0.0171)  ( cond )                     i IBC bwd bwd-src
BB04 [0003]  1       BB11                 98.29 9611 [015..027)-> BB06(0.933),BB05(0.0666)  ( cond )                     i IBC loophead bwd bwd-target
BB05 [0004]  1       BB04                  6.55  640 [027..02B)-> BB06(1)                 (always)                     i IBC bwd
BB12 [0006]  1       BB11                  1.71  167 [034..050)-> BB02(0.994),BB13(0.00595)   ( cond )                     i IBC bwd
BB02 [0001]  1       BB12                  1.71  167 [00C..013)-> BB03(1)                 (always)                     i IBC loophead nullcheck bwd bwd-target
BB03 [0002]  1       BB02                  1.71  167 [???..???)-> BB06(1)                 (always)                     i IBC keep internal bwd
BB13 [0010]  1       BB12                  0.01    1 [050..059)                           (return)                     i IBC
BB14 [0025]  1       BB06                  0       0 [02B..02C)                           (throw )                     i IBC rare hascall gcsafe bwd
BB15 [0030]  0                             0         [???..???)                           (throw )                     i rare keep internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    And the final layout:

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight    IBC [IL range]   [jump]                            [EH region]        [flags]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
BB01 [0011]  1                             1      98 [???..???)-> BB06(1)                 (always)                     i IBC internal
BB05 [0004]  1       BB04                  6.55  640 [027..02B)-> BB06(1)                 (always)                     i IBC bwd
BB06 [0005]  4       BB01,BB03,BB04,BB05 100    9778 [02B..02C)-> BB14(0.00016),BB07(1)   ( cond )                     i IBC bwd bwd-src osr-entry
BB07 [0020]  1       BB06                 99.98 9776 [02B..02C)-> BB10(0.161),BB08(0.839) ( cond )                     i IBC bwd
BB08 [0021]  1       BB07                 83.93 8207 [02B..02C)-> BB11(1)                 (always)                     i IBC idxlen bwd
BB11 [0023]  2       BB08,BB10           100    9778 [02B..034)-> BB04(0.983),BB12(0.0171)  ( cond )                     i IBC bwd bwd-src
BB04 [0003]  1       BB11                 98.29 9611 [015..027)-> BB06(0.933),BB05(0.0666)  ( cond )                     i IBC loophead bwd bwd-target
BB10 [0026]  1       BB07                 16.07 1571 [02B..02C)-> BB11(1)                 (always)                     i IBC bwd
BB12 [0006]  1       BB11                  1.71  167 [034..050)-> BB02(0.994),BB13(0.00595)   ( cond )                     i IBC bwd
BB02 [0001]  1       BB12                  1.71  167 [00C..013)-> BB03(1)                 (always)                     i IBC loophead nullcheck bwd bwd-target
BB03 [0002]  1       BB02                  1.71  167 [???..???)-> BB06(1)                 (always)                     i IBC keep internal bwd
BB13 [0010]  1       BB12                  0.01    1 [050..059)                           (return)                     i IBC
BB14 [0025]  1       BB06                  0       0 [02B..02C)                           (throw )                     i IBC rare hascall gcsafe bwd
BB15 [0030]  0                             0         [???..???)                           (throw )                     i rare keep internal
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

    The decision to move BB05 ends up breaking fallthrough from BB01 into BB06 without introducing any new fallthrough. We could fix this in fgMoveHotBlocks by not moving backward jumps if the previous block falls into the block with the jump, and breaking this fallthrough won't introduce new fallthrough into the next block. But this starts to get a bit pattern-matchy, so perhaps we accept this for now until we have a more general (i.e. 3-opt) solution.

  19. amanasifkhalid commented on Jun 15, 2024

    @amanasifkhalid
    amanasifkhalid
    on Jun 15, 2024
    Contributor

    Sorry for all the comment spam. I've looked at the remaining top regressions, and I haven't seen anything noteworthy -- in general, #102927 seems to do a good job of maximizing fallthrough on hot paths, so long as the profile data is sensible. As such, I'm planning on getting that merged for now, and coming back to 3-opt after Preview 6.

  20. amanasifkhalid
    added a commit that references this issue on Jun 17, 2024

    JIT: Move targets of hot jumps to create fallthrough post-RPO layout (#…

    6e88cf8

  21. amanasifkhalid commented on Jun 17, 2024

    @amanasifkhalid
    amanasifkhalid
    on Jun 17, 2024
    Contributor

    I've merged #102927 -- fingers crossed we see an improvement in the next triage...

  22. amanasifkhalid commented on Jul 26, 2024

    @amanasifkhalid
    amanasifkhalid
    on Jul 26, 2024
    Contributor

    I'm punting this to .NET 10, as we're planning to continue to iterate on the new layout. We'll re-evaluate any remaining regressions here with each change.

  23. amanasifkhalid
    modified the milestones: 9.0.0, 10.0.0 on Jul 26, 2024

  24. amanasifkhalid commented on Apr 11, 2025

    @amanasifkhalid
    amanasifkhalid
    on Apr 11, 2025
    Contributor

    There's been far too much churn since these reports came out for us to be able to pinpoint root causes for changes since then, but I thought it would be interesting to look at how the benchmarks have changed as a proxy for .NET 9 vs 10 layout perf. Here are the arm64 regressions:

    Notes Recent Score Orig Score Ubuntu arm64 Windows arm64 Benchmark
    1.40 1.14 1.40
    1.14    		 System.Collections.Perf_Frozen(Int16).ToFrozenDictionary(Count: 512)
    1.37 1.33 1.37
    1.33    		 System.Collections.IterateForEach(String).ImmutableStack(Size: 512)
    1.34 1.56 1.34
    1.56    		 Benchstone.BenchI.CSieve.Test
    1.33 1.16 1.33
    1.16    		 System.Collections.Perf_Frozen(Int16).ToFrozenSet(Count: 512)
    1.29 2.22 1.29
    2.22    		 GuardedDevirtualization.TwoClassVirtual.Call(testInput: pB = 0.10)
    1.28 1.90 1.28
    1.90    		 System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfNumbers)
    1.22 1.46 1.03
    1.33    		 1.45
    1.61    		 System.Collections.Sort(Int32).LinqOrderByExtension(Size: 512)
    1.21 1.07 1.21
    1.07    		 System.Collections.Perf_Frozen(NotKnownComparable).ToFrozenDictionary(Count: 64)
    1.18 1.47 1.18
    1.47    		 System.MathBenchmarks.Double.Max
    1.16 1.05 1.16
    1.05    		 System.Collections.Perf_Frozen(NotKnownComparable).ToFrozenSet(Count: 64)
    1.04 1.40 1.04
    1.40    		 System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: NumericProperties)
    1.04 1.11 1.04
    1.11    		 System.Collections.CreateAddAndClear(Int32).Dictionary(Size: 512)
    1.02 1.11 1.02
    1.11    		 System.Collections.CreateAddAndClear(Int32).IDictionary(Size: 512)
    1.01 1.13 1.01
    1.13    		 System.Collections.Tests.Perf_PriorityQueue(Guid, Guid).Dequeue_And_Enqueue(Size: 100)
    1.00 1.08 1.00
    1.08    		 System.Collections.CreateAddAndClear(Int32).HashSet(Size: 512)
    1.00 2.05 1.00
    2.05    		 GuardedDevirtualization.TwoClassInterface.Call(testInput: pB = 0.80)
    1.00 1.53 1.00
    1.53    		 LinqBenchmarks.Count00ForX
    0.98 1.16 0.98
    1.16    		 System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: StringProperties)
    0.98 1.06 0.98
    1.06    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes", Options: None)
    0.98 1.06 0.98
    1.06    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock Holmes", Options: None)
    0.98 1.06 0.98
    1.06    		 System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 100, TestCase: LongMultiLine)
    0.97 1.07 0.97
    1.07    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock\s+Holmes", Options: None)
    0.97 1.82 0.97
    1.82    		 GuardedDevirtualization.TwoClassInterface.Call(testInput: pB = 0.30)
    0.97 1.07 0.97
    1.07    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?m)^Sherlock Holmes
    0.97 1.06 0.97
    1.06    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?m)^Sherlock Holmes
    0.97 1.06 0.97
    1.06    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock", Options: None)
    0.97 1.45 0.97
    1.45    		 System.Collections.IterateForEachNonGeneric(String).Stack(Size: 512)
    0.97 2.80 0.97
    2.80    		 GuardedDevirtualization.TwoClassVirtual.Call(testInput: pB = 0.70)
    0.96 1.06 0.96
    1.06    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock Holmes", Options: NonBacktracking)
    0.95 1.06 0.95
    1.06    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock", Options: NonBacktracking)
    0.93 1.06 0.93
    1.06    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock\s+Holmes", Options: NonBacktracking)
    0.93 1.99 0.93
    1.99    		 GuardedDevirtualization.ThreeClassInterface.Call(testInput: pB=0.00 pD=0.67)
    0.93 1.08 0.93
    1.08    		 System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfStrings)
    0.92 1.18 0.92
    1.18    		 System.Collections.Perf_Frozen(NotKnownComparable).ToFrozenSet(Count: 512)
    0.92 3.00 0.92
    3.00    		 GuardedDevirtualization.TwoClassVirtual.Call(testInput: pB = 0.60)
    0.92 1.31 0.92
    1.31    		 System.Collections.Sort(Int32).LinqQuery(Size: 512)
    0.91 1.18 0.91
    1.18    		 System.Collections.Sort(Int32).Array_ComparerClass(Size: 512)
    0.90 1.05 0.90
    1.05    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes", Options: NonBacktracking)
    0.90 1.23 0.90
    1.23    		 System.Collections.Sort(Int32).Array_Comparison(Size: 512)
    0.89 1.48 0.89
    1.48    		 System.Collections.Sort(IntStruct).LinqOrderByExtension(Size: 512)
    0.89 1.09 0.89
    1.09    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: NonBacktracking)
    0.84 1.12 0.84
    1.12    		 System.Collections.Perf_Frozen(NotKnownComparable).ToFrozenDictionary(Count: 512)
    0.84 1.96 0.84
    1.96    		 GuardedDevirtualization.TwoClassVirtual.Call(testInput: pB = 0.80)
    0.84 2.07 0.84
    2.07    		 GuardedDevirtualization.TwoClassVirtual.Call(testInput: pB = 0.20)
    0.76 2.20 0.76
    2.20    		 GuardedDevirtualization.ThreeClassVirtual.Call(testInput: pB=0.67 pD=0.33)
    0.75 2.28 0.75
    2.28    		 GuardedDevirtualization.ThreeClassVirtual.Call(testInput: pB=0.67 pD=0.00)
    0.75 2.26 0.75
    2.26    		 GuardedDevirtualization.ThreeClassVirtual.Call(testInput: pB=0.33 pD=0.00)
    0.73 1.96 0.73
    1.96    		 GuardedDevirtualization.ThreeClassInterface.Call(testInput: pB=0.67 pD=0.00)
    0.72 1.64 0.72
    1.64    		 GuardedDevirtualization.ThreeClassInterface.Call(testInput: pB=0.67 pD=0.33)
    0.71 2.03 0.71
    2.03    		 GuardedDevirtualization.TwoClassVirtual.Call(testInput: pB = 0.90)
    0.70 1.33 0.70
    1.33    		 System.Collections.Sort(BigStruct).LinqOrderByExtension(Size: 512)
    0.65 1.91 0.65
    1.91    		 GuardedDevirtualization.ThreeClassVirtual.Call(testInput: pB=0.00 pD=0.33)
    0.64 2.20 0.64
    2.20    		 GuardedDevirtualization.ThreeClassVirtual.Call(testInput: pB=0.00 pD=0.67)
    0.62 1.85 0.62
    1.85    		 GuardedDevirtualization.TwoClassVirtual.Call(testInput: pB = 0.40)
    0.56 1.48 0.56
    1.48    		 GuardedDevirtualization.TwoClassVirtual.Call(testInput: pB = 0.50)
    0.37 1.48 0.37
    1.48    		 GuardedDevirtualization.ThreeClassVirtual.Call(testInput: pB=0.33 pD=0.67)

    The Windows arm64 regressions are too noisy to be enlightening. The Ubuntu arm64 regressions are clearly from #113491, and are tracked in #113912. The GDV benchmarks at the bottom show nice improvements.

  25. amanasifkhalid commented on Apr 12, 2025

    @amanasifkhalid
    amanasifkhalid
    on Apr 12, 2025
    Contributor

    And here are the improved arm64 benchmarks compared to main:

    Notes Recent Score Orig Score Ubuntu arm64 Benchmark
    1.41 0.89 1.41
    0.89    		 System.Collections.ContainsTrue(String).Array(Size: 512)
    0.96 0.92 0.96
    0.92    		 System.Memory.Span(Char).LastIndexOfValue(Size: 512)
    0.93 0.93 0.93
    0.93    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock
    0.93 0.93 0.93
    0.93    		 System.Collections.Tests.Perf_PriorityQueue(Int32, Int32).HeapSort(Size: 1000)
    0.92 0.92 0.92
    0.92    		 System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: InvariantCultureIgno
    0.90 0.92 0.90
    0.92    		 System.Buffers.Text.Tests.Utf8ParserTests.TryParseInt64(value: 9223372036854775807)
    0.89 0.89 0.89
    0.89    		 System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64(value: 18446744073709551615)
    0.89 0.84 0.89
    0.84    		 Benchstone.BenchI.Fib.Test
    0.88 0.82 0.88
    0.82    		 System.Collections.Tests.Perf_PriorityQueue(Int32, Int32).Dequeue_And_Enqueue(Size: 100)
    0.87 0.87 0.87
    0.87    		 System.Collections.Tests.Perf_PriorityQueue(Int32, Int32).Enumerate(Size: 100)
    0.86 0.86 0.86
    0.86    		 System.Collections.ContainsTrue(Int32).Span(Size: 512)
    0.84 0.83 0.84
    0.83    		 System.Collections.IterateForEach(Int32).List(Size: 512)
    0.83 0.83 0.83
    0.83    		 System.Collections.ContainsTrue(Int32).ICollection(Size: 512)
    0.83 0.84 0.83
    0.84    		 System.Collections.ContainsTrue(Int32).ImmutableArray(Size: 512)
    0.82 0.82 0.82
    0.82    		 System.Collections.ContainsTrue(Int32).Queue(Size: 512)
    0.82 0.83 0.82
    0.83    		 System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
    0.82 0.83 0.82
    0.83    		 System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ
    0.82 0.83 0.82
    0.83    		 System.Collections.ContainsTrue(Int32).List(Size: 512)
    0.81 0.90 0.81
    0.90    		 System.Collections.IterateForEach(Int32).Dictionary(Size: 512)
    0.80 0.84 0.80
    0.84    		 System.Collections.ContainsTrue(Int32).Array(Size: 512)
    0.77 0.82 0.77
    0.82    		 System.Memory.Span(Char).IndexOfValue(Size: 512)
    0.77 0.76 0.77
    0.76    		 System.Memory.Span(Byte).SequenceCompareTo(Size: 4)
    0.76 0.87 0.76
    0.87    		 System.Memory.Span(Byte).IndexOfValue(Size: 512)
    0.76 0.75 0.76
    0.75    		 System.Memory.Span(Char).SequenceCompareToDifferent(Size: 33)
    0.75 0.83 0.75
    0.83    		 System.Collections.ContainsTrue(Int32).ImmutableSortedSet(Size: 512)
    0.73 0.82 0.73
    0.82    		 System.Collections.ContainsTrueComparer(Int32).ImmutableSortedSet(Size: 512)
    0.72 0.75 0.72
    0.75    		 System.Collections.IterateForEach(Int32).FrozenSet(Size: 512)
    0.71 0.90 0.71
    0.90    		 System.Collections.TryGetValueTrue(Int32, Int32).ImmutableSortedDictionary(Size: 512)
    0.71 0.88 0.71
    0.88    		 System.Collections.ContainsTrue(String).LinkedList(Size: 512)
    0.67 0.92 0.67
    0.92    		 System.Collections.TryGetValueTrue(Int32, Int32).ImmutableDictionary(Size: 512)
    0.67 0.65 0.67
    0.65    		 System.Collections.TryGetValueFalse(Int32, Int32).ImmutableDictionary(Size: 512)
    0.58 0.79 0.58
    0.79    		 System.Collections.ContainsTrueComparer(Int32).ImmutableHashSet(Size: 512)
    0.56 0.84 0.56
    0.84    		 System.Collections.ContainsFalse(Int32).ImmutableHashSet(Size: 512)
    0.53 0.73 0.53
    0.73    		 System.Collections.ContainsTrue(Int32).ImmutableHashSet(Size: 512)

    I don't see anything in the commit range for the regression in System.Collections.ContainsTrue.Array that points to layout. Everything else looks pretty stable.

  26. amanasifkhalid commented on Apr 12, 2025

    @amanasifkhalid
    amanasifkhalid
    on Apr 12, 2025
    Contributor

    I'm not putting a lot of weight into them due to all the JCC errata instances, but here are the Intel regressions:

    Notes Recent Score Orig Score TigerUbuntu TigerWIndows Benchmark
    1.56 1.19 1.56
    1.19    		 System.IO.Tests.Perf_StreamWriter.WriteCharArray(writeLength: 100)
    1.47 1.45 1.47
    1.45    		 System.Collections.Tests.Perf_BitArray.BitArrayRightShift(Size: 512)
    1.38 1.13 1.38
    1.13    		 System.IO.Tests.Perf_StreamWriter.WritePartialCharArray(writeLength: 100)
    1.34 1.16 1.34
    1.16    		 System.Linq.Tests.Perf_Enumerable.Select(input: IList)
    1.34 1.18 1.34
    1.18    		 Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 8, KeysCountPerProvider: 20)
    1.33 1.16 1.33
    1.16    		 System.Linq.Tests.Perf_Enumerable.Select(input: IEnumerable)
    1.30 1.07 1.30
    1.07    		 System.Buffers.Text.Tests.Base64Tests.ConvertToBase64CharArray(NumberOfBytes: 1000)
    1.29 1.16 1.29
    1.16    		 System.Buffers.Tests.ReadOnlySequenceTests(Char).IterateGetPositionTenSegments
    1.25 1.24 1.25
    1.24    		 System.Buffers.Tests.ReadOnlySequenceTests(Char).IterateForEachTenSegments
    1.16 1.10 1.16
    1.10    		 System.Text.Tests.Perf_Encoding.GetBytes(size: 512, encName: "utf-8")
    1.14 1.12 1.14
    1.12    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+", Options: None)
    1.13 1.12 1.13
    1.12    		 System.Numerics.Tests.Perf_VectorOf(UInt16).DivisionOperatorBenchmark
    1.13 1.08 1.13
    1.08    		 System.Buffers.Text.Tests.Base64Tests.Base64Encode(NumberOfBytes: 1000)
    1.12 1.10 1.12
    1.10    		 System.Numerics.Tests.Perf_VectorOf(Int16).DivisionOperatorBenchmark
    1.12 1.24 1.12
    1.24    		 System.Perf_Convert.FromBase64Chars
    1.12 1.12 1.12
    1.12    		 System.Numerics.Tests.Perf_VectorOf(Int16).DivideBenchmark
    1.12 1.11 1.12
    1.11    		 System.Numerics.Tests.Perf_VectorOf(UInt16).DivideBenchmark
    1.11 1.19 1.11
    1.19    		 System.Linq.Tests.Perf_Enumerable.WhereSelect(input: Array)
    1.11 1.09 1.11
    1.09    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[^\n]*", Options: None)
    1.11 1.12 1.11
    1.12    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{L}", Options: None)
    1.11 1.05 1.11
    1.05    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Ll}", Options: None)
    1.11 1.18 1.11
    1.18    		 System.Buffers.Text.Tests.Base64Tests.Base64EncodeDestinationTooSmall(NumberOfBytes: 1000)
    1.10 1.18 1.10
    1.18    		 System.Linq.Tests.Perf_Enumerable.WhereSingleOrDefault_LastElementMatches(input: IEnumerable)
    1.08 1.08 1.08
    1.08    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: None)
    1.08 1.18 1.08
    1.18    		 System.Linq.Tests.Perf_Enumerable.Take_All(input: IEnumerable)
    1.06 1.13 1.06
    1.13    		 System.IO.Tests.Perf_Path.GetDirectoryName
    1.06 2.45 1.06
    2.45    		 System.IO.Hashing.Tests.Crc32_AppendPerf.Append(BufferSize: 128)
    1.04 1.12 1.04
    1.12    		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: ".*", Options: None)
    1.03 1.08 1.03
    1.08    		 System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 32)
    1.03 1.22 1.03
    1.22    		 System.Linq.Tests.Perf_Enumerable.Select(input: Array)
    1.02 1.15 1.06
    1.16    		 0.99
    1.14    		 System.Collections.Perf_Frozen(ReferenceType).ToFrozenDictionary(Count: 64)
    1.02 1.08 1.02
    1.08    		 System.Perf_Convert.ChangeType
    1.02 1.07 1.02
    1.07    		 System.Collections.ContainsKeyFalse(Int32, Int32).FrozenDictionary(Size: 512)
    1.02 1.38 1.02
    1.38    		 System.IO.Tests.Perf_Path.GetFullPathNoRedundantSegments
    1.02 3.20 1.02
    3.20    		 System.IO.Hashing.Tests.Crc32_AppendPerf.Append(BufferSize: 10240)
    1.02 1.10 1.02
    1.10    		 System.Collections.IndexerSetReverse(Int32).List(Size: 512)
    1.01 2.64 1.01
    2.64    		 System.IO.Hashing.Tests.Crc32_AppendPerf.Append(BufferSize: 16)
    1.01 1.11 1.01
    1.11    		 System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)
    1.01 1.07 1.01
    1.07    		 System.Linq.Tests.Perf_Enumerable.Skip_One(input: IEnumerable)
    1.01 1.67 1.01
    1.67    		 System.Linq.Tests.Perf_Enumerable.SelectToList(input: IList)
    1.00 1.30 1.00
    1.30    		 System.Collections.Perf_Frozen(ReferenceType).ToFrozenDictionary(Count: 512)
    1.00 1.13 1.00
    1.13    		 System.Collections.IterateFor(String).IList(Size: 512)
    1.00 1.11 1.00
    1.11    		 System.Collections.CtorFromCollection(String).FrozenSet(Size: 512)
    1.00 1.09 1.00
    1.09    		 System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: IEnumerable)
    1.00 1.17 1.00
    1.17    		 System.Linq.Tests.Perf_Enumerable.SkipHalfTakeHalf(input: IEnumerable)
    1.00 1.21 1.04
    1.13    		 0.97
    1.29    		 System.Collections.Perf_Frozen(ReferenceType).ToFrozenSet(Count: 64)
    1.00 1.13 1.00
    1.13    		 System.Perf_Convert.FromBase64String
    0.99 1.15 0.99
    1.15    		 System.Linq.Tests.Perf_Enumerable.OrderBy(input: IEnumerable)
    0.99 1.18 0.99
    1.18    		 System.Linq.Tests.Perf_Enumerable.SelectToArray(input: IList)
    0.99 1.09 0.99
    1.09    		 System.Text.Json.Tests.Perf_Depth.ReadSpanEmptyLoop(Depth: 512)
    0.98 1.08 0.98
    1.08    		 System.Linq.Tests.Perf_Enumerable.ToList(input: IEnumerable)
    0.98 1.09 0.98
    1.09    		 System.Linq.Tests.Perf_Enumerable.Select(input: List)
    0.97 1.35 0.97
    1.35    		 System.IO.Tests.Perf_Path.GetFullPathForTypicalLongPath
    0.97 1.11 0.97
    1.11    		 System.Collections.Perf_Frozen(ReferenceType).TryGetValue_True(Count: 64)
    0.97 1.16 0.97
    1.16    		 System.Collections.AddGivenSize(Int32).HashSet(Size: 512)
    0.96 1.31 0.99
    1.34    		 0.93
    1.29    		 System.Linq.Tests.Perf_Enumerable.Contains_ElementNotFound(input: IEnumerable)
    0.96 1.13 0.96
    1.13    		 System.IO.Tests.Perf_Path.GetFullPathForLegacyLength
    0.95 1.29 0.95
    1.29    		 System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IOrderedEnumerable)
    0.94 1.13 0.94
    1.13    		 System.Collections.CtorFromCollection(String).FrozenDictionaryOptimized(Size: 512)
    0.93 1.21 0.93
    1.21    		 System.Collections.Perf_Frozen(ReferenceType).ToFrozenSet(Count: 512)
    0.92 1.11 0.92
    1.11    		 System.Collections.CtorFromCollection(String).ConcurrentDictionary(Size: 512)
    0.92 1.07 0.92
    1.07    		 System.Collections.Perf_Frozen(ReferenceType).Contains_True(Count: 4)
    0.91 1.15 0.91
    1.15    		 System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IOrderedEnumerable)
    0.90 1.08 0.90
    1.08    		 System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)
    0.90 1.22 0.90
    1.22    		 System.Buffers.Tests.SearchValuesCharTests.LastIndexOfAny(Values: "abcdefABCDEF0123456789Ü")
    0.87 1.39 0.87
    1.39    		 System.Collections.Sort(IntStruct).LinqOrderByExtension(Size: 512)
    0.86 1.72 0.86
    1.72    		 GuardedDevirtualization.ThreeClassVirtual.Call(testInput: pB=0.33 pD=0.33)
    0.86 1.76 0.86
    1.76    		 System.Collections.Sort(IntStruct).Array_ComparerClass(Size: 512)
    0.85 1.92 0.85
    1.92    		 System.Collections.Sort(IntStruct).Array_Comparison(Size: 512)
    0.84 1.24 0.84
    1.24    		 System.Linq.Tests.Perf_Enumerable.SelectToList(input: Array)
    0.75 1.12 0.75
    1.12    		 System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 8000)
    0.74 1.13 0.74
    1.13    		 System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 8000)
    0.68 1.28 0.68
    1.28    		 System.Collections.Sort(IntStruct).LinqQuery(Size: 512)

    The System.IO.Tests.Perf_StreamWriter regressions are probably from #113715. For the remaining ones, some never recovered from before RPO layout, while others have spiked up and down as we churned the layout through .NET 10. In spite of the JCC errata instances, most of these benchmarks seem to have improved.

  27. amanasifkhalid commented on Apr 12, 2025

    @amanasifkhalid
    amanasifkhalid
    on Apr 12, 2025
    Contributor

    Since many of the regressed benchmarks have returned or improved upon their baseline on non-Intel platforms, and remaining regressions are tracked, I'm going to close this.

  28. amanasifkhalid
    closed this as completedon Apr 12, 2025
  29. github-actions
    locked and limited conversation to collaborators on May 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions