G
YouTube TikTok Games
Skip to content

Tracking issue for CI build timeouts #76454

New issue
New issue
Closed
Closed
Tracking issue for CI build timeouts#76454
Labels
area-Infrastructureblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Milestone
9.0.0

Description

@jkotas
jkotas
opened on Sep 30, 2022
 {
    "ErrorMessage" : "##[error]The operation was canceled.",
    "BuildRetry": false
 }

See dotnet/arcade#11072 for more context.

Report

Build Definition Step Name Console log Pull Request
2409598 dotnet-runtime Build product Log
610058 dotnet/runtime Send to Helix Log #97792
610152 dotnet/runtime Send to Helix Log
610011 dotnet/runtime Send tests to Helix (Unix) Log #99992
609972 dotnet/runtime Send tests to Helix (Unix) Log #100014
609923 dotnet/runtime Send to Helix Log
609966 dotnet/runtime Send to Helix Log #100018
609950 dotnet/runtime Send to Helix Log #99608
609925 dotnet/runtime Send to Helix Log #100016
609818 dotnet/runtime Send tests to Helix (Unix) Log #100011
609824 dotnet/runtime Send tests to Helix (Unix) Log #100012
609832 dotnet/runtime Send tests to Helix (Unix) Log #99859
609788 dotnet/runtime Send tests to Helix (Unix) Log #99953
609791 dotnet/runtime Send to Helix Log #99909
609762 dotnet/runtime Send to Helix Log #99371
2409365 dotnet-runtime Send job to Helix (Windows) Log
609735 dotnet/runtime Send to Helix Log
609729 dotnet/runtime Send to Helix Log
609736 dotnet/runtime Send to Helix Log
609730 dotnet/runtime Send to Helix Log
609709 dotnet/runtime Send to Helix Log
609711 dotnet/runtime Send to Helix Log
609742 dotnet/runtime Send to Helix Log
609715 dotnet/runtime Send to Helix Log
609716 dotnet/runtime Send to Helix Log
609720 dotnet/runtime Send to Helix Log
609701 dotnet/runtime Send tests to Helix (Windows) Log #99778
609693 dotnet/runtime Send tests to Helix (Windows) Log #100004
609667 dotnet/runtime Send to Helix Log #99608
609662 dotnet/runtime Send to Helix Log #100005
609653 dotnet/runtime Send to Helix Log #100001
609637 dotnet/runtime Send to Helix Log #100002
609610 dotnet/runtime Send to Helix Log #99972
609607 dotnet/runtime Send to Helix Log #100000
609604 dotnet/runtime Send tests to Helix (Unix) Log #99999
609600 dotnet/runtime Send to Helix Log #99998
609196 dotnet/runtime Send tests to Helix (Unix) Log #99985
609587 dotnet/runtime Send tests to Helix (Unix) Log
609327 dotnet/runtime Send to Helix Log
609031 dotnet/runtime Send tests to Helix (Unix) Log #99688
608673 dotnet/runtime Send to Helix Log #99836
609589 dotnet/runtime Send to Helix Log
609577 dotnet/runtime Send to Helix Log
609546 dotnet/runtime Send tests to Helix (Unix) Log #99996
608721 dotnet/runtime Send tests to Helix (Unix) Log #99956
608715 dotnet/runtime Send to Helix Log #99873
608540 dotnet/runtime Send tests to Helix (Windows) Log #99908
609321 dotnet/runtime Send to Helix Log
609516 dotnet/runtime Send tests to Helix (Unix) Log #99995
2409150 dotnet-runtime Send job to Helix (Windows) Log
609454 dotnet/runtime Send tests to Helix (Unix) Log #99849
609442 dotnet/runtime Send tests to Helix (Unix) Log #99982
609436 dotnet/runtime Send tests to Helix (Unix) Log #97529
609405 dotnet/runtime Send to Helix Log #99183
608686 dotnet/runtime Send to Helix Log #99973
609391 dotnet/runtime Send to Helix Log #99992
609368 dotnet/runtime Send tests to Helix (Windows) Log #99991
608833 dotnet/runtime Send to Helix Log
609361 dotnet/runtime Send to Helix Log #99990
609316 dotnet/runtime Send tests to Helix (Unix) Log
609355 dotnet/runtime Send to Helix Log #99662
609318 dotnet/runtime Send tests to Helix (Unix) Log
609323 dotnet/runtime Send tests to Helix (Unix) Log
609345 dotnet/runtime Send tests to Helix (Unix) Log #99784
609324 dotnet/runtime Send tests to Helix (Unix) Log
2408902 dotnet-runtime Send job to Helix (Windows) Log
609315 dotnet/runtime Send tests to Helix (Unix) Log
609332 dotnet/runtime Send tests to Helix (Unix) Log #99970
609329 dotnet/runtime Send to Helix Log #99989
609312 dotnet/runtime Send tests to Helix (Unix) Log
2408912 dotnet-runtime Build product Log
609326 dotnet/runtime Send to Helix Log
609320 dotnet/runtime Send to Helix Log
609229 dotnet/runtime Send to Helix Log #99987
609199 dotnet/runtime Send tests to Helix (Unix) Log #99986
609077 dotnet/runtime Send tests to Helix (Unix) Log
609073 dotnet/runtime Send tests to Helix (Unix) Log
2408699 dotnet-runtime Send job to Helix (Windows) Log
609163 dotnet/runtime Send to Helix Log #99983
608351 dotnet/runtime Send tests to Helix (Unix) Log #99958
609089 dotnet/runtime Send to Helix Log #99584
609081 dotnet/runtime Send to Helix Log
609092 dotnet/runtime Send to Helix Log #99584
609062 dotnet/runtime Send to Helix Log #99981
609059 dotnet/runtime Send to Helix Log #99981
609069 dotnet/runtime Send tests to Helix (Unix) Log
2408649 dotnet-runtime Send job to Helix (Windows) Log
608477 dotnet/runtime Send to Helix Log #99889
609048 dotnet/runtime Build native test components Log #99980
609053 dotnet/runtime Build product Log #99980
609003 dotnet/runtime Send to Helix Log #99909
609026 dotnet/runtime Send to Helix Log #99781
609080 dotnet/runtime Send to Helix Log
608992 dotnet/runtime Send to Helix Log #99183
608987 dotnet/runtime Send tests to Helix (Unix) Log #99566
608889 dotnet/runtime Send tests to Helix (Windows) Log #99468
608897 dotnet/runtime Send to Helix Log
608843 dotnet/runtime Send to Helix Log #99955
608820 dotnet/runtime Send to Helix Log #99976
608830 dotnet/runtime Send tests to Helix (Unix) Log
Displaying 100 of 670 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
98 258 662

Known issue validation

Build: 🔎
Result validation: ⚠️ Provided build not found. Provide a valid build in the "Build: 🔎" line.
Validation performed at: 11/30/2023 9:39:02 PM UTC

Activity

  1. jkotas
    added
    area-Infrastructure
    Known Build ErrorUse this to report build issues in the .NET Helix tab
    on Sep 30, 2022
  2. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Sep 30, 2022
  4. jkotas
    added
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    on Oct 1, 2022
  5. jeffschwMSFT
    added this to the 8.0.0 milestone on Oct 4, 2022
  6. jeffschwMSFT
    removed
    untriagedNew issue has not been triaged by the area owner
    on Oct 4, 2022
  7. hoyosjs
    added a commit that references this issue on Feb 22, 2023

    Revert update dependencies from dotnet/roslyn (#82466)

    98bf267

  8. kg commented on Feb 23, 2023

    @kg
    kg
    on Feb 23, 2023
    Contributor

    #82476 can be ignored, it's intentionally running tests in a slower mode to flush out failures

  9. janvorli commented on Apr 27, 2023

    @janvorli
    janvorli
    on Apr 27, 2023
    Member

    I have noticed that the osx-x64 Release NativeAOT timeouted in my PR #85366, but from its log, it is obvious that it has completed running (https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/252395/logs/1833). The leg took few seconds over two hours. Maybe we could bump the timeout just a bit.

  10. jkotas commented on Apr 27, 2023

    @jkotas
    jkotas
    on Apr 27, 2023
    MemberAuthor

    I have noticed that the osx-x64 Release NativeAOT timeouted in my PR #85366, but from its log, it is obvious that it has completed running (https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/252395/logs/1833)

    The log shows that the leg was waiting for 1+ hour for a Helix machine. This happens when we are sending more work to a Helix queue than what it has capacity for. Increasing the timeouts is not going to fix it.

  11. janvorli commented on Apr 27, 2023

    @janvorli
    janvorli
    on Apr 27, 2023
    Member

    Ah, I can see it now, I've missed that.

  12. hoyosjs commented on Apr 27, 2023

    @hoyosjs
    hoyosjs
    on Apr 27, 2023
    Member

    Looking at this, our queue depth is not looking great for OSX 12.

    image

    Should we be trying to spill some work to 13 and 11?

    cc: @dotnet/runtime-infrastructure @dotnet/dnceng

  13. hoyosjs commented on Apr 27, 2023

    @hoyosjs
    hoyosjs
    on Apr 27, 2023
    Member

    Hmm, These are the largest work producers:

    image

    With PRs like #85389 #85343 being a ton of the work.

  14. hoyosjs commented on Apr 27, 2023

    @hoyosjs
    hoyosjs
    on Apr 27, 2023
    Member

    image

    The runtime time distribution seems to be consistent over the last couple weeks. We are 263043/307320 workitems sent over the last 3 days to the queue. That being said. It looks like in terms of workitem compute time, it looks like SDK is sending very long lived workitems what makes SDK take so much compute time is the different branches.

  15. 2 remaining items

  16. lambdageek commented on Jun 16, 2023

    @lambdageek
    lambdageek
    on Jun 16, 2023
    Member

    There's another issue about the recent AlmaLinux timeouts #87667 - the AlmaLinux leg has been removed by #87668

  17. agocke
    modified the milestones: 8.0.0, 9.0.0 on Sep 5, 2023
  18. build-analysis
    removed this from the 9.0.0 milestone on Nov 15, 2023
  19. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Nov 15, 2023
  20. akoeplinger
    added this to the 9.0.0 milestone on Nov 24, 2023
  21. ghost removed
    untriagedNew issue has not been triaged by the area owner
    on Nov 24, 2023

  22. rzikm commented on Nov 30, 2023

    @rzikm
    rzikm
    on Nov 30, 2023
    Member

    This issue seems to match also generic test timeout failures, such as

    System.Net.Http.HttpRequestException : Requesting HTTP version 3.0 with version policy RequestVersionExact while unable to establish HTTP/3 connection.
---- System.OperationCanceledException : The operation was canceled.

    is that the intent here?

  23. jkotas commented on Nov 30, 2023

    @jkotas
    jkotas
    on Nov 30, 2023
    MemberAuthor

    is that the intent here?

    It is not the indent. I have tweaked the matching string.

  24. JulieLeeMSFT
    removed
    Known Build ErrorUse this to report build issues in the .NET Helix tab
    on Mar 20, 2024

  25. JulieLeeMSFT commented on Mar 20, 2024

    @JulieLeeMSFT
    JulieLeeMSFT
    on Mar 20, 2024
    Member

    Removing Known Build Error label because it is marking Build Analysis green. We want developers either to wait until infra timeout issue is resolved and rerun their tests or to use their judgement to use the escape path to merge.

  26. JulieLeeMSFT commented on Mar 20, 2024

    @JulieLeeMSFT
    JulieLeeMSFT
    on Mar 20, 2024
    Member

    Known Build Error label is useful to get stats on the infra issues such as timeout. On the other hand, it allows PRs to get merged whle tests are not fully run. So, it is something we need to think about how we should handle such cases.

  27. jkotas commented on Mar 20, 2024

    @jkotas
    jkotas
    on Mar 20, 2024
    MemberAuthor

    Known Build Error label is useful to get stats on the infra issues such as timeout

    This was the sole purpose of this tracking issue. If we do not have it marked as Known Build Error, it can be closed.

  28. jkotas
    closed this as completedon Mar 20, 2024
  29. github-actions
    locked and limited conversation to collaborators on Apr 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-Infrastructureblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions