analytics: batch api calls by iMichka · Pull Request #24561 · Homebrew/brew

iMichka · 2026-11-10T22:32:44Z

We have more than 1 billion API calls per month to store analytics. When sending this through a CDN, as we do right now, this is really expensive (even above flat-rate premium plans from AWS)

Batch API calls, so that for example if you run a "brew install" with many dependencies, the number of events is reduced

In InfluxDB line protocol each line is one point, and the format defines \n as the separator between points.

/api/v2/write, the endpoint brew posts to, is built for batches: InfluxDB recommends sending up to about 5,000 lines per request. A batch of 10 is well within that.
InfluxDB 3 accepts the same /api/v2/write request, so this holds even if the server moves to v3.

Have you followed our Contributing guidelines?
Have you checked for other open Pull Requests for the same change?
Have you explained what your changes do? Performance claims (e.g. "this is faster") must include brew benchmark results.
Have you explained why you'd like these changes included, not just what they do?
For bug fixes, have you given step-by-step brew commands to reproduce the bug?
Have you written new tests (excluding integration tests)? Here's an example.
Have you successfully run brew lgtm (style, typechecking and tests) locally?

I did not use AI/LLM to create this PR, or I disclosed the tool/model below and reviewed its output; every commit has a human author or co-author and I will answer maintainer questions and review comments myself without AI/LLM.

I have used Claude Opus 5.5 for this

iMichka · 2026-11-10T22:34:00Z

This is a quick implementation I tested. Batch size set to 10 is arbitrary

@SMillerDev

iMichka · 2026-11-10T22:34:46Z

I won't be able to work on this tomorrow so please test / give feedback (and pick the PR up if you are faster than me to respond) :)

coderabbitai · 2026-11-10T22:39:07Z

Summary

Summary by CodeRabbit

Improvements
- Analytics events are now sent in batches, reducing the number of individual reporting requests.
- Any remaining queued events are sent when the process exits, helping ensure they are included in reporting.

Influx analytics events are queued and sent in batches of ten. A process-local queue resets when inherited by another process, and the code registers a flush at process exit.

Changes: Influx analytics batching

Layer / File(s)	Summary
Queue, batch, and flush lifecycle `Library/Homebrew/utils/analytics.rb`, `Library/Homebrew/test/utils/analytics_spec.rb`	`report_influx` queues formatted events and flushes when the queue reaches ten lines. `flush_influx_lines` sends queued records as newline-separated data. The queue resets when the process ID changes, and the code registers a process-exit flush. The updated examples flush queued events after reporting.

Merge Risk: Low · up to 2bc13

Analytics batching mostly works, and the earlier test breakage is fixed. Two small edge cases remain: an event may be sent after a user opts out in the same run, and commands that replace the Ruby process may lose their analytics event. Both are bounded, so the change is mergeable with follow-up.

Pre-merge checks: 4

Check name	Status	Explanation
Title check		The title clearly and concisely describes the main change: batching analytics API calls.
Description check		The description explains the purpose of batching analytics API calls, the expected cost reduction, and the InfluxDB batching rationale.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

coderabbitai

Actionable comments posted: 1

Review info

Run configuration

Source mode: Auto

Repository: Unavailable

Branch: Unavailable

File: No configuration file from this repository applied

Commit: Unavailable

Layers: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: `7bfb937e-c135-4173-8ff0-ac34f91c9a60`

Commits

Reviewing files that changed from the base of the PR and between `95d4938` and `8cd1d38`.

Files selected for processing: 1

`Library/Homebrew/utils/analytics.rb`

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

We have more than 1 billion API calls per month to store analytics. When sending this through a CDN, as we do right now, this is really expensive (even above flat-rate premium plans from AWS) Batch API calls, so that for example if you run a "brew install" with many dependencies, the number of events is reduced Even if we decide to go away from the CDN solution, reducing the number of calls is worthwile.

iMichka · 2026-11-11T07:53:20Z

BTW some more information about this idea:

In InfluxDB line protocol each line is one point, and the format defines \n as the separator between points.

/api/v2/write, the endpoint brew posts to, is built for batches: InfluxDB recommends sending up to about 5,000 lines per request. A batch > of 10 is well within that.

InfluxDB 3 accepts the same /api/v2/write request, so this holds even if the server moves to v3.

coderabbitai

Actionable comments posted: 2

Review info

Run configuration

Source mode: Auto

Repository: Unavailable

Branch: Unavailable

File: No configuration file from this repository applied

Commit: Unavailable

Layers: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: `cf4476a1-8228-4e65-b3b6-a6160c388e1d`

Commits

Reviewing files that changed from the base of the PR and between `8cd1d38` and `504ca3e`.

Files selected for processing: 4

`Library/Homebrew/env_config.rb`

`Library/Homebrew/sorbet/rbi/dsl/homebrew/env_config.rbi`

`Library/Homebrew/test/utils/analytics_spec.rb`

`Library/Homebrew/utils/analytics.rb`

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

coderabbitai · 2026-11-11T07:56:13Z

+        lines = pending_influx_lines
+        return if lines.empty?


Minor · Security & Privacy · Quick win

Discard queued events when analytics has been disabled.

With analytics initially enabled, brew analytics off queues its command event before running disable!. Command reporting precedes command execution in Library/Homebrew/brew.rb, Lines 132-134. The off subcommand calls disable! in Library/Homebrew/analytics/subcommand/off.rb, Lines 19-20. (raw.githubusercontent.com)

The exit callback then sends that event without checking the updated opt-out state. Before dispatch, recheck disabled? and not_this_run?. Clear the queue without sending when either check is true.

coderabbitai · 2026-11-11T07:56:13Z

+        end
+        @flush_influx_lines_at_exit ||= T.let(
+          begin
+            at_exit { flush_influx_lines }


Minor · Data Integrity & Integration · Quick win

Flush pending events before process replacement.

Command reporting runs before command execution in Library/Homebrew/brew.rb, Lines 132-134. However, brew ruby calls exec in Library/Homebrew/dev-cmd/ruby.rb, Lines 30-33. (raw.githubusercontent.com)

With the default batch size, its command event remains queued. Successful exec replaces the Ruby process, so this exit callback cannot send the event. Flush before supported process-replacement paths; retain the exit callback for normal termination. (docs.ruby-lang.org)

Yes, this is something we'll need to do. Honestly I'd consider something like a Thread as well/instead and just flush after no more events after a given period of time.

@MikeMcQuaid An idle-time flush could send partial batches during long-running commands.

It would not replace a flush before exec: the process could be replaced before the timer fires. If we add a thread, the queue also needs synchronisation and fork-aware thread state. The exit flush can still handle normal termination.

_{You are interacting with an AI system.}

MikeMcQuaid

Thanks, good idea, few thoughts.

MikeMcQuaid · 2026-11-11T08:48:18Z

+        end
+        @flush_influx_lines_at_exit ||= T.let(
+          begin
+            at_exit { flush_influx_lines }


Yes, this is something we'll need to do. Honestly I'd consider something like a Thread as well/instead and just flush after no more events after a given period of time.

MikeMcQuaid · 2026-11-11T08:49:44Z

+
+      # Sends all queued events in a single request, one line protocol record per line.
+      sig { void }
+      def flush_influx_lines


I wonder if this approach could be simplified a bit and instead just get all the curl calls per e.g. brew install run and send them at once i.e. optimise the callers rather than change the underlying behaviour. That would allow both potentially larger batches and avoid the exit handling which is unlikely to be as reliable as the current approach.

We have more than 1 billion API calls per month to store analytics. When sending this through a CDN, as we do right now, this is really expensive (even above flat-rate premium plans from AWS) Batch API calls, so that for example if you run a "brew install" with many dependencies, the number of events is reduced

coderabbitai

Nitpick comments: 1

Library/Homebrew/test/utils/analytics_spec.rb: 1

151-151: Trivial · Maintainability & Code Quality · Quick win

Test the automatic batch boundary.

Add a test that checks no request is sent for nine events and that the tenth sends one request containing all ten newline-separated records.

Review info

Run configuration

Source mode: Auto

Repository: Unavailable

Branch: Unavailable

File: No configuration file from this repository applied

Commit: Unavailable

Layers: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: `d75cd06e-d687-4c32-a76a-260e768fc691`

Commits

Reviewing files that changed from the base of the PR and between `0bcf90c` and `2bc134f`.

Files selected for processing: 2

`Library/Homebrew/test/utils/analytics_spec.rb`

`Library/Homebrew/utils/analytics.rb`

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

coderabbitai Bot reviewed Nov 10, 2026

View reviewed changes

Comment thread Library/Homebrew/utils/analytics.rb Resolved

iMichka force-pushed the analytics branch from 8cd1d38 to 504ca3e Compare November 11, 2026 07:50

coderabbitai Bot reviewed Nov 11, 2026

View reviewed changes

iMichka force-pushed the analytics branch from 504ca3e to 0bcf90c Compare November 11, 2026 08:16

MikeMcQuaid reviewed Nov 11, 2026

View reviewed changes

iMichka force-pushed the analytics branch from 0bcf90c to 2bc134f Compare November 11, 2026 09:03

coderabbitai Bot reviewed Nov 11, 2026

View reviewed changes

Conversation

iMichka commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iMichka commented Nov 10, 2026

Uh oh!

iMichka commented Nov 10, 2026

Uh oh!

coderabbitai Bot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iMichka commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MikeMcQuaid Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

MikeMcQuaid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MikeMcQuaid Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

MikeMcQuaid Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

iMichka commented Nov 10, 2026 •

edited

Loading

coderabbitai Bot commented Nov 10, 2026 •

edited

Loading

iMichka commented Nov 11, 2026 •

edited

Loading

coderabbitai Bot Nov 11, 2026 •

edited

Loading