Repository navigation
build(python): make numpy an optional dependency - #11084
Draft
ZephyrYWZhou wants to merge 1 commit into
Draft
ZephyrYWZhou wants to merge 1 commit into
ZephyrYWZhou wants to merge 1 commit into
Conversation
pyarrow dropped its required numpy dependency in 18.0, so pylance no longer needs numpy as a hard dependency. - Move numpy out of required deps into a new `numpy` optional extra, and raise the pyarrow floor to >=18 (the first release where pyarrow does not pull numpy in). - Route the two remaining direct `import numpy` statements (indices/builder.py, _dataset/scanner.py) through the existing lazy `_LazyModule` helper, matching pandas/polars/torch. - Gate the isinstance/issubclass numpy checks in _dataset/scanner.py and pydantic.py behind `_check_for_numpy` so they do not trigger the lazy proxy when numpy is absent. - Add a test asserting `import lance` works with numpy absent and that numpy-only paths raise a clear error.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1216.
numpy was a required dependency of pylance mainly because pyarrow required it. pyarrow dropped its required numpy dependency in 18.0 and has kept it optional through the current 26.x, so pylance can make numpy optional too.
Changes
pyproject.toml: movenumpyout of required dependencies into a newnumpyoptional extra. Bump the pyarrow floor to>=18, the first release that does not pull numpy in.import numpystatements (indices/builder.py,_dataset/scanner.py) through the existing lazy_LazyModulehelper inlance/dependencies.py, matching how pandas/polars/torch are already handled. Most modules were already lazy.isinstance/issubclassnumpy checks in_dataset/scanner.pyandpydantic.pybehind_check_for_numpy, so they don't trigger the lazy proxy when numpy is absent.test_numpy_optional.py:import lanceworks with numpy absent, and a numpy-only path raises a clear error.Genuine numpy paths (vector math, sampling, kmeans) still need numpy and will raise a helpful error if it's missing. Install with
pip install pylance[numpy]to get it.One open question
pyproject previously allowed
pyarrow>=14, but numpy-free only holds onpyarrow>=18. This PR bumps the floor to 18. Keeping>=14and instead documenting that numpy-free needs pyarrow 18+ is also an option, if the community prefers that.Testing
test_numpy_optional.py(blocks numpy import in a subprocess).Opening as a draft for early feedback.