G
YouTube TikTok Games
Skip to content

fix(python): write Enum fields of Pydantic models - #11083

Open
maharanay22 wants to merge 2 commits into
lance-format:mainfrom
maharanay22:fix-pydantic-enum-write
Open

maharanay22 wants to merge 2 commits into
lance-format:mainfrom
maharanay22:fix-pydantic-enum-write

Conversation

@maharanay22

Copy link
Copy Markdown

Summary

Writing Pydantic model instances with a plain Enum field failed. pydantic_to_schema maps the field to the type of the enum's values (a dictionary of strings for string values), but the rows come from model_dump(), which keeps the enum members. Arrow can only convert members that also subclass str or int, so lance.write_dataset(models, uri) and LanceDataset.from_pydantic_model raised ArrowTypeError: Expected bytes, got a 'Color' object for a model like this.

class Color(enum.Enum):
    RED = "red"
    BLUE = "blue"

class Item(pydantic.BaseModel):
    color: Color

lance.write_dataset([Item(color=Color.BLUE)], "/tmp/items.lance")

model_to_dict now replaces each enum member with its value, including inside nested models and lists, which matches the schema the model maps to.

Testing

Added test_pydantic_enum_fields_store_enum_values to test_dataset.py for both write_dataset and from_pydantic_model, with a string enum, an integer enum and an enum inside a nested model list. It fails on main and passes with the fix. The existing Pydantic tests in test_dataset.py and test_pydantic.py still pass, and ruff format --check and ruff check are clean on the touched files.

pydantic_to_schema stores an Enum field as the type of its values, but
model_dump keeps the Enum members. Arrow cannot convert members of a
plain Enum, so write_dataset and LanceDataset.from_pydantic_model failed
with ArrowTypeError for any model with such a field. Store each member's
value, including in nested models and lists.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-python Python bindings bug Something isn't working labels Nov 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 14, 2026
Walking every element visited each float of embedding vectors, making
Pydantic writes several times slower even without enums. Return lists of
plain scalars unchanged, as lancedb#4357 does.
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Nov 14, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The scalar-list fast path fixes the previous vector-ingestion regression. Enum values still round-trip through both write entry points, including nested models and lists, and vector ingestion is back near its previous throughput.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant