feat: Support table format: Iceberg, Delta, and Hudi #5650

HaoXuAI · 2025-10-05T21:47:54Z

What this PR does / why we need it:

examples:

  from feast.table_format import IcebergFormat
  from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource

  # Create Iceberg format with snapshot option
  iceberg_format = IcebergFormat()
  iceberg_format.set_property("snapshot-id", "your_snapshot_id")

  spark_source = SparkSource(
      name="iceberg_features",
      path="my_catalog.my_database.my_table",  # Catalog table name
      table_format=iceberg_format
  )

  Generated Spark code:
  spark.read.format("iceberg").option("snapshot-id", "your_snapshot_id").load("my_catalog.my_database.my_table")

  ✅ Delta Example:

  from feast.table_format import DeltaFormat

  delta_format = DeltaFormat()
  delta_format.set_property("versionAsOf", "1")

  spark_source = SparkSource(
      name="delta_features",
      path="s3://bucket/delta-table",
      table_format=delta_format
  )

  Generated Spark code:
  spark.read.format("delta").option("versionAsOf", "1").load("s3://bucket/delta-table")

  ✅ Hudi Example:

  from feast.table_format import HudiFormat

  hudi_format = HudiFormat()
  hudi_format.set_property("hoodie.datasource.query.type", "snapshot")
  hudi_format.set_property("as.of.instant", "20231010120000")

  spark_source = SparkSource(
      name="hudi_features",
      path="s3://bucket/hudi-table",
      table_format=hudi_format
  )

  Generated Spark code:
  spark.read.format("hudi")\
    .option("hoodie.datasource.query.type", "snapshot")\
    .option("as.of.instant", "20231010120000")\
    .load("s3://bucket/hudi-table")

Which issue(s) this PR fixes:

Misc

Signed-off-by: HaoXuAI <[email protected]>

HaoXuAI · 2025-10-05T22:29:38Z

protos/feast/core/DataSource.proto

    string date_partition_column_format = 5;
+
+    // Table Format (e.g. iceberg, delta, etc)
+    string table_format = 6;


TODO, create TableFormat proto, consolidate with FileFormat proto

tokoko · 2025-10-06T09:58:50Z

+1 on the inclusion of all 3 formats. Still I think we might be able to better design data-source side such that data source definitions don't tie the sources to specific offline stores. For example right now FileSource already supports specifying delta as a format... I don't like that design either because delta and especially iceberg might require more configuration that plain files, but the upside in that design was that Source class wasn't tied to a single offline store and it is actually usable from both dask and duckdb offline stores. Hiding formats behind SparkSource will mean that when a user stores a data source instance in the registry only spark offline store can be used to retrieve the data.

I think we can have best of both worlds if we instead go for adding all these formats as separate independent data sources (DeltaSource, IcebergSource, HudiSource) and then simply teach spark engine (and in the future others) how to read it.

franciscojavierarceo · 2025-10-06T12:21:05Z

sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py

            query: The query to be executed in Spark.
            path: The path to file data.
-            file_format: The format of the file data.
+            file_format: The underlying file format (parquet, avro, csv, json).


why not consolidate now?

franciscojavierarceo · 2025-10-06T12:22:28Z

I think we can have best of both worlds if we instead go for adding all these formats as separate independent data sources (DeltaSource, IcebergSource, HudiSource) and then simply teach spark engine (and in the future others) how to read it.

+1

HaoXuAI · 2025-10-06T22:58:09Z

@franciscojavierarceo @tokoko consolidation with FilleFormat and new data sources could break the backward compatibility, so I want to do it pace by pace.

franciscojavierarceo · 2025-10-07T00:02:09Z

That makes sense

tokoko · 2025-10-07T08:46:29Z

@HaoXuAI Why would new data sources break backwards compatibility though?

HaoXuAI · 2025-10-09T05:01:50Z

@HaoXuAI Why would new data sources break backwards compatibility though?

There will be some proto changes, no 100% sure if there will be API changes exposed to users but I think might be the case

Signed-off-by: HaoXuAI <[email protected]>

HaoXuAI · 2025-10-10T05:11:42Z

@franciscojavierarceo @ntkathole mind take a look

franciscojavierarceo

@HaoXuAI i don't see use actually using or testing Spark Table, Iceberg, or Hudi format's outside of our definitions, can you add that?

Can you also add documentation that these formats are now supported?

Otherwise lgtm.

HaoXuAI · 2025-10-14T21:20:25Z

@HaoXuAI i don't see use actually using or testing Spark Table, Iceberg, or Hudi format's outside of our definitions, can you add that?

Can you also add documentation that these formats are now supported?

Otherwise lgtm.

HaoXuAI · 2025-10-14T21:21:36Z

@HaoXuAI i don't see use actually using or testing Spark Table, Iceberg, or Hudi format's outside of our definitions, can you add that?
Can you also add documentation that these formats are now supported?
Otherwise lgtm.

Gonna update to add the TableFormat proto in the next PR, after that I'll add the docs. And I think the test will need to be changed as well.

Signed-off-by: hao-xu5 <[email protected]>

HaoXuAI · 2025-11-06T01:49:49Z

@franciscojavierarceo mind take another look?

franciscojavierarceo

Lgtm

* add support for table format such as Iceberg, Delta, Hudi etc. Signed-off-by: HaoXuAI <[email protected]> * linting Signed-off-by: HaoXuAI <[email protected]> * linting Signed-off-by: HaoXuAI <[email protected]> * add tests Signed-off-by: HaoXuAI <[email protected]> * fix tests Signed-off-by: HaoXuAI <[email protected]> * fix tests Signed-off-by: HaoXuAI <[email protected]> * linting Signed-off-by: HaoXuAI <[email protected]> * add tableformat proto Signed-off-by: hao-xu5 <[email protected]> * update Signed-off-by: hao-xu5 <[email protected]> * update doc Signed-off-by: hao-xu5 <[email protected]> * fix linting Signed-off-by: hao-xu5 <[email protected]> * fix test Signed-off-by: hao-xu5 <[email protected]> --------- Signed-off-by: HaoXuAI <[email protected]> Signed-off-by: hao-xu5 <[email protected]> Co-authored-by: hao-xu5 <[email protected]>

# [0.57.0](v0.56.0...v0.57.0) (2025-11-13) ### Bug Fixes * Improve trino to feast type mapping with (real,varchar,timestamp,decimal) ([#5691](#5691)) ([f855ad2](f855ad2)) * Materialize API - ODFV views not looked-up (thinks views non existant) - crashes materialize ([#5716](#5716)) ([1b050b3](1b050b3)) * Support historical feature retrieval with start_date/end_date in RemoteOfflineStore ([#5703](#5703)) ([ad32756](ad32756)) * Thread safe Clickhouse offline store ([#5710](#5710)) ([5f446ed](5f446ed)) ### Features * Add annotations to cronjob CRDs ([#5701](#5701)) ([be6e6c2](be6e6c2)) * Add batch commit mode for MySQL OnlineStore ([#5699](#5699)) ([3cfe4eb](3cfe4eb)) * Add possibility to materialize only latest values, to increase performance ([#5713](#5713)) ([8d77b72](8d77b72)) * Support table format: Iceberg, Delta, and Hudi ([#5650](#5650)) ([2915ad1](2915ad1))

HaoXuAI added 3 commits October 5, 2025 14:42

add support for table format such as Iceberg, Delta, Hudi etc.

8ae5ed5

Signed-off-by: HaoXuAI <[email protected]>

linting

456b97f

Signed-off-by: HaoXuAI <[email protected]>

linting

afa1444

Signed-off-by: HaoXuAI <[email protected]>

HaoXuAI requested a review from a team as a code owner October 5, 2025 21:47

HaoXuAI changed the title ~~feat: Support table format~~ feat: Support table format: Iceberg, Delta, and Hudi Oct 5, 2025

HaoXuAI added the ok-to-test label Oct 5, 2025

HaoXuAI added 2 commits October 5, 2025 15:08

add tests

7a34eaf

Signed-off-by: HaoXuAI <[email protected]>

fix tests

99f85b1

Signed-off-by: HaoXuAI <[email protected]>

HaoXuAI commented Oct 5, 2025

View reviewed changes

HaoXuAI requested review from franciscojavierarceo and ntkathole October 5, 2025 22:34

franciscojavierarceo reviewed Oct 6, 2025

View reviewed changes

HaoXuAI added 3 commits October 8, 2025 23:01

fix tests

eb15424

Signed-off-by: HaoXuAI <[email protected]>

Merge branch 'master' into support-table-format

05e3a65

linting

b11c083

Signed-off-by: HaoXuAI <[email protected]>

HaoXuAI requested a review from franciscojavierarceo October 10, 2025 04:47

Merge branch 'master' into support-table-format

021f164

franciscojavierarceo requested changes Oct 10, 2025

View reviewed changes

HaoXuAI closed this Oct 14, 2025

HaoXuAI reopened this Oct 14, 2025

Merge branch 'master' into support-table-format

48b166b

HaoXuAI and others added 7 commits October 29, 2025 21:36

Merge branch 'master' into support-table-format

429958a

add tableformat proto

e19467d

Signed-off-by: hao-xu5 <[email protected]>

update

931fdbd

Signed-off-by: hao-xu5 <[email protected]>

update doc

4919fba

Signed-off-by: hao-xu5 <[email protected]>

fix linting

63bba87

Signed-off-by: hao-xu5 <[email protected]>

Merge branch 'master' into support-table-format

b9bd58e

fix test

c014d6e

Signed-off-by: hao-xu5 <[email protected]>

franciscojavierarceo approved these changes Nov 6, 2025

View reviewed changes

HaoXuAI merged commit 2915ad1 into master Nov 6, 2025
19 of 20 checks passed

feat: Support table format: Iceberg, Delta, and Hudi #5650

feat: Support table format: Iceberg, Delta, and Hudi #5650

Uh oh!

Conversation

HaoXuAI commented Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Misc

Uh oh!

HaoXuAI Oct 5, 2025

Choose a reason for hiding this comment

Uh oh!

tokoko commented Oct 6, 2025

Uh oh!

franciscojavierarceo Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

franciscojavierarceo commented Oct 6, 2025

Uh oh!

HaoXuAI commented Oct 6, 2025

Uh oh!

franciscojavierarceo commented Oct 7, 2025

Uh oh!

tokoko commented Oct 7, 2025

Uh oh!

HaoXuAI commented Oct 9, 2025

Uh oh!

HaoXuAI commented Oct 10, 2025

Uh oh!

franciscojavierarceo left a comment

Choose a reason for hiding this comment

Uh oh!

HaoXuAI commented Oct 14, 2025

Uh oh!

HaoXuAI commented Oct 14, 2025

Uh oh!

HaoXuAI commented Nov 6, 2025

Uh oh!

franciscojavierarceo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HaoXuAI commented Oct 5, 2025 •

edited

Loading