feat: Feature/upgrade 3.11 and remove bytewax #4046
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pendencies. Signed-off-by: Lokesh Rangineni <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]>
lokeshrangineni
changed the title
…hould fix the failure of python unit test cases. Signed-off-by: Lokesh Rangineni <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]>
lokeshrangineni
changed the title
Signed-off-by: Lokesh Rangineni <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]>
…-bytewax # Conflicts: # sdk/python/feast/infra/materialization/contrib/bytewax/bytewax_materialization_engine.py
lokeshrangineni
changed the title
Contributor
Author
|
I am looking at the PR again to modify the the logic without actually removing the bytewax engine. If this approach works then probably i will update the PR otherwise we can go ahead with this approach. |
This was referenced Apr 9, 2024
jeremyary
changed the title
…1-remove-bytewax # Conflicts: # .github/workflows/build_wheels.yml # .github/workflows/nightly-ci.yml # .github/workflows/unit_tests.yml # sdk/python/feast/infra/materialization/kubernetes/Dockerfile # sdk/python/feast/infra/materialization/kubernetes/k8s_materialization_engine.py # sdk/python/feast/infra/materialization/kubernetes/k8s_materialization_job.py # sdk/python/feast/infra/materialization/kubernetes/k8s_materialization_task.py # sdk/python/feast/infra/materialization/kubernetes/main.py # setup.py
Signed-off-by: Lokesh Rangineni <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]> This reverts commit 4bec6b0.
…dev#4140) * chore: Move feast install to docker build in java it tests (feast-dev#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <[email protected]> * remove commented out lines in compose file Signed-off-by: tokoko <[email protected]> * make local compose mode default Signed-off-by: tokoko <[email protected]> * limit COPY contents Signed-off-by: tokoko <[email protected]> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <[email protected]> * include pyproject.toml in dockerfile Signed-off-by: tokoko <[email protected]> * change links to depends_on Signed-off-by: tokoko <[email protected]> * try updating setup-python to v5 Signed-off-by: tokoko <[email protected]> * pin macos image to macos-12 Signed-off-by: tokoko <[email protected]> * force rerun Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * chore: Incorporate release 0.37.1 to master (feast-dev#4127) * chore(release): release 0.37.1 ## [0.37.1](feast-dev/feast@v0.37.0...v0.37.1) (2024-04-17) ### Bug Fixes * Pgvector patch ([feast-dev#4108](feast-dev#4108)) ([1a1f0b1](feast-dev@1a1f0b1)) ### Reverts * Reverts "fix: Using version args to install the correct feast version" ([feast-dev#4112](feast-dev#4112)) ([d5ded69](feast-dev@d5ded69)), closes [feast-dev#3953](feast-dev#3953) * chore: Move feast install to docker build in java it tests (feast-dev#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <[email protected]> * remove commented out lines in compose file Signed-off-by: tokoko <[email protected]> * make local compose mode default Signed-off-by: tokoko <[email protected]> * limit COPY contents Signed-off-by: tokoko <[email protected]> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <[email protected]> * include pyproject.toml in dockerfile Signed-off-by: tokoko <[email protected]> * change links to depends_on Signed-off-by: tokoko <[email protected]> * try updating setup-python to v5 Signed-off-by: tokoko <[email protected]> * pin macos image to macos-12 Signed-off-by: tokoko <[email protected]> * force rerun Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Co-authored-by: feast-ci-bot <[email protected]> Co-authored-by: Tornike Gurgenidze <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * chore: Install python dependencies with uv in workflows (feast-dev#4086) * install dependencies in unit-tests with uv Signed-off-by: tokoko <[email protected]> * install dependencies in unit-tests with uv Signed-off-by: tokoko <[email protected]> * enable caching, change linter job Signed-off-by: tokoko <[email protected]> * change local integration tests to uv Signed-off-by: tokoko <[email protected]> * change all installs to uv Signed-off-by: tokoko <[email protected]> * try adding uv cache Signed-off-by: tokoko <[email protected]> * fix lambda cache step name Signed-off-by: tokoko <[email protected]> * reenable caches for uv Signed-off-by: tokoko <[email protected]> * remove dangling cache step Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * feat: Make arrow primary interchange for online ODFV execution (feast-dev#4143) * rewrite online flow to use transform_arrow Signed-off-by: tokoko <[email protected]> * fix transformation server Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * feat: Feast/IKV online store documentation (feast-dev#4146) * feat: Feast/IKV online store documentation Signed-off-by: Pushkar Gupta <[email protected]> * functionality matric Signed-off-by: Pushkar Gupta <[email protected]> * more changes Signed-off-by: Pushkar Gupta <[email protected]> * mount dir Signed-off-by: Pushkar Gupta <[email protected]> --------- Signed-off-by: Pushkar Gupta <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * fix: Default value is not set in Redis connection string using environment variable (feast-dev#4136) Removed documentation of Redis connection string supporting default values when using environment variables as it isn't supported Fixes feast-dev#3669 Signed-off-by: Theodor Mihalache <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * fix: Make sure schema is used when calling `get_table_query_string` method for Snowflake datasource (feast-dev#4131) * Fix get_table_query_string method for Snowflake datasource Signed-off-by: TomSteenbergen <[email protected]> * Add quotes to table string Signed-off-by: TomSteenbergen <[email protected]> --------- Signed-off-by: TomSteenbergen <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * fix: Change checkout action back to v3 from v5 which isn't released yet (feast-dev#4147) Signed-off-by: Jeremy Ary <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * changed the code the way mysql container is initialized. Trying to fix the issue - feast-dev#4128 Also going to check if this change will be resolved in the github actions as well. Signed-off-by: Lokesh Rangineni <[email protected]> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <[email protected]> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> Signed-off-by: Pushkar Gupta <[email protected]> Signed-off-by: Theodor Mihalache <[email protected]> Signed-off-by: TomSteenbergen <[email protected]> Signed-off-by: Jeremy Ary <[email protected]> Co-authored-by: Tornike Gurgenidze <[email protected]> Co-authored-by: Jeremy Ary <[email protected]> Co-authored-by: feast-ci-bot <[email protected]> Co-authored-by: Pushkar Gupta <[email protected]> Co-authored-by: Theodor Mihalache <[email protected]> Co-authored-by: Tom Steenbergen <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]> This reverts commit 8d77135.
Signed-off-by: Lokesh Rangineni <[email protected]>
lokeshrangineni
force-pushed
the
feature/upgrade-3.11-remove-bytewax
branch
from
279b043 to
bada729
Compare
jeremyary
reviewed
Apr 26, 2024
.github/workflows/unit_tests.yml
Outdated
| python-version: [ "3.9", "3.10" ] | ||
| os: [ ubuntu-latest, macos-12 ] | ||
| python-version: [ "3.9", "3.10", "3.11"] | ||
| os: [ ubuntu-latest, macOS-12 ] |
Collaborator
There was a problem hiding this comment.
SUPER nitpicky of me, but casing here is fine in lowercase which is how GitHub references them too
| sniffio==1.3.1 | ||
| # via anyio | ||
| sqlalchemy[mypy]==2.0.29 | ||
| sqlalchemy[mypy]==1.4.52 |
Collaborator
There was a problem hiding this comment.
not saying it's wrong, more curious, can you please share what's causing the downgrade of sqlalchemy?
Contributor
Author
There was a problem hiding this comment.
I have just re-ran the command to regenerate this file but not sure why this was modified. I have not modified the file manually. I will pay attention to it.
| "uvicorn[standard]>=0.14.0,<1", | ||
| "gunicorn; platform_system != 'Windows'", | ||
| "dask[dataframe]>=2021.1.0", | ||
| "dask[dataframe]>=2024.4.2", |
Collaborator
There was a problem hiding this comment.
I feel like I saw an answer in slack somewhere that I can't find at the moment, but this is a pretty big version jump, was Dask having issues with new Python version?
Contributor
Author
There was a problem hiding this comment.
We had a issue with Dask with new python version. Build is failing so had to upgrade but i have not tried the immediate version so let me give a shot with the version released in 2022.
Collaborator
There was a problem hiding this comment.
you're good! I was just looking for reason - since we need to upgrade we might as well catch up
| "googleapis-common-protos>=1.52.0,<2", | ||
| "google-cloud-bigquery[pandas]>=2,<3.13.0", | ||
| "google-cloud-bigquery-storage >= 2.0.0,<3", | ||
| "google-cloud-bigquery-storage >= 2.23.0,<3", |
Collaborator
There was a problem hiding this comment.
Did we encounter an issue w/BigQuery dependency too?
Contributor
Author
There was a problem hiding this comment.
Thank you for pointing it out. I had modified this version in between to resolve an issue with dask library. probably we don't need this change. i will remove this upgrade and see what happens.
Collaborator
|
@lokeshrangineni looking really good, appreciate the diligence, I know you've been sticking with this one for a while! |
update doc Signed-off-by: cmuhao <[email protected]>
…pendencies. Signed-off-by: Lokesh Rangineni <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]>
Signed-off-by: Lokesh Rangineni <[email protected]> This reverts commit 8d77135.
Signed-off-by: Lokesh Rangineni <[email protected]>
lokeshrangineni
requested review from
a team,
DvirDukhan,
kevjumba,
niklasvm,
sfc-gh-madkins,
sudohainguyen and
younggyu-oh
as code owners
…e/upgrade-3.11-remove-bytewax
Collaborator
|
work continuing here due to conflict complexity |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #