fix: Verify the existence of Registry tables in snowflake before calling CREATE sql command. Allow read-only user to call feast apply. #3851
Conversation
…ing CREATE sql command. Allow read-only user to call feast apply. Signed-off-by: Shuchu Han <[email protected]>
|
Please add tests to make sure the fix works as intended |
|
Can you please add more details on how to test the fix ? |
|
I did few manual tests with my free Snowflake account while I was creating this PR. To test it, we need to configure the Snowflake as I mentioned in the PR and check the result of "feast apply" on the tester's terminal. It will be great if someone has a Snowflake account and test it. |
|
This PR can not fix the feature that allows READ-only users to access the DB. There is an optimization in the get_historical_features() that will create a temporary table in the (offline store) DB for the point-of-time correct Join. (https://www.hopsworks.ai/dictionary/point-in-time-correct-joins#:~:text=A%20point%2Din%2Dtime%20correct,a%20specific%20point%20in%20time.) For example, in line 815-822 of bigquery.py (https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/offline_stores/bigquery.py), there is a short description about this. The temporary table is deleted as in Line 298 after the usage. On snowflake, if we can separate the offline store DB from the registry repo, I believe we can solve this problem by allow READ-only to registry repo DB, but allow Write+Read to the offline store DB. |
3 participants
What this PR does / why we need it:
For snowflake users who only have "USAGE" and "SELECT" privileges, we want to allow them able to run the "feast apply" command and do READ-only access existing Registry tables in snowflake cloud.
Please be aware that the Feast Registry is designed to allow every user can modify the records of registry tables under their own "project" name. This PR didn't change the original design. It only changes the previous implementation logic of snowflake.py which always calls "CREATE" sql command while initializing the SnowflakeRegistry(), which will be called while run statement:
store = FeatureStore()
This PR is tested with snowflake cloud privilege for a Role with setting as below:
for feast Registry database : SELECT - FUTURE TABLE, SELECT - FUTURE VIEW, USAGE
for feast Registry 11 tables: SELECT
for feast schema (default name PUBLIC): SELECT - FUTURE TABLE, SELECT - FUTURE VIEW, USAGE
for warehouse: USAGE
The test will expect the above specific role can call "feast apply" and does not see errors like:
snowflake.connector.errors.ProgrammingError: 003001 (42501): SQL access control error: Insufficient privileges to operate on *****
during the init() function call of SnowflakeRegistry().
Which issue(s) this PR fixes:
Fixes #3844