Skip to content

Fix storage API failing to return DF for experiments without any tunables #889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bpkroth merged 6 commits into from
Dec 5, 2024
Merged

Fix storage API failing to return DF for experiments without any tunables #889

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d390f24
Initial fix
Dec 4, 2024
6a49e8f
Merge branch 'main' into eujingchua/fix-empty-tunables
motus Dec 4, 2024
2cfbf8d
Change to inner join instead
Dec 4, 2024
a9710cc
Add unit test
Dec 4, 2024
74e6186
Address linting
Dec 4, 2024
5bf80fe
Additional check in test for no config columns
Dec 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion mlos_bench/mlos_bench/storage/sql/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,6 @@ def get_results_df(
.join(
schema.config_param,
schema.config_param.c.config_id == schema.trial.c.config_id,
isouter=True,
)
.order_by(
schema.trial.c.trial_id,
Expand Down
13 changes: 13 additions & 0 deletions mlos_bench/mlos_bench/tests/storage/exp_data_test.py
View file
Open in desktop
Copy link
Contributor

@bpkroth bpkroth Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per the discussions, can you please also add a test to check that pending trials that don't yet have results, still show up in the results_df, with empty results columns?

Copy link
Member

@motus motus Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can also add some failed trials for good measure

Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,19 @@ def test_exp_data_results_df(exp_data: ExperimentData, tunable_groups: TunableGr
)


def test_exp_no_tunables_data_results_df(exp_no_tunables_data: ExperimentData) -> None:
"""Tests the results_df property of ExperimentData when there are no tunables."""
results_df = exp_no_tunables_data.results_df
expected_trials_count = CONFIG_COUNT * CONFIG_TRIAL_REPEAT_COUNT
assert len(results_df) == expected_trials_count
assert len(results_df["trial_id"].unique()) == expected_trials_count
obj_target = next(iter(exp_no_tunables_data.objectives))
assert (
len(results_df[ExperimentData.RESULT_COLUMN_PREFIX + obj_target]) == expected_trials_count
)
assert not results_df.columns.str.startswith(ExperimentData.CONFIG_COLUMN_PREFIX).any()


def test_exp_data_tunable_config_trial_group_id_in_results_df(exp_data: ExperimentData) -> None:
"""
Tests the tunable_config_trial_group_id property of ExperimentData.results_df.
Expand Down
Loading