Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: microsoft/MLOS
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.3.2
Choose a base ref
...
head repository: microsoft/MLOS
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v0.4.2
Choose a head ref
  • 11 commits
  • 68 files changed
  • 2 contributors

Commits on Jan 19, 2024

  1. Configuration menu
    Copy the full SHA
    08b3a94 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. Remove TunableGroups from Storage classes and add TrialData tests (#644)

    Work split out from #633
    
    - Reorgs the unit test fixtures for reuse by mlos_viz (future PR)
    - Removes TunablesGroups from Storage classes. Currently unnecessary,
    belongs in Experiment, and causes inconsistencies with ExperimentData
    interactions.
    - Adjusts the initialization code to match.
    
    ---------
    
    Co-authored-by: Sergiy Matusevych <[email protected]>
    bpkroth and motus authored Jan 23, 2024
    Configuration menu
    Copy the full SHA
    c786a25 View commit details
    Browse the repository at this point in the history
  2. Refactor mlos_bench.storage and add TunableConfigTrialGroup prope…

    …rty for `TrialData` and `ExperimentData` (#648)
    
    Useful for grouping trials by the config they used. In use by upcoming
    #633 for generating graphs with variance error bars for repeated
    configs.
    
    Also refactors a number of other things:
    - Standardize on `experiment_id` instead of `exp_id`
      (but not the db schema for now)
    - Standardize on `tunable_config_id` instead of `config_id` in the API
    since we also call it `tunable_config` for the object fetching property
    to distinguish from the `config` dict used internally.
      (but not the db schema for now)
    - Rework the idea of `TunableConfigTrialGroup` as an object inaddition
    to an ID (further methods can be added later to move back and forth
    between types when doing interactive analysis).
    - Rework the idea of a `TunableConfig` as an object for fetching tunable
    value assignments (similar justification - easier grouping in the future
    by fetching trial across experiments based on config - eventually could
    be used to house the experiment merge logic).
    - Rename `results` APIs to `results_df` (similar for others that return
    `pandas.DataFrame`) to match the `results_dict` that return `dict`
    - Refactor test fixtures to match other styles and for future use (moved
    to #644).
    - Expand tests
    
    NOTE:
    - We cut a new version with this commit since there are potentially
    breaking API changes (e.g., `results` -> to `results_df` and `exp_id` ->
    `experiment_id`).
    
    Currently builds off of #644 and splits work out of #633
    
    ---------
    
    Co-authored-by: Sergiy Matusevych <[email protected]>
    bpkroth and motus authored Jan 23, 2024
    Configuration menu
    Copy the full SHA
    d8ea64f View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2024

  1. Add default_tunable_config_id property to ExperimentData (#650)

    Splitting work out from #633.
    
    Having an easy way to identify what is the "default" config group for an
    Experiment makes analysis of improvements easier.
    
    One challenge with this is that we don't currently strictly enforce
    non-changes between resumptions of an Experiment, so it's currently
    possible for the user to change the tunables config and adjust what the
    "default".
    
    Moreover, in the past we didn't store what the default config was in the
    DB.
    
    By convention we take the first one for now as a fallback.
    bpkroth authored Jan 26, 2024
    Configuration menu
    Copy the full SHA
    56e84fd View commit details
    Browse the repository at this point in the history
  2. Remove mistakenly added columns (#652)

    #628 mistakenly included an early attempt at adding
    `optimization_target` and `optimization_direction` to the `experiment`
    table in the `mlos_bench.storage.sql` backend.
    
    In that PR we later moved it to its own `objectives` table to eventually
    support multi-objectives.
    
    Nothing accesses those columns now, however including them in the
    metadata makes it impossible to load storage backends previously created
    with the old schema since adjusting columns with sqlalchemy's
    `create_all()` API only considers table existence.
    
    On the contrary, the latter means that we will automatically support old
    storage backends with the new code for the `objectives` table.
    
    Removing these two columns in the `metadata` schema description simply
    allows that to proceed without error.
    
    See Also: #649
    
    Co-authored-by: Sergiy Matusevych <[email protected]>
    bpkroth and motus authored Jan 26, 2024
    Configuration menu
    Copy the full SHA
    5be96f2 View commit details
    Browse the repository at this point in the history
  3. Fixes to max_iterations calculation when `trial_config_repeat_count…

    …` is given. (#651)
    
    Splitting work out from #633.
    Bug fix for #642.
    
    Co-authored-by: Sergiy Matusevych <[email protected]>
    bpkroth and motus authored Jan 26, 2024
    Configuration menu
    Copy the full SHA
    d073fc1 View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2024

  1. Address some pandas FuturesWarnings (#653)

    Polish for a demo.
    
    See Also: https://stackoverflow.com/a/61185987
    
    ---------
    
    Co-authored-by: Sergiy Matusevych <[email protected]>
    bpkroth and motus authored Jan 27, 2024
    Configuration menu
    Copy the full SHA
    3a36797 View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2024

  1. Improve mlos-viz for multiple repeats of a config and add tests (#633)

    - Mark `mlos_viz` as `typed` for `mypy`
    - Bump version
    - Mock calls to matplotlib/dabl for testing
    - Add plotting of top-N configs
    - Improve plots for handling repeat config trials via variance error
    bars
    
    ---------
    
    Co-authored-by: Sergiy Matusevych <[email protected]>
    bpkroth and motus authored Jan 29, 2024
    Configuration menu
    Copy the full SHA
    a45f97d View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. Configuration menu
    Copy the full SHA
    5287c24 View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. Configuration menu
    Copy the full SHA
    415674c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f5c357a View commit details
    Browse the repository at this point in the history
Loading