Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Finish computing intersection of time periods #274

Merged
merged 11 commits into from
Oct 25, 2021

Conversation

JackKelly
Copy link
Member

@JackKelly JackKelly commented Oct 22, 2021

Pull Request

Description

This is a bite-sized part of a larger issue (#223). This is the last PR for #223. After this PR is merged, #223 will be done.

The ultimate aim of #223 is to change the way nowcasting_dataset computes the set available datetimes across all DataSources.

This PR implements these features:

  • Implement a DataSource.get_contigous_t0_time_periods() -> pd.DataFrame which goes through each period and chops off history_duration from the beginning of the period, and chops off forecast_duration from the end of the period.
  • Enable NowcastingDataModule to compute the intersection of all the lists of t0 time periods from each DataSource. Use nd_time.intersection_of_2_dataframes_of_periods().
  • Compute t0 datetimes across all those periods (using a user-specified frequency, e.g. '5 minutes').
  • As before, split those t0 datetimes into train, valid, test
  • Remove nd_time.get_start_datetimes(), nd_time.intersection_of_datetimeindexes(), DataSource.get_t0_datetimes(), and their tests, and use grep to check they're not called from anywhere I've missed.

Fixes #223

How Has This Been Tested?

New tests have been written. All tests pass.

  • No
  • Yes

Checklist:

  • My code follows OCF's coding style guidelines
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked my code and corrected any misspellings

@JackKelly JackKelly added enhancement New feature or request refactoring labels Oct 22, 2021
@JackKelly JackKelly self-assigned this Oct 22, 2021
@JackKelly JackKelly changed the title Jack/finish intersection of periods 2 Finish computing intersection of time periods Oct 22, 2021
@JackKelly JackKelly marked this pull request as ready for review October 22, 2021 18:21
@JackKelly JackKelly requested a review from jacobbieker October 25, 2021 07:45
Copy link
Member

@jacobbieker jacobbieker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@JackKelly JackKelly merged commit 8249329 into main Oct 25, 2021
@JackKelly JackKelly deleted the jack/finish-intersection-of-periods-2 branch October 25, 2021 09:56
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request refactoring
Projects
No open projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Get intersection of _Periods_ of available data across all DataSources, instead of using datetimes.
2 participants