Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Gridded VIIRS from the NOAA AWS bucket and more #164

Open
wants to merge 49 commits into
base: develop
Choose a base branch
from
Open

Add Gridded VIIRS from the NOAA AWS bucket and more #164

wants to merge 49 commits into from
+1,129 −306

Conversation

bbakernoaa
Copy link
Member

Added a new function to stream data from AWS from the noaa-jpss bucket into xarray for monetio. This should be useful for satellite comparisons.

It can handle both SNPP and NOAA20 satellites as well as varying resolutions (0.05, 0.1, 0.25 degree) and different time aggregations that are preprocessed by NESDIS (daily, weekly, monthly).

Data availability is from 2012 -> current.

to use:

nests_viirs_aod_gridded.open_dataset('2013-01-01', 'SNPP', '0.100', 'daily')

nests_viirs_aod_gridded.open_mfdataset(dates, 'SNPP', '0.050', 'daily')

@angehung5

@bbakernoaa bbakernoaa added the enhancement New feature or request label Mar 12, 2024
@bbakernoaa bbakernoaa requested a review from zmoon March 12, 2024 15:02
zmoon
zmoon reviewed Mar 12, 2024
View reviewed changes
Copy link
Member

@zmoon zmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you run the pre-commit hooks?

How long does it take to get a small amount of data (for test)?

@bbakernoaa
Copy link
Member Author

just a few seconds at most

@bbakernoaa
Copy link
Member Author

Can you run the pre-commit hooks?

How long does it take to get a small amount of data (for test)?

Not quite sure how to do that. Can you provide a little instruction?

@bbakernoaa
Copy link
Member Author

I added back in the NRT data. Seems the AWS server didn't have a the latest data from the past 3 years. in the NRT data we don't have the 0.05 data but we do have the 0.1 and 0.25 degree data.

@zmoon
Copy link
Member

zmoon commented Mar 13, 2024

Not quite sure how to do that.

  1. Install pre-commit to your env if you don't have it available already (it's available on conda-forge)
  2. pre-commit install --install-hooks
  3. pre-commit run --all-files

@zmoon
Copy link
Member

zmoon commented Mar 13, 2024

just a few seconds at most

Sweet, do you want a write a test? I can give it a go if not.

@bbakernoaa
Copy link
Member Author

Not quite sure how to do that.

  1. Install pre-commit to your env if you don't have it available already (it's available on conda-forge)
  2. pre-commit install --install-hooks
  3. pre-commit run --all-files

Thanks! Added it and got these rolling. Seems the tests are failing but in unrelated spots where the code was changed

@zmoon zmoon mentioned this pull request Mar 13, 2024
Open
@bbakernoaa bbakernoaa mentioned this pull request Mar 13, 2024
Open
@zmoon
sp
9212c80
@zmoon
Raise in mf version too
efa245b
@zmoon
'both' option doesn't work currently
915e9fe 
since sat_name doesn't get set
@zmoon
Initial mf test
535dc12
@zmoon
Update mf docstring
0d4a410
@zmoon
Skip missing daily file with warning by default
4a6a1be 
as long as have at least one
@zmoon
Skip for monthly as well
efc3b33
@zmoon
Always error for not found file in open_dataset
cec5823 
since nothing to return, unlike mfdataset, where we still
have something as long as there is at least one file that does
exist
@zmoon
Fix test for pre 3.10
62ea648
@bbakernoaa bbakernoaa linked an issue Mar 20, 2024 that may be closed by this pull request
Add reader for VIIRS AOD on S3 #132
Open
@zmoon zmoon marked this pull request as draft August 7, 2024 22:10
@bbakernoaa bbakernoaa marked this pull request as ready for review August 14, 2024 13:57
@zmoon
Copy link
Member

zmoon commented Aug 14, 2024 •
edited
Loading

Current NDVI failure seems to be an issue decoding the times in current xarray (works in my local older version) pydata/xarray#8957 TIMEOFDAY variable is the offender.

@zmoon
AVHRR AOT
402067b
@zmoon
Drop decode error culprit
45679c4 
ds.TIMEOFDAY
<xarray.DataArray 'TIMEOFDAY' (time: 1, latitude: 3600, longitude: 7200)>
[25920000 values with dtype=datetime64[ns]]
Coordinates:
  * latitude   (latitude) float32 89.97 89.93 89.88 ... -89.88 -89.93 -89.97
  * longitude  (longitude) float32 -180.0 -179.9 -179.9 ... 179.9 179.9 180.0
  * time       (time) datetime64[ns] 2023-01-01
Attributes:
    scale_factor:  0.01
    add_offset:    0.0
    long_name:     Time since Start of Data Day
    valid_range:   [   0 2399]
    _FillValue:    -9999
    grid_mapping:  crs
@zmoon
Hopefully avoid TIMEOFDAY warnings
f88a7fe
@zmoon
AVHRR AOT test
957aa4a
@zmoon
NESDIS VIIRS AOD NRT
ddd6fca
@zmoon
Add NESDIS VIIRS AOD NRT tests
1d3fd7c
@zmoon
Remove warns
2dbf8d6
@zmoon zmoon changed the title Add Gridded VIIRS from the NOAA AWS bucket Add Gridded VIIRS from the NOAA AWS bucket and more Aug 15, 2024
@zmoon
Copy link
Member

zmoon commented Sep 12, 2024 •
edited
Loading

@bbakernoaa besides the question of whether the module names need to have _gridded, I also wondered if you wanted to add a bit more documentation about the data sources, some links or something.

@bbakernoaa
Copy link
Member Author

@bbakernoaa besides the question whether the module names need to have _gridded, I also wondered if you wanted to add a bit more documentation about the data sources, some links or something.

sure where you like me to add these?

@zmoon
Copy link
Member

zmoon commented Sep 19, 2024 •
edited
Loading

sure where you like me to add these?

It could be in module docstring at the top (probably best), or in the docstring of the function that gets the data, or some combination. Or send me or point me to the relevant info and I can put it in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zmoon zmoon zmoon left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add reader for VIIRS AOD on S3
2 participants
@bbakernoaa @zmoon