[Thread Prev][Thread Next][Index]

Re: [ferret_users] grid compressed after subtract area average



Hi,

The listing in SHOW GRID is incorrect, but the variable does in fact have a 3-D grid.  In fact this is a bug which has been found and fixed, so the next release will have the correct SHOW GRID listing:  https://github.com/NOAA-PMEL/Ferret/issues/1961

Another way to really check what's happening is to use the "stat" command to load data and show its shape:


yes? use coads_climatology

yes? let sstave=sst[x=@AVE,Y=@AVE]

! This grid is indeed compressed in X and Y
yes? stat/brief sstave
 
 Total # of data points: 12 (1*1*1*12*1*1)
 # flagged as bad  data: 0
 Minimum value: 19.037
 Maximum value: 21.322
 Mean    value: 20.264 (unweighted average)

! This grid is not compressed, it gets its X-Y axes from sst

yes? let test=sst-sstave
yes? stat/br test
 
 Total # of data points: 194400 (180*90*1*12*1*1)
 # flagged as bad  data: 89622
 Minimum value: -23.622
 Maximum value: 12.722
 Mean    value: -2.1127 (unweighted average)

Now, I do think that I see the behavior that you were asking about.  I don't think it is because of the cosine correction but I don't yet see what is happening.  The definite integral in XY of SST is expressed in degrees_C* m^2.  So is thesecond listing, the integral of sst minus its overall average.  But the integral of the variable "test" seems wrong. 

yes? list sst[x=@din,y=@din]
             VARIABLE : SEA SURFACE TEMPERATURE (Deg C)
             FILENAME : coads_climatology.cdf
             FILEPATH : /home/users/tmap/ferret/linux/fer_dsets/data/
             SUBSET   : 12 points (TIME)
             LONGITUDE: 20E to 20E(380) (XY integ.)
             LATITUDE : 90S to 90N (XY integ.)
 16-JAN      /  1:  6.963E+15
 15-FEB      /  2:  7.009E+15
 17-MAR      /  3:  7.035E+15
 16-APR      /  4:  6.999E+15
...

yes? let sst_m_ave = sst - `sst[x=@ave,y=@ave,t=@ave]`
 !-> DEFINE VARIABLE sst_m_ave = sst - 20.22408770542379
yes? list sst_m_ave[x=@din,y=@din]
             VARIABLE : SST - 20.22408770542379
             FILENAME : coads_climatology.cdf
             FILEPATH : /home/users/tmap/ferret/linux/fer_dsets/data/
             SUBSET   : 12 points (TIME)
             LONGITUDE: 20E to 20E(380) (XY integ.)
             LATITUDE : 90S to 90N (XY integ.)
 16-JAN      /  1: -4.341E+14
 15-FEB      /  2: -4.144E+14
 17-MAR      /  3: -3.452E+14
 16-APR      /  4:  8.673E+13

...


yes? list test[x=@din,y=@din]
             VARIABLE : SST-SSTAVE
             FILENAME : coads_climatology.cdf
             FILEPATH : /home/users/tmap/ferret/linux/fer_dsets/data/
             SUBSET   : 12 points (TIME)
             LONGITUDE: 20E to 20E(380) (XY integ.)
             LATITUDE : 90S to 90N (XY integ.)
 16-JAN      /  1: -142.2
 15-FEB      /  2: -125.0
 17-MAR      /  3:  -93.7
 16-APR      /  4: -137.5
...



On 1/22/2020 1:33 PM, Xiaoyu Bai wrote:
Hi Ansley,

I am subtracting averaged value from a full subset variable. Like the variable test here:

use coads_climatology
yes? let sstave=sst[x=@AVE,Y=@AVE]
yes? sh grid sstave
    GRID GSQ1
 name       axis              # pts   start                end                 subset
 COADSX    LONGITUDE          180mr   21E                  19E(379)            compressed
 COADSY    LATITUDE            90 r   89S                  89N                 compressed
 normal    Z
 TIME      TIME                12mr   16-JAN 06:00         16-DEC 01:20        full
yes? let test=sst-sstave
yes? sh grid test
    GRID GSQ1
 name       axis              # pts   start                end                 subset
 COADSX    LONGITUDE          180mr   21E                  19E(379)            compressed
 COADSY    LATITUDE            90 r   89S                  89N                 compressed
 normal    Z
 TIME      TIME                12mr   16-JAN 06:00         16-DEC 01:20        full
yes? sh grid sst
    GRID GSQ1
 name       axis              # pts   start                end                 subset
 COADSX    LONGITUDE          180mr   21E                  19E(379)            full
 COADSY    LATITUDE            90 r   89S                  89N                 full
 normal    Z
 TIME      TIME                12mr   16-JAN 06:00         16-DEC 01:20        full

What I am expecting to see is, after sst minus sstave, the result still have a full subset like sst. But ferret tends to give sstave's grid to the new variable. This behavior is blocking me from applying @DIN because the cosine correction was no longer applying. adj_EB_land_CNRM_AM5 in my analysis is like the variable TEST here, it is defined as the result of a 3D (x,y,t) variable minus a 1D (X=@AVE,Y=@AVE,t) variable. The 1D value will be subtracted from each grid of the 3D variable. Although this calculation may be weird to ferret, the result should be a 3D variable since the grids are still there.

I hope I made things clearer this time.

Thank you so much for your time.

Xiaoyu   

On Wed, Jan 22, 2020 at 2:15 PM Ansley C. Manke <ansley.b.manke@xxxxxxxx> wrote:

Hi,

Once the data has been averaged, applying @DIN in the same directions does not make sense, as there is nothing to integrate. I'm not sure how adj_EB_land_CNRM_AM5 is computed but if its grid in the X and Y directions shows as compressed, then there is no dependence on those directions and pyferret should refuse to even compute a @DIN transformation.  To show a simple example,


yes? use coads_climatology
yes? let sstave = sst[x=@ave,y=@ave]

yes? show grid sstave
    GRID GSQ1
 name       axis              # pts   start                end                 subset
 COADSX    LONGITUDE          180mr   21E                  19E(379)            compressed
 COADSY    LATITUDE            90 r   89S                  89N                 compressed
 normal    Z
 TIME      TIME                12mr   16-JAN 06:00         16-DEC 01:20        full

yes? list sstave[x=@din,y=@din]
**ERROR: illegal use of transformation: SSTAVE[X=20E:20E(380)@IN4,D=1] applied to point


So, I am not sure I understand what you are doing.  How is adj_EB_land_CNRM_AM5  defined?

Sometimes if the order of operations that PyFerret uses is not what you need, a method can be to write intermediate results to a file and then read that back in to, say, subtract from another _expression_.


-Ansley


On 1/21/2020 12:05 PM, Xiaoyu Bai wrote:
Hi Ansley,

What I am trying to do is to subtract global average (@AVE, which is only ONE value for the whole globe) from model output (which has value at every grid). This calculation may be weird to pyferret but I have to use this method to adjust the model's energy budget.    
yes? sh grid adj_EB_land_CNRM_AM5 (adjusted value), AHT_land_CNRM_AM5 (energy budget)
    GRID (G033)
 name       axis              # pts   start                end                 subset
 LON       LONGITUDE          256mr   0E                   1.406W(358.59)      compressed
 LAT       LATITUDE           128 i   88.928S              88.928N             compressed
 PLEV      Z (Pa)              17 i-  1000                 100000              compressed
 MONTH_GREGORIAN TIME          12mi   16-JAN 12:00         15-DEC 17:49        full
    GRID (G033)
 name       axis              # pts   start                end                 subset
 LON       LONGITUDE          256mr   0E                   1.406W(358.59)      full
 LAT       LATITUDE           128 i   88.928S              88.928N             full
 PLEV      Z (Pa)              17 i-  1000                 100000              compressed
 MONTH_GREGORIAN TIME          12mi   16-JAN 12:00         15-DEC 17:49        full

The incorrect grid affects my further calculation. I need to apply @DIN to my adj_EB_land_CNRM_AM5. Since the grid is compressed, pyferret doesn't apply cosine correction anymore. My proof for this is the value for AHT_land_CNRM_AM5 is around -100 to 50. The @AVE value is 4.021. So the adj_EB_land_CNRM_AM5 value is roughly around -104 to 46. If I apply @DIN for both latitude and longitude to AHT_land_CNRM_AM5, the outcome is 2.051E+15 W/m2 which makes sense. If I apply @DIN to adj_EB_land_CNRM_AM5, the outcome is -61.33 W/m2 which doesn't make sense. 

list/L=7 AHT_land_CNRM_AM5[X=@DIN,Y=@DIN], adj_EB_land_CNRM_AM5[X=@DIN,Y=@Din]
             LONGITUDE: 0.7W(-0.7) to 0.7W(359.3) (XY integ.)
             LATITUDE: 90S to 90N (XY integ.)
             TIME: 15-JUL 17:49
 Column  1: AHT_LAND_CNRM_AM5 is SWABS_LAND_CNRM_AM5-OLR_LAND_CNRM_AM5+SHF_LAND_CNRM_AM5-STORAGE_LAND_CNRM_AM5
 Column  2: ADJ_EB_LAND_CNRM_AM5 is AHT_LAND_CNRM_AM5-EB_LAND_CNRM_AM5
          AHT_LAND_  ADJ_EB_LAND_CNRM_AM5
I / *:     2.051E+15  -61.33

So I tried with a single month to use AHT_land_CNRM_AM5 just minus the ave value which is 4.021.
   
yes? let test=AHT_land_CNRM_AM5-4.021
yes? list/L=7 AHT_land_CNRM_AM5[X=@DIN,Y=@DIN], adj_EB_land_CNRM_AM5[X=@DIN,Y=@DIN], test[X=@DIN,Y=@DIN]
             LONGITUDE: 0.7W(-0.7) to 0.7W(359.3) (XY integ.)
             LATITUDE: 90S to 90N (XY integ.)
             TIME: 15-JUL 17:49
 Column  1: AHT_LAND_CNRM_AM5 is SWABS_LAND_CNRM_AM5-OLR_LAND_CNRM_AM5+SHF_LAND_CNRM_AM5-STORAGE_LAND_CNRM_AM5
 Column  2: ADJ_EB_LAND_CNRM_AM5 is AHT_LAND_CNRM_AM5-EB_LAND_CNRM_AM5
 Column  3: TEST is AHT_LAND_CNRM_AM5-4.021
          AHT_LAND_  ADJ_EB_    TEST
I / *:     2.051E+15  -61.33  1.507E+11

These results are indicating that pyferret was not applying cosine correction to compressed latitude and longitude (which makes sense to ferret since it's compressed). But I have 10 models with 12 months' data, it's hard for me to subtract the value one by one. I am wondering whether there are some methods that I can transfer the @AVE value into a numeric or I can regrid adj_EB_land_CNRM_AM5 to be like AHT_land_CNRM_AM5 (I already tried with let/like=AHT_land_CNRM_AM5 adj_EB_land_CNRM_AM5=AHT_land_CNRM_AM5-EB_land_CNRM_AM5(this the @AVE variable) and set variable/name=tt adj_EB_land_CNRM_AM5 \  let adj_EB_land_CNRM_AM5=tt[g=AHT_land_CNRM_AM5,gt=@ASN] but neither of them worked). I tend to use transfering @AVE to a numeric method now. Any suggestions?

Thank you very much.

Xiaoyu

 

On Wed, Dec 4, 2019 at 1:04 PM Ansley C. Manke <ansley.b.manke@xxxxxxxx> wrote:

Hi,

Maybe this is just some detail in the syntax.  The coordinate axes for the variable "temp" are not actually compressed, it's just that SHOW GRID is incorrectly saying they are.  Are you seeing an error message?  If this doesn't resolve things, please send more about the commands you are using and the results.

Your original example was with levitus_climatology, so let's go back to that.

yes? use levitus_climatology
yes? let test=TEMP-TEMP[X=@AVE,Y=@AVE]

yes? list test[x=@din,y=@din]
             VARIABLE : TEMP-TEMP[X=@AVE,Y=@AVE]
             FILENAME : levitus_climatology.cdf
             FILEPATH : /home/users/tmap/ferret/linux/fer_dsets/data/
             SUBSET   : 20 points (DEPTH (m))
             LONGITUDE: 20E to 20E(380) (XY integ.)
             LATITUDE : 90S to 90N (XY integ.)
 0       /  1: -538.5
 10      /  2: -633.9
 20      /  3: -530.8
 30      /  4: -575.1
 50      /  5: -335.4
 75      /  6: -440.2
 100     /  7: -459.0
 150     /  8: -405.4
 200     /  9: -287.2
 300     / 10: -305.7
 400     / 11: -234.4
 600     / 12: -118.7
 800     / 13: -113.0
 1000    / 14:  -73.9
 1200    / 15:  -52.0
 1500    / 16:  -35.0
 2000    / 17:   -6.6
 3000    / 18:  -13.9
 4000    / 19:    7.7
 5000    / 20:    0.5

On 12/3/2019 6:02 PM, Xiaoyu Bai wrote:
I picked the wrong word. I do want to remove the mean over lat and Lon. After that I want to apply @DIN. The compressed coordinates blocked @DIN to work. I tried with TEMP-Temp[x=@ave, y=@ave] and TEMP-number. They gave me very different results.

Thank you 😊 

Xiaoyu

在 2019年12月3日,下午8:13,Ansley C. Manke <ansley.b.manke@xxxxxxxx> 写道:



Hi Xiaoyu,

First, what you have done with TEMP-TEMP[X=@AVE,Y=@AVE] is to remove the mean over longitude and latitude.  That will not be less noisy.  If you want to work with a less-noisy version of TEMP, you might use one of the smoothing transformations such as the boxcar smoother, @SBX.  Here is a list of all the transformations:

https://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/variables-xpressions/XPRESSIONS#Chapter3-Transformations, and further down that page are detailed descriptions of the different transformations.


Now, it looks as if you've discovered an error in the SHOW GRID output.  It should not be listing the x and y axes as "compressed".  So we'll look into that.

yes? use coads_climatology
yes? let sst_demean =  sst - sst[x=@ave,y=@ave]

yes? show grid sst_demean
    GRID GSQ1
 name       axis              # pts   start                end                 subset
 COADSX    LONGITUDE          180mr   21E                  19E(379)            compressed
 COADSY    LATITUDE            90 r   89S                  89N                 compressed
 normal    Z
 TIME      TIME                12mr   16-JAN 06:00         16-DEC 01:20        full

However sst_demean does still vary in longitude and latitude.  See how STAT shows 180 points in X and 90 points in Y, and a plot at one time and latitude varies in X.

 
yes? stat/brief sst_demean
 
 Total # of data points: 194400 (180*90*1*12*1*1)
 # flagged as bad  data: 89622
 Minimum value: -23.622
 Maximum value: 12.722
 Mean    value: -2.1127 (unweighted average)

yes? plot/L=1/y=0 sst_demean


First, what you have done with TEMP-TEMP[X=@AVE,Y=@AVE] is merely to remove the mean.  If you want a less-noisy version of TEMP, you might look at the smoothing transformations. Here is a list of all the transformations:

https://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/variables-xpressions/XPRESSIONS#Chapter3-Transformations




On 12/3/2019 4:38 PM, Xiaoyu Bai wrote:
Dear ferreters,

I noticed that when I subtract area average from a variable, it will make the variable's Lat and Lon become compressed instead of full. For example:

use levi_climatology
let test=TEMP-TEMP[X=@AVE,Y=@AVE]
sh grid TEMP,test
    GRID GMS1
 name       axis              # pts   start                end                 subset
 XAXLEVITR LONGITUDE          360mr   20.5E                19.5E(379.5)        full
 YAXLEVITR LATITUDE           180 r   89.5S                89.5N               full
 ZAXLEVITR DEPTH (m)           20 i-  0                    5000                full
 normal    T
    GRID GMS1
 name       axis              # pts   start                end                 subset
 XAXLEVITR LONGITUDE          360mr   20.5E                19.5E(379.5)        compressed
 YAXLEVITR LATITUDE           180 r   89.5S                89.5N               compressed
 ZAXLEVITR DEPTH (m)           20 i-  0                    5000                full
 normal    T

What I want is to get a less noisy variable, then apply @DIN to it. But this compressed grid was blocking me to use @DIN.

So I am wondering how to get my coordinate to full again.

Thank you in advance.
 
--
Xiaoyu


--
Xiaoyu


--
Xiaoyu

[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement