Hello again,
The latitude correction is being done. It's just that subtracting a global average and the computing the definite integral over the entire earth are essentially repeating the same computation.
The operations sst[x=@AVE,Y=@AVE] and sst[x=@DIN,Y=@DIN] differ only in that the averaging computation is weighted by the surface area of the grid cells as it's computed, and the definite integral is not. This means that once we have subtracted the global average from the data, then integrating that difference again over the entire globe the result is essentially zero.
Try this: Define a variable with value 1 everywhere, compute the definite integral over the earth:
yes? let _one_ = 1+0*missing(sst,0)
yes? list one[x=@din,y=@din]
VARIABLE : 1+0*MISSING(SST,0)
FILENAME : coads_climatology.cdf
FILEPATH : /home/users/tmap/ferret/linux/fer_dsets/data/
SUBSET : 12 points (TIME)
LONGITUDE: 20E to 20E(380) (XY integ.)
LATITUDE : 90S to 90N (XY integ.)
16-JAN / 1: 5.101E+14
15-FEB / 2: 5.101E+14
17-MAR / 3: 5.101E+14
16-APR / 4: 5.101E+14
16-MAY / 5: 5.101E+14
16-JUN / 6: 5.101E+14
16-JUL / 7: 5.101E+14
16-AUG / 8: 5.101E+14
15-SEP / 9: 5.101E+14
16-OCT / 10: 5.101E+14
15-NOV / 11: 5.101E+14
16-DEC / 12: 5.101E+14
! Compute the earth from the earth radius in meters:
yes? let m_radius = 6371e3
yes? let pi = 3.141592653
yes? let earth_area = 4*pi*m_radius^2
yes? list earth_area
VARIABLE : 4*PI*M_RADIUS^2
5.101E+14
On 1/22/2020 5:20 PM, Ansley C. Manke wrote:
Hi,
The listing in SHOW GRID is incorrect, but the variable does in fact have a 3-D grid. In fact this is a bug which has been found and fixed, so the next release will have the correct SHOW GRID listing: https://github.com/NOAA-PMEL/Ferret/issues/1961
Another way to really check what's happening is to use the "stat" command to load data and show its shape:
yes? use coads_climatology
yes? let sstave=sst[x=@AVE,Y=@AVE]
! This grid is indeed compressed in X and Y
yes? stat/brief sstave
Total # of data points: 12 (1*1*1*12*1*1)
# flagged as bad data: 0
Minimum value: 19.037
Maximum value: 21.322
Mean value: 20.264 (unweighted average)! This grid is not compressed, it gets its X-Y axes from sst
yes? let test=sst-sstave
yes? stat/br test
Total # of data points: 194400 (180*90*1*12*1*1)
# flagged as bad data: 89622
Minimum value: -23.622
Maximum value: 12.722
Mean value: -2.1127 (unweighted average)
Now, I do think that I see the behavior that you were asking about. I don't think it is because of the cosine correction but I don't yet see what is happening. The definite integral in XY of SST is expressed in degrees_C* m^2. So is thesecond listing, the integral of sst minus its overall average. But the integral of the variable "test" seems wrong.
yes? list sst[x=@din,y=@din]
VARIABLE : SEA SURFACE TEMPERATURE (Deg C)
FILENAME : coads_climatology.cdf
FILEPATH : /home/users/tmap/ferret/linux/fer_dsets/data/
SUBSET : 12 points (TIME)
LONGITUDE: 20E to 20E(380) (XY integ.)
LATITUDE : 90S to 90N (XY integ.)
16-JAN / 1: 6.963E+15
15-FEB / 2: 7.009E+15
17-MAR / 3: 7.035E+15
16-APR / 4: 6.999E+15
...yes? let sst_m_ave = sst - `sst[x=@ave,y=@ave,t=@ave]`
!-> DEFINE VARIABLE sst_m_ave = sst - 20.22408770542379
yes? list sst_m_ave[x=@din,y=@din]
VARIABLE : SST - 20.22408770542379
FILENAME : coads_climatology.cdf
FILEPATH : /home/users/tmap/ferret/linux/fer_dsets/data/
SUBSET : 12 points (TIME)
LONGITUDE: 20E to 20E(380) (XY integ.)
LATITUDE : 90S to 90N (XY integ.)
16-JAN / 1: -4.341E+14
15-FEB / 2: -4.144E+14
17-MAR / 3: -3.452E+14
16-APR / 4: 8.673E+13
...
yes? list test[x=@din,y=@din]
VARIABLE : SST-SSTAVE
FILENAME : coads_climatology.cdf
FILEPATH : /home/users/tmap/ferret/linux/fer_dsets/data/
SUBSET : 12 points (TIME)
LONGITUDE: 20E to 20E(380) (XY integ.)
LATITUDE : 90S to 90N (XY integ.)
16-JAN / 1: -142.2
15-FEB / 2: -125.0
17-MAR / 3: -93.7
16-APR / 4: -137.5
...
On 1/22/2020 1:33 PM, Xiaoyu Bai wrote:
Hi Ansley,
I am subtracting averaged value from a full subset variable. Like the variable test here:
use coads_climatology
yes? let sstave=sst[x=@AVE,Y=@AVE]
yes? sh grid sstave
GRID GSQ1
name axis # pts start end subset
COADSX LONGITUDE 180mr 21E 19E(379) compressed
COADSY LATITUDE 90 r 89S 89N compressed
normal Z
TIME TIME 12mr 16-JAN 06:00 16-DEC 01:20 full
yes? let test=sst-sstave
yes? sh grid test
GRID GSQ1
name axis # pts start end subset
COADSX LONGITUDE 180mr 21E 19E(379) compressed
COADSY LATITUDE 90 r 89S 89N compressed
normal Z
TIME TIME 12mr 16-JAN 06:00 16-DEC 01:20 full
yes? sh grid sst
GRID GSQ1
name axis # pts start end subset
COADSX LONGITUDE 180mr 21E 19E(379) full
COADSY LATITUDE 90 r 89S 89N full
normal Z
TIME TIME 12mr 16-JAN 06:00 16-DEC 01:20 full
What I am expecting to see is, after sst minus sstave, the result still have a full subset like sst. But ferret tends to give sstave's grid to the new variable. This behavior is blocking me from applying @DIN because the cosine correction was no longer applying. adj_EB_land_CNRM_AM5 in my analysis is like the variable TEST here, it is defined as the result of a 3D (x,y,t) variable minus a 1D (X=@AVE,Y=@AVE,t) variable. The 1D value will be subtracted from each grid of the 3D variable. Although this calculation may be weird to ferret, the result should be a 3D variable since the grids are still there.
I hope I made things clearer this time.
Thank you so much for your time.
Xiaoyu
On Wed, Jan 22, 2020 at 2:15 PM Ansley C. Manke <ansley.b.manke@xxxxxxxx> wrote:
Hi,
Once the data has been averaged, applying @DIN in the same directions does not make sense, as there is nothing to integrate. I'm not sure how adj_EB_land_CNRM_AM5 is computed but if its grid in the X and Y directions shows as compressed, then there is no dependence on those directions and pyferret should refuse to even compute a @DIN transformation. To show a simple example,
yes? use coads_climatology
yes? let sstave = sst[x=@ave,y=@ave]yes? show grid sstave
GRID GSQ1
name axis # pts start end subset
COADSX LONGITUDE 180mr 21E 19E(379) compressed
COADSY LATITUDE 90 r 89S 89N compressed
normal Z
TIME TIME 12mr 16-JAN 06:00 16-DEC 01:20 full
yes? list sstave[x=@din,y=@din]
**ERROR: illegal use of transformation: SSTAVE[X=20E:20E(380)@IN4,D=1] applied to point
So, I am not sure I understand what you are doing. How is adj_EB_land_CNRM_AM5 defined?
Sometimes if the order of operations that PyFerret uses is not what you need, a method can be to write intermediate results to a file and then read that back in to, say, subtract from another _expression_.
-Ansley
On 1/21/2020 12:05 PM, Xiaoyu Bai wrote:
Hi Ansley,
What I am trying to do is to subtract global average (@AVE, which is only ONE value for the whole globe) from model output (which has value at every grid). This calculation may be weird to pyferret but I have to use this method to adjust the model's energy budget.yes? sh grid adj_EB_land_CNRM_AM5 (adjusted value), AHT_land_CNRM_AM5 (energy budget)
GRID (G033)
name axis # pts start end subset
LON LONGITUDE 256mr 0E 1.406W(358.59) compressed
LAT LATITUDE 128 i 88.928S 88.928N compressed
PLEV Z (Pa) 17 i- 1000 100000 compressed
MONTH_GREGORIAN TIME 12mi 16-JAN 12:00 15-DEC 17:49 full
GRID (G033)
name axis # pts start end subset
LON LONGITUDE 256mr 0E 1.406W(358.59) full
LAT LATITUDE 128 i 88.928S 88.928N full
PLEV Z (Pa) 17 i- 1000 100000 compressed
MONTH_GREGORIAN TIME 12mi 16-JAN 12:00 15-DEC 17:49 full
The incorrect grid affects my further calculation. I need to apply @DIN to my adj_EB_land_CNRM_AM5. Since the grid is compressed, pyferret doesn't apply cosine correction anymore. My proof for this is the value for AHT_land_CNRM_AM5 is around -100 to 50. The @AVE value is 4.021. So the adj_EB_land_CNRM_AM5 value is roughly around -104 to 46. If I apply @DIN for both latitude and longitude to AHT_land_CNRM_AM5, the outcome is 2.051E+15 W/m2 which makes sense. If I apply @DIN to adj_EB_land_CNRM_AM5, the outcome is -61.33 W/m2 which doesn't make sense.
list/L=7 AHT_land_CNRM_AM5[X=@DIN,Y=@DIN], adj_EB_land_CNRM_AM5[X=@DIN,Y=@Din]
LONGITUDE: 0.7W(-0.7) to 0.7W(359.3) (XY integ.)
LATITUDE: 90S to 90N (XY integ.)
TIME: 15-JUL 17:49
Column 1: AHT_LAND_CNRM_AM5 is SWABS_LAND_CNRM_AM5-OLR_LAND_CNRM_AM5+SHF_LAND_CNRM_AM5-STORAGE_LAND_CNRM_AM5
Column 2: ADJ_EB_LAND_CNRM_AM5 is AHT_LAND_CNRM_AM5-EB_LAND_CNRM_AM5
AHT_LAND_ ADJ_EB_LAND_CNRM_AM5
I / *: 2.051E+15 -61.33
So I tried with a single month to use AHT_land_CNRM_AM5 just minus the ave value which is 4.021.yes? let test=AHT_land_CNRM_AM5-4.021
yes? list/L=7 AHT_land_CNRM_AM5[X=@DIN,Y=@DIN], adj_EB_land_CNRM_AM5[X=@DIN,Y=@DIN], test[X=@DIN,Y=@DIN]
LONGITUDE: 0.7W(-0.7) to 0.7W(359.3) (XY integ.)
LATITUDE: 90S to 90N (XY integ.)
TIME: 15-JUL 17:49
Column 1: AHT_LAND_CNRM_AM5 is SWABS_LAND_CNRM_AM5-OLR_LAND_CNRM_AM5+SHF_LAND_CNRM_AM5-STORAGE_LAND_CNRM_AM5
Column 2: ADJ_EB_LAND_CNRM_AM5 is AHT_LAND_CNRM_AM5-EB_LAND_CNRM_AM5
Column 3: TEST is AHT_LAND_CNRM_AM5-4.021
AHT_LAND_ ADJ_EB_ TEST
I / *: 2.051E+15 -61.33 1.507E+11
These results are indicating that pyferret was not applying cosine correction to compressed latitude and longitude (which makes sense to ferret since it's compressed). But I have 10 models with 12 months' data, it's hard for me to subtract the value one by one. I am wondering whether there are some methods that I can transfer the @AVE value into a numeric or I can regrid adj_EB_land_CNRM_AM5 to be like AHT_land_CNRM_AM5 (I already tried with let/like=AHT_land_CNRM_AM5 adj_EB_land_CNRM_AM5=AHT_land_CNRM_AM5-EB_land_CNRM_AM5(this the @AVE variable) and set variable/name=tt adj_EB_land_CNRM_AM5 \ let adj_EB_land_CNRM_AM5=tt[g=AHT_land_CNRM_AM5,gt=@ASN] but neither of them worked). I tend to use transfering @AVE to a numeric method now. Any suggestions?
Thank you very much.
Xiaoyu
On Wed, Dec 4, 2019 at 1:04 PM Ansley C. Manke <ansley.b.manke@xxxxxxxx> wrote:
Hi,
Maybe this is just some detail in the syntax. The coordinate axes for the variable "temp" are not actually compressed, it's just that SHOW GRID is incorrectly saying they are. Are you seeing an error message? If this doesn't resolve things, please send more about the commands you are using and the results.
Your original example was with levitus_climatology, so let's go back to that.
yes? use levitus_climatology
yes? let test=TEMP-TEMP[X=@AVE,Y=@AVE]
yes? list test[x=@din,y=@din]
VARIABLE : TEMP-TEMP[X=@AVE,Y=@AVE]
FILENAME : levitus_climatology.cdf
FILEPATH : /home/users/tmap/ferret/linux/fer_dsets/data/
SUBSET : 20 points (DEPTH (m))
LONGITUDE: 20E to 20E(380) (XY integ.)
LATITUDE : 90S to 90N (XY integ.)
0 / 1: -538.5
10 / 2: -633.9
20 / 3: -530.8
30 / 4: -575.1
50 / 5: -335.4
75 / 6: -440.2
100 / 7: -459.0
150 / 8: -405.4
200 / 9: -287.2
300 / 10: -305.7
400 / 11: -234.4
600 / 12: -118.7
800 / 13: -113.0
1000 / 14: -73.9
1200 / 15: -52.0
1500 / 16: -35.0
2000 / 17: -6.6
3000 / 18: -13.9
4000 / 19: 7.7
5000 / 20: 0.5
On 12/3/2019 6:02 PM, Xiaoyu Bai wrote:
I picked the wrong word. I do want to remove the mean over lat and Lon. After that I want to apply @DIN. The compressed coordinates blocked @DIN to work. I tried with TEMP-Temp[x=@ave, y=@ave] and TEMP-number. They gave me very different results.
Thank you 😊
Xiaoyu
在 2019年12月3日，下午8:13，Ansley C. Manke <ansley.b.manke@xxxxxxxx> 写道：
Hi Xiaoyu,
First, what you have done with TEMP-TEMP[X=@AVE,Y=@AVE] is to remove the mean over longitude and latitude. That will not be less noisy. If you want to work with a less-noisy version of TEMP, you might use one of the smoothing transformations such as the boxcar smoother, @SBX. Here is a list of all the transformations:
https://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/variables-xpressions/XPRESSIONS#Chapter3-Transformations, and further down that page are detailed descriptions of the different transformations.
Now, it looks as if you've discovered an error in the SHOW GRID output. It should not be listing the x and y axes as "compressed". So we'll look into that.
yes? use coads_climatology
yes? let sst_demean = sst - sst[x=@ave,y=@ave]
yes? show grid sst_demean
GRID GSQ1
name axis # pts start end subset
COADSX LONGITUDE 180mr 21E 19E(379) compressed
COADSY LATITUDE 90 r 89S 89N compressed
normal Z
TIME TIME 12mr 16-JAN 06:00 16-DEC 01:20 full
However sst_demean does still vary in longitude and latitude. See how STAT shows 180 points in X and 90 points in Y, and a plot at one time and latitude varies in X.
yes? stat/brief sst_demean
Total # of data points: 194400 (180*90*1*12*1*1)
# flagged as bad data: 89622
Minimum value: -23.622
Maximum value: 12.722
Mean value: -2.1127 (unweighted average)
yes? plot/L=1/y=0 sst_demean
First, what you have done with TEMP-TEMP[X=@AVE,Y=@AVE] is merely to remove the mean. If you want a less-noisy version of TEMP, you might look at the smoothing transformations. Here is a list of all the transformations:
On 12/3/2019 4:38 PM, Xiaoyu Bai wrote:
Dear ferreters,
I noticed that when I subtract area average from a variable, it will make the variable's Lat and Lon become compressed instead of full. For example:
use levi_climatologylet test=TEMP-TEMP[X=@AVE,Y=@AVE]
sh grid TEMP,test
GRID GMS1
name axis # pts start end subset
XAXLEVITR LONGITUDE 360mr 20.5E 19.5E(379.5) full
YAXLEVITR LATITUDE 180 r 89.5S 89.5N full
ZAXLEVITR DEPTH (m) 20 i- 0 5000 full
normal T
GRID GMS1
name axis # pts start end subset
XAXLEVITR LONGITUDE 360mr 20.5E 19.5E(379.5) compressed
YAXLEVITR LATITUDE 180 r 89.5S 89.5N compressed
ZAXLEVITR DEPTH (m) 20 i- 0 5000 full
normal T
What I want is to get a less noisy variable, then apply @DIN to it. But this compressed grid was blocking me to use @DIN.
So I am wondering how to get my coordinate to full again.
Thank you in advance.--
Xiaoyu
--
Xiaoyu
--
Xiaoyu