Re: varaince and var_n jnl files

[Ansley Manke <ansley.b.manke@xxxxxxxx>]

Hi all,
Thanks to Sudheer for reporting this, and Mick for seeing what the trouble
was.  The script var_n did not apply the n/n-1 correction to all of the 
variables
that are part of the definition of CORREL.  The corrected var_n.jnl is 
below,
and we'll get it fixed in the FAQ and the Ferret distribution.

Ansley Manke

----------------------------------------------------------------------------
CANCEL MODE VERIFY

! var_n.jnl
! Description: Correct "GO variance" definitions by n/n-1 factor

! 6/24/2004  add new definitions for masked variables, so that
! CORREL will be corrected (see the FAQ on computing correleations).

! compute n/n-1 factor to get the variances exact
let l1 = l
let nm1 = l1[l=@max] - l1[l=@min]
let ndnm1 = (nm1+1)/nm1

! redefine terms with n/n-1 factor
let/title="VARIANCE OF P"        P_VAR = P_DSQ[l=@ave] * ndnm1
let/title="VARIANCE OF Q"        Q_VAR = Q_DSQ[l=@ave] * ndnm1
let/title="COVARIANCE OF P AND Q"  COVAR = PQ_DSQ[l=@ave] * ndnm1


let/quiet/title="VARIANCE OF P WHEN Q PRESENT" P_VAR_MASK = 
P_DSQ_MASK[l=@ave] * ndnm1
let/quiet/title="VARIANCE OF Q WHEN P PRESENT" Q_VAR_MASK = 
Q_DSQ_MASK[l=@ave] * ndnm1


SET GRID P
SET MODE/LAST VERIFY
----------------------------------------------------------------------------


Mick Spillane wrote:

> Hello Sudheer,
> 	I agree that there is something wrong in the way variance.jnl and
> var_n.jnl combine in the demonstration link
> 	http://ferret.pmel.noaa.gov/Ferret/FAQ/analysis/correlations.html
> The reported maximum correlation of "Maximum value: 1.0841" near the end
> of the page is clear indication that something is wrong, and it is not
> just roundoff error.
> 	The trouble lies in the use of var_n.jnl I think.  The purpose of
> the n/n-1 factor is to get unbiased estimates of the variances and the
> covariance.  But the correction factor should cancel when the correlation
> coefficient is computed because it appears in both the numerator and the
> denominator.  So in the demo, while it is correct that the covariance
> value (and the p and q variances) should change before and after the use
> of var_n.jnl, the value of "correl" should not have changed. I think that
> some of the "masked" variables are not getting the n/n-1 correction.
>
> 	If I am right in this, if you want to compute and plot "correl"
> you should do so without using var_n.jnl
> !---------------------------------------------
> SET DATA coads_climatology
> LET p = sst[x=150:220,y=0:40]
> LET q = airt[x=150:220,y=0:40]
> go variance
> stat correl
> !...
> Maximum value: 0.9938  !!! That's more like it !!!
> !---------------------------------------------
> Then, if you were interested in reporting or plotting the unbiassed
> variances or covariance, you could define the correction factor
> yourself (12/11 = 1.090909 in this example) and apply it, skipping
> var_n.jnl entirely.
>
> Good luck,
> Mick
>
> |!!! Mick.Spillane@noaa.gov !!!|
> |__Room 2070 Bldg#3 NOAA/PMEL__|
> |____Phone_:_(206)526-6780_____|
>
>
>