Re: [ferret_users] LIST/CLOBBER doesn't trim the file

To: Ryo Furue <furue@xxxxxxxxxx>, ferret users <ferret_users@xxxxxxxx>

Subject: Re: [ferret_users] LIST/CLOBBER doesn't trim the file

From: "Ansley C. Manke" <ansley.b.manke@xxxxxxxx>

Date: Tue, 4 Dec 2018 11:03:54 -0800

Arc-authentication-results: i=2; mx.google.com; dkim=pass header.i=@noaa.gov header.s=google header.b=fz0xJ1Gx; spf=pass (google.com: domain of ansley.b.manke@xxxxxxxx designates 209.85.220.41 as permitted sender) smtp.mailfrom=ansley.b.manke@xxxxxxxx; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=noaa.gov

Arc-authentication-results: i=1; mx.google.com; dkim=pass header.i=@noaa.gov header.s=google header.b=fz0xJ1Gx; spf=pass (google.com: domain of ansley.b.manke@xxxxxxxx designates 209.85.220.41 as permitted sender) smtp.mailfrom=ansley.b.manke@xxxxxxxx; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=noaa.gov

Arc-message-signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-archive:list-help:list-post:list-id:mailing-list:precedence :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:to:subject:dkim-signature; bh=Z9OO7Vcc1C4kMjL5/AzGMy20uErr0t2MwCzXwSvdATc=; b=t/agqmmjDLNyRwJYPFMbgKD34hBzcP2RdWScyInph9BlwAx3lbrdFRBFvQtEo6Dhqp +gk/64n33LwU/xQ/nEaUpJGOvgionUnfli5bCMW6cxS56pb3O3yeH+rUnQ3b8uBK3/6n TCSfSDfokmp/0ZZEFhs1Fx6RoYLiUxtYMWtdYTXM3JqNKvS+0RyscjhdDxUzALADiqoh l0WQ0s0CxbwlxzGpCTMUyPL5T1Tll36dgZiegRMzX1HsEZMwCHM7IdBf5bs/jZS0L8yJ CiUS+tWOGpsmnIGLjfgkz3iIVg91m9NdfKcqXO5vehqM//aX+Z9q/gqPGTE+KBeD+iNS SThQ==

Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:to:subject:dkim-signature; bh=Z9OO7Vcc1C4kMjL5/AzGMy20uErr0t2MwCzXwSvdATc=; b=uNbl+qdIcSmmVCoD8p1QQWNEP9daO440vSoths3WZQ4WRQFaoNTDy09/+zh0t/IDdn AY8ND00nao27QpwGxFZXtFD8KKR7lK0Qe9f3giJbfcRNbobu+1ymCWqnTfIBkflXqlX4 nm3sH6/+kArWp6jRmKQMG5ap+y+5MaC2kcjy3lw2mlE46SqRwmuvQQrlhTZ3WSH65xej qg97dEMZToHze39ObwQHPR7xcZmxFB64kqsTyZGM87+/fPRb28rw4e1U3Wb7fNBMY7jL jvLj3N0d6Fa5uIhMzKUZWwrJZzgNjM7MhB86dwKBZSMr2QsjGG1+TyLxDPiiMhP5pOpu BRog==

Arc-seal: i=2; a=rsa-sha256; t=1543950237; cv=pass; d=google.com; s=arc-20160816; b=JV3bDkRIN1Tvi+1PFaSxeUvZv7nrQlA3nL92Vtv4tyvPN280MRPAWiRXCNc7/TNkF8 tqtp1SeucDSZx4Zz2oWVP0oFM3xrZb6/RexidbCFYUY7HuTLY5sJz0G6AG99DIkfcAX8 0J1o0URcjqJ/1tFFxkI4JotTaG636GMeReK7njFwqfIHqu10ZzHHA+ShYrtsNull+wy1 x4RvklvvvW6QR4S6Iok0TuujLsft8o0XP8DdOFUXHRFLOhTNu9WOuG+aDccWeHSK9GNF ySM6nz8kXZcrd7Ypv7r06i/bD5pm9htVCmwFahRtLEP9OZOtb1zCCl7E3mA99F8yf+qy 3Qwg==

Arc-seal: i=1; a=rsa-sha256; t=1543950237; cv=none; d=google.com; s=arc-20160816; b=CvYn06mWiEs3WEqJ3KM2zo4pzoM0e7Doy7jSt+qergTz0gVCOoJzRax91DULo5e+Nw FM4eY3uO5V2b+ERA9voTnd/NdARmAnWL2zybc3iAy2lpCJdSIovOcoPePph4ISF6e2+U ceB4GtMDGgW8j1ThjWlbk9spEiWwL3g2aCeEBJq9e33t0QAbrHucNaUaT0i3zD/H5Ewv O0ETL9DLt2awKfH0uSjx+zmyAyS0YcmX4Vb2VUGquSyZBXUVem3sDh/YW/+FObcn4vIh 9XoL3xu92HrRGO0oXCym4uVdZ6B5kMLsiW9UCh4gT7k3y8mz8GM6VXHlbAQad00Eci7z 5DjA==

Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=noaa.gov; s=google; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :list-post:list-help:list-archive; bh=Z9OO7Vcc1C4kMjL5/AzGMy20uErr0t2MwCzXwSvdATc=; b=aQR7Zp+3fQIix9vJO6ShRocGgKnXAt/R84IUi20luS9tO1yoxR266QSRCgxnu5API0 C8rbJv3RYaa5piFY7AKak7hEK8kbWqwyx2vil6YipJQunDDk8gY0/inYzu51oNCUxKSi ZmQ3FqB63yfumb9WNNyVbBzDF+0yQ5gO2J3engaei+TW4/XQ111BdaJg3WBYi2yleTtO VPq8psTnnIg7/2z57g94fbOxIX9+oZUIRWYmCuwUhtetkpOo8Mh2aZwMkPlQ7Tg7eiPW FGMEp0cLgCDhuj5rnp44WxI/RKS58gkD/N1tu2h/IubSxtvUz5/DXWbSOykbVE2h9USS 8h9g==

In-reply-to: <CAEwQpaJ9yQ96iUjoLY8RbBbHPjK57NsVC2RHJM=j_FEvNP_gVw@mail.gmail.com>

List-archive: <https://groups.google.com/a/noaa.gov/group/ferret_users/>

List-help: <https://support.google.com/a/noaa.gov/bin/topic.py?topic=25838>, <mailto:ferret_users+help@noaa.gov>

List-id: <ferret_users.noaa.gov>

List-post: <https://groups.google.com/a/noaa.gov/group/ferret_users/post>, <mailto:ferret_users@noaa.gov>

Mailing-list: list ferret_users@xxxxxxxx; contact ferret_users+owners@xxxxxxxx

References: <CAEwQpaJ9yQ96iUjoLY8RbBbHPjK57NsVC2RHJM=j_FEvNP_gVw@mail.gmail.com>

Sender: owner-ferret_users@xxxxxxxx

User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.2

Hi Ryo,

You're right that of course /CLOBBER should open the file in such a way that it's wiped out and contains nothing but the data currently being written.

Thank you for the example - that same behavior goes back to all the older versions of Ferret. Looking into the code, this seems to be intentional, but I can't imagine why this would have been implemented this way (maybe it's just an oversight). There's nothing about this in the documentation or old tickets. Clobber is clobber. We'll change it.

Ansley

On 12/4/2018 9:24 AM, Ryo Furue wrote:

Hi Ferret users,

I've found that LIST/CLOBBER/FORMAT=STREAM doesn't trim the file if it already exists and is longer than the data written.

Suppose that our file "out.bin" is currently 1000-byte long and we are now going to overwrite it with 16 bytes of data:

yes? list/clobber/format=stream/file=out.bin sixteen_byte_var

Then, only the first 16 bytes of "out.bin" is overwritten and the remaining bytes retain their original values. That, I suppose, doesn't agree with the spirit of CLOBBER.

[In Fortran, CLOBBER should mean OPEN(STATUS="REPLACE"), I suppose.]

I provide a self-contained example below. I tested the code on

PyFerret v7.43 (optimized)

Darwin 16.7.0 - 09/14/18

on macOS Mojave.

Best regards,

Ryo

-----------------

set data levitus_climatology

! Create a file of 2 KiB filled with zeros.

SP dd if=/dev/zero of=out2.bin bs=1024 count=2

list/clobber/format=stream/file=out1.bin temp[x=180,y=0]

list/clobber/format=stream/file=out2.bin temp[x=180,y=0]