[ferret_users] memory strategies for handling large computational requests

To: ferret_users@xxxxxxxx

Subject: [ferret_users] memory strategies for handling large computational requests

From: Patrick Brockmann <patrick.brockmann@xxxxxxxxxxxx>

Date: Wed, 22 Nov 2017 11:57:04 +0100 (CET)

Arc-authentication-results: i=3; mx.google.com; arc=pass (i=2); spf=softfail (google.com: domain of transitioning patrick.brockmann@xxxxxxxxxxxx does not designate 52.222.5.176 as permitted sender) smtp.mailfrom=patrick.brockmann@xxxxxxxxxxxx

Arc-authentication-results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=lsce.ipsl.fr); spf=softfail (google.com: domain of transitioning patrick.brockmann@xxxxxxxxxxxx does not designate 65.127.216.226 as permitted sender) smtp.mailfrom=patrick.brockmann@xxxxxxxxxxxx

Arc-authentication-results: i=1; mx.google.com; spf=pass (google.com: domain of patrick.brockmann@xxxxxxxxxxxx designates 194.254.240.33 as permitted sender) smtp.mailfrom=patrick.brockmann@xxxxxxxxxxxx

Arc-message-signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=thread-topic:thread-index:mime-version:subject:to:from:date :arc-authentication-results:arc-message-signature :arc-authentication-results:arc-message-signature:message-id :arc-authentication-results; bh=hsw2YwW1Clbti0Y0ZRknI3EpCObbJBaPmvSRawx6qb8=; b=04RLXUPE6HyLbkJY/a/zM377bCUH8Kc6+Voez6p5VroJw+4CzSMZ3gFA0V6D6/Vxun csDK5W9hXQ4nyfc3GfhaoZqDH4RS7J9Rkl95ukQx3yWAZ+1QU4Ykxz37H9LLGhVaqnlK p1kJBQAOAl4wBmu9g8D79NBhjKjXj38wfHVFv2oFsM7kDOoj/XyiwetU/sjAWXYJptyt puUMeBgnlHGJR9Sj9Q8+TpEu9oV1sj9BxRo9VJ+GK0kyv3oOgTg3mXCDNwD058UHLSXA AjxfHFLB6UZ3Ppzr//r+PdQb8KM2R7iufPUJZ6mN/RIpct4PtfaR86M48CX4f0JXdhaI Pf7A==

Arc-message-signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=thread-topic:thread-index:mime-version:subject:message-id:to:from :date:arc-authentication-results:arc-message-signature :arc-authentication-results; bh=hsw2YwW1Clbti0Y0ZRknI3EpCObbJBaPmvSRawx6qb8=; b=oU4wzBQUVLbH5R3d02ictxKhRJ91Kxv/sQi5LdCvO+e8g478gN0zsXJDIqt+47pLMg R7vCGy0vTpBcNZyxC67e8zK7cm2HKFbSA7nUvmMjBAoaEWW1CSVWgbs0ZbB44foxC5b5 hCnkXYIx+yk4Owk6jgrKfDqLQqomD7HhGtCYbZJeYI48NZjUQhRAZ+7/7ISiNY6YdgDk 5vYz1UD5acYs1relFRTAgjkG7njqDSYSzULtLhx3kZDiLRFB7+/XtmFA0QxyrRQpeRZJ UFjqMHfnxFzflhhV3jQp0aJj4KSbZ2ifCfuWIxWIBZobX/a88wNIIPvuUxT7YHJPPCDE Gjnw==

Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=thread-topic:thread-index:mime-version:subject:message-id:to:from :date:arc-authentication-results; bh=hsw2YwW1Clbti0Y0ZRknI3EpCObbJBaPmvSRawx6qb8=; b=K0gqlm8CyiwqfKyX0A+rcYj3JxLbvVaPy6gjOj5X+qAYcLZi5+fHJzLSAEUmj+hhRt NSFsVZrBo2nCtqpg4JR33u5Wup2abalpPgxwqS0BnuOhsK6Gchp2ttwNXXQtw1i0AQHi UzAp6M+sEVHTCXtkdphToKt5vOQmCU8chuuYfNFi87X1SdxxkKqMk9o6kAtHyj2dOlRX OE0RsBrSlB+2WIvNjnJqr160k8Ux50PDHiu2SDgSdDDsFdb0W06wOjoRbSvoIxU7V5La SN+BW+ALLb0GUkqa8mLzXciVFn3OWJdw8s8wLnt3Eb4kUa5LdmjHbkrByHGTzg3uTdH/ Q+AQ==

Arc-seal: i=3; a=rsa-sha256; t=1511348238; cv=pass; d=google.com; s=arc-20160816; b=oHpBDHcRtaNszsLAG90Xqv7XEHrUOXBAKmfCAoyuX0tCbdEA6MiN2dS0NM4NAbrQB0 xYMGdNrEzGUgvTyp1DD8a11Q/kbTURI+z3bKrddgSzGDSG5SgrONp1nqfpdtxdawIeXV K/zhfu0Rn1lBl2QXTcqdmw/x7sz6SKjgVVSExDMNraiRke5T5Xh0JN9Ab79pvW044/3G t2HmFvtEaGu4pCoDhzPuy3KCITNULOkHFgjzq58v2EV8h6U4yBd6S3aQ1fLy0R5jEWy+ qNuoEcPxTzWj6XTGds+s8GREJkJX2tnZKgPFCQmS3eXK+NU4f4U6JOqKp0pWFJBFm8K7 mwSw==

Arc-seal: i=2; a=rsa-sha256; t=1511348231; cv=pass; d=google.com; s=arc-20160816; b=cR4dX47eaAy71IBesp21Cem2CA9JFIfVJvjKYVStA/nJ1qQG5GXxySILlYb41cP86V 9YYUfRi6SrmrMOqEm4WGR259atOMQr4Jsf3LNRAhKEfRQi5c6p52S8ZmVlXRmqslb9Sf 81hajxvjno1YU81hhD1WEQoLxWQ3A5eQcIQtG8QvIEkTotr60LXVsi8ffI+XVPT5TZsC A/NQXY39mAfUoGYqprF7CIFPGz6muUP0qU/eB3lV7lWUriYdjVlF4N5nDlobTWUReX7q /5eNGdZAuhuKDiDxXst1SjeVAMOzljWEJ0RaPQcBjBS48IAti1q6Oe5/XKmqkKPT/gAW xYZg==

Arc-seal: i=1; a=rsa-sha256; t=1511348226; cv=none; d=google.com; s=arc-20160816; b=LrjzTkkH3AtmQbwdaMVmh198tv6U43PO4EiVAlK5DLlpvCZajKsT5Wh3Lc6w31Q0qe uTvvWY3ILTgR9vFbGZictRKlwiBVcLqqqHFo4k2jQcnieKDuGvZwbKu0n769oMU03G9D VCc6CsB/huGmi0gP5GhSRuJo/jxsoWbnymVCtH2ukIIl10wGKYCPUxHvdYk5LozHvdcr A8BLE5wnCy6TZwd6aKNfBJxSLrBvC+9kiyAHbKyKEziOFuwjkSGG0w4+cNZxjG2vpneN +Ao0e1lKxrxF7T3YRLqZBDT4qeyTrMW2Hf6QTBti3FFuyxrutoecZystueROiUckGFys Vhag==

Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=noaa-gov.20150623.gappssmtp.com; s=20150623; h=message-id:date:from:to:subject:mime-version:thread-index :thread-topic:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive; bh=hsw2YwW1Clbti0Y0ZRknI3EpCObbJBaPmvSRawx6qb8=; b=t2FgUwHZbtFywOg5RiFz3+UccZYeu2Ydu9AojsS9YVIAcpDQ2wNHyDNnA36aCYyM/g BHN4p08p/g5iyDROqY5yy06xDa1kEzK794iwFgfBJmuAN78a02qrOxou5C9QgEzpOVUL /ExylUUsVEhQesw2yEWEaEAGIKCDWBm0hQv/EQaLEGZkLTxJT03/CXXyTYHbq0/zORI2 isnTPEVeUZv91DXPLiIS1qsJ7F84bOKCY9l6XI5UPWq5miKFH7aES6YMTghwivbZJhn+ +DNURsbORSZK3avDtn976f+RYL2P69qxSEqp07XCJSBoytfOyzyXDXizjwVo1xByvu2J Y0EA==

List-archive: <https://groups.google.com/a/noaa.gov/group/ferret_users/>

List-help: <https://support.google.com/a/noaa.gov/bin/topic.py?topic=25838>, <mailto:ferret_users+help@noaa.gov>

List-id: <ferret_users.noaa.gov>

List-post: <https://groups.google.com/a/noaa.gov/group/ferret_users/post>, <mailto:ferret_users@noaa.gov>

Mailing-list: list ferret_users@xxxxxxxx; contact ferret_users+owners@xxxxxxxx

Sender: owner-ferret_users@xxxxxxxx

Thread-index: UF1BJ8gnuibMcxi/LZfazRGXCNVAbw==

Thread-topic: memory strategies for handling large computational requests

Hi ferreters,

I would like to plot time series with quite huge file (8.3G) from a variable XYZT (144x143x13x8280).

I have worked with last 7.2 ferret release and tried different increases of memory without success.

I always get **ERROR: request exceeds memory setting

Next step would be as suggested from doc (http://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/computing-environment/MEMORY-USE)

to break up my request into fragments.

Is it the best solution in my case ?

My ressource is available from

https://vesg.ipsl.upmc.fr/thredds/catalog/IPSLFS/brocksce/tmp/catalog.html?dataset=DatasetScanIPSLFS/brocksce/tmp/CM6012.1-pi-ttop-02_23200101_30091231_1M_transpir.nc

Typical code lines are:

yes? use CM6012.1-pi-ttop-02_23200101_30091231_1M_transpir.nc

yes? let var=TRANSPIR[k=@sum, x=@ave, y=@ave]
yes? plot var[l=@sbx:120]

! the following pass because I have limites time range (1:1200)

yes? plot var[l=1:1200@sbx:120]

Any help welcome.

Regards

Patrick

--
Data Analysis and Visualization Engineer
LSCE/IPSL, CEA-CNRS-UVSQ laboratory
LSCE - Climate and Environment Sciences Laboratory
IPSL - Institut Pierre Simon Laplace
--