[ferret_users] RE: memory strategies for handling large computational requests

To: Patrick Brockmann <patrick.brockmann@xxxxxxxxxxxx>, "ferret_users@xxxxxxxx" <ferret_users@xxxxxxxx>

Subject: [ferret_users] RE: memory strategies for handling large computational requests

From: "Kaduk, Jorg D. (Dr.)" <jk61@xxxxxxxxxxxxxxx>

Date: Wed, 22 Nov 2017 12:45:34 +0000

Accept-language: en-GB, en-US

Arc-authentication-results: i=4; mx.google.com; dkim=pass header.i=@uniofleicester.onmicrosoft.com header.s=selector1-leicester-ac-uk header.b=NdF4Y7g9; arc=pass (i=2 dkim=pass dkdomain=uniofleicester.onmicrosoft.com); spf=neutral (google.com: 52.222.5.176 is neither permitted nor denied by best guess record for domain of jk61@xxxxxxxxxxxxxxx) smtp.mailfrom=jk61@xxxxxxxxxxxxxxx

Arc-authentication-results: i=3; mx.google.com; dkim=pass header.i=@uniofleicester.onmicrosoft.com header.s=selector1-leicester-ac-uk header.b=NdF4Y7g9; arc=pass (i=2 dkim=pass dkdomain=uniofleicester.onmicrosoft.com); spf=neutral (google.com: 52.222.5.176 is neither permitted nor denied by best guess record for domain of jk61@xxxxxxxxxxxxxxx) smtp.mailfrom=jk61@xxxxxxxxxxxxxxx

Arc-authentication-results: i=2; mx.google.com; dkim=pass header.i=@uniofleicester.onmicrosoft.com header.s=selector1-leicester-ac-uk header.b=NdF4Y7g9; arc=pass (i=1 dkim=pass dkdomain=uniofleicester.onmicrosoft.com); spf=neutral (google.com: 65.127.216.227 is neither permitted nor denied by best guess record for domain of jk61@xxxxxxxxxxxxxxx) smtp.mailfrom=jk61@xxxxxxxxxxxxxxx

Arc-authentication-results: i=1; mx.google.com; dkim=pass header.i=@uniofleicester.onmicrosoft.com header.s=selector1-leicester-ac-uk header.b=NdF4Y7g9; spf=neutral (google.com: 2001:630:306:10::24 is neither permitted nor denied by best guess record for domain of jk61@xxxxxxxxxxxxxxx) smtp.mailfrom=jk61@xxxxxxxxxxxxxxx

Arc-message-signature: i=4; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-archive:list-help:list-post:list-id:mailing-list:precedence :mime-version:spamdiagnosticmetadata:spamdiagnosticoutput :content-language:accept-language:in-reply-to:references:date :thread-index:thread-topic:subject:to:from :arc-authentication-results:arc-message-signature :arc-authentication-results:arc-message-signature:message-id :arc-authentication-results:arc-message-signature:dkim-signature :arc-authentication-results; bh=g3DRrfSGglIm+jY5anqU0Zyh/KFfbYFsMZKDdvKZjWU=; b=S0AyTSOjqygHJQTOzoUgZdEtVJIgnaMZdBl29pv3OgfL9Cfq4Nt2vSpHZaAeqIm/tc EdWLZe6OG90tQ5ldkNCPRfX/Qjjvm9jg+QQJS8mocjUpHc35B1sD66wb5ojQNQR3/zjD mnCXYjZ42D+huvENMgNXUFaZWBDkVeUq38mUrqVN1bfQUJjuhDrGur77plV8yNBBNouw gjmAVxB9QFv/Wj6laNDbmse5/+n7SrhPnqxK8GDqB2h+p6oCGOyRN+X2sWm7nyEek+5C U1ZJUUeio7K1cBkuWcn1fTG5mH+/TdiF9ybC16wja33wZcCn2uhOBTFHnO4hgNuJ4dN4 qISg==

Arc-message-signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:spamdiagnosticmetadata:spamdiagnosticoutput :content-language:accept-language:in-reply-to:references:date :thread-index:thread-topic:subject:to:from:dkim-signature :arc-authentication-results:arc-message-signature :arc-authentication-results:arc-message-signature:message-id :arc-authentication-results; bh=g3DRrfSGglIm+jY5anqU0Zyh/KFfbYFsMZKDdvKZjWU=; b=zcVO4ib7fKErcOGWH4ZGe/pNNtmoJUr35IOHUHoH5662rAP5sz3ln/Rpt6tdcpYpCR BrYashWYSbdaIjsiiRvFv0vDSNiRugmFB7O45wYhPqOVGHYq24aOb4IFmGHqYi87voPh fzotQXv74xRtLy7qakqAFwwcDkXiJD8VvvMx07hBUs1RneBSbj19sV2xq+Hrs4fvPqW9 GRSB0oQwjJt2x8yk7+9akFFPlSpD/h9JY4TXPFQUO401pQEm+XmleCO4BFLqnQeKR1TG KIy3i/5rp6LuUX3342hM1/DFdm1qDawhAO3PAjdabTdrJrJ3XYJbiBTOCd99Itf+5kRa ZTKg==

Arc-message-signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:spamdiagnosticmetadata:spamdiagnosticoutput :content-language:accept-language:in-reply-to:references:message-id :date:thread-index:thread-topic:subject:to:from:dkim-signature :arc-authentication-results:arc-message-signature :arc-authentication-results; bh=g3DRrfSGglIm+jY5anqU0Zyh/KFfbYFsMZKDdvKZjWU=; b=cvgrLOKy5DDiWululzj8RROp0bURvhP/ZJw2Ya4RFmmdL9stAzqClvwkhLkZzlFq28 Wr//mIP6mVWMCTt7VL52Pa+6uLll+XjXaSICj6QTX83xK1DkSFI0BbzE6kYPyQwKi8bn EFNloQpqH1FYLiEc1IcaIjtg0Fad1F54oEnw//+U/t1MzwNkth2H3Mza6yNHiCEf6+pI EDVggMhNWnPDFwwKpK2pFtZURTjXu6lj+svpLq+C8VcXZoWPRgT6m3dR4bSxHrleBLbu gBSut8AvY6kZhuVRWkuFJQt9Zrs6k9jnMVq2VOJICWAUWcmD5qD71znfoBtRKcSC6x4Y wHHg==

Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:spamdiagnosticmetadata:spamdiagnosticoutput :content-language:accept-language:in-reply-to:references:message-id :date:thread-index:thread-topic:subject:to:from:dkim-signature :arc-authentication-results; bh=g3DRrfSGglIm+jY5anqU0Zyh/KFfbYFsMZKDdvKZjWU=; b=Czi5dXoxEWaXCy7kAd68YF4zlztmtyQDbD0K7X3PwVXJzaMeWuKOyym/fm4VHz0z6C LEb+2jnP6c4PT3szAfsLc0MyB+HY4eqmSx/978hYWBwT6g6ht5dWlaaXeRfrD0QbcVMT d8W58tooqWozl4ukEpEhno/w329OtskUBaZIsC253kddZe69rUiuWZq1s1MGzNgXGcyV vFnbsWcuYyUvd2jYZZXkRGELGawrhvJHBHIkpXBgdwBCn1mzW0vIDF88w5QXv8Qt5Iat G3SBWZSefGiVGHL+i/HOkBwYEpiFvrMztE9kwAAP/J24PKUuYGWSaIibVzXvFX6SsKdi 4/xg==

Arc-seal: i=4; a=rsa-sha256; t=1511354750; cv=pass; d=google.com; s=arc-20160816; b=i6TT4PM+3XlZ2WhEEGp4G9qwCTZ0JZKgbGHeGZL82jTIZKprdFNbXSqeu81HHtOo/N P82PZLLprqVQDhl+2cJ76bTIfyX9Wkr/sTPxe5KleTQMcv/g0jcjWyjS7NptSxXo5A4m pb+f8XbkUD7d3TByUQVHK1PTxD7IF73wN3RavSul6MjYf2eMaCAyTEJ6HzjB9qTiTjUC be09n6twVdhPA0h5pJlQzJHsBJg8ahrOQUdmgnrhC437KATqM1Mkb1HmPrRietmrU+m9 KybbXgt5ojOnfrqA+wzKefwr3dUWtk0TdO9961dTI5qQmzgSZ8o5s/eQ/8lYVmukRQIp 4s4w==

Arc-seal: i=3; a=rsa-sha256; t=1511354749; cv=pass; d=google.com; s=arc-20160816; b=jmQShAkQ9ropPwUTBZfKZzwf9QPtH7BrUny8mgy1f/eKVWBB3pir1mdhEV39IAg8Bm ROHOkzB10W/+OH3BQt0r7aP2oKoykTzGXmlv6b363J1kByOicRTGUT9fJIY4NShUZDH8 KpxPx57tgK4WGgimPhwf8Um7EmJMF0ops7pAW0I1j7htAJS/ZHzmnd25l8ZqUvdQCtKc hDO0FptSGotJUG9uVQSsGZevSD0SVzWx4Ka/cOZnpxnPXxmoULM7TNDv/nUN+qwwB8xY 7zyz2S0F2jGvSJJ+vvzdkCbPBrPLYiL8OaXGU3Oh3JHWk6C80tEwBRhd65wQIaR39MS0 hD6g==

Arc-seal: i=2; a=rsa-sha256; t=1511354744; cv=pass; d=google.com; s=arc-20160816; b=cPY85tcsmXNWCzu671OaZLayfXRlCnR9wh8r+54RGQWem5LnmPULO5xgDYrVF0CcHj IvpEq/GJ9kUoOHWoSi4/Jnmgm6qNeC3+fdLDcMPlEX/qAWxxg7LSuQyM+g1hGW+kEQp1 xVMknpkjxqy6OoUg6lgVPPio+gJBYcd67fKLIS7iB9Uts5McNC+24sKRK7eLAWhu+Q2q auPPLo+J9kbICTt557XO6mQqvwsbbI5CTjbhMTOdYV95FCawJTqvp/9YKvsAI4r8XM1w eciN7P2qxIJFBXAlYke49N9ZVj/F/ly7z21j34nLMSz2fYfURL2B2L47DeyyhnVxO6Bt MH5g==

Arc-seal: i=1; a=rsa-sha256; t=1511354736; cv=none; d=google.com; s=arc-20160816; b=mm+IjqsYnpuktWtsu9tKgCHI4USTuFWaP8i1p3imvV2KZcmV0lUz7vz9BD69kjoLmQ OwC3+6prUgjvvsov36ADvDcouRHrDbbVrlfQp/dsJ4DMEftw7+YNeNUXMhsj0r6fK5ku KVyImArq2yJoX1JMN7JdcY2SypF1b3AL8tDLMRsNsPxhS4F/RKgpBOk6IvZZAnQ7o6uM hWz+Bp2ycyRNAwdv2g31LuiEfqODnbVRojvTOrk23KyRBry+mK7RhoeR4cezMhSwRzzZ s+3znmM2K++VbKu/ftohBFkdWo64XV1ks7g/hjg44pzhPFdNS5xZHv12WsjZLqX39js2 G93A==

Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=noaa-gov.20150623.gappssmtp.com; s=20150623; h=message-id:from:to:subject:thread-topic:thread-index:date :references:in-reply-to:accept-language:content-language :spamdiagnosticoutput:spamdiagnosticmetadata:mime-version :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive; bh=g3DRrfSGglIm+jY5anqU0Zyh/KFfbYFsMZKDdvKZjWU=; b=xFpwtsy3Pz/N4eagI+UgAoe+cSfzo9LFpJosSNzVAlceoodJoMtlmAsrXd2JGIBA+V 06JCsS/wMeWprj3vngU7Q3vhlGZBtg1wlQc2XlZCqd/UPkzZvfuQlLpIros5628NULL9 k/kOG6mixtnnMQmsmH0n9/Ab8VcqOsuWcNiAMzsZJTCQtvbX1vLcFlTsY52dVX6j279J a+mL5yOMKv590PL667/tgCvYjKCGyqhDAM8Dyjms2djswF/D69HzrUA81D9eeK0AZnmy uMCGYW9YzZDYGXs1XQGbfeX+OvLZ/qOlq3vwRF0K04QjBnXN2+aN0y2FkMHr0jALrdf1 7KEg==

In-reply-to: <400769719.1005202.1511348224834.JavaMail.zimbra@lsce.ipsl.fr>

List-archive: <https://groups.google.com/a/noaa.gov/group/ferret_users/>

List-help: <https://support.google.com/a/noaa.gov/bin/topic.py?topic=25838>, <mailto:ferret_users+help@noaa.gov>

List-id: <ferret_users.noaa.gov>

List-post: <https://groups.google.com/a/noaa.gov/group/ferret_users/post>, <mailto:ferret_users@noaa.gov>

Mailing-list: list ferret_users@xxxxxxxx; contact ferret_users+owners@xxxxxxxx

References: <400769719.1005202.1511348224834.JavaMail.zimbra@lsce.ipsl.fr>

Sender: owner-ferret_users@xxxxxxxx

Spamdiagnosticmetadata: NSPM

Spamdiagnosticoutput: 1:99

Thread-index: UF1BJ8gnuibMcxi/LZfazRGXCNVAb+xE0ALw

Thread-topic: memory strategies for handling large computational requests

Hi Patrick,

Seeing that you work with summed/averaged data, what about using nco to do the calculations first and then feed the averages into ferret? This should be much faster and is only a couple of nco commands.

I found nco really fast and easy to use – though my files have not been as big.

Just a thought,

Best

Jörg

Jörg Kaduk

Associate Professor for Environmental Modelling

Plagiarism Officer of the School of Geography, Geology and the Environment

http://www2.le.ac.uk/departments/geography/people/jkaduk

@jkaduk

http://orcid.org/0000-0003-4051-3081

e: jk61@xxxxxxxx

t: +44 (0)116 252 3848

School of Geography, Geology and the Environment, University of Leicester, University Road, Leicester LE1 7RH, UK

Elite Without Being Elitist

From: owner-ferret_users@xxxxxxxx [mailto:owner-ferret_users@xxxxxxxx] On Behalf Of Patrick Brockmann
Sent: 22 November 2017 10:57
To: ferret_users@xxxxxxxx
Subject: [ferret_users] memory strategies for handling large computational requests

Hi ferreters,

I would like to plot time series with quite huge file (8.3G) from a variable XYZT (144x143x13x8280).

I have worked with last 7.2 ferret release and tried different increases of memory without success.

I always get **ERROR: request exceeds memory setting

Next step would be as suggested from doc (http://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/computing-environment/MEMORY-USE)

to break up my request into fragments.

Is it the best solution in my case ?

My ressource is available from

https://vesg.ipsl.upmc.fr/thredds/catalog/IPSLFS/brocksce/tmp/catalog.html?dataset=DatasetScanIPSLFS/brocksce/tmp/CM6012.1-pi-ttop-02_23200101_30091231_1M_transpir.nc

Typical code lines are:

yes? use CM6012.1-pi-ttop-02_23200101_30091231_1M_transpir.nc

yes? let var=TRANSPIR[k=@sum, x=@ave, y=@ave]
yes? plot var[l=@sbx:120]

! the following pass because I have limites time range (1:1200)

yes? plot var[l=1:1200@sbx:120]

Any help welcome.

Regards

Patrick

--
Data Analysis and Visualization Engineer
LSCE/IPSL, CEA-CNRS-UVSQ laboratory
LSCE - Climate and Environment Sciences Laboratory
IPSL - Institut Pierre Simon Laplace
--