Why does my kernel restarts/dies when displaying or plotting values in an xarray DataArray?

[shreyatrivedi26]

I am working with a NetCDF file (ds) which has one variable 'zg' and 4 dimensions (time,levels,lat,lon). I am trying to select a coordinate point at one level. However when I made the selection and tried to look at the DataArray (using ds.values) or plot the time-series (by using ds.values.plot()), my kernel freezes and dies in sometime. I get a pop error that "Kernel is restarting". I am not able to figure out the exact problem as this problem is not occuring when I plot the spatial plots (for example a single time-slice ds[0,:,:].plot()). It has not happened before as I have worked with multiple CMIP6 model files before.

Following is my code:

import pandas as pd
import matplotlib.pyplot as plt
import xarray as xr
import numpy as np
import dask
xr.set_options(display_style='html')
import intake
from xmip.preprocessing import rename_cmip6,promote_empty_dims
import cftime

cat_url = "https://storage.googleapis.com/cmip6/pangeo-cmip6.json" #not a local but object storage col = intake.open_esm_datastore(cat_url)

#Using Geopotential hts for calculation:

cat = col.search(experiment_id=['piControl'], table_id=['Amon'], variable_id=['zg'],source_id=['CESM2'],member_id = ['r1i1p1f1'], grid_label=['gn'])
z_kwargs = {'consolidated': True, 'use_cftime':True} dset_dict = cat.to_dataset_dict(zarr_kwargs=z_kwargs)

#selecting the 500HPa level from the dataset:

dset_dict['CMIP.NCAR.CESM2.piControl.Amon.gn'] = dset_dict['CMIP.NCAR.CESM2.piControl.Amon.gn'].squeeze() #remove empty dims
GPT_500Hpa = dset_dict['CMIP.NCAR.CESM2.piControl.Amon.gn'].zg[:,5,:,:]

GPT_500Hpa[0].plot() #this works!

#Following don't work and kill the kernel:
GPT_500Hpa.mean(dim='time').plot() 
GPT_500Hpa.mean(dim=('lon','lat')).plot()

#Not even the .value works: 
GPT_500Hpa.mean(dim=('lon','lat')).value

#It doesn't even let me save it to a new netcdf file: GPT_500Hpa.to_netcdf('/CMIP.NCAR.CESM2.piControl.Amon.gn_500Hpa.nc')

I have upgraded my matplotlib as well as pip and python but nothing worked.
I downgraded the freetype assuming that my matplotlib is behind the problem but it didn't help.

I have been struggling with this for sometime now and it will be extremely helpful if I get some guidance on this!
Thank you!

[TomNicholas]

However when I made the selection and tried to look at the DataArray (using ds.values) or plot the time-series (by using ds.values.plot()

How large is your GPT_500Hpa dataset? (You can find out via GPT_500Hpa.nbytes.) Could it just be that you are loading an array into memory that is bigger than your available RAM? When you call .values (or .plot()) on the whole array you are loading the whole thing as a single in-memory numpy array.

[shreyatrivedi26]

Thank you, Tom!
The size of GPT_500Hpa dataset is 3185049600. And after finding the area-average, the one dimensional time-series which I am trying to plot is of size 57600.
I agree! it's definitely a memory issue. Therefore, I tried to chunk the data by using the following:

GPT_500Hpa = dset_dict['CMIP.NCAR.CESM2.piControl.Amon.gn'].zg.sel(plev=50000).chunk(chunks={"time": 500})

However, my kernel is still dying and I see no difference in the data size as well even after breaking it into chunks.

Any help will be appreciated!