Creating a dataset

[davidgodinez]

Greetings!

I have this code and if I just run it as it is, it works. But if I try to write it to a nwbfile it gives an exception.

    if tracking.Tracking & session_key:
        behav_acq = pynwb.behavior.BehavioralTimeSeries(name='BehavioralTimeSeries')
        nwbfile.add_acquisition(behav_acq)
        for table in tables:
            if table[0] & session_key:
                time_stamps,x_vals, y_vals, likelihood_vals = [], [], [], []
                for tracking_device in (tracking.TrackingDevice & (table[0] & session_key)).fetch('KEY'):
                    trials, trial_event_times, x, y, likelihood, tracking_samples = ((tracking.Tracking 
                                                                                    * table[0]
                                                                                    * experiment.TrialEvent)
                                                                                    & tracking_device
                                                                                    & session_key).fetch(
                                                                                        'trial', 'trial_event_time',f'{table[1]}_x', 
                                                                                        f'{table[1]}_y',f'{table[1]}_likelihood', 'tracking_samples') 
                    for trial in trial_event_times:
                        trial_time = float(trial)
                        tracking_sample = tracking_samples[int(trial)]
                        time_stamp = np.linspace(0,trial_time,tracking_sample)
                        time_stamps.append(time_stamp)

                    x_vals.append(x)
                    y_vals.append(y)
                    likelihood_vals.append(likelihood)
                    x_data = [d.flatten() for d in x_vals]
                    y_data = [d.flatten() for d in y_vals]
                    likelihood_data = [d.flatten() for d in likelihood_vals]

                    behav_acq.create_timeseries(name=f'BehavioralTimeSeries_{table[0]}_x_y_data', 
                                                data=np.vstack([x_data,y_data]),
                                                timestamps=np.hstack(time_stamps),
                                                description='video description',
                                                unit='a.u.', 
                                                conversion=1.0)
                    behav_acq.create_timeseries(name=f'BehavioralTimeSeries_{table[0]}_likelihood', 
                            data=likelihood_data,
                            timestamps=np.hstack(time_stamps),
                            description='video description',
                            unit='a.u.', 
                            conversion=1.0)

Exception: Could not create dataset data in /acquisition/BehavioralTimeSeries/BehavioralTimeSeries_<class 'pipeline.tracking.Tracking.NoseTracking'>_x_y_data with shape (2, 1904), dtype <class 'numpy.object_'>, and iosettings {}. Object dtype dtype('O') has no native HDF5 equivalent

[bendichter]

The problem is with np.vstack([x_data, y_data]). numpy uses the "object" type when it is unable to create a standard numeric matrix. Maybe x_data and y_data use the Decimal data type? Can you print x_data, y_data, and np.vstack([x_data, y_data])?

[davidgodinez]

I checked up on it and x_data, y_data, and likelihood data use the numpy.float64.

[bendichter]

Can you copy paste the output of:

print(x_data)

x_data.shape

print(y_data)

y_data.shape

[davidgodinez]

[davidgodinez]

[bendichter]

Why are your x_data and y_data lists of arrays? I would expect them to both be just arrays

On Tue, Jan 11, 2022 at 9:39 AM davidgodinez ***@***.***> wrote: [image: Screen Shot 2022-01-11 at 8 38 37 AM] <https://user-images.githubusercontent.com/43123708/148963124-62236ef8-4846-460f-9689-441fbb35ddf5.png> — Reply to this email directly, view it on GitHub <#19 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGOEEUKIHWX5LQ6BHOVAP3UVQ6IXANCNFSM5LU5UHCQ> . You are receiving this because you commented.Message ID: <NeurodataWithoutBorders/helpdesk/repo-discussions/19/comments/1948194@ github.com>

-- Ben Dichter, PhD Data Science Consultant personal website <http://bendichter.com/>

[davidgodinez]

I figured It would help to flatten the data. But what I can do is just to the np.vstack() on x_vals[0] and y_vals[0] since those are arrays. x_data is a list and then x_vals is a list but if I index into x and y vals then I get the array of arrays.

[davidgodinez]

never mind I tried that but it got the same error.

[bendichter]

We met and determined that x_data and y_data were a list of data per trial. The solution was to concatenate this data first.

[davidgodinez]

Thank you for your help!