How to open RAW folder via python script?

[khoroshyy]

Hi.
I cannot figure out how to open a RAW folder via a python script.
Could you give me a hint?
Thanks.

[michaelmarty]

Hey!
Happy to help. Here's some example code:

`
path = "D:\Data\strep.RAW"

import unidec_modules.data_reader as dr

im = dr.DataImporter(path)

spec1 = im.get_data(scan_range=[0, 0])
`

The data reader class will detect the file type and import it. It will import the ThermoDataImporter class from unidec_modules.thermo_reader.ThermoImporter.py. You can look at this file to see the various functions that can be called from the im variable above. Let me know how that goes!
Thanks,
MTM

[khoroshyy]

Thank you. I will try.
spec1 in the example above I guess will be an array of the mass spectrum from a range of scans.

[michaelmarty]

That's correct! You can also use time_range=[0,1] instead of scan_range to get a specific time window. It will average everything within that range into a single spectrum. Should be n x 2 dimension with the first column as the m/z and second as intensity.

[khoroshyy]

I am getting some errors:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Registering...
Checking for access to MS data formats...
Failed Interface Registration: registerInterfaces() must be run with administrator priviledges!
NOTE: TRY RUNNING AS ADMINISTRATOR
Traceback (most recent call last):
File "E:\UniDec-v.5.0.1\unidec_modules\data_reader.py", line 48, in init
self.msrun = mzFile(path)
File "C:\Python38\lib\site-packages\multiplierz\mzAPI_init_.py", line 120, in init
raw.mzFile.init(self, data_file, **kwargs)
File "C:\Python38\lib\site-packages\multiplierz\mzAPI\raw.py", line 41, in init
if self.source.InstrumentCount > 1:
System.Exception: No open raw file.
at ThermoFisher.CommonCore.RawFileReader.RawFileAccessBase.ValidateOpenFile()
at ThermoFisher.CommonCore.RawFileReader.RawFileAccessBase.get_InstrumentCount()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "xzml_processing.py", line 165, in
batch_process_raw_folders(abs_dirname, t_range)
File "xzml_processing.py", line 114, in batch_process_raw_folders
raw_file = dr.DataImporter(i)
File "E:\UniDec-v.5.0.1\unidec_modules\data_reader.py", line 51, in init
self.msrun = mzFile(path)
File "C:\Python38\lib\site-packages\multiplierz\mzAPI_init_.py", line 120, in init
raw.mzFile.init(self, data_file, **kwargs)
File "C:\Python38\lib\site-packages\multiplierz\mzAPI\raw.py", line 41, in init

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

And if I run as Administrator I get

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Microsoft .NET Framework Assembly Registration Utility version 4.8.3761.0
for Microsoft .NET Framework version 4.8.3761.0
Copyright (C) Microsoft Corporation. All rights reserved.

RegAsm : warning RA0000 : Registering an unsigned assembly with /codebase can cause your assembly to interfere with other applications that may be installed on the same computer. The /codebase switch is intended to be used only with signed assemblies. Please give your assembly a strong name and re-register it.
Types registered successfully
RegAsm : error RA0000 : An error occurred while saving the exported type library: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Registration operations completed.
Results:

File Type Registered Before RegAsm Return Code Registered Now
T2D False 0 False
D True 100 True
WIFF False 0 False
RAW True 100 True

Traceback (most recent call last):
File "E:\UniDec-v.5.0.1\unidec_modules\data_reader.py", line 48, in init
self.msrun = mzFile(path)
File "C:\Python38\lib\site-packages\multiplierz\mzAPI_init_.py", line 120, in init
raw.mzFile.init(self, data_file, **kwargs)
File "C:\Python38\lib\site-packages\multiplierz\mzAPI\raw.py", line 41, in init
if self.source.InstrumentCount > 1:
System.Exception: No open raw file.
at ThermoFisher.CommonCore.RawFileReader.RawFileAccessBase.ValidateOpenFile()
at ThermoFisher.CommonCore.RawFileReader.RawFileAccessBase.get_InstrumentCount()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "xzml_processing.py", line 165, in
batch_process_raw_folders(abs_dirname, t_range)
File "xzml_processing.py", line 114, in batch_process_raw_folders
raw_file = dr.DataImporter(i)
File "E:\UniDec-v.5.0.1\unidec_modules\data_reader.py", line 51, in init
self.msrun = mzFile(path)
File "C:\Python38\lib\site-packages\multiplierz\mzAPI_init_.py", line 120, in init
raw.mzFile.init(self, data_file, **kwargs)
File "C:\Python38\lib\site-packages\multiplierz\mzAPI\raw.py", line 41, in init
if self.source.InstrumentCount > 1:
System.Exception: No open raw file.
at ThermoFisher.CommonCore.RawFileReader.RawFileAccessBase.ValidateOpenFile()
at ThermoFisher.CommonCore.RawFileReader.RawFileAccessBase.get_InstrumentCount()

E:\UniDec_Windows\Example Data\test\Unidec_script_test>

[michaelmarty]

Looks like an issue with MultiplierZ. We switched libraries a while ago, so I haven't tested that one recently. It's only used for Agilent files I think. Anyway, I think we can use the other library. Try switching the code to:

import unidec_modules.thermo_reader.ThermoImporter as dr

im = dr.ThermoDataImporter(path)

spec1 = im.get_data(scan_range=[0, 0])

If you want something more generic, try:

import unidec_modules.unidectools as ucd

im = ucd.get_importer(path)

The latter will automatically grab the right data importer. Let me know how it goes!

[AlexanderMoysa]

Hi Michael I tried to process a RAW file, but I got an error. Could you check?
here is my output:

UniDec Engine v.7.0.3
UniDec Path: C:\Users\Alexander.Moysa\AppData\Local\Programs\Python\Python312\Lib\site-packages\unidec\bin\unidec.exe
Launching Thermo Importer. If it fails after this step, try this:
Delete your whole UniDec folder but keep the zip file.
Right click on the zip file and open properties. You should see a box to Unblock it. Check that.
Click ok. Unzip it again. Try it once more.
2024-12-13 12:35:59,761 - INFO - Configuration loaded from C:\Users\Alexander.Moysa\PycharmProjects\UniDec\data\conf.dat
2024-12-13 12:35:59,761 - INFO - Processing RAW file: W522_DMSO_16.raw
Reading Thermo Data: C:\Users\Alexander.Moysa\PycharmProjects\UniDec\data\W522_DMSO_16.raw
Number of Scans 182
Thermo Scan Range: [1 1]
2024-12-13 12:36:00,561 - ERROR - Error processing RAW file W522_DMSO_16.raw: too many indices for array: array is 1-dimensional, but 2 were indexed

I checked spec1 data, and it looks OK
spec1 shape: (1669, 2)
spec1 data (first 10 values): [[594.03008307 0. ]
[594.03694982 0. ]
[594.0438167 0. ]
[594.05068369 0. ]
[610.12263584 0. ]
[610.12978351 0. ]
[610.1369313 0. ]
[610.14407923 0. ]
[610.15122728 8.74436951]
[610.15837545 71.04295349]]

import os
import logging
import unidec
import unidec.tools as ucd

# Set up logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")

# Define the folder path and configuration file path
folder = r"C:\Users\Alexander.Moysa\PycharmProjects\UniDec\data"
config_file = os.path.join(folder, "conf.dat")

# Initialize the UniDec engine
eng = unidec.UniDec()

# Load configuration from conf.dat
try:
    eng.load_config(config_file)
    logging.info(f"Configuration loaded from {config_file}")
except Exception as e:
    logging.error(f"Failed to load configuration from {config_file}: {e}")
    raise

# Process each RAW file in the folder
for file_name in os.listdir(folder):
    file_path = os.path.join(folder, file_name)  # Get full path to the file

    # Check if the file is a RAW file
    if file_name.lower().endswith(".raw"):
        logging.info(f"Processing RAW file: {file_name}")

        try:
            # Import RAW data 
            im = ucd.get_importer(file_path)
            spec1 = im.get_data(scan_range=[0, 0])

            # Process RAW data with UniDec
            eng.process_data()
            eng.run_unidec(silent=True)
            eng.pick_peaks()

            logging.info(f"Successfully processed RAW file: {file_name}")

        except Exception as e:
            logging.error(f"Error processing RAW file {file_name}: {e}")

[AlexanderMoysa]

Finally, I overcome this issue by storing data in the dictionary

import os
import logging
import unidec
import unidec.tools as ucd
import numpy as np

# Set up logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")

# Define the folder path and configuration file path
folder = r"C:\Users\Alexander.Moysa\PycharmProjects\UniDec\data"
config_file = os.path.join(folder, "conf.dat")

# Initialize the UniDec engine
eng = unidec.UniDec()

# Load configuration from conf.dat
try:
    eng.load_config(config_file)
    logging.info(f"Configuration loaded from {config_file}")
except Exception as e:
    logging.error(f"Failed to load configuration from {config_file}: {e}")
    raise

# Process each RAW file in the folder
for file_name in os.listdir(folder):
    file_path = os.path.join(folder, file_name)  # Get full path to the file

    # Check if the file is a RAW file
    if file_name.lower().endswith(".raw"):
        logging.info(f"Processing RAW file: {file_name}")

        try:
            # Import RAW data using ThermoDataImporter
            importer = ucd.get_importer(file_path)
            spec1 = importer.get_data(scan_range=[40, 80])  # Adjust scan range as needed

            # Check if spec1 is a valid NumPy array
            if isinstance(spec1, np.ndarray):
                logging.info(f"Loaded RAW data with shape: {spec1.shape}")

                # Save spec1 to an in-memory dictionary
                stored_data = {
                    "header": ["m/z", "Intensity"],
                    "data": spec1
                }

                # Pass the data into UniDec engine
                eng.pass_data_in(spec1)  # Feed the NumPy array to UniDec
                eng.process_data()       # Process the data
                eng.run_unidec(silent=True)  # Run UniDec analysis
                peaks = eng.pick_peaks()  # Pick peaks
                logging.info(f"Successfully processed data for file: {file_name}")

                # Save peaks to a text file
                output_file = os.path.join(folder, f"{os.path.splitext(file_name)[0]}_peaks.txt")
                try:
                    np.savetxt(output_file, peaks, fmt="%.6f", header="m/z\tIntensity", delimiter="\t")
                    logging.info(f"Peak data successfully saved to: {output_file}")
                except Exception as e:
                    logging.error(f"Error saving peak data to file {output_file}: {e}")

            else:
                logging.warning(f"spec1 is not a valid NumPy array for file: {file_name}")

        except Exception as e:
            logging.error(f"Error processing RAW file {file_name}: {e}")

[michaelmarty]

Hi Alex,
You are doing great with the API! The pass_data_in function will work if you want to open in yourself. You can also use the eng.open_file(file_path) to bypass that. It should open the raw file for you and handle the rest. I'd probably recommend the open_file for most applications, but it's up to you.

One thing to note, we are about to release version 8, which has major changes to the importer. We restructured the whole thing to make it more consistent across file types, faster, and cleaner. For example, some data formats (Waters and Agilent) are indexed such that the first scan is scan 0. Thermo and mzML are indexed with the first scan as 1. In our new importer, we are using the Thermo/mzML model and doing 1 indexing on scans. The function names should also be much easier to understand now. You can preview some of the changes in the v3 branch. The importer will be located with: from unidec.UniDecImporter import get_importer and get_data will change to get_avg_scan. However, open_file should still handle all of that for you. Let me know if you have questions on that. Thanks!
MTM

[AlexanderMoysa]

Hi Michale,
Thank you very much for your support!

Best,
Alex

[KoryoKwao]

Hello Dr. Marty,
I am trying to import Agilent .d files for subsequent analyses using the API.

Here's my code, based on previous suggestions in this thread:

from unidec import engine
import unidec_modules.data_reader as dr

eng=engine.UniDec()
path=r"C:\007-090_GA95_AFPS222_POSTLYO_60MIN_.d"
im = dr.DataImporter(path)
spec = im.get_data(scan_range=[0, 0])

eng.process_data(spec)
eng.run_unidec(silent=True)
eng.pick_peaks()

Error:
ModuleNotFoundError Traceback (most recent call last)
Cell In[78], line 2
1 from unidec import engine
----> 2 import unidec_modules.data_reader as dr
4 eng=engine.UniDec()
5 path=r"C:\007-090_GA95_AFPS222_POSTLYO_60MIN_.d"

ModuleNotFoundError: No module named 'unidec_modules'

Can you help me locate this module if it has been moved? I'm using version 8.0.0. Thank you!

[michaelmarty]

Yep, it's moved. We refactored all of our file opening API into UniDecImporter. Editing the relevant lines to something like this should work:

from UniDecImporter.ImporterFactory import ImporterFactory

and

im= ImporterFactory.create_importer(path)