Skip to content

zjunlp/OceanGPT

BranchesTags

Repository files navigation

✨ OceanGPT Beginner's Guide officially published!

We have published a detailed beginner's guide for OceanGPT, which will help you quickly grasp the model’s capabilities and begin applying this ocean-focused large model in your work.

Important

We regularly update our open-source models, so their capabilities may differ from previous versions. We warmly welcome your feedback to help us continuously improve the application of LLMs in the ocean domain.

Table of Contents

🔔News

  • 2025-06-17, we release the OceanGPT-coder-0.6B.
  • 2025-05-29, we deploy the OceanGPT MCP Server to support sonar image interpretation.
  • 2025-04-20, we release the OceanGPT-o-7B and OceanGPT-coder-7B.
  • 2025-02-01, we collect sonar data for model training and test OceanGPT-coder.
  • 2024-12-01, we collect more publicly available sonar data and scientific images for model training.
  • 2024-08-01, we launch bilingual (Chinese-English) multimodal large language model OceanGPT-o with sonar and ocean science image data collection and training.
  • 2024-07-04, we release the OceanGPT-basic-14B/2B and the updated version of OceanGPT-basic-7B (v0.2).
  • 2024-06-04, OceanGPT is accepted by ACL 2024. 🎉🎉
  • 2023-10-04, we release the paper "OceanGPT: A Large Language Model for Ocean Science Tasks" and release OceanGPT-basic-7B (v0.1) based on LLaMA2.
  • 2023-05-01, we launch the OceanGPT (沧渊) project.

Models

Model Name ModelScope HuggingFace
OceanGPT-o-7B (based on Qwen, recommended) 7B 7B
OceanGPT-coder-7B (based on Qwen, recommended) 7B 7B
OceanGPT-basic-8B (based on Qwen, recommended) 8B 8B
OceanGPT-basic-14B (based on Qwen, legacy) 14B 14B
OceanGPT-basic-7B (based on Qwen, legacy) 7B 7B
OceanGPT-basic-2B (based on MiniCPM, legacy) 2B 2B
OceanGPT-coder-0.6B (based on Qwen3) 0.6B 0.6B
  • ❗Please note that the ocean domain Q&A in the online demo system (including the video) is based on knowledge base augmentation and a "general-specialized integration" approach, and the generated content differs from that of the open-source models (注意：在线演示系统和视频里的海洋专业问答采用了知识增强与通专结合等技术，因此和开源模型存在差异)！
  • ❗Due to limited computing resources, OceanGPT-o is currently only applicable for natural language interpretation and generation of certain types of sonar images and marine science images. It is recommended to use a GPU that is greater than or equal to 24GB.

Instruction Data

Data Name HuggingFace ModelScope
OceanInstruct-v0.2 50K 50K
OceanInstruct-o 50K 50K
OceanInstruct-v0.1 10K 10K
  • ❗Some of the instruction data are synthetic data; we apologize for any inaccuracies that may exist (部分指令数据为合成数据，如存在错误敬请谅解)！

🌟Overview

This is the OceanGPT (沧渊) project, which aims to build ocean foundation model.

⏩Quickstart

conda create -n py3.11 python=3.11
conda activate py3.11
pip install -r requirements.txt

Download the model

Download from HuggingFace

# use git lfs
git lfs install
git clone https://huggingface.co/zjunlp/OceanGPT-o-7B

or

# use huggingface-cli
pip install -U huggingface_hub
huggingface-cli download --resume-download zjunlp/OceanGPT-o-7B --local-dir OceanGPT-o-7B --local-dir-use-symlinks False

Download from ModelScope

# use git lfs
git lfs install
git clone https://www.modelscope.cn/ZJUNLP/OceanGPT-o-7B.git

or

# use modelscope
pip install modelscope
modelscope download --model ZJUNLP/OceanGPT-o-7B

Inference

OceanGPT-basic-8B

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "zjunlp/OceanGPT-basic-8B"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

question = "<Your Question>"
messages = [
    {"role": "user", "content": question}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False 
)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=8192
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

try:
    index = len(output_ids) - output_ids[::-1].index(151668)  # </think> token ID
except ValueError:
    index = 0

content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")
print(content)

OceanGPT-o-7B

# It's highly recommanded to use `[decord]` feature for faster video loading.
pip install qwen-vl-utils[decord]==0.0.8
pip install transformers
from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, Qwen2VLProcessor
from qwen_vl_utils import process_vision_info
import torch

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "zjunlp/OceanGPT-o-7B", torch_dtype=torch.bfloat16, device_map="auto"
)
processor = Qwen2VLProcessor.from_pretrained("zjunlp/OceanGPT-o-7B")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "file:///path/to/your/image.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")


generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

OceanGPT-coder-7B

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "zjunlp/OceanGPT-coder-7B", torch_dtype=torch.float16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("zjunlp/OceanGPT-coder-7B")
messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
    {"role": "user", "content": "请为水下机器人生成MOOS代码，实现如下任务：先回到（50,20）点，然后以（15,20）点为圆形，做半径为30的圆周运动，持续时间200s，速度4 m/s。"}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
    **model_inputs,
    top_p=0.6,
    temperature=0.6,
    max_new_tokens=2048
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Inference by vllm

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

path = 'YOUR-MODEL-PATH'

tokenizer = AutoTokenizer.from_pretrained(path)

prompt = "Which is the largest ocean in the world?"
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

sampling_params = SamplingParams(temperature=0.8, top_k=50)
llm = LLM(model=path)

response = llm.generate(text, sampling_params)

🤗Chat with Our Demo on Gradio

Local WebUI Demo

You can easily deploy the interactive interface locally using the code we provide.

🔧 Before running, modify the model path (OceanGPT/OceanGPT-o/OceanGPT-coder's path) in app.py to your local model path.

python app.py

Open https://localhost:7860/ in browser and enjoy the interaction with OceanGPT.

Online Demo

Marine Expertise Q&A

 You can use OceanGPT-basic for marine expertise Q&A.
  1. Input your query (optional: upload an Word/PDF).
  2. Choose the generation hyparameters.
  3. Run and get results.

Marine Science Image Interpretation

 You can use OceanGPT-o for marine science image interpretation.
  1. Input your query and upload an image.
  2. Choose the generation hyparameters.
  3. Run and get results.

Marine Sonar Image Interpretation

 You can use OceanGPT-o for marine sonar image interpretation.
  1. Input your query and upload an image.
  2. Choose the generation hyparameters.
  3. Run and get results.

Underwater Robot MOOS Code Generation

 You can use OceanGPT-coder for moos code generation.
  1. Input your query.
  2. Choose the generation hyparameters.
  3. Run and get code.

Fine-tuning OceanGPT for Building Customized QA Applications

This tutorial is based on the open-source OceanGPT·Cangyuan large model, the EasyDataset open-source tool, and the Llama Factory open-source tool, covering the following key steps:

  • Model Acquisition
  • EasyDataset Data Engineering
  • Domain Fine-tuning with Llama Factory
  • Building a Web Application
  • User Usage and Effect Validation

This guide provides a practical engineering solution to help you quickly build a professional Q&A system for the maritime domain. For detailed setup instructions and usage examples, see the CustomQA_CN.md or CustomQA_EN.md.

Using MCP Server for Sonar Image Caption

The mcp_userver directory contains the Model Context Protocol (MCP) server for OceanGPT to implement some features.

For detailed setup instructions and usage examples, see the MCP server README.

📌Inference

Efficient Inference with sglang, vLLM, ollama, llama.cpp

sglang now officially supports Models based Qwen2.5-VL and Qwen2.5. Click to see.
  1. Install sglang:
pip install --upgrade pip
pip install uv
uv pip install "sglang[all]>=0.4.6.post4"
  1. Launch Server:
import requests
from openai import OpenAI
from sglang.test.test_utils import is_in_ci

if is_in_ci():
    from patch import launch_server_cmd
else:
    from sglang.utils import launch_server_cmd

from sglang.utils import wait_for_server, print_highlight, terminate_process


server_process, port = launch_server_cmd(
    "python3 -m sglang.launch_server --model-path zjunlp/OceanGPT-o-7B --host 0.0.0.0"
)

wait_for_server(f"http://localhost:{port}")
  1. Chat with Model
import requests

url = f"http://localhost:{port}/v1/chat/completions"

data = {
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
                    },
                },
            ],
        }
    ],
    "max_tokens": 300,
}

response = requests.post(url, json=data)
print_highlight(response.text)
vLLM now officially supports Models based Qwen2.5-VL and Qwen2.5. Click to see.
  1. Install vLLM(>=0.7.3):
pip install vllm
  1. Run Example:
ollama now officially supports Models based Qwen2.5. Click to see.

Create a file named Modelfile

FROM ./OceanGPT.gguf
TEMPLATE "[INST] {{ .Prompt }} [/INST]"

Create the model in Ollama:

ollama create example -f Modelfile

Running the model:

ollama run example "What is your favourite condiment?"
llama.cpp now officially supports Models based Qwen2.5-hf convert to gguf. Click to see.

Download OceanGPT PyTorch model from huggingface to "OceanGPT" folder.

Clone llama.cpp and make:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make llama-cli

And then convert PyTorch model to gguf files:

python convert-hf-to-gguf.py OceanGPT --outfile OceanGPT.gguf

Running the model:

./llama-cli -m OceanGPT.gguf \
    -co -cnv -p "Your prompt" \
    -fa -ngl 80 -n 512

🌻Acknowledgement

OceanGPT (沧渊) is trained based on the open-sourced large language models including Qwen, MiniCPM, LLaMA.

OceanGPT is trained based on the open-sourced data and tools including Moos, UATD, Forward-looking Sonar Detection Dataset, NKSID, SeabedObjects-KLSG, Marine Debris.

Thanks for their great contributions!

Limitations

  • The model may have hallucination issues.

  • Due to limited computational resources, OceanGPT-o currently only supports natural language generation for certain types of sonar images and ocean science images. OceanGPT-coder currently only supports MOOS code generation.

  • We did not optimize the identity and the model may generate identity information similar to that of Qwen/MiniCPM/LLaMA/GPT series models.

  • The model's output is influenced by prompt tokens, which may result in inconsistent results across multiple attempts.

🚩Citation

Please cite the following paper if you use OceanGPT in your work.

@article{bi2024oceangpt,
  title={OceanGPT: A Large Language Model for Ocean Science Tasks},
  author={Bi, Zhen and Zhang, Ningyu and Xue, Yida and Ou, Yixin and Ji, Daxiong and Zheng, Guozhou and Chen, Huajun},
  journal={arXiv preprint arXiv:2310.02031},
  year={2024}
}

Contributors

Ningyu Zhang, Yida Xue, Zhen Bi, Xiaozhuan Liang, Zhisong Qiu, Kewei Xu, Chenxi Wang, Shumin Deng, Xiangyuan Ru, Jintian Zhang, Shuofei Qiao, Guozhou Zheng, Huajun Chen

Community Contributors: Junjie Zheng, Zhe Ma, Shuwei Peng, Song Gao

About

[沧渊] [ACL 2024] OceanGPT: A Large Language Model for Ocean Science Tasks

OceanGPT.org/

Topics

machine-learning natural-language-processing artificial-intelligence oceanography marine ocean-science large-language-models multimodal-large-language-models oceangpt oceanbench oceaninstructions cangyuan

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

70 stars

Watchers

6 watching

Forks

9 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 12

Languages