inference speed

[zcablii]

Great work!
Is it normal for an A40 GPU (45G) to take several minutes to perform inference on the single demo image?

[p-vacek]

I used two 24GB RTX 3090 and it took around 30 seconds for single demo image inference. I think you do not correctly map the model to GPUs.

[VoyagerXvoyagerx]

I used two 24GB RTX 3090 and it took around 30 seconds for single demo image inference. I think you do not correctly map the model to GPUs.

Hi, could you please share your implementation demo for multi-GPU inference? Thanks!

[p-vacek]

Hi, I can provide you with the forked pip-package (The requirements are the same as for official GeoPixel repository and are not installed by the package), we adjusted based on the official repository, where is the multi-gpu inference implemented by adjusting code with automatic mapping to device via transformers library and transferring the features in the forward pass to always remap to the same device, where required. It should be sufficient to just install the package and run the inference script.

Install the repo:

pip install git+https://github.com/SpaceKnow/GeoPixel.git

and now if you import the model and map to the device:

from GeoPixel.model.geopixel import GeoPixelForCausalLM
version = "MBZUAI/GeoPixel-7B"
tokenizer = transformers.AutoTokenizer.from_pretrained(
    version,
    padding_side="right",
    use_fast=False,
    trust_remote_code=True,
)
tokenizer.pad_token = tokenizer.unk_token
seg_token_idx, bop_token_idx, eop_token_idx = (
    tokenizer(token, add_special_tokens=False).input_ids[0]
    for token in ["[SEG]", "<p>", "</p>"]
)

kwargs = {"torch_dtype": torch.bfloat16}
geo_model_args = {
    "vision_pretrained": "facebook/sam2-hiera-large",
    "seg_token_idx": seg_token_idx,  # segmentation token index
    "bop_token_idx": bop_token_idx,  # begining of phrase token index
    "eop_token_idx": eop_token_idx,  # end of phrase token index
}

model = GeoPixelForCausalLM.from_pretrained(
    f"{version}",
    low_cpu_mem_usage=True,
    device_map="auto",
    **kwargs,
    **geo_model_args,
)

It should now run in multi-gpu setting automatically.

[VoyagerXvoyagerx]

Thanks a million! @p-vacek

[sparkhuang12]

使用一张RTX3090是否可以正常推理

[Erxucomeon]

使用一张RTX3090是否可以正常推理
应该是不行，显存不够

[Erxucomeon]

Hi, I can provide you with the forked pip-package (The requirements are the same as for official GeoPixel repository and are not installed by the package), we adjusted based on the official repository, where is the multi-gpu inference implemented by adjusting code with automatic mapping to device via transformers library and transferring the features in the forward pass to always remap to the same device, where required. It should be sufficient to just install the package and run the inference script.嗨，我可以为您提供分叉的 pip 包（要求与官方 GeoPixel 存储库相同，并且没有安装该包），我们根据官方存储库进行了调整，其中通过调整代码实现了多 GPU 推理，通过 transformers 库自动映射到设备，并在向前传递中传输功能，以便始终重新映射到相同的设备，在需要时。只需安装包并运行推理脚本就足够了。

Install the repo:  安装 repo：

pip install git+https://github.com/SpaceKnow/GeoPixel.git
and now if you import the model and map to the device:现在，如果您将模型和映射导入到设备：

from GeoPixel.model.geopixel import GeoPixelForCausalLM
version = "MBZUAI/GeoPixel-7B"
tokenizer = transformers.AutoTokenizer.from_pretrained(
version,
padding_side="right",
use_fast=False,
trust_remote_code=True,
)
tokenizer.pad_token = tokenizer.unk_token
seg_token_idx, bop_token_idx, eop_token_idx = (
tokenizer(token, add_special_tokens=False).input_ids[0]
for token in ["[SEG]", "

", "

"]
)

kwargs = {"torch_dtype": torch.bfloat16}
geo_model_args = {
"vision_pretrained": "facebook/sam2-hiera-large",
"seg_token_idx": seg_token_idx, # segmentation token index
"bop_token_idx": bop_token_idx, # begining of phrase token index
"eop_token_idx": eop_token_idx, # end of phrase token index
}

model = GeoPixelForCausalLM.from_pretrained(
f"{version}",
low_cpu_mem_usage=True,
device_map="auto",
**kwargs,
**geo_model_args,
)
It should now run in multi-gpu setting automatically.它现在应该在多 GPU 设置自动运行。

Can this method be used to train multiple graphics cards with small video memory? Thank you