Custom Keypoints detection multiple labels

[wikyburns]

I've trained a model for keypoints detection

Exported that to mlpackage with my macOs
`
model = YOLO("foam_boxes_model.pt")

Export the model to CoreML format
model.export(format="coreml")
`

Just to make sure it's working propertly I just tested out with the packageml generated:

`
image = Image.open('dual_foambox.jpg').convert('RGB')
coreml_model = YOLO("foam_boxes_model.mlpackage", task="pose")

results = coreml_model(image)
results[0].show()
`

and the result is correct:

But when It's imported to swift it just returns the boundingBox of one of the results(without the keypoints) and always appears "person", the labels are applied.

There is the result on iOS:

What I noticed it's on PoserEstimation.swift the hardcoded "person" class, ignoring the labels loaded.

Appreciate any help.

[UltralyticsAssistant]

👋 Hello @wikyburns, thank you for submitting a ultralytics/yolo-ios-app 🚀 Issue. This is an automated response to help streamline triage. An Ultralytics engineer will review and assist you here shortly 🙂

To help us address your concern efficiently, please ensure you've provided the following information:

1. For bug reports:

   • A clear and concise description of the bug (what you expected vs. what you’re seeing on iOS)
   • A minimum reproducible example MRE that demonstrates the issue (e.g., minimal Swift snippet calling the CoreML model and showing the incorrect “person” label and missing keypoints)
   • Your environment details:
     • macOS version, Xcode version, iOS version, device or simulator model
     • Ultralytics package version used to export the model
     • CoreML export command and options used
   • Model details:
     • Task and model used (pose), number of keypoints, and class names/labels
     • Confirm the labels file used during training and export
   • Any relevant logs or screenshots from Xcode or runtime
   • If applicable, the specific code paths in your app (e.g., the section of PoseEstimation.swift where class names are handled) that demonstrate the hardcoded label behavior

2. For feature requests:

   • A clear and concise description of the proposed feature
   • The problem this feature would solve
   • Any alternative solutions you've considered

3. For questions:

   • Provide as much context as possible about your question
   • Include any research you've already done on the topic
   • Specify which parts of the documentation, if any, you've already consulted

Please make sure you've searched existing issues to avoid duplicates. If you need to add any additional information, please comment on this issue. Thanks for helping us improve yolo-ios-app! 🙌

[pderrenger]

Thanks for the clear repro and screenshots—nice job verifying the .mlpackage in Python. You’re right: the sample PoseEstimation.swift currently assumes COCO pose (“person”) and a fixed 17‑keypoint layout, so custom class names and keypoints won’t show by default. To fix your case: load class names and kpt_shape from the model’s CoreML user-defined metadata, then use those to label detections and reshape/draw keypoints.

Minimal pointers:

• Read metadata in Swift: let meta = model.modelDescription.metadata[.creatorDefinedKey] as? [String:String]; let names = meta?["names"]; let kptShape = meta?["kpt_shape"]; parse names into [String] and kptShape into (K, D), then replace the hardcoded "person" with names[classIdx] and reshape the flattened keypoint slice using K and D before drawing. The exporter writes these fields into userDefinedMetadata during export. See the CoreML export reference for where metadata is set, and note that built‑in NMS is only added for Detect, not Pose. (github.com)
• Confirm your Swift postprocess isn’t limiting results (e.g., top‑K or maxDet set to 1).
• Pull the latest yolo-ios-app main and re-export with the latest ultralytics pip to ensure you have current exporter behavior and metadata fields. See the CoreML integration guide and Pose task docs for expected keypoint structure and defaults. (github.com)

If you can share a small diff (replace the hardcoded label with metadata-driven names and use kpt_shape to reshape), we’ll review and help land it so others with multi-class pose models benefit too. See the Ultralytics Docs for details on Pose and CoreML export. (github.com)

[wikyburns]

Hello Paula,

I've tried to review the result and there is what I found.

When my model has 1 class it export
Shape of [1,14,8400]

But when my model has 2 classes it exports:
📦 Prediction shape: [1, 15, 8400]
➡️ numAnchors: 8400 | featureCount: 15
all values: [376.25, 352.5, 145.5, 220.5, 0.8588867, 0.008605957, 351.75, 306.0, 0.99658203, 361.0, 444.25, 0.95166016, 416.75, 330.25, 1.0]

After printing it as you can see the result for the class it's 0.0086, but it normally has to export 2 results right? for class_1 and class_2

But there it's just 1 class, why this is happening? any advice?

`
func DebugAnchors(prediction: MLMultiArray, labels: [String]) {
let numAnchors = prediction.shape[2].intValue
let featureCount = prediction.shape[1].intValue
print("📦 Prediction shape:", prediction.shape)
print("➡️ numAnchors:", numAnchors, " | featureCount:", featureCount)

let featurePointer = UnsafeMutablePointer<Float>(OpaquePointer(prediction.dataPointer))

let sampleCount = min(8400, numAnchors)

for j in 0..<sampleCount {

    var values: [Float] = []
    for i in 0..<featureCount {
        let value = featurePointer[i * numAnchors + j]
        values.append(value)
    }

    if values.count >= 15 && values[4] > 0.8{
       print("all values: ", values)
        print(String(format: "x: %.2f, y: %.2f, w: %.2f, h: %.2f, obj_conf: %.2f",
                     values[0], values[1], values[2], values[3], values[4]))
        if labels.count > 1 {
          let classScores = zip(labels, values[5..<5+labels.count]).map { "\($0): \(String(format: "%.3f", $1))" }.joined(separator: " | ")
          print("class scores:", classScores)
        } else if labels.count == 1 {
            print(String(format: "class '%@' score: %.3f", labels[0], values[5]))
        } else {
            print(String(format: "class score: %.3f", values[5]))
        }
          var kpIndex = 5 + labels.count
          while kpIndex + 2 < values.count {
              let kx = values[kpIndex]
              let ky = values[kpIndex + 1]
              let kc = values[kpIndex + 2]
              print(String(format: "kp: (%.2f, %.2f) conf: %.2f", kx, ky, kc))
              kpIndex += 3
          }
      } else {
        // print("❌ Anchor les than 15 values", values)
      }
  }

}
`

[pderrenger]

Great investigation—the shapes you’re seeing are expected. For CoreML pose exports there’s no separate objectness channel; the per‑anchor vector is [x, y, w, h, class_scores (nc), keypoints (KD)]. That’s why featureCount = 4 + nc + KD, so 14→15 when you go from 1→2 classes. In your parser, start classes at index 4 (not 5), then keypoints at 4 + nc. Minimal fix:

let nc = labels.count
let clsStart = 4
let kptStart = clsStart + nc

let clsScores = Array(values[clsStart..<kptStart])
let (bestIdx, bestScore) = clsScores.enumerated().max(by: { $0.element < $1.element })!
let label = labels[bestIdx]  // predicted class
// iterate keypoints from kptStart in steps of 3: (x, y, conf)

This should show two class scores for your 2‑class model and align keypoints correctly. For reference on export behavior and output layout, see the Pose head forward/export notes and that Pose CoreML exports return boxes+cls+kpts without NMS: Pose head reference and kpts decode, imx pose export showing boxes/cls/kpts, CoreML exporter notes (no NMS for Pose). See the Pose head reference, Pose export path, and CoreML exporter docs for details.