Skip to content

MediaProcessing.asCIImageSequence should produce an array of downsampled frames #223

Open
@davidkoski

Description

@davidkoski

MediaProcessing.asCIImageSequence should take a visitor that can apply processing to frames and downsample (requires rendering as CIImage is just a promise!)

Right now we do this:

            var videoFrameResult = await try MediaProcessing.asCIImageSequence(
                video, maxFrames: maxVideoFrames, targetFPS: targetVideoFPS)

            var processedFrames: [MLXArray] = []
            for frame in videoFrameResult.frames {
                let image =
                    frame
                    .toSRGB()
                    .resampled(
                        to: CGSize(width: fixedImageSize, height: fixedImageSize), method: .lanczos
                    )
                    .normalized(mean: config.imageMeanTuple, std: config.imageStdTuple)
                    .asMLXArray()
                processedFrames.append(image)
            }

if the video is 4k, we might have 64 very large frames that we then downsample rather than downsampling as we read.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions