MPSImage(3)

NAME
MPSImage -

SYNOPSIS

#import <MPSImage.h>

Inherits NSObject.

Inherited by MPSTemporaryImage.

  Instance Methods
(nonnull instancetype) - initWithDevice:imageDescriptor:
(nonnull instancetype) - initWithTexture:featureChannels:
(nonnull instancetype) - NS_UNAVAILABLE
(MPSPurgeableState) - setPurgeableState:

  Properties
id< MTLDevice > device
NSUInteger width
NSUInteger height
NSUInteger featureChannels
NSUInteger numberOfImages
MTLTextureType textureType
MTLPixelFormat pixelFormat
NSUInteger precision
MTLTextureUsage usage
size_t pixelSize
id< MTLTexture > texture
NSString * label

Detailed Description

This depends on Metal.framework A MPSImage object describes a MTLTexture that may have more than 4 channels. Some image types, such as those found in convolutional neural networks (CNN) image differs from a standard texture in that it may have more than 4 channels per image. While the channels could hold RGBA data, they will more commonly hold a number of structural permutations upon a RGBA image as the neural network progresses. It is not uncommon for each pixel to have 32 or 64 channels in it.

Since a standard MTLTexture may have no more than 4 channels, the additional channels are stored in slices of 2d texture array (i.e. texture type is MTLTextureType2DArray) such that 4 consecutive channels are stored in each slice of this array. If the number of feature channels is N, number of array slices needed is (N+3)/4. E.g. a CNN image with width 3 and height 2 with 9 channels will be stored as

slice 2         R???   R???   R???
                R???   R???   R???
slice 1      RGBA   RGBA   RGBA
             RGBA   RGBA   RGBA         (ASCII art /diagonal offset/ intended to show a Z dimension)
slice 0   RGBA   RGBA  RGBA
          RGBA   RGBA  RGBA

 Thus width and height of underlying 2d texture array is same as width and height of MPSImage and array length is equal to (featureChannels + 3) / 4. Channels marked with ? are just for padding and should not contain NaNs or Infs.

MPSImage can be container of multiple CNN images for batch processing. In order to create MPSImage that contains N images, create MPSImageDescriptror with numberOfImages set to N.

Length of 2d texture array (i.e. number of slices) will be equal to ((featureChannels +3)/4)*numberOfImage where consecutive (featureChannels+3)/4 slices of this array represent one image.

Although MPSImage can contain numberOfImages > 1, the actual number of images among these processed by MPSCNNKernel is controlled by z-dimension of clipRect. MPSCNNKernel processes n=clipRect.size.depth images from this collection. Starting index of image to process from source MPSImage is given by offset.z. Starting index of image in destination MPSImage where this processed image is written to is given by clipRect.origin.z. Thus MPSCNNKernel takes n=clipRect.size.depth image from source at indices [offset.z, offset.z+n], processed each independently and stores the result in destination at indices [clipRect.origin.z, clipRect.origin.z+n] respectively. Thus offset.z+n should be <= [src numberOfImage] and clipRect.origin.z+n should be <= [dest numberOfImages] and offset.z must be >= 0. Example: Suppose MPSCNNConvolution takes an input image with 16 channels and outputs an image with 32 channels. This number of slices needed in source 2d texture array is 2 and number of slices nneded in destination 2d array is 4. Suppose source batch size is 5 and desination batch size is 4. Thus number of source slices will be 2*5=10 and number of destination slices will be 4*4=16. If you want to process image 2 and 3 of source and store the result at index 1 and 2 in destination, you can achieve this by setting offset.z=2, clipRect.origin.z=1 and clipRect.size.depth=2. MPSCNNConvoluiton will take, in this case, slice 4 and 5 of source and produce slice 4 to 7 of destination. Similarly slice 6 and 7 will be used to produce slice 8 to 11 of destination.

All MPSCNNKernels process images in the batch independently. That is, calling a MPSCNNKernel on an batch is formally the same as calling it on each image in the batch sequentially. Computational and GPU work submission overhead will be amortized over more work if batch processing is used. This is especially important for better performance on small images.

If the number of feature channel is <= 4 and numberOfImages=1 i.e. only one slice is needed by represent MPSImage, underlying metal texture type is choosen to be MTLTextureType2D rather than MTLTextureType2DArray as explained above.

There are also MPSTemporaryImages, intended for use for very short-lived image data that is produced and consumed immediately in the same MTLCommandBuffer. They are a useful way to minimize CPU-side texture allocation costs and greatly reduce the amount of memory used by your image pipeline.

Creation of the underlying texture may in some cases occur lazily. You should in general avoid calling MPSImage.texture except when unavoidable to avoid materializing memory for longer than necessary. When possible, use the other MPSImage properties to get information about the MPSImage instead.

Method Documentation

  - (nonnull instancetype) initWithDevice: (nonnull id< MTLDevice >) device(const MPSImageDescriptor *__nonnull) imageDescriptor
Initialize an empty image object

Parameters:
    device The device that the image will be used. May not be NULL.
    imageDescriptor The MPSImageDescriptor. May not be NULL.

Returns:
    A valid MPSImage object or nil, if failure. Storage to store data needed is allocated lazily on first use of MPSImage or when application calls MPSImage.texture

Reimplemented in MPSTemporaryImage.

  - (nonnull instancetype) initWithTexture: (nonnull id< MTLTexture >) texture(NSUInteger) featureChannels
Initialize an MPSImage object using Metal texture. Metal texture has been created by user for specific number of feature channels and number of images.

Parameters:
    texture The MTLTexture allocated by the user to be used as backing for MPSImage.
    featureChannels Number of feature channels this texture contains.

Returns:
    A valid MPSImage object or nil, if failure. Application can let MPS framework allocate texture with properties specified in imageDescriptor using initWithDevice:MPSImageDescriptor API above. However in memory intensive application, you can save memory (and allocation/deallocation time) by using MPSTemporaryImage where MPS framework aggressively reuse memory underlying textures on same command buffer. See MPSTemporaryImage class for details below. However, in certain cases, application developer may want more control on allocation, placement, reusing/recycling of memory backing textures used in application using Metal Heaps API. In this case, application can create MPSImage from pre-allocated texture using initWithTexture:featureChannels.

MTLTextureType of texture can be MTLTextureType2D ONLY if featureChannels <= 4 in which case MPSImage.numberOfImages is set to 1. Else it should be MTLTextureType2DArray with arrayLength == numberOfImage * ((featureChannels + 3)/4). MPSImage.numberOfImages is set to texture.arrayLength / ((featureChannels + 3)/4).

For MTLTextures containing typical image data which application may obtain from MetalKit or other libraries such as that drawn from a JPEG or PNG, featureChannels should be set to number of valid color channel e.g. for RGB data, even thought MTLPixelFormat will be MTLPixelFormatRGBA, featureChannels should be set to 3.

Reimplemented in MPSTemporaryImage.

  - (MPSPurgeableState) setPurgeableState: (MPSPurgeableState) state
setPurgeableState Set (or query) the purgeability state of a MPSImage Usage is per [MTLResource setPurgeableState:], except that the MTLTexture might be MPSPurgeableStateAllocationDeferred, which means there is no texture to mark volatile / nonvolatile. Attempts to set purgeability on MTLTextures that have not been allocated will be ignored.

Property Documentation

  - device [read], [nonatomic], [retain]
The device on which the MPSImage will be used

  - featureChannels [read], [nonatomic], [assign]
The number of feature channels per pixel.

  - height [read], [nonatomic], [assign]
The formal height of the image in pixels.

  - label [read], [write], [atomic], [copy]
A string to help identify this object.

  - numberOfImages [read], [nonatomic], [assign]
numberOfImages for batch processing

  - pixelFormat [read], [nonatomic], [assign]
The MTLPixelFormat of the underlying texture

  - pixelSize [read], [nonatomic], [assign]
Number of bytes from the first byte of one pixel to the first byte of the next pixel in storage order. (Includes padding.)

  - precision [read], [nonatomic], [assign]
The number of bits of numeric precision available for each feature channel. This is precision, not size. That is, float is 24 bits, not 32. half precision floating-point is 11 bits, not 16. SNorm formats have one less bit of precision for the sign bit, etc. For formats like MTLPixelFormatB5G6R5Unorm it is the precision of the most precise channel, in this case 6. When this information is unavailable, typically compressed formats, 0 will be returned.

  - texture [read], [nonatomic], [assign]
The associated MTLTexture object. This is a 2D texture if numberOfImages is 1 and number of feature channels <= 4. It is a 2D texture array otherwise. To avoid the high cost of premature allocation of the underlying texture, avoid calling this property except when strictly necessary. [MPSCNNKernel encode...] calls typically cause their arguments to become allocated. Likewise, MPSImages initialized with -initWithTexture: featureChannels: have already been allocated.

  - textureType [read], [nonatomic], [assign]
The type of the underlying texture, typically MTLTextureType2D or MTLTextureType2DArray

  - usage [read], [nonatomic], [assign]
Description of texture usage.

  - width [read], [nonatomic], [assign]
The formal width of the image in pixels.

Author

Generated automatically by Doxygen for MetalPerformanceShaders.framework from the source code.