UE ReplayKit for iOS插件使用

iOSReplayKit插件使用

iOSReplayKit用途
开始使用
修改代码
增加MIC录制
完整代码
Objective-C和C++混合编码

iOSReplayKit用途

ReplayKit是苹果为iOS/tvOS/macOS平台视频直播和视频录制提供的工具包（Record or stream video from the screen, and audio from the app and microphone.），整个插件使用很简单（因为大部分功能都被苹果限制死了），调用StartXxxxWithHandler开始录制，提供回调函数接收调用结果和数据包；StopXxxxWithHandler结束录制，提供回调函数接收调用结果。

iOSReplayKit（ReplayKit for iOS）插件是提供给UE使用iOS平台ReplayKit软件包的桥接器。插件包含一个蓝图类UIOSReplayKitControl，提供调用iOS ReplayKit API函数的蓝图接口；一个Objective-C语言写的ReplayKitRecorder实现类。

开始使用

因为要在Unreal开发的iOS平台APP使用视频录制功能，查找资料发现苹果提供的ReplayKit可以支持视频和麦克风录制。进一步查找Unreal平台插件，在github上找到一个replaykit for ios的项目
PushkinStudio/PsReplayKit
但这个项目最后更新已经是2018年了。
后来查看Unreal引擎源码，发现Unreal引擎里面已经集成ReplayKit插件。
直接在项目引入插件，重启
插件使用很简单，一共提供了四个函数

Unreal本着已经提供了源代码没必要再提供文档的原则，查看帮助文档，得到的信息也不会比函数名更多了。

至于StartRecording和StartCaptureToFile有什么区别，查看源代码，发现确实是Apple的锅，Apple ReplayKit提供startRecordingWithHandler和startCaptureWithHandler两个方法，startCaptureWithHandler提供了回调函数，可以自己处理录制过程；startRecordingWithHandler不提供过程回调，仅在stop时提供预览（调用系统预览框），全程开发者无法干预。

创建两个按钮，点击绑定调用StartCaptureToFile和StopCapture，打包输出，在真机上运行点击Start录制，成功唤起权限确认对话框。

点击Stop结束，显示存储到相册权限提示，看起来一切正常。
在测试几次，结果发现问题了，除了一开始成功的一次，后面基本上都是失败的，并没有任何文件存储到相册。
然后换了StartRecording和StopRecording接口，一样的问题，偶尔成功，大部分失败。
查看源代码，发现StartCaptureToFile调用startCapture，该函数实现代码是自己将回调数据包写入mp4文件，其中有处理文件的代码

  // create the asset writer[_assetWriter release] ;// todo: do we care about the file name? support cleaning up old captures?auto fileName = FString::Printf(TEXT("%s.mp4"), *FGuid::NewGuid().ToString());_captureFilePath = [captureDir stringByAppendingFormat : @"/%@", fileName.GetNSString()];[_captureFilePath retain] ;_assetWriter = [[AVAssetWriter alloc]initWithURL: [NSURL fileURLWithPath : _captureFilePath] fileType : AVFileTypeMPEG4 error : nil];// _assetWriter = [AVAssetWriter assetWriterWithURL : [NSURL fileURLWithPath : _captureFilePath] fileType : AVFileTypeMPEG4 error : nil];[_assetWriter retain] ;

通过USB连接iPAD，在文件浏览里面发现Capture目录下大部分录制的mp4都是0字节文件，说明在写入环节出现问题了

查看写入代码，增加Unreal屏幕打印调式信息功能

        if (_assetWriter.status == AVAssetWriterStatusUnknown){[_assetWriter startWriting] ;[_assetWriter startSessionAtSourceTime : CMSampleBufferGetPresentationTimeStamp(sampleBuffer)] ;if (GEngine) {CMTime time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Blue, FString::Printf(TEXT("_assetWriter startWriting: %d"), (int)CMTimeGetSeconds(time)));}}if (_assetWriter.status == AVAssetWriterStatusFailed){NSLog(@"%d: %@", (int)_assetWriter.error.code, _assetWriter.error.localizedDescription);if (GEngine) {const char* dest = [_assetWriter.error.localizedDescription UTF8String];GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Green, FString::Printf(TEXT("_assetWriter failed: %d %s"), (int)_assetWriter.error.code, UTF8_TO_TCHAR(dest)));}}

运行后屏幕打印的调试信息表明回调函数没有问题，问题出在AVAssetWriter状态是AVAssetWriterStatusFailed，错误代码为-11823，错误描述cannot save。
经历漫长编译-输出-安装-调试，发现蓝色提示信息_assetWriter startWriting 16600

GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Blue, FString::Printf(TEXT("_assetWriter startWriting: %d"), (int)CMTimeGetSeconds(time)));

夹杂在绿色-11823，cannot save错误信息之间显示了两条。蓝色提示信息是在调用_assetWriter.startWriting时显示的，显然调用了两次startWriting导致AV视频写入器报错了。

        if (_assetWriter.status == AVAssetWriterStatusUnknown){[_assetWriter startWriting] ;// ...}

代码在调用startWriting时判断了写入器状态是不是无状态，调用startWriting以后写入器的状态应该会发生变化，线性调用是不会导致调用两次的，除非是多线程同时调用。连续调用了两次startWriting说明在状态改变之前又有新的回调函数调用了，猜想视频数据包和音频数据包处理是在不同线程进行的，几乎同时调用了回调函数，而回调函数并没有采用任何的线程锁机制，导致startWriting被调用了两次，从而引起AVAssetWriterStatusFailed错误，而偶尔成功的几次，应该是多线程没有同时回调，侥幸成功。

修改代码

增加一个frames自增变量，仅在0==frames时调用startWriting函数

        if (0 == _frames++) {[_assetWriter startWriting] ;[_assetWriter startSessionAtSourceTime : CMSampleBufferGetPresentationTimeStamp(sampleBuffer)] ;if (GEngine) {CMTime time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Blue, FString::Printf(TEXT("_assetWriter startWriting: %d"), (int)CMTimeGetSeconds(time)));}}

这里没有用线程锁，用了++自增操作，++操作仅一个CPU指令周期，几乎不可能被中断。

增加MIC录制

          AVAssetWriterInput* input = nullptr;// ...else if (bufferType == RPSampleBufferTypeAudioMic){// todo?}// ...

bufferType == RPSampleBufferTypeAudioMic表示这是一个Microphone采集回调数据，但代码中仅有一个todo?注释，并没有任何代码实现，也就是直接使用插件是不能录制麦克风声音的。（直到5.1版本的iOSReplayKit都没有实现Mic采集）
仿照RPSampleBufferTypeAudioApp内录音频，创建一个Mic录制写入器

  // create the audio input[_audioInput release] ;AudioChannelLayout acl;bzero(&acl, sizeof(acl));acl.mChannelLayoutTag = kAudioChannelLayoutTag_Stereo;auto audioSettings = @{AVFormatIDKey: @(kAudioFormatMPEG4AAC),AVSampleRateKey: @(44100),AVChannelLayoutKey: [NSData dataWithBytes : &acl length : sizeof(acl)] ,};_audioInput = [AVAssetWriterInput assetWriterInputWithMediaType : AVMediaTypeAudio outputSettings : audioSettings];_audioInput.expectsMediaDataInRealTime = YES;[_audioInput retain] ;// create the microphone input[_microInput release] ;AudioChannelLayout mic;bzero(&mic, sizeof(mic));mic.mChannelLayoutTag = kAudioChannelLayoutTag_Mono;auto micSettings = @{AVFormatIDKey: @(kAudioFormatMPEG4AAC),AVSampleRateKey: @(44100),AVChannelLayoutKey: [NSData dataWithBytes : &mic length : sizeof(mic)] ,};_microInput = [AVAssetWriterInput assetWriterInputWithMediaType : AVMediaTypeAudio outputSettings : micSettings];_microInput.expectsMediaDataInRealTime = YES;[_microInput retain] ;// add the input to the writer// ...if ([_assetWriter canAddInput : _audioInput]){[_assetWriter addInput : _audioInput] ;}if ([_assetWriter canAddInput : _microInput]){[_assetWriter addInput : _microInput] ;}

todo?注释增加mic input输入处理

          AVAssetWriterInput* input = nullptr;if (bufferType == RPSampleBufferTypeVideo){input = _videoInput;}else if (bufferType == RPSampleBufferTypeAudioApp){input = _audioInput;}else if (bufferType == RPSampleBufferTypeAudioMic){// todo?input = _microInput;}if (input && input.isReadyForMoreMediaData){[input appendSampleBuffer : sampleBuffer] ;}

增加上面代码后，启动startCapture时勾选麦克风，就可以录制麦克风声音了（不知道官方插件为啥不实现这个简单的功能）

完整代码

// Copyright Epic Games, Inc. All Rights Reserved.#include "ReplayKitRecorder.h"#if PLATFORM_IOS
#include "IOSAppDelegate.h"
#include "IOS/IOSView.h"
#include "Misc/Paths.h"
#include "Engine/Engine.h"
#include "Engine/GameViewportClient.h"@implementation ReplayKitRecorderRPScreenRecorder* _Nullable _screenRecorder;
RPBroadcastActivityViewController* _Nullable _broadcastActivityController;
RPBroadcastController* _Nullable _broadcastController;// stuff used when capturing to file
AVAssetWriter* _Nullable _assetWriter;
AVAssetWriterInput* _Nullable _videoInput;
AVAssetWriterInput* _Nullable _audioInput;
AVAssetWriterInput* _Nullable _microInput;
NSString* _Nullable _captureFilePath;
int _frames;- (void) initializeWithMicrophoneEnabled:(BOOL)bMicrophoneEnabled withCameraEnabled:(BOOL)bCameraEnabled {_screenRecorder = [RPScreenRecorder sharedRecorder];[_screenRecorder setDelegate:self];
#if !PLATFORM_TVOS[_screenRecorder setMicrophoneEnabled:bMicrophoneEnabled];[_screenRecorder setCameraEnabled:bCameraEnabled];
#endif
}- (void)startRecording {// NOTE(omar): stop any live broadcasts before staring a local recordingif( _broadcastController != nil ) {[self stopBroadcast];}if( [_screenRecorder isAvailable] ) {[_screenRecorder startRecordingWithHandler:^(NSError * _Nullable error) {if( error ) {NSLog( @"error starting screen recording");}}];}
}- (void)stopRecording {if( [_screenRecorder isAvailable] && [_screenRecorder isRecording] ) {[_screenRecorder stopRecordingWithHandler:^(RPPreviewViewController * _Nullable previewViewController, NSError * _Nullable error) {[previewViewController setPreviewControllerDelegate:self];// automatically show the video preview when recording is stoppedpreviewViewController.popoverPresentationController.sourceView = (UIView* _Nullable)[IOSAppDelegate GetDelegate].IOSView;[[IOSAppDelegate GetDelegate].IOSController presentViewController:previewViewController animated:YES completion:nil];}];}
}- (void)createCaptureContext
{auto fileManager = [NSFileManager defaultManager];auto docDir = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) objectAtIndex:0];auto captureDir = [docDir stringByAppendingFormat : @"/Captures"];[fileManager createDirectoryAtPath : captureDir withIntermediateDirectories : YES attributes : nil error : nil] ;[_captureFilePath release] ;// create the asset writer[_assetWriter release] ;// todo: do we care about the file name? support cleaning up old captures?auto fileName = FString::Printf(TEXT("%s.mp4"), *FGuid::NewGuid().ToString());_captureFilePath = [captureDir stringByAppendingFormat : @"/%@", fileName.GetNSString()];[_captureFilePath retain] ;_assetWriter = [[AVAssetWriter alloc]initWithURL: [NSURL fileURLWithPath : _captureFilePath] fileType : AVFileTypeMPEG4 error : nil];// _assetWriter = [AVAssetWriter assetWriterWithURL : [NSURL fileURLWithPath : _captureFilePath] fileType : AVFileTypeMPEG4 error : nil];[_assetWriter retain] ;// create the video input[_videoInput release] ;auto view = [IOSAppDelegate GetDelegate].IOSView;auto width = [NSNumber numberWithFloat : view.frame.size.width];auto height = [NSNumber numberWithFloat : view.frame.size.height];if (GEngine && GEngine->GameViewport && GEngine->GameViewport->Viewport){auto viewportSize = GEngine->GameViewport->Viewport->GetSizeXY();width = [NSNumber numberWithInt : viewportSize.X];height = [NSNumber numberWithInt : viewportSize.Y];}auto videoSettings = @{AVVideoCodecKey: AVVideoCodecTypeH264,AVVideoWidthKey : width,AVVideoHeightKey : height};_videoInput = [AVAssetWriterInput assetWriterInputWithMediaType : AVMediaTypeVideo outputSettings : videoSettings];_videoInput.expectsMediaDataInRealTime = YES;[_videoInput retain] ;// create the audio input[_audioInput release] ;AudioChannelLayout acl;bzero(&acl, sizeof(acl));acl.mChannelLayoutTag = kAudioChannelLayoutTag_Stereo;auto audioSettings = @{AVFormatIDKey: @(kAudioFormatMPEG4AAC),AVSampleRateKey: @(44100),AVChannelLayoutKey: [NSData dataWithBytes : &acl length : sizeof(acl)] ,};_audioInput = [AVAssetWriterInput assetWriterInputWithMediaType : AVMediaTypeAudio outputSettings : audioSettings];_audioInput.expectsMediaDataInRealTime = YES;[_audioInput retain] ;// create the microphone input[_microInput release] ;AudioChannelLayout mic;bzero(&mic, sizeof(mic));mic.mChannelLayoutTag = kAudioChannelLayoutTag_Mono;auto micSettings = @{AVFormatIDKey: @(kAudioFormatMPEG4AAC),AVSampleRateKey: @(44100),AVChannelLayoutKey: [NSData dataWithBytes : &mic length : sizeof(mic)] ,};_microInput = [AVAssetWriterInput assetWriterInputWithMediaType : AVMediaTypeAudio outputSettings : micSettings];_microInput.expectsMediaDataInRealTime = YES;[_microInput retain] ;// add the input to the writerif ([_assetWriter canAddInput : _videoInput]){[_assetWriter addInput : _videoInput] ;}if ([_assetWriter canAddInput : _audioInput]){[_assetWriter addInput : _audioInput] ;}if ([_assetWriter canAddInput : _microInput]){[_assetWriter addInput : _microInput] ;}
}- (void)startCapture
{if (_broadcastController){[self stopBroadcast] ;}if ([_screenRecorder isAvailable]){_frames = 0;[self createCaptureContext] ;[_screenRecorder startCaptureWithHandler : ^ (CMSampleBufferRef sampleBuffer, RPSampleBufferType bufferType, NSError * error){if (CMSampleBufferDataIsReady(sampleBuffer)){if (0 == _frames++) {[_assetWriter startWriting] ;[_assetWriter startSessionAtSourceTime : CMSampleBufferGetPresentationTimeStamp(sampleBuffer)] ;if (GEngine) {CMTime time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Blue, FString::Printf(TEXT("_assetWriter startWriting: %d"), (int)CMTimeGetSeconds(time)));}}/*if (_assetWriter.status == AVAssetWriterStatusUnknown){[_assetWriter startWriting] ;[_assetWriter startSessionAtSourceTime : CMSampleBufferGetPresentationTimeStamp(sampleBuffer)] ;if (GEngine) {CMTime time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Blue, FString::Printf(TEXT("_assetWriter startWriting: %d"), (int)CMTimeGetSeconds(time)));}}*/if (_assetWriter.status == AVAssetWriterStatusFailed){// NSLog(@"%d: %@", (int)_assetWriter.error.code, _assetWriter.error.localizedDescription);if (GEngine) {const char* dest = [_assetWriter.error.localizedDescription UTF8String];GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Green, FString::Printf(TEXT("_assetWriter failed: %d %s"), (int)_assetWriter.error.code, UTF8_TO_TCHAR(dest)));}}else {AVAssetWriterInput* input = nullptr;if (bufferType == RPSampleBufferTypeVideo){input = _videoInput;}else if (bufferType == RPSampleBufferTypeAudioApp){input = _audioInput;}else if (bufferType == RPSampleBufferTypeAudioMic){// todo?input = _microInput;}if (input && input.isReadyForMoreMediaData){[input appendSampleBuffer : sampleBuffer] ;}}}}completionHandler: ^ (NSError * error){if (error){NSLog(@"completionHandler: %@", error);if (GEngine) {GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Red, FString::Printf(TEXT("error at startCapture")));}}}];}
}- (void)stopCapture
{if ([_screenRecorder isAvailable]){[_screenRecorder stopCaptureWithHandler:^(NSError *error){if (error){NSLog(@"stopCaptureWithHandler: %@", error);}if (_assetWriter){[_assetWriter finishWritingWithCompletionHandler:^(){NSLog(@"finishWritingWithCompletionHandler");#if !PLATFORM_TVOSif (UIVideoAtPathIsCompatibleWithSavedPhotosAlbum(_captureFilePath)){UISaveVideoAtPathToSavedPhotosAlbum(_captureFilePath, nil, nil, nil);NSLog(@"capture saved to album");}
#endif[_captureFilePath release];_captureFilePath = nullptr;[_videoInput release];_videoInput = nullptr;[_audioInput release];_audioInput = nullptr;[_microInput release];_microInput = nullptr;[_assetWriter release];_assetWriter = nullptr;}];}}];}
}//
// livestreaming functionality
//- (void)startBroadcast {// NOTE(omar): ending any local recordings that might be active before starting a broadcastif( [_screenRecorder isRecording] ) {[self stopRecording];}[RPBroadcastActivityViewController loadBroadcastActivityViewControllerWithHandler:^(RPBroadcastActivityViewController * _Nullable broadcastActivityViewController, NSError * _Nullable error) {_broadcastActivityController = broadcastActivityViewController;[_broadcastActivityController setDelegate:self];[[IOSAppDelegate GetDelegate].IOSController presentViewController:_broadcastActivityController animated:YES completion:nil];}];
}- (void)pauseBroadcast {if( [_broadcastController isBroadcasting] ) {[_broadcastController pauseBroadcast];}
}- (void)resumeBroadcast {if( [_broadcastController isPaused] ) {[_broadcastController resumeBroadcast];}
}- (void)stopBroadcast {if( [_broadcastController isBroadcasting ] ) {[_broadcastController finishBroadcastWithHandler:^(NSError * _Nullable error) {if( error ) {NSLog( @"error finishing broadcast" );}[_broadcastController release];_broadcastController = nil;}];}
}//
// delegates
//// screen recorder delegate
- (void)screenRecorder:(RPScreenRecorder* _Nullable)screenRecorder didStopRecordingWithError:(NSError* _Nullable)error previewViewController:(RPPreviewViewController* _Nullable)previewViewController {NSLog(@"RTRScreenRecorderDelegate::didStopRecrodingWithError");[previewViewController dismissViewControllerAnimated:YES completion:nil];
}- (void)screenRecorderDidChangeAvailability:(RPScreenRecorder* _Nullable)screenRecorder {NSLog(@"RTRScreenRecorderDelegate::screenRecorderDidChangeAvailability");
}// screen recorder preview view controller delegate
- (void)previewControllerDidFinish:(RPPreviewViewController* _Nullable)previewController {NSLog( @"RTRPreviewViewControllerDelegate::previewControllerDidFinish" );[previewController dismissViewControllerAnimated:YES completion:nil];
}- (void)previewController:(RPPreviewViewController* _Nullable)previewController didFinishWithActivityTypes:(NSSet <NSString*> * _Nullable)activityTypes __TVOS_PROHIBITED {NSLog( @"RTRPreviewViewControllerDelegate::didFinishWithActivityTypes" );[previewController dismissViewControllerAnimated:YES completion:nil];
}// broadcast activity view controller delegate
- (void)broadcastActivityViewController:(RPBroadcastActivityViewController* _Nullable)broadcastActivityViewController didFinishWithBroadcastController:(RPBroadcastController* _Nullable)broadcastController error:(NSError* _Nullable)error {NSLog( @"RPBroadcastActivityViewControllerDelegate::didFinishWithBroadcastController" );[broadcastActivityViewController dismissViewControllerAnimated:YES completion:^{_broadcastController = [broadcastController retain];[_broadcastController setDelegate:self];[_broadcastController startBroadcastWithHandler:^(NSError* _Nullable _error) {if( _error ) {NSLog( @"error starting broadcast" );}}];}];
}// broadcast controller delegate
- (void)broadcastController:(RPBroadcastController* _Nullable)broadcastController didFinishWithError:(NSError* _Nullable)error {NSLog( @"RPBroadcastControllerDelegate::didFinishWithError" );
}- (void)broadcastController:(RPBroadcastController* _Nullable)broadcastController didUpdateServiceInfo:(NSDictionary <NSString*, NSObject <NSCoding>*> *_Nullable)serviceInfo {NSLog( @"RPBroadcastControllerDelegate::didUpdateServiceInfo" );
}@end#endif

Objective-C和C++混合编码

Objective-C和C++虽然都是C语言，但他们语法差异还是很大的。Objective-C和C++混合编程很简单，将Objective-C代码放在.h和.cpp文件中，就可以直接写Objective-C代码，也可以直接写C++代码，不需要做任何特殊处理，编译器可以直接编译混合了Objective-C代码和C++代码的.h文件和.cpp文件。