WebRTC的JitterBuffer笔记

接收类图不全

一主要的类

类video_coding::PacketBuffer是接收RTP包，类RtpVideoStreamReceiver2中有用到。

packet_buffer_(clock_, kPacketBufferStartSize, PacketBufferMaxSize())，最小值512，最大2048。

类video_coding::FrameBuffer是一帧完整的视频数据，类VideoReceiveStream2中有用到。

二判断一帧数据是否完整

判断第一包、最后一包是否在一帧中

parsed_payload->type.Video.is_first_packet_in_frame = first_fragment;

//FU_A第二字节的SER的S

video_header.is_last_packet_in_frame |= rtp_packet.Marker(); //RTP包的mark位

void RtpVideoStreamReceiver2::OnInsertedPacket(video_coding::PacketBuffer::InsertResult result) {for (auto& packet : result.packets) {...//把rtp包写成完整一帧数据，给video_coding::RtpFrameReferenceFinder管理OnAssembledFrame(std::make_unique<video_coding::RtpFrameObject>()}
}RtpVideoStreamReceiver2::OnAssembledFrame(
--| reference_finder_->ManageFrame(std::move(frame));

三帧间完整性判断—gop--没懂

把最后一包的seq作为pid-图片唯一标识。

四 FrameBuffer用在那？解码

VideoReceiveStream2::OnCompleteFrame()
frame_buffer_->InsertFrame(std::move(frame));VideoReceiveStream2::StartNextDecode() {
//NextFrame()获得frame，给HandleEncodedFrame解码
frame_buffer_->NextFrame()--|  frame = absl::WrapUnique(GetNextFrame());
HandleEncodedFrame(std::move(frame));
}

五 JitterDelay的计算

//用卡尔曼kalman滤波估计(或预测) JitterDelay
计算公式如下：跟最大帧长度，平均帧长度等有关
JitterDelay = theta[0] * (MaxFS – AvgFS) + [noiseStdDevs * sqrt(varNoise) – noiseStdDevOffset]double VCMJitterEstimator::CalculateEstimate()，返回值就是JitterDelay。VCMJitterEstimator::UpdateEstimate
--| KalmanEstimateChannel(frameDelayMS, deltaFS);--更新公式参数的值//获取jitter估计值，单位是毫秒
int VCMJitterEstimator::GetJitterEstimate(double rttMultiplier,absl::optional<double> rttMultAddCapMs)// Updates the estimates with the new measurements.
void VCMJitterEstimator::UpdateEstimate(int64_t frameDelayMS,uint32_t frameSizeBytes,bool incompleteFrame /* = false */)

六 JitterDelay：1 作用于从接收队列取数据的时间（意义在那？） 2 赋值给视频帧渲染时间

//帧间延时-->JitterDelay-->wait_ms：从接收队列取frame的时间
//wait_ms值在[0、3000]
int64_t VCMTiming::RenderTimeMsInternal(
{int64_t estimated_complete_time_ms =ts_extrapolator_->ExtrapolateLocalTime(frame_timestamp); //?return estimated_complete_time_ms + actual_delay;
}EncodedFrame* FrameBuffer::GetNextFrame() {
//帧间延时frame_delay传给滤波
if (inter_frame_delay_.CalculateDelay(first_frame->Timestamp(),&frame_delay, receive_time_ms)) {jitter_estimator_.UpdateEstimate(frame_delay, superframe_size);
}
...
timing_->SetJitterDelay(jitter_estimator_.GetJitterEstimate(rtt_mult, rtt_mult_add_cap_ms));
...
}int64_t FrameBuffer::FindNextFrame(int64_t now_ms) {EncodedFrame* frame = frame_it->second.frame.get();if (frame->RenderTime() == -1) {       frame->SetRenderTime(timing_->RenderTimeMs(frame->Timestamp(), now_ms));       }//frame->RenderTime()即期望渲染时间-当前时间-解码所需时间-渲染延迟时间(值10)wait_ms = timing_->MaxWaitingTime(frame->RenderTime(), now_ms);return wait_ms;
}

七 RenderTime视频帧渲染时间，算出渲染等待时间

看有的文章说，动态调整jitterbuffer大小，没有，只是调整了渲染的时间。

void IncomingVideoStream::Dequeue() {  absl::optional<VideoFrame> frame_to_render = render_buffers_.FrameToRender();if (frame_to_render)callback_->OnFrame(*frame_to_render);//即VideoReceiveStream2::OnFrame(//过wait_time，再去渲染if (render_buffers_.HasPendingFrames()) {uint32_t wait_time = render_buffers_.TimeToNextFrameRelease();  incoming_render_queue_.PostDelayedTask([this]() { Dequeue(); }, wait_time);}
}uint32_t VideoRenderFrames::TimeToNextFrameRelease() {//render_time_ms()是frame->SetRenderTime()设置进去的时间。const int64_t time_to_release = incoming_frames_.front().render_time_ms() -render_delay_ms_ - rtc::TimeMillis();return time_to_release < 0 ? 0u : static_cast<uint32_t>(time_to_release);
}
//赋值的地方
frame_info.renderTimeMs = frame.RenderTimeMs();
decodedImage.set_timestamp_us(frameInfo->renderTimeMs *rtc::kNumMicrosecsPerMillisec);

八 Jitterbuffer调优参数

1 rtp_video_header.playout_delay，rtp扩展头字段。

2 jitterbuffer和nack模块关系？没有直接关系