ECM技术学习：基于模板的帧内模式推导(Template based intra mode derivation )

和基于梯度的帧内模式推导类似，ECM中还使用了基于模板的帧内模式推导。

基本原理：对于当前待预测CU，在模板区域计算预测像素和重建像素的SATD，从MPM列表中选出两个SATD最小的预测模式mode1和mode2，根据mode1和mode2的SATD，决定是否应用加权融合。如果costMode2 < 2*costMode1，则使用mode1和mode2预测当前CU，并将mode1和mode2得到的预测值加权融合作为CU的最终预测值，其中weight1 = costMode2/(costMode1+costMode2)，weight2 = 1 – weight1；否则，仅使用mode1预测当前CU。

如下图所示，模板区域即上相邻重建区域和左相邻重建区域，模板的宽度和高度与CU尺寸相关：

当CU的高度大于8时，模板的宽度L1 = 4, 否则，L1 = 2
当CU的宽度大于8时，模板的高度L2 = 4，否则，L2 = 2

原理分析：TIMD假定模板区域和当前待预测区域的分布特性一致，由于模板区域的像素已重建，通过遍历MPM列表，计算模板区域的预测像素，求预测像素和重建像素的SATD，可以选出最优的模式，将该模式作为待预测区域的帧内模式。在解码端，可以通过相同的推导方式推导帧内预测模式，从而可以大幅降低模式的编码比特。

TIMD模式的模式推导流程：

先判断相邻CU（左、左下、右上、上、左上）是否存在角度预测模式，如果不存在角度预测模式，则仅在模板区域遍历Planar模式和DC模式，选出最佳的一种模式作为当前CU的预测模式
否则，检查MPM + DC/HOR/VER模式，从模板区域中选出SATD最小的两个模式iBestMode和iSecondaryMode
检查iBestMode和iSecondaryMode相邻的两个角度模式（+1和-1）
若(uiSecondaryCost - uiBestCost) < uiBestCost，则使用融合预测的方式，否则仅使用iBestMode预测当前CU。

在TIMD中，为了进一步提高预测精度，将2-66模式细化为2-130，如下图所示。

TIMD模式推导的代码及注释如下所示：

int IntraPrediction::deriveTimdMode( const CPelBuf &recoBuf, const CompArea &area, CodingUnit &cu )
{int channelBitDepth = cu.slice->getSPS()->getBitDepth(CHANNEL_TYPE_LUMA);SizeType uiWidth = cu.lwidth();SizeType uiHeight = cu.lheight();static Pel PredLuma[(MAX_CU_SIZE + DIMD_MAX_TEMP_SIZE) * (MAX_CU_SIZE + DIMD_MAX_TEMP_SIZE)];memset(PredLuma, 0, (MAX_CU_SIZE + DIMD_MAX_TEMP_SIZE) * (MAX_CU_SIZE + DIMD_MAX_TEMP_SIZE) * sizeof(Pel));Pel* piPred = PredLuma;uint32_t uiPredStride = MAX_CU_SIZE + DIMD_MAX_TEMP_SIZE;int  iCurX  = cu.lx(); // 当前CU的x坐标int  iCurY  = cu.ly(); // 当前CU的y坐标int  iRefX  = -1, iRefY = -1;uint32_t uiRefWidth = 0, uiRefHeight = 0;//参考像素的宽度和高度int iTempWidth = 4, iTempHeight = 4; // 模板的宽度和高度if(uiWidth <= 8){iTempWidth = 2;}if(uiHeight <= 8){iTempHeight = 2;}// 模板类型// LEFT_NEIGHBOR or ABOVE_NEIGHBOR or LEFT_ABOVE_NEIGHBORTEMPLATE_TYPE eTempType = CU::deriveTimdRefType(iCurX, iCurY, uiWidth, uiHeight, iTempWidth, iTempHeight, iRefX, iRefY, uiRefWidth, uiRefHeight);if (eTempType != NO_NEIGHBOR){const CodingStructure& cs   = *cu.cs;m_ipaParam.multiRefIndex        = iTempWidth;Pel* piOrg = cs.picture->getRecoBuf( area ).buf;int iOrgStride = cs.picture->getRecoBuf( area ).stride;piOrg += (iRefY - iCurY) * iOrgStride + (iRefX - iCurX); // 移动到两个模板中间的左上角位置DistParam distParamSad[2]; // above, leftdistParamSad[0].applyWeight = false;distParamSad[0].useMR = false;distParamSad[1].applyWeight = false;distParamSad[1].useMR = false;if(eTempType == LEFT_ABOVE_NEIGHBOR){m_timdSatdCost->setTimdDistParam(distParamSad[0], piOrg + iTempWidth, piPred + iTempWidth, iOrgStride, uiPredStride, channelBitDepth, COMPONENT_Y, uiWidth, iTempHeight, 0, 1, true); // Use HAD (SATD) costm_timdSatdCost->setTimdDistParam(distParamSad[1], piOrg + iTempHeight * iOrgStride, piPred + iTempHeight * uiPredStride, iOrgStride, uiPredStride, channelBitDepth, COMPONENT_Y, iTempWidth, uiHeight, 0, 1, true); // Use HAD (SATD) cost}else if(eTempType == LEFT_NEIGHBOR){m_timdSatdCost->setTimdDistParam(distParamSad[1], piOrg, piPred, iOrgStride, uiPredStride, channelBitDepth, COMPONENT_Y, iTempWidth, uiHeight, 0, 1, true);}else if(eTempType == ABOVE_NEIGHBOR){m_timdSatdCost->setTimdDistParam(distParamSad[0], piOrg, piPred, iOrgStride, uiPredStride, channelBitDepth, COMPONENT_Y, uiWidth, iTempHeight, 0, 1, true);}// 获取参考像素initTimdIntraPatternLuma(cu, area, eTempType != ABOVE_NEIGHBOR ? iTempWidth : 0, eTempType != LEFT_NEIGHBOR ? iTempHeight : 0, uiRefWidth, uiRefHeight);// 记录相邻PU的预测模式// modeIdx表示加入到候选模式列表的模式数目uint32_t uiIntraDirNeighbor[5] = {0}, modeIdx = 0; bool includedMode[EXT_VDIA_IDX + 1];memset(includedMode, false, (EXT_VDIA_IDX + 1) * sizeof(bool));auto &pu = *cu.firstPU;uint32_t uiRealW = uiRefWidth + (eTempType == LEFT_NEIGHBOR? iTempWidth : 0);// 实际宽度uint32_t uiRealH = uiRefHeight + (eTempType == ABOVE_NEIGHBOR? iTempHeight : 0);// 实际高度uint64_t maxCost = (uint64_t)(iTempWidth * cu.lheight() + iTempHeight * cu.lwidth()); // 最大代价，相当于每个位置的失真为1uint64_t uiBestCost = MAX_UINT64;int iBestMode = PLANAR_IDX;uint64_t uiSecondaryCost = MAX_UINT64;int iSecondaryMode = PLANAR_IDX;const Position posLTx = pu.Y().topLeft();const Position posRTx = pu.Y().topRight();const Position posLBx = pu.Y().bottomLeft();// left 左侧PUconst PredictionUnit *puLeftx = pu.cs->getPURestricted(posLBx.offset(-1, 0), pu, pu.chType);if (puLeftx && CU::isIntra(*puLeftx->cu)){uiIntraDirNeighbor[modeIdx] = PU::getIntraDirLuma( *puLeftx );if (!puLeftx->cu->timd){uiIntraDirNeighbor[modeIdx] = MAP67TO131(uiIntraDirNeighbor[modeIdx]);}if( !includedMode[uiIntraDirNeighbor[modeIdx]] ){includedMode[uiIntraDirNeighbor[modeIdx]] = true;modeIdx++;}}// above 上侧PUconst PredictionUnit *puAbovex = pu.cs->getPURestricted(posRTx.offset(0, -1), pu, pu.chType);if (puAbovex && CU::isIntra(*puAbovex->cu) && CU::isSameCtu(*pu.cu, *puAbovex->cu)){uiIntraDirNeighbor[modeIdx] =PU::getIntraDirLuma( *puAbovex );if (!puAbovex->cu->timd){uiIntraDirNeighbor[modeIdx] = MAP67TO131(uiIntraDirNeighbor[modeIdx]);}if( !includedMode[uiIntraDirNeighbor[modeIdx]] ){includedMode[uiIntraDirNeighbor[modeIdx]] = true;modeIdx++;}}// below left 左下PUconst PredictionUnit *puLeftBottomx = cs.getPURestricted( posLBx.offset( -1, 1 ), pu, pu.chType );if (puLeftBottomx && CU::isIntra(*puLeftBottomx->cu)){uiIntraDirNeighbor[modeIdx] = PU::getIntraDirLuma( *puLeftBottomx );if (!puLeftBottomx->cu->timd){uiIntraDirNeighbor[modeIdx] = MAP67TO131(uiIntraDirNeighbor[modeIdx]);}if( !includedMode[uiIntraDirNeighbor[modeIdx]] ){includedMode[uiIntraDirNeighbor[modeIdx]] = true;modeIdx++;}}// above right 右上PUconst PredictionUnit *puAboveRightx = cs.getPURestricted( posRTx.offset( 1, -1 ), pu, pu.chType );if (puAboveRightx && CU::isIntra(*puAboveRightx->cu)){uiIntraDirNeighbor[modeIdx] = PU::getIntraDirLuma( *puAboveRightx );if (!puAboveRightx->cu->timd){uiIntraDirNeighbor[modeIdx] = MAP67TO131(uiIntraDirNeighbor[modeIdx]);}if( !includedMode[uiIntraDirNeighbor[modeIdx]] ){includedMode[uiIntraDirNeighbor[modeIdx]] = true;modeIdx++;}}//above left 左上PUconst PredictionUnit *puAboveLeftx = cs.getPURestricted( posLTx.offset( -1, -1 ), pu, pu.chType );if( puAboveLeftx && CU::isIntra(*puAboveLeftx->cu) ){uiIntraDirNeighbor[modeIdx] = PU::getIntraDirLuma( *puAboveLeftx );if (!puAboveLeftx->cu->timd){uiIntraDirNeighbor[modeIdx] = MAP67TO131(uiIntraDirNeighbor[modeIdx]);}if( !includedMode[uiIntraDirNeighbor[modeIdx]] ){includedMode[uiIntraDirNeighbor[modeIdx]] = true;modeIdx++;}}bool bNoAngular = false;if(modeIdx >= 2){bNoAngular = true;for(uint32_t i = 0; i < modeIdx; i++){if(uiIntraDirNeighbor[i] > DC_IDX){bNoAngular = false;break;}}}// 如果相邻模式中不存在角度模式，则遍历DC模式和Planar模式if (bNoAngular){for(int iMode = 0; iMode <= 1; iMode ++){uint64_t uiCost = 0;initPredTimdIntraParams(pu, area, iMode);predTimdIntraAng(COMPONENT_Y, pu, iMode, piPred, uiPredStride, uiRealW, uiRealH, eTempType, (eTempType == ABOVE_NEIGHBOR)? 0: iTempWidth, (eTempType == LEFT_NEIGHBOR)? 0: iTempHeight);if(eTempType == LEFT_ABOVE_NEIGHBOR){uiCost += distParamSad[0].distFunc(distParamSad[0]);uiCost += distParamSad[1].distFunc(distParamSad[1]);}else if(eTempType == LEFT_NEIGHBOR){uiCost = distParamSad[1].distFunc(distParamSad[1]);}else if(eTempType == ABOVE_NEIGHBOR){uiCost += distParamSad[0].distFunc(distParamSad[0]);}else{assert(0);}if(uiCost < uiBestCost){uiBestCost = uiCost;iBestMode = iMode;}if(uiBestCost <= maxCost){break;}}cu.timdMode = iBestMode;cu.timdIsBlended = false;return iBestMode;}
#if SECONDARY_MPMuint8_t mpmList[NUM_MOST_PROBABLE_MODES];uint8_t intraNonMPM[NUM_NON_MPM_MODES];// 获取MPM列表PU::getIntraMPMs(pu, mpmList, intraNonMPM);
#elseunsigned mpmList[NUM_MOST_PROBABLE_MODES];PU::getIntraMPMs(pu, mpmList);
#endifunsigned mpmExtraList[NUM_MOST_PROBABLE_MODES + 3]; // +DC/VER/HORint maxModeNum = NUM_MOST_PROBABLE_MODES;unsigned modeCandList[3] = {DC_IDX, HOR_IDX, VER_IDX};bool bNotExist[3] = {true, true, true}; // 指的是DC/VER/HOR三种模式分别是否在MPM列表中// 遍历MPM列表for (int i = 0; i < NUM_MOST_PROBABLE_MODES; i++){mpmExtraList[i] = mpmList[i]; // 将MPM列表复制给mpmExtraList中if (bNotExist[0] && mpmList[i] == DC_IDX){bNotExist[0] = false;}if (bNotExist[1] && mpmList[i] == HOR_IDX){bNotExist[1] = false;}if (bNotExist[2] && mpmList[i] == VER_IDX){bNotExist[2] = false;}}for (int i = 0; i < 3; i++){if (bNotExist[i]) // 如果DC/VER/HOR不在MPM列表中，则将其加入进去{mpmExtraList[maxModeNum++] = modeCandList[i];}}// 遍历全部候选模式for(int i = 0; i < maxModeNum; i ++){uint64_t uiCost = 0;int iMode = mpmExtraList[i];if (iMode > DC_IDX) // 对于角度模式，映射到130{iMode = MAP67TO131(iMode);}initPredTimdIntraParams(pu, area, iMode); // 初始化预测参数predTimdIntraAng(COMPONENT_Y, pu, iMode, piPred, uiPredStride, uiRealW, uiRealH, eTempType, (eTempType == ABOVE_NEIGHBOR)? 0: iTempWidth, (eTempType == LEFT_NEIGHBOR)? 0: iTempHeight);if(eTempType == LEFT_ABOVE_NEIGHBOR){uiCost += distParamSad[0].distFunc(distParamSad[0]);uiCost += distParamSad[1].distFunc(distParamSad[1]);}else if(eTempType == LEFT_NEIGHBOR){uiCost = distParamSad[1].distFunc(distParamSad[1]);}else if(eTempType == ABOVE_NEIGHBOR){uiCost += distParamSad[0].distFunc(distParamSad[0]);}else{assert(0);}if( uiCost < uiBestCost ){uiSecondaryCost = uiBestCost;iSecondaryMode  = iBestMode;uiBestCost  = uiCost;iBestMode = iMode;}else if (uiCost < uiSecondaryCost){uiSecondaryCost = uiCost;iSecondaryMode  = iMode;}if (uiSecondaryCost <= maxCost){break;}}int midMode = iBestMode; if (midMode > DC_IDX && uiBestCost > maxCost){// 检查最佳模式的相邻两个角度模式for (int i = -1; i <= 1; i+=2){int iMode = midMode + i;if (iMode <= DC_IDX || iMode > EXT_VDIA_IDX){continue;}initPredTimdIntraParams(pu, area, iMode);predTimdIntraAng(COMPONENT_Y, pu, iMode, piPred, uiPredStride, uiRealW, uiRealH, eTempType, (eTempType == ABOVE_NEIGHBOR)? 0: iTempWidth, (eTempType == LEFT_NEIGHBOR)? 0: iTempHeight);uint64_t uiCost = 0;if(eTempType == LEFT_ABOVE_NEIGHBOR){uiCost += distParamSad[0].distFunc(distParamSad[0]);uiCost += distParamSad[1].distFunc(distParamSad[1]);}else if(eTempType == LEFT_NEIGHBOR){uiCost = distParamSad[1].distFunc(distParamSad[1]);}else if(eTempType == ABOVE_NEIGHBOR){uiCost += distParamSad[0].distFunc(distParamSad[0]);}else{assert(0);}if(uiCost < uiBestCost){uiBestCost  = uiCost;iBestMode = iMode;}if(uiBestCost <= maxCost){break;}}}midMode = iSecondaryMode;if (midMode > DC_IDX && uiSecondaryCost > maxCost){// 检查次优模式的相邻两个角度模式for (int i = -1; i <= 1; i+=2){int iMode = midMode + i;if (iMode <= DC_IDX || iMode > EXT_VDIA_IDX){continue;}initPredTimdIntraParams(pu, area, iMode);predTimdIntraAng(COMPONENT_Y, pu, iMode, piPred, uiPredStride, uiRealW, uiRealH, eTempType, (eTempType == ABOVE_NEIGHBOR)? 0: iTempWidth, (eTempType == LEFT_NEIGHBOR)? 0: iTempHeight);uint64_t uiCost = 0;if(eTempType == LEFT_ABOVE_NEIGHBOR){uiCost += distParamSad[0].distFunc(distParamSad[0]);uiCost += distParamSad[1].distFunc(distParamSad[1]);}else if(eTempType == LEFT_NEIGHBOR){uiCost = distParamSad[1].distFunc(distParamSad[1]);}else if(eTempType == ABOVE_NEIGHBOR){uiCost += distParamSad[0].distFunc(distParamSad[0]);}else{assert(0);}if(uiCost < uiSecondaryCost){uiSecondaryCost  = uiCost;iSecondaryMode = iMode;}if(uiSecondaryCost <= maxCost){break;}}}if ((uiSecondaryCost - uiBestCost) < uiBestCost){cu.timdMode         = iBestMode;cu.timdIsBlended    = true;cu.timdModeSecondary = iSecondaryMode;const int blend_sum_weight = 6;int       sum_weight       = 1 << blend_sum_weight;double dRatio       = 0.0;dRatio              = (double) uiSecondaryCost / (double) (uiBestCost + uiSecondaryCost);int iRatio          = static_cast<int>(dRatio * sum_weight + 0.5);cu.timdFusionWeight[0] = iRatio;cu.timdFusionWeight[1] = sum_weight - iRatio;}else{cu.timdMode      = iBestMode;cu.timdIsBlended = false;}return iBestMode;}else{cu.timdMode = PLANAR_IDX;cu.timdIsBlended = false;return PLANAR_IDX;}
}

TIMD模式加权预测代码如下：

  if (pu.cu->timd && pu.cu->timdIsBlended && isLuma(compID)){int width = piPred.width;int height = piPred.height;const UnitArea localUnitArea( pu.chromaFormat, Area( 0, 0, width, height ) );PelBuf predFusion = m_tempBuffer[1].getBuf( localUnitArea.Y() );const bool applyPdpc = m_ipaParam.applyPDPC;PredictionUnit pu2 = pu;pu2.intraDir[0] = pu.cu->timdModeSecondary;initPredIntraParams(pu2, pu.Y(), *(pu.cs->sps));switch (pu.cu->timdModeSecondary) // 次优模式{case(PLANAR_IDX): xPredIntraPlanar(srcBuf, predFusion); break;case(DC_IDX):     xPredIntraDc(srcBuf, predFusion, channelType, false); break;default:          xPredIntraAng(srcBuf, predFusion, channelType, clpRng, bExtIntraDir); break;}m_ipaParam.applyPDPC = applyPdpc;// do blendingconst int log2WeightSum = 6;Pel *pelPred = piPred.buf;Pel *pelPredFusion = predFusion.buf;int  w0 = pu.cu->timdFusionWeight[0], w1 = pu.cu->timdFusionWeight[1];for( int y = 0; y < height; y++ ){for( int x = 0; x < width; x++ ){int blend = pelPred[x] * w0;blend += pelPredFusion[x] * w1;pelPred[x] = (Pel)(blend >> log2WeightSum); // 加权融合}pelPred += piPred.stride;pelPredFusion += predFusion.stride;}}

ECM技术学习：基于模板的帧内模式推导(Template based intra mode derivation )相关推荐

ECM技术学习：解码端帧内模式推导(Decoder-side Intra Mode Derivation )
解码端帧内模式推导(DIMD)技术是之前在VVC标准化的过程中提出的技术,因为其在解码端引入的复杂度较高,因此没有被VVC采纳.为了探索下一代压缩标准,JVET最近设立了最新的ECM参考平台,将DIM ...
JVET-AB0117-基于模板的帧内推导的方向性融合
本提案是针对 ECM 中的 TIMD(基于模板的帧内模式推导,Template based intra mode derivation ) 的技术的加权方式的改进.具体地,本提案提出使用方向混合(di ...
ECM技术学习：重叠块运动补偿（Overlapped Block Motion Compensation）
重叠块运动补偿(Overlapped Block Motion Compensation,OBMC)技术是在当前块运动补偿完成之后,将使用相邻块的运动信息进行运动补偿得到的块与当前预测块进行加权,主要 ...
ECM技术学习：卷积跨分量帧内预测模型（Convolutional cross-component intra prediction model）
卷积交叉分量模型(convolutional cross-component model,CCCM)基本思想和CCLM模式类似,建立亮度和色度之间模型实现从亮度重建像素预测色度像素.和CCLM一样,预 ...
H.266/VVC相关技术学习笔记20：帧间预测技术中的MMVD技术（Merge mode with MVD）
今天介绍一下帧间预测技术中的MMVD技术(Merge mode with MVD),也称带有运动矢量差的融合技术,MMVD也属于基于Merge的技术中的一种,在解码端的语法元素中也属于Merge分支. ...
H.266/VVC相关技术学习笔记18：帧间预测中的AMVR技术（自适应运动适量精度）
AMVR技术也称为自适应运动适量精度技术,就是在以前的HEVC中,MVD的精度只有一个默认的1/4像素精度,但是由于要适应不同分辨率的图像,仅仅使用一个精度去表示MVD是远远不够的,因此在VTM6.0 ...
H.266/VVC相关技术学习笔记21：帧间预测中五种Merge模式的熵编码方式
今天主要详细讲一下帧间预测中五种Merge模式的熵编码方式,以及对应的VTM的代码中的编码方式的实现.现阶段VTM6.0中Merge模式大致上分为五种,分别是Subblock_Merge.MMVD_M ...
H.266/VVC技术学习之帧内模式编码
在HEVC中,支持33种角度模式.DC模式和Planar模式,为了减少编码比特,使用长度为3的最可能模式列表.在VVC中,引入了ISP模式.MRL模式.MIP模式等,帧内模式编码时需要先对这些模式的f ...
TIP 2019开源论文：基于深度学习的HEVC多帧环路滤波方法
作者丨李天一学校丨北京航空航天大学博士生研究方向丨视频编码与深度学习本文概述的是作者近期在 IEEE TIP 期刊上发表的论文:A Deep Learning Approach for Mult ...

ECM技术学习：基于模板的帧内模式推导(Template based intra mode derivation )

ECM技术学习：基于模板的帧内模式推导(Template based intra mode derivation )相关推荐

最新文章

热门文章