快手在2020年中旬开源了一个线上OOM监控上报的框架：KOOM，这里简单研究下。

一、官方项目介绍

1.1 描述：

KOOM是快手性能优化团队在处理移动端OOM问题的过程中沉淀出的一套完整解决方案。其中Android Java内存部分在LeakCanary的基础上进行了大量优化，解决了线上内存监控的性能问题，在不影响用户体验的前提下线上采集内存镜像并解析。从 2020 年春节后在快手主APP上线至今解决了大量OOM问题，其性能和稳定性经受住了海量用户与设备的考验，因此决定开源以回馈社区。

1.2 特点：

比leakCanary更丰富的泄漏场景检测；

比leakCanary更好的检测性能；

功能全面的支持线上大规模部署的闭环监控系统；

1.3 KOOM框架

1.4 快手KOOM核心流程包括：

配置下发决策；

监控内存状态；

采集内存镜像；

解析镜像文件(以下简称hprof)生成报告并上传；

问题聚合报警与分配跟进。

1.5 泄漏检测触发机制优化：

泄漏检测触发机制leakCanary做法是GC过后对象WeakReference一直不被加入 ReferenceQueue，它可能存在内存泄漏。这个过程会主动触发GC做确认，可能会造成用户可感知的卡顿，而KOOM采用内存阈值监控来触发镜像采集，将对象是否泄漏的判断延迟到了解析时，阈值监控只要在子线程定期获取关注的几个内存指标即可，性能损耗很低。

1.6 heap dump优化：

传统方案会冻结应用几秒，KOOM会fork新进程来执行dump操作，对父进程的正常执行没有影响。暂停虚拟机需要调用虚拟机的art::Dbg::SuspendVM函数，谷歌从Android 7.0开始对调用系统库做了限制，快手自研了kwai-linker组件，通过caller address替换和dl_iterate_phdr解析绕过了这一限制。

随机采集线上真实用户的内存镜像，普通dump和fork子进程dump阻塞用户使用的耗时如下：

image.png

而从官方给出的测试数据来看，效果似乎是非常好的。

二、官方demo演示

这里就直接跑下官方提供的koom-demo

点击按钮，经过dump heap -> heap analysis -> report cache/koom/report/三个流程(heap analysis时间会比较长，但是完全不影响应用的正常操作),最终在应用的cache/koom/report里生成json报告：

cepheus:/data/data/com.kwai.koom.demo/cache/koom/report # ls

2020-12-08_15-23-32.json

模拟一个最简单的单例CommonUtils持有LeakActivity实例的内存泄漏，看下json最终上报的内容是个啥：

{

"analysisDone":true,

"classInfos":[

{

"className":"android.app.Activity",

"instanceCount":4,

"leakInstanceCount":3

},

{

"className":"android.app.Fragment",

"instanceCount":4,

"leakInstanceCount":3

},

{

"className":"android.graphics.Bitmap",

"instanceCount":115,

"leakInstanceCount":0

},

{

"className":"libcore.util.NativeAllocationRegistry",

"instanceCount":1513,

"leakInstanceCount":0

},

{

"className":"android.view.Window",

"instanceCount":4,

"leakInstanceCount":0

}

],

"gcPaths":[

{

"gcRoot":"Local variable in native code",

"instanceCount":1,

"leakReason":"Activity Leak",

"path":[

{

"declaredClass":"java.lang.Thread",

"reference":"android.os.HandlerThread.contextClassLoader",

"referenceType":"INSTANCE_FIELD"

},

{

"declaredClass":"java.lang.ClassLoader",

"reference":"dalvik.system.PathClassLoader.runtimeInternalObjects",

"referenceType":"INSTANCE_FIELD"

},

{

"declaredClass":"",

"reference":"java.lang.Object[]",

"referenceType":"ARRAY_ENTRY"

},

{

"declaredClass":"com.kwai.koom.demo.CommonUtils",

"reference":"com.kwai.koom.demo.CommonUtils.context",

"referenceType":"STATIC_FIELD"

},

{

"reference":"com.kwai.koom.demo.LeakActivity",

"referenceType":"instance"

}

],

"signature":"378fc01daea06b6bb679bd61725affd163d026a8"

}

],

"runningInfo":{

"analysisReason":"RIGHT_NOW",

"appVersion":"1.0",

"buildModel":"MI 9 Transparent Edition",

"currentPage":"LeakActivity",

"dumpReason":"MANUAL_TRIGGER",

"jvmMax":512,

"jvmUsed":2,

"koomVersion":1,

"manufacture":"Xiaomi",

"nowTime":"2020-12-08_16-07-34",

"pss":32,

"rss":123,

"sdkInt":29,

"threadCount":17,

"usageSeconds":40,

"vss":5674

}

}

这里主要分三个部分：类信息、gc引用路径、运行基本信息。这里从gcPaths中能看出LeakActivity被CommonUtils持有了引用。

三、框架解析

3.1 类图

3.2 时序图

KOOM初始化流程

KOOM执行初始化方法，10秒延迟之后会在threadHandler子线程中先通过check状态判断是否开始工作，工作内容是先检查是不是有未完成分析的文件，如果有就就触发解析，没有则监控内存。

heap dump流程

HeapDumpTrigger

startTrack：监控自动触发dump hprof操作。开启内存监控，子线程5s触发一次检测，看当前是否满足触发heap dump的条件。条件是由一系列阀值组织，这部分后面详细分析。满足阀值后会通过监听回调给HeapDumpTrigger去执行trigger。

trigger:主动触发dump hprof操作。这里是fork子进程来处理的，这部分也到后面详细分析。dump完成之后通过监听回调触发HeapAnalysisTrigger.startTrack触发heap分析流程。

heap analysis流程

HeapAnalysisTrigger

startTrack 根据策略触发hprof文件分析。

trigger 直接触发hprof文件分析。由单独起进程的service来处理，工作内容主要分内存泄漏检测(activity/fragment/bitmap/window)和泄漏数据整理缓存为json文件以供上报。

四、核心源码解析

经过前面的分析，基本上对框架的使用和结构有了一个宏观了解，这部分就打算对一些实现细节进行简单分析。

4.1 内存监控触发dump规则

这里主要是研究HeapMonitor中isTrigger规则，每隔5S都会循环判断该触发条件。

com/kwai/koom/javaoom/monitor/HeapMonitor.java

@Override

public boolean isTrigger() {

if (!started) {

return false;

}

HeapStatus heapStatus = currentHeapStatus();

if (heapStatus.isOverThreshold) {

if (heapThreshold.ascending()) {

if (lastHeapStatus == null || heapStatus.used >= lastHeapStatus.used) {

currentTimes++;

} else {

currentTimes = 0;

}

} else {

currentTimes++;

}

} else {

currentTimes = 0;

}

lastHeapStatus = heapStatus;

return currentTimes >= heapThreshold.overTimes();

}

private HeapStatus lastHeapStatus;

private HeapStatus currentHeapStatus() {

HeapStatus heapStatus = new HeapStatus();

heapStatus.max = Runtime.getRuntime().maxMemory();

heapStatus.used = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();

heapStatus.isOverThreshold = 100.0f * heapStatus.used / heapStatus.max > heapThreshold.value();

return heapStatus;

}

com/kwai/koom/javaoom/common/KConstants.java

public static class HeapThreshold {

public static int VM_512_DEVICE = 510;

public static int VM_256_DEVICE = 250;

public static int VM_128_DEVICE = 128;

public static float PERCENT_RATIO_IN_512_DEVICE = 80;

public static float PERCENT_RATIO_IN_256_DEVICE = 85;

public static float PERCENT_RATIO_IN_128_DEVICE = 90;

public static float getDefaultPercentRation() {

int maxMem = (int) (Runtime.getRuntime().maxMemory() / MB);

if (maxMem >= VM_512_DEVICE) {

return KConstants.HeapThreshold.PERCENT_RATIO_IN_512_DEVICE;

} else if (maxMem >= VM_256_DEVICE) {

return KConstants.HeapThreshold.PERCENT_RATIO_IN_256_DEVICE;

} else if (maxMem >= VM_128_DEVICE) {

return KConstants.HeapThreshold.PERCENT_RATIO_IN_128_DEVICE;

}

return KConstants.HeapThreshold.PERCENT_RATIO_IN_512_DEVICE;

}

public static int OVER_TIMES = 3;

public static int POLL_INTERVAL = 5000;

}

这里就是针对不同内存大小做了不同的阀值比例：

应用内存>512M 80%

应用内存>256M 85%

应用内存>128M 90%

低于128M的默认按80%

应用已使用内存/最大内存超过该比例则会触发heapStatus.isOverThreshold。连续满足3次触发heap dump，但是这个过程会考虑内存增长性，3次范围内出现了使用内存下降或者使用内存/最大内存低于对应阀值了则清零。

因此规则总结为：3次满足>阀值条件且内存一直处于上升期才触发。这样能减少无效的dump。

4.2 fork进程执行dump操作实现

目前项目中默认使用ForkJvmHeapDumper来执行dump。

com/kwai/koom/javaoom/dump/ForkJvmHeapDumper.java

@Override

public boolean dump(String path) {

boolean dumpRes = false;

try {

int pid = trySuspendVMThenFork();//暂停虚拟机，copy-on-write fork子进程

if (pid == 0) {//子进程中

Debug.dumpHprofData(path);//dump hprof

exitProcess();//_exit(0) 退出进程

} else {//父进程中

resumeVM();//resume当前虚拟机

dumpRes = waitDumping(pid);//waitpid异步等待pid进程结束

}

} catch (Exception e) {

e.printStackTrace();

}

return dumpRes;

}

谷歌从Android 7.0开始对调用系统库做了限制，快手自研了kwai-linker组件，通过caller address替换和dl_iterate_phdr解析绕过了这一限制，详细分析在后续的这篇文章有展开：KOOM V1.0.5 fork dump方案解析。

4.3 内存泄漏检测实现

内存泄漏检测核心代码在于SuspicionLeaksFinder.find

public Pair, List> find() {

boolean indexed = buildIndex();

if (!indexed) {

return null;

}

initLeakDetectors();

findLeaks();

return findPath();

}

4.3.1 buildIndex()

private boolean buildIndex() {

Hprof hprof = Hprof.Companion.open(hprofFile.file());

//选择可以作为gcroot的类类型

KClass[] gcRoots = new KClass[]{

Reflection.getOrCreateKotlinClass(GcRoot.JniGlobal.class),

//Reflection.getOrCreateKotlinClass(GcRoot.JavaFrame.class),

Reflection.getOrCreateKotlinClass(GcRoot.JniLocal.class),

//Reflection.getOrCreateKotlinClass(GcRoot.MonitorUsed.class),

Reflection.getOrCreateKotlinClass(GcRoot.NativeStack.class),

Reflection.getOrCreateKotlinClass(GcRoot.StickyClass.class),

Reflection.getOrCreateKotlinClass(GcRoot.ThreadBlock.class),

Reflection.getOrCreateKotlinClass(GcRoot.ThreadObject.class),

Reflection.getOrCreateKotlinClass(GcRoot.JniMonitor.class)};

//解析hprof文件为HeapGraph对象

heapGraph = HprofHeapGraph.Companion.indexHprof(hprof, null,

kotlin.collections.SetsKt.setOf(gcRoots));

return true;

}

fun indexHprof(

hprof: Hprof,

proguardMapping: ProguardMapping? = null,

indexedGcRootTypes: Set> = setOf(

JniGlobal::class,

JavaFrame::class,

JniLocal::class,

MonitorUsed::class,

NativeStack::class,

StickyClass::class,

ThreadBlock::class,

ThreadObject::class,

JniMonitor::class

)

): HeapGraph {

//确认对应的record的index

val index = HprofInMemoryIndex.createReadingHprof(hprof, proguardMapping, indexedGcRootTypes)

//HprofHeapGraph是HeapGraph的实现类

return HprofHeapGraph(hprof, index)

}

HprofInMemoryIndex.createReadingHprof核心逻辑：读取hprof文件，将不同内容封装为不同的record，然后将record转为索引化的index封装，之后查找内容可以通过index去索引到。

HprofReader.readHprofRecords() 封装record

LoadClassRecord

InstanceSkipContentRecord

ObjectArraySkipContentRecord

PrimitiveArraySkipContentRecord

HprofInMemoryIndex.onHprofRecord() 封装index：

classIndex

instanceIndex

objectArrayIndex

primitiveArrayIndex

class HprofHeapGraph internal constructor(

private val hprof: Hprof,

private val index: HprofInMemoryIndex

) : HeapGraph {

...

override val gcRoots: List

get() = index.gcRoots()

override val objects: Sequence

get() {

return index.indexedObjectSequence().map {wrapIndexedObject(it.second, it.first)}

}

override val classes: Sequence

get() {

return index.indexedClassSequence().map {val objectId = it.first

val indexedObject = it.second

HeapClass(this, indexedObject, objectId)

}

}

override val instances: Sequence

get() {

return index.indexedInstanceSequence().map {val objectId = it.first

val indexedObject = it.second

val isPrimitiveWrapper = index.primitiveWrapperTypes.contains(indexedObject.classId)

HeapInstance(this, indexedObject, objectId, isPrimitiveWrapper)

}

}

override val objectArrays: Sequence

get() = index.indexedObjectArraySequence().map {val objectId = it.first

val indexedObject = it.second

val isPrimitiveWrapper = index.primitiveWrapperTypes.contains(indexedObject.arrayClassId)

HeapObjectArray(this, indexedObject, objectId, isPrimitiveWrapper)

}

override val primitiveArrays: Sequence

get() = index.indexedPrimitiveArraySequence().map {val objectId = it.first

val indexedObject = it.second

HeapPrimitiveArray(this, indexedObject, objectId)

}

Hprof 经过层层转换最终封装为HprofHeapGraph。

简而言之，这部分功能主要是将Hrpof文件按照扫描的格式解析为结构化的索引关系图，索引化后的内容封装为HprofHeapGraph，由它去通过对应的起始索引去定位每类数据。没有细抠这部分的实现细节，实现这个功能的库之前是squere的HAHA，现在改为shark,但是提供的功能大同小异。

4.3.2 initLeakDetectors() 与findLeaks()

初始化泄漏检测者：

private void initLeakDetectors() {

addDetector(new ActivityLeakDetector(heapGraph));

addDetector(new FragmentLeakDetector(heapGraph));

addDetector(new BitmapLeakDetector(heapGraph));

addDetector(new NativeAllocationRegistryLeakDetector(heapGraph));

addDetector(new WindowLeakDetector(heapGraph));

ClassHierarchyFetcher.initComputeGenerations(computeGenerations);

leakReasonTable = new HashMap<>();

}

初始化各类型泄漏的检测者，主要包含Activity、Fragment、Bitmap+NativeAllocationRegistry、window的泄漏检测。

其次是梳理以上几类对象类继承关系串，检测覆盖到他们的子类。

public void findLeaks() {

KLog.i(TAG, "start find leaks");

//从HprofHeapGraph中获取所有instance

Sequence instances = heapGraph.getInstances();

Iterator instanceIterator = instances.iterator();

while (instanceIterator.hasNext()) {

HeapObject.HeapInstance instance = instanceIterator.next();

if (instance.isPrimitiveWrapper()) {

continue;

}

ClassHierarchyFetcher.process(instance.getInstanceClassId(),

instance.getInstanceClass().getClassHierarchy());

for (LeakDetector leakDetector : leakDetectors) {

//是检测对象的子类&满足对应泄漏条件

if (leakDetector.isSubClass(instance.getInstanceClassId())

&& leakDetector.isLeak(instance)) {

ClassCounter classCounter = leakDetector.instanceCount();

if (classCounter.leakInstancesCount <=

SAME_CLASS_LEAK_OBJECT_GC_PATH_THRESHOLD) {

leakingObjects.add(instance.getObjectId());

leakReasonTable.put(instance.getObjectId(), leakDetector.leakReason());

}

}

}

}

//关注class和对应instance数量，加入json

HeapAnalyzeReporter.addClassInfo(leakDetectors);

findPrimitiveArrayLeaks();

findObjectArrayLeaks();

}

这里重点看看各类型对象是如何判断泄漏的：

ActivityLeakDetector:

private static final String ACTIVITY_CLASS_NAME = "android.app.Activity";

private static final String FINISHED_FIELD_NAME = "mFinished";

private static final String DESTROYED_FIELD_NAME = "mDestroyed";

public boolean isLeak(HeapObject.HeapInstance instance) {

activityCounter.instancesCount++;

HeapField destroyField = instance.get(ACTIVITY_CLASS_NAME, DESTROYED_FIELD_NAME);

HeapField finishedField = instance.get(ACTIVITY_CLASS_NAME, FINISHED_FIELD_NAME);

assert destroyField != null;

assert finishedField != null;

boolean abnormal = destroyField.getValue().getAsBoolean() == null

|| finishedField.getValue().getAsBoolean() == null;

if (abnormal) {

return false;

}

boolean leak = destroyField.getValue().getAsBoolean()

|| finishedField.getValue().getAsBoolean();

if (leak) {

activityCounter.leakInstancesCount++;

}

return leak;

}

mDestroyed和mFinish字段为true，但是实例还存在的Activity是疑似泄漏对象。

FragmentLeakDetector:

private static final String NATIVE_FRAGMENT_CLASS_NAME = "android.app.Fragment";

// native android Fragment, deprecated as of API 28.

private static final String SUPPORT_FRAGMENT_CLASS_NAME = "android.support.v4.app.Fragment";

// pre-androidx, support library version of the Fragment implementation.

private static final String ANDROIDX_FRAGMENT_CLASS_NAME = "androidx.fragment.app.Fragment";

// androidx version of the Fragment implementation

private static final String FRAGMENT_MANAGER_FIELD_NAME = "mFragmentManager”;

private static final String FRAGMENT_MCALLED_FIELD_NAME = "mCalled”;//Used to verify that subclasses call through to super class.

public FragmentLeakDetector(HeapGraph heapGraph) {

HeapObject.HeapClass fragmentHeapClass =

heapGraph.findClassByName(ANDROIDX_FRAGMENT_CLASS_NAME);

fragmentClassName = ANDROIDX_FRAGMENT_CLASS_NAME;

if (fragmentHeapClass == null) {

fragmentHeapClass = heapGraph.findClassByName(NATIVE_FRAGMENT_CLASS_NAME);

fragmentClassName = NATIVE_FRAGMENT_CLASS_NAME;

}

if (fragmentHeapClass == null) {

fragmentHeapClass = heapGraph.findClassByName(SUPPORT_FRAGMENT_CLASS_NAME);

fragmentClassName = SUPPORT_FRAGMENT_CLASS_NAME;

}

assert fragmentHeapClass != null;

fragmentClassId = fragmentHeapClass.getObjectId();

fragmentCounter = new ClassCounter();

}

public boolean isLeak(HeapObject.HeapInstance instance) {

if (VERBOSE_LOG) {

KLog.i(TAG, "run isLeak");

}

fragmentCounter.instancesCount++;

boolean leak = false;

HeapField fragmentManager = instance.get(fragmentClassName, FRAGMENT_MANAGER_FIELD_NAME);

if (fragmentManager != null && fragmentManager.getValue().getAsObject() == null) {

HeapField mCalledField = instance.get(fragmentClassName, FRAGMENT_MCALLED_FIELD_NAME);

boolean abnormal = mCalledField == null || mCalledField.getValue().getAsBoolean() == null;

if (abnormal) {

KLog.e(TAG, "ABNORMAL mCalledField is null");

return false;

}

leak = mCalledField.getValue().getAsBoolean();

if (leak) {

if (VERBOSE_LOG) {

KLog.e(TAG, "fragment leak : " + instance.getInstanceClassName());

}

fragmentCounter.leakInstancesCount++;

}

}

return leak;

}

这里分了三种fragment:

android.app.Fragment

android.support.v4.app.Fragment

androidx.fragment.app.Fragment

对应的FragmentManager实例为null(这表示fragment被remove了)且满足对应的mCalled为true,即非perform状态，而是对应生命周期被回调状态(onDestroy)，但是实例还存在的Fragment是疑似泄漏对象。

BitmapLeakDetector

private static final String BITMAP_CLASS_NAME = "android.graphics.Bitmap”;

public boolean isLeak(HeapObject.HeapInstance instance) {

if (VERBOSE_LOG) {

KLog.i(TAG, "run isLeak");

}

bitmapCounter.instancesCount++;

HeapField fieldWidth = instance.get(BITMAP_CLASS_NAME, "mWidth");

HeapField fieldHeight = instance.get(BITMAP_CLASS_NAME, "mHeight");

assert fieldHeight != null;

assert fieldWidth != null;

boolean abnormal = fieldHeight.getValue().getAsInt() == null

|| fieldWidth.getValue().getAsInt() == null;

if (abnormal) {

KLog.e(TAG, "ABNORMAL fieldWidth or fieldHeight is null");

return false;

}

int width = fieldWidth.getValue().getAsInt();

int height = fieldHeight.getValue().getAsInt();

boolean suspicionLeak = width * height >= KConstants.BitmapThreshold.DEFAULT_BIG_BITMAP;

if (suspicionLeak) {

KLog.e(TAG, "bitmap leak : " + instance.getInstanceClassName() + " " +

"width:" + width + " height:" + height);

bitmapCounter.leakInstancesCount++;

}

return suspicionLeak;

}

这里是针对Bitmap size做判断，超过768*1366这个size的认为泄漏。

另外，NativeAllocationRegistryLeakDetector和WindowLeakDetector两类还没做具体泄漏判断规则，不参与对象泄漏检测，只是做了统计。

总结：

整体看下来，KOOM有两个值得借鉴的点：

1.触发内存泄漏检测，常规是watcher activity/fragment的onDestroy，而KOOM是定期轮询查看当前内存是否到达阀值；

2.dump hprof，常规是对应进程dump，而KOOM是fork进程dump。