Using GDB To Trace Into a Parallel Worker Spawned By Postmaster During a Large Query

1. Introduction

我正在研究一个新的PostgreSQL特性，它重新定义了确定元组可见性状态的方式。在我开始执行一个大型SELECT查询之前，这个特性工作得很好，这会触发PostgreSQL生成多个并行工作程序来处理请求。当这种情况发生时，我正在研究的功能开始产生错误的结果。返回的数据元组中有很大一部分是丢失的，因为它们被认为是不可见的，而其中的一部分仍然可见。我立刻注意到，我正在研究的新功能不能在并行工作模式下工作，不知为什么，我需要找到一种方法来调试生成的并行工作程序，以检查它是如何计算可见性的，以及内部缺少什么。
在这个博客中，我想和大家分享我如何使用GDB调试和跟踪Postmaster生成的新并行工作程序，以解决可见性问题。
I am working on a new PostgreSQL feature that redefines the way a tuple’s visibility status is determined. The feature is working very nicely until I start doing a large SELECT query, which triggers PostgreSQL to spawn multiple parallel workers to process the request. When this happens, the feature I am working on start to yield incorrect results. A good portion of the data tuples returned are missing because they are considered as invisible, while some portion of it remains visible. It immediately came to my attention that the new feature I am working on does not work in parallel worker mode and somehow I need to find a way to debug into a spawned parallel worker to examine how it is computing the visibility and what is missing inside.In this blog, I would like to share with you how I use GDB to debug and trace into a new parallel worker spawned by Postmaster in order to fix the visibility issue.

2. GDB Basics

我之前写过另一篇博客，展示了如何使用GDB跟踪和调试PostgreSQL问题，并分享了我每天使用的一些最常见的命令来解决软件问题。如果你是GDB的新手，我建议在这里读一读这个博客 I wrote another blog previously that shows how to use GDB to trace and debug a PostgreSQL issues and share some of the most common commands that I use every day to resolve software issues. If you are new to GDB, I suggest giving this blog a read here

3. How and When does PG Spawn A New Parallel Worker

当您使用psql连接到PostgreSQL数据库时，它将产生一个新的后端工作进程来服务于这个连接客户端。您提供的大多数查询都将由这个后端进程处理，包括SELECT、UPDATE、INSERT…等。默认情况下，如果SELECT查询需要对8MB的数据进行顺序扫描，它将尝试使用并行工作程序来加快处理速度。此8MB阈值可以通过postgresql.conf中的min_paralle_table_scan_size参数进行配置。还有一个配置参数max_paralle_workers控制允许生成的最大并行工作线程数。默认值为8。When you use psql to connect to a PostgreSQL database, it will spawn a new backend worker process to serve this connecting client. Most of the queries you provide will be processed by this backend process, includes SELECT, UPDATE, INSERT…etc. By default, if your SELECT query will require doing a sequential scan over 8MB of data, it will try to use a parallel worker to help speed up the processing. This 8MB threshold can be configured by the min_parallel_table_scan_size parameter in postgresql.conf . There is another configuration parameter max_parallel_workers that controls the maximum number of parallel workers is allowed to be spawned. The default is 8.

从技术上讲，我可以通过将min_paralle_table_scan_size设置为一个巨大的数字或将max_paralle_workers设置为0来避免可见性问题。但这真的不是我的风格，我想保留PG提供的所有好东西，同时能够解决问题。Technically, I can avoid my visibility issues simply by either setting min_parallel_table_scan_size to a huge number, or setting max_parallel_workers to 0. But this is really not my style, I would like to keep all these goodies that PG provides while being able to solve the problem.

为了生成并行工作程序，psql后端将在全局进程表中初始化并行工作程序上下文，并基于共享内存初始化消息队列，以便与后端进行通信。然后，它向邮政局长发送信号，通知全局进程表已更新。To spawn a parallel worker, the psql backend will initialize a parallel worker context in the global process table and a message queue based on shared memory for communication with the backend. Then it sends a signal to postmaster to notify that the global process table has been updated.

当邮局主管收到信号时，它将加载全局进程表，并发现需要生成一个新的并行工作程序。它将根据提供的上下文信息继续fork一个新的并行工作程序。此信息确定并行工作程序的入口点，以及生成后要执行的操作。在处理过程中，并行工作程序和psql后端使用消息队列来回传递元组，最后psql后端将收集所有数据元组并向用户返回最终结果。When postmaster receives the signal, it will load the global process table and found that it needs to spawn a new parallel worker. It will proceed to fork a new parallel worker according to the context information supplied. This information determines the entry point for the parallel worker and what to do once spawned. During processing, the parallel worker and the psql backend use the message queue to communicate tuples back and forth and finally the psql backend will gather together all the data tuples and produce a final result back to the user.

4. Can We Use GDB to attach This Parallel Worker’s PID When Spawned?

技术上是的，但这个并行工作程序的生命周期可能很短，当您从ps-ef命令中看到它的PID时，该工作程序可能已经完成了它的工作并退出。这意味着，对我来说，启动GDB并附加到它的PID太晚了。相反，我今天要向您展示的技术将从并行工作程序开始的那一刻开始跟踪它。
Technically yes, but the life time of this parallel worker may be very short, by the time you see its PID from the ps -ef command, the worker may have already done its job and exited. This means, it is too late for me to start GDB and attach to its PID. Instead, the technique I am going to show you today will trace the parallel worker from the moment it starts.

5. Tracing the Parallel Worker

I will be using this instance of PostgreSQL server (version 12.5) as an example where PID 11976 is the psql backend process serving the psql client.

Pre-Condition: Connect psql to the PostgreSQL server, create an example table and inserted about 2.5M rows of data. This will for sure trigger parallel workers.

$ psql -d postgres -U postgres -p 6660
psql (12.5)
Type "help" for help.
postgres=# create table test(a int, b int);
CREATE TABLE
postgres=# insert into test values(generate_series(1,500000),1);
INSERT 0 500000
postgres=# insert into test values(generate_series(1,500000),1);
INSERT 0 500000
postgres=# insert into test values(generate_series(1,500000),1);
INSERT 0 500000
postgres=# insert into test values(generate_series(1,500000),1);
INSERT 0 500000
postgres=# insert into test values(generate_series(1,500000),1);
INSERT 0 500000
postgres=#

Step 1: Attach GDB to the psql backend having PID = 11976 and Set a Break Point
I am setting a break point at the function RegisterDynamicBackgroundWorker. This is called when parallel worker is required to complete the query. Setting a breakpoint allows us more control as to when to proceed with a parallel worker spawn. 我正在设置RegisterDynamicBackgroundWorker函数的断点。当需要并行工作程序来完成查询时，将调用此函数。设置断点可以让我们更好地控制何时继续并行工作进程派生。

gdb postgres
(gdb) attach 11976
(gdb) b RegisterDynamicBackgroundWorker

Step 2: Attach GDB to the Postmaster having PID = 11959 and Set 2 Break Points
We are using a second GDB to attach to the postmaster and set 2 break points there. fork_process is the function before postmaster actually spawns a new parallel worker using the system fork() call. ParallelWorkerMain is the main function for the parallel worker after it has been spawned. 我们正在使用第二个GDB连接到postmaster，并在那里设置两个断点。fork_process是postmaster使用系统fork调用实际生成新的并行工作程序之前的函数。ParallelWorkerMain是并行工作程序生成后的主要函数。

gdb postgres
(gdb) attach 11959
(gdb) b fork_process
(gdb) b ParallelWorkerMain

Step 3: Execute a Large SELECT Query On psql To Trigger the Break Points
postgres=# select count(*) from test;
The RegisterDynamicBackgroundWorker break point will be hit on the first GDB session having attached PID = 11959. Use the continue or c GDB command to continue to spawn the worker. RegisterDynamicBackgroundWorker断点将在附加了PID=11959的第一个GDB会话上命中使用continue或c GDB命令继续派生工作程序

Breakpoint 1, RegisterDynamicBackgroundWorker (worker=0x7ffd867f3c80, handle=0x55a009b77388) at bgworker.c:1002
1002            bool            success = false;
(gdb) c
Continuing.

当您继续第一个GDB会话时，第二个GDB将由于收到SIGUSR1信号而暂停。这个信号告诉邮局主管重新加载全局进程表，然后生成一个并行工作程序。使用continue命令将命中fork_process处的第一个断点。As you continue the first GDB session, the second GDB session will pause due to receipt of a SIGUSR1 signal. This signal tells postmaster to reload the global process table and then spawn a parallel worker. Using the continue command will hit the first break point at fork_process

Program received signal SIGUSR1, User defined signal 1.
0x00007f301b97d0f7 in __GI___select (nfds=5, readfds=0x7ffd867f47d0, writefds=0x0, exceptfds=0x0, timeout=0x7ffd867f4740)at ../sysdeps/unix/sysv/linux/select.c:41
41      in ../sysdeps/unix/sysv/linux/select.c
(gdb) c
Continuing.Breakpoint 1, fork_process () at fork_process.c:47
47              fflush(stdout);(gdb)

You may use the continue command after setting it to follow child. Then immediately the GDB will switch to the new child process having PID = 12198 below and hit our second break point ParallelWorkerMain. So, Now the GDB is debugging the parallel worker process instead of the original postmaster. 您可以在将continue命令设置为跟随child之后使用它。然后，GDB将立即切换到PID＝12198以下的新子进程，并到达我们的第二个断点ParallelWorkerMain。所以，现在GDB正在调试并行工作进程，而不是原来的邮政局长。

(gdb) c
Continuing.
[New process 12198]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Switching to Thread 0x7f301ca79740 (LWP 12198)]Thread 2.1 "postgres" hit Breakpoint 2, ParallelWorkerMain (main_arg=1544458234) at parallel.c:1207
1207    {

Step 5: Continue To Debug The Parallel Process
Using the ps -ef | grep postgres command, we can see a new parallel worker being spawned having PID = 12198

At this point, you are free to explore the process flow of the parallel worker. For me, I am debugging the visibility issues, so I will set additional break points at HeapTupleSatisfiesMVCC and TransactionIdIsCurrentTransactionId. In your case, you may be debugging some other functionalities.Being able to debugging into a parallel worker with GDB allows me to see the problems I was having and being able to fix quickly.If you are having trouble tracing into a parallel workers spawned by PostgreSQL during run time, I hope this blog will be helpful to you. 此时，您可以自由探索并行工作程序的流程。对我来说，我正在调试可见性问题，因此我将在HeapTupleSatisfiesMVCC和TransactionIdIsCurrentTransactionId处设置其他断点。在您的情况下，您可能正在调试一些其他功能。能够用GDB调试成并行工作程序，可以让我看到我遇到的问题，并能够快速修复。如果您在运行时跟踪PostgreSQL生成的并行工作程序时遇到问题，我希望这篇博客对您有所帮助。

Cary is a Senior Software Developer in HighGo Software Canada with 8 years of industrial experience developing innovative software solutions in C/C++ in the field of smart grid & metering prior to joining HighGo. He holds a bachelor degree in Electrical Engineering from University of British Columnbia (UBC) in Vancouver in 2012 and has extensive hands-on experience in technologies such as: Advanced Networking, Network & Data security, Smart Metering Innovations, deployment management with Docker, Software Engineering Lifecycle, scalability, authentication, cryptography, PostgreSQL & non-relational database, web services, firewalls, embedded systems, RTOS, ARM, PKI, Cisco equipment, functional and Architecture Design. Cary是HighGo Software Canada的高级软件开发人员，在加入HighGo之前，在智能电网和计量领域拥有8年的C/C++开发创新软件解决方案的行业经验。他于2012年在温哥华不列颠哥伦比亚大学（UBC）获得电气工程学士学位，在以下技术方面拥有丰富的实践经验：高级网络、网络和数据安全、智能计量创新、Docker部署管理、软件工程生命周期、可扩展性、认证、加密，PostgreSQL和非关系数据库、web服务、防火墙、嵌入式系统、RTOS、ARM、PKI、Cisco设备、功能和架构设计。

https://www.highgo.ca/2020/11/07/how-to-analyze-a-postgresql-crash-dump-file/
https://www.highgo.ca/2021/07/09/using-gdb-to-trace-into-a-parallel-worker-spawned-by-postmaster-during-a-large-query/