valgrind包含多个工具，通过--tool=xxx指定，最被大家熟知是memcheck，主要解决内存泄露，越界访问，未初始化却去引用等问题，它是默认选项，如果未指定--tool，默认就是memcheck了。而在多线程编程中，最常见的bug有：数据竞争（data race），死锁，错误的使用POSIX接口等问题。这些问题可以通过valgrind工具集里面的helgrind来探测。即：
valgrind --tool=helgrind yourprogram
-
利用诊断输出文件静态分析
一般探测出来的诊断信息会直接输出在终端上，但是有时候被诊断的程序可能也会输出信息到终端上，两者混淆在一起不方便阅读。这时可以通过参数--log-file=指定诊断日志文件来解决这个问题，诊断信息会输出到你指定的文件里面。
valgrind --tool=helgrind --log-file=./dump.log yourprogram
甚至还可以是fd, socket。分别由参数--log-fd=，--log-socket=指定。
诊断信息通常如下：
==11525== Possible data race during read of size 1 at 0x6532D40 by thread #3
==11525== Locks held: 1, at address 0x1FFEFFFCA0
==11525== at 0x5068856: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x1140C6: mqttclient_subscribe (mqttwrapper.c:1138)
==11525== by 0x1162CB: bar (test_mqtt.c:99)
==11525== by 0x4C36C26: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==11525== by 0x4E496DA: start_thread (pthread_create.c:463)
==11525== by 0x585DA3E: clone (clone.S:95)
==11525==
==11525== This conflicts with a previous write of size 1 by thread #6
==11525== Locks held: 1, at address 0x6532BA0
==11525== at 0x506948B: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x5068A65: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x5065D58: mosquitto_loop_read (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x50660D0: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x5066465: mosquitto_loop_forever (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x506B25A: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x4C36C26: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==11525== by 0x4E496DA: start_thread (pthread_create.c:463)
==11525== Address 0x6532d40 is 656 bytes inside a block of size 832 alloc'd
==11525== at 0x4C32F45: calloc (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==11525== by 0x50650CE: mosquitto_new (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x11271C: client_create (mqttwrapper.c:239)
==11525== by 0x1139B6: mqttclient_create (mqttwrapper.c:837)
==11525== by 0x1160F1: mqtt_init (test_mqtt.c:32)
==11525== by 0x1162B3: bar (test_mqtt.c:97)
==11525== by 0x4C36C26: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==11525== by 0x4E496DA: start_thread (pthread_create.c:463)
==11525== by 0x585DA3E: clone (clone.S:95)
==11525== Block was alloc'd by thread #3
如上面的信息所示：
==11525== Possible data race during read of size 1 at 0x6532D40 by thread #3
==11525== Locks held: 1, at address 0x1FFEFFFCA0
==11525== at 0x5068856: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x1140C6: mqttclient_subscribe (mqttwrapper.c:1138)
==11525== by 0x1162CB: bar (test_mqtt.c:99)
==11525== by 0x4C36C26: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==11525== by 0x4E496DA: start_thread (pthread_create.c:463)
==11525== by 0x585DA3E: clone (clone.S:95)
这里输出了调用栈，也写明了dada race所操作的地址0x6532D40。但是没法判断具体是哪个变量，并且错误产生时也无法暂停程序，用gdb attach来分析。但是动态分析程序，像用gdb那样分析程序对分析问题是有极大帮助的。
-
使用gdb + valgrind动态分析
好在valgrind有一个叫vgdb的东西，可以运行一个gdb server，然后我们用gdb的 target remote来连接它，于是可以愉快的使用gdb动态分析问题，并且可以在问题产生时，立马停止程序，分析现场。
--vgdb=yes|no使用或关闭vgdb--vgdb-error=0探测到多少个错误就暂停程序，如果是0，则表示发现错误立即暂停。
具体命令如下：
valgrind --tool=helgrind --vgdb=yes --vgdb-error=0 ./test_mqtt
这时gdb server就暂停程序，等待gdb来连接才开始继续运行下去。另起一个终端，直接运行gdb，然后输入 target remote | vgdb , continue
thomas@thomas-virtual-machine:~$ gdb
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target remote | vgdb
(gdb) c
用gdb来连接gdbserver总不太方便，借助IDE可以更舒服的分析问题。推荐使用Clion这款IDE，详细设置见官方文档。稍微不同的是'target remote' args不是文档里面gdbserver ip : port的形式，而是| vgdb这种管道形式。然后就是设置你的符号文件，即你应用程序。