What is this GWP (Google Wide Prifile) for ? It try to give answer to the following question :
What are the hottest processes, routines, or code regions?
How does performance differ across software versions?
Which locks are most contended?
Which processes are memory hogs?
Does a particular memory allocation scheme benefit a particular class of applications?
What is the cycles per instruction (CPI) for applications across platforms?
From paper, we can get some notes :
1. Data center monitor is different data center profiling tools, Monitor is coarse degree, and profiling tools is fine grained.
2. Sample at two demension : select part of machines and sample selected events
3. Result shows when sampling rate reach 0.01, we can get what we wanted, so higher sampling is not necessary.
4. Every machine has a lightweight daemon to collect machine and event info
5. It seems more than one kind of OS were installed in one datacenter, the reason maybe different app can get higher performance on some special kind of OS.
6. Profling lib act as a http server with hanlders for each type of profiler so what developer only need to do is link the profiling lib and remote client can get prifiling info from that service.
7. It seems symbol of online app process was stripped, but I think we can stripped job process but keep service process symbol.
8. Web interface can provide:
- qeury any part of data
- call graph and execution time
- source code align
9. The scale of GWP storage cluster will limit sample rate
10. Some interesting observation :
- zlib consume 5% cpu, so we can concentrate on zip optimization
- online coverage meaurement
- job scheduler optimization, some kind of job instance perform better on some spedical CPU architecture.arious
- Collect many demension, such as function name, compile, shared lib version, ... to support group aggregation to prifile performance of different unit, such as date center, compiler