How to do? Solution is to move pdbs into a custimized directory and only leave those you use frequently in default pdb directory. You can write things like below into a bat file:
md X:\lib\debug\pdbs
move X:\lib\debug\*.pdb X:\lib\debug\pdbs
rem - edit the follwing lines to restore the PDBs for modules you debug in frequently
rem move X:\lib\debug\pdbs\*.pdb X:\lib\debug
And do corresponding change in VS Tools --> options.
Note here, keep on learning.