Linux system level monitoring

htop, sar, iostat, or vmstat do help, but they only give you the 10,000 foot overview of resource usage. They don't allow you to see what part of which program or process is eating up too much RAM or which part is creating race conditions on lock files.

A much more powerful took is "Atop", a powerful monitor program that allows you to see system-level counters concerning utilization of CPU and memory/swap, as well as see disk I/O and network utilization counters at the system level -- in real time or historically

Modern tools like DTrace provide insight into almost every aspect of system behavior