Finding Hidden Bottlenecks in Go Apps: A Lazy, Hacky, and Bruteforce Method

When developing pgrwl - a PostgreSQL WAL receiver - performance is a critical concern. Every part of the program must be predictable. There should be no hidden bottlenecks. But what about: typos that silently degrade performance? missing tests that fail to catch inefficiencies? slow logic introduced "just for now"? accidental O(n^2) behavior? These issues are often hard to detect. The Problem For instance, you may concatenate a huge template in a loop, but that may be done once outside of the loop. And this will work fine, until the heavy load is reveal that. Of course you should profile CPU/RAM, and there are a lot of great tools, but sometimes it's not enough. The Bruteforce Idea A lazy decision for measuring the whole picture is to to trace each function execution time, and the total number that function being called. Yeah, that cannot help you to inspect all the loops, nasty conditions, memory leaks, etc... But may help a LOT to find as fast as possible really heavily loaded functi