But depending on the problem CPUs aren’t as bad as you indicate. It is almost the same as with GPUs, some algorithms fit the architecture and some not. Generally speaking the probability that a given problem tanks on a GPU (obtained performance <10% of peak) is a lot higher than on a CPU.

Just as an example, I have implemented an embarrassingly parallel (perfectly suited to GPUs) algorithm using double precision for CPUs as well as for ATI GPUs. It runs with about 40% (Core2, K10) to 60% (Core i7, HT helps a lot here) of theoretical peak performance on CPUs and 62% of theoretical peak on ATI GPUs (150GFlops sustained on a HD4870, peak would be 240GFlops).
But because it is not a 50:50 ratio of multiplication and additions (or MADs for ATI) it is actually quite close to the maximal obtainable performance for this algorithm. There are virtually no stalls on ATI GPUs, it really executes one instruction per unit and clock cycle more than 95% of the time. The Core i7 is actually not that far behind. The hyperthreading like execution of the parallel tasks on the GPU really helps to avoid stalls and boosts the utilization (same is true for the i7) in such cases.

And if you wonder what algorithm I am talking about, it’s Milkyway@home.