Parallelized or Paralyzed? The Essential High Performance Debate