Well, the 6 spd trans tend to be sluggish in shifting, and it takes more time to change gears.
A slow change is not a problem if you do normal driving without trying to shift the gears really fast. The 6 spd trans is also heavier.
For the guys who race, this will be a big problem.
Thus the reason the racers go for 5-spd