I say keep it as is, it creates "Party play".
BS- Come up with a better excuse. Party play doesnt happen because some one doesnt have haste or extended haste. It happens because monsters or areas are too tough to solo, you are a social person or an event occurs. Period.
I dont see any reason other than keeping the caster one more step above. Permahaste should be revisited.