Neither shall he go out of the sanctuary, nor profane the sanctuary of his God; for the crown of the anointing oil of his God is upon him: I am the Lord. (Leviticus 21:12)
The common priests each served only for a short time every year; the high priest was on continuous duty. He must be available at all times, hence could not go on a journey. One priest could officiate for another in case of emergency, but no one could officiate for the high priest, though in later times he did have a substitute.