A marriage at age 35 isn’t unreasonable and could also explain why Edward and Mary had just four children.
Taking the three census returns 1841 – 1861 gives an average birth year for Edward of 1789. Doing the same for Mary over four censuses gives her an average birth year of 1790. If as you say, Edward was born in 1803 then either he married Mary, 13 years his senior or four census returns were consistently 13 – 14 years in error.
Baptism showing for Edward Price on 21 Dec 1803, is this the one you looked at?
St Nicholas, Hereford and at Weybridge, Hereford (non-conformist) Parents John and Margaret