I've found discrepancies in adult ages can be significant. Sometimes you can make sense of it through age mother would be when she gave birth. Since we will hope that Mary wasn't 12 when she gave birth to James, you can assume that her age was wildly off in the 1841 census.
The other ages line up well. Seem like they had a boarder in 1841 and a young servant living with them in 1851.
(I always imagine the census taker asking the youngest and least reliable child in the family how old everyone is. Most kids have no idea how old their parents are and '30' could sound as old to them as '40'.)