There are quite a lot of James Rice's born in Waddesdon on Ancestry. Don't know if you have this info or not. The three sons you mention on the 1841 living with James are Joseph who was 14, John who was 12 and James who was 10. He would have been the youngest child born to Ann around about 1831-might be rounded up I suppose being the 1841. In 1851 it looks as if James 1831 was living as a lodger aged 19 years.
There is also a James Rice born 1782 Waddesdon on the 1861, could this have been his father, husband of Elizabeth?
There is also a James Rice born 1786 Waddesdon aged 55 years on the 1841, wife Ann born 1786, and son Philip born 1811-might be a relative!
regards.