In 1851 you have John Richardson, aged 48, in Aston near Birmingham with wife Martha and children George, Jean, John, James, Margaret, Catherine and Thomas. In 1861 he says he was 58.
Assuming that his age is accurate, this means that he was more likely born in 1802 than in 1803, because the census was held at the end of March, and would only have had his 48th birthday in 1851, and his 58th birthday in 1861, if he was born between 1 January and late March.
There are no surviving records of the birth of a John Richardson, father George, between 1801 and 1804. There is one in Edinburgh St Cuthberts in 1800, and one in Coldingham in 1805. There could of course be others whose baptism records have not survived, and these could include your one.
Unfortunately I can't find him in a later census to see if he gave a more precise place of birth than just 'Scotland', or if he was being visited by a sibling, niece or nephew who might provide a clue.
