The only James Clark* b.c. 1837 in Scotland with father Richard that I've been able to find on the 1841 is living at 58 Hadden Street, Woodside, Old Machar, Aberdeen (168/12/11)
Richard Clarke, 38, linen bleacher
Kynoch, 37
Eliza, 10, linen spinner
Margt, 8
James, 4
All born Aberdeenshire
The usual sources list them in a very large household but the original page shows them as seperate - the houses in this part of Aberdeen are of the tenement type and would have had many families living in one house.
I can't find this family on the 1851.
Gadget