1841 census
Dundee, Hospital Park No 7
James Milne, aged 30, a porter with Elizabeth, also 30 plus David, 7, Margaret, 5 and Ann, 3 months
1851 census
Liff and Benvie, 19 Blackness Road
James Mill, aged 41, a general labourer, born Dundee, with Elizabeth, aged 38 born Errol, Perthshire with David, 17, Margaret, 15, Catherine, 6, James, 4 and James' father, David, (?not William as you state?) a mason's labourer, born Glamis in 1788
Have found David Milne on the 1841 census, also in Hospital Park, Dundee, a carter, living alone. I couldn't trace him on the 1861 census.
(This info taken from Ancestry so check originals on Scotland's People to be sure ... sometimes Ancestry transcriptions not the best!)