Similar names on 1860/1870 in New York but occupation differs. ?
1860 census Brooklyn Ward 14 District 1, Kings, New York
There is a Patrick Clifford born 1810 Ireland with a wife Ellen and children James Clifford; Catherine Clifford; Patrick Clifford.  However Patrick is a tailor.
Patrick Clifford   50 Tailor
Ellen Clifford   45
Catherine Clifford 8
Patrick Clifford   1
James Clifford
and same family in 1870  Brooklyn Ward 14, Kings, New York.
Pat Clifford   60 Tailor
Ellen Clifford   52
Phillip Clifford   30
James Clifford   23
Cath Clifford   18
Pat Clifford   11
Sandra
Added Ancestry tree shows :-
Patrick Clifford 1807–1882
birth ABT 1807  EnnisKillen, Fermanagh
died 21 NOV 1882 -  New York City, New York.
Ellen Clifford 1816–1874
birth 1816 - Ireland
died 1874 -  Brooklyn, Kings County, New York