Hello Claire, Welcome to Rootschat!

I have found the entry (set below), It seems that the whole family are classed as INDEPENDENT. They do have a William Stiffe aged 50 with them born c.1791 & an Elizabeth Stiffe born c. 1806 ~ the No means none were born in Glamorgan
Glamorgan Family History Society 1841 Census CD
District Name: Swansea Enumeration District: 6 Swansea Town Piece: 1426/03 Folio: 043 Schedule: 138
Address: PICTON PLACE
S 33 043 138 PICTON PLACE
STIFFE WILLIAM 50 (c1791)
INDEPENDENT NO
S 33 STIFFE ELIZABETH EVERITT 35 (c1806)
INDEPENDENT NO
S 33 STIFFE FRANCIS W.E. 11 (c1830)
INDEPENDENT NO
S 33 044 STIFFE ARTHUR WILLIAM 10 (c1831)
INDEPENDENT NO
S 33 STIFFE FLORENCE ANNE 9 (c1832)
INDEPENDENT NO
S 33 STIFFE SYDNEY BURROWS 7 (c1834)
- NO
S 33 STIFFE MADELINE ROSE 5 (c1836)
- NO
Many people moved to Swansea simply for employment, Swansea had booming industries many people came from Bristol, Devon & Ireland to live in Swansea.
The only Stiff family (no Stiffe) listed in 1861 are these in Cardiff, I have listed them simply due to the fact that they are from Gloucestershire:
Glamorgan Family History Society 1861 Census CD
4101 1861 CENSUS OF SWANSEA LOWER. REF.R.G.9/4101, E.D.2.
Piece: 4101 Folio: 047 Schedule: 226
Address: FROG STREET 4
STIFF EDWARD LG U 24 SEAMAN MERCHANT SERVICE
GLS IRON ACTON (c1837)
The year on this entry above should be c.1813 not c.1837 (probably a census error???)
I also found a death of an Edward Stiff in Swansea on the Free BMD website:
Surname First name(s) Age District Vol Page
Deaths Dec 1867
Stiff Edward 32 Swansea 11a 331 this one would have been born around 1835
Glamorgan Family History Society 1861 Census CD
4035 1861 CENSUS OF CARDIFF ST.MARY. REF.R.G.9/4035, E.D.14.Piece: 4035 Folio: 105 Schedule: 300
Address: AUGUSTA STREET 50
STIFF JOHN HD M 44 BLACKSMITH
GLS IRON ACTON (c1817)
STIFF CAROLINE WI M 40 -
GLS IRON ACTON (c1821)
STIFF JULIA L. DA U 9M -
GLA CARDIFF (c1860)
PURNELL LEONARD LG U 24 CARPENTER
SOM CHILCOMPTON (c1837)
Both John and Caroline are still living in Cardiff in 1881:
Dwelling: 10 Silver St
Census Place: Roath, Glamorgan, Wales
Source: FHL Film 1342271 PRO Ref RG11 Piece 5285 Folio 104 Page 45
Marr Age Sex Birthplace
John STIFF M 65 M Ironacton, Gloucester, England
Rel: Head
Occ: Smiths Helper
Caroline STIFF M 60 F Ironacton, Gloucester, England
Rel: Wife
Occ: Smiths Helper Wife