I think I
may have found him......or a clue, at least.

I didn't find it at first, as it was under WITNEY surname.
Chinnor baptism - William
Witney-
Folly. Born 30 Nov 1847. Baptised - 23/4/1848. Son of
Abraham(labourer) &
Letitia(Lacemaker).
Abraham Witney c1821. Stokenchurch, Oxfordshire, is quite easy to trace in census. He's in Oxford County Jail in 1851.
Letitia/Leticia Folley c1821. Stokenchurch, Oxfordshire, is a bit harder. 1841 census is ok, 1851 census, she is down as Seticca Folley 1821. Stokenchurch.
Letitia Folley was baptised 12 Aug 1821 @ Stokenchurch, Oxfordshire. Daughter of William & Susannah. (as per 1841 census).
If you look in 1851, there is a WILLIAM FOLLEY aged 3 years old. born Stokenchurch. They are living with Richard Folley c1825, Stokenchurch. (He is Letitia's brother.). Richard was living with possible Grandparents in 1841.

There is a marriage for Leticia Folley in j/a/s - 1856. Wycombe district. Vol 3a. P.515. I think she married Edmund Steptoe, @ St. Peter & St. Paul, Stokenchurch.
In 1861 census, Leticia Steptoe c1819. Stokenchurch, has a son Thomas Steptoe c1847. Saunderton, Bucks. I think this maybe William Folley/Witney.
Even though Stokenchurch is in Oxfordshire, civil registration transferred to Buckinghamshire.....(I cannot exactly remember when.).
I would also hazard a guess, that William Folley/Witney, was born elsewhere(Saunderton), as an illegitimate child of Leticia Folley, but Abraham Witney did not marry her.
I have no explanation as to why he gives his father as John on his marriage details, but it's not unheard of to fabricate a father from thin air, or stretch the truth.
If you buy his birth cert, I would
guess he was born @ Saunderton, Bucks, Workhouse in 1847.
I hope you can follow this.

Steve.
Info from OxFHS- Chinnor/Stokenchurch PR's CD. And BucksFHS - Church finder facility.