When we cant find the obvious we need to look at the not so obvious , there could be many reasons unfortunately. Gray is very common can be mis spelt ie. Grey. None of the family may have been born where the 1851 census states, that was very common, I notice that Emma is 27 and Charles looks to be 37 ,10 years her senior, she could have been widowed young, and the children may not have been born as Gray, but as they were so young they went down under the step fathers name.
Also deaths as I have already suggested, life span could be short then, or remarriage for Emma, I have looked for that but can find none.

I am not suggesting that any of the above is the case but all points need to be thought about carefully and persued.
I will continue to look around to see if I can come up with anything.