I suppose it would be more helpful to give you 1861 data, rather than just tell you about reasons there might be age and birthplace differences. I like to be useful.
It's a pest to read, but I'll give it a try - it's really light. Banwell, district 2, page 27. Note this is not a proper transcript, I'm using my own abbreviations etc.

Harding's Lane
Family 139
John Harding, head, Mar, 48, Agr Lab, Banwell, Somerset
Ann do (ditto), wife, Mar, 54[that could say 34, too the line is right at the upper space line, looking at the other 3's and 5's, though, makes me think it's very probably a 5] Wedmore, Somerset
George Harding, son, Un, 22, Ag Lab, Banwell, Somerset

They have a boarder and a lodger. The lodger is a wid named James... something [might be Baker] aged.. I'm going to go with 74. He was born in ... Barrow[?] does that make sense? is there a Barrow or Berrow in Somerset? The boarder is Thomas King, age 6, born Banwell.

Family 140 is an Elizabeth Jones, Widow, 77 Tailoress
Family 141 appears to be lodging in Elizabeth's place.
William Gunningham, lodger, unm, 20, Lawyer [that's a young lawyer, isn't it? that 0 could be an off 9, there is a very tiny tail at the bottom, but it's very different from another obvious 9 on the page.]
James Webb, do, unm, 18, labourer, Banwell
Sidney Harding, do, unm, 22, do, do,
Thomas Harding, do, do, 19, do, do... [the do's are kind of catchy]
William Fisher, do, do, 20, do, do
John Davis, do, do, do, do, do

Looking back up over John and Ann I'm seeing that they are all part of another house containing Charles Thomas and family (he appears to be the main householder, or was first to the door, the James Price, or Rice family, and then your Hardings.

Contact me at genesaplenty at gmail dotcom if you have any questions.

Maer