David Lee is supposedly my 5th great grandfather but I am still researching it.

this is what I don't understand,
How can david be my 5th great grandfather?
He was born in Egg Harbor, NJ
Married Hester Smith in NJ
from what I can see, all of his children were also born in NJ.

My 4th Great Grandfather James was born in Virginia.
Emma, my 3rd great grandfather, was living with Furman Garrison, their son Frank, and little Kate who was just 13 when the twins (George and James) walked to Jersey from Virginia to stay with them after the cival war.

I see there are several people who have James Sr listed as David's son, but the siblings before and after were all born in Jersey. It doesn't add up..