I've been told we came from Ireland or Scottland.

The documented info unfortunately stops at Wyatt C. I do know he was born and migrated from Brunswick Co. Va. . Early 1800 Virginia census of that area has a Wyatt Allen as the head of the house hold. Wyatt C was not born or a baby at this time. Brunswick Co. Va. married records has a Wyatt Allen married to Nancy Tarver in 1805. Nancy’s father was Andrew Tarver and her mother’s maiden name was Ann Young. From what I put together they had three children that I know of.
Robert Lunsford, Wyatt Carter and Andrew Young Allen. I need to verify this, but I recall on the same census a William and John Allen. My thought is William may be Wyatt Carters grand father because he named one of his sons William. It’s probably a stretch but some times it’s all you have to go by. He also named a son Robert and a daughter Ann.
I believe Wyatt and Nancy moved the Twiggs Co. Georgia and Wyatt C and Andrew moves to north florida. By 1850 Andrew was living in Lake city and Wyatt C purchased land outside of Lulu, Fl. The same property is still in the family today. I’m related to Wyatt through his son Allison Hamilton Allen. This is all I have but your inquiry has inspired me to continue my research.