James Waller and Elizabeth Ellis of Hancock County, GA did have a son, James Bozeman Waller born February 05, 1802. This James Waller born December 15, 1768 in Delaware or Maryland was the son of Joseph Waller and may be the brother of Joseph E. Waller of Delaware and then eventually Louisiana.
From "The Georgians" by J. H. Austin, Joseph Waller Will and LDS.