According to contemporary chroniclers Pope Leo IX did not give a reason. But the church’s objections would typically be based on consanguinity, affinity or spiritual affinity. Consanguinity is being related by blood within seven prohibited degrees at that time, affinity is relationship through a marriage and followed the same rules of consanguinity while spiritual affinity related to God-parentage. For centuries historians have believed his reasons to be based on consanguinity and that William and Matilda were related through a common ancestor. As it turns out their closest blood relationship is that they were third cousins (once removed). But W.H. Blaauw, in ‘Remarks on Matilda, Queen of William the Conqueror and Her Daughter Gundrada,' Archaeologia, Vol. 32, 1847, p. 110 pointed out that Matilda’s mother, Adela of France had been previously married to Richard III, 5th Duke of Normandy, William’s uncle. And although the marriage had probably not been consummated, under canon law a second-degree affinity existed between William and Matilda, which is much closer than third cousins and an additional impediment to the marriage.
Besides the affinity information, Blaauw’s article also joins several others commenting on the parentage of Gundrada, wife of William de Warenne, 1st Earl of Surrey. Orderic Vitalis in his Ecclesiastical History stated that Gundrada was the sister of Gerbod the Fleming, Earl of Chester (who in turn was the son of another Gerbod, the Advocate of St. Bertin). The Hyde Chronicle confirms this. Sometime during the sixteenth century, however, a copy of a charter of Lewes Priory was found with an inserted passage leading many to assume Gundrada was a daughter of William the Conqueror and Matilda of Flanders. Then in the 1840s Stapleton theorized that Gundrada was not a daughter of the Conqueror but rather of Matilda of Flanders by an earlier marriage. Further, that this was the reason for the papal ban on the marriage between William the Conqueror and Matilda—that Matilda had a still living husband. By the 1880s several historians had discredited both theories. These included Waters, Clay, and Freeman as well as Blaauw. There is much more detail to the discussion of Gundrada but as she was not the Conqueror’s daughter she could not have been the reason that his marriage to Matilda of Flanders was banned.
Because the marriage between William and Matilda did take place, by most accounts about 1053, defying the pope, all of Normandy came under papal interdiction that wasn’t lifted for several years. William and Matilda finally obtained a dispensation from pope Nicholas II in 1059 which act legalized the marriage and legitimized their children born during the interdict. I’ve yet to see any chronicler or later historian refer to their children as ever being illegitimate though.
I hope this helps.