Elayne denies that Galad is her brother. Thom was around at the right time. Thom clearly had something to do with Taringail's disappearance. Thom was Morgase's lover. In [TFOH: 19, Memories, 252], Morgase thinks about Taringail, and how the only good to come of the marriage were "two beautiful children." This could be Elayne and Gawyn, or she could mean Galad (adopted) and Gawyn.
Evidence from the Glossary: "A Royal Prince of Cairhien, he married Tigraine and fathered Galadedrid. When Tigraine disappeared and was declared dead, he married Morgase and fathered Elayne and Gawyn." [TEOTW: Glossary, entry "Damodred, Prince Taringail", 661]
In [TSR: 17, Deceptions, 194], Moiraine says Thom was "Morgase's lover for a time, after Taringail died" (emphasis mine). Here, Moiraine is trying to impress Thom with how much she knows. She wouldn't include a detail she had any doubt about. To be wrong about something in such a situation would indicate faulty research, and Moiraine would not risk showing any gap in her knowledge. If she had any doubt at all, she wouldn't have mentioned the timing of events. Plus, Thom didn't say or think anything to contradict Moiraine's statement.
If Morgase had been having an affair with Thom while still married to Taringail, it is very unlikely that she would have been able to keep it entirely under wraps; surely some rumor would have been around, especially in Cairhien. However, we see no evidence of such a rumor existing. Specifically, in [LOC: 50, Thorns, 628] when Rand mentions to some Cairhienin that he means to have Elayne rule Cairhien, the Cairhienin think she'd be a good choice, due to her descent from her father Taringail Damodred. If Morgase was unfaithful, surely there would have been some doubt as to Elayne's right to the throne of Cairhien.
At a post-ACOS signing [Dunwoody, GA, 9 October, 1996], RJ strongly denied that Thom was Elayne or Gawyn's father. "Thom is exactly who he says he is."