From the moment she stepped into the spotlight, Beyoncé has been the center of her fair share of rumors.

But even PEOPLE’s 2012 Most Beautiful Woman was caught off guard when, while pregnant with her first child, whispers started swirling that she and husband Jay-Z had settled on a surrogate to carry their daughter.

“That was crazy. It wasn’t hurtful, it was just crazy,” the songstress, 30, tells PEOPLE. “[I thought] ‘Where did they come up with this?’”

But perhaps the person most affected by the surrogacy speculation was not the expectant mom herself, but rather her own mother, Tina Knowles.

“I thought it was very unfair and very cruel that someone would think that someone would be that diabolical to keep up a charade like that for nine months,” she says. ”As a mother it was painful for me to hear the crazy rumors. And I even had people ask me, which was so ridiculous.”

Alluding to an Australian interview in which it was later suggested that Beyoncé had been sporting a prosthetic bump, Knowles says, “It was a fabric that folded — does fabric not fold? Oh my gosh, so stupid.”

Read More: People Magazine: Beyoncé: Surrogacy Rumors Were ‘Just Crazy’