I can't speak for all fans, but for me, yes. If that makes me a bad fan, oh well. I don't care a single bit about ULL, ULM, MTSU, et al. If you asked the majority of casual college football fans what division those schools were in, they'd say "D2". And they'd be wrong on several levels, but it speaks to the perception of the league.
So yeah, I'd much rather play Tulsa, Rice, and UTEP than anyone in the Sunbelt, times infinity. No offense.
And I understand the head to head records of the Sunbelt schools vs other conferences and all that, and I get that the league is improving, but literally no one outside the fans of schools that are members of this conference cares about any of that.