An SEC that is so overwhelming that they will soon get all of the 4 slots. Top eight conf champs, with some ability to replace the ones near the bottom with at larges, if certain conditions happen is a better way.
For example, you could have a rule that the committee ranked #2 always gets to take a slot. That makes sense to me, too, because the playoffs would lose credibility without it. Other rules could be that no one ranked over #25 gets in, if some at large is ranked higher, and so on.
After accommodating #1 and #2, the arguments for guaranteed inclusion get weaker and weaker, and are outweighed by the need to preserve a reasonable level of parity. A skipped #3 would only ever be the third best team in any league. I'd rather have a slot taken by Coastal Carolina than have a third SEC team included.