It's a least part of the reason i think. If one were passionate enough i am quite certain there would be a metric ton of crap that could be ripped out while having the result still be capable of booting into an useful environment - even a graphical one. It just depends on which kind of graphical ui (aka desktop) is desired. Gnome for example is already dead once you cut systemd (likely the first thing to go if you were aiming for minimalism and very likely a huge offender in relation to requiring kernel bloat on top of that). Messing with the kernel is not needed for that at all.