The best way of building a very compact raspbian would be to start from debootstrap. (There are some tutorials on this around the net).
If removing stuff, you could remove xorg /xserver and related packages – anything that relies on it would then get removed too. You can also remove many libraries, icons / fonts etc. check out /usr/lib / usr/share and do “du -sh | sort -h” and you will get a sorted listing of what uses what. You can then google what stuff is and work out if it would be ok to remove.
Or just start with my minimal retro pie and go from there :)