I'm not sure why I didn't think to try this before, but I decided to create a new user to try to run it there, and it works.
Also, on a hunch I logged out of XFCE, stopped the lightdm service, and started the X server with "startxfce4" in my ~/.xinitrc, and it worked, so it might have something to do with lightdm.
Update: Ok, so I tried it with SDDM, and it still won't start. I'm going to try to build it from source and use gdm to see if I can find some useful information.