I fixed it. As I thought I mixed up the radiator brackets. The top bracket has the 120mm screw holes off-set. The front bracket has them centered. I put the front up top and the top in the front. After swapping them back to the right spot, the front fans are centered properly.
If you want to see pictures, the 570x radiator trays look the same as the ones in my case.
https://www.corsair.com/us/en/Categories/Products/Accessories-%7C-Parts/PC-Components/Cases/Crystal-570X-Front-Fan-Tray/p/CC-8900038
https://www.corsair.com/us/en/Categories/Products/Accessories-%7C-Parts/PC-Components/Cases/Crystal-570X-Top-Fan-Tray/p/CC-8900037