What are the downsides of raising under 506c (in public)?

You/your fund admin, will need to re-verify LPs after a certain time period to ensure they are accredited investors. Very large LPs are not going to be comfortable disclosing their financial or tax information and it's very likely that they will pass on investing in your fund.