What if I put my entire $5M fund into an SPV and operate off of that?

This is a sub-optimal approach for building an during VC firm. VCs are setup with 3 entity structures and it's highly recommended not deviate from this approach. LPs will not count the SPV as fund 1. Therefore, on the next fund, the GP will face the same fundraising challenges of all other new managers on Fund 1.