Hello,
You need to find the right balance.
Generally, it is advisable to have a large population before advancing to the next age; however, if you wait too long, you will fall behind in technology.
That said, a larger population usually means gathering more resources, so you should have enough to advance (it isn't that expensive).
Best regards,