Coming from a gameplay perspective, I would say a big issue with naval battles (other than their size/pathfinding troubles) is garrisoning for effectiveness. Overall, this makes boats essentially siege towers, which isn't very fun. Also, the firepower of each boat depends on garrisoning which introduces probability into boat fights. Garrisoning individual boats for effectiveness is tedious (made worse by size and pathfinding). After doing this, you then have much less pop to fight with on land.
I think ships should be dedicated fighters with garrisoning only used for transportation. They should use unitAI instead of building AI. From here, you can introduce ship classes: light (or scout), medium, siege, and special. Instead of bigger = better, they should have different ranges, speeds, and unit characteristics. Fireships can go in special, along with new additions to this category mentioned above. Some kind of capturing ship would be good, but it should be clear it is for capturing so you can avoid it.
I will put together something like this in the community mod after a27 is released, and once those are play tested we could try adding new abilities like ramming, special ships, technologies etc.