Turns out they hadn't built as much as I thought. Running through it again, found they quickly built another civ center when I destroyed the first one. Destroying 1 mill with ships didn't start them building, when my land forces destroyed their only other mill, they started building and completed one before I got to their main complex. At that time, all they had was 1 civ center, 1 Persian Palace, 1 Tower (2 more under construction), a Market, and a bunch of houses. Their gatherers acually worked right next to mine for some time before fighting broke out. When my guys started killing theirs, they started to fight back. They also started making more troops, which they hadn't been doing. Didn't send any organized attack, it stayed skirmishing between their gatherers and mine. Once they get stuck, it takes a lot to wake them up.