demolishing buildings

why oh why .. i cant destroy a building in my province ?
i know there is a deactivate button
but it really bugs me with the harbours
i really dont mind destroying it and spending another 10k lumber just to build a new harbour in the right place .

i know this feature could be used in a fishy way ( taking another nations proving . then destroy all the buildings if u know u cant hold the province ) but oh well ... thats what war is like :D

im sure others face the same problem as me O.o
or is there a destroy building button i dont know about ?