It doesn't really matter if it's in someone's home, near someone's home, their farm, their shop, their town, or anything else they've built.
There is nothing that is free in minetown, unless you have a screenshot of the owner saying it was free. If it is not natural-generated terrain and you did not build it, then it is not yours. Destroying or stealing from it is against the rules, unless someone explicitly gave you permission and asked you to help build/tear it down.
This was a decent-sized grief/theft; I'll unban you 24 hours from your response to this, and you will need to serve 3 hours in-game jail time afterwards, but I would not expect similar leniency from myself or another mod if something like this happens again. The diamond blocks have been removed from your inventory and returned to their owner.