The expression comes from the heyday of sailing ships, all war ships carried iron cannons, those cannons fired round iron cannon balls, it was necessary to keep a good supply near the cannon, but to prevent them from rolling about the deck, the best storage method was to make a square-based pyramid with sixteen balls at the bottom building up to one at the top. But the problem was how to prevent the bottom layer from sliding or rolling from under the others. So they devised a metal plate called a "Monkey" with 16 round indentations.
The plate was made of iron so the iron balls would quickly rust to it. The solution to the rusting was to make "Brass Monkeys", but brass contracts much more and much faster than iron when cold. So when the temperature dropped too much, the brass indentations would shrink so much that the cannonballs would come right off the monkey. So this is were the quote, "Cold enough to freeze the balls off a brass monkey" comes from
I live about 3 miles from Historic Dockyard in Chatham