Hi Milkman,
Were the boxers in any kind of packaging? If they were, and you haven't opened the item, then you can go back to the store, and tell them that the item is unfit for its intended purpose, under the Sale Of Goods Act, and then the shop are legally obliged to refund or exchange the item.
The fact that the "intended purpose" being that the undies don't fit Mr Milkman isn't the issue. As far as the law is concerned, if any item is not fit for its intended usage or purpose, then you are legally entitled to a full refund or exchange.
Now, I'd politely ask that you return to the store, and ask to speak to the manager, and explain the situation to them. Also mention that all the boxer shorts were on the wrong hangers, and hence everyone who purchases these items will also have the wrong sized garments, and that they (the shop) could be potentially sued for false misrepresentation of goods!
Hopefully, that should all be enough to get them to exchange the offending undies for ones that are of the right size.
Also mention that if they fail to exchange the item in question, you will be contacting the store's Head Office to complain about the situation, and naming the manager of the store by name in your letter. PLus, you will be contacting the local Trading Standards offices too, to tell them how badly the store is treating you.
As long as you are firm, but fair, and willing to sound like you might actually be willing to undergo all of the "threats" you say you will do, then that should be enough to get the store to simply offer you an exchange or full refund. If you can do this at a busy time of day (lunchtimes, etc), then that also helps your chances, because the store will not want an angry customer being seen by lots of other customers, at one of the store's busiest times of the year! Nor would they like you to humiliate the store, over something so trivial, because it will make the store and the staff look like total numpties!
Let us know how you get on, but be prepared to actually complain to their head office, by demanding the address and phone number, plus the store manager's full name, in case they think you are simply trying to pull a fast one.

In other words, have the courage of your convictions!
Pooch