So I looked into the protocols that are used in vending machines a bit more. It turns out that the MDB / ICP (Multi Drop Bus / Internal Communication Protocol) which is a simple serial protocol. This is freely available from the NAMA (National Automatic Merchandising Association) website http://www.vending.org/technology/MDB_Version_4.pdf
I'll draw your attention to the attached page from the earlier version of the protocol. It shows what happens when a vend fails.
VEND REQUEST contains the amount that the item costs, and it is up to the reader to decide if it should allow the vend. If it does, it sends a VEND APPROVED with the amount it had deducted.
The VEND FAILURE doesn't have any information about how much to refund though – it's entirely in the hands of the card reader. This goes against what I thought in the earlier post, and points to there being a problem with the card reader itself. Unfortunately, there isn't much information about the reader – I'm fairly sure it is a rebadged version of an OEM reader but have yet to find it.
Another interesting thing is that the reader doesn't appear to have any connection to a network – any data offloaded needs to come through an infrared data port on the front. This uses the EMA CVS spec http://www.vending-europe.eu/en/standards/eva-cvs.html – which isn't truely free. Let's see if they give me access!