retrieval issues: TreeHash doesn't match #115
Description
In retrieving files from glacier using mtglacier, we often get an error that the TreeHash doesn't match. Our job is configured to retry the retrieval when it fails, so it may get this error 10 or 15 times, and then finally a retrieval will succeed. This happens about once every dozen or so retrieval requests.
The offset in the file where the TreeHash mismatch occurs is in a different location on each failure, and since the file eventually is restored it appears to not be corrupt at glacier. So, something is wrong with the transfer or the TreeHash calculation routine.
ERROR (child 26241): TreeHash for received segment of file "/data/glacier-stg/86346" (position 2550136832, size 134217728) does not match. TreeHash reported by server af0b2467868b783ab298faa03f58133cdff7b88309c0fcb928bae650420d6a8f, Calculated TreeHash bfed84480067e86258b3e14026a8f0d2362286459e31389674b632af9b66348e