IWF-1153: checking all S3 objects aginst Temporal

[ktrops]

Description

Checklist

• Code compiles correctly
• Tests for the changes have been added
• All tests passing
• This PR change is backwards-compatible
• This PR CONTAINS a (planned) breaking change (it is not backwards compatible)

Related Issue

Closes #issue_number

[codecov]

Codecov Report

❌ Patch coverage is 73.68421% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.70%. Comparing base (b357107) to head (6de36d0).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
service/interpreter/activityImpl.go	61.53%	4 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (73.68%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #598      +/-   ##
==========================================
+ Coverage   65.65%   65.70%   +0.04%     
==========================================
  Files          64       64              
  Lines        6464     6461       -3     
==========================================
+ Hits         4244     4245       +1     
+ Misses       1895     1893       -2     
+ Partials      325      323       -2     

Files with missing lines	Coverage Δ
service/interpreter/cadence/workflow.go	100.00% <100.00%> (ø)
service/interpreter/temporal/workflow.go	100.00% <100.00%> (ø)
service/interpreter/workflowImpl.go	86.81% <100.00%> (+0.03%)	⬆️
service/interpreter/activityImpl.go	78.20% <61.53%> (+0.73%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dillia23]

we're returning and throwing an error here if describe workflow fails because that means the workflow is likely no longer running?

[ktrops]

The other errors describe workflow can return besides NotFound error are:
// - serviceerror.InvalidArgument
// - serviceerror.Internal
// - serviceerror.Unavailable

Describe workflow is an expensive call and can take up lots of resources. So if we get any of the above errors, I think we'd want to fix the issue or wait before running it again.

[dillia23]

overall looks good to me! I would approve this if I could!