align deterministic models list for xpu (#169640)

Summary:
Fixes intel/torch-xpu-ops#1863
In this PR, we align the deterministic models list with cpu/cuda's for xpu devices.
For your information, models like  detectron2_fasterrcnn_r_50_fpn and detectron2_maskrcnn_r_50_c4 will encounter the OOM issue if using the deterministic roi_align implementation in https://github.com/pytorch/vision/blob/main/torchvision/ops/roi_align.py#L115.

X-link: pytorch/pytorch#169640
Approved by: https://github.com/jianyizh, https://github.com/EikanWang, https://github.com/guangyey, https://github.com/desertfire

Reviewed By: wdvr

Differential Revision: D91862770

fbshipit-source-id: e43188fabb61fe6ab1d3293d4fcdf3cd82cd240f

Author: generatedunixname499836121

userbenchmark/dynamo/dynamobench/common.py

@@ -3829,7 +3829,7 @@ def setup_determinism_for_accuracy_test(args):
         # some of the models do not support use_deterministic_algorithms
         torch.use_deterministic_algorithms(True)
 
-    if args.devices == ["rocm"] or args.devices == ["xpu"]:
+    if args.devices == ["rocm"]:
         torch.use_deterministic_algorithms(True, warn_only=True)
 
     torch.backends.cudnn.deterministic = True