dill 0.3.5.1 vs python 3.11 vs globus-compute-sdk 2.2.1

[benclifford]

Describe the bug
I think I have encountered this dill behaviour: uqfoundation/dill#514

Stack trace 1 below gives the traceback.

Following that issue I forcibly violated the globus-compute-sdk packaging requirements using this command line:

$ pip install 'dill>0.3.5.1'

and now get further in execution, to what looks like an execution failure remotely... (presumably because of incompatible dill or python versions between my submit side and the tutorial endpoint - so I think that means that execution proceeded further but still doesn't work. See stack trace 2 below.

To Reproduce
Install globus-compute-sdk 2.2.1 vs python 3.11.2. Run quickstart example.

STACK TRACE 1 - DILL 0.3.5.1 VS PYTHON 3.11.2

benc@parsl-dev-3-11-5704:~/parsl/src/parsl$ python3 parsl2funcx.py 
Unable to register function: add_func
Traceback (most recent call last):
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/serialize/facade.py", line 63, in serialize
    return strategy.serialize(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/serialize/concretes.py", line 137, in serialize
    x = codecs.encode(dill.dumps(data), "base64").decode()
                      ^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 364, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 336, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 620, in dump
    StockPickler.dump(self, obj)
  File "/usr/local/lib/python3.11/pickle.py", line 487, in dump
    self.save(obj)
  File "/usr/local/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 1140, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/usr/local/lib/python3.11/pickle.py", line 692, in save_reduce
    save(args)
  File "/usr/local/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pickle.py", line 902, in save_tuple
    save(element)
  File "/usr/local/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/dill/_dill.py", line 1187, in save_code
    obj.co_firstlineno, obj.co_lnotab, obj.co_endlinetable,
                                       ^^^^^^^^^^^^^^^^^^^
AttributeError: 'code' object has no attribute 'co_endlinetable'. Did you mean: 'co_linetable'?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl2funcx.py", line 15, in <module>
    future = gce.submit(add_func, 5, 10)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/executor.py", line 322, in submit
    self.register_function(fn)
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/executor.py", line 281, in register_function
    func_reg_id = self.funcx_client.register_function(fn, **reg_kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/login_manager/decorators.py", line 17, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/client.py", line 654, in register_function
    data = FunctionRegistrationData(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/web_client.py", line 44, in __init__
    function_code = _get_packed_code(function, serializer=serializer)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/sdk/web_client.py", line 26, in _get_packed_code
    return serializer.pack_buffers([serializer.serialize(func)])
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/globus_compute_sdk/serialize/facade.py", line 66, in serialize
    raise SerializationError(err_msg) from e
globus_compute_sdk.serialize.base.SerializationError: Code serialization strategy DillCode failed

STACK TRACE 2 - DILL 0.3.6 VS TUTORIAL ENDPOINT

benc@parsl-dev-3-11-5704:~/parsl/src/parsl$ python3 parsl2funcx.py 
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl2funcx.py", line 18, in <module>
    print(future.result())
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
globus_compute_sdk.errors.error_types.TaskExecutionFailed: 
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/facade.py", line 116, in unpack_and_deserialize
        deserialized = self.deserialize(current)
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/facade.py", line 67, in deserialize
        result = self.methods_for_code[header].deserialize(payload)
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/concretes.py", line 137, in deserialize
        function = dill.loads(codecs.decode(chomped.encode(), "base64"))
      File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 387, in loads
        return load(file, ignore, **kwds)
      File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 373, in load
        return Unpickler(file, ignore=ignore, **kwds).load()
      File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 646, in load
        obj = StockUnpickler.load(self)
      File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 805, in _create_code
        return CodeType(args[0], 0, 0, *args[1:])
    TypeError: code expected at most 16 arguments, got 21

Expected behavior
The quickstart tutorial should proceed as documented

Environment
my laptop, in containerised debian-ish dev environment

• Python version @ client: custom built python 3.11.2
• globus-compute-sdk version @ client: 2.2.1
• Python version @ endpoint: tutorial endpoint:
• globus-compute-endpoint version @ endpoint: tutorial endpoint:

Distributed Environment

• Where are you running the funcX script from? [e.g. Laptop/Workstation, Login node, Compute node]
  laptop
• Where does the endpoint run? [e.g. Laptop/Workstation, Login node]
  globus hq
• What is your endpoint-uuid?
  4b116d3c-1703-4f8f-9f6f-39921e5864df
• Attach endpoint logs at ~/.globus_compute/<ENDPOINT_NAME> if this is an endpoint issue.
  globus hq should have these alreayd

[benclifford]

(this may be a separate issue, or not... but it follows on from the above narrative)

I reset dill to 0.3.5.1

I tried using the CombinedCode serializer like this:

from globus_compute_sdk.serialize import CombinedCode
from globus_compute_sdk import Client

c= Client(code_serialization_strategy=CombinedCode())
from globus_compute_sdk import Executor

def add_func(a, b):
    return a + b

tutorial_endpoint_id = '4b116d3c-1703-4f8f-9f6f-39921e5864df' # Public tutorial endpoint
# ... then create the executor, ...
with Executor(endpoint_id=tutorial_endpoint_id, funcx_client=c) as gce:
    # ... then submit for execution, ...
    future = gce.submit(add_func, 5, 10)

    # ... and finally, wait for the result
    print(future.result())

and receive this AssertionError (at the tutorial endpoint, I think?)

benc@parsl-dev-3-11-5704:~/parsl/src/parsl$ !pyt
python3 parsl2funcx.py 
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl2funcx.py", line 18, in <module>
    print(future.result())
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
globus_compute_sdk.errors.error_types.TaskExecutionFailed: 
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/facade.py", line 116, in unpack_and_deserialize
        deserialized = self.deserialize(current)
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/facade.py", line 67, in deserialize
        result = self.methods_for_code[header].deserialize(payload)
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/concretes.py", line 182, in deserialize
        for serial_id, encoded_func in self.get_multiple_payloads(payload):
      File "/usr/local/lib/python3.10/site-packages/globus_compute_sdk/serialize/concretes.py", line 155, in get_multiple_payloads
        assert len(parts) == 2 * len(COMBINED_SERIALIZE_METHODS)
    AssertionError

[benclifford]

With DillCodeSource as a serializer, I'm able to get a successful function execution in this environment.

[kommav]

I have also experienced this:
GlobusCompute Versions:
globus-compute-common 0.2.0
globus-compute-sdk 2.2.3
globus-sdk 3.23.0
MacOS Version: 12.5.1
Python Version: 3.11.4