What is this file: bn_17.sn_360.epoch_9.ckpt.data-00000-of-00001

[Arkadiy-Garber]

I am downloading the trained models, but I am not getting any files ending with .ckpt.

Here are the contents of one of the directories:

bn_17.sn_360.epoch_9.ckpt.data-00000-of-00001 

bn_17.sn_360.epoch_9.ckpt.index 

bn_17.sn_360.epoch_9.ckpt.meta 

checkpoint

Which of these files should i provide to the --model_path command?

Traceback (most recent call last):
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 255, in _fast5s_q_to_pred_str_q
    saver.restore(sess, model_path)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file /data/MAB/bin/deepsignal/models/model.CpG.R9.4_1D.human_hx1.bn17.sn360.v0.1.7+/bn_17.sn_360.epoch_9.ckpt.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]

Caused by op 'save/RestoreV2', defined at:
  File "/data/MAB/miniconda3/envs/deepsignalenv/bin/deepsignal", line 33, in <module>
    sys.exit(load_entry_point('deepsignal==0.2.0', 'console_scripts', 'deepsignal')())
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 424, in main
    args.func(args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 88, in main_call_mods
    f5_args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 448, in call_mods
    is_rnn, is_base, is_cnn)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 326, in _call_mods_from_fast5s_cpu
    p.start()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 254, in _fast5s_q_to_pred_str_q
    saver = tf.train.Saver()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal
    restore_sequentially, reshape)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

DataLossError (see above for traceback): Unable to open table file /data/MAB/bin/deepsignal/models/model.CpG.R9.4_1D.human_hx1.bn17.sn360.v0.1.7+/bn_17.sn_360.epoch_9.ckpt.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]

write process 1316634 ending
0 of 110340 fast5 files failed..
call_mods costs 29.43 seconds..

[PengNi]

@Arkadiy-Garber , hi, just use the prefix /path/to/bn_17.sn_360.epoch_9.ckpt.

[Arkadiy-Garber]

Thanks, that seemed to worked to fix the previous issue. However, I am not getting the following output at then end of the stdout (with the command):

deepsignal call_mods --input_path single_barcode12/ --model_path /data/MAB/bin/deepsignal/models/model.GATC.R9_2D.tem.puc19.bn17.sn360/bn_17.sn_360.epoch_5.ckpt --result_file single_barcode12.gatc.call_mods.tsv --corrected_group RawGenomeCorrected_001 --nproc 10 --is_gpu no

/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/sklearn/utils/validation.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LARGE_SPARSE_SUPPORTED = LooseVersion(scipy_version) >= '0.14.0'
# ===============================================
## parameters: 
input_path:
	single_barcode12/
f5_batch_num:
	50
model_path:
	/data/MAB/bin/deepsignal/models/model.GATC.R9_2D.tem.puc19.bn17.sn360/bn_17.sn_360.epoch_5.ckpt
is_cnn:
	yes
is_rnn:
	yes
is_base:
	yes
kmer_len:
	17
cent_signals_len:
	360
batch_size:
	512
learning_rate:
	0.001
class_num:
	2
result_file:
	single_barcode12.gatc.call_mods.tsv
recursively:
	yes
corrected_group:
	RawGenomeCorrected_001
basecall_subgroup:
	BaseCalled_template
is_dna:
	yes
normalize_method:
	mad
motifs:
	CG
mod_loc:
	0
positions:
	None
reference_path:
	None
nproc:
	1
is_gpu:
	no
# ===============================================
110340 fast5 files in total..
parse the motifs string..
read genome reference file..
read position file if it is not None..
call_mods process 1352721 starts
write process 1352722 starts
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:49: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:66: MultiRNNCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:71: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell)`, which is equivalent to this API
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:626: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py:1259: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:185: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:191: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:235: average_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.average_pooling2d instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:77: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
2023-03-24 19:35:20.760064: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2023-03-24 19:35:20.786036: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
2023-03-24 19:35:20.790056: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55c23ba56f60 executing computations on platform Host. Devices:
2023-03-24 19:35:20.790083: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2023-03-24 19:35:23.436685: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
Process Process-1:
Traceback (most recent call last):
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[{{node save/RestoreV2}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]

Caused by op 'save/RestoreV2', defined at:
  File "/data/MAB/miniconda3/envs/deepsignalenv/bin/deepsignal", line 33, in <module>
    sys.exit(load_entry_point('deepsignal==0.2.0', 'console_scripts', 'deepsignal')())
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 424, in main
    args.func(args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 88, in main_call_mods
    f5_args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 448, in call_mods
    is_rnn, is_base, is_cnn)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 326, in _call_mods_from_fast5s_cpu
    p.start()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 254, in _fast5s_q_to_pred_str_q
    saver = tf.train.Saver()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal
    restore_sequentially, reshape)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
    names_to_keys = object_graph_key_mapping(save_path)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1591, in object_graph_key_mapping
    checkpointable.OBJECT_GRAPH_PROTO_KEY)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 370, in get_tensor
    status)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 255, in _fast5s_q_to_pred_str_q
    saver.restore(sess, model_path)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1292, in restore
    err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]

Caused by op 'save/RestoreV2', defined at:
  File "/data/MAB/miniconda3/envs/deepsignalenv/bin/deepsignal", line 33, in <module>
    sys.exit(load_entry_point('deepsignal==0.2.0', 'console_scripts', 'deepsignal')())
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 424, in main
    args.func(args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 88, in main_call_mods
    f5_args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 448, in call_mods
    is_rnn, is_base, is_cnn)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 326, in _call_mods_from_fast5s_cpu
    p.start()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 254, in _fast5s_q_to_pred_str_q
    saver = tf.train.Saver()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal
    restore_sequentially, reshape)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]

write process 1352722 ending
0 of 110340 fast5 files failed..
call_mods costs 26.37 seconds..

The results file (single_barcode12.gatc.call_mods.tsv) that is produced is empty.