Fill command fails to validate transaction traces (--traces) #2092
Description
Metadata
- Python: CPython 3.12.12
- Operating System: OSX
pip freeze
annotated-types==0.7.0
asn1crypto==1.5.1
attrs==25.4.0
babel==2.17.0
backrefs==5.9
beautifulsoup4==4.14.2
bracex==2.6
cairocffi==1.7.1
cairosvg==2.8.2
cattrs==25.3.0
certifi==2025.10.5
cffi==2.0.0
charset-normalizer==3.4.4
ckzg==2.1.5
click==8.3.0
codespell==2.4.1
coincurve==20.0.0
colorama==0.4.6
colorlog==6.10.1
coverage==7.11.0
cryptography==45.0.7
cssselect2==0.8.0
cytoolz==1.1.0
defusedxml==0.7.1
docc==0.3.1
eth-abi==5.2.0
eth-hash==0.7.1
eth-typing==5.2.1
eth-utils==5.3.1
ethash==1.1.0
ethereum-hive==0.1.0a2
ethereum-rlp==0.1.4
ethereum-types==0.2.4
execnet==2.1.1
filelock==3.20.0
fladrif==0.2.0
ghp-import==2.1.0
gitdb==4.0.12
gitpython==3.1.45
griffe==1.14.0
hexbytes==1.3.1
html5lib==1.1
idna==3.11
importlib-resources==6.5.2
inflection==0.5.1
iniconfig==2.3.0
jinja2==3.1.6
joblib==1.5.2
libcst==1.8.5
lxml==6.0.2
markdown==3.8
markdown-it-py==4.0.0
markupsafe==3.0.3
mdurl==0.1.2
mergedeep==1.3.4
mike==1.1.2
mistletoe==1.5.0
mkdocs==1.6.1
mkdocs-autorefs==1.4.3
mkdocs-click==0.9.0
mkdocs-gen-files==0.5.0
mkdocs-get-deps==0.2.0
mkdocs-git-authors-plugin==0.10.0
mkdocs-glightbox==0.5.2
mkdocs-literate-nav==0.6.2
mkdocs-material==9.6.22
mkdocs-material-extensions==1.3.1
mkdocstrings==0.30.1
mkdocstrings-python==1.18.2
mypy==1.17.0
mypy-extensions==1.1.0
packaging==25.0
paginate==0.5.7
parsimonious==0.10.0
pathspec==0.12.1
pillow==10.4.0
platformdirs==4.5.0
pluggy==1.6.0
prompt-toolkit==3.0.52
py-ecc==8.0.0
pycparser==2.23
pycryptodome==3.23.0
pydantic==2.12.3
pydantic-core==2.41.4
pygments==2.19.2
pyjwt==2.10.1
pymdown-extensions==10.16.1
pyspelling==2.12
pytest==8.4.2
pytest-cov==4.1.0
pytest-custom-report==1.0.1
pytest-html==4.1.1
pytest-json-report==1.5.0
pytest-metadata==3.1.1
pytest-regex==0.2.0
pytest-xdist==3.8.0
python-dateutil==2.9.0.post0
pyyaml==6.0.3
pyyaml-env-tag==1.1
questionary==2.1.1
regex==2025.10.23
requests==2.32.5
requests-cache==1.2.1
requests-unixsocket2==1.0.1
rich==13.9.4
rlp==4.1.0
ruff==0.13.2
rust-pyspec-glue==0.0.9
selectolax==0.4.0
semver==3.0.4
six==1.17.0
smmap==5.0.2
sortedcontainers==2.4.0
soupsieve==2.8
tenacity==9.1.2
tinycss2==1.4.0
tomli==2.3.0
toolz==1.1.0
trie==3.1.0
types-pyyaml==6.0.12.20250915
types-requests==2.32.4.20250913
typing-extensions==4.15.0
typing-inspection==0.4.2
url-normalize==2.2.1
urllib3==2.5.0
verspec==0.1.0
vulture==2.14
watchdog==6.0.0
wcmatch==10.1
wcwidth==0.2.14
webencodings==0.5.1
What was wrong?
When running fill with --traces option, filling of some tests fails. For example,
$ uv run fill tests/frontier/opcodes/test_calldatacopy.py --traces --fork FrontierTool output
========================================================================= test session starts ==========================================================================
platform darwin -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
Generating fixtures for: Frontier
Only generating fixtures with stable/deployed forks: Specify an upcoming fork via --until=fork to add forks under development.
2.19.0
Log file: logs/fill-tests/frontier/opcodes/test_calldatacopy.py-20260128-181545-main.log
rootdir: /Users/avlasov/Documents/GitHub/execution-specs
configfile: packages/testing/src/execution_testing/cli/pytest_commands/pytest_ini_files/pytest-fill.ini
plugins: regex-0.2.0, html-4.1.1, xdist-3.8.0, json-report-1.5.0, metadata-3.1.1, cov-4.1.0, custom-report-1.0.1
collecting ... pytest-regex selected 24 tests to run for regex: .*
collected 24 items
tests/frontier/opcodes/test_calldatacopy.py ......F.......F. [16/16]
=============================================================================== FAILURES ===============================================================================
________________________________________________________ test_calldatacopy[fork_Frontier-state_test-underflow] _________________________________________________________
tests/frontier/opcodes/test_calldatacopy.py:205: in test_calldatacopy
state_test(pre=pre, post=post, tx=tx)
packages/testing/src/execution_testing/cli/pytest_commands/plugins/filler/filler.py:1385: in __init__
fixture = self.generate(
packages/testing/src/execution_testing/specs/state.py:507: in generate
return self.make_state_test_fixture(t8n)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
packages/testing/src/execution_testing/specs/state.py:352: in make_state_test_fixture
transition_tool_output = t8n.evaluate(
packages/testing/src/execution_testing/client_clis/clis/execution_specs.py:152: in evaluate
self.collect_traces(
packages/testing/src/execution_testing/client_clis/transition_tool.py:223: in collect_traces
traces.append(TransactionTraces.from_file(trace_file_path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
packages/testing/src/execution_testing/client_clis/cli_types.py:115: in from_file
return cls.model_validate(trace_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E pydantic_core._pydantic_core.ValidationError: 1 validation error for TransactionTraces
E error
E Extra inputs are not permitted [type=extra_forbidden, input_value='OutOfGasError', input_type=str]
E For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
______________________________________________ test_calldatacopy[fork_Frontier-blockchain_test_from_state_test-underflow] ______________________________________________
tests/frontier/opcodes/test_calldatacopy.py:205: in test_calldatacopy
state_test(pre=pre, post=post, tx=tx)
packages/testing/src/execution_testing/cli/pytest_commands/plugins/filler/filler.py:1385: in __init__
fixture = self.generate(
packages/testing/src/execution_testing/specs/state.py:503: in generate
return self.generate_blockchain_test().generate(
packages/testing/src/execution_testing/specs/blockchain.py:1010: in generate
return self.make_fixture(t8n)
^^^^^^^^^^^^^^^^^^^^^^
packages/testing/src/execution_testing/specs/blockchain.py:805: in make_fixture
built_block = self.generate_block_data(
packages/testing/src/execution_testing/specs/blockchain.py:600: in generate_block_data
transition_tool_output = t8n.evaluate(
packages/testing/src/execution_testing/client_clis/clis/execution_specs.py:152: in evaluate
self.collect_traces(
packages/testing/src/execution_testing/client_clis/transition_tool.py:223: in collect_traces
traces.append(TransactionTraces.from_file(trace_file_path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
packages/testing/src/execution_testing/client_clis/cli_types.py:115: in from_file
return cls.model_validate(trace_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E pydantic_core._pydantic_core.ValidationError: 1 validation error for TransactionTraces
E error
E Extra inputs are not permitted [type=extra_forbidden, input_value='OutOfGasError', input_type=str]
E For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
=========================================================================== warnings summary ===========================================================================
.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:812
/Users/avlasov/Documents/GitHub/execution-specs/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:812: PytestAssertRewriteWarning: Module already imported so cannot be rewritten; execution_testing.cli.pytest_commands.plugins.shared.execute_fill
self.import_plugin(arg, consider_entry_points=True)
.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:812
/Users/avlasov/Documents/GitHub/execution-specs/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:812: PytestAssertRewriteWarning: Module already imported so cannot be rewritten; execution_testing.cli.pytest_commands.plugins.forks.forks
self.import_plugin(arg, consider_entry_points=True)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
--------------- Log file: /Users/avlasov/Documents/GitHub/execution-specs/logs/fill-tests/frontier/opcodes/test_calldatacopy.py-20260128-181545-main.log ---------------
---------------------------- Generated html report: file:///Users/avlasov/Documents/GitHub/execution-specs/fixtures/.meta/report_fill.html -----------------------------
===================================== No tests executed - the test fixtures in "fixtures/" may now be executed against a client ======================================
======================================================================= short test summary info ========================================================================
FAILED tests/frontier/opcodes/test_calldatacopy.py::test_calldatacopy[fork_Frontier-state_test-underflow] - pydantic_core._pydantic_core.ValidationError: 1 validation error for TransactionTraces
FAILED tests/frontier/opcodes/test_calldatacopy.py::test_calldatacopy[fork_Frontier-blockchain_test_from_state_test-underflow] - pydantic_core._pydantic_core.ValidationError: 1 validation error for TransactionTraces
=============================================================== 2 failed, 14 passed, 2 warnings in 1.30s ===============================================================
The problem is that the last line of .jsonl trace file has an additional error field:
{"pc":64,"op":85,"gas":"0x49d","gasCost":"0x0","memory":"0x0000000000000000000000000000001234567890abcdef01234567890abcdef000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","memSize":96,"stack":["0x0","0x0"],"returnData":"0x","depth":1,"refund":0,"opName":"SSTORE","error":"OutOfGasError"}
{"output":"","gasUsed":"0x13488","error":"OutOfGasError"}
Other t8n tool can output traces, which cannot be validated also (e.g. "output" rows in the middle of a trace).
How can it be fixed?
TransactionTraces.from_files needs to be updated. However, I'm not sure whether one should just skip extra fields or error field should be added to TransactionTraces.