CI: add synthesizer test automation

[JehyukJang]

Description

Add synthesizer ERC20 config matrix + main-from-configs test runners, enforce permutation/instance_description checks, and wire synthesizer tests into CI for PRs to dev/main.

Related Issue

Closes #

Type of Change

• 🔥 Breaking Change
• 🌟 New Feature
• 🐛 Bug Fix
• 📝 Documentation Update
• 🎨 Style
• 🔧 Code Refactoring
• 📈 Performance Improvements
• ✅ Test
• 🤖 Build
• 🔁 CI
• 📦 Chore (Release)
• ⏩ Revert

Testing

• Unit Tests
• Integration Tests
• Manual Tests

[gemini-code-assist]

Summary of Changes

Hello @JehyukJang, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly upgrades the synthesizer's testing capabilities by automating the generation and execution of ERC20 transfer test scenarios. It introduces a robust framework for creating diverse test configurations, running them, and validating their outputs for data integrity. This enhancement aims to streamline the testing process, ensure the reliability of ERC20 transfer logic, and provide a more consistent environment for future development.

Highlights

• Automated ERC20 Test Configuration: Introduced new scripts to automatically generate a matrix of ERC20 transfer test configurations for both mainnet and Sepolia, covering various contract addresses and sender/recipient permutations.
• Enhanced Test Execution and Validation: Developed a script to run ERC20 transfer tests using the generated configurations, archiving outputs and enforcing consistency checks for permutation.json and error detection in instance_description.json.
• Simplified Test Scripting and Git Hooks: Streamlined the main npm test script in package.json to execute the new automated test flows and added a pre-commit Git hook to automatically run synthesizer tests.
• Centralized RPC/Environment Handling: Refactored RPC URL retrieval and environment variable handling into a shared utility module, improving consistency and maintainability across the synthesizer package.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• packages/frontend/synthesizer/.githooks/pre-commit
  • Added a new pre-commit hook to automatically run npm run test:node for the synthesizer package, ensuring local tests are executed before commits.
• packages/frontend/synthesizer/agents/tasks/lessons.md
  • Added a new lesson advising against suggesting README updates for developer-only workflow changes unless explicitly requested.
• packages/frontend/synthesizer/agents/tasks/todo.md
  • Updated the TODO and Review sections to reflect progress on GitHub Actions workflow for synthesizer tests, git hook automation, ERC20 config generation, and RPC/network utility refactoring.
• packages/frontend/synthesizer/examples/erc20Transfers/main.ts
  • Updated the import path for getRpcUrlFromEnv to use the new centralized src/interface/node/env.ts module.
  • Modified the call to getRpcUrlFromEnv to pass network, process.env, and options for more flexible RPC URL resolution.
• packages/frontend/synthesizer/examples/erc20Transfers/utils.ts
  • Removed the local getRpcUrlFromEnv function, centralizing RPC URL logic.
  • Exported EXAMPLES_ENV_PATH for consistent environment path usage across examples.
  • Removed the RPC_URL_ENV_KEY constant as its functionality is now handled by the centralized RPC utility.
• packages/frontend/synthesizer/package.json
  • Removed several specific test:* scripts, simplifying the testing commands.
  • Simplified the main test script to execute the new run-erc20-config-matrix.ts and run-erc20-main-from-configs.ts scripts.
  • Added hooks:install and hooks:uninstall scripts to manage git hooks for the package.
  • Removed coverage, coverage:test, and formatTest scripts.
• packages/frontend/synthesizer/scripts/generate-erc20-config.ts
  • Integrated minimist for robust CLI argument parsing, enabling non-interactive config generation.
  • Introduced CliConfig and BaseInputs types to structure CLI inputs.
  • Implemented parseCliInputs to handle CLI arguments for automated config creation.
  • Modified finalizeConfig to support different output paths and update a canonical config file.
  • Updated buildConfig to accept an optional baseOverride for non-interactive mode.
  • Refactored runPipeline to accept finalPath and baseOverride parameters.
  • Updated the main function to leverage parseCliInputs for determining interactive or non-interactive execution flow.
• packages/frontend/synthesizer/src/interface/cli/index.ts
  • Removed dotenv import and related environment loading logic, indicating centralization of environment variable handling.
• packages/frontend/synthesizer/src/interface/node/env.ts
  • Added a new file defining getRpcUrlFromEnv, a centralized utility to retrieve RPC URLs based on network and Alchemy API key, with support for custom environment paths.
• packages/frontend/synthesizer/src/interface/rpc/index.ts
  • No functional changes; removed a trailing newline character.
• packages/frontend/synthesizer/src/interface/rpc/types.ts
  • Added ALCHEMY_API_KEY_ENV_KEY and NETWORK_ENV_KEY constants for consistent environment variable naming.
  • Defined ALCHEMY_RPC_URLS for mainnet and Sepolia, centralizing RPC endpoint configurations.
  • Exported AlchemyNetwork type for network type safety.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-2be5e8c109e2197d077d13a82daead6a9b3433c5-s0-r3.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0x2be5e8c109e2197D077D13A82dAead6a9b3433C5, sender index 0, and recipient index 3.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-2be5e8c109e2197d077d13a82daead6a9b3433c5-s1-r2.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0x2be5e8c109e2197D077D13A82dAead6a9b3433C5, sender index 1, and recipient index 2.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-2be5e8c109e2197d077d13a82daead6a9b3433c5-s2-r1.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0x2be5e8c109e2197D077D13A82dAead6a9b3433C5, sender index 2, and recipient index 1.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-2be5e8c109e2197d077d13a82daead6a9b3433c5-s3-r0.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0x2be5e8c109e2197D077D13A82dAead6a9b3433C5, sender index 3, and recipient index 0.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48-s0-r3.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48, sender index 0, and recipient index 3.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48-s1-r2.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48, sender index 1, and recipient index 2.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48-s2-r1.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0xA0b86991c6218b36c1d19d4a2e9eb0ce3606eb48, sender index 2, and recipient index 1.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48-s3-r0.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0xA0b86991c6218b36c1d19d4a2e9eb0ce3606eb48, sender index 3, and recipient index 0.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-dac17f958d2ee523a2206206994597c13d831ec7-s0-r3.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0xdAC17F958D2ee523a2206206994597C13D831ec7, sender index 0, and recipient index 3.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-dac17f958d2ee523a2206206994597c13d831ec7-s1-r2.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0xdAC17F958D2ee523a2206206994597C13D831ec7, sender index 1, and recipient index 2.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-dac17f958d2ee523a2206206994597c13d831ec7-s2-r1.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0xdAC17F958D2ee523a2206206994597C13D831ec7, sender index 2, and recipient index 1.
• packages/frontend/synthesizer/tests/configs/config-mainnet-p4-i10-dac17f958d2ee523a2206206994597c13d831ec7-s3-r0.json
  • Added a new ERC20 transfer test configuration for mainnet, with 4 participants, 10 iterations, contract 0xdAC17F958D2ee523a2206206994597C13D831ec7, sender index 3, and recipient index 0.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-1c7d4b196cb0c7b01d743fbc6116a902379c7238-s0-r3.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0x1c7D4B196Cb0C7B01d743Fbc6116a902379C7238, sender index 0, and recipient index 3.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-1c7d4b196cb0c7b01d743fbc6116a902379c7238-s1-r2.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0x1c7D4B196Cb0C7B01d743Fbc6116a902379C7238, sender index 1, and recipient index 2.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-1c7d4b196cb0c7b01d743fbc6116a902379c7238-s2-r1.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0x1c7D4B196Cb0C7B01d743Fbc6116a902379C7238, sender index 2, and recipient index 1.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-1c7d4b196cb0c7b01d743fbc6116a902379c7238-s3-r0.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0x1c7D4B196Cb0C7B01d743Fbc6116a902379C7238, sender index 3, and recipient index 0.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-42d3b260c761cd5da022db56fe2f89c4a909b04a-s0-r3.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0x42d3b260c761cD5da022dB56Fe2F89c4A909b04A, sender index 0, and recipient index 3.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-42d3b260c761cd5da022db56fe2f89c4a909b04a-s1-r2.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0x42d3b260c761cD5da022dB56Fe2F89c4A909b04A, sender index 1, and recipient index 2.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-42d3b260c761cd5da022db56fe2f89c4a909b04a-s2-r1.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0x42d3b260c761cD5da022dB56Fe2F89c4A909b04A, sender index 2, and recipient index 1.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-42d3b260c761cd5da022db56fe2f89c4a909b04a-s3-r0.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0x42d3b260c761cD5da022dB56Fe2F89c4A909b04A, sender index 3, and recipient index 0.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-a30fe40285b8f5c0457dbc3b7c8a280373c40044-s0-r3.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0xa30fe40285B8f5c0457DbC3B7C8A280373c40044, sender index 0, and recipient index 3.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-a30fe40285b8f5c0457dbc3b7c8a280373c40044-s1-r2.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0xa30fe40285B8f5c0457DbC3B7C8A280373c40044, sender index 1, and recipient index 2.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-a30fe40285b8f5c0457dbc3b7c8a280373c40044-s2-r1.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0xa30fe40285B8f5c0457DbC3B7C8A280373c40044, sender index 2, and recipient index 1.
• packages/frontend/synthesizer/tests/configs/config-sepolia-p4-i10-a30fe40285b8f5c0457dbc3b7c8a280373c40044-s3-r0.json
  • Added a new ERC20 transfer test configuration for Sepolia, with 4 participants, 10 iterations, contract 0xa30fe40285B8f5c0457DbC3B7C8A280373c40044, sender index 3, and recipient index 0.
• packages/frontend/synthesizer/tests/scripts/run-erc20-config-matrix.ts
  • Added a new script to automate the generation of ERC20 transfer test configurations.
  • Iterates through predefined networks (mainnet, Sepolia), contract addresses, and sender/recipient index pairs.
  • Utilizes generate-erc20-config.ts in non-interactive mode to create a comprehensive set of test config files in tests/configs.
• packages/frontend/synthesizer/tests/scripts/run-erc20-main-from-configs.ts
  • Added a new script to execute ERC20 transfer tests based on the configurations generated by run-erc20-config-matrix.ts.
  • Runs examples/erc20Transfers/main.ts for each configuration file.
  • Archives the outputs of each test run into tests/outputs/<config-name>/.
  • Includes validation logic to compare permutation.json consistency across test groups (same network, contract, and transfer selector).
  • Checks instance_description.json for any 'Error:' strings, failing the test if found.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/synthesizer-tests.yml

Activity

• Verified that traceKeyOrigin correctly resolves reported keys to push32 using the current outputs/analysis/step_log.json.
• Confirmed the addition of a GitHub Actions workflow to run synthesizer npm test on PRs targeting dev or main.
• Verified that the ./.githooks/pre-commit script failed as expected when no test files were found, confirming intentional behavior.
• Confirmed the addition of tests/scripts/run-erc20-main-from-configs.ts and verified its structural integrity.
• Successfully executed tsx tests/scripts/run-erc20-main-from-configs.ts, which exited with code 0 and archived outputs correctly.
• Confirmed the implementation of permutation comparison logic and the presence of the script file.
• Verified that the script was updated to fail when instance_description.json contains 'Error:', and that a full run completed successfully with validations.
• Confirmed that RPC URL selection now correctly uses ALCHEMY_API_KEY and distinguishes between mainnet and Sepolia network settings.
• Confirmed that duplicated RPC/network logic has been consolidated into shared src/interface/rpc utilities.
• Confirmed that Alchemy RPC constants/types are now defined in src/interface/rpc/types.ts and correctly imported/consumed.
• Confirmed that Node-specific environment helpers are now located in src/interface/node/env.ts and are being used by callers.
• Confirmed that Node environment helpers exclusively return RPC URLs and do not modify process.env.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive test automation suite for the synthesizer, including scripts for generating a matrix of test configurations and running tests against them. The changes also include significant refactoring of environment variable and RPC URL handling, making it more robust and centralized. Additionally, a pre-commit hook is added to run tests automatically. Overall, this is a great step towards improving the test coverage and reliability of the synthesizer. I've identified a critical issue with the new pre-commit hook and a high-severity issue in the test generation script's error handling that should be addressed. I also have a suggestion to improve the maintainability of the config generation script.