Agora ESP32 Large-Model Intelligent Conversation

Example Overview

This example demonstrates how to use the Seeed Studio XIAO ESP32S3 to simulate a typical large-model intelligent conversation scenario, enabling real-time interactive dialogue with a large model.

File Structure

├── CMakeLists.txt
├── components                                  Agora iot sdk component
│   ├── agora_iot_sdk
│   │   ├── CMakeLists.txt
│   │   ├── include                             Agora iot sdk header files
│   │   │   ├── agora_rtc_api.h
│   │   └── libs                                Agora iot sdk libraries
│   │       ├── libagora-cjson.a
│   │       ├── libahpl.a
│   │       ├── librtsa.a
|   ├── esp32-camera                           esp32-camera component submodule
├── main                                        LLM Demo code
│   ├── ai_agent.h
│   ├── app_config.h
│   ├── common.h
│   ├── audio_proc.h
│   ├── rtc_proc.h
│   ├── CMakeLists.txt
│   ├── Kconfig.projbuild
|   ├── ai_agent.c
|   ├── audio_proc.c
|   ├── rtc_proc.c
│   └── llm_main.c
├── partitions.csv                              partition table
├── README.en.md
├── README.md
├── sdkconfig.defaults
└── sdkconfig.defaults.esp32s3

Directory / Files at a Glance

llm_main.c Program entry app_main(): WiFi → AIC3104 init → ai_agent_generate() → create RTC → prompt button to start Agent → periodic ping + print memory.
app_config.h Business configuration (TEN AI Agent service URL, Graph, Channel, UID, AppID, etc.).
common.h Global state g_app, audio codec macro selection (G711U/G722), and parameters such as sample rate/frame length.
ai_agent.c / ai_agent.h HTTP client: requests TEN AI Agent service (token/generate, start, stop, ping), parses responses and writes into g_app.app_id / g_app.token.
rtc_proc.c / rtc_proc.h Agora RTC wrapper: join channel, event callbacks, send audio frames send_rtc_audio_frame(), receive mixed audio and play via playback_stream_write().
audio_proc.c / audio_proc.h ESP-ADF audio pipeline: I2S recording → AEC (algo_stream) → raw → send to RTC; downlink audio writes to raw → I2S playback. Note: audio_set_volume() is currently a placeholder print (AIC3104 volume register control not implemented).
aic3104_ng.c / aic3104_ng.h Minimal AIC3104 Codec I2C driver (probe / write registers / default initialization).
xvf3800.c / xvf3800.h XVF3800 (XMOS) I2C control and key polling task:
- SET → ai_agent_start()
- MUTE → ai_agent_stop()
video_proc.c / video_proc.h (kept but disabled by default) video_proc.c is commented out in CMakeLists, and start_video_proc() is commented out in llm_main.c.

Environment Setup

Hardware Requirements

This example is ported from the ESP32-S3-Korvo-2 V3 development board and supports Seeed Studio Respeaker XVF3800.

Build and Flash

esp32-camera

To build and run this example, the esp32-camera component is required. It has been added as a submodule under components/esp32-camera. Run the following command to clone the submodule:

git submodule update --init --recursive

Agora IOT SDK

To build and run this example, the Agora IoT SDK is required. You can download it from here. Place agora_iot_sdk.tar into the esp32-client/components directory and run:

cd esp32-client/components
tar -xvf agora_iot_sdk.tar

Linux

Default IDF Branch

This example supports IDF tag v[5.2.3] and later. By default it uses IDF tag v[5.2.3] (commit id: c9763f62dd00c887a1a8fafe388db868a7e44069).

To select the IDF branch:

cd $IDF_PATH
git checkout v5.2.3
git pull
git submodule update --init --recursive

This example supports ADF v2.7 tag (commit id: 9cf556de500019bb79f3bb84c821fda37668c052).

Apply the IDF Patch

This example also requires merging one patch into IDF. Run:

export ADF_PATH=~/esp/esp-adf
cd $IDF_PATH
git apply $ADF_PATH/idf_patches/idf_v5.2_freertos.patch

Build Firmware

Copy this example directory (esp32-client) to ~/esp. Run:

$ . $HOME/esp/esp-idf/export.sh
$ cd ~/esp/esp32-client
$ idf.py set-target esp32s3
$ idf.py menuconfig  --> Agora Demo for ESP32 --> (configure WIFI SSID and Password)
$ idf.py build

Configure FreeRTOS forward compatibility: In menuconfig: Component config --> FreeRTOS --> Kernel, enable configENABLE_BACKWARD_COMPATIBILITY.

Windows

Default IDF Branch

Download IDF and choose the v5.2.3 offline version. This example uses IDF tag v[5.2.3] by default: https://docs.espressif.com/projects/esp-idf/zh_CN/v5.2.3/esp32/get-started/windows-setup.html

Download ADF; the ADF directory is under Espressif/frameworks. To support ADF v2.7 tag (commit id: 9cf556de500019bb79f3bb84c821fda37668c052): https://docs.espressif.com/projects/esp-adf/zh_CN/latest/get-started/index.html#step-2-get-esp-adf

Apply the IDF Patch

Method 1: Add ADF_PATH to system environment variables: E:\esp32s3\Espressif\frameworks\esp-adf

Method 2: Add ADF_PATH in the command line:

$ setx ADF_PATH Espressif/frameworks/esp-adf

Note: After setting the ADF_PATH environment variable, restart ESP-IDF 5.2 PowerShell for it to take effect.

This example also requires merging one patch into IDF. Run:

cd $IDF_PATH
git apply $ADF_PATH/idf_patches/idf_v5.2_freertos.patch

Modify ESP-ADF Board Pin Configuration

This is the most critical step! You need to modify the Board-layer configuration in the ESP-ADF framework.

File location (example) Depending on your own ESP-ADF directory:

Windows:

D:\Espressif\frameworks\esp-adf\components\audio_board\esp32_s3_korvo2_v3\board_pins_config.c

Linux/Mac:

~/esp/esp-adf/components/audio_board/esp32_s3_korvo2_v3/board_pins_config.c

Or based on your $ADF_PATH environment variable:

$ADF_PATH/components/audio_board/esp32_s3_korvo2_v3/board_pins_config.c

Modify the I2C pin configuration

Find the get_i2c_pins() function and change it to:

esp_err_t get_i2c_pins(i2c_port_t port, i2c_config_t *i2c_config)
{
    // Comment out the original Korvo-2 configuration
    // AUDIO_NULL_CHECK(TAG, i2c_config, return ESP_FAIL);
    // if (port == I2C_NUM_0 || port == I2C_NUM_1) {
    //     i2c_config->sda_io_num = GPIO_NUM_17;
    //     i2c_config->scl_io_num = GPIO_NUM_18;
    // } else {
    //     i2c_config->sda_io_num = -1;
    //     i2c_config->scl_io_num = -1;
    //     ESP_LOGE(TAG, "i2c port %d is not supported", port);
    //     return ESP_FAIL;
    // }

    // ReSpeaker XVF3800 I2C configuration
    i2c_config->sda_io_num = GPIO_NUM_5;   // ReSpeaker I2C SDA
    i2c_config->scl_io_num = GPIO_NUM_6;   // ReSpeaker I2C SCL
    return ESP_OK;
}

Modify the I2S Pin Configuration

Find the get_i2s_pins() function and change it to:

esp_err_t get_i2s_pins(int port, board_i2s_pin_t *i2s_config)
{
    // Comment out the original Korvo-2 configuration
    // AUDIO_NULL_CHECK(TAG, i2s_config, return ESP_FAIL);
    // if (port == 0) {
    //     i2s_config->bck_io_num = GPIO_NUM_9;
    //     i2s_config->ws_io_num = GPIO_NUM_45;
    //     i2s_config->data_out_num = GPIO_NUM_8;
    //     i2s_config->data_in_num = GPIO_NUM_10;
    //     i2s_config->mck_io_num = GPIO_NUM_16;
    // } else if (port == 1) {
    //     i2s_config->bck_io_num = -1;
    //     i2s_config->ws_io_num = -1;
    //     i2s_config->data_out_num = -1;
    //     i2s_config->data_in_num = -1;
    //     i2s_config->mck_io_num = -1;
    // } else {
    //     memset(i2s_config, -1, sizeof(board_i2s_pin_t));
    //     ESP_LOGE(TAG, "i2s port %d is not supported", port);
    //     return ESP_FAIL;
    // }

    // ReSpeaker XVF3800 I2S configuration
    i2s_config->bck_io_num   = GPIO_NUM_8;   // BCLK
    i2s_config->ws_io_num    = GPIO_NUM_7;   // WS/LRCK
    i2s_config->data_out_num = GPIO_NUM_44;  // DOUT
    i2s_config->data_in_num  = GPIO_NUM_43;  // DIN
    i2s_config->mck_io_num   = -1;           // Disable MCLK (ensure stability first)
    return ESP_OK;
}

⚠️ Important Notes:

This file is in the ESP-ADF framework directory, not in the project directory.
Changes will affect all projects using this Board configuration.
It’s recommended to back up the original file: cp board_pins_config.c board_pins_config.c.backup

Configure Your Own AI Agent

Configure your own AI Agent in the app_config.h file.
Change TENAI_AGENT_URL to your own TEN-Agent server URL (usually the 8080 service started via task run).
Change AI_AGENT_CHANNEL_NAME to your own AI Agent channel name.
Configure WIFI SSID and Password.
Rebuild and flash to the chip.

Notes:

Make sure at least one speaker is connected to the development board.
The server-side agora task is already running.

Build Firmware

Copy this example directory (esp32-client) into the Espressif/frameworks directory. Run:

$ cd ../esp32-client
$ idf.py set-target esp32s3
$ idf.py menuconfig  --> Agora Demo for ESP32 --> (configure WIFI SSID and Password)
$ idf.py build

Configure FreeRTOS forward compatibility: In menuconfig: Component config --> FreeRTOS --> Kernel, enable configENABLE_BACKWARD_COMPATIBILITY.

Flash Firmware

Run:

$ idf.py -p /dev/ttyUSB0 flash monitor

Note: On Linux, you may encounter permission issues with /dev/ttyUSB0. Run sudo usermod -aG dialout $USER.

After a successful flash, the example will run automatically. Once the device joins the RTC channel, you will see the serial log: "Agora: Press [SET] key to Join the Ai Agent ...".

FAQ

Q1: `i2c driver install error` during compilation

Cause: I2C driver conflict—old and new APIs are both used.

Solution:

Ensure aic3104_ng.c uses driver/i2c.h (old API).
Ensure driver/i2c_master.h (new API) is not used.
Check whether the code calls i2c_driver_delete() to delete the old driver.

Q2: Runtime I2C timeout `ESP_ERR_TIMEOUT`

Possible causes:

Hardware wiring issues
Incorrect pin configuration
Incorrect I2C address
Missing pull-up resistors

Debug steps:

Check the I2C scan result in logs:

W (xxxx) AIC3104_NG: Scanning I2C bus...
W (xxxx) AIC3104_NG: Found device at address 0x??

If no device is detected, check the wiring.
If a device is detected but the address is not 0x18, modify AIC3104_ADDR in aic3104_ng.h.

Q3: Compilation error `audio_board_init` not defined

Cause: Related code in audio_proc.c wasn’t properly commented out.

Solution: Ensure all code that calls board_handle in the setup_audio() function is commented out.

Q4: No audio output

Possible causes:

AIC3104 initialization failed
Incorrect I2S pin configuration
Audio data path issues

Debug steps:

Check logs to confirm AIC3104 initialization succeeded:

AIC3104 detected, page register = 0x00
~~~~~AIC3104 Codec initialized successfully~~~~

Confirm the I2S pin configuration in board_pins_config.c is correct.
Use an oscilloscope to check whether the I2S signal lines have waveforms.

Q5: Network buffer error `Not enough space`

Symptom:

[7479.598][err] -1/12:Not enough space
[7480.643][err][][netc_send_udp_data:185] 28 calls suppressed

Cause: This is a runtime network issue and does not affect hardware initialization.

Solution:

/ (idf.py menuconfig)
Check network quality
Reduce audio bitrate

Q6: Errors still occur after modifying `board_pins_config.c`

Cause: The file path modified is incorrect, or ESP-ADF was not rebuilt.

Solution:

Confirm you modified the correct file path (use echo $ADF_PATH or echo %ADF_PATH% to check).
Run idf.py fullclean for a full clean.
Rebuild with idf.py build.

Verification

Successful Startup Log

If everything is configured correctly, you should see logs similar to:

I (xxxx) wifi:connected with YourWiFi, aid = 1, channel 6, BW20, bssid = xx:xx:xx:xx:xx:xx
got ip: 10.103.4.61

~~~~~Initializing AIC3104 Codec~~~~
E (2342) i2c: i2c_driver_delete(457): i2c driver install error
W (2349) AIC3104_NG: init done: port=0 SDA=5 SCL=6 speed=100000
W (2355) AIC3104_NG: Scanning I2C bus...
W (2360) AIC3104_NG: Found device at address 0x18
W (2365) AIC3104_NG: Found 1 I2C device(s)
W (2370) AIC3104_NG: probe: write page 0
W (2375) AIC3104_NG: probe ok: page reg=0x00
AIC3104 detected, page register = 0x00
W (2380) AIC3104_NG: default setup applied
~~~~~AIC3104 Codec initialized successfully~~~~

I (xxxx) AUDIO_PIPELINE: Pipeline started
~~~~~agora_rtc_join_channel success~~~~
Agora: Press [SET] key to join the Ai Agent ...

Key Success Indicators

✅ WiFi connected - WiFi connected successfully
✅ got ip: xxx.xxx.xxx.xxx - IP address obtained
✅ Found device at address 0x18 - AIC3104 chip detected
✅ AIC3104 detected, page register = 0x00 - AIC3104 probe successful
✅ AIC3104 Codec initialized successfully - Codec initialization successful
✅ agora_rtc_join_channel success - RTC joined channel successfully

Summary

Core Adaptation Points

Codec driver replacement: Replace ES8311/ES7210 with AIC3104
I2C API compatibility: Use the old I2C API to avoid driver conflicts
Pin remapping: Modify the Board-layer configuration to adapt to new hardware
Skip Board initialization: Initialize AIC3104 directly without relying on the ESP-ADF Board layer

Modified File Count

New files: 2
Modified project files: 3
Modified framework files: 1 (ESP-ADF)

Known Limitations

Volume control: Temporarily disabled; needs to be implemented via AIC3104 registers
MCLK: Disabled; can be enabled and pin-configured if needed
Network buffers: You may see buffer-insufficient warnings at runtime

References

License

This adaptation guide is based on the MIT license of the TEN Framework.

Document Version: 1.0 Creation Date: 2025-01-05 Applicable Hardware: ReSpeaker XVF3800 + ESP32-S3 Applicable Software: TEN Framework esp32-client + ESP-IDF 5.2.3 + ESP-ADF 2.7

About Agora

Agora Audio & Video IoT platform solutions rely on Agora’s self-built underlying real-time transport network, Agora SD-RTN™ (Software Defined Real-time Network), providing audio and video [...] for all Linux/RTOS devices that support networking.

Technical Support

Please use the links below to obtain technical support:

If you find bugs in the sample code or have other questions, you can contact the community lead directly.

We will reply as soon as possible.

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Agora ESP32 Large-Model Intelligent Conversation

Example Overview

File Structure

Directory / Files at a Glance

Environment Setup

Hardware Requirements

Build and Flash

esp32-camera

Agora IOT SDK

Linux

Default IDF Branch

Apply the IDF Patch

Build Firmware

Windows

Default IDF Branch

Apply the IDF Patch

Modify ESP-ADF Board Pin Configuration

Modify the I2S Pin Configuration

Configure Your Own AI Agent

Build Firmware

Flash Firmware

FAQ

Q1: i2c driver install error during compilation

Q2: Runtime I2C timeout ESP_ERR_TIMEOUT

Q3: Compilation error audio_board_init not defined

Q4: No audio output

Q5: Network buffer error Not enough space

Q6: Errors still occur after modifying board_pins_config.c

Verification

Successful Startup Log

Key Success Indicators

Summary

Core Adaptation Points

Modified File Count

Known Limitations

References

License

About Agora

Technical Support

Q1: `i2c driver install error` during compilation

Q2: Runtime I2C timeout `ESP_ERR_TIMEOUT`

Q3: Compilation error `audio_board_init` not defined

Q5: Network buffer error `Not enough space`

Q6: Errors still occur after modifying `board_pins_config.c`