diff --git a/assets/contributors.csv b/assets/contributors.csv index 7f897b0613..fc5902ceed 100644 --- a/assets/contributors.csv +++ b/assets/contributors.csv @@ -133,7 +133,7 @@ Jaxson Han,Arm,povergoing,,, Tirui Wu,Arm,,,, William Watson,Arm,,,, John O'Hara,Arm,JohnOhara-Arm,johnaohara,JohnnyDoItAll, -Usamah Zaheer,Arm,usamahz,,, +Usamah Zaheer,Arm,usamahz,usamahzaheer,,https://usamah.me Kostiantyn Perehuda,Arm,,,, Yogesh Deshpande,Arm,,,, Roy Jamil,Ac6,RoyAc6,royjamil,,https://www.ac6.fr/en/ diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/_index.md b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/_index.md new file mode 100644 index 0000000000..93497fcd31 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/_index.md @@ -0,0 +1,67 @@ +--- +title: Deploy MobileSAM on Alif Ensemble E8 with ExecuTorch +description: Export MobileSAM, deploy camera and NPU inference on the Alif E8, and validate live progress through UART and retained integer-model references. +minutes_to_complete: 120 +draft: true +cascade: + draft: true +who_is_this_for: This Learning Path is for embedded ML developers who want to deploy live MobileSAM segmentation on an Alif Ensemble E8 using ExecuTorch and Zephyr. +learning_objectives: + - Trace the fixed-prompt and quantized tensor contracts from model export to live camera inference + - Export a MobileSAM model for the E8 HP core and Ethos-U85-256 + - Build a Zephyr application within its scratch and application RAM budgets + - Program and verify the model and firmware, then validate camera capture and NPU inference through UART + - Distinguish integer execution agreement from image accuracy and the optional dashboard's known DSI errors +prerequisites: + - An [Alif Ensemble E8 Development Kit](https://alifsemi.com/support/kits/ensemble-e8devkit/), revision A0 (`AE822FA0E5597LS0`), with its MT9M114 selfie camera; the default application does not use the display + - Internet access to download the supplied public calibration image; optionally, an RGB image representative of your own camera scenes + - Experience with PyTorch model export, C++, and Zephyr builds + - An Apple silicon Mac with Python 3.12, Git, CMake, Ninja, and a C++ build toolchain + - Zephyr SDK 0.17.0, west, Alif Security Toolkit, SEGGER J-Link, and Expect + - Access to the HP UART4 console through PRG USB and the board's SW4 selector +author: Usamah Zaheer +generate_summary_faq: false +rerun_summary: false +rerun_faqs: false +skilllevels: Advanced +subjects: AI/ML +armips: + - Ethos-U +tools_software_languages: + - ExecuTorch + - PyTorch + - MobileSAM + - Zephyr + - Python + - CMake +operatingsystems: + - macOS + - RTOS +further_reading: + - resource: + title: Run MobileSAM prompt segmentation on Arm Ethos-U85 with ExecuTorch + link: /learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-ethos-u/ + type: documentation + - resource: + title: Set up the Alif Ensemble E8 board + link: /learning-paths/embedded-and-microcontrollers/alif-image-classification/board-setup/ + type: documentation + - resource: + title: Alif Ensemble E8 DevKit support page + link: https://alifsemi.com/support/kits/ensemble-e8devkit/ + type: website + - resource: + title: ExecuTorch Arm Ethos-U backend documentation + link: https://docs.pytorch.org/executorch/stable/embedded-arm-ethos-u.html + type: documentation + - resource: + title: MobileSAM source repository + link: https://github.com/ChaoningZhang/MobileSAM + type: repository + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 +layout: learningpathall +learning_path_main_page: "yes" +--- diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/_next-steps.md b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/_next-steps.md new file mode 100644 index 0000000000..727b395ddd --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # The weight controls the order of the pages. _index.md always has weight 1. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/alif-e8-camera-display.patch b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/alif-e8-camera-display.patch new file mode 100644 index 0000000000..2b7b714952 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/alif-e8-camera-display.patch @@ -0,0 +1,335 @@ +diff --git a/drivers/display/display_cdc200.c b/drivers/display/display_cdc200.c +--- a/drivers/display/display_cdc200.c ++++ b/drivers/display/display_cdc200.c +@@ -18,6 +18,13 @@ + #include + + LOG_MODULE_REGISTER(CDC200, CONFIG_DISPLAY_LOG_LEVEL); ++ ++enum cdc200_stop_state { ++ CDC200_STOP_IDLE, ++ CDC200_STOP_ARMING, ++ CDC200_STOP_PENDING, ++ CDC200_STOP_CLAIMED, ++}; + + /* Helper functions */ + /* Global configuration setup functions. */ +@@ -636,6 +643,40 @@ + } + } + ++int cdc200_disable_at_vblank(const struct device *dev, k_timeout_t timeout) ++{ ++ struct cdc200_data *data = dev->data; ++ const struct cdc200_config *config = dev->config; ++ uintptr_t regs = DEVICE_MMIO_GET(dev); ++ int ret; ++ ++ if (!config->panel_cfg.vfp) ++ return -ENOTSUP; ++ /* Reap an ISR completion that arrived after an earlier caller timed out. */ ++ if (atomic_get(&data->stop_state) == CDC200_STOP_CLAIMED && ++ k_sem_take(&data->stop_sem, K_NO_WAIT) == 0) ++ atomic_set(&data->stop_state, CDC200_STOP_IDLE); ++ if (!atomic_cas(&data->stop_state, CDC200_STOP_IDLE, CDC200_STOP_ARMING)) ++ return -EBUSY; ++ if (!(sys_read32(regs + CDC_GLB_CTRL) & CDC_GLB_CTRL_CDC_EN)) { ++ atomic_set(&data->stop_state, CDC200_STOP_IDLE); ++ return 0; ++ } ++ ++ k_sem_reset(&data->stop_sem); ++ atomic_set(&data->stop_state, CDC200_STOP_PENDING); ++ ret = k_sem_take(&data->stop_sem, timeout); ++ if (ret) { ++ /* A claimed request must finish before another caller can arm a stop. */ ++ atomic_cas(&data->stop_state, CDC200_STOP_PENDING, CDC200_STOP_IDLE); ++ LOG_ERR("CDC did not stop at vertical blanking: position=0x%08x", ++ sys_read32(regs + CDC_POS_STAT)); ++ return ret; ++ } ++ atomic_set(&data->stop_state, CDC200_STOP_IDLE); ++ return 0; ++} ++ + void cdc200_swap_fb(const struct device *dev, uint8_t idx, struct cdc200_fb_desc *fb) + { + struct cdc200_data *data = dev->data; +@@ -683,6 +724,18 @@ + + if (irq_st & CDC_IRQ_LINE) { + cdc200_set_irq_clear(regs, CDC_IRQ_LINE); ++ uint32_t ypos = sys_read32(regs + CDC_POS_STAT) & CDC_POS_STAT_CYPOS_MASK; ++ uint32_t active_end = sys_read32(regs + CDC_ACTW_CFG) & ++ CDC_ACTW_CFG_AAH_MASK; ++ uint32_t total_end = sys_read32(regs + CDC_TOTALW_CFG) & ++ CDC_TOTALW_CFG_TOTALH_MASK; ++ ++ /* A delayed line IRQ must not truncate the next active frame. */ ++ if (ypos > active_end && ypos <= total_end && ++ atomic_cas(&data->stop_state, CDC200_STOP_PENDING, CDC200_STOP_CLAIMED)) { ++ cdc200_global_disable(regs); ++ k_sem_give(&data->stop_sem); ++ } + + bool swapped = false; + +@@ -787,6 +840,8 @@ + + data->dev = dev; + k_sem_init(&data->vsync_sem, 0, 1); ++ k_sem_init(&data->stop_sem, 0, 1); ++ atomic_set(&data->stop_state, CDC200_STOP_IDLE); + + #if DT_ANY_INST_HAS_PROP_STATUS_OKAY(clocks) + if (!cdc200_pixel_clock_valid(dev)) { +diff --git a/drivers/display/display_cdc200.h b/drivers/display/display_cdc200.h +--- a/drivers/display/display_cdc200.h ++++ b/drivers/display/display_cdc200.h +@@ -9,6 +9,7 @@ + #include + #include + #include ++#include + + /* Global Registers */ + #define CDC_HW_VER 0x00 /* HW Version Register */ +@@ -336,6 +337,8 @@ + uint8_t *next_fb[CDC_LAYER_MAX]; + + struct k_sem vsync_sem; ++ struct k_sem stop_sem; ++ atomic_t stop_state; + }; + + #endif /* _DISPLAY_CDC200_H_ */ +diff --git a/drivers/mipi_dsi/dsi_dw.c b/drivers/mipi_dsi/dsi_dw.c +--- a/drivers/mipi_dsi/dsi_dw.c ++++ b/drivers/mipi_dsi/dsi_dw.c +@@ -746,10 +746,70 @@ + dsi_dw_video_mode_config(dev); + } else { + /* Setup the DSI as Command mode. */ +- sys_write32(DSI_MODE_CFG_CMD_MODE, regs + DSI_MODE_CFG); ++ dsi_dw_cmd_mode_config(dev); + } + dsi_dw_pwr_up(regs); + return 0; ++} ++ ++int dsi_dw_suspend_video(const struct device *dev) ++{ ++ struct dsi_dw_data *data = dev->data; ++ uintptr_t regs = DEVICE_MMIO_GET(dev); ++ uint32_t mask = DSI_PHY_STATUS_PHY_STOPSTATECLKLANE | ++ DSI_PHY_STATUS_PHY_STOPSTATE0LANE; ++ uint32_t status; ++ int ret; ++ ++ if (data->phy.num_lanes == 2) ++ mask |= DSI_PHY_STATUS_PHY_STOPSTATE1LANE; ++ else if (data->phy.num_lanes != 1) ++ return -EINVAL; ++ ++ if (data->curr_mode == DSI_DW_VIDEO_MODE) { ++ uint32_t empty_mask = DSI_VID_PKT_STATUS_DPI_BUFF_PLD_EMPTY | ++ DSI_VID_PKT_STATUS_DPI_PLD_W_EMPTY | ++ DSI_VID_PKT_STATUS_DPI_CMD_W_EMPTY; ++ uint32_t data_mask = mask & ~DSI_PHY_STATUS_PHY_STOPSTATECLKLANE; ++ uint32_t video_status; ++ bool drained = false; ++ ++ /* CDC must already be stopped at VFP; COMMAND mode clears these status bits. */ ++ for (int i = 0; i < 100; i++) { ++ video_status = sys_read32(regs + DSI_VID_PKT_STATUS); ++ status = sys_read32(regs + DSI_PHY_STATUS); ++ if ((video_status & empty_mask) == empty_mask && ++ (status & (data_mask | DSI_PHY_STATUS_PHY_PHY_DIRECTION)) == data_mask) { ++ drained = true; ++ break; ++ } ++ k_sleep(K_MSEC(1)); ++ } ++ if (!drained) { ++ LOG_ERR("DSI video did not drain: VID_PKT_STATUS=0x%08x PHY_STATUS=0x%08x", ++ video_status, status); ++ return -ETIMEDOUT; ++ } ++ } ++ ++ ret = dsi_dw_set_mode(dev, DSI_DW_COMMAND_MODE); ++ if (ret) ++ return ret; ++ ++ /* HS command panels otherwise keep the clock lane active while idle. */ ++ sys_clear_bits(regs + DSI_LPCLK_CTRL, DSI_LPCLK_CTRL_PHY_TXREQUESTCLKHS); ++ for (int i = 0; i < 100; i++) { ++ status = sys_read32(regs + DSI_PHY_STATUS); ++ if ((status & (mask | DSI_PHY_STATUS_PHY_PHY_DIRECTION)) == mask) { ++ dsi_dw_pwr_down(regs); ++ return 0; ++ } ++ k_sleep(K_MSEC(1)); ++ } ++ ++ dsi_dw_pwr_down(regs); ++ LOG_ERR("DSI lanes did not stop before suspension: PHY_STATUS=0x%08x", status); ++ return -ETIMEDOUT; + } + + /* Generic APIs */ +diff --git a/drivers/mipi_dsi/dsi_dw.h b/drivers/mipi_dsi/dsi_dw.h +--- a/drivers/mipi_dsi/dsi_dw.h ++++ b/drivers/mipi_dsi/dsi_dw.h +@@ -75,6 +75,10 @@ + #define DSI_VID_VFP_LINES_ACT 0x15C /* Video VFP Current Config Register */ + #define DSI_VID_VACT_LINES_ACT 0x160 /* Video VA Current Config Register */ + #define DSI_VID_PKT_STATUS 0x168 /* Video Pkt Status Register */ ++ ++#define DSI_VID_PKT_STATUS_DPI_BUFF_PLD_EMPTY BIT(16) ++#define DSI_VID_PKT_STATUS_DPI_PLD_W_EMPTY BIT(2) ++#define DSI_VID_PKT_STATUS_DPI_CMD_W_EMPTY BIT(0) + #define DSI_SDF_3D_ACT 0x190 /* 3D Current Config Register */ + + /* Registers Bit-fields */ +diff --git a/drivers/video/isp_pico.c b/drivers/video/isp_pico.c +--- a/drivers/video/isp_pico.c ++++ b/drivers/video/isp_pico.c +@@ -205,6 +205,7 @@ + const struct isp_config *config = dev->config; + struct isp_data *data = dev->data; + struct video_buffer *vbuf = NULL; ++ struct video_buffer *completed_vbuf = NULL; + + int ret; + +@@ -238,7 +239,7 @@ + + vbuf->timestamp = k_uptime_get_32(); + +- k_fifo_put(&data->fifo_out, vbuf); ++ completed_vbuf = vbuf; + + vbuf = k_fifo_peek_head(&data->fifo_in); + if (vbuf == NULL) { +@@ -260,8 +261,22 @@ + + isp_bottom_done: + if (!data->is_streaming) { +- video_stream_stop(config->controller); +- data->curr_vid_buf = 0; ++ ret = video_stream_stop(config->controller); ++ if (ret) { ++ LOG_ERR("Failed to stop controller before returning video buffer: %d", ret); ++ signal_status = VIDEO_BUF_ERROR; ++ if (completed_vbuf) { ++ k_fifo_put(&data->fifo_in, completed_vbuf); ++ completed_vbuf = NULL; ++ } ++ } else { ++ data->curr_vid_buf = 0; ++ } ++ } ++ ++ /* A returned buffer lets the application switch the shared MIPI route. */ ++ if (completed_vbuf) { ++ k_fifo_put(&data->fifo_out, completed_vbuf); + } + + LOG_DBG("current video buffer - 0x%08x", data->curr_vid_buf); +diff --git a/drivers/video/video_alif.c b/drivers/video/video_alif.c +--- a/drivers/video/video_alif.c ++++ b/drivers/video/video_alif.c +@@ -544,17 +544,24 @@ + hw_enable_interrupts(regs, INTR_VSYNC | INTR_BRESP_ERR | INTR_OUTFIFO_OVERRUN | + INTR_INFIFO_OVERRUN | INTR_STOP); + +- /* Start the MIPI CSI-2 IP in case the MIPI CSI is available. */ ++ data->is_streaming = true; ++ hw_cam_start_video_capture(dev); ++ ++ /* Arm the consumer before upstream starts producing pixels. */ + ret = video_stream_start(config->endpoint_dev); + if (ret) { ++ data->is_streaming = false; ++ hw_disable_interrupts(regs, INTR_VSYNC | INTR_BRESP_ERR | INTR_OUTFIFO_OVERRUN | ++ INTR_INFIFO_OVERRUN | INTR_STOP); ++ k_work_cancel_sync(&data->cb_work, &sync); ++ hw_cam_soft_reset(regs); ++ sys_write32(sys_read32(regs + CAM_INTR), regs + CAM_INTR); ++ data->curr_vid_buf = 0; + LOG_ERR("Failed to start streaming of Video pipeline!"); + return -EIO; + } + +- hw_cam_start_video_capture(dev); + LOG_DBG("Stream started"); +- +- data->is_streaming = true; + + return 0; + } +diff --git a/drivers/video/video_csi_dw.c b/drivers/video/video_csi_dw.c +--- a/drivers/video/video_csi_dw.c ++++ b/drivers/video/video_csi_dw.c +@@ -425,6 +425,12 @@ + return 0; + } + ++ if (!data->streaming_map) { ++ /* Discard frame state retained while IPI was disabled. */ ++ sys_clear_bits(regs + CSI_IPI_SOFTRSTN, CSI_IPI_SOFTRSTN_RSTN); ++ sys_set_bits(regs + CSI_IPI_SOFTRSTN, CSI_IPI_SOFTRSTN_RSTN); ++ } ++ + /* Enable CSI streaming */ + sys_set_bits(regs + CSI_IPI_MODE, CSI_IPI_MODE_ENABLE); + LOG_DBG("Stream started"); +@@ -433,10 +439,12 @@ + if (config->sensor[data->current_sensor]) { + ret = video_stream_start(config->sensor[data->current_sensor]); + if (ret) { ++ sys_clear_bits(regs + CSI_IPI_MODE, CSI_IPI_MODE_ENABLE); + LOG_ERR("Failed to start sensor stream!"); + return ret; + } + } else { ++ sys_clear_bits(regs + CSI_IPI_MODE, CSI_IPI_MODE_ENABLE); + LOG_ERR("Incorrect sensor selected!"); + return -ENODEV; + } +diff --git a/include/zephyr/drivers/display/cdc200.h b/include/zephyr/drivers/display/cdc200.h +--- a/include/zephyr/drivers/display/cdc200.h ++++ b/include/zephyr/drivers/display/cdc200.h +@@ -93,6 +93,13 @@ + bool enable); + + /* ++ * Disable CDC at the next vertical front porch. Returns -EAGAIN on timeout. ++ * The caller must serialize all CDC enable/disable requests, including calls to ++ * this function, and framebuffer changes. ++ */ ++int cdc200_disable_at_vblank(const struct device *dev, k_timeout_t timeout); ++ ++/* + * Swap Framebuffers for specified layer. The buffer change will happen at next vblank + */ + void cdc200_swap_fb(const struct device *dev, +diff --git a/include/zephyr/drivers/mipi_dsi/dsi_dw.h b/include/zephyr/drivers/mipi_dsi/dsi_dw.h +--- a/include/zephyr/drivers/mipi_dsi/dsi_dw.h ++++ b/include/zephyr/drivers/mipi_dsi/dsi_dw.h +@@ -25,6 +25,13 @@ + int dsi_dw_set_mode(const struct device *dev, + enum dsi_dw_mode mode); + ++/* ++ * After CDC is stopped at vertical blanking, drain video and stop the HS clock, ++ * wait for all active lanes to stop, then hold the host powered down. ++ * Resume with dsi_dw_set_mode(dev, DSI_DW_VIDEO_MODE). ++ */ ++int dsi_dw_suspend_video(const struct device *dev); ++ + #ifdef __cplusplus + } + #endif diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/alif-live-demo.jpg b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/alif-live-demo.jpg new file mode 100644 index 0000000000..64e55fdcdf Binary files /dev/null and b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/alif-live-demo.jpg differ diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/build-the-firmware.md b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/build-the-firmware.md new file mode 100644 index 0000000000..3c848b749c --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/build-the-firmware.md @@ -0,0 +1,93 @@ +--- +title: Build the Zephyr firmware within the E8 memory budget +description: Build for the correct HP and U85 target and check model metadata, scratch capacity, and linker memory usage. +weight: 5 +layout: learningpathall +--- + +## Build the camera and UART application + +The default build captures camera frames, executes MobileSAM, selects masks, and reports progress through UART4. `CONFIG_MOBILESAM_DISPLAY=n` leaves CDC, DSI, and the panel disabled; the camera route stays selected after initialization. Build with the pinned Alif SDK and the matching PTE/metadata pair: + +```bash +cd "$EXECUTORCH_ROOT" +west build \ + -d "$MOBILE_SAM_OUTPUT/build-mobilesam-live" \ + -p always \ + -b alif_e8_dk/ae822fa0e5597xx0/rtss_hp \ + "$EXECUTORCH_ROOT/zephyr/samples/mobilesam-ethosu" -- \ + -DZEPHYR_EXTRA_MODULES="$EXECUTORCH_ROOT;$SDK_ALIF" \ + -DSDK_ALIF="$SDK_ALIF" \ + -DET_PTE_FILE_PATH="$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \ + -DMODEL_METADATA_PATH="$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.json" \ + -DPython3_EXECUTABLE="$(command -v python)" +``` + +The board identifier is tied to the pinned vendor tree. Replacing it with a similar identifier from a newer Zephyr tutorial can select a different configuration. This sample sets `CONFIG_ARM_ETHOS_U85_256=y` and `ETHOSU_TARGET_NPU_CONFIG=ethos-u85-256` explicitly. + +CMake finds the official `serial_camera_mt9m114_selfie.overlay` in `sdk-alif` and adds the sample's deferred-initialization overlay. It generates model metadata, inspects the PTE for unsupported CPU operators, and extracts the Ethos-U scratch requirement. The default target excludes `display.cpp` and its logo from the linked application. + +Check the generated configuration before programming: + +```bash +rg 'CONFIG_(RTSS_HP|ARM_ETHOS_U85_256|MOBILESAM_DISPLAY|DISPLAY|MIPI_DSI)' \ + "$MOBILE_SAM_OUTPUT/build-mobilesam-live/zephyr/.config" +! rg -q '^CONFIG_(MOBILESAM_DISPLAY|DISPLAY|MIPI_DSI)=y$' \ + "$MOBILE_SAM_OUTPUT/build-mobilesam-live/zephyr/.config" +``` + +Require the HP/U85-256 target and no enabled display or DSI option. A display that stays blank is expected for this configuration. + +## Understand the three allocator pools + +The supplied `prj.conf` sets: + +| Setting | Bytes | Purpose | +| --- | ---: | --- | +| `CONFIG_EXECUTORCH_METHOD_ALLOCATOR_POOL_SIZE` | 655,360 | Method state and planned buffers allocated by the sample | +| `CONFIG_EXECUTORCH_TEMP_ALLOCATOR_POOL_SIZE` | 6,488,064 | Main Ethos-U tensor/scratch arena | +| `CONFIG_EXECUTORCH_FAST_TEMP_ALLOCATOR_POOL_SIZE` | 393,216 | Fast scratch arena | + +CMake rejects a PTE whose extracted scratch requirement exceeds the configured main scratch pool. Passing that check does not prove the complete application fits: camera buffers, stacks, method allocations, and linker regions also consume memory. The optional dashboard adds display buffers and drivers. + +The clean default build with the pinned source, public-image export, and independently cloned vendor workspace reported: + +| Linker region | Used bytes | Capacity bytes | Remaining bytes | +| --- | ---: | ---: | ---: | +| Application RAM | 7,580,640 | 7,602,176 | 21,536 | +| SRAM0 | 64 | 786,432 | 786,368 | + +It used **171,192 bytes** of FLASH, with raw binary SHA256 `3be06219380e7d2444d0b18bef8ebaa42284f401aed2d8e3e5d720c50edc4a36`. The build checks confirmed disabled display/DSI nodes, no linked display or diagnostic-trace functions, matching model metadata, and hard-float Cortex-M55/MVE settings. Read your own linker report and `zephyr.map`: application RAM remains close to its limit even though SRAM0 no longer holds the dashboard framebuffer. Changing the model, toolchain, logging, or buffer placement requires another memory and execution check. + +## Keep model storage separate from firmware storage + +The PTE remains in memory-mapped external OSPI flash at `0xC0000000`. The firmware records its size and address and opens it with `BufferDataLoader`. It does not link the 13 MB model into the firmware image. + +Changing the external-flash address alone does not enlarge RAM or fix a linker overflow. Changing the model size also requires rebuilding the firmware so its compiled PTE size and generated metadata remain consistent. + +Before flashing, locate `build-mobilesam-live/zephyr/zephyr.elf`, inspect the linker report, and confirm that the build accepted your PTE. The next lesson programs model data and application firmware as separate operations. + +## Build the optional dashboard separately + +The dashboard is an experimental extension: earlier physical runs displayed advancing camera frames and masks but retained DSI acknowledgment and packet errors. Keep the default build and its evidence intact. To investigate the extension, build into a different directory with both its configuration and overlay: + +```bash +west build \ + -d "$MOBILE_SAM_OUTPUT/build-mobilesam-dashboard" \ + -p always \ + -b alif_e8_dk/ae822fa0e5597xx0/rtss_hp \ + "$EXECUTORCH_ROOT/zephyr/samples/mobilesam-ethosu" -- \ + -DZEPHYR_EXTRA_MODULES="$EXECUTORCH_ROOT;$SDK_ALIF" \ + -DSDK_ALIF="$SDK_ALIF" \ + -DEXTRA_CONF_FILE="$EXECUTORCH_ROOT/zephyr/samples/mobilesam-ethosu/dashboard.conf" \ + -DEXTRA_DTC_OVERLAY_FILE="$EXECUTORCH_ROOT/zephyr/samples/mobilesam-ethosu/boards/dashboard.overlay" \ + -DET_PTE_FILE_PATH="$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \ + -DMODEL_METADATA_PATH="$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.json" \ + -DPython3_EXECUTABLE="$(command -v python)" +``` + +Both files are required: they enable `CONFIG_MOBILESAM_DISPLAY=y` and the matching display devicetree nodes. This optional configuration cross-built successfully, but has not been accepted as a clean physical output path. It switches the shared MIPI route between capture and display and retains the known unresolved DSI limitation. A successful build or updating screen does not establish an error-free run. The remaining programming and primary validation commands use `build-mobilesam-live`, the default camera-and-UART build. + +## What you've accomplished + +You've built the default HP camera/NPU application, generated matching model metadata, and checked its scratch and linker capacity. Next, program the model and firmware as separate artifacts. diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/export-the-model.md b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/export-the-model.md new file mode 100644 index 0000000000..0719f9a427 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/export-the-model.md @@ -0,0 +1,163 @@ +--- +title: Export and inspect the live MobileSAM model +description: Freeze four independent prompts, export int8 model I/O, and inspect delegation and metadata before building firmware. +weight: 4 +layout: learningpathall +--- + +## Prepare the pinned MobileSAM source + +Keep the environment from the previous lesson active and run these commands from the patched ExecuTorch checkout. Its exporter has live-camera options that are absent from some later versions of the public FVP example. + +```bash +cd "$EXECUTORCH_ROOT" +python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/prepare_mobilesam.py \ + --source-dir "$MOBILE_SAM_SOURCE" +``` + +The preparation step uses MobileSAM revision `f706ad9c4eb7f219c00d9050e46328518ffb65d2` and applies the configurable-image-size patch. The exporter verifies the official checkpoint SHA256: + +```output +6dbb90523a35330fedd7f1d3dfc66f995213d81b29a5ca8108dbcdd4e37d6c2f +``` + +## Select a calibration image + +Use the public dog image from the Segment Anything example, pinned to a source revision. Download it and verify its SHA256 before export: + +```bash +export MOBILE_SAM_CALIBRATION_IMAGE="$MOBILE_SAM_OUTPUT/sam-example-dog.jpg" +curl --fail --location \ + https://raw.githubusercontent.com/facebookresearch/segment-anything/dca509fe793f601edb92606367a655c15ac00fdf/notebooks/images/dog.jpg \ + --output "$MOBILE_SAM_CALIBRATION_IMAGE" +printf '%s %s\n' \ + bf76876b90e3ebd521f9882b9177ba8f33e80cb7ec09c630f179b122edd125e1 \ + "$MOBILE_SAM_CALIBRATION_IMAGE" | shasum -a 256 -c +``` + +Require the checksum result `OK`. The image comes from the [Segment Anything repository](https://github.com/facebookresearch/segment-anything/tree/dca509fe793f601edb92606367a655c15ac00fdf/notebooks/images), which provides its [license](https://github.com/facebookresearch/segment-anything/blob/dca509fe793f601edb92606367a655c15ac00fdf/LICENSE). + +With the released dependencies and the configuration below, this image passed the unchanged host quality gate with selected-mask IoU `0.9756` and pixel agreement `0.9919`. The gate selected zero-based prompt indices 2 and 3 (the third and fourth prompts); the later section explains this selection rule. This image is different from `examples/models/dinov2/dog.jpg`. + +The exporter uses this image for both quantization calibration and its host comparison. You can replace it with an RGB image representative of your camera scenes, but require your export to pass the unchanged quality gate before continuing. Calibration input affects both the result and the generated model. A passing comparison on the calibration image does not measure accuracy on new scenes. + +## Export four independent prompts + +Use the live-camera configuration: + +```bash +python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/export_mobilesam.py \ + --output-path "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \ + --mobile-sam-source "$MOBILE_SAM_SOURCE" \ + --image "$MOBILE_SAM_CALIBRATION_IMAGE" \ + --input-size 384 \ + --point 96 96 \ + --point 288 96 \ + --point 96 288 \ + --point 288 288 \ + --multimask-output \ + --quantized-io +``` + +The points sample the four image quadrants. In this multimask mode they are independent prompt batches, giving three candidate masks and associated quality scores per point. This is different from asking SAM for one mask constrained jointly by all four points. + +The image encoder and decoder run on the NPU. The prompt encoder runs during export, and its fixed embeddings are stored in the model. Re-export when you change prompt coordinates, input size, or the output contract. + +This model contract is the same for the default camera-and-UART firmware and optional dashboard. Enabling a display does not require a different export. + +The export applies A8W8 globally and A16W8 to TinyViT attention, then lowers through the Arm backend and Vela for `ethos-u85-256`. The compiled configuration uses `Ethos_U85_SYS_DRAM_Mid` and `Dedicated_Sram_384KB`. Those are compiler settings; actual firmware storage and scratch placement still need to match the E8 memory map. + +## Inspect artifacts before continuing + +The exporter writes the `.pte` and matching `.json` metadata beside it, together with mask-agreement metrics, a delegation summary, and TOSA/Vela debug artifacts. Inspect them: + +```bash +ls -lh "$MOBILE_SAM_OUTPUT" +python -m json.tool "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.json" +cat "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask_metrics.json" +cat "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask_delegation.txt" +``` + +Require successful export and inspect the generated masks and both quality metrics. This exporter accepts selected-mask IoU at least `0.9` **or** pixel agreement at least `0.96`. It compares selected candidates at prompts whose predicted quantized quality reaches `0.9`, using all prompts as fallback when none reaches that threshold. This does not require `0.9` IoU for every candidate or every prompt. The device's later mask-selection threshold is `0.8`. + +Using the same image for calibration and evaluation does not measure segmentation accuracy on new scenes. Inspect the generated masks, then validate the programmed model on representative camera scenes. Your calibration image and released dependencies can produce a different artifact from the retained board binary; record your own model and metadata hashes. + +The live application expects this tensor interface: + +| Tensor | Shape | Type | +| --- | --- | --- | +| Image input | `[1,3,384,384]` | int8 | +| Mask output | `[4,3,96,96]` | int8 | +| Quality output | `[4,3]` | int8 | + +CMake rejects known CPU operators, while the runtime checks input type/byte count and some output dimensions. The metadata-only header generator does not validate `output_shape`. Check the actual PTE tensor interface against the JSON and this configuration before building: + +```bash +python - <<'PY' +import hashlib +import json +import math +import os +from pathlib import Path + +from executorch.exir._serialize._program import deserialize_pte_binary +from executorch.exir.scalar_type import ScalarType +from executorch.exir.schema import Tensor + +pte_path = Path(os.environ["MOBILE_SAM_OUTPUT"]) / "mobilesam_live_384_grid4_multimask.pte" +json_path = pte_path.with_suffix(".json") +metadata = json.loads(json_path.read_text()) +program = deserialize_pte_binary(pte_path.read_bytes()).program +plan, = [p for p in program.execution_plan if p.name == "forward"] +shapes = [[1, 3, 384, 384], [4, 3, 96, 96], [4, 3]] +assert len(plan.inputs) == 1 and len(plan.outputs) == 2 +for index, shape in zip(plan.inputs + plan.outputs, shapes): + tensor = plan.values[index].val + assert isinstance(tensor, Tensor) and tensor.scalar_type == ScalarType.CHAR + assert tensor.sizes == shape, (tensor.sizes, shape) +assert metadata["input_shape"] == shapes[0] +assert metadata["output_shape"] == shapes[1] +assert metadata["input_size"] == 384 and metadata["multimask_output"] is True +assert metadata["point_prompts_xy"] == [[96, 96], [288, 96], [96, 288], [288, 288]] +preprocessing = metadata["preprocessing"] +assert preprocessing["resize"] == "longest_side_then_zero_pad" +for key, expected in (("pixel_mean", [123.675, 116.28, 103.53]), + ("pixel_std", [58.395, 57.12, 57.375])): + values = preprocessing[key] + assert len(values) == 3 and all( + math.isclose(value, reference, rel_tol=0, abs_tol=0.0001) + for value, reference in zip(values, expected) + ) +for name in ("input", "output", "quality"): + qparams = metadata["io_quantization"][name] + assert qparams["dtype"] == "int8" + assert math.isfinite(qparams["scale"]) and qparams["scale"] > 0 + assert isinstance(qparams["zero_point"], int) and -128 <= qparams["zero_point"] <= 127 +print("PTE tensor interface and JSON configuration checks passed") +for path in (pte_path, json_path): + print(hashlib.sha256(path.read_bytes()).hexdigest(), path.name) +PY +``` + +Require the success message and retain both printed hashes with your export. An assertion failure means the files do not satisfy this configuration; resolve it before continuing. The PTE's tensor schema does not store quantization parameters, so this check cannot prove that the JSON scales and zero points belong to that PTE. Keep the matching PTE and JSON from the same export, alongside their hashes, and check target output during execution. + +An old `[1,1,112,112]` floating-point FVP result belongs to the earlier 448×448 single-mask configuration. It is not the output contract of this 384×384 live model. + +## Prepare an identical model file for programming + +J-Link and the recovery writer consume a raw `.bin` file. This is an unchanged copy of the PTE, without a new container or header: + +```bash +cp "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \ + "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.bin" +cmp "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \ + "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.bin" +``` + +The fresh released-dependency export using this public image is 13,317,584 bytes, with SHA256 `a908bca09a74bac884895fa72b650d15e944cdfc7b0351b99e40ee63b6b3a7b5`. Its metadata JSON has SHA256 `36884d2ffafa1b9ee46c71c0a855b9b83b01da57e3e87a3a99fdb56f547770f7`. Its extracted scratch requirement is 5,898,256 bytes with a 393,216-byte fast scratch requirement. These identify the fresh host export; firmware and hardware checks follow in later lessons. Generated JSON records local paths, so its hash also depends on your workspace. Record the size and hashes of your own matching pair. + +Before proceeding, explain why `--quantized-io` matters: it removes the host-style floating-point I/O boundary from the device program, while the application performs camera normalization, quantization, and mask post-processing. + +## What you've accomplished + +You've exported the four-prompt model, checked delegation and quantized tensor metadata, and prepared an identical binary for programming. Next, build firmware with this PTE and JSON and check its memory use. diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/mobilesam-alif-e8.patch b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/mobilesam-alif-e8.patch new file mode 100644 index 0000000000..58522481a3 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/mobilesam-alif-e8.patch @@ -0,0 +1,4943 @@ +diff --git a/README-wheel.md b/README-wheel.md +--- a/README-wheel.md ++++ b/README-wheel.md +@@ -7,6 +7,22 @@ + The `executorch` pip package is in beta. + * Supported python versions: 3.10, 3.11, 3.12, 3.13, 3.14 + * Compatible systems: Linux x86_64, Linux aarch64, macOS aarch64 ++ ++Backend export tools can require optional Python dependencies. For example, ++install the Ethos-U AOT dependencies with: ++ ++```bash ++pip install 'executorch[ethos_u]' ++``` ++ ++The `ethos_u` extra installs host-side export dependencies such as the Vela ++compiler; it does not install an Arm cross compiler, an FVP, or a target C++ ++runtime. There is no `cortex_m` package extra. CMSIS-NN is a separate Cortex-M ++export and runtime path that requires an ExecuTorch source checkout. Installing ++an extra does not make unsupported Ethos-U nodes use CMSIS-NN automatically. ++See the ++[Arm examples](https://github.com/pytorch/executorch/tree/main/examples/arm) ++for that setup. + + To build a minimal wheel from source, set + `EXECUTORCH_BUILD_MINIMAL=1` when running `pip wheel` or `pip install`. +diff --git a/README.md b/README.md +--- a/README.md ++++ b/README.md +@@ -64,6 +64,11 @@ + pip install executorch + ``` + ++Backend export tools can require optional dependencies. For example, use ++`pip install 'executorch[ethos_u]'` for Ethos-U AOT export. Embedded ++toolchains, simulators, and target runtimes are installed separately; see the ++[Arm examples](examples/arm/README.md) for the complete flow. ++ + For platform-specific setup (Android, iOS, embedded systems), see the [Quick Start](https://docs.pytorch.org/executorch/main/quick-start-section.html) documentation for additional info. + + ### Export and Deploy in 3 Steps +diff --git a/backends/arm/requirements-arm-tosa.txt b/backends/arm/requirements-arm-tosa.txt +--- a/backends/arm/requirements-arm-tosa.txt ++++ b/backends/arm/requirements-arm-tosa.txt +@@ -6,7 +6,7 @@ + # These dependencies need to match pyproject.toml + + ml_dtypes == 0.5.1 +-flatbuffers == 24.3.25 ++flatbuffers == 25.2.10 + tosa-adapter-model-explorer == 0.1.0 + ai-edge-model-explorer >= 0.1.16 + pytest-timeout == 2.4.0 +diff --git a/backends/arm/scripts/corstone_utils.cmake b/backends/arm/scripts/corstone_utils.cmake +--- a/backends/arm/scripts/corstone_utils.cmake ++++ b/backends/arm/scripts/corstone_utils.cmake +@@ -37,7 +37,7 @@ + BINARY_DIR + ${ETHOS_SDK_PATH} + SUBBUILD_DIR +- ${ETHOS_SDK_PATH}/../ethos_u-subbuild ++ ${CMAKE_BINARY_DIR}/ethos_u-subbuild + SOURCE_SUBDIR + none + ) +diff --git a/backends/arm/test/misc/test_mobilesam_export.py b/backends/arm/test/misc/test_mobilesam_export.py +new file mode 100644 +--- /dev/null ++++ b/backends/arm/test/misc/test_mobilesam_export.py +@@ -0,0 +1,44 @@ ++# Copyright 2026 Arm Limited and/or its affiliates. ++# ++# This source code is licensed under the BSD-style license found in the ++# LICENSE file in the root directory of this source tree. ++ ++from pathlib import Path ++ ++import pytest ++ ++from examples.arm.mobilesam_prompt_segmentation_example_ethos_u.model_export.export_mobilesam import ( ++ INPUT_SIZE, ++ parse_args, ++ resolve_point_prompts, ++) ++ ++ ++def test_minimal_export_arguments() -> None: ++ args = parse_args( ++ [ ++ "--output-path", ++ "model.pte", ++ "--image", ++ "image.jpg", ++ "--point", ++ "219", ++ "193", ++ ] ++ ) ++ ++ assert args.output_path == Path("model.pte") ++ assert args.image == Path("image.jpg") ++ assert args.point == [[219.0, 193.0]] ++ assert not args.local_files_only ++ assert not args.quantized_io ++ ++ ++def test_point_prompts_must_be_inside_fixed_input() -> None: ++ assert resolve_point_prompts([[0, 0], [INPUT_SIZE - 1, INPUT_SIZE - 1]]) == [ ++ (0.0, 0.0), ++ (447.0, 447.0), ++ ] ++ ++ with pytest.raises(ValueError, match="must be inside"): ++ resolve_point_prompts([[INPUT_SIZE, 0]]) +diff --git a/backends/arm/test/misc/test_mobilesam_visualization.py b/backends/arm/test/misc/test_mobilesam_visualization.py +new file mode 100644 +--- /dev/null ++++ b/backends/arm/test/misc/test_mobilesam_visualization.py +@@ -0,0 +1,24 @@ ++# Copyright 2026 Arm Limited and/or its affiliates. ++# ++# This source code is licensed under the BSD-style license found in the ++# LICENSE file in the root directory of this source tree. ++ ++from pathlib import Path ++ ++from examples.arm.mobilesam_prompt_segmentation_example_ethos_u.runtime.visualize_fvp_output import ( ++ parse_rle_mask, ++) ++ ++ ++def test_parse_rle_mask_tolerates_non_utf8_fvp_output(tmp_path: Path) -> None: ++ log_path = tmp_path / "fvp.log" ++ log_path.write_bytes( ++ b"\xffFVP startup noise\n" ++ b"Segmentation mask RLE begin\n" ++ b"Segmentation mask RLE chunk 0:2,1:3,\n" ++ b"\x80UART noise\n" ++ b"Segmentation mask RLE chunk 0:1,\n" ++ b"Segmentation mask RLE end\n" ++ ) ++ ++ assert parse_rle_mask(log_path, expected_pixels=6) == [0, 0, 1, 1, 1, 0] +diff --git a/backends/arm/test/test_arm_ootb.sh b/backends/arm/test/test_arm_ootb.sh +--- a/backends/arm/test/test_arm_ootb.sh ++++ b/backends/arm/test/test_arm_ootb.sh +@@ -170,10 +170,8 @@ + local example_dir="${et_root_dir}/examples/arm/mobilesam_prompt_segmentation_example_ethos_u" + local work_root="${et_root_dir}/arm_test/mobilesam_ootb_smoke" + local export_dir="${work_root}/export" +- local artifact_dir="${work_root}/artifacts" +- local debug_dir="${work_root}/debug" ++ local debug_dir="${export_dir}/debug" + local et_build_dir="${work_root}/cmake-out-arm" +- local quantized_aot_build_dir="${work_root}/quantized_ops_aot" + local build_dir="${work_root}/runtime" + local mobile_sam_source="${work_root}/mobile_sam/source" + local image_path="${et_root_dir}/examples/models/dinov2/dog.jpg" +@@ -181,11 +179,10 @@ + local metadata_path="${export_dir}/mobilesam_prompt_smoke.json" + local fvp_log="${work_root}/fvp.log" + local toolchain_file="${et_root_dir}/examples/arm/ethos-u-setup/arm-none-eabi-gcc.cmake" +- local input_size=448 + local fvp_timelimit="${FVP_TIMELIMIT:-300}" + echo "${FUNCNAME}: Work directory: ${work_root}; existing artifacts will be reused if present" + +- mkdir -p "${export_dir}" "${artifact_dir}" "${debug_dir}" "${build_dir}" ++ mkdir -p "${export_dir}" "${build_dir}" + + setup_path_script=${et_root_dir}/examples/arm/arm-scratch/setup_path.sh + source ${setup_path_script} +@@ -197,60 +194,26 @@ + cmake --preset arm-baremetal -B "${et_build_dir}" + cmake --build "${et_build_dir}" --target install -j"$n_proc" + +- echo "${FUNCNAME}: Building host quantized AOT library" + local python_executable + python_executable="$(python3 -c 'import sys; print(sys.executable)')" +- cmake \ +- -S "${et_root_dir}" \ +- -B "${quantized_aot_build_dir}" \ +- -DCMAKE_BUILD_TYPE=Release \ +- -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ +- -DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=ON \ +- -DEXECUTORCH_BUILD_XNNPACK=OFF \ +- -DPYTHON_EXECUTABLE="${python_executable}" +- cmake --build "${quantized_aot_build_dir}" --target quantized_ops_aot_lib -j"$n_proc" +- +- local quantized_ops_library +- quantized_ops_library="$( +- find "${quantized_aot_build_dir}/kernels/quantized" \ +- -name 'libquantized_ops_aot_lib.*' \ +- -type f \ +- -print \ +- -quit +- )" +- [[ -n "${quantized_ops_library}" ]] || { +- echo "${FUNCNAME}: Missing quantized AOT library under ${quantized_aot_build_dir}" +- return 1 +- } +- +- echo "${FUNCNAME}: Installing example requirements" +- pip install -r "${example_dir}/requirements.txt" + + echo "${FUNCNAME}: Preparing pinned MobileSAM source" + python3 "${example_dir}/model_export/prepare_mobilesam.py" \ + --source-dir "${mobile_sam_source}" + + echo "${FUNCNAME}: Exporting quantized MobileSAM PTE" +- env EXECUTORCH_QUANTIZED_OPS_AOT_LIBRARY="${quantized_ops_library}" \ +- python3 "${example_dir}/model_export/export_mobilesam.py" \ ++ python3 "${example_dir}/model_export/export_mobilesam.py" \ + --output-path "${pte_path}" \ +- --calibration-image "${image_path}" \ +- --eval-image "${image_path}" \ ++ --image "${image_path}" \ + --mobile-sam-source "${mobile_sam_source}" \ +- --input-size "${input_size}" \ +- --point 219 193 \ +- --num-calibration-samples 1 \ +- --num-eval-samples 1 \ +- --num-debug-samples 1 \ +- --minimum-fp32-quantized-iou 0.9 \ +- --artifact-dir "${artifact_dir}" \ +- --debug-output-dir "${debug_dir}" ++ --point 219 193 + + for artifact in \ + "${pte_path}" \ + "${metadata_path}" \ + "${export_dir}/mobilesam_prompt_smoke_delegation.txt" \ +- "${export_dir}/mobilesam_prompt_smoke_metrics.json"; do ++ "${export_dir}/mobilesam_prompt_smoke_metrics.json" \ ++ "${debug_dir}/dog/quantized_mask.png"; do + [[ -f "${artifact}" ]] || { + echo "${FUNCNAME}: Missing export artifact ${artifact}" + return 1 +@@ -265,9 +228,7 @@ + -B "${build_dir}" \ + -DCMAKE_TOOLCHAIN_FILE="${toolchain_file}" \ + -DET_PTE_FILE_PATH="${pte_path}" \ +- -DMODEL_METADATA_PATH="${metadata_path}" \ + -DIMAGE_PATH="${image_path}" \ +- -DMASK_THRESHOLD=0.0 \ + -DET_SEGMENTATION_DUMP_MASK=ON \ + -DPYTHON_EXECUTABLE="${python_executable}" \ + -DET_BUILD_DIR_PATH="${et_build_dir}" +diff --git a/docs/source/backends/arm-ethos-u/tutorials/ethos-u-getting-started.md b/docs/source/backends/arm-ethos-u/tutorials/ethos-u-getting-started.md +--- a/docs/source/backends/arm-ethos-u/tutorials/ethos-u-getting-started.md ++++ b/docs/source/backends/arm-ethos-u/tutorials/ethos-u-getting-started.md +@@ -35,10 +35,25 @@ + + ### Software + +-First, you will need to install ExecuTorch. Please follow the recommended tutorials to set up a working ExecuTorch development environment. +- +-In addition to this, you need to install a number of SDK dependencies for generating Ethos-U command streams. Scripts to automate this are available in the main [ExecuTorch repository](https://github.com/pytorch/executorch/tree/main/examples/arm/). +-To install Ethos-U dependencies, run ++Use a source checkout because this tutorial builds an embedded target runtime. ++From the repository root, create a Python environment and install the current ++checkout with the Ethos-U AOT dependencies: ++ ++```bash ++python3.12 -m venv .venv ++source .venv/bin/activate ++./install_executorch.sh --optional-dependency ethos_u ++``` ++ ++The released-package equivalent for host-side export is ++`pip install 'executorch[ethos_u]'`. The `ethos_u` extra does not install an Arm ++cross compiler, FVP, or target runtime. There is no `cortex_m` package extra; ++CMSIS-NN is a separate Cortex-M export and runtime path. Installing an extra ++does not make unsupported Ethos-U nodes use CMSIS-NN automatically. ++ ++Install the Arm cross-compilation toolchain, simulators, and remaining backend ++dependencies with: ++ + ```bash + ./examples/arm/setup.sh --i-agree-to-the-contained-eula + ``` +diff --git a/docs/source/getting-started.md b/docs/source/getting-started.md +--- a/docs/source/getting-started.md ++++ b/docs/source/getting-started.md +@@ -23,7 +23,22 @@ + pip install executorch + ``` + +-To build the framework from source, see [Building From Source](using-executorch-building-from-source.md). Backend delegates may require additional dependencies. See the appropriate backend documentation for more information. ++Some backends provide optional Python dependencies for ahead-of-time export. ++For example, install the Ethos-U export dependencies with: ++ ++```bash ++pip install 'executorch[ethos_u]' ++``` ++ ++The extra installs host-side AOT tools; it does not install an embedded ++toolchain, simulator, or target runtime. There is no `cortex_m` package extra. ++CMSIS-NN is a separate Cortex-M export and runtime path that requires a source ++checkout. Installing an extra does not make unsupported Ethos-U nodes use ++CMSIS-NN automatically. See the ++[Arm examples](https://github.com/pytorch/executorch/tree/main/examples/arm) ++for the complete setup. ++ ++To build the framework from source, see [Building From Source](using-executorch-building-from-source.md). Other backend delegates may require additional dependencies. See the appropriate backend documentation for more information. + + > **_NOTE:_** On Windows, ExecuTorch requires a [Visual Studio Developer Powershell](https://learn.microsoft.com/en-us/visualstudio/ide/reference/command-prompt-powershell?view=vs-2022). Running from outside of a developer prompt will manifest as errors related to CL.exe. + +diff --git a/docs/source/using-executorch-building-from-source.md b/docs/source/using-executorch-building-from-source.md +--- a/docs/source/using-executorch-building-from-source.md ++++ b/docs/source/using-executorch-building-from-source.md +@@ -79,7 +79,27 @@ + * `--clean`: Removes build artifacts. + * `--editable`: Install the ExecuTorch python package in editable mode (see [Editable Install](#editable-install)). + * `--minimal`: Install only the minimal set of dependencies required to run ExecuTorch. Do not install dependencies for examples. ++ * `--optional-dependency `: Install an optional Python dependency set. ++ Repeat the flag to select more than one. Supported names are `ethos_u`, ++ `vgf`, and `openvino`. + * `--use-pt-pinned-commit`: Install the pinned PyTorch commit or release version. When not specified, the latest PyTorch nightly build is installed. ++ ++ For example, install the current checkout with the Ethos-U AOT dependencies: ++ ++ ```bash ++ ./install_executorch.sh --optional-dependency ethos_u ++ ``` ++ ++ After the base dependencies have already been installed, the equivalent ++ editable package command is: ++ ++ ```bash ++ pip install -e '.[ethos_u]' --no-build-isolation ++ ``` ++ ++ Optional dependency sets install host-side Python tools only. Embedded ++ toolchains, simulators, and target runtimes are configured separately by the ++ backend setup and build instructions. + + For Intel-based macOS systems, use `--use-pt-pinned-commit --minimal`. As PyTorch does not provide pre-built binaries for Intel Mac, installation requires building PyTorch from source. Instructions can be found in [PyTorch Installation](https://github.com/pytorch/pytorch#installation). + +diff --git a/examples/arm/README.md b/examples/arm/README.md +--- a/examples/arm/README.md ++++ b/examples/arm/README.md +@@ -10,6 +10,28 @@ + This directory contains documentation and scripts to + help you setup and run a PyTorch model on the Arm backend + via ExecuTorch. ++ ++## Python package setup ++ ++For Ethos-U examples, install the current checkout and its AOT dependencies in ++a clean Python environment: ++ ++```bash ++python3.12 -m venv .venv ++source .venv/bin/activate ++./install_executorch.sh --optional-dependency ethos_u ++``` ++ ++After the base dependencies are installed, the equivalent editable package ++command is `pip install -e '.[ethos_u]' --no-build-isolation`. The `ethos_u` ++extra provides host-side export dependencies; it does not install the Arm ++toolchain, FVPs, or target runtime. ++ ++There is no `cortex_m` package extra. CMSIS-NN is a separate Cortex-M export ++and runtime path that uses code and build configuration from the ExecuTorch ++source checkout. Installing an extra does not make unsupported Ethos-U nodes ++use CMSIS-NN automatically. Run `setup.sh` below, then follow the relevant ++Cortex-M example. + + ## setup.sh + +diff --git a/examples/arm/ethos_u_minimal_example.ipynb b/examples/arm/ethos_u_minimal_example.ipynb +--- a/examples/arm/ethos_u_minimal_example.ipynb ++++ b/examples/arm/ethos_u_minimal_example.ipynb +@@ -21,11 +21,17 @@ + "This guide demonstrates the full flow for running a module on Arm Ethos-U55 using ExecuTorch.\n", + "Tested on Linux x86_64 and macOS aarch64. If something is not working for you, please raise a GitHub issue and tag Arm.\n", + "\n", +- "Before you begin:\n", +- "1. (In a clean virtual environment with a compatible Python version) Install executorch using `./install_executorch.sh`\n", +- "2. Install Arm cross-compilation toolchain and simulators using `./examples/arm/setup.sh --i-agree-to-the-contained-eula`\n", +- "\n", +- "With all commands executed from the base `executorch` folder.\n", ++ "Before you begin, run these commands from the base `executorch` folder:\n", ++ "\n", ++ "```bash\n", ++ "python3.12 -m venv .venv\n", ++ "source .venv/bin/activate\n", ++ "./install_executorch.sh --optional-dependency ethos_u\n", ++ "./examples/arm/setup.sh --i-agree-to-the-contained-eula\n", ++ "source examples/arm/arm-scratch/setup_path.sh\n", ++ "```\n", ++ "\n", ++ "The `ethos_u` extra installs the host-side AOT dependencies. The Arm setup script installs the cross compiler and FVPs needed later in the notebook. There is no `cortex_m` package extra. CMSIS-NN is a separate Cortex-M export and runtime path; installing an extra does not make unsupported Ethos-U nodes use CMSIS-NN automatically.\n", + "\n", + "\n", + "\n", +diff --git a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/README.md b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/README.md +--- a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/README.md ++++ b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/README.md +@@ -4,17 +4,75 @@ + ExecuTorch for transformer-based prompt segmentation. MobileSAM predicts a + binary mask for fixed positive point prompts rather than semantic class IDs. + The host debug flow validates quantization by comparing FP32 and quantized +-masks, with an optional binary reference mask when one is available. ++masks before lowering. + +-It covers: ++**Prompt contract:** the runtime accepts only an image tensor. The positive ++point coordinates are embedded in the `.pte` during export. You can reuse the ++`.pte` for a new image, but changing the points requires re-exporting the model. + +-- Loading the MobileSAM `vit_t` checkpoint. +-- Freezing one or more positive point prompts into the exported graph. +-- Applying post-training quantization with the Ethos-U quantizer. +-- Lowering the quantized model to an Ethos-U85-256 ExecuTorch program. +-- Producing validation and debugging artifacts such as masks, overlays, +- mismatch heatmaps, metrics, and delegation summaries. +-- Building a bare-metal Corstone-320 runtime app and running it on FVP. ++## Clean-clone setup ++ ++Run every command in this guide from the ExecuTorch repository root. Use the ++checkout's installer rather than installing an unrelated `executorch` wheel ++from PyPI: ++ ++```bash ++python3.12 -m venv .venv ++source .venv/bin/activate ++ ++./install_executorch.sh --optional-dependency ethos_u ++ ++./examples/arm/setup.sh --i-agree-to-the-contained-eula ++source examples/arm/arm-scratch/setup_path.sh ++``` ++ ++`install_executorch.sh` installs the current checkout and Ethos-U Python tools ++into the active virtual environment. `examples/arm/setup.sh` installs the ++pinned Arm GNU bare-metal toolchain and Corstone FVP used by this example. ++For a later editable reinstall, the equivalent package command is ++`python -m pip install -e '.[ethos_u]' --no-build-isolation`. There is no ++`cortex_m` package extra; CMSIS-NN kernels are selected when building a target ++runtime, and this example does not need them because its graph is fully ++delegated to Ethos-U. ++ ++On macOS, install the ++[FVPs-on-Mac](https://github.com/Arm-Examples/FVPs-on-Mac) Docker wrapper and ++add its `bin` directory to `PATH` before running the setup or smoke commands. ++Run `command -v FVP_Corstone_SSE-320` and confirm it resolves inside the ++FVPs-on-Mac `bin` directory rather than to a downloaded Linux FVP binary. ++ ++## One-command end-to-end smoke test ++ ++After setup, export, build, run on Corstone-320 FVP, and validate the target ++mask with: ++ ++```bash ++./backends/arm/test/test_arm_ootb.sh run_mobilesam_e2e_ethos_u ++``` ++ ++The smoke test uses the demo image for calibration and validation. Its main ++outputs are: ++ ++- `.pte`: `arm_test/mobilesam_ootb_smoke/export/mobilesam_prompt_smoke.pte` ++- Host quantized mask: ++ `arm_test/mobilesam_ootb_smoke/export/debug/dog/quantized_mask.png` ++- FVP log: `arm_test/mobilesam_ootb_smoke/fvp.log` ++- Side-by-side result: ++ `arm_test/mobilesam_ootb_smoke/fvp_visual/fvp_comparison.png` ++- FVP validation metrics: ++ `arm_test/mobilesam_ootb_smoke/fvp_visual/metrics.json` ++ ++## What the example does ++ ++1. Loads the pinned official MobileSAM `vit_t` source and checkpoint. ++2. Builds a wrapper with fixed positive point prompts. ++3. Calibrates PT2E quantization with `EthosUQuantizer`. ++4. Applies a stable softmax decomposition to transformer attention blocks. ++5. Lowers the complete quantized graph with `EthosUPartitioner`. ++6. Writes a `.pte`, builds the bare-metal runtime, executes it on FVP, and ++ compares the target mask with the host quantized result. ++7. Optionally builds a Zephyr application for the Alif Ensemble E8 DevKit and ++ runs the same export on Ethos-U85-256 hardware. + + The default export uses a reduced `448x448` image input and returns one + low-resolution `[1, 1, 112, 112]` mask-logit tensor. The example prepares the +@@ -31,6 +89,12 @@ + numerically sensitive after export and quantization, so this example keeps the + target graph focused on the fixed-prompt image encoder and mask decoder. + ++## Manual workflow ++ ++For the educational flow and configurable export options, continue with ++`model_export/README.md`, then build and execute the generated program using ++`runtime/README.md`. ++ + ## Layout + + - `model_export/prepare_mobilesam.py` - Prepares the pinned external MobileSAM +@@ -41,3 +105,5 @@ + Corstone-320 FVP execution. + - `runtime/visualize_fvp_output.py` - Decodes the target mask dump, creates an + overlay, and compares FVP output with the host quantized mask. ++- `../../../zephyr/samples/mobilesam-ethosu/README.md` - Alif Ensemble E8 ++ build, flash, execution, and visualization instructions. +diff --git a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/README.md b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/README.md +--- a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/README.md ++++ b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/README.md +@@ -1,192 +1,96 @@ +-# MobileSAM Export, Quantization, and Debugging ++# Export MobileSAM for Ethos-U + +-This directory exports MobileSAM `vit_t` for the ExecuTorch Ethos-U backend. +-The exporter freezes one or more positive point prompts into the graph, so the +-runtime app has one tensor input: the preprocessed image. The model returns one +-mask-logit tensor for those prompts. The default export uses a `448x448` input, +-which produces a `112x112` mask-logit tensor. ++This flow exports MobileSAM `vit_t` as a fixed-prompt, fully delegated ++Ethos-U85 program. The image remains a runtime input, while positive point ++coordinates are embedded in the `.pte`. Changing the points requires another ++export. + +-Production SAM applications usually pass prompts dynamically from a UI or +-tracking pipeline. This example freezes the prompt embeddings so the FVP +-runtime stays small and demonstrates the Ethos-U flow with the same image +-encoder and mask decoder used by MobileSAM. ++## Export flow + +-## Requirements ++1. Load the pinned official MobileSAM source and checkpoint. ++2. Freeze one or more positive point prompts into the model. ++3. Use the demonstration image for PT2E calibration. ++4. Quantize globally to A8W8 and use A16W8 for TinyViT attention. ++5. Compare FP32 and quantized masks and require at least `0.9` IoU. ++6. Lower the complete graph with `EthosUPartitioner` and save the `.pte`. ++7. Write TOSA/Vela intermediates, masks, overlays, metrics, and delegation ++ information beside the model. + +-- Python 3.10+ with `executorch`. +-- Dependencies from +- `examples/arm/mobilesam_prompt_segmentation_example_ethos_u/requirements.txt`. +-- Git and internet access to prepare the pinned external MobileSAM checkout and +- download the official checkpoint, unless both are already cached. +-- Ethos-U dependencies from `examples/arm/setup.sh`. +- +-MobileSAM's A16W8 attention requires `ethos-u-vela>=5.1.0`. Vela 5.0 produces +-incorrect Ethos-U85 INT16 reductions, so the exporter rejects that version +-before generating a `.pte`. +- +-## Export +- +-Run from the ExecuTorch repo root: ++Complete the clean-clone setup in the top-level example `README.md`, then run ++from the ExecuTorch repository root: + + ```bash +-MOBILE_SAM_SOURCE="$HOME/.cache/executorch/mobilesam/f706ad9c4eb7f219c00d9050e46328518ffb65d2/source" ++MOBILE_SAM_WORK_DIR="${MOBILE_SAM_WORK_DIR:-$PWD/arm_test/mobilesam_manual}" ++MOBILE_SAM_SOURCE="$MOBILE_SAM_WORK_DIR/mobile_sam/source" ++MOBILE_SAM_IMAGE="$PWD/examples/models/dinov2/dog.jpg" ++MOBILE_SAM_PTE="$MOBILE_SAM_WORK_DIR/export/mobilesam_point_ethos_u85_448.pte" ++ + python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/prepare_mobilesam.py \ + --source-dir "$MOBILE_SAM_SOURCE" + + python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/export_mobilesam.py \ +- --output-path ./mobilesam_point_ethos_u85_448.pte \ +- --mobile-sam-source "$MOBILE_SAM_SOURCE" \ +- --calibration-image examples/models/dinov2/dog.jpg \ +- --eval-image examples/models/dinov2/dog.jpg \ ++ --output-path "$MOBILE_SAM_PTE" \ ++ --image "$MOBILE_SAM_IMAGE" \ + --point 219 193 \ +- --artifact-dir ./mobilesam_point_artifacts \ +- --debug-output-dir ./mobilesam_point_debug ++ --mobile-sam-source "$MOBILE_SAM_SOURCE" + ``` + +-The default configuration is: ++The example intentionally fixes the deployment configuration: + +-- Model source: `https://github.com/ChaoningZhang/MobileSAM` +-- MobileSAM source revision: `f706ad9c4eb7f219c00d9050e46328518ffb65d2` +-- External source patch: `0001-Make-TinyViT-image-size-configurable.patch` +-- Checkpoint URL: +- `https://github.com/ChaoningZhang/MobileSAM/raw/f706ad9c4eb7f219c00d9050e46328518ffb65d2/weights/mobile_sam.pt` +-- Checkpoint SHA256: +- `6dbb90523a35330fedd7f1d3dfc66f995213d81b29a5ca8108dbcdd4e37d6c2f` +-- MobileSAM source-code license: Apache-2.0 +-- Static input shape: `[1, 3, 448, 448]` +-- Output mask logits shape: `[1, 1, 112, 112]` +-- Positive point prompt: `(224, 224)` in the padded `448x448` model input +- frame ++- Input: `[1, 3, 448, 448]` ++- Output mask logits: `[1, 1, 112, 112]` + - Target: `ethos-u85-256` +-- System config: `Ethos_U85_SYS_DRAM_Mid` ++- System configuration: `Ethos_U85_SYS_DRAM_Mid` + - Memory mode: `Dedicated_Sram_384KB` +-- Calibration samples: `4` +-- Validation samples: `4` ++- Mask threshold: `0.0` + +-The MobileSAM source and checkpoint are not redistributed by this example. The +-preparation script clones the pinned official source into a managed external +-cache and applies the configurable-input patch. The exporter downloads the +-pinned official checkpoint and verifies its SHA256 unless `--checkpoint-path` +-is provided. The source repository and checkpoint may have separate terms; the +-export metadata records only the source-code license and does not assign a +-license to the checkpoint. ++Repeat `--point X Y` to export a multi-point positive prompt. Pass ++`--local-files-only` after the source and checkpoint have been cached. Pass a ++local checkpoint with `--checkpoint-path` when network access is unavailable. + +-For a quick offline smoke test after caching the official checkpoint and source +-checkout: +- +-```bash +-python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/prepare_mobilesam.py \ +- --source-dir "$MOBILE_SAM_SOURCE" \ +- --local-files-only +- +-python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/export_mobilesam.py \ +- --output-path /tmp/mobilesam_smoke.pte \ +- --local-files-only \ +- --mobile-sam-source "$MOBILE_SAM_SOURCE" \ +- --calibration-image examples/models/dinov2/dog.jpg \ +- --eval-image examples/models/dinov2/dog.jpg \ +- --point 219 193 \ +- --num-calibration-samples 1 \ +- --num-eval-samples 1 \ +- --minimum-fp32-quantized-iou 0.9 +-``` +- +-Repeat `--point X Y` to freeze a multi-point prompt into the graph. Validate +-every prompt set because adding positive points can substantially change the +-predicted object after quantization: +- +-```bash +-python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/export_mobilesam.py \ +- --output-path /tmp/mobilesam_multipoint_smoke.pte \ +- --local-files-only \ +- --mobile-sam-source "$MOBILE_SAM_SOURCE" \ +- --calibration-image examples/models/dinov2/dog.jpg \ +- --eval-image examples/models/dinov2/dog.jpg \ +- --point 166 158 \ +- --point 219 193 \ +- --point 289 184 \ +- --num-calibration-samples 1 \ +- --num-eval-samples 1 \ +- --debug-output-dir /tmp/mobilesam_multipoint_debug +-``` +- +-Pass `--input-size 1024` to reproduce the original MobileSAM resolution. Smaller +-inputs export and run faster, but very small inputs such as `224` or `256` +-usually produce lower-quality masks because the checkpoint was trained for +-`1024x1024` images. +- +-To validate against a known binary mask, pass one `--eval-mask` per +-`--eval-image`. Non-zero mask pixels are treated as foreground. When no +-reference mask is provided, validation reports FP32/quantized mask agreement. +-Use `--minimum-fp32-quantized-iou` in automated runs to reject an inaccurate +-quantized model before lowering. +- +-The export flow: +- +-1. Loads the MobileSAM `vit_t` checkpoint through the patched external API. +-2. Builds a fixed-prompt wrapper containing the image encoder and mask decoder. +-3. Calibrates PT2E quantization with `EthosUQuantizer`. +-4. Uses stable softmax decomposition for transformer attention blocks. +-5. Lowers the quantized graph with `EthosUPartitioner`. +-6. Writes an ExecuTorch `.pte` program. +- +-The quantization recipe uses int8 activations and int8 weights globally, with +-A16W8 quantization for the TinyViT attention modules. Static int8 activation +-quantization collapses MobileSAM attention features, while the selective A16W8 +-attention path preserves mask quality and still lowers as one Ethos-U delegate. ++The Alif E8 Zephyr sample additionally requires `--quantized-io`. This exposes ++int8 model input and output tensors and records their quantization parameters ++in the metadata JSON. + + ## Outputs + +-For `--output-path ./mobilesam_point_ethos_u85_448.pte`, the script writes: ++For the command above, the export directory contains: + +-- `mobilesam_point_ethos_u85_448.pte` - Ethos-U-ready ExecuTorch program. +-- `mobilesam_point_ethos_u85_448.json` - Export metadata. +-- `mobilesam_point_ethos_u85_448_metrics.json` - FP32/quantized mask +- agreement and optional reference-mask IoU. +-- `mobilesam_point_ethos_u85_448_delegation.txt` - Operator delegation +- summary. +-- `mobilesam_point_artifacts/` - Optional TOSA/Vela intermediate artifacts. +-- `mobilesam_point_debug/` - Optional per-sample masks, overlays, mismatch +- heatmaps, and mask summaries. ++- `mobilesam_point_ethos_u85_448.pte` - ExecuTorch program. ++- `mobilesam_point_ethos_u85_448.json` - shapes, preprocessing, prompt, and ++ provenance metadata. ++- `mobilesam_point_ethos_u85_448_metrics.json` - FP32/quantized mask agreement. ++- `mobilesam_point_ethos_u85_448_delegation.txt` - delegation summary. ++- `artifacts/` - TOSA and Vela intermediate files. ++- `debug/dog/` - input, FP32/quantized masks, and colored overlays. + +-## Interpreting the Debug Artifacts ++The host quantized mask used for FVP validation is: + +-Each debug sample contains: ++```text ++$MOBILE_SAM_WORK_DIR/export/debug/dog/quantized_mask.png ++``` + +-- `input.png` - The resized RGB input used by the exported model. +-- `reference_mask.png` - Optional binary reference mask resized to the model +- output-mask size. +-- `fp32_mask.png` - Host-side FP32 model prediction. +-- `fp32_overlay.png` - Colored FP32 prediction blended over the input image. +-- `quantized_mask.png` - Host-side PT2E quantized prediction before lowering. +-- `quantized_overlay.png` - Colored quantized prediction blended over the input +- image. +-- `mismatch_heatmap.png` - Green for FP32/quantized agreement and red for +- mismatch. +-- `mask_summary.json` - Foreground/background pixel counts and FP32/quantized +- IoU. ++## Source and checkpoint + +-The runtime app thresholds mask logits on target, logs a mask hash and +-foreground/background counts, and can dump the thresholded mask as RLE chunks. +-Host-side debug masks are intentionally generated before lowering so users can +-inspect quantization quality without needing target-side image output. ++The preparation script creates a managed external checkout of ++`https://github.com/ChaoningZhang/MobileSAM` at revision ++`f706ad9c4eb7f219c00d9050e46328518ffb65d2` and applies ++`0001-Make-TinyViT-image-size-configurable.patch` there. The exporter downloads ++the official `mobile_sam.pt` checkpoint and verifies SHA256 ++`6dbb90523a35330fedd7f1d3dfc66f995213d81b29a5ca8108dbcdd4e37d6c2f`. ++ ++Neither source nor weights are redistributed in ExecuTorch. The source is ++Apache-2.0; the source repository and checkpoint can have separate terms, so ++the export metadata does not assign the source license to the checkpoint. + + ## Limitations + +-- The example freezes positive point prompts into the exported graph to keep +- the target app to a single image input. Changing the prompt requires +- re-exporting the `.pte`. +-- The export uses `multimask_output=False` and does not include candidate-mask +- argmax selection, upsampling, or thresholding in the graph. Keep those steps +- in host-side or target-side post-processing when comparing mask quality. +-- The runtime app logs a mask hash and foreground/background counts; it does +- not render a color image on target. ++- The `.pte` accepts one image tensor. Changing prompt coordinates requires ++ re-exporting. ++- The graph uses `multimask_output=False`; candidate-mask selection, upsampling, ++ and thresholding remain post-processing steps. ++- `448x448` is the smallest locally tested size that retained at least `0.95` ++ FP32/quantized IoU on the demonstration image. + - The first supported runtime target is Corstone-320/Ethos-U85-256. +-- Reduced input sizes require the patch applied by `prepare_mobilesam.py`; the +- MobileSAM source remains outside the ExecuTorch checkout. +-- The default `448x448` input is the smallest size in the local sweep that +- retained at least `0.95` host quantized/FP32 mask IoU on the demo image. +- `512x512` retained about `0.975` IoU, while smaller sizes were inconsistent. +-- MobileSAM PTQ is sensitive to the calibration set and the mask-logit +- threshold. Inspect `*_metrics.json` and the debug overlays before treating +- the quantized mask as an accuracy result. ++- Calibration uses the demonstration image to keep the example quick. Validate ++ with representative data before using the quantized model in a product. +diff --git a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/export_mobilesam.py b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/export_mobilesam.py +--- a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/export_mobilesam.py ++++ b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/export_mobilesam.py +@@ -10,7 +10,6 @@ + import importlib + import inspect + import json +-import os + import sys + import urllib.error + import urllib.request +@@ -21,7 +20,6 @@ + + import numpy as np + import torch +-import tqdm # type: ignore[import] + from executorch.backends.arm.common.pipeline_config import ( + ArmPassPipelineConfig, + SoftmaxDecompositionConfig, +@@ -38,6 +36,7 @@ + ExecutorchBackendConfig, + to_edge_transform_and_lower, + ) ++from executorch.exir.passes.quantize_io_pass import QuantizeInputs, QuantizeOutputs + from executorch.extension.export_util.utils import save_pte_program + from packaging.version import Version + from PIL import Image +@@ -49,6 +48,7 @@ + MOBILE_SAM_SOURCE_URL = "https://github.com/ChaoningZhang/MobileSAM" + MOBILE_SAM_SOURCE_REVISION = "f706ad9c4eb7f219c00d9050e46328518ffb65d2" + MOBILE_SAM_PATCH = "0001-Make-TinyViT-image-size-configurable.patch" ++MOBILE_SAM_SOURCE_LICENSE = "Apache-2.0" + DEFAULT_CHECKPOINT_FILENAME = "mobile_sam.pt" + DEFAULT_CHECKPOINT_URL = ( + f"{MOBILE_SAM_SOURCE_URL}/raw/{MOBILE_SAM_SOURCE_REVISION}/weights/" +@@ -57,10 +57,17 @@ + DEFAULT_CHECKPOINT_SHA256 = ( + "6dbb90523a35330fedd7f1d3dfc66f995213d81b29a5ca8108dbcdd4e37d6c2f" + ) +-MOBILE_SAM_SOURCE_LICENSE = "Apache-2.0" ++ ++INPUT_SIZE = 448 ++MASK_THRESHOLD = 0.0 ++MINIMUM_FP32_QUANTIZED_IOU = 0.9 ++MINIMUM_FP32_QUANTIZED_PIXEL_AGREEMENT = 0.96 ++MINIMUM_MASK_QUALITY = 0.8 ++MINIMUM_VALIDATION_MASK_QUALITY = 0.9 ++TARGET = "ethos-u85-256" ++SYSTEM_CONFIG = "Ethos_U85_SYS_DRAM_Mid" ++MEMORY_MODE = "Dedicated_Sram_384KB" + MINIMUM_VELA_VERSION = Version("5.1.0") +-DEFAULT_INPUT_SIZE = 448 +-MOBILE_SAM_INPUT_ALIGNMENT = 16 + + + @dataclass +@@ -68,197 +75,80 @@ + name: str + image: Image.Image + pixel_values: torch.Tensor +- labels: torch.Tensor | None +- +- +-def load_mobile_sam( +- checkpoint_path: str, +- mobile_sam_source: str | None, +- input_size: int, +-) -> torch.nn.Module: +- mobile_sam_module = import_mobile_sam_module(mobile_sam_source) +- builder = mobile_sam_module.sam_model_registry["vit_t"] +- if "image_size" not in inspect.signature(builder).parameters: +- raise RuntimeError( +- "The MobileSAM checkout does not provide configurable image sizes. " +- "Run model_export/prepare_mobilesam.py and pass the prepared checkout " +- "with --mobile-sam-source." +- ) +- return builder(checkpoint=checkpoint_path, image_size=input_size).eval() +- +- +-class MobileSAMFixedPrompt(torch.nn.Module): +- image_encoder: Any +- mask_decoder: Any +- +- def __init__( +- self, +- sam: torch.nn.Module, +- point_prompts: list[tuple[float, float]], +- ) -> None: +- super().__init__() +- sam = cast(Any, sam) +- +- self.image_encoder = sam.image_encoder +- self.mask_decoder = sam.mask_decoder +- +- with torch.no_grad(): +- points = ( +- torch.tensor([point_prompts], dtype=torch.float32), +- torch.ones((1, len(point_prompts)), dtype=torch.int64), +- ) +- sparse_embeddings, dense_embeddings = sam.prompt_encoder( +- points=points, +- boxes=None, +- masks=None, +- ) +- image_pe = sam.prompt_encoder.get_dense_pe() +- +- self.register_buffer("sparse_prompt_embeddings", sparse_embeddings) +- self.register_buffer("dense_prompt_embeddings", dense_embeddings) +- self.register_buffer("image_pe", image_pe) +- +- def forward(self, pixel_values: torch.Tensor) -> torch.Tensor: +- image_embeddings = self.image_encoder(pixel_values) +- low_res_masks, _ = self.mask_decoder( +- image_embeddings=image_embeddings, +- image_pe=self.image_pe, +- sparse_prompt_embeddings=self.sparse_prompt_embeddings, +- dense_prompt_embeddings=self.dense_prompt_embeddings, +- multimask_output=False, +- ) +- return low_res_masks +- +- +-def parse_args() -> argparse.Namespace: ++ ++ ++def default_source_dir() -> Path: ++ return ( ++ Path.home() ++ / ".cache" ++ / "executorch" ++ / "mobilesam" ++ / MOBILE_SAM_SOURCE_REVISION ++ / "source" ++ ) ++ ++ ++def parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser( +- description="Export fixed-prompt MobileSAM segmentation for Ethos-U." +- ) +- parser.add_argument( +- "--checkpoint-path", +- default=None, +- help="Optional local MobileSAM checkpoint path.", +- ) +- parser.add_argument( +- "--mobile-sam-source", +- default=None, +- help=( +- "Optional MobileSAM checkout prepared by prepare_mobilesam.py. " +- "Used before importing the mobile_sam package." +- ), +- ) +- parser.add_argument( +- "--local-files-only", +- action="store_true", +- help="Load the checkpoint from the local cache only; do not download.", +- ) +- parser.add_argument( +- "--calibration-image", +- action="append", +- default=[], +- help="Local RGB image used for PTQ calibration. Can be repeated.", +- ) +- parser.add_argument( +- "--eval-image", +- action="append", +- default=[], +- help="Local RGB image used for validation/debugging. Can be repeated.", +- ) +- parser.add_argument( +- "--eval-mask", +- action="append", +- default=[], +- help=( +- "Optional binary reference mask used for validation/debugging. " +- "Can be repeated and must match --eval-image count when provided." +- ), ++ description="Export fixed-prompt MobileSAM object segmentation for Ethos-U85." ++ ) ++ parser.add_argument( ++ "--output-path", ++ type=Path, ++ required=True, ++ help="Output path for the ExecuTorch .pte program.", ++ ) ++ parser.add_argument( ++ "--image", ++ type=Path, ++ required=True, ++ help="RGB image used for calibration and host validation.", + ) + parser.add_argument( + "--point", + type=float, + nargs=2, + action="append", +- default=[], ++ required=True, + metavar=("X", "Y"), + help=( +- "Positive point prompt in the resized square input frame. " +- "Can be repeated for multi-point prompts." ++ "Positive point prompt in the square input frame. Repeat once per " ++ "independently segmented object." + ), +- ) +- parser.add_argument( +- "--mask-threshold", +- type=float, +- default=0.0, +- help="Mask-logit threshold used for metrics and debug masks.", +- ) +- parser.add_argument( +- "--output-path", +- type=str, +- required=True, +- help="Path to save the exported ExecuTorch program.", + ) + parser.add_argument( + "--input-size", + type=int, +- default=DEFAULT_INPUT_SIZE, +- help="Square MobileSAM input size. Must be divisible by 16.", +- ) +- parser.add_argument( +- "--num-calibration-samples", +- type=int, +- default=4, +- help="Number of local samples used for PTQ calibration.", +- ) +- parser.add_argument( +- "--num-eval-samples", +- type=int, +- default=4, +- help="Number of local samples used for host-side validation.", +- ) +- parser.add_argument( +- "--num-debug-samples", +- type=int, +- default=4, +- help="Number of validation samples written as visual debug artifacts.", +- ) +- parser.add_argument( +- "--minimum-fp32-quantized-iou", +- type=float, +- default=None, +- help="Fail before lowering when host quantized/FP32 mask IoU is lower.", +- ) +- parser.add_argument( +- "--target", +- default="ethos-u85-256", +- help="Ethos-U target passed to Vela.", +- ) +- parser.add_argument( +- "--system-config", +- default="Ethos_U85_SYS_DRAM_Mid", +- help="Vela system configuration.", +- ) +- parser.add_argument( +- "--memory-mode", +- default="Dedicated_Sram_384KB", +- help="Vela memory mode.", +- ) +- parser.add_argument( +- "--extra-vela-flag", +- action="append", +- default=[], +- help="Additional Vela flag. Can be provided multiple times.", +- ) +- parser.add_argument( +- "--artifact-dir", +- default=None, +- help="Optional directory for intermediate TOSA/Vela artifacts.", +- ) +- parser.add_argument( +- "--debug-output-dir", +- default=None, +- help="Optional directory for masks, overlays, and validation summaries.", +- ) +- return parser.parse_args() ++ default=INPUT_SIZE, ++ help=f"Square model input size (default: {INPUT_SIZE}).", ++ ) ++ parser.add_argument( ++ "--mobile-sam-source", ++ type=Path, ++ default=default_source_dir(), ++ help="MobileSAM checkout prepared by prepare_mobilesam.py.", ++ ) ++ parser.add_argument( ++ "--checkpoint-path", ++ type=Path, ++ help="Optional local MobileSAM checkpoint; otherwise use the pinned cache.", ++ ) ++ parser.add_argument( ++ "--local-files-only", ++ action="store_true", ++ help="Require the checkpoint to be present in the local cache.", ++ ) ++ parser.add_argument( ++ "--quantized-io", ++ action="store_true", ++ help="Expose int8 input/output tensors for the Alif E8 sample.", ++ ) ++ parser.add_argument( ++ "--multimask-output", ++ action="store_true", ++ help="Return three candidate masks and quality scores for each point.", ++ ) ++ return parser.parse_args(argv) + + + def write_json(path: Path, payload: dict[str, Any]) -> None: +@@ -283,24 +173,84 @@ + return installed_version + + +-def import_mobile_sam_module(mobile_sam_source: str | None) -> Any: +- if mobile_sam_source is not None: +- sys.path.insert(0, str(Path(mobile_sam_source).expanduser().resolve())) ++def import_mobile_sam_module(source_dir: Path) -> Any: ++ source_dir = source_dir.expanduser().resolve() ++ if not source_dir.is_dir(): ++ raise FileNotFoundError( ++ f"Prepared MobileSAM source not found: {source_dir}. Run " ++ "model_export/prepare_mobilesam.py first." ++ ) ++ sys.path.insert(0, str(source_dir)) + try: + return importlib.import_module("mobile_sam") + except ImportError as error: +- raise ImportError( +- "Could not import the patched mobile_sam package. Run " +- "model_export/prepare_mobilesam.py and pass its checkout with " +- "--mobile-sam-source." +- ) from error +- +- +-def find_module_type(module: torch.nn.Module, class_name: str) -> type[torch.nn.Module]: +- for child in module.modules(): +- if child.__class__.__name__ == class_name: +- return child.__class__ +- raise ValueError(f"Could not find module type {class_name} in {module.__class__}.") ++ raise ImportError(f"Could not import mobile_sam from {source_dir}.") from error ++ ++ ++def load_mobile_sam( ++ checkpoint_path: Path, ++ source_dir: Path, ++ image_size: int = INPUT_SIZE, ++) -> torch.nn.Module: ++ mobile_sam_module = import_mobile_sam_module(source_dir) ++ builder = mobile_sam_module.sam_model_registry["vit_t"] ++ if "image_size" not in inspect.signature(builder).parameters: ++ raise RuntimeError( ++ "The MobileSAM checkout does not support configurable image sizes. " ++ "Prepare it with model_export/prepare_mobilesam.py." ++ ) ++ return builder( ++ checkpoint=str(checkpoint_path), ++ image_size=image_size, ++ ).eval() ++ ++ ++class MobileSAMFixedPrompt(torch.nn.Module): ++ image_encoder: Any ++ mask_decoder: Any ++ ++ def __init__( ++ self, ++ sam: torch.nn.Module, ++ point_prompts: list[tuple[float, float]], ++ multimask_output: bool = False, ++ ) -> None: ++ super().__init__() ++ sam = cast(Any, sam) ++ self.image_encoder = sam.image_encoder ++ self.mask_decoder = sam.mask_decoder ++ self.multimask_output = multimask_output ++ ++ with torch.no_grad(): ++ points = ( ++ torch.tensor(point_prompts, dtype=torch.float32).unsqueeze(1), ++ torch.ones((len(point_prompts), 1), dtype=torch.int64), ++ ) ++ sparse_embeddings, dense_embeddings = sam.prompt_encoder( ++ points=points, ++ boxes=None, ++ masks=None, ++ ) ++ image_pe = sam.prompt_encoder.get_dense_pe() ++ ++ self.register_buffer("sparse_prompt_embeddings", sparse_embeddings) ++ self.register_buffer("dense_prompt_embeddings", dense_embeddings) ++ self.register_buffer("image_pe", image_pe) ++ ++ def forward( ++ self, pixel_values: torch.Tensor ++ ) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]: ++ image_embeddings = self.image_encoder(pixel_values) ++ low_res_masks, quality_scores = self.mask_decoder( ++ image_embeddings=image_embeddings, ++ image_pe=self.image_pe, ++ sparse_prompt_embeddings=self.sparse_prompt_embeddings, ++ dense_prompt_embeddings=self.dense_prompt_embeddings, ++ multimask_output=self.multimask_output, ++ ) ++ if self.multimask_output: ++ return low_res_masks, quality_scores ++ return low_res_masks + + + def file_sha256(path: Path) -> str: +@@ -311,27 +261,26 @@ + return digest.hexdigest() + + +-def default_checkpoint_cache_dir() -> Path: ++def default_checkpoint_path() -> Path: + return ( +- Path.home() / ".cache" / "executorch" / "mobilesam" / MOBILE_SAM_SOURCE_REVISION +- ) +- +- +-def verify_checkpoint(path: Path, expected_sha256: str) -> None: +- actual_sha256 = file_sha256(path) +- if actual_sha256 != expected_sha256: +- raise RuntimeError( +- f"Checkpoint SHA256 mismatch for {path}: expected {expected_sha256}, " +- f"got {actual_sha256}." +- ) +- +- +-def download_checkpoint(url: str, path: Path) -> None: ++ Path.home() ++ / ".cache" ++ / "executorch" ++ / "mobilesam" ++ / MOBILE_SAM_SOURCE_REVISION ++ / DEFAULT_CHECKPOINT_FILENAME ++ ) ++ ++ ++def download_checkpoint(path: Path) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + temp_path = path.with_suffix(path.suffix + ".tmp") + try: + with ( +- urllib.request.urlopen(url, timeout=60) as response, # nosec B310 ++ urllib.request.urlopen( # nosec B310 ++ DEFAULT_CHECKPOINT_URL, ++ timeout=60, ++ ) as response, + temp_path.open("wb") as file, + ): + for chunk in iter(lambda: response.read(1024 * 1024), b""): +@@ -340,46 +289,50 @@ + except (OSError, urllib.error.URLError) as error: + temp_path.unlink(missing_ok=True) + raise RuntimeError( +- f"Failed to download MobileSAM checkpoint from {url}." ++ f"Failed to download MobileSAM checkpoint from {DEFAULT_CHECKPOINT_URL}." + ) from error + + + def resolve_checkpoint( +- args: argparse.Namespace, +-) -> tuple[str, str | None, str | None]: +- if args.checkpoint_path is not None: +- checkpoint_path = Path(args.checkpoint_path).expanduser().resolve() +- if not checkpoint_path.is_file(): +- raise FileNotFoundError(f"Checkpoint not found: {checkpoint_path}") +- return str(checkpoint_path), None, None +- +- checkpoint_path = default_checkpoint_cache_dir() / DEFAULT_CHECKPOINT_FILENAME +- if not checkpoint_path.is_file(): +- if args.local_files_only: ++ checkpoint_path: Path | None, ++ *, ++ local_files_only: bool, ++) -> tuple[Path, str | None, str]: ++ if checkpoint_path is not None: ++ resolved_path = checkpoint_path.expanduser().resolve() ++ if not resolved_path.is_file(): ++ raise FileNotFoundError(f"Checkpoint not found: {resolved_path}") ++ return resolved_path, None, file_sha256(resolved_path) ++ ++ resolved_path = default_checkpoint_path() ++ if not resolved_path.is_file(): ++ if local_files_only: + raise FileNotFoundError( +- f"Checkpoint not found in local cache: {checkpoint_path}" ++ f"Checkpoint not found in local cache: {resolved_path}" + ) +- download_checkpoint(DEFAULT_CHECKPOINT_URL, checkpoint_path) +- +- verify_checkpoint(checkpoint_path, DEFAULT_CHECKPOINT_SHA256) +- return ( +- str(checkpoint_path.resolve()), +- DEFAULT_CHECKPOINT_URL, +- DEFAULT_CHECKPOINT_SHA256, +- ) +- +- +-def preprocess_image( +- image: Image.Image, +- segmentation_map: Image.Image | None, +- *, +- name: str, +- input_size: int, ++ download_checkpoint(resolved_path) ++ ++ actual_sha256 = file_sha256(resolved_path) ++ if actual_sha256 != DEFAULT_CHECKPOINT_SHA256: ++ raise RuntimeError( ++ f"Checkpoint SHA256 mismatch for {resolved_path}: expected " ++ f"{DEFAULT_CHECKPOINT_SHA256}, got {actual_sha256}." ++ ) ++ return resolved_path.resolve(), DEFAULT_CHECKPOINT_URL, actual_sha256 ++ ++ ++def prepare_sample( ++ image_path: Path, + pixel_mean: np.ndarray, + pixel_std: np.ndarray, +- output_mask_size: tuple[int, int] | None, ++ input_size: int = INPUT_SIZE, + ) -> PreparedSample: +- rgb_image = image.convert("RGB") ++ image_path = image_path.expanduser().resolve() ++ if not image_path.is_file(): ++ raise FileNotFoundError(f"Input image not found: {image_path}") ++ ++ with Image.open(image_path) as input_image: ++ rgb_image = input_image.convert("RGB") + width, height = rgb_image.size + scale = input_size / max(height, width) + resized_size = (round(width * scale), round(height * scale)) +@@ -393,101 +346,53 @@ + pixel_values = torch.from_numpy(image_np).permute(2, 0, 1).unsqueeze(0) + pixel_values = torch.nn.functional.pad( + pixel_values, +- ( +- 0, +- input_size - resized_size[0], +- 0, +- input_size - resized_size[1], +- ), +- ) +- pixel_values = pixel_values.contiguous() +- +- labels = None +- if segmentation_map is not None: +- if output_mask_size is None: +- raise ValueError("An output mask size is required for reference masks.") +- resized_mask = segmentation_map.convert("L").resize( +- resized_size, +- Image.Resampling.NEAREST, +- ) +- padded_mask = Image.new("L", (input_size, input_size)) +- padded_mask.paste(resized_mask, (0, 0)) +- resized_mask = padded_mask.resize(output_mask_size, Image.Resampling.NEAREST) +- mask_np = np.asarray(resized_mask, dtype=np.uint8) +- labels = torch.from_numpy((mask_np > 0).astype(np.uint8)).to(torch.long) +- ++ (0, input_size - resized_size[0], 0, input_size - resized_size[1]), ++ ) + return PreparedSample( +- name=name, ++ name=image_path.stem, + image=padded_image, +- pixel_values=pixel_values, +- labels=labels, +- ) +- +- +-def load_local_samples( +- image_paths: list[str], +- mask_paths: list[str], +- limit: int, +- *, +- include_labels: bool, +- input_size: int, +- pixel_mean: np.ndarray, +- pixel_std: np.ndarray, +- output_mask_size: tuple[int, int] | None = None, +-) -> list[PreparedSample]: +- if include_labels and len(mask_paths) not in (0, len(image_paths)): +- raise ValueError("--eval-mask must be omitted or match --eval-image count.") +- +- samples: list[PreparedSample] = [] +- for index, image_path in enumerate(image_paths): +- if len(samples) >= limit: +- break +- image = Image.open(image_path) +- mask = None +- if include_labels and len(mask_paths) > 0: +- mask = Image.open(mask_paths[index]) +- samples.append( +- preprocess_image( +- image, +- mask, +- name=Path(image_path).stem or f"local_sample_{index:04d}", +- input_size=input_size, +- pixel_mean=pixel_mean, +- pixel_std=pixel_std, +- output_mask_size=output_mask_size, +- ) +- ) +- if len(samples) == 0: +- raise ValueError("No local samples were loaded.") +- return samples +- +- +-def run_mask_logits(model: torch.nn.Module, input_tensor: torch.Tensor) -> torch.Tensor: ++ pixel_values=pixel_values.contiguous(), ++ ) ++ ++ ++def run_model_outputs( ++ model: torch.nn.Module, input_tensor: torch.Tensor ++) -> tuple[torch.Tensor, torch.Tensor | None]: + output = model(input_tensor) + if isinstance(output, (tuple, list)): +- output = output[0] +- if not isinstance(output, torch.Tensor): +- raise TypeError(f"Expected tensor mask logits, got {type(output)}") +- return output +- +- +-def predict_mask(logits: torch.Tensor, threshold: float) -> np.ndarray: +- mask = logits.detach().cpu().squeeze(0).squeeze(0).numpy() > threshold +- return mask.astype(np.uint8) ++ if len(output) != 2 or not all( ++ isinstance(value, torch.Tensor) for value in output ++ ): ++ raise TypeError("Expected mask logits and quality score tensors.") ++ return output[0], output[1] ++ if isinstance(output, torch.Tensor): ++ return output, None ++ raise TypeError(f"Expected tensor model output, got {type(output)}") ++ ++ ++def run_mask_logits(model: torch.nn.Module, input_tensor: torch.Tensor) -> torch.Tensor: ++ return run_model_outputs(model, input_tensor)[0] ++ ++ ++def predict_masks(logits: torch.Tensor) -> np.ndarray: ++ if logits.ndim != 4: ++ raise ValueError( ++ "Expected mask logits with shape [points, candidates, H, W], " ++ f"got {logits.shape}." ++ ) ++ masks = logits.detach().cpu().numpy() > MASK_THRESHOLD ++ masks = masks.reshape(-1, masks.shape[-2], masks.shape[-1]) ++ return masks.astype(np.uint8) + + + def binary_iou(mask_a: np.ndarray, mask_b: np.ndarray) -> float: + intersection = np.logical_and(mask_a, mask_b).sum() + union = np.logical_or(mask_a, mask_b).sum() +- if union == 0: +- return 1.0 +- return float(intersection / union) +- +- +-def save_binary_mask(path: Path, mask: np.ndarray) -> Image.Image: +- image = Image.fromarray((mask.astype(np.uint8) * 255), mode="L") +- image.save(path) +- return image.convert("RGB") ++ return 1.0 if union == 0 else float(intersection / union) ++ ++ ++def save_binary_mask(path: Path, mask: np.ndarray) -> None: ++ Image.fromarray(mask.astype(np.uint8) * 255, mode="L").save(path) + + + def save_mask_overlay( +@@ -496,182 +401,176 @@ + mask: np.ndarray, + color: tuple[int, int, int], + ) -> None: +- rgb_image = image.convert("RGB") +- resized_mask = Image.fromarray(mask.astype(np.uint8) * 255, mode="L").resize( +- rgb_image.size, ++ resized_mask = Image.fromarray(mask * 255, mode="L").resize( ++ image.size, + Image.Resampling.NEAREST, + ) + mask_np = np.asarray(resized_mask, dtype=np.uint8) > 0 +- overlay_np = np.asarray(rgb_image, dtype=np.float32) +- color_np = np.asarray(color, dtype=np.float32) +- overlay_np[mask_np] = overlay_np[mask_np] * 0.55 + color_np * 0.45 +- Image.fromarray(np.clip(overlay_np, 0, 255).astype(np.uint8), mode="RGB").save(path) +- +- +-def write_debug_artifacts( ++ overlay_np = np.asarray(image.convert("RGB"), dtype=np.float32) ++ overlay_np[mask_np] = ( ++ overlay_np[mask_np] * 0.55 + np.asarray(color, dtype=np.float32) * 0.45 ++ ) ++ Image.fromarray(overlay_np.astype(np.uint8), mode="RGB").save(path) ++ ++ ++def evaluate_and_write_debug_artifacts( ++ fp32_model: torch.nn.Module, ++ quantized_model: torch.nn.Module, ++ sample: PreparedSample, + debug_dir: Path, +- sample: PreparedSample, +- fp32_mask: np.ndarray, +- quantized_mask: np.ndarray, +-) -> None: ++) -> dict[str, Any]: ++ fp32_logits, fp32_quality = run_model_outputs(fp32_model, sample.pixel_values) ++ quantized_logits, quantized_quality = run_model_outputs( ++ quantized_model, sample.pixel_values ++ ) ++ fp32_masks = predict_masks(fp32_logits) ++ quantized_masks = predict_masks(quantized_logits) ++ if fp32_masks.shape != quantized_masks.shape: ++ raise ValueError( ++ f"FP32 masks {fp32_masks.shape} do not match quantized masks " ++ f"{quantized_masks.shape}." ++ ) ++ ious = [ ++ binary_iou(fp32_mask, quantized_mask) ++ for fp32_mask, quantized_mask in zip(fp32_masks, quantized_masks) ++ ] ++ agreements = [ ++ float(np.mean(fp32_mask == quantized_mask)) ++ for fp32_mask, quantized_mask in zip(fp32_masks, quantized_masks) ++ ] ++ metrics = { ++ "fp32_quantized_iou": float(np.mean(ious)), ++ "fp32_quantized_pixel_agreement": float(np.mean(agreements)), ++ "minimum_per_object_fp32_quantized_iou": min(ious), ++ "minimum_per_object_fp32_quantized_pixel_agreement": min(agreements), ++ "per_object_fp32_quantized_iou": ious, ++ "per_object_fp32_quantized_pixel_agreement": agreements, ++ } ++ if fp32_quality is not None and quantized_quality is not None: ++ fp32_quality_np = fp32_quality.detach().cpu().numpy() ++ quantized_quality_np = quantized_quality.detach().cpu().numpy() ++ fp32_grouped = fp32_masks.reshape( ++ fp32_logits.shape[0], fp32_logits.shape[1], *fp32_masks.shape[-2:] ++ ) ++ quantized_grouped = quantized_masks.reshape( ++ quantized_logits.shape[0], ++ quantized_logits.shape[1], ++ *quantized_masks.shape[-2:], ++ ) ++ fp32_best = fp32_quality_np.argmax(axis=1) ++ quantized_best = quantized_quality_np.argmax(axis=1) ++ selected_ious = [] ++ selected_agreements = [] ++ accepted_points = [] ++ for point in range(fp32_grouped.shape[0]): ++ fp32_mask = fp32_grouped[point, fp32_best[point]] ++ quantized_mask = quantized_grouped[point, quantized_best[point]] ++ selected_ious.append(binary_iou(fp32_mask, quantized_mask)) ++ selected_agreements.append(float(np.mean(fp32_mask == quantized_mask))) ++ if ( ++ quantized_quality_np[point, quantized_best[point]] ++ >= MINIMUM_VALIDATION_MASK_QUALITY ++ ): ++ accepted_points.append(point) ++ quality_gate_points = accepted_points or list(range(fp32_grouped.shape[0])) ++ metrics.update( ++ { ++ "fp32_quality_scores": fp32_quality_np.tolist(), ++ "quantized_quality_scores": quantized_quality_np.tolist(), ++ "fp32_best_candidate": fp32_best.tolist(), ++ "quantized_best_candidate": quantized_best.tolist(), ++ "quality_gate_points": quality_gate_points, ++ "selected_fp32_quantized_iou": float( ++ np.mean([selected_ious[point] for point in quality_gate_points]) ++ ), ++ "selected_fp32_quantized_pixel_agreement": float( ++ np.mean( ++ [selected_agreements[point] for point in quality_gate_points] ++ ) ++ ), ++ "per_point_selected_fp32_quantized_iou": selected_ious, ++ "per_point_selected_fp32_quantized_pixel_agreement": ( ++ selected_agreements ++ ), ++ } ++ ) ++ + sample_dir = debug_dir / sample.name + sample_dir.mkdir(parents=True, exist_ok=True) +- + sample.image.save(sample_dir / "input.png") +- save_binary_mask(sample_dir / "fp32_mask.png", fp32_mask) +- save_binary_mask(sample_dir / "quantized_mask.png", quantized_mask) +- save_mask_overlay( +- sample_dir / "fp32_overlay.png", +- sample.image, +- fp32_mask, +- (0, 220, 120), +- ) +- save_mask_overlay( +- sample_dir / "quantized_overlay.png", +- sample.image, +- quantized_mask, +- (0, 170, 255), +- ) +- if sample.labels is not None: ++ for object_index, (fp32_mask, quantized_mask) in enumerate( ++ zip(fp32_masks, quantized_masks), start=1 ++ ): ++ save_binary_mask(sample_dir / f"fp32_mask_{object_index}.png", fp32_mask) + save_binary_mask( +- sample_dir / "reference_mask.png", +- sample.labels.detach().cpu().numpy().astype(np.uint8), +- ) +- +- mismatch = fp32_mask != quantized_mask +- heatmap = np.zeros((*fp32_mask.shape, 3), dtype=np.uint8) +- heatmap[~mismatch] = [0, 128, 0] +- heatmap[mismatch] = [255, 0, 0] +- Image.fromarray(heatmap, mode="RGB").save(sample_dir / "mismatch_heatmap.png") +- +- write_json( +- sample_dir / "mask_summary.json", +- { +- "foreground_pixels": int(quantized_mask.sum()), +- "background_pixels": int(quantized_mask.size - quantized_mask.sum()), +- "fp32_quantized_iou": binary_iou(fp32_mask, quantized_mask), +- }, +- ) +- +- +-def evaluate_and_debug( +- fp32_model: torch.nn.Module, +- quantized_model: torch.nn.Module, +- eval_samples: list[PreparedSample], +- debug_dir: Path | None, +- num_debug_samples: int, +- threshold: float, +-) -> dict[str, float]: +- fp32_quantized_ious: list[float] = [] +- fp32_quantized_pixel_agreements: list[float] = [] +- reference_ious: list[float] = [] +- +- if debug_dir is not None: +- debug_dir.mkdir(parents=True, exist_ok=True) +- +- print("\nEvaluating quantized MobileSAM on validation samples...") +- for index, sample in enumerate(tqdm.tqdm(eval_samples)): +- fp32_logits = run_mask_logits(fp32_model, sample.pixel_values) +- quantized_logits = run_mask_logits(quantized_model, sample.pixel_values) +- fp32_mask = predict_mask(fp32_logits, threshold) +- quantized_mask = predict_mask(quantized_logits, threshold) +- +- fp32_quantized_ious.append(binary_iou(fp32_mask, quantized_mask)) +- fp32_quantized_pixel_agreements.append( +- float(np.mean(fp32_mask == quantized_mask)) +- ) +- if sample.labels is not None: +- labels = sample.labels.detach().cpu().numpy().astype(np.uint8) +- reference_ious.append(binary_iou(quantized_mask, labels)) +- +- if debug_dir is not None and index < num_debug_samples: +- write_debug_artifacts(debug_dir, sample, fp32_mask, quantized_mask) +- +- metrics = { +- "num_samples": float(len(eval_samples)), +- "fp32_quantized_mean_iou": float(np.mean(fp32_quantized_ious)), +- "fp32_quantized_pixel_agreement": float( +- np.mean(fp32_quantized_pixel_agreements) +- ), +- } +- if len(reference_ious) > 0: +- metrics["reference_mean_iou"] = float(np.mean(reference_ious)) ++ sample_dir / f"quantized_mask_{object_index}.png", quantized_mask ++ ) ++ save_mask_overlay( ++ sample_dir / f"fp32_overlay_{object_index}.png", ++ sample.image, ++ fp32_mask, ++ (0, 220, 120), ++ ) ++ save_mask_overlay( ++ sample_dir / f"quantized_overlay_{object_index}.png", ++ sample.image, ++ quantized_mask, ++ (0, 170, 255), ++ ) + return metrics + + + def quantize_model( + model: torch.nn.Module, + quantizer: EthosUQuantizer, +- example_inputs: tuple[torch.Tensor], +- calibration_samples: list[PreparedSample], ++ input_tensor: torch.Tensor, + ) -> torch.export.ExportedProgram: +- exported = torch.export.export(model, example_inputs) ++ exported = torch.export.export(model, (input_tensor,)) + prepared = prepare_pt2e(exported.module(), quantizer) +- +- print("\nCalibrating MobileSAM...") +- for sample in tqdm.tqdm(calibration_samples): +- prepared(sample.pixel_values) +- ++ print("Calibrating MobileSAM with the demonstration image...") ++ prepared(input_tensor) + quantized = convert_pt2e(prepared) +- return torch.export.export(quantized, example_inputs) +- +- +-def has_quantized_out_variants() -> bool: ++ return torch.export.export(quantized, (input_tensor,)) ++ ++ ++def ensure_quantized_ops_loaded() -> None: ++ try: ++ import executorch.kernels.quantized # noqa: F401 ++ except ImportError as error: ++ raise RuntimeError( ++ "MobileSAM export requires ExecuTorch quantized kernels. Install the " ++ "current checkout with ./install_executorch.sh." ++ ) from error ++ + try: + _ = torch.ops.quantized_decomposed.quantize_per_tensor.out + _ = torch.ops.quantized_decomposed.dequantize_per_tensor.out +- return True +- except AttributeError: +- return False +- +- +-def load_quantized_ops_library(library_path: Path) -> Path: +- if not library_path.is_file(): +- raise FileNotFoundError(f"Quantized ops library not found: {library_path}") +- torch.ops.load_library(str(library_path)) +- if has_quantized_out_variants(): +- return library_path +- raise RuntimeError( +- f"Quantized ops library did not register required out variants: {library_path}" +- ) +- +- +-def ensure_quantized_ops_loaded() -> Path | None: +- if has_quantized_out_variants(): +- return None +- +- quantized_ops_library = os.environ.get("EXECUTORCH_QUANTIZED_OPS_AOT_LIBRARY") +- if quantized_ops_library: +- return load_quantized_ops_library( +- Path(quantized_ops_library).expanduser().resolve() +- ) +- +- try: +- import executorch.kernels.quantized # noqa: F401 +- except ImportError: +- pass +- else: +- if has_quantized_out_variants(): +- return None +- +- repo_root = Path(__file__).resolve().parents[4] +- search_patterns = ( +- "cmake-out/kernels/quantized/libquantized_ops_aot_lib.*", +- "arm_test/*/kernels/quantized/libquantized_ops_aot_lib.*", +- "arm_test/**/kernels/quantized/libquantized_ops_aot_lib.*", +- ) +- for pattern in search_patterns: +- for candidate in sorted(repo_root.glob(pattern)): +- if not candidate.is_file(): +- continue +- return load_quantized_ops_library(candidate) +- +- raise RuntimeError( +- "MobileSAM int8 export requires the quantized ops out-variant library. " +- "Build or install ExecuTorch quantized kernels so that " +- "`quantized_decomposed::quantize_per_tensor.out` and " +- "`quantized_decomposed::dequantize_per_tensor.out` are available." +- ) ++ except AttributeError as error: ++ raise RuntimeError( ++ "The ExecuTorch installation does not provide quantized out variants. " ++ "Re-run ./install_executorch.sh." ++ ) from error ++ ++ ++def find_module_type(module: torch.nn.Module, class_name: str) -> type[torch.nn.Module]: ++ for child in module.modules(): ++ if child.__class__.__name__ == class_name: ++ return child.__class__ ++ raise ValueError(f"Could not find module type {class_name} in {module.__class__}.") ++ ++ ++def resolve_point_prompts( ++ points: list[list[float]], input_size: int = INPUT_SIZE ++) -> list[tuple[float, float]]: ++ point_prompts = [(float(x), float(y)) for x, y in points] ++ if any( ++ not (0 <= point_x < input_size and 0 <= point_y < input_size) ++ for point_x, point_y in point_prompts ++ ): ++ raise ValueError( ++ f"Point prompts must be inside the {input_size}x{input_size} input." ++ ) ++ return point_prompts + + + def write_delegation_report(edge_program_manager: Any, report_path: Path) -> None: +@@ -681,113 +580,69 @@ + report_path.write_text(delegation_info.get_summary() + "\n") + + +-def resolve_point_prompts(args: argparse.Namespace) -> list[tuple[float, float]]: +- if len(args.point) > 0: +- point_prompts = [(float(x), float(y)) for x, y in args.point] +- else: +- point_prompts = [(args.input_size / 2, args.input_size / 2)] +- +- for point_x, point_y in point_prompts: +- if not (0 <= point_x <= args.input_size and 0 <= point_y <= args.input_size): +- raise ValueError("Point prompts must be inside the square input.") +- return point_prompts +- +- +-def validate_export_args(args: argparse.Namespace) -> None: +- if args.input_size < 224 or args.input_size % MOBILE_SAM_INPUT_ALIGNMENT != 0: +- raise ValueError("--input-size must be at least 224 and divisible by 16.") +- if args.num_calibration_samples <= 0: +- raise ValueError("--num-calibration-samples must be positive.") +- if args.num_eval_samples <= 0: +- raise ValueError("--num-eval-samples must be positive.") +- if args.minimum_fp32_quantized_iou is not None and not ( +- 0.0 <= args.minimum_fp32_quantized_iou <= 1.0 +- ): +- raise ValueError("--minimum-fp32-quantized-iou must be between 0 and 1.") +- if len(args.calibration_image) == 0: +- raise ValueError("At least one --calibration-image is required.") +- if len(args.eval_image) == 0: +- args.eval_image = list(args.calibration_image) +- if len(args.eval_mask) not in (0, len(args.eval_image)): +- raise ValueError("--eval-mask must be omitted or match --eval-image count.") +- +- +-def main() -> None: +- args = parse_args() +- validate_export_args(args) ++def main(argv: list[str] | None = None) -> None: ++ args = parse_args(argv) ++ if args.input_size < 128 or args.input_size % 32 != 0: ++ raise ValueError("--input-size must be at least 128 and divisible by 32.") + vela_version = validate_vela_version() +- point_prompts = resolve_point_prompts(args) +- quantized_ops_library = ensure_quantized_ops_loaded() +- if quantized_ops_library is not None: +- print(f"Loaded quantized ops library from {quantized_ops_library}") +- +- output_path = Path(args.output_path).resolve() ++ ensure_quantized_ops_loaded() ++ point_prompts = resolve_point_prompts(args.point, args.input_size) ++ ++ output_path = args.output_path.expanduser().resolve() ++ if output_path.suffix != ".pte": ++ raise ValueError("--output-path must end in .pte.") + output_path.parent.mkdir(parents=True, exist_ok=True) + metadata_path = output_path.with_suffix(".json") + metrics_path = output_path.with_name(f"{output_path.stem}_metrics.json") + delegation_path = output_path.with_name(f"{output_path.stem}_delegation.txt") +- debug_dir = Path(args.debug_output_dir).resolve() if args.debug_output_dir else None +- +- checkpoint_path, checkpoint_url, checkpoint_sha256 = resolve_checkpoint(args) ++ artifact_dir = output_path.parent / "artifacts" ++ debug_dir = output_path.parent / "debug" ++ artifact_dir.mkdir(exist_ok=True) ++ ++ checkpoint_path, checkpoint_url, checkpoint_sha256 = resolve_checkpoint( ++ args.checkpoint_path, ++ local_files_only=args.local_files_only, ++ ) + mobile_sam = load_mobile_sam( +- checkpoint_path, +- args.mobile_sam_source, +- args.input_size, ++ checkpoint_path, args.mobile_sam_source, args.input_size + ) + pixel_mean = cast(Any, mobile_sam).pixel_mean.detach().cpu().reshape(-1).numpy() + pixel_std = cast(Any, mobile_sam).pixel_std.detach().cpu().reshape(-1).numpy() + if pixel_mean.shape != (3,) or pixel_std.shape != (3,): + raise ValueError("MobileSAM preprocessing must provide three RGB values.") +- wrapped_model = MobileSAMFixedPrompt(mobile_sam, point_prompts).eval() +- +- calibration_samples = load_local_samples( +- args.calibration_image, +- [], +- args.num_calibration_samples, +- include_labels=False, +- input_size=args.input_size, +- pixel_mean=pixel_mean, +- pixel_std=pixel_std, +- ) +- example_inputs = (calibration_samples[0].pixel_values,) ++ ++ sample = prepare_sample(args.image, pixel_mean, pixel_std, args.input_size) ++ wrapped_model = MobileSAMFixedPrompt( ++ mobile_sam, point_prompts, args.multimask_output ++ ).eval() + with torch.no_grad(): +- output_shape = list(run_mask_logits(wrapped_model, example_inputs[0]).shape) +- if len(output_shape) != 4 or output_shape[:2] != [1, 1]: ++ output_shape = list(run_mask_logits(wrapped_model, sample.pixel_values).shape) ++ if ( ++ len(output_shape) != 4 ++ or output_shape[0] != len(point_prompts) ++ or output_shape[1] != (3 if args.multimask_output else 1) ++ ): + raise ValueError( +- f"Expected MobileSAM output shape [1, 1, height, width], got {output_shape}." +- ) +- output_mask_size = (output_shape[3], output_shape[2]) +- +- eval_samples = load_local_samples( +- args.eval_image, +- args.eval_mask, +- args.num_eval_samples, +- include_labels=True, +- input_size=args.input_size, +- pixel_mean=pixel_mean, +- pixel_std=pixel_std, +- output_mask_size=output_mask_size, +- ) ++ "Expected MobileSAM output shape " ++ f"[{len(point_prompts)}, " ++ f"{3 if args.multimask_output else 1}, height, width], got " ++ f"{output_shape}." ++ ) + + compile_spec = EthosUCompileSpec( +- target=args.target, +- system_config=args.system_config, +- memory_mode=args.memory_mode, +- extra_flags=args.extra_vela_flag, ++ target=TARGET, ++ system_config=SYSTEM_CONFIG, ++ memory_mode=MEMORY_MODE, + ) + compile_spec.set_pass_pipeline_config( + ArmPassPipelineConfig(softmax=SoftmaxDecompositionConfig.STABLE) + ) +- if args.artifact_dir is not None: +- artifact_dir = Path(args.artifact_dir).resolve() +- artifact_dir.mkdir(parents=True, exist_ok=True) +- compile_spec.dump_intermediate_artifacts_to(str(artifact_dir)) ++ compile_spec.dump_intermediate_artifacts_to(str(artifact_dir)) + + quantizer = EthosUQuantizer(compile_spec) + quantizer.set_global(get_symmetric_quantization_config()) +- attention_module_type = find_module_type(wrapped_model.image_encoder, "Attention") + quantizer.set_module_type( +- attention_module_type, ++ find_module_type(wrapped_model.image_encoder, "Attention"), + get_symmetric_a16w8_quantization_config(), + ) + +@@ -795,34 +650,37 @@ + quantized_program = quantize_model( + wrapped_model, + quantizer, +- example_inputs, +- calibration_samples, +- ) +- quantized_module = quantized_program.module() +- metrics = evaluate_and_debug( ++ sample.pixel_values, ++ ) ++ metrics = evaluate_and_write_debug_artifacts( + wrapped_model, +- quantized_module, +- eval_samples, ++ quantized_program.module(), ++ sample, + debug_dir, +- args.num_debug_samples, +- args.mask_threshold, +- ) +- write_json(metrics_path, metrics) +- print( +- "Validation metrics: " +- f"fp32_quantized_mean_iou={metrics['fp32_quantized_mean_iou']:.4f} " +- "fp32_quantized_pixel_agreement=" +- f"{metrics['fp32_quantized_pixel_agreement']:.4f}" +- ) +- if ( +- args.minimum_fp32_quantized_iou is not None +- and metrics["fp32_quantized_mean_iou"] < args.minimum_fp32_quantized_iou +- ): +- raise RuntimeError( +- "Host quantized/FP32 mask IoU " +- f"{metrics['fp32_quantized_mean_iou']:.4f} is below " +- f"{args.minimum_fp32_quantized_iou:.4f}." +- ) ++ ) ++ write_json(metrics_path, metrics) ++ print( ++ "Validation: " ++ f"FP32/quantized IoU={metrics['fp32_quantized_iou']:.4f}, " ++ f"agreement={metrics['fp32_quantized_pixel_agreement']:.4f}" ++ ) ++ validation_iou = metrics.get( ++ "selected_fp32_quantized_iou", metrics["fp32_quantized_iou"] ++ ) ++ validation_agreement = metrics.get( ++ "selected_fp32_quantized_pixel_agreement", ++ metrics["fp32_quantized_pixel_agreement"], ++ ) ++ if ( ++ validation_iou < MINIMUM_FP32_QUANTIZED_IOU ++ and validation_agreement < MINIMUM_FP32_QUANTIZED_PIXEL_AGREEMENT ++ ): ++ raise RuntimeError( ++ f"Selected FP32/quantized mask IoU {validation_iou:.4f} is below " ++ f"the required {MINIMUM_FP32_QUANTIZED_IOU:.4f}, and pixel agreement " ++ f"{validation_agreement:.4f} is below " ++ f"{MINIMUM_FP32_QUANTIZED_PIXEL_AGREEMENT:.4f}." ++ ) + + edge_program_manager = to_edge_transform_and_lower( + programs=quantized_program, +@@ -831,6 +689,33 @@ + ) + write_delegation_report(edge_program_manager, delegation_path) + ++ io_quantization = None ++ if args.quantized_io: ++ quantize_inputs = QuantizeInputs(edge_program_manager, [0]) ++ output_indices = [0, 1] if args.multimask_output else [0] ++ quantize_outputs = QuantizeOutputs(edge_program_manager, output_indices) ++ edge_program_manager = edge_program_manager.transform( ++ [quantize_inputs, quantize_outputs] ++ ) ++ ++ def serialize_qparams(values: tuple[Any, ...]) -> dict[str, Any]: ++ return { ++ "scale": float(values[0]), ++ "zero_point": int(values[1]), ++ "quant_min": int(values[2]), ++ "quant_max": int(values[3]), ++ "dtype": str(values[4]).removeprefix("torch."), ++ } ++ ++ io_quantization = { ++ "input": serialize_qparams(quantize_inputs.quant_args[0]), ++ "output": serialize_qparams(quantize_outputs.dequant_args[0]), ++ } ++ if args.multimask_output: ++ io_quantization["quality"] = serialize_qparams( ++ quantize_outputs.dequant_args[1] ++ ) ++ + executorch_program_manager = edge_program_manager.to_executorch( + config=ExecutorchBackendConfig(extract_delegate_segments=False) + ) +@@ -844,15 +729,15 @@ + metadata_path, + { + "model_name": "MobileSAM vit_t", +- "checkpoint_filename": DEFAULT_CHECKPOINT_FILENAME, +- "checkpoint_path": checkpoint_path, ++ "checkpoint_filename": checkpoint_path.name, ++ "checkpoint_path": str(checkpoint_path), + "checkpoint_url": checkpoint_url, + "checkpoint_sha256": checkpoint_sha256, + "mobile_sam_source_license": MOBILE_SAM_SOURCE_LICENSE, + "mobile_sam_source_url": MOBILE_SAM_SOURCE_URL, + "mobile_sam_source_revision": MOBILE_SAM_SOURCE_REVISION, + "mobile_sam_patch": MOBILE_SAM_PATCH, +- "input_shape": list(example_inputs[0].shape), ++ "input_shape": list(sample.pixel_values.shape), + "output_shape": output_shape, + "input_size": args.input_size, + "preprocessing": { +@@ -861,34 +746,33 @@ + "resize": "longest_side_then_zero_pad", + }, + "point_prompts_xy": point_prompts, +- "mask_threshold": args.mask_threshold, +- "target": args.target, ++ "multimask_output": args.multimask_output, ++ "mask_threshold": MASK_THRESHOLD, ++ "minimum_mask_quality": MINIMUM_MASK_QUALITY, ++ "target": TARGET, + "vela_version": vela_version, +- "system_config": args.system_config, +- "memory_mode": args.memory_mode, +- "extra_vela_flags": args.extra_vela_flag, ++ "system_config": SYSTEM_CONFIG, ++ "memory_mode": MEMORY_MODE, + "quantization": { + "global": "int8 activations and int8 weights", + "tinyvit_attention": "int16 activations and int8 weights", + }, +- "num_calibration_samples": len(calibration_samples), +- "num_eval_samples": len(eval_samples), +- "calibration_images": args.calibration_image, +- "eval_images": args.eval_image, +- "eval_masks": args.eval_mask, ++ "io_quantization": io_quantization, ++ "image": str(args.image.expanduser().resolve()), + "output_path": str(output_path), + "metrics_path": str(metrics_path), + "delegation_path": str(delegation_path), +- "debug_output_dir": str(debug_dir) if debug_dir is not None else None, ++ "artifact_dir": str(artifact_dir), ++ "debug_dir": str(debug_dir), + }, + ) + +- print(f"\nExported model saved to {output_path}") +- print(f"Metadata saved to {metadata_path}") +- print(f"Metrics saved to {metrics_path}") +- print(f"Delegation summary saved to {delegation_path}") +- if debug_dir is not None: +- print(f"Debug artifacts saved to {debug_dir}") ++ print(f"Exported model: {output_path}") ++ print(f"Metadata: {metadata_path}") ++ print(f"Metrics: {metrics_path}") ++ print(f"Delegation report: {delegation_path}") ++ print(f"TOSA/Vela artifacts: {artifact_dir}") ++ print(f"Debug masks and overlays: {debug_dir}") + + + if __name__ == "__main__": +diff --git a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/requirements.txt b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/requirements.txt +deleted file mode 100644 +--- a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/requirements.txt ++++ /dev/null +@@ -1,6 +0,0 @@ +-# Copyright 2026 Arm Limited and/or its affiliates. +-# +-# This source code is licensed under the BSD-style license found in the +-# LICENSE file in the root directory of this source tree. +- +-tqdm == 4.67.1 +diff --git a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/CMakeLists.txt b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/CMakeLists.txt +--- a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/CMakeLists.txt ++++ b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/CMakeLists.txt +@@ -58,6 +58,60 @@ + option(ET_SEGMENTATION_SEMIHOSTING_OUTPUT + "Emit validation logs through Arm semihosting for FVP smoke runs" ON + ) ++ ++function(validate_arm_baremetal_toolchain) ++ set(setup_hint ++ "Run examples/arm/setup.sh, source examples/arm/arm-scratch/setup_path.sh, and configure a fresh build directory." ++ ) ++ foreach(compiler IN ITEMS "${CMAKE_C_COMPILER}" "${CMAKE_CXX_COMPILER}") ++ execute_process( ++ COMMAND "${compiler}" -dumpmachine ++ OUTPUT_VARIABLE compiler_target ++ OUTPUT_STRIP_TRAILING_WHITESPACE ++ RESULT_VARIABLE compiler_target_result ++ ) ++ if(compiler_target_result OR NOT compiler_target STREQUAL "arm-none-eabi") ++ message( ++ FATAL_ERROR ++ "MobileSAM requires the Arm GNU bare-metal toolchain, but ${compiler} targets '${compiler_target}'. ${setup_hint}" ++ ) ++ endif() ++ endforeach() ++ ++ include(CheckCSourceCompiles) ++ check_c_source_compiles( ++ [=[ ++ #include ++ const char *unsigned_format = "%" PRIu64; ++ const char *hex_format = "%" PRIx64; ++ int main(void) { ++ return unsigned_format[0] == hex_format[0]; ++ } ++ ]=] ++ ET_MOBILESAM_HAS_INTTYPES_FORMAT_MACROS ++ ) ++ if(NOT ET_MOBILESAM_HAS_INTTYPES_FORMAT_MACROS) ++ message( ++ FATAL_ERROR ++ "The selected Arm C library does not provide PRIu64 and PRIx64 from inttypes.h. ${setup_hint}" ++ ) ++ endif() ++ ++ execute_process( ++ COMMAND "${CMAKE_CXX_COMPILER}" -print-file-name=libstdc++.a ++ OUTPUT_VARIABLE cxx_standard_library ++ OUTPUT_STRIP_TRAILING_WHITESPACE ++ RESULT_VARIABLE cxx_standard_library_result ++ ) ++ if(cxx_standard_library_result OR NOT EXISTS "${cxx_standard_library}") ++ message( ++ FATAL_ERROR ++ "The selected Arm C++ compiler cannot locate libstdc++.a. ${setup_hint}" ++ ) ++ endif() ++endfunction() ++ ++validate_arm_baremetal_toolchain() + + if(NOT EXISTS "${IMAGE_PATH}") + message( +diff --git a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/README.md b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/README.md +--- a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/README.md ++++ b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/README.md +@@ -1,84 +1,80 @@ + # MobileSAM Runtime Example + +-This directory builds a bare-metal Corstone-320 application for a MobileSAM +-fixed-prompt `.pte` generated by `model_export/export_mobilesam.py`. ++This directory builds a bare-metal Corstone-320 application for the fixed-prompt ++MobileSAM `.pte` generated by `model_export/export_mobilesam.py`. Complete the ++top-level clean-clone setup and the manual export before running these commands. ++ ++All commands below run from the ExecuTorch repository root and use the same ++paths as `model_export/README.md`: ++ ++```bash ++MOBILE_SAM_EXAMPLE="$PWD/examples/arm/mobilesam_prompt_segmentation_example_ethos_u" ++MOBILE_SAM_WORK_DIR="${MOBILE_SAM_WORK_DIR:-$PWD/arm_test/mobilesam_manual}" ++MOBILE_SAM_PTE="$MOBILE_SAM_WORK_DIR/export/mobilesam_point_ethos_u85_448.pte" ++MOBILE_SAM_METADATA="$MOBILE_SAM_WORK_DIR/export/mobilesam_point_ethos_u85_448.json" ++MOBILE_SAM_IMAGE="$PWD/examples/models/dinov2/dog.jpg" ++ ++test -f "$MOBILE_SAM_PTE" ++test -f "$MOBILE_SAM_METADATA" ++``` ++ ++The point `(219, 193)` is already embedded in this `.pte`. Changing ++`MOBILE_SAM_IMAGE` does not require another export, but it does require ++rebuilding the runtime so CMake regenerates the embedded image header. Changing ++the point requires another export. + + ## Build ExecuTorch for Arm + +-Run the Arm setup first if it has not already been run: ++Load the toolchain installed during clean-clone setup, then build and install ++the bare-metal ExecuTorch libraries: + + ```bash +-./examples/arm/setup.sh --i-agree-to-the-contained-eula + source examples/arm/arm-scratch/setup_path.sh ++ ++cmake --preset arm-baremetal -B "$MOBILE_SAM_WORK_DIR/cmake-out-arm" ++cmake --build "$MOBILE_SAM_WORK_DIR/cmake-out-arm" \ ++ --target install --parallel + ``` + +-Build and install the Arm bare-metal ExecuTorch libraries from +-`examples/arm`: ++## Configure and compile the runtime ++ ++The documented configuration enables the mask dump required by the ++visualization step: + + ```bash +-cmake --preset arm-baremetal \ +- -DCMAKE_BUILD_TYPE=Release \ +- -B../../cmake-out-arm ../.. +-cmake --build ../../cmake-out-arm --target install -j$(nproc) ++cmake -S "$MOBILE_SAM_EXAMPLE/runtime" \ ++ -B "$MOBILE_SAM_WORK_DIR/runtime" \ ++ -DCMAKE_TOOLCHAIN_FILE="$PWD/examples/arm/ethos-u-setup/arm-none-eabi-gcc.cmake" \ ++ -DET_BUILD_DIR_PATH="$MOBILE_SAM_WORK_DIR/cmake-out-arm" \ ++ -DET_PTE_FILE_PATH="$MOBILE_SAM_PTE" \ ++ -DIMAGE_PATH="$MOBILE_SAM_IMAGE" \ ++ -DET_SEGMENTATION_DUMP_MASK=ON \ ++ -DPYTHON_EXECUTABLE="$(command -v python)" ++ ++cmake --build "$MOBILE_SAM_WORK_DIR/runtime" \ ++ --target mobilesam_prompt_segmentation_example --parallel + ``` + +-The Arm bare-metal preset installs into the build directory. If you use a +-different `-B` path, pass the same path as `-DET_BUILD_DIR_PATH` when +-configuring the runtime app. ++The runtime finds the metadata JSON beside the `.pte`. The generated image ++header reads its input shape and normalization values, resizes the longest ++image side, and pads the shorter side with zeros. The generated `448x448` ++float input is placed in the Corstone-320 DDR input section so it does not ++consume BRAM. + +-## Configure the Runtime App +- +-Use the `.pte` from the export step and any RGB image: +- +-```bash +-cmake \ +- -DCMAKE_TOOLCHAIN_FILE=$(pwd)/ethos-u-setup/arm-none-eabi-gcc.cmake \ +- -DET_BUILD_DIR_PATH=../../cmake-out-arm \ +- -DET_PTE_FILE_PATH= \ +- -DMODEL_METADATA_PATH= \ +- -DIMAGE_PATH= \ +- -DMASK_THRESHOLD=0.0 \ +- -DSYSTEM_CONFIG=Ethos_U85_SYS_DRAM_Mid \ +- -DMEMORY_MODE=Dedicated_Sram_384KB \ +- -Bmobilesam_point_runtime \ +- mobilesam_prompt_segmentation_example_ethos_u/runtime +-``` +- +-The generated image header reads the input shape and normalization values from +-the exporter metadata, resizes the longest side, and pads the shorter side with +-zeros. `MODEL_METADATA_PATH` defaults to the `.json` file beside the `.pte`. +-The generated `448x448` float input is placed in the Corstone-320 DDR input +-section so it does not consume BRAM. +- +-The `MEMORY_MODE` value must match the value used during export. The default +-`Dedicated_Sram_384KB` places the Ethos-U tensor arena in DDR and the fast +-scratch buffer in SRAM, matching the Corstone-320 linker script. The default +-method allocator pool is sized for MobileSAM's reduced `448x448` input tensor. +- +-The runtime emits validation logs through Arm semihosting by default so the FVP +-smoke run below prints the segmentation summary. Configure with +-`-DET_SEGMENTATION_SEMIHOSTING_OUTPUT=OFF` when targeting an environment +-without semihosting support. +- +-The default target-side mask threshold is `0.0`, matching MobileSAM's mask-logit +-threshold. The runtime logs foreground counts for several thresholds so users +-can inspect and retune post-processing without re-exporting the graph. +- +-## Compile +- +-```bash +-cmake --build mobilesam_point_runtime -j$(nproc) -- mobilesam_prompt_segmentation_example +-``` ++The defaults match the exporter: mask threshold `0.0`, system configuration ++`Ethos_U85_SYS_DRAM_Mid`, and memory mode `Dedicated_Sram_384KB`. The memory ++mode places the Ethos-U tensor arena in DDR and the fast scratch buffer in ++SRAM. Keep these values unchanged for this example. + + ## Run on Corstone-320 FVP + + ```bash +-../../backends/arm/scripts/run_fvp.sh \ +- --elf=mobilesam_point_runtime/mobilesam_prompt_segmentation_example \ ++backends/arm/scripts/run_fvp.sh \ ++ --elf="$MOBILE_SAM_WORK_DIR/runtime/mobilesam_prompt_segmentation_example" \ + --target=ethos-u85-256 \ + --timeout=300 \ +- --semihosting-cwd=$(pwd)/mobilesam_point_runtime \ +- --fast 2>&1 | tee mobilesam_fvp.log ++ --semihosting-cwd="$MOBILE_SAM_WORK_DIR/runtime" \ ++ --fast 2>&1 | tee "$MOBILE_SAM_WORK_DIR/fvp.log" + ``` + + Expected logs include: +@@ -90,21 +86,44 @@ + - Foreground/background pixel counts. + - `Model executed successfully.` + +-To dump the predicted binary mask as run-length encoded log chunks, configure +-with `-DET_SEGMENTATION_DUMP_MASK=ON`. This is useful for debugging but makes +-the UART output larger. ++## Visualize and validate the FVP mask + +-Capture that FVP output and reconstruct the target mask and overlay with: ++The manual export writes the host quantized mask for the demo image to ++`$MOBILE_SAM_WORK_DIR/export/debug/dog/quantized_mask.png`. Reconstruct the ++target mask from the FVP log and compare the two results with: + + ```bash +-python mobilesam_prompt_segmentation_example_ethos_u/runtime/visualize_fvp_output.py \ +- --fvp-log=mobilesam_fvp.log \ +- --input-image= \ +- --metadata= \ +- --reference-mask= \ ++python "$MOBILE_SAM_EXAMPLE/runtime/visualize_fvp_output.py" \ ++ --fvp-log="$MOBILE_SAM_WORK_DIR/fvp.log" \ ++ --input-image="$MOBILE_SAM_IMAGE" \ ++ --metadata="$MOBILE_SAM_METADATA" \ ++ --reference-mask="$MOBILE_SAM_WORK_DIR/export/debug/dog/quantized_mask.png" \ + --minimum-iou=0.9 \ +- --output-dir=mobilesam_fvp_visual ++ --output-dir="$MOBILE_SAM_WORK_DIR/fvp_visual" + ``` + +-The tool writes the decoded target mask, an input/prompt image, a colored FVP +-overlay, a side-by-side comparison, and target/reference agreement metrics. ++The tool tolerates non-UTF-8 transport bytes in the FVP log and writes: ++ ++- Target mask: `$MOBILE_SAM_WORK_DIR/fvp_visual/fvp_mask.png` ++- Input and prompt: `$MOBILE_SAM_WORK_DIR/fvp_visual/input_with_prompts.png` ++- Target overlay: `$MOBILE_SAM_WORK_DIR/fvp_visual/fvp_overlay.png` ++- Side-by-side result: `$MOBILE_SAM_WORK_DIR/fvp_visual/fvp_comparison.png` ++- Metrics: `$MOBILE_SAM_WORK_DIR/fvp_visual/metrics.json` ++ ++## Toolchain errors ++ ++Runtime configuration checks that CMake selected the Arm GNU bare-metal C and ++C++ compilers, that `` provides `PRIu64` and `PRIx64`, and that the ++C++ compiler can locate `libstdc++.a`. If this check fails, rerun setup, source ++the generated path file, and use a fresh runtime build directory: ++ ++```bash ++./examples/arm/setup.sh --i-agree-to-the-contained-eula ++source examples/arm/arm-scratch/setup_path.sh ++ ++arm-none-eabi-gcc -dumpmachine ++arm-none-eabi-g++ -print-file-name=libstdc++.a ++``` ++ ++The first command must print `arm-none-eabi`; the second must print an existing ++absolute path rather than only `libstdc++.a`. +diff --git a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/image_to_array.py b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/image_to_array.py +--- a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/image_to_array.py ++++ b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/image_to_array.py +@@ -15,9 +15,17 @@ + import os + from argparse import ArgumentParser + from pathlib import Path ++from typing import Any + + import numpy as np + from PIL import Image ++ ++ ++def c_float_literal(value: float) -> str: ++ literal = f"{value:.9g}" ++ if "." not in literal and "e" not in literal.lower(): ++ literal += ".0" ++ return literal + "f" + + + def convert_image_to_c_array( +@@ -26,6 +34,8 @@ + image_size: tuple[int, int], + pixel_mean: tuple[float, float, float], + pixel_std: tuple[float, float, float], ++ io_quantization: dict[str, Any] | None = None, ++ point_prompts_xy: list[tuple[float, float]] | None = None, + array_name: str = "image_data", + ) -> None: + image = Image.open(image_path).convert("RGB") +@@ -46,29 +56,89 @@ + padded_data[: resized_size[1], : resized_size[0], :] = data + data = np.transpose(padded_data, (2, 0, 1)).flatten() + ++ c_type = "float" ++ quantized_input = False ++ input_scale = 1.0 ++ input_zero_point = 0 ++ output_scale = 1.0 ++ output_zero_point = 0 ++ if io_quantization is not None: ++ input_qparams = io_quantization.get("input") ++ output_qparams = io_quantization.get("output") ++ if not isinstance(input_qparams, dict) or not isinstance(output_qparams, dict): ++ raise ValueError("Quantized I/O metadata must contain input and output.") ++ if input_qparams.get("dtype") != "int8": ++ raise ValueError( ++ "MobileSAM board input generation currently supports int8." ++ ) ++ if output_qparams.get("dtype") != "int8": ++ raise ValueError( ++ "MobileSAM board output generation currently supports int8." ++ ) ++ ++ input_scale = float(input_qparams["scale"]) ++ input_zero_point = int(input_qparams["zero_point"]) ++ quant_min = int(input_qparams["quant_min"]) ++ quant_max = int(input_qparams["quant_max"]) ++ if not np.isfinite(input_scale) or input_scale <= 0: ++ raise ValueError("Input quantization scale must be finite and positive.") ++ data = np.clip( ++ np.round(data / input_scale + input_zero_point), quant_min, quant_max ++ ).astype(np.int8) ++ output_scale = float(output_qparams["scale"]) ++ output_zero_point = int(output_qparams["zero_point"]) ++ c_type = "int8_t" ++ quantized_input = True ++ + array_lines = [] + for i in range(0, len(data), 12): +- line = ", ".join(f"{value:.8f}" for value in data[i : i + 12]) ++ values = data[i : i + 12] ++ if quantized_input: ++ line = ", ".join(str(int(value)) for value in values) ++ else: ++ line = ", ".join(f"{value:.8f}" for value in values) + array_lines.append(" " + line + ",") ++ ++ if not point_prompts_xy: ++ point_prompts_xy = [(image_size[0] / 2, image_size[1] / 2)] ++ prompt_x_values = ", ".join(str(round(point[0])) for point in point_prompts_xy) ++ prompt_y_values = ", ".join(str(round(point[1])) for point in point_prompts_xy) + + c_array = f"""#include + #include + + const size_t image_width = {image_size[0]}; + const size_t image_height = {image_size[1]}; ++const size_t image_resized_width = {resized_size[0]}; ++const size_t image_resized_height = {resized_size[1]}; + const size_t image_channels = 3; +-__attribute__((section("input_data_sec"), aligned(16))) float {array_name}[{len(data)}] = {{ ++const float model_pixel_mean[3] = {{{", ".join(c_float_literal(value) for value in pixel_mean)}}}; ++const float model_pixel_std[3] = {{{", ".join(c_float_literal(value) for value in pixel_std)}}}; ++const float model_input_scale = {c_float_literal(input_scale)}; ++const int32_t model_input_zero_point = {input_zero_point}; ++const float model_output_scale = {c_float_literal(output_scale)}; ++const int32_t model_output_zero_point = {output_zero_point}; ++const size_t point_prompt_count = {len(point_prompts_xy)}; ++const size_t point_prompt_x[{len(point_prompts_xy)}] = {{{prompt_x_values}}}; ++const size_t point_prompt_y[{len(point_prompts_xy)}] = {{{prompt_y_values}}}; ++__attribute__((section("input_data_sec"), aligned(16))) const {c_type} {array_name}[{len(data)}] = {{ + {os.linesep.join(array_lines)} + }}; + """ + with open(output_path, "w") as output_file: + output_file.write(c_array) +- print(f"Converted '{image_path}' to '{output_path}' ({len(data)} floats)") ++ print(f"Converted '{image_path}' to '{output_path}' ({len(data)} {c_type} values)") + + + def load_model_metadata( + metadata_path: str, +-) -> tuple[tuple[int, int], tuple[float, float, float], tuple[float, float, float]]: ++) -> tuple[ ++ tuple[int, int], ++ tuple[float, float, float], ++ tuple[float, float, float], ++ dict[str, Any] | None, ++ list[tuple[float, float]], ++]: + metadata = json.loads(Path(metadata_path).read_text()) + input_shape = metadata.get("input_shape") + if not isinstance(input_shape, list) or len(input_shape) != 4: +@@ -89,16 +159,81 @@ + image_size = (int(input_shape[3]), int(input_shape[2])) + pixel_mean = tuple(float(value) for value in pixel_mean_values) + pixel_std = tuple(float(value) for value in pixel_std_values) ++ point_prompts = metadata.get("point_prompts_xy") ++ point_prompts_xy: list[tuple[float, float]] = [] ++ if point_prompts is not None: ++ if not isinstance(point_prompts, list) or not point_prompts: ++ raise ValueError("MobileSAM point_prompts_xy must contain [x, y] pairs.") ++ for point_prompt in point_prompts: ++ if not isinstance(point_prompt, list) or len(point_prompt) != 2: ++ raise ValueError( ++ "MobileSAM point_prompts_xy must contain [x, y] pairs." ++ ) ++ point_prompts_xy.append((float(point_prompt[0]), float(point_prompt[1]))) + return ( + image_size, + (pixel_mean[0], pixel_mean[1], pixel_mean[2]), + (pixel_std[0], pixel_std[1], pixel_std[2]), +- ) ++ metadata.get("io_quantization"), ++ point_prompts_xy, ++ ) ++ ++ ++def convert_metadata_to_c_header(metadata_path: str, output_path: str) -> None: ++ metadata = json.loads(Path(metadata_path).read_text()) ++ ( ++ image_size, ++ pixel_mean, ++ pixel_std, ++ io_quantization, ++ point_prompts_xy, ++ ) = load_model_metadata(metadata_path) ++ if not isinstance(io_quantization, dict): ++ raise ValueError("Live camera mode requires quantized I/O metadata.") ++ input_qparams = io_quantization.get("input") ++ output_qparams = io_quantization.get("output") ++ quality_qparams = io_quantization.get("quality") ++ if not all( ++ isinstance(qparams, dict) ++ for qparams in (input_qparams, output_qparams, quality_qparams) ++ ): ++ raise ValueError( ++ "Live multimask mode requires input, output, and quality qparams." ++ ) ++ if not point_prompts_xy: ++ raise ValueError("Live multimask mode requires automatic point prompts.") ++ ++ prompt_x_values = ", ".join(str(round(point[0])) for point in point_prompts_xy) ++ prompt_y_values = ", ".join(str(round(point[1])) for point in point_prompts_xy) ++ minimum_mask_quality = float(metadata.get("minimum_mask_quality", 0.9)) ++ header = f"""#include ++#include ++ ++const size_t image_width = {image_size[0]}; ++const size_t image_height = {image_size[1]}; ++const size_t image_resized_width = {image_size[0]}; ++const size_t image_resized_height = {image_size[1]}; ++const size_t image_channels = 3; ++const float model_pixel_mean[3] = {{{", ".join(c_float_literal(value) for value in pixel_mean)}}}; ++const float model_pixel_std[3] = {{{", ".join(c_float_literal(value) for value in pixel_std)}}}; ++const float model_input_scale = {c_float_literal(float(input_qparams["scale"]))}; ++const int32_t model_input_zero_point = {int(input_qparams["zero_point"])}; ++const float model_output_scale = {c_float_literal(float(output_qparams["scale"]))}; ++const int32_t model_output_zero_point = {int(output_qparams["zero_point"])}; ++const float model_quality_scale = {c_float_literal(float(quality_qparams["scale"]))}; ++const int32_t model_quality_zero_point = {int(quality_qparams["zero_point"])}; ++const float minimum_mask_quality = {c_float_literal(minimum_mask_quality)}; ++const size_t point_prompt_count = {len(point_prompts_xy)}; ++const size_t point_prompt_x[{len(point_prompts_xy)}] = {{{prompt_x_values}}}; ++const size_t point_prompt_y[{len(point_prompts_xy)}] = {{{prompt_y_values}}}; ++""" ++ Path(output_path).write_text(header) ++ print(f"Converted '{metadata_path}' to '{output_path}'") + + + def main() -> None: + parser = ArgumentParser() +- parser.add_argument("--image", required=True, help="Path to an RGB image.") ++ parser.add_argument("--image", help="Path to an RGB image.") + parser.add_argument( + "--output", required=True, help="Output path for the generated C array." + ) +@@ -107,15 +242,34 @@ + required=True, + help="Exporter metadata containing input shape and preprocessing values.", + ) ++ parser.add_argument( ++ "--metadata-only", ++ action="store_true", ++ help="Generate constants for live camera capture without a static image.", ++ ) + args = parser.parse_args() + +- image_size, pixel_mean, pixel_std = load_model_metadata(args.metadata) ++ if args.metadata_only: ++ convert_metadata_to_c_header(args.metadata, args.output) ++ return ++ if not args.image: ++ parser.error("--image is required unless --metadata-only is used.") ++ ++ ( ++ image_size, ++ pixel_mean, ++ pixel_std, ++ io_quantization, ++ point_prompts_xy, ++ ) = load_model_metadata(args.metadata) + convert_image_to_c_array( + args.image, + args.output, + image_size, + pixel_mean, + pixel_std, ++ io_quantization, ++ point_prompts_xy, + ) + + +diff --git a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/main.cpp b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/main.cpp +--- a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/main.cpp ++++ b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/main.cpp +@@ -14,6 +14,7 @@ + #include + + #include ++#include + #include + #include + +@@ -185,10 +186,6 @@ + out.dim() == 4, + "Expected mask logits with shape [1, 1, height, width], got rank %zd", + out.dim()); +- ET_CHECK_MSG( +- out.scalar_type() == ScalarType::Float, +- "Expected float mask logits, got dtype %d", +- out.scalar_type()); + ET_CHECK_MSG(out.size(0) == 1, "Only batch size 1 is supported."); + ET_CHECK_MSG( + out.size(1) == 1, +@@ -198,15 +195,30 @@ + const size_t height = static_cast(out.size(2)); + const size_t width = static_cast(out.size(3)); + const auto strides = out.strides(); +- const float* data = out.const_data_ptr(); ++ const float* float_data = nullptr; ++ const int8_t* int8_data = nullptr; ++ if (out.scalar_type() == ScalarType::Float) { ++ float_data = out.const_data_ptr(); ++ } else if (out.scalar_type() == ScalarType::Char) { ++ int8_data = out.const_data_ptr(); ++ } else { ++ ET_CHECK_MSG(false, "Expected float or int8 mask logits."); ++ } + + size_t foreground_pixels = 0; + #if defined(ET_SEGMENTATION_DUMP_MASK) + std::vector mask(height * width, 0); + #endif + uint32_t mask_hash = 2166136261u; +- float min_score = data[0]; +- float max_score = data[0]; ++ const auto score_at = [&](size_t index) { ++ if (float_data != nullptr) { ++ return float_data[index]; ++ } ++ return (static_cast(int8_data[index]) - model_output_zero_point) * ++ model_output_scale; ++ }; ++ float min_score = score_at(0); ++ float max_score = min_score; + double score_sum = 0.0; + const float threshold_sweep[] = { + 0.0f, +@@ -225,7 +237,7 @@ + + for (size_t y = 0; y < height; ++y) { + for (size_t x = 0; x < width; ++x) { +- const float score = data[y * strides[2] + x * strides[3]]; ++ const float score = score_at(y * strides[2] + x * strides[3]); + min_score = std::min(min_score, score); + max_score = std::max(max_score, score); + score_sum += score; +@@ -371,15 +383,26 @@ + expected_elems, + image_elements); + ET_CHECK_MSG( +- input_tensor.scalar_type() == ScalarType::Float, +- "Expected float input tensor, got dtype %d", +- input_tensor.scalar_type()); +- +- float* input_data = input_tensor.mutable_data_ptr(); ++ input_tensor.nbytes() == sizeof(image_data), ++ "Input tensor expects %zu bytes, but image_data has %zu bytes", ++ input_tensor.nbytes(), ++ sizeof(image_data)); + write_runtime_format("Copying %zu input elements", expected_elems); +- for (size_t i = 0; i < expected_elems; ++i) { +- input_data[i] = image_data[i]; +- } ++ if (input_tensor.scalar_type() == ScalarType::Float) { ++ ET_CHECK_MSG( ++ sizeof(image_data[0]) == sizeof(float), ++ "Float model input requires float image data."); ++ } else if (input_tensor.scalar_type() == ScalarType::Char) { ++ ET_CHECK_MSG( ++ sizeof(image_data[0]) == sizeof(int8_t), ++ "Int8 model input requires int8 image data."); ++ } else { ++ ET_CHECK_MSG( ++ false, ++ "Expected float or int8 input tensor, got dtype %d", ++ input_tensor.scalar_type()); ++ } ++ std::memcpy(input_tensor.mutable_data_ptr(), image_data, sizeof(image_data)); + + write_runtime_line("Running model execution"); + Error status_inference = method->execute(); +diff --git a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/visualize_fvp_output.py b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/visualize_fvp_output.py +--- a/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/visualize_fvp_output.py ++++ b/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/visualize_fvp_output.py +@@ -31,7 +31,7 @@ + def parse_rle_mask(log_path: Path, expected_pixels: int) -> list[int]: + mask: list[int] = [] + in_dump = False +- for line in log_path.read_text().splitlines(): ++ for line in log_path.read_text(encoding="utf-8", errors="replace").splitlines(): + if "executorch:main.cpp:" in line: + continue + if "Segmentation mask RLE begin" in line: +diff --git a/install_requirements.py b/install_requirements.py +--- a/install_requirements.py ++++ b/install_requirements.py +@@ -13,25 +13,11 @@ + + from install_utils import determine_torch_url, is_intel_mac_os, python_is_compatible + +-# The pip repository that hosts nightly torch packages. +-# This will be dynamically set based on CUDA availability and CUDA backend enabled/disabled. +-TORCH_URL_BASE = "https://download.pytorch.org/whl/test" +-TORCHAO_URL_BASE = "https://download.pytorch.org/whl/nightly" +-TORCHAO_NIGHTLY_VERSION = "0.18.0.dev20260729" +- +-# Since ExecuTorch often uses main-branch features of pytorch, only the nightly +-# pip versions will have the required features. +-# +-# NOTE: If a newly-fetched version of the executorch repo changes the value of +-# NIGHTLY_VERSION, you should re-run this script to install the necessary +-# package versions. +-# +-# NOTE: If you're changing, make the corresponding change in .ci/docker/ci_commit_pins/pytorch.txt +-# by picking the hash from the same date in +-# https://hud.pytorch.org/hud/pytorch/pytorch/nightly/ @lint-ignore +-# +-# NOTE: If you're changing, make the corresponding supported CUDA versions in +-# SUPPORTED_CUDA_VERSIONS in install_utils.py if needed. ++# Use released packages for the Alif E8 Learning Path. The platform selector ++# appends the CPU or CUDA channel to these stable index roots. ++TORCH_URL_BASE = "https://download.pytorch.org/whl" ++TORCHAO_URL_BASE = "https://download.pytorch.org/whl" ++TORCHAO_RELEASE_VERSION = "0.18.0" + + + def install_requirements(use_pytorch_nightly): +@@ -66,7 +52,7 @@ + # that we don't need to set any version number there because they have already + # been installed on CI before this step, so pip won't reinstall them + ("torch==2.13.0" if use_pytorch_nightly else "torch"), +- f"torchao=={TORCHAO_NIGHTLY_VERSION}", ++ f"torchao=={TORCHAO_RELEASE_VERSION}", + ] + + # Install the requirements for core ExecuTorch package. +diff --git a/zephyr/samples/mobilesam-ethosu/CMakeLists.txt b/zephyr/samples/mobilesam-ethosu/CMakeLists.txt +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/CMakeLists.txt +@@ -0,0 +1,243 @@ ++# Copyright (c) Meta Platforms, Inc. and affiliates. ++# All rights reserved. ++# Copyright 2026 Arm Limited and/or its affiliates. ++# ++# SPDX-License-Identifier: Apache-2.0 ++ ++cmake_minimum_required(VERSION 3.24) ++ ++set(CONFIG_ARM_ETHOS_U85_256 ++ y ++ CACHE BOOL "E8 Ethos-U85, 256 MACs" FORCE ++) ++set(ETHOSU_TARGET_NPU_CONFIG ++ "ethos-u85-256" ++ CACHE STRING "Arm Ethos-U HAL target" FORCE ++) ++set(MAX_KERNEL_NUM ++ 32 ++ CACHE STRING "ExecuTorch kernel registry capacity" FORCE ++) ++set(EXECUTORCH_LOG_LEVEL ++ Error ++ CACHE STRING "ExecuTorch runtime log level" FORCE ++) ++set(CMAKE_SKIP_INSTALL_RULES ++ ON ++ CACHE BOOL "" FORCE ++) ++ ++set(ETHOSU_DRIVER_PATH ++ "$ENV{ZEPHYR_BASE}/../modules/hal/ethos_u" ++ CACHE PATH "Zephyr workspace Ethos-U core driver" ++) ++if(NOT EXISTS "${ETHOSU_DRIVER_PATH}/include/ethosu_driver.h") ++ message(FATAL_ERROR "ETHOSU_DRIVER_PATH must point to the Zephyr Ethos-U HAL module.") ++endif() ++# Compile against the same driver that Zephyr links, without fetching another SDK. ++set(ETHOS_SDK_PATH "${CMAKE_CURRENT_BINARY_DIR}/ethos-u" CACHE PATH "" FORCE) ++file(MAKE_DIRECTORY "${ETHOS_SDK_PATH}/core_platform") ++file(COPY "${ETHOSU_DRIVER_PATH}/include" ++ DESTINATION "${ETHOS_SDK_PATH}/core_software/core_driver") ++ ++set(ET_PTE_FILE_PATH ++ "" ++ CACHE FILEPATH "Path to the quantized-I/O MobileSAM PTE" ++) ++set(MODEL_METADATA_PATH ++ "" ++ CACHE FILEPATH "Path to MobileSAM exporter metadata" ++) ++set(ET_MODEL_PTE_ADDRESS ++ "0xC0000000" ++ CACHE STRING "Memory-mapped OSPI address containing the PTE" ++) ++set(MASK_THRESHOLD ++ "0.0" ++ CACHE STRING "MobileSAM mask logit threshold" ++) ++ ++set(SDK_ALIF ++ "$ENV{SDK_ALIF}" ++ CACHE PATH "Path to the sdk-alif checkout" ++) ++ ++foreach(required_file ET_PTE_FILE_PATH MODEL_METADATA_PATH) ++ if(NOT DEFINED ${required_file} OR "${${required_file}}" STREQUAL "") ++ message(FATAL_ERROR "${required_file} must be provided.") ++ endif() ++ if(NOT IS_ABSOLUTE "${${required_file}}") ++ get_filename_component( ++ ${required_file} "${${required_file}}" ABSOLUTE BASE_DIR ++ "${CMAKE_CURRENT_SOURCE_DIR}" ++ ) ++ endif() ++ if(NOT EXISTS "${${required_file}}") ++ message(FATAL_ERROR "Could not find ${required_file}: ${${required_file}}") ++ endif() ++endforeach() ++ ++if(NOT SDK_ALIF OR NOT EXISTS "${SDK_ALIF}/zephyr/module.yml") ++ message( ++ FATAL_ERROR ++ "SDK_ALIF must point to a sdk-alif checkout. Set the SDK_ALIF " ++ "environment variable or pass -DSDK_ALIF=/path/to/sdk-alif." ++ ) ++endif() ++set(MT9M114_OVERLAY ++ "${SDK_ALIF}/samples/drivers/video/boards/serial_camera_mt9m114_selfie.overlay" ++) ++if(NOT EXISTS "${MT9M114_OVERLAY}") ++ message( ++ FATAL_ERROR ++ "Could not find the sdk-alif MT9M114 overlay: ${MT9M114_OVERLAY}" ++ ) ++endif() ++list( ++ APPEND ++ EXTRA_DTC_OVERLAY_FILE ++ "${MT9M114_OVERLAY}" ++ "${CMAKE_CURRENT_LIST_DIR}/boards/mt9m114_selfie_deferred.overlay" ++) ++ ++find_package( ++ Python3 ++ COMPONENTS Interpreter ++ REQUIRED ++) ++ ++execute_process( ++ COMMAND ++ ${Python3_EXECUTABLE} ++ "${CMAKE_CURRENT_LIST_DIR}/../../../codegen/tools/gen_oplist.py" ++ --model_file_path=${ET_PTE_FILE_PATH} ++ --output_path=${CMAKE_CURRENT_BINARY_DIR}/temp.yaml ++ RESULT_VARIABLE GEN_OPLIST_RESULT ++ OUTPUT_VARIABLE GEN_OPLIST_OUTPUT ++ ERROR_VARIABLE GEN_OPLIST_ERROR ++) ++if(NOT GEN_OPLIST_RESULT EQUAL 0) ++ message( ++ FATAL_ERROR ++ "gen_oplist.py failed with exit code ${GEN_OPLIST_RESULT}: ${GEN_OPLIST_ERROR}" ++ ) ++endif() ++file(READ "${CMAKE_CURRENT_BINARY_DIR}/temp.yaml" GEN_OPLIST_CONTENT) ++if(GEN_OPLIST_CONTENT MATCHES "aten::" OR GEN_OPLIST_CONTENT MATCHES ++ "dim_order_ops::" ++) ++ message( ++ FATAL_ERROR ++ "The Alif sample requires a fully delegated --quantized-io PTE with no CPU operators." ++ ) ++endif() ++ ++execute_process( ++ COMMAND ++ ${Python3_EXECUTABLE} ++ "${CMAKE_CURRENT_LIST_DIR}/../../../backends/arm/scripts/get_ethosu_scratch_from_pte.py" ++ ${ET_PTE_FILE_PATH} ++ RESULT_VARIABLE SCRATCH_RESULT ++ OUTPUT_VARIABLE SCRATCH_OUTPUT ++ ERROR_VARIABLE SCRATCH_ERROR ++) ++if(NOT SCRATCH_RESULT EQUAL 0) ++ message(FATAL_ERROR "Could not inspect PTE scratch size: ${SCRATCH_ERROR}") ++endif() ++string(REGEX MATCH "max_scratch_size=([0-9]+)" _ "${SCRATCH_OUTPUT}") ++if(NOT CMAKE_MATCH_1) ++ message(FATAL_ERROR "PTE did not report an Ethos-U scratch size.") ++endif() ++set(ET_REQUIRED_SCRATCH_SIZE ${CMAKE_MATCH_1}) ++ ++find_package(Zephyr REQUIRED HINTS $ENV{ZEPHYR_BASE}) ++project(executorch_mobilesam_ethosu) ++ ++if(${CONFIG_EXECUTORCH_TEMP_ALLOCATOR_POOL_SIZE} LESS ++ ${ET_REQUIRED_SCRATCH_SIZE} ++) ++ message( ++ FATAL_ERROR ++ "CONFIG_EXECUTORCH_TEMP_ALLOCATOR_POOL_SIZE=${CONFIG_EXECUTORCH_TEMP_ALLOCATOR_POOL_SIZE} is smaller than the PTE requirement ${ET_REQUIRED_SCRATCH_SIZE}." ++ ) ++endif() ++ ++if(NOT DEFINED EXECUTORCH_DIR) ++ if(DEFINED ZEPHYR_EXECUTORCH_MODULE_DIR) ++ set(EXECUTORCH_DIR ${ZEPHYR_EXECUTORCH_MODULE_DIR}) ++ else() ++ message( ++ FATAL_ERROR ++ "ExecuTorch module not found. Include it in ZEPHYR_EXTRA_MODULES." ++ ) ++ endif() ++endif() ++ ++include(${EXECUTORCH_DIR}/tools/cmake/Utils.cmake) ++ ++set(MODEL_HEADER "${CMAKE_CURRENT_BINARY_DIR}/generated/model_metadata.h") ++set(LOGO_HEADER "${CMAKE_CURRENT_BINARY_DIR}/generated/logos.h") ++add_custom_command( ++ OUTPUT "${MODEL_HEADER}" ++ COMMAND ++ ${Python3_EXECUTABLE} ++ ${EXECUTORCH_DIR}/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/image_to_array.py ++ --metadata ${MODEL_METADATA_PATH} --metadata-only --output ${MODEL_HEADER} ++ DEPENDS ++ ${MODEL_METADATA_PATH} ++ ${EXECUTORCH_DIR}/examples/arm/mobilesam_prompt_segmentation_example_ethos_u/runtime/image_to_array.py ++ COMMENT "Generating MobileSAM model metadata" ++ VERBATIM ++) ++ ++add_custom_command( ++ OUTPUT "${LOGO_HEADER}" ++ COMMAND ++ ${Python3_EXECUTABLE} ++ ${CMAKE_CURRENT_LIST_DIR}/scripts/png_to_logo_header.py --executorch-logo ++ ${EXECUTORCH_DIR}/docs/source/_static/img/executorch-chip-logo-circle-32.png ++ --pytorch-logo ++ ${EXECUTORCH_DIR}/docs/source/_static/img/generic-pytorch-logo.png --output ++ ${LOGO_HEADER} ++ DEPENDS ++ ${CMAKE_CURRENT_LIST_DIR}/scripts/png_to_logo_header.py ++ ${EXECUTORCH_DIR}/docs/source/_static/img/executorch-chip-logo-circle-32.png ++ ${EXECUTORCH_DIR}/docs/source/_static/img/generic-pytorch-logo.png ++ COMMENT "Generating ExecuTorch and PyTorch logos" ++ VERBATIM ++) ++ ++target_sources( ++ app ++ PRIVATE ++ src/camera.cpp src/main.cpp ${MODEL_HEADER} ++ ${EXECUTORCH_DIR}/examples/arm/common/arm_memory_allocator.cpp ++) ++target_sources_ifdef(CONFIG_MOBILESAM_DISPLAY app PRIVATE src/display.cpp ${LOGO_HEADER}) ++target_include_directories( ++ app PRIVATE "${CMAKE_CURRENT_BINARY_DIR}/generated" ++ ${EXECUTORCH_DIR}/examples/arm/common ++) ++ ++file(SIZE "${ET_PTE_FILE_PATH}" ET_MODEL_PTE_SIZE) ++target_compile_definitions( ++ app ++ PRIVATE ++ ET_MODEL_PTE_ADDRESS=${ET_MODEL_PTE_ADDRESS} ++ ET_MODEL_PTE_SIZE=${ET_MODEL_PTE_SIZE} ++ ET_SEGMENTATION_MASK_THRESHOLD=${MASK_THRESHOLD} ++ ET_ARM_METHOD_ALLOCATOR_POOL_SIZE=${CONFIG_EXECUTORCH_METHOD_ALLOCATOR_POOL_SIZE} ++ ET_ARM_BAREMETAL_SCRATCH_TEMP_ALLOCATOR_POOL_SIZE=${CONFIG_EXECUTORCH_TEMP_ALLOCATOR_POOL_SIZE} ++ ET_ARM_BAREMETAL_FAST_SCRATCH_TEMP_ALLOCATOR_POOL_SIZE=${CONFIG_EXECUTORCH_FAST_TEMP_ALLOCATOR_POOL_SIZE} ++) ++ ++target_link_libraries(app PRIVATE libexecutorch) ++if(TARGET executorch_delegate_ethos_u) ++ executorch_target_link_options_shared_lib(executorch_delegate_ethos_u) ++ target_link_libraries(app PRIVATE executorch_delegate_ethos_u) ++else() ++ message(FATAL_ERROR "ExecuTorch Ethos-U delegate target was not built.") ++endif() ++if(TARGET ethosu_core_driver) ++ target_link_libraries(app PRIVATE ethosu_core_driver) ++endif() +diff --git a/zephyr/samples/mobilesam-ethosu/Kconfig b/zephyr/samples/mobilesam-ethosu/Kconfig +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/Kconfig +@@ -0,0 +1,34 @@ ++# Copyright (c) Meta Platforms, Inc. and affiliates. ++# All rights reserved. ++# Copyright 2026 Arm Limited and/or its affiliates. ++# ++# SPDX-License-Identifier: Apache-2.0 ++ ++source "Kconfig.zephyr" ++ ++menu "ExecuTorch MobileSAM Ethos-U sample configuration" ++ ++config MOBILESAM_DISPLAY ++ bool "Enable the optional on-board dashboard" ++ depends on DISPLAY && MIPI_DSI && DT_HAS_FOCUSLCDS_MW405_ENABLED ++ default n ++ help ++ Render camera frames and masks on the on-board display. This optional ++ mode still reports unresolved DSI link errors on the tested board. ++ ++config EXECUTORCH_METHOD_ALLOCATOR_POOL_SIZE ++ int "Method allocator pool size in bytes" ++ default 1048576 ++ depends on EXECUTORCH ++ ++config EXECUTORCH_TEMP_ALLOCATOR_POOL_SIZE ++ int "Ethos-U tensor arena size in bytes" ++ default 5709824 ++ depends on EXECUTORCH ++ ++config EXECUTORCH_FAST_TEMP_ALLOCATOR_POOL_SIZE ++ int "Ethos-U fast scratch size in bytes" ++ default 393216 ++ depends on EXECUTORCH ++ ++endmenu +diff --git a/zephyr/samples/mobilesam-ethosu/README.md b/zephyr/samples/mobilesam-ethosu/README.md +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/README.md +@@ -0,0 +1,210 @@ ++## Default camera and NPU mode ++ ++The default build captures camera frames, executes the quantized model, selects masks, ++and reports progress over UART. CDC200, DSI and the panel are disabled, and the MIPI ++route stays on the camera. `CONFIG_MOBILESAM_DISPLAY` defaults to `n`. ++ ++The optional dashboard retains unresolved DSI acknowledgment/packet errors on the ++tested hardware. To build that extension, add both of these CMake arguments: ++ ++```bash ++-DEXTRA_CONF_FILE="$EXECUTORCH_ROOT/zephyr/samples/mobilesam-ethosu/dashboard.conf" \ ++-DEXTRA_DTC_OVERLAY_FILE="$EXECUTORCH_ROOT/zephyr/samples/mobilesam-ethosu/boards/dashboard.overlay" ++``` ++ ++Do not use the optional display extension as the error-free deployment reference. ++ ++# Live MobileSAM on Alif Ensemble E8 ++ ++This Zephyr sample captures the Alif E8 Development Kit's on-board MT9M114 ++selfie camera, runs MobileSAM on the Cortex-M55 HP core and Ethos-U85-256, and ++draws up to four distinct masks on the 480x800 display. The dashboard includes ++the ExecuTorch and PyTorch logos, inference time, and detected-object count. ++ ++The model is stored separately in external OSPI flash at `0xC0000000`. Point ++prompts are fixed during export; the recommended four-point grid samples each ++image quadrant and target-side post-processing removes duplicate masks. ++ ++## Requirements ++ ++- Alif Ensemble E8 Development Kit with its display and selfie camera attached. ++- Alif's `zephyr_alif` and `sdk-alif` checkouts. These revisions are tested: ++ - `zephyr_alif`: `f002a4d8499c35fc70ec4d5324faa7b01c152c72` ++ - `sdk-alif`: `1db82c9d0cc594e2b548a13699e8a4157e6dcd4c` ++- Zephyr SDK 0.17.0 and a Python environment containing `west`, `fdt`, and the ++ Alif Zephyr requirements. ++- ExecuTorch's Arm dependencies from `examples/arm/setup.sh`. ++- Alif Security Toolkit and SEGGER J-Link software. ++ ++Use the [Arm Alif E8 board setup guide](https://learn.arm.com/learning-paths/embedded-and-microcontrollers/alif-image-classification/board-setup/) ++for a new board. Leave SW4 in its default SEUART position. Close terminal ++programs connected to the SEUART port before using SETOOLS; no further SW4 ++changes are required by this workflow. ++ ++The commands below assume: ++ ++```bash ++export EXECUTORCH_ROOT=/absolute/path/to/executorch ++export ZEPHYR_BASE=/absolute/path/to/zephyr_alif ++export SDK_ALIF=/absolute/path/to/sdk-alif ++export ZEPHYR_SDK_INSTALL_DIR=/absolute/path/to/zephyr-sdk-0.17.0 ++export ZEPHYR_TOOLCHAIN_VARIANT=zephyr ++export MOBILE_SAM_OUTPUT=/absolute/path/to/mobilesam-alif-e8 ++export MOBILE_SAM_SOURCE="$MOBILE_SAM_OUTPUT/mobile-sam/source" ++ ++mkdir -p "$MOBILE_SAM_OUTPUT" ++cd "$EXECUTORCH_ROOT" ++``` ++ ++Initialize the west workspace and Python dependencies once: ++ ++```bash ++west init -l "$ZEPHYR_BASE" ++west update ++python -m pip install -r "$ZEPHYR_BASE/scripts/requirements.txt" ++python -m pip install fdt west ++west zephyr-export ++``` ++ ++Keep this ExecuTorch Python environment active for export and build. CMake uses ++its interpreter to inspect the PTE and generate model-specific sources. ++ ++## Export the live multi-object model ++ ++Prepare the pinned MobileSAM checkout, then export a 384x384 quantized-I/O ++model with four independent point prompts and three candidates per point: ++ ++```bash ++python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/prepare_mobilesam.py \ ++ --source-dir "$MOBILE_SAM_SOURCE" ++ ++python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/export_mobilesam.py \ ++ --output-path "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \ ++ --mobile-sam-source "$MOBILE_SAM_SOURCE" \ ++ --image examples/models/dinov2/dog.jpg \ ++ --input-size 384 \ ++ --point 96 96 \ ++ --point 288 96 \ ++ --point 96 288 \ ++ --point 288 288 \ ++ --multimask-output \ ++ --quantized-io ++``` ++ ++The export also writes ++`mobilesam_live_384_grid4_multimask.json`. The sample rejects models that have ++CPU operators, non-int8 I/O, or an incompatible output layout. ++ ++Copy the PTE byte-for-byte for J-Link: ++ ++```bash ++cp "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \ ++ "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.bin" ++cmp "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \ ++ "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.bin" ++``` ++ ++## Build ++ ++`SDK_ALIF` lets CMake select sdk-alif's official MT9M114 selfie-camera overlay. ++The application adds the deferred sensor initialization needed by its explicit ++power-up sequence. ++ ++```bash ++west build \ ++ -d "$MOBILE_SAM_OUTPUT/build-mobilesam-live" \ ++ -p always \ ++ -b alif_e8_dk/ae822fa0e5597xx0/rtss_hp \ ++ "$EXECUTORCH_ROOT/zephyr/samples/mobilesam-ethosu" -- \ ++ -DZEPHYR_EXTRA_MODULES="$EXECUTORCH_ROOT;$SDK_ALIF" \ ++ -DSDK_ALIF="$SDK_ALIF" \ ++ -DET_PTE_FILE_PATH="$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \ ++ -DMODEL_METADATA_PATH="$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.json" \ ++ -DPython3_EXECUTABLE="$(command -v python)" ++``` ++ ++A tested build used 7,580,784 bytes of the 7,602,176-byte application RAM ++region and 768,064 bytes of the 786,432-byte SRAM0 region. The application ++checks the PTE's required Ethos-U scratch size during configuration. ++ ++## Flash a new board ++ ++First complete the one-time SETOOLS board setup from the Arm guide. Keep SW4 in ++SEUART, disconnect any serial terminal, and set the Security Toolkit path: ++ ++```bash ++export ALIF_SE_TOOLS_DIR=/absolute/path/to/alif-security-toolkit ++export ALIF_DEVICE=AE822FA0E5597_HP ++export JLINK_COMMANDER=/absolute/path/to/JLinkExe ++``` ++ ++Flash a small idle application before changing OSPI contents. This prevents a ++running XIP application from holding the flash controller: ++ ++```bash ++west build \ ++ -d "$MOBILE_SAM_OUTPUT/build-ospi-idle" \ ++ -p always \ ++ -b alif_e8_dk/ae822fa0e5597xx0/rtss_hp \ ++ "$ZEPHYR_BASE/samples/hello_world" -- \ ++ -DZEPHYR_EXTRA_MODULES="$SDK_ALIF" ++ ++west flash -d "$MOBILE_SAM_OUTPUT/build-ospi-idle" \ ++ --runner alif_flash -- --device "$ALIF_DEVICE" ++``` ++ ++Power-cycle the board, then start J-Link Commander: ++ ++```bash ++"$JLINK_COMMANDER" \ ++ -device AE822FA0E5597LS0_M55_HP \ ++ -if SWD \ ++ -speed 4000 \ ++ -autoconnect 1 ++``` ++ ++At the J-Link prompt, program and verify the model while the target is halted: ++ ++```text ++halt ++loadbin /absolute/path/to/mobilesam_live_384_grid4_multimask.bin,0xC0000000 ++verifybin /absolute/path/to/mobilesam_live_384_grid4_multimask.bin,0xC0000000 ++exit ++``` ++ ++Flash the final firmware through SETOOLS and power-cycle once more: ++ ++```bash ++west flash -d "$MOBILE_SAM_OUTPUT/build-mobilesam-live" \ ++ --runner alif_flash -- --device "$ALIF_DEVICE" ++``` ++ ++The PTE only needs to be programmed again after re-exporting the model. Normal ++firmware rebuilds require only the final `west flash` command. ++ ++## Expected result ++ ++After the sensor's initial stabilization interval, the display continuously ++updates with the live camera image and colored segmentation overlays. A tested ++E8 processed a frame in about 1.34 seconds and selected two distinct objects in ++a representative scene. The exact mask count and timing depend on the scene. ++ ++When a console is available, each completed frame also prints: ++ ++```text ++Frame 9: 1340 ms, 2 objects ++``` ++ ++## Limitations ++ ++- This sample targets the Alif E8 DK hardware and its bundled MT9M114 camera; ++ other boards need their own camera, display, memory, and power configuration. ++- The four prompt coordinates are compiled into the PTE. Re-export to change ++ the grid or input resolution. ++- Up to four distinct masks are displayed. The model emits three candidates ++ per point; the application filters low-quality, oversized, and duplicate ++ masks. ++- Camera capture and display share the board's external MIPI route, so the ++ application alternates them after each frame. ++- The PTE is not linked into the firmware and must remain at OSPI address ++ `0xC0000000`. +diff --git a/zephyr/samples/mobilesam-ethosu/boards/alif_e8_dk_ae822fa0e5597xx0_rtss_hp.conf b/zephyr/samples/mobilesam-ethosu/boards/alif_e8_dk_ae822fa0e5597xx0_rtss_hp.conf +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/boards/alif_e8_dk_ae822fa0e5597xx0_rtss_hp.conf +@@ -0,0 +1,12 @@ ++# Copyright 2026 Arm Limited and/or its affiliates. ++# ++# SPDX-License-Identifier: Apache-2.0 ++ ++CONFIG_ARM_ETHOS_U=y ++CONFIG_ARM_ETHOS_U85_256=y ++ ++CONFIG_FLASH=y ++CONFIG_OSPI_FLASH=y ++CONFIG_OSPI_FLASH_IS25WX=y ++CONFIG_ALIF_OSPI_FLASH_XIP=y ++CONFIG_USE_ALIF_HAL_OSPI=y +diff --git a/zephyr/samples/mobilesam-ethosu/boards/alif_e8_dk_ae822fa0e5597xx0_rtss_hp.overlay b/zephyr/samples/mobilesam-ethosu/boards/alif_e8_dk_ae822fa0e5597xx0_rtss_hp.overlay +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/boards/alif_e8_dk_ae822fa0e5597xx0_rtss_hp.overlay +@@ -0,0 +1,156 @@ ++/* Copyright 2026 Arm Limited and/or its affiliates. ++ * ++ * SPDX-License-Identifier: Apache-2.0 ++ */ ++ ++#include ++#include ++ ++/delete-node/ &sram0; ++/delete-node/ &sram1; ++ ++&{/soc} { ++ sram0: sram@2000000 { ++ compatible = "zephyr,memory-region", "mmio-sram"; ++ reg = <0x02000000 DT_SIZE_K(768)>; ++ zephyr,memory-region = "SRAM0"; ++ zephyr,memory-attr = ; ++ }; ++ ++ app_sram: sram@20c0000 { ++ compatible = "zephyr,memory-region", "mmio-sram"; ++ reg = <0x020c0000 0x740000>; ++ zephyr,memory-region = "APP_RAM"; ++ zephyr,memory-attr = ; ++ }; ++}; ++ ++/ { ++ chosen { ++ zephyr,sram = &app_sram; ++ /delete-property/ zephyr,display; ++ }; ++}; ++ ++&cdc200 { ++ status = "disabled"; ++ width = <480>; ++ height = <800>; ++ hfront-porch = <5>; ++ hback-porch = <5>; ++ hsync-len = <4>; ++ vfront-porch = <10>; ++ vback-porch = <10>; ++ vsync-len = <2>; ++ clock-frequency = <24364080>; ++ hsync-active = <0>; ++ vsync-active = <0>; ++ de-active = <1>; ++ pixelclk-active = <0>; ++ bg-color = <0>; ++ enable-l1 = <1>; ++ pixel-fmt-l1 = "rgb-565"; ++ def-back-color-l1 = <0>; ++ win-x0-l1 = <0>; ++ win-y0-l1 = <0>; ++ win-x1-l1 = <480>; ++ win-y1-l1 = <800>; ++ blend-factor1-l1 = <4>; ++ blend-factor2-l1 = <5>; ++ const-alpha-l1 = <0xff>; ++ enable-l2 = <0>; ++ pixel-fmt-l2 = "undefined"; ++ win-x0-l2 = <0>; ++ win-y0-l2 = <0>; ++ win-x1-l2 = <0>; ++ win-y1-l2 = <0>; ++ blend-factor1-l2 = <6>; ++ blend-factor2-l2 = <7>; ++ const-alpha-l2 = <0xff>; ++}; ++ ++&aipm_run_default { ++ ip-clock-gating = <(ALIF_IP_CLK_GATE_NPU_HP_MASK | ++ ALIF_IP_CLK_GATE_OSPI_1_MASK | ++ ALIF_IP_CLK_GATE_CDC200_MASK | ++ ALIF_IP_CLK_GATE_CAMERA_MASK | ++ ALIF_IP_CLK_GATE_MIPI_CSI_MASK | ++ ALIF_IP_CLK_GATE_MIPI_DSI_MASK)>; ++ phy-pwr-gating = <(ALIF_PHY_GATE_MIPI_TX_DPHY_MASK | ++ ALIF_PHY_GATE_MIPI_RX_DPHY_MASK | ++ ALIF_PHY_GATE_MIPI_PLL_DPHY_MASK | ++ ALIF_PHY_GATE_LDO_MASK)>; ++}; ++ ++&clk_38p4m { ++ status = "okay"; ++}; ++ ++&mipi_dsi { ++ status = "disabled"; ++}; ++ ++&gpio14 { ++ status = "okay"; ++ ++ display_mux { ++ gpio-hog; ++ gpios = <2 GPIO_ACTIVE_HIGH>; ++ output-high; ++ }; ++}; ++ ++&gpio6 { ++ status = "okay"; ++}; ++ ++&gpio9 { ++ status = "okay"; ++}; ++ ++&lpgpio { ++ status = "okay"; ++}; ++ ++&pinctrl { ++ pinctrl_mobilesam_flash: pinctrl_mobilesam_flash { ++ group0 { ++ pinmux = , ++ ; ++ read-enable = <0x0>; ++ drive-strength = <12>; ++ slew-rate = <0x1>; ++ }; ++ group1 { ++ pinmux = , ++ , ++ , ++ , ++ , ++ , ++ , ++ , ++ ; ++ read-enable = <0x1>; ++ drive-strength = <12>; ++ slew-rate = <0x1>; ++ }; ++ }; ++}; ++ ++&ospi1 { ++ pinctrl-0 = <&pinctrl_mobilesam_flash>; ++ status = "okay"; ++}; ++ ++&ospi_flash { ++ status = "okay"; ++}; ++ ++ðosu1 { ++ status = "okay"; ++}; ++ ++&mw405 { ++ status = "disabled"; ++}; +diff --git a/zephyr/samples/mobilesam-ethosu/boards/mt9m114_selfie_deferred.overlay b/zephyr/samples/mobilesam-ethosu/boards/mt9m114_selfie_deferred.overlay +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/boards/mt9m114_selfie_deferred.overlay +@@ -0,0 +1,8 @@ ++/* Copyright 2026 Arm Limited and/or its affiliates. ++ * ++ * SPDX-License-Identifier: Apache-2.0 ++ */ ++ ++&mt9m114_selfie { ++ zephyr,deferred-init; ++}; +diff --git a/zephyr/samples/mobilesam-ethosu/prj.conf b/zephyr/samples/mobilesam-ethosu/prj.conf +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/prj.conf +@@ -0,0 +1,44 @@ ++# Copyright (c) Meta Platforms, Inc. and affiliates. ++# All rights reserved. ++# Copyright 2026 Arm Limited and/or its affiliates. ++# ++# SPDX-License-Identifier: Apache-2.0 ++ ++CONFIG_EXECUTORCH=y ++ ++CONFIG_CPP=y ++CONFIG_CRC=y ++CONFIG_STD_CPP17=y ++ ++CONFIG_MAIN_STACK_SIZE=16384 ++CONFIG_HEAP_MEM_POOL_SIZE=8192 ++ ++CONFIG_REQUIRES_FLOAT_PRINTF=y ++CONFIG_DISPLAY=n ++CONFIG_MIPI_DSI=n ++CONFIG_VIDEO=y ++CONFIG_VIDEO_BUFFER_POOL_SZ_MAX=64 ++CONFIG_VIDEO_BUFFER_POOL_NUM_MAX=1 ++CONFIG_VIDEO_ALIF_CAM_EXTENDED=y ++CONFIG_LOG=y ++CONFIG_LOG_MODE_IMMEDIATE=y ++CONFIG_LOG_DEFAULT_LEVEL=3 ++CONFIG_VIDEO_LOG_LEVEL_DBG=y ++CONFIG_MT9M114_PARALLEL_INIT=n ++CONFIG_I2C=y ++CONFIG_I2C_TARGET=y ++CONFIG_I2C_DW_CLOCK_SPEED=100 ++CONFIG_USE_ALIF_ISP_LIB=y ++CONFIG_ISP_LIB_SCALAR_MODULE=y ++CONFIG_ISP_LIB_DMSC_MODULE=y ++CONFIG_ISP_LIB_CCM_MODULE=n ++CONFIG_ISP_LIB_WBM_MODULE=n ++CONFIG_ISP_LIB_WB_MODULE=n ++CONFIG_EXECUTORCH_BUILD_PORTABLE_OPS=n ++ ++CONFIG_FPU=y ++CONFIG_FP_HARDABI=y ++ ++CONFIG_EXECUTORCH_METHOD_ALLOCATOR_POOL_SIZE=655360 ++CONFIG_EXECUTORCH_TEMP_ALLOCATOR_POOL_SIZE=6488064 ++CONFIG_EXECUTORCH_FAST_TEMP_ALLOCATOR_POOL_SIZE=393216 +diff --git a/zephyr/samples/mobilesam-ethosu/sample.yaml b/zephyr/samples/mobilesam-ethosu/sample.yaml +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/sample.yaml +@@ -0,0 +1,16 @@ ++# Copyright 2026 Arm Limited and/or its affiliates. ++# ++# SPDX-License-Identifier: Apache-2.0 ++ ++sample: ++ name: ExecuTorch live MobileSAM on Alif E8 ++ description: Segments live MT9M114 camera frames on an Alif E8 Ethos-U85. ++common: ++ tags: ++ - executorch ++ - ethos-u ++ build_only: true ++ platform_allow: ++ - alif_e8_dk/ae822fa0e5597xx0/rtss_hp ++tests: ++ sample.executorch.mobilesam_ethosu: {} +diff --git a/zephyr/samples/mobilesam-ethosu/scripts/png_to_logo_header.py b/zephyr/samples/mobilesam-ethosu/scripts/png_to_logo_header.py +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/scripts/png_to_logo_header.py +@@ -0,0 +1,93 @@ ++# Copyright 2026 Arm Limited and/or its affiliates. ++# ++# SPDX-License-Identifier: Apache-2.0 ++ ++from __future__ import annotations ++ ++import argparse ++from pathlib import Path ++ ++import numpy as np ++from PIL import Image ++ ++ ++def resize_contained(image: Image.Image, width: int, height: int) -> Image.Image: ++ image = image.copy() ++ image.thumbnail((width, height), Image.Resampling.LANCZOS) ++ canvas = Image.new("RGBA", (width, height), (0, 0, 0, 0)) ++ canvas.alpha_composite( ++ image, ((width - image.width) // 2, (height - image.height) // 2) ++ ) ++ return canvas ++ ++ ++def prepare_executorch_logo(path: Path) -> Image.Image: ++ return resize_contained(Image.open(path).convert("RGBA"), 40, 40) ++ ++ ++def prepare_pytorch_logo(path: Path) -> Image.Image: ++ source = np.asarray(Image.open(path).convert("RGB"), dtype=np.uint8) ++ alpha = np.max(255 - source, axis=2).astype(np.uint8) ++ foreground = alpha > 0 ++ orange = foreground & (source[:, :, 0] > source[:, :, 1] * 1.5) ++ ++ rgba = np.zeros((*source.shape[:2], 4), dtype=np.uint8) ++ rgba[foreground, :3] = 255 ++ rgba[orange, :3] = (238, 76, 44) ++ rgba[:, :, 3] = alpha ++ image = Image.fromarray(rgba, mode="RGBA") ++ bounding_box = image.getchannel("A").getbbox() ++ if bounding_box is None: ++ raise ValueError(f"PyTorch logo has no visible pixels: {path}") ++ return resize_contained(image.crop(bounding_box), 120, 30) ++ ++ ++def rgb565(red: int, green: int, blue: int) -> int: ++ return ((red & 0xF8) << 8) | ((green & 0xFC) << 3) | (blue >> 3) ++ ++ ++def emit_logo(name: str, image: Image.Image) -> str: ++ data = np.asarray(image, dtype=np.uint8) ++ pixels = [rgb565(*pixel[:3]) for pixel in data.reshape(-1, 4)] ++ alpha = data[:, :, 3].reshape(-1) ++ ++ def rows(values: list[int] | np.ndarray, per_row: int, formatter: object) -> str: ++ return "\n".join( ++ " " ++ + ", ".join(formatter(value) for value in values[index : index + per_row]) ++ + "," ++ for index in range(0, len(values), per_row) ++ ) ++ ++ return f"""const size_t {name}_width = {image.width}; ++const size_t {name}_height = {image.height}; ++const uint16_t {name}_pixels[{len(pixels)}] = {{ ++{rows(pixels, 12, lambda value: f'0x{value:04x}')} ++}}; ++const uint8_t {name}_alpha[{len(alpha)}] = {{ ++{rows(alpha, 20, lambda value: str(int(value)))} ++}}; ++""" ++ ++ ++def main() -> None: ++ parser = argparse.ArgumentParser() ++ parser.add_argument("--executorch-logo", type=Path, required=True) ++ parser.add_argument("--pytorch-logo", type=Path, required=True) ++ parser.add_argument("--output", type=Path, required=True) ++ args = parser.parse_args() ++ ++ header = """#include ++#include ++ ++""" ++ header += emit_logo( ++ "executorch_logo", prepare_executorch_logo(args.executorch_logo) ++ ) ++ header += emit_logo("pytorch_logo", prepare_pytorch_logo(args.pytorch_logo)) ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(header) ++ ++ ++if __name__ == "__main__": ++ main() +diff --git a/zephyr/samples/mobilesam-ethosu/src/camera.cpp b/zephyr/samples/mobilesam-ethosu/src/camera.cpp +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/src/camera.cpp +@@ -0,0 +1,287 @@ ++/* Copyright 2026 Arm Limited and/or its affiliates. ++ * ++ * SPDX-License-Identifier: Apache-2.0 ++ */ ++ ++#include "camera.h" ++#include "display.h" ++ ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++ ++#include ++#include ++ ++#include "soc_common.h" ++ ++extern "C" int log_level() { ++ return 0; ++} ++ ++extern "C" volatile uint32_t camera_diagnostics[24]; ++volatile uint32_t camera_diagnostics[24]; ++ ++namespace { ++ ++const struct device* camera_device; ++struct video_buffer camera_buffer; ++bool frame_dequeued; ++const struct gpio_dt_spec mipi_camera_select = ++ GPIO_DT_SPEC_GET(DT_NODELABEL(camera_select), select_gpios); ++ ++int enable_camera_pixel_clock() { ++ sys_write32(0x140001, CLKCTRL_PER_MST_CAMERA_PIXCLK_CTRL); ++ return 0; ++} ++ ++SYS_INIT(enable_camera_pixel_clock, PRE_KERNEL_1, 46); ++ ++} // namespace ++ ++bool initialize_camera( ++ uint8_t* frame_buffer, ++ size_t frame_buffer_size, ++ size_t width, ++ size_t height) { ++ camera_diagnostics[0] = 0x43414d33; ++ camera_device = DEVICE_DT_GET_ONE(vsi_isp_pico); ++ camera_diagnostics[1] = device_is_ready(camera_device); ++ if (!device_is_ready(camera_device)) { ++ printk("ERROR camera ISP is not ready\n"); ++ return false; ++ } ++ ++ const struct device* camera_sensor = DEVICE_DT_GET_ONE(aptina_mt9m114); ++ const struct gpio_dt_spec sensor_power = ++ GPIO_DT_SPEC_GET(DT_NODELABEL(csi_cam_power), enable_gpios); ++ const struct gpio_dt_spec module_reset{ ++ DEVICE_DT_GET(DT_NODELABEL(gpio9)), 1, GPIO_ACTIVE_LOW}; ++ const struct gpio_dt_spec sensor_sleep{ ++ DEVICE_DT_GET(DT_NODELABEL(gpio6)), 3, GPIO_ACTIVE_LOW}; ++#ifdef CONFIG_MOBILESAM_DISPLAY ++ if (!pause_display()) { ++ return false; ++ } ++#endif ++ int sensor_init_status = ++ gpio_pin_configure_dt(&mipi_camera_select, GPIO_OUTPUT_INACTIVE); ++ if (sensor_init_status == 0) { ++ sensor_init_status = gpio_pin_set_dt(&mipi_camera_select, 1); ++ } ++ if (sensor_init_status == 0) { ++ sensor_init_status = ++ gpio_pin_configure_dt(&module_reset, GPIO_OUTPUT_ACTIVE); ++ } ++ if (sensor_init_status == 0) { ++ sensor_init_status = ++ gpio_pin_configure_dt(&sensor_power, GPIO_OUTPUT_INACTIVE); ++ } ++ if (sensor_init_status == 0) { ++ sensor_init_status = ++ gpio_pin_configure_dt(&sensor_sleep, GPIO_OUTPUT_ACTIVE); ++ } ++ if (sensor_init_status == 0) { ++ k_usleep(100); ++ sensor_init_status = gpio_pin_set_dt(&sensor_power, 1); ++ } ++ if (sensor_init_status == 0) { ++ sensor_init_status = gpio_pin_set_dt(&module_reset, 0); ++ k_usleep(400); ++ } ++ if (sensor_init_status == 0) { ++ sensor_init_status = gpio_pin_set_dt(&sensor_sleep, 0); ++ k_usleep(39); ++ } ++ if (sensor_init_status == 0 && !device_is_ready(camera_sensor)) { ++ sensor_init_status = device_init(camera_sensor); ++ } ++ camera_diagnostics[2] = static_cast(sensor_init_status); ++ camera_diagnostics[9] = static_cast(sensor_init_status); ++ camera_diagnostics[10] = device_is_ready(camera_sensor); ++ if (sensor_init_status != 0 || !device_is_ready(camera_sensor)) { ++ printk( ++ "ERROR MT9M114 camera sensor initialization failed: %d\n", ++ sensor_init_status); ++ return false; ++ } ++ ++ const struct i2c_dt_spec sensor_i2c = ++ I2C_DT_SPEC_GET(DT_NODELABEL(mt9m114_selfie)); ++ uint8_t register_address[2] = {0x00, 0x00}; ++ uint8_t register_value[2]{}; ++ const int probe_status = i2c_write_read_dt( ++ &sensor_i2c, ++ register_address, ++ sizeof(register_address), ++ register_value, ++ sizeof(register_value)); ++ camera_diagnostics[3] = static_cast(probe_status); ++ camera_diagnostics[4] = ++ static_cast(register_value[0] << 8) | register_value[1]; ++ ++ struct video_caps caps{}; ++ const int caps_status = ++ video_get_caps(camera_device, VIDEO_EP_IN, &caps); ++ camera_diagnostics[11] = static_cast(caps_status); ++ if (caps_status != 0) { ++ printk("ERROR camera capabilities unavailable: %d\n", caps_status); ++ return false; ++ } ++ ++ struct video_format format{}; ++ for (size_t i = 0; caps.format_caps[i].pixelformat != 0; ++i) { ++ const auto& candidate = caps.format_caps[i]; ++ if (candidate.pixelformat == VIDEO_PIX_FMT_Y10P) { ++ format.pixelformat = candidate.pixelformat; ++ format.width = candidate.width_min; ++ format.height = candidate.height_min; ++ } ++ } ++ camera_diagnostics[12] = format.pixelformat; ++ camera_diagnostics[13] = (format.width << 16) | format.height; ++ if (format.pixelformat == 0) { ++ printk("ERROR camera RAW10 format unavailable\n"); ++ return false; ++ } ++ ++ format.pitch = format.width * 2; ++ // Sensor set-format starts streaming; restore LP11 before configuring CSI. ++ const int sensor_format_status = ++ video_set_format(camera_sensor, VIDEO_EP_OUT, &format); ++ camera_diagnostics[5] = static_cast(sensor_format_status); ++ if (sensor_format_status != 0) { ++ printk("ERROR camera sensor RAW10 format failed: %d\n", sensor_format_status); ++ return false; ++ } ++ const int sensor_stop_status = video_stream_stop(camera_sensor); ++ camera_diagnostics[6] = static_cast(sensor_stop_status); ++ if (sensor_stop_status != 0) { ++ printk("ERROR camera sensor suspend failed: %d\n", sensor_stop_status); ++ return false; ++ } ++ const int input_status = ++ video_set_format(camera_device, VIDEO_EP_IN, &format); ++ camera_diagnostics[14] = static_cast(input_status); ++ if (input_status != 0) { ++ printk( ++ "ERROR camera input format failed: %d (RAW10 %ux%u pitch %u)\n", ++ input_status, ++ format.width, ++ format.height, ++ format.pitch); ++ return false; ++ } ++ ++ format.pixelformat = VIDEO_PIX_FMT_RGB888_PLANAR_PRIVATE; ++ format.width = width; ++ format.height = height; ++ format.pitch = width * 3; ++ const size_t required_size = format.pitch * format.height; ++ if (frame_buffer_size < required_size) { ++ printk( ++ "ERROR camera needs %zu bytes, model input has %zu\n", ++ required_size, ++ frame_buffer_size); ++ return false; ++ } ++ const int output_status = ++ video_set_format(camera_device, VIDEO_EP_OUT, &format); ++ camera_diagnostics[15] = static_cast(output_status); ++ if (output_status != 0) { ++ printk("ERROR camera output format failed: %d\n", output_status); ++ return false; ++ } ++ ++ camera_buffer = {}; ++ camera_buffer.buffer = frame_buffer; ++ camera_buffer.size = required_size; ++ const int enqueue_status = ++ video_enqueue(camera_device, VIDEO_EP_OUT, &camera_buffer); ++ camera_diagnostics[16] = static_cast(enqueue_status); ++ if (enqueue_status != 0) { ++ printk("ERROR camera buffer enqueue failed: %d\n", enqueue_status); ++ return false; ++ } ++ // The MT9M114 needs this stabilization interval before its first stream. ++ k_msleep(7000); ++ const int stream_status = video_stream_start(camera_device); ++ camera_diagnostics[17] = static_cast(stream_status); ++ if (stream_status != 0) { ++ printk("ERROR camera stream start failed: %d\n", stream_status); ++ return false; ++ } ++ ++ printk( ++ "Camera ready: MT9M114 to planar RGB %zux%zu\n", width, height); ++ return true; ++} ++ ++bool capture_camera_frame() { ++ struct video_buffer* completed_buffer = nullptr; ++ const int status = video_dequeue( ++ camera_device, VIDEO_EP_OUT, &completed_buffer, K_FOREVER); ++ camera_diagnostics[18] = static_cast(status); ++ if (status != 0 || completed_buffer != &camera_buffer) { ++ printk("ERROR camera frame dequeue failed: %d\n", status); ++ return false; ++ } ++ frame_dequeued = true; ++#ifdef CONFIG_MOBILESAM_DISPLAY ++ const int route_status = gpio_pin_set_dt(&mipi_camera_select, 0); ++ if (route_status != 0) { ++ printk("ERROR MIPI display routing failed: %d\n", route_status); ++ return false; ++ } ++ k_usleep(100); ++ return resume_display(); ++#else ++ return true; ++#endif ++} ++ ++bool release_camera_frame() { ++ if (!frame_dequeued) { ++ return false; ++ } ++#ifdef CONFIG_MOBILESAM_DISPLAY ++ if (!pause_display()) { ++ return false; ++ } ++ const int route_status = gpio_pin_set_dt(&mipi_camera_select, 1); ++ if (route_status != 0) { ++ printk("ERROR MIPI camera routing failed: %d\n", route_status); ++ return false; ++ } ++ k_usleep(100); ++#endif ++ const int enqueue_status = ++ video_enqueue(camera_device, VIDEO_EP_OUT, &camera_buffer); ++ camera_diagnostics[22] = static_cast(enqueue_status); ++ if (enqueue_status != 0) { ++ printk("ERROR camera buffer requeue failed: %d\n", enqueue_status); ++ return false; ++ } ++ const int stream_status = video_stream_start(camera_device); ++ if (stream_status != 0 && stream_status != -EBUSY) { ++ printk("ERROR camera stream restart failed: %d\n", stream_status); ++ return false; ++ } ++ frame_dequeued = false; ++ return true; ++} ++ ++void record_live_status( ++ uint32_t frame_number, ++ uint32_t inference_ms, ++ uint32_t object_count) { ++ camera_diagnostics[19] = frame_number; ++ camera_diagnostics[20] = inference_ms; ++ camera_diagnostics[21] = object_count; ++} +diff --git a/zephyr/samples/mobilesam-ethosu/src/camera.h b/zephyr/samples/mobilesam-ethosu/src/camera.h +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/src/camera.h +@@ -0,0 +1,21 @@ ++/* Copyright 2026 Arm Limited and/or its affiliates. ++ * ++ * SPDX-License-Identifier: Apache-2.0 ++ */ ++ ++#pragma once ++ ++#include ++#include ++ ++bool initialize_camera( ++ uint8_t* frame_buffer, ++ size_t frame_buffer_size, ++ size_t width, ++ size_t height); ++bool capture_camera_frame(); ++bool release_camera_frame(); ++void record_live_status( ++ uint32_t frame_number, ++ uint32_t inference_ms, ++ uint32_t object_count); +diff --git a/zephyr/samples/mobilesam-ethosu/src/display.cpp b/zephyr/samples/mobilesam-ethosu/src/display.cpp +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/src/display.cpp +@@ -0,0 +1,482 @@ ++/* Copyright 2026 Arm Limited and/or its affiliates. ++ * ++ * SPDX-License-Identifier: Apache-2.0 ++ */ ++ ++#include "display.h" ++ ++#include ++#include ++#include ++#include ++#include ++ ++#include ++#include ++#include ++#include ++ ++#include "logos.h" ++#include "model_metadata.h" ++ ++namespace { ++ ++constexpr size_t kScreenWidth = 480; ++constexpr size_t kScreenHeight = 800; ++constexpr size_t kImageSize = 448; ++constexpr size_t kImageX = 16; ++constexpr size_t kImageY = 80; ++constexpr float kMaskThreshold = ET_SEGMENTATION_MASK_THRESHOLD; ++ ++constexpr uint16_t rgb565(uint8_t red, uint8_t green, uint8_t blue) { ++ return static_cast( ++ ((red & 0xf8u) << 8) | ((green & 0xfcu) << 3) | (blue >> 3)); ++} ++ ++constexpr uint16_t kBackground = rgb565(5, 10, 20); ++constexpr uint16_t kHeader = rgb565(8, 18, 34); ++constexpr uint16_t kCard = rgb565(14, 29, 47); ++constexpr uint16_t kCardBorder = rgb565(35, 58, 78); ++constexpr uint16_t kWhite = rgb565(242, 247, 250); ++constexpr uint16_t kMuted = rgb565(133, 158, 176); ++constexpr uint16_t kCyan = rgb565(22, 224, 210); ++constexpr uint16_t kPink = rgb565(255, 86, 145); ++constexpr uint16_t kGreen = rgb565(73, 224, 137); ++constexpr uint16_t kYellow = rgb565(255, 213, 64); ++ ++struct MaskColor { ++ uint8_t red; ++ uint8_t green; ++ uint8_t blue; ++ uint16_t boundary; ++}; ++ ++constexpr MaskColor kMaskColors[] = { ++ {22, 224, 210, kCyan}, ++ {255, 86, 145, kPink}, ++ {255, 213, 64, kYellow}, ++ {73, 224, 137, kGreen}, ++}; ++ ++constexpr uint8_t kFont[][5] = { ++ {0x00, 0x00, 0x00, 0x00, 0x00}, {0x00, 0x00, 0x5f, 0x00, 0x00}, ++ {0x00, 0x07, 0x00, 0x07, 0x00}, {0x14, 0x7f, 0x14, 0x7f, 0x14}, ++ {0x24, 0x2a, 0x7f, 0x2a, 0x12}, {0x23, 0x13, 0x08, 0x64, 0x62}, ++ {0x36, 0x49, 0x55, 0x22, 0x50}, {0x00, 0x05, 0x03, 0x00, 0x00}, ++ {0x00, 0x1c, 0x22, 0x41, 0x00}, {0x00, 0x41, 0x22, 0x1c, 0x00}, ++ {0x14, 0x08, 0x3e, 0x08, 0x14}, {0x08, 0x08, 0x3e, 0x08, 0x08}, ++ {0x00, 0x50, 0x30, 0x00, 0x00}, {0x08, 0x08, 0x08, 0x08, 0x08}, ++ {0x00, 0x60, 0x60, 0x00, 0x00}, {0x20, 0x10, 0x08, 0x04, 0x02}, ++ {0x3e, 0x51, 0x49, 0x45, 0x3e}, {0x00, 0x42, 0x7f, 0x40, 0x00}, ++ {0x42, 0x61, 0x51, 0x49, 0x46}, {0x21, 0x41, 0x45, 0x4b, 0x31}, ++ {0x18, 0x14, 0x12, 0x7f, 0x10}, {0x27, 0x45, 0x45, 0x45, 0x39}, ++ {0x3c, 0x4a, 0x49, 0x49, 0x30}, {0x01, 0x71, 0x09, 0x05, 0x03}, ++ {0x36, 0x49, 0x49, 0x49, 0x36}, {0x06, 0x49, 0x49, 0x29, 0x1e}, ++ {0x00, 0x36, 0x36, 0x00, 0x00}, {0x00, 0x56, 0x36, 0x00, 0x00}, ++ {0x08, 0x14, 0x22, 0x41, 0x00}, {0x14, 0x14, 0x14, 0x14, 0x14}, ++ {0x00, 0x41, 0x22, 0x14, 0x08}, {0x02, 0x01, 0x51, 0x09, 0x06}, ++ {0x32, 0x49, 0x79, 0x41, 0x3e}, {0x7e, 0x11, 0x11, 0x11, 0x7e}, ++ {0x7f, 0x49, 0x49, 0x49, 0x36}, {0x3e, 0x41, 0x41, 0x41, 0x22}, ++ {0x7f, 0x41, 0x41, 0x22, 0x1c}, {0x7f, 0x49, 0x49, 0x49, 0x41}, ++ {0x7f, 0x09, 0x09, 0x09, 0x01}, {0x3e, 0x41, 0x49, 0x49, 0x7a}, ++ {0x7f, 0x08, 0x08, 0x08, 0x7f}, {0x00, 0x41, 0x7f, 0x41, 0x00}, ++ {0x20, 0x40, 0x41, 0x3f, 0x01}, {0x7f, 0x08, 0x14, 0x22, 0x41}, ++ {0x7f, 0x40, 0x40, 0x40, 0x40}, {0x7f, 0x02, 0x0c, 0x02, 0x7f}, ++ {0x7f, 0x04, 0x08, 0x10, 0x7f}, {0x3e, 0x41, 0x41, 0x41, 0x3e}, ++ {0x7f, 0x09, 0x09, 0x09, 0x06}, {0x3e, 0x41, 0x51, 0x21, 0x5e}, ++ {0x7f, 0x09, 0x19, 0x29, 0x46}, {0x46, 0x49, 0x49, 0x49, 0x31}, ++ {0x01, 0x01, 0x7f, 0x01, 0x01}, {0x3f, 0x40, 0x40, 0x40, 0x3f}, ++ {0x1f, 0x20, 0x40, 0x20, 0x1f}, {0x3f, 0x40, 0x38, 0x40, 0x3f}, ++ {0x63, 0x14, 0x08, 0x14, 0x63}, {0x07, 0x08, 0x70, 0x08, 0x07}, ++ {0x61, 0x51, 0x49, 0x45, 0x43}, ++}; ++ ++const struct device* display_device; ++struct cdc200_fb_desc framebuffer; ++uint16_t* pixels; ++bool display_ready; ++ ++void fill_rect( ++ size_t x, ++ size_t y, ++ size_t width, ++ size_t height, ++ uint16_t color) { ++ const size_t x_end = std::min(x + width, kScreenWidth); ++ const size_t y_end = std::min(y + height, kScreenHeight); ++ for (size_t row = y; row < y_end; ++row) { ++ std::fill( ++ pixels + row * kScreenWidth + x, ++ pixels + row * kScreenWidth + x_end, ++ color); ++ } ++} ++ ++const uint8_t* glyph(char character) { ++ if (character >= 'a' && character <= 'z') { ++ character -= 'a' - 'A'; ++ } ++ if (character < ' ' || character > 'Z') { ++ character = '?'; ++ } ++ return kFont[character - ' ']; ++} ++ ++void draw_text( ++ size_t x, ++ size_t y, ++ const char* text, ++ size_t scale, ++ uint16_t color) { ++ while (*text != '\0') { ++ const uint8_t* columns = glyph(*text++); ++ for (size_t column = 0; column < 5; ++column) { ++ for (size_t row = 0; row < 7; ++row) { ++ if ((columns[column] & (1u << row)) != 0) { ++ fill_rect(x + column * scale, y + row * scale, scale, scale, color); ++ } ++ } ++ } ++ x += 6 * scale; ++ } ++} ++ ++void draw_centered(size_t y, const char* text, size_t scale, uint16_t color) { ++ const size_t width = std::strlen(text) * 6 * scale - scale; ++ draw_text((kScreenWidth - width) / 2, y, text, scale, color); ++} ++ ++uint16_t alpha_blend(uint16_t background, uint16_t foreground, uint8_t alpha) { ++ const uint32_t inverse_alpha = 255 - alpha; ++ const uint32_t red = (((background >> 11) & 0x1f) * inverse_alpha + ++ ((foreground >> 11) & 0x1f) * alpha + 127) / ++ 255; ++ const uint32_t green = (((background >> 5) & 0x3f) * inverse_alpha + ++ ((foreground >> 5) & 0x3f) * alpha + 127) / ++ 255; ++ const uint32_t blue = ((background & 0x1f) * inverse_alpha + ++ (foreground & 0x1f) * alpha + 127) / ++ 255; ++ return static_cast((red << 11) | (green << 5) | blue); ++} ++ ++void draw_logo( ++ size_t x, ++ size_t y, ++ size_t width, ++ size_t height, ++ const uint16_t* logo_pixels, ++ const uint8_t* logo_alpha) { ++ for (size_t row = 0; row < height; ++row) { ++ for (size_t column = 0; column < width; ++column) { ++ const size_t logo_index = row * width + column; ++ uint16_t& screen_pixel = pixels[(y + row) * kScreenWidth + x + column]; ++ screen_pixel = alpha_blend( ++ screen_pixel, logo_pixels[logo_index], logo_alpha[logo_index]); ++ } ++ } ++} ++ ++uint8_t model_pixel( ++ const int8_t* frame_data, ++ size_t channel, ++ size_t x, ++ size_t y) { ++ const size_t index = ++ channel * image_width * image_height + y * image_width + x; ++ const float normalized = ++ (static_cast(frame_data[index]) - model_input_zero_point) * ++ model_input_scale; ++ const int value = static_cast( ++ normalized * model_pixel_std[channel] + model_pixel_mean[channel] + 0.5f); ++ return static_cast(std::clamp(value, 0, 255)); ++} ++ ++uint16_t blend_mask(uint16_t color, const MaskColor& mask_color) { ++ const uint8_t red = static_cast(((color >> 11) & 0x1f) << 3); ++ const uint8_t green = static_cast(((color >> 5) & 0x3f) << 2); ++ const uint8_t blue = static_cast((color & 0x1f) << 3); ++ return rgb565( ++ static_cast((red * 5 + mask_color.red * 3) / 8), ++ static_cast((green * 5 + mask_color.green * 3) / 8), ++ static_cast((blue * 5 + mask_color.blue * 3) / 8)); ++} ++ ++void draw_image( ++ const int8_t* frame_data, ++ const int8_t* masks, ++ const size_t* mask_offsets, ++ size_t object_count, ++ size_t mask_height, ++ size_t mask_width, ++ size_t row_stride, ++ size_t column_stride) { ++ for (size_t y = 0; y < kImageSize; ++y) { ++ for (size_t x = 0; x < kImageSize; ++x) { ++ uint16_t color = rgb565(8, 14, 22); ++ if (frame_data != nullptr) { ++ const size_t source_x = x * image_width / kImageSize; ++ const size_t source_y = y * image_height / kImageSize; ++ color = rgb565( ++ model_pixel(frame_data, 0, source_x, source_y), ++ model_pixel(frame_data, 1, source_x, source_y), ++ model_pixel(frame_data, 2, source_x, source_y)); ++ } ++ pixels[(kImageY + y) * kScreenWidth + kImageX + x] = color; ++ } ++ } ++ ++ if (masks != nullptr) { ++ for (size_t object = 0; object < object_count; ++object) { ++ const int8_t* mask = masks + mask_offsets[object]; ++ const MaskColor& mask_color = ++ kMaskColors[object % (sizeof(kMaskColors) / sizeof(kMaskColors[0]))]; ++ for (size_t y = 0; y < kImageSize; ++y) { ++ const size_t mask_y = y * mask_height / kImageSize; ++ const size_t y_in_cell = y * mask_height % kImageSize; ++ for (size_t x = 0; x < kImageSize; ++x) { ++ const size_t mask_x = x * mask_width / kImageSize; ++ const size_t index = mask_y * row_stride + mask_x * column_stride; ++ const bool foreground = ++ (static_cast(mask[index]) - model_output_zero_point) * ++ model_output_scale > ++ kMaskThreshold; ++ if (!foreground) { ++ continue; ++ } ++ ++ const size_t x_in_cell = x * mask_width % kImageSize; ++ const auto is_foreground = [&](size_t query_x, size_t query_y) { ++ const size_t query_index = ++ query_y * row_stride + query_x * column_stride; ++ return (static_cast(mask[query_index]) - ++ model_output_zero_point) * ++ model_output_scale > ++ kMaskThreshold; ++ }; ++ const bool boundary = ++ (x_in_cell < mask_width && ++ (mask_x == 0 || !is_foreground(mask_x - 1, mask_y))) || ++ (x_in_cell >= kImageSize - mask_width && ++ (mask_x + 1 == mask_width || ++ !is_foreground(mask_x + 1, mask_y))) || ++ (y_in_cell < mask_height && ++ (mask_y == 0 || !is_foreground(mask_x, mask_y - 1))) || ++ (y_in_cell >= kImageSize - mask_height && ++ (mask_y + 1 == mask_height || ++ !is_foreground(mask_x, mask_y + 1))); ++ uint16_t& pixel = pixels[(kImageY + y) * kScreenWidth + kImageX + x]; ++ pixel = ++ boundary ? mask_color.boundary : blend_mask(pixel, mask_color); ++ } ++ } ++ } ++ } ++ ++ fill_rect(kImageX - 2, kImageY - 2, kImageSize + 4, 2, kCardBorder); ++ fill_rect(kImageX - 2, kImageY + kImageSize, kImageSize + 4, 2, kCardBorder); ++ fill_rect(kImageX - 2, kImageY, 2, kImageSize, kCardBorder); ++ fill_rect(kImageX + kImageSize, kImageY, 2, kImageSize, kCardBorder); ++ ++} ++ ++void draw_card( ++ size_t x, ++ const char* label, ++ const char* value, ++ uint16_t accent) { ++ fill_rect(x, 606, 216, 116, kCardBorder); ++ fill_rect(x + 2, 608, 212, 112, kCard); ++ fill_rect(x + 2, 608, 5, 112, accent); ++ draw_text(x + 18, 622, label, 1, kMuted); ++ draw_text(x + 18, 658, value, 3, kWhite); ++} ++ ++void draw_dashboard( ++ const int8_t* frame_data, ++ const int8_t* masks, ++ const size_t* mask_offsets, ++ size_t object_count, ++ size_t mask_height, ++ size_t mask_width, ++ size_t row_stride, ++ size_t column_stride, ++ uint64_t inference_ms, ++ const size_t* foreground_pixels) { ++ fill_rect(0, 0, kScreenWidth, kScreenHeight, kBackground); ++ fill_rect(0, 0, kScreenWidth, 70, kHeader); ++ fill_rect(0, 68, kScreenWidth, 2, kCyan); ++ fill_rect(10, 10, 48, 48, kWhite); ++ draw_logo( ++ 14, ++ 14, ++ executorch_logo_width, ++ executorch_logo_height, ++ executorch_logo_pixels, ++ executorch_logo_alpha); ++ draw_logo( ++ 350, ++ 20, ++ pytorch_logo_width, ++ pytorch_logo_height, ++ pytorch_logo_pixels, ++ pytorch_logo_alpha); ++ draw_centered(12, "MOBILE SAM", 3, kWhite); ++ draw_centered(48, "ALIF E8 / ETHOS-U85", 1, kMuted); ++ draw_image( ++ frame_data, ++ masks, ++ mask_offsets, ++ object_count, ++ mask_height, ++ mask_width, ++ row_stride, ++ column_stride); ++ ++ fill_rect(16, 548, 448, 42, kCard); ++ fill_rect(16, 548, 6, 42, masks == nullptr ? kYellow : kGreen); ++ char status_text[32]; ++ std::snprintf( ++ status_text, ++ sizeof(status_text), ++ masks == nullptr ? "STARTING LIVE CAMERA" : "%zu OBJECTS SEGMENTED", ++ object_count); ++ draw_text( ++ 34, ++ 562, ++ status_text, ++ 2, ++ masks == nullptr ? kYellow : kGreen); ++ ++ char left_value[24]; ++ char right_value[24]; ++ char footer[64]; ++ if (masks == nullptr) { ++ std::snprintf(left_value, sizeof(left_value), "U85-256"); ++ std::snprintf(right_value, sizeof(right_value), "RUNNING"); ++ std::snprintf( ++ footer, sizeof(footer), "%zu OBJECT PROMPTS READY", point_prompt_count); ++ draw_card(16, "NPU", left_value, kCyan); ++ draw_card(248, "STATE", right_value, kYellow); ++ } else { ++ std::snprintf( ++ left_value, sizeof(left_value), "%" PRIu64 " MS", inference_ms); ++ std::snprintf(right_value, sizeof(right_value), "%zu", object_count); ++ size_t total_foreground = 0; ++ for (size_t object = 0; object < object_count; ++object) { ++ total_foreground += foreground_pixels[object]; ++ } ++ std::snprintf( ++ footer, ++ sizeof(footer), ++ "LIVE CAMERA / %zu MASK PIXELS", ++ total_foreground); ++ draw_card(16, "INFERENCE", left_value, kCyan); ++ draw_card(248, "OBJECTS", right_value, kGreen); ++ } ++ draw_centered(754, footer, 1, kMuted); ++ draw_centered(778, "EXECUTORCH ON DEVICE AI", 1, kMuted); ++ sys_cache_data_flush_range(framebuffer.fb_addr, framebuffer.fb_size); ++} ++ ++} // namespace ++ ++bool initialize_display() { ++ display_device = DEVICE_DT_GET(DT_CHOSEN(zephyr_display)); ++ if (!device_is_ready(display_device)) { ++ printk("ERROR display is not ready\n"); ++ return false; ++ } ++ ++ const struct device* dsi = DEVICE_DT_GET(DT_NODELABEL(mipi_dsi)); ++ if (!device_is_ready(dsi)) { ++ printk("ERROR MIPI-DSI host is not ready\n"); ++ return false; ++ } ++ if (dsi_dw_set_mode(dsi, DSI_DW_VIDEO_MODE) != 0) { ++ printk("ERROR could not enable MIPI-DSI video mode\n"); ++ return false; ++ } ++ ++ cdc200_get_framebuffer(display_device, 0, &framebuffer); ++ if (framebuffer.fb_addr == nullptr || ++ framebuffer.fb_size != kScreenWidth * kScreenHeight * sizeof(uint16_t)) { ++ printk( ++ "ERROR unexpected display framebuffer size: %zu\n", ++ framebuffer.fb_size); ++ return false; ++ } ++ pixels = reinterpret_cast(framebuffer.fb_addr); ++ std::fill(pixels, pixels + kScreenWidth * kScreenHeight, kBackground); ++ sys_cache_data_flush_range(framebuffer.fb_addr, framebuffer.fb_size); ++ cdc200_set_enable(display_device, true); ++ display_ready = true; ++ return true; ++} ++ ++bool pause_display() { ++ if (!display_ready) { ++ printk("ERROR cannot pause an unavailable display\n"); ++ return false; ++ } ++ const int stop_status = ++ cdc200_disable_at_vblank(display_device, K_MSEC(100)); ++ if (stop_status != 0) { ++ printk("ERROR CDC frame-boundary stop failed: %d\n", stop_status); ++ return false; ++ } ++ const struct device* dsi = DEVICE_DT_GET(DT_NODELABEL(mipi_dsi)); ++ const int status = dsi_dw_suspend_video(dsi); ++ if (status != 0) { ++ printk("ERROR MIPI-DSI suspension failed: %d\n", status); ++ return false; ++ } ++ return true; ++} ++ ++bool resume_display() { ++ if (!display_ready) { ++ printk("ERROR cannot resume an unavailable display\n"); ++ return false; ++ } ++ const struct device* dsi = DEVICE_DT_GET(DT_NODELABEL(mipi_dsi)); ++ const int status = dsi_dw_set_mode(dsi, DSI_DW_VIDEO_MODE); ++ if (status != 0) { ++ printk("ERROR MIPI-DSI video resume failed: %d\n", status); ++ return false; ++ } ++ cdc200_set_enable(display_device, true); ++ return true; ++} ++ ++void display_running() { ++ if (display_ready) { ++ draw_dashboard(nullptr, nullptr, nullptr, 0, 0, 0, 0, 0, 0, nullptr); ++ } ++} ++ ++void display_result( ++ const int8_t* frame_data, ++ const int8_t* masks, ++ const size_t* mask_offsets, ++ size_t object_count, ++ size_t mask_height, ++ size_t mask_width, ++ size_t row_stride, ++ size_t column_stride, ++ uint64_t inference_ms, ++ const size_t* foreground_pixels) { ++ if (display_ready) { ++ draw_dashboard( ++ frame_data, ++ masks, ++ mask_offsets, ++ object_count, ++ mask_height, ++ mask_width, ++ row_stride, ++ column_stride, ++ inference_ms, ++ foreground_pixels); ++ } ++} +diff --git a/zephyr/samples/mobilesam-ethosu/src/display.h b/zephyr/samples/mobilesam-ethosu/src/display.h +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/src/display.h +@@ -0,0 +1,25 @@ ++/* Copyright 2026 Arm Limited and/or its affiliates. ++ * ++ * SPDX-License-Identifier: Apache-2.0 ++ */ ++ ++#pragma once ++ ++#include ++#include ++ ++bool initialize_display(); ++bool pause_display(); ++bool resume_display(); ++void display_running(); ++void display_result( ++ const int8_t* frame_data, ++ const int8_t* masks, ++ const size_t* mask_offsets, ++ size_t object_count, ++ size_t mask_height, ++ size_t mask_width, ++ size_t row_stride, ++ size_t column_stride, ++ uint64_t inference_ms, ++ const size_t* foreground_pixels); +diff --git a/zephyr/samples/mobilesam-ethosu/src/main.cpp b/zephyr/samples/mobilesam-ethosu/src/main.cpp +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/src/main.cpp +@@ -0,0 +1,431 @@ ++/* Copyright (c) Meta Platforms, Inc. and affiliates. ++ * All rights reserved. ++ * Copyright 2026 Arm Limited and/or its affiliates. ++ * ++ * SPDX-License-Identifier: Apache-2.0 ++ */ ++ ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++ ++#include ++#include ++#include ++#include ++#include ++#include ++ ++#include "camera.h" ++#include "display.h" ++#include "model_metadata.h" ++ ++using executorch::aten::ScalarType; ++using executorch::aten::Tensor; ++using executorch::extension::BufferDataLoader; ++using executorch::runtime::Error; ++using executorch::runtime::EValue; ++using executorch::runtime::HierarchicalAllocator; ++using executorch::runtime::MemoryManager; ++using executorch::runtime::Method; ++using executorch::runtime::MethodMeta; ++using executorch::runtime::Program; ++using executorch::runtime::Result; ++using executorch::runtime::Span; ++ ++alignas(16) static uint8_t method_pool[ET_ARM_METHOD_ALLOCATOR_POOL_SIZE]; ++__attribute__((section(".bss.tensor_arena"), aligned(16))) static uint8_t ++ temp_pool[ET_ARM_BAREMETAL_SCRATCH_TEMP_ALLOCATOR_POOL_SIZE]; ++ ++extern "C" { ++size_t ethosu_fast_scratch_size = ++ ET_ARM_BAREMETAL_FAST_SCRATCH_TEMP_ALLOCATOR_POOL_SIZE; ++__attribute__((section(".bss.ethosu_scratch"), aligned(16))) static uint8_t ++ fast_scratch[ET_ARM_BAREMETAL_FAST_SCRATCH_TEMP_ALLOCATOR_POOL_SIZE]; ++uint8_t* ethosu_fast_scratch = fast_scratch; ++} ++ ++namespace { ++ ++constexpr size_t kMaxPlannedBuffers = 4; ++constexpr size_t kCandidatesPerPoint = 3; ++constexpr size_t kMaxCandidates = 16; ++constexpr size_t kMaxObjects = 4; ++constexpr float kMaskThreshold = ET_SEGMENTATION_MASK_THRESHOLD; ++constexpr float kDuplicateIou = 0.75f; ++ ++struct Candidate { ++ size_t offset; ++ size_t foreground; ++ float quality; ++}; ++ ++struct SelectedMasks { ++ size_t count; ++ size_t offsets[kMaxObjects]; ++ size_t foreground[kMaxObjects]; ++}; ++ ++float mask_score(const int8_t* data, size_t index) { ++ return (static_cast(data[index]) - model_output_zero_point) * ++ model_output_scale; ++} ++ ++bool is_foreground( ++ const int8_t* data, ++ size_t offset, ++ size_t y, ++ size_t x, ++ size_t row_stride, ++ size_t column_stride) { ++ return mask_score(data, offset + y * row_stride + x * column_stride) > ++ kMaskThreshold; ++} ++ ++size_t count_foreground( ++ const int8_t* data, ++ size_t offset, ++ size_t height, ++ size_t width, ++ size_t row_stride, ++ size_t column_stride) { ++ size_t foreground = 0; ++ for (size_t y = 0; y < height; ++y) { ++ for (size_t x = 0; x < width; ++x) { ++ foreground += is_foreground( ++ data, offset, y, x, row_stride, column_stride); ++ } ++ } ++ return foreground; ++} ++ ++float mask_iou( ++ const int8_t* data, ++ size_t first_offset, ++ size_t second_offset, ++ size_t height, ++ size_t width, ++ size_t row_stride, ++ size_t column_stride) { ++ size_t intersection = 0; ++ size_t union_count = 0; ++ for (size_t y = 0; y < height; ++y) { ++ for (size_t x = 0; x < width; ++x) { ++ const bool first = is_foreground( ++ data, first_offset, y, x, row_stride, column_stride); ++ const bool second = is_foreground( ++ data, second_offset, y, x, row_stride, column_stride); ++ intersection += first && second; ++ union_count += first || second; ++ } ++ } ++ return union_count == 0 ++ ? 1.0f ++ : static_cast(intersection) / static_cast(union_count); ++} ++ ++bool select_masks( ++ const Tensor& masks, ++ const Tensor& quality, ++ SelectedMasks& selected) { ++ if (masks.dim() != 4 || masks.size(0) != point_prompt_count || ++ masks.size(1) != kCandidatesPerPoint || ++ masks.scalar_type() != ScalarType::Char || quality.dim() != 2 || ++ quality.size(0) != point_prompt_count || ++ quality.size(1) != kCandidatesPerPoint || ++ quality.scalar_type() != ScalarType::Char) { ++ printk("ERROR unexpected multimask output tensors\n"); ++ return false; ++ } ++ ++ const size_t candidate_count = masks.size(0) * masks.size(1); ++ if (candidate_count > kMaxCandidates) { ++ printk("ERROR too many mask candidates: %zu\n", candidate_count); ++ return false; ++ } ++ ++ const int8_t* mask_data = masks.const_data_ptr(); ++ const int8_t* quality_data = quality.const_data_ptr(); ++ const auto mask_strides = masks.strides(); ++ const auto quality_strides = quality.strides(); ++ const size_t height = masks.size(2); ++ const size_t width = masks.size(3); ++ const size_t pixel_count = height * width; ++ const size_t minimum_area = std::max(1, pixel_count / 200); ++ const size_t maximum_area = pixel_count * 85 / 100; ++ ++ std::array candidates{}; ++ size_t accepted_count = 0; ++ for (size_t point = 0; point < static_cast(masks.size(0)); ++point) { ++ size_t best_candidate = 0; ++ float best_quality = -1.0f; ++ for (size_t candidate = 0; ++ candidate < static_cast(masks.size(1)); ++ ++candidate) { ++ const size_t quality_index = ++ point * quality_strides[0] + candidate * quality_strides[1]; ++ const float quality_score = ++ (static_cast(quality_data[quality_index]) - ++ model_quality_zero_point) * ++ model_quality_scale; ++ if (quality_score > best_quality) { ++ best_candidate = candidate; ++ best_quality = quality_score; ++ } ++ } ++ if (best_quality < minimum_mask_quality) { ++ continue; ++ } ++ ++ const size_t offset = ++ point * mask_strides[0] + best_candidate * mask_strides[1]; ++ const size_t foreground = count_foreground( ++ mask_data, ++ offset, ++ height, ++ width, ++ mask_strides[2], ++ mask_strides[3]); ++ if (foreground < minimum_area || foreground > maximum_area) { ++ continue; ++ } ++ candidates[accepted_count++] = {offset, foreground, best_quality}; ++ } ++ ++ std::sort( ++ candidates.begin(), ++ candidates.begin() + accepted_count, ++ [](const Candidate& first, const Candidate& second) { ++ return first.quality > second.quality; ++ }); ++ ++ selected.count = 0; ++ for (size_t index = 0; ++ index < accepted_count && selected.count < kMaxObjects; ++ ++index) { ++ bool duplicate = false; ++ for (size_t object = 0; object < selected.count; ++object) { ++ if (mask_iou( ++ mask_data, ++ candidates[index].offset, ++ selected.offsets[object], ++ height, ++ width, ++ mask_strides[2], ++ mask_strides[3]) > kDuplicateIou) { ++ duplicate = true; ++ break; ++ } ++ } ++ if (!duplicate) { ++ selected.offsets[selected.count] = candidates[index].offset; ++ selected.foreground[selected.count] = candidates[index].foreground; ++ ++selected.count; ++ } ++ } ++ ++ printk( ++ "Mask proposals: %zu accepted, %zu distinct\n", ++ accepted_count, ++ selected.count); ++ return true; ++} ++ ++void quantize_camera_frame(int8_t* data) { ++ const size_t plane_size = image_width * image_height; ++ auto* camera_pixels = reinterpret_cast(data); ++ for (size_t channel = 0; channel < image_channels; ++channel) { ++ for (size_t pixel = 0; pixel < plane_size; ++pixel) { ++ const size_t index = channel * plane_size + pixel; ++ const float normalized = ++ (static_cast(camera_pixels[index]) - ++ model_pixel_mean[channel]) / ++ model_pixel_std[channel]; ++ const int32_t quantized = static_cast( ++ std::round(normalized / model_input_scale)) + ++ model_input_zero_point; ++ data[index] = static_cast(std::clamp(quantized, -128, 127)); ++ } ++ } ++} ++ ++} // namespace ++ ++int main() { ++ printk("\nLive MobileSAM / Ethos-U85-256 / Alif E8\n"); ++ executorch::runtime::runtime_init(); ++#ifdef CONFIG_MOBILESAM_DISPLAY ++ if (!initialize_display()) { ++ printk("ERROR display initialization failed\n"); ++ return 1; ++ } ++ display_running(); ++#else ++ printk("Camera + NPU mode; on-board dashboard disabled\n"); ++#endif ++ ++#if DT_NODE_EXISTS(DT_NODELABEL(ospi_flash)) ++ const struct device* ospi_flash = DEVICE_DT_GET(DT_NODELABEL(ospi_flash)); ++ if (!device_is_ready(ospi_flash)) { ++ printk("ERROR OSPI flash is not ready\n"); ++ return 1; ++ } ++#else ++ printk("ERROR E8 OSPI flash device is unavailable\n"); ++ return 1; ++#endif ++ ++#if !defined(CONFIG_ALIF_OSPI_FLASH_XIP) ++ printk("ERROR OSPI XIP must be enabled\n"); ++ return 1; ++#else ++ const auto* pte = reinterpret_cast(ET_MODEL_PTE_ADDRESS); ++ printk( ++ "PTE address: 0x%08x, size: %u bytes\n", ++ static_cast(ET_MODEL_PTE_ADDRESS), ++ static_cast(ET_MODEL_PTE_SIZE)); ++ ++ BufferDataLoader loader(pte, ET_MODEL_PTE_SIZE); ++ Result program_result = Program::load(&loader); ++ if (!program_result.ok()) { ++ printk( ++ "ERROR Program::load: 0x%08" PRIx32 "\n", ++ static_cast(program_result.error())); ++ return 1; ++ } ++ Program program = std::move(program_result.get()); ++ Result meta_result = program.method_meta("forward"); ++ if (!meta_result.ok()) { ++ printk("ERROR method metadata\n"); ++ return 1; ++ } ++ MethodMeta meta = std::move(meta_result.get()); ++ ++ ArmMemoryAllocator method_allocator(sizeof(method_pool), method_pool); ++ Span planned_spans[kMaxPlannedBuffers]; ++ const size_t planned_count = meta.num_memory_planned_buffers(); ++ if (planned_count > kMaxPlannedBuffers) { ++ printk("ERROR too many planned buffers: %zu\n", planned_count); ++ return 1; ++ } ++ for (size_t i = 0; i < planned_count; ++i) { ++ const auto size_result = meta.memory_planned_buffer_size(i); ++ if (!size_result.ok()) { ++ return 1; ++ } ++ const size_t size = size_result.get(); ++ auto* buffer = static_cast(method_allocator.allocate(size, 16)); ++ if (buffer == nullptr) { ++ printk("ERROR planned buffer %zu (%zu bytes)\n", i, size); ++ return 1; ++ } ++ planned_spans[i] = Span(buffer, size); ++ printk("Planned buffer %zu: %zu bytes\n", i, size); ++ } ++ ++ HierarchicalAllocator planned_memory{ ++ Span>(planned_spans, planned_count)}; ++ ArmMemoryAllocator temp_allocator(sizeof(temp_pool), temp_pool); ++ MemoryManager memory_manager( ++ &method_allocator, &planned_memory, &temp_allocator); ++ Result method_result = ++ program.load_method("forward", &memory_manager); ++ if (!method_result.ok()) { ++ printk( ++ "ERROR load_method: 0x%08" PRIx32 "\n", ++ static_cast(method_result.error())); ++ return 1; ++ } ++ Method method = std::move(method_result.get()); ++ ++ EValue input; ++ Error status = method.get_inputs(&input, 1); ++ if (status != Error::Ok || !input.isTensor()) { ++ printk("ERROR get input: 0x%08" PRIx32 "\n", (uint32_t)status); ++ return 1; ++ } ++ Tensor input_tensor = input.toTensor(); ++ const size_t camera_frame_size = image_channels * image_width * image_height; ++ if (input_tensor.scalar_type() != ScalarType::Char || ++ input_tensor.nbytes() != camera_frame_size) { ++ printk( ++ "ERROR expected %zu-byte int8 input, got %zu bytes type %d\n", ++ camera_frame_size, ++ input_tensor.nbytes(), ++ static_cast(input_tensor.scalar_type())); ++ return 1; ++ } ++ auto* input_data = input_tensor.mutable_data_ptr(); ++ if (!initialize_camera( ++ reinterpret_cast(input_data), ++ input_tensor.nbytes(), ++ image_width, ++ image_height)) { ++ while (true) { ++ k_sleep(K_SECONDS(1)); ++ } ++ } ++ ++ size_t frame_number = 0; ++ while (true) { ++ if (!capture_camera_frame()) { ++ return 1; ++ } ++ ++frame_number; ++ quantize_camera_frame(input_data); ++ ++ const uint64_t start_cycles = k_cycle_get_64(); ++ status = method.execute(); ++ const uint64_t elapsed_cycles = k_cycle_get_64() - start_cycles; ++ if (status != Error::Ok) { ++ printk("ERROR execute: 0x%08" PRIx32 "\n", (uint32_t)status); ++ return 1; ++ } ++ const uint64_t inference_ms = k_cyc_to_ms_floor64(elapsed_cycles); ++ ++ EValue outputs[2]; ++ status = method.get_outputs(outputs, 2); ++ if (status != Error::Ok || !outputs[0].isTensor() || ++ !outputs[1].isTensor()) { ++ printk("ERROR get multimask outputs\n"); ++ return 1; ++ } ++ Tensor masks = outputs[0].toTensor(); ++ Tensor quality = outputs[1].toTensor(); ++ SelectedMasks selected{}; ++ if (!select_masks(masks, quality, selected)) { ++ return 1; ++ } ++ ++#ifdef CONFIG_MOBILESAM_DISPLAY ++ const auto mask_strides = masks.strides(); ++ display_result( ++ input_data, ++ masks.const_data_ptr(), ++ selected.offsets, ++ selected.count, ++ masks.size(2), ++ masks.size(3), ++ mask_strides[2], ++ mask_strides[3], ++ inference_ms, ++ selected.foreground); ++#endif ++ record_live_status(frame_number, inference_ms, selected.count); ++ printk( ++ "Frame %zu: %" PRIu64 " ms, %zu objects\n", ++ frame_number, ++ inference_ms, ++ selected.count); ++ ++ if (!release_camera_frame()) { ++ return 1; ++ } ++ } ++#endif ++} +diff --git a/zephyr/samples/mobilesam-ethosu/dashboard.conf b/zephyr/samples/mobilesam-ethosu/dashboard.conf +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/dashboard.conf +@@ -0,0 +1,6 @@ ++# Copyright 2026 Arm Limited and/or its affiliates. ++# SPDX-License-Identifier: Apache-2.0 ++ ++CONFIG_DISPLAY=y ++CONFIG_MIPI_DSI=y ++CONFIG_MOBILESAM_DISPLAY=y +diff --git a/zephyr/samples/mobilesam-ethosu/boards/dashboard.overlay b/zephyr/samples/mobilesam-ethosu/boards/dashboard.overlay +new file mode 100644 +--- /dev/null ++++ b/zephyr/samples/mobilesam-ethosu/boards/dashboard.overlay +@@ -0,0 +1,21 @@ ++/* Copyright 2026 Arm Limited and/or its affiliates. ++ * SPDX-License-Identifier: Apache-2.0 ++ */ ++ ++/ { ++ chosen { ++ zephyr,display = &cdc200; ++ }; ++}; ++ ++&cdc200 { ++ status = "okay"; ++}; ++ ++&mipi_dsi { ++ status = "okay"; ++}; ++ ++&mw405 { ++ status = "okay"; ++}; diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/ospi-writer.zip b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/ospi-writer.zip new file mode 100644 index 0000000000..f5408d153f Binary files /dev/null and b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/ospi-writer.zip differ diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/prepare-the-environment.md b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/prepare-the-environment.md new file mode 100644 index 0000000000..091fe02e24 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/prepare-the-environment.md @@ -0,0 +1,152 @@ +--- +title: Prepare ExecuTorch, Zephyr, and the Alif E8 +description: Install the matching MobileSAM sample source, pin the Alif SDKs, and configure the E8 programming tools. +weight: 3 +layout: learningpathall +--- + +## Check the host and dependency requirements + +Use an Apple silicon Mac. The pinned Arm setup and Python installer do not support Intel macOS for this workflow. + +{{% notice Important %}} +The supplied sample patch uses released PyTorch and TorchAO packages by default. Install the exporter from this source checkout so it matches the ExecuTorch runtime used to build the firmware. +{{% /notice %}} + +## Set up the MobileSAM sample source + +The live-camera application uses int8 I/O, independent prompts, camera capture, and UART reporting. The display is disabled by default; a separate configuration enables an experimental dashboard. Download the [sample patch](../mobilesam-alif-e8.patch), then apply it to the matching public upstream base. + +Choose a new workspace and the installed Zephyr SDK path: + +```bash +export ALIF_WORK="$HOME/mobilesam-alif-e8" +export EXECUTORCH_ROOT="$ALIF_WORK/executorch" +export ZEPHYR_BASE="$ALIF_WORK/zephyr_alif" +export SDK_ALIF="$ALIF_WORK/sdk-alif" +export ZEPHYR_SDK_INSTALL_DIR=/absolute/path/to/zephyr-sdk-0.17.0 +export ZEPHYR_TOOLCHAIN_VARIANT=zephyr +export MOBILE_SAM_OUTPUT="$ALIF_WORK/model-and-builds" +export MOBILE_SAM_SOURCE="$MOBILE_SAM_OUTPUT/mobile-sam/source" + +mkdir -p "$ALIF_WORK" "$MOBILE_SAM_OUTPUT" +``` + +Save the downloaded patch as `$ALIF_WORK/mobilesam-alif-e8.patch`. Check its SHA256 with `shasum -a 256`; the expected value is: + +```output +e49e11ce7becf49d4b58f3618f97f38e8bdbc8e31dd43f441fbde75e84a794ec +``` + +Clone ExecuTorch and apply the patch to the pinned base: + +```bash +git clone https://github.com/pytorch/executorch.git "$EXECUTORCH_ROOT" +cd "$EXECUTORCH_ROOT" +git checkout --detach baafd7e8eaa4ed7f7e1cacaa42db6ae16f247b2d +git apply --check "$ALIF_WORK/mobilesam-alif-e8.patch" +git apply "$ALIF_WORK/mobilesam-alif-e8.patch" +git submodule update --init --recursive +``` + +The patch starts from the live-camera sample at revision `488e4fe96d97af1189d66916477fb017b52a2245`, uses released CPU dependencies, and stages the Ethos-U headers from the pinned Zephyr driver's own module. Its default `CONFIG_MOBILESAM_DISPLAY=n` excludes display initialization and keeps the MIPI route on the camera. It removes a misleading checksum log; the writer's CRC and flash readback provide the model-integrity checks. Applying the patch leaves local modifications on the pinned upstream base. Installation and the public-image export passed with the released dependencies. The validation lesson records the exact firmware tested, integer-reference agreement, and the failed original strict PT2E comparison. + +## Pin the vendor repositories + +Clone the vendor repositories into the new workspace and select the tested revisions: + +```bash +git clone https://github.com/alifsemi/zephyr_alif.git "$ZEPHYR_BASE" +git -C "$ZEPHYR_BASE" checkout --detach \ + f002a4d8499c35fc70ec4d5324faa7b01c152c72 + +git clone https://github.com/alifsemi/sdk-alif.git "$SDK_ALIF" +git -C "$SDK_ALIF" checkout --detach \ + 1db82c9d0cc594e2b548a13699e8a4157e6dcd4c +``` + +The application also requires the [camera and display driver patch](../alif-e8-camera-display.patch) for this exact vendor revision. Save it as `$ALIF_WORK/alif-e8-camera-display.patch`. Its SHA256 is: + +```output +8bf1b9a9a6bcd4bafe854ead8645eb1f6370d67b56b6aa2bdd5395e0d8ee3961 +``` + +Apply it before building: + +```bash +git -C "$ZEPHYR_BASE" apply --check "$ALIF_WORK/alif-e8-camera-display.patch" +git -C "$ZEPHYR_BASE" apply "$ALIF_WORK/alif-e8-camera-display.patch" +``` + +The camera changes arm the receiver before starting the upstream sensor, publish completed frames only after the pipeline stops, and reset the idle CSI IPI at stream start. The patch also supplies frame-boundary display suspension for the optional dashboard. The default build does not enable that display path. Dashboard runs retained DSI error reports despite visibly advancing frames; enabling the extension requires accepting and investigating that known limitation. Peripheral error reporting remains enabled. + +The original validated build used Zephyr 4.1.0, Zephyr SDK 0.17.0, Python 3.12.13, west 1.5.0, and J-Link Commander 9.66. Install the SDK and vendor tools for your host before continuing. Use macOS on Apple silicon, matching the validated host. + +## Establish the Python and Arm tools + +Create a Python 3.12 environment and install the matching ExecuTorch source and Arm tools: + +```bash +python3.12 -m venv "$ALIF_WORK/.venv" +source "$ALIF_WORK/.venv/bin/activate" +cd "$EXECUTORCH_ROOT" +CMAKE_GENERATOR=Ninja ./install_executorch.sh --minimal --optional-dependency ethos_u +python -m pip install torchvision==0.28.0 timm==1.0.7 west==1.5.0 fdt +``` + +The patched installer selects released `torch==2.13.0` and `torchao==0.18.0` packages. Its `ethos_u` extra supplies Vela 5.1.0 and TOSA tools 2026.5.0. `--minimal` skips unrelated example dependencies; the explicit package command supplies this workflow’s image and Zephyr tools. The installer still builds the ExecuTorch host components, and the Zephyr firmware is built separately in a later lesson. Keep this Python environment active for export and firmware metadata generation. + +The separately configured Zephyr SDK supplies the firmware toolchain. + +## Initialize the west workspace once + +Use the same active Python environment for export and CMake metadata generation: + +```bash +cd "$ALIF_WORK" +west init -l "$ZEPHYR_BASE" +west update +west manifest --freeze --active-only > "$MOBILE_SAM_OUTPUT/west-manifest.yml" +python -m pip install -r "$ZEPHYR_BASE/scripts/requirements-base.txt" +python -m pip check +west zephyr-export +python -m pip freeze > "$MOBILE_SAM_OUTPUT/python-requirements.txt" +cd "$EXECUTORCH_ROOT" +``` + +`--active-only` freezes the active projects selected by this manifest; unused optional projects do not need to be cloned. Run `west init` only for a new workspace. If resuming an existing workspace, verify its manifest and revisions instead of initializing again. The vendor manifest includes moving references, including LVGL's `generic-arm-cortex-m-flagging` branch and imported debug metadata on `main`. Pinning `zephyr_alif` alone therefore does not freeze every west dependency. The clean vendor bootstrap completed all 58 active projects, and its frozen manifest was used for an independent firmware build with the public calibration model. Preserve the manifest generated above and your resolved Python dependencies with your validation results. + +## Prepare board programming and debug access + +With board power disconnected, retain the vendor's default jumper configuration and verify that **SW3 (JLOB) is set to EN**, as shown in Figure 1 of the [official Alif DK-E8 User Guide (AUGD0023)](https://alifsemi.com/download/AUGD0023). + +Use the separate **SW4 UART selector on SEUART** for SETOOLS ISP programming. SW3 enables the on-board J-Link block; SW4 selects which UART reaches the USB-to-UART bridge. The programming lesson uses J-Link SWD for application MRAM, which does not depend on SW4's UART selection. + +Connect a data-capable USB cable to the E8's **PRG USB** port. A power LED alone does not establish USB data connectivity. Check that the host can discover both the serial device used by SETOOLS and the J-Link probe. + +For first-time provisioning, follow the Security Toolkit and debug-probe portions of the [Arm E8 board setup guide](https://learn.arm.com/learning-paths/embedded-and-microcontrollers/alif-image-classification/board-setup/). That guide's later CMSIS image-classification application is a separate project; this tutorial uses Zephyr. Close any terminal holding SEUART before programming. + +Set the paths to your installed tools: + +```bash +export ALIF_SE_TOOLS_DIR=/absolute/path/to/alif-security-toolkit +export JLINK_COMMANDER=/absolute/path/to/JLinkExe +export ALIF_DEVICE=AE822FA0E5597LS0_M55_HP +export ALIF_APP_CONFIG="$ALIF_SE_TOOLS_DIR/build/config/app-cpu-stubs.json" +command -v west +command -v expect +test -x "$JLINK_COMMANDER" +test -s "$ALIF_APP_CONFIG" +``` + +This workflow targets the tested E8 revision A0 device, `AE822FA0E5597LS0`. Use the Security Toolkit configuration for that silicon revision; an A0/A1 mismatch prevents correct programming. The full device name above selects its HP core. Other silicon revisions require separately validated board and tool configurations. + +Use `ALIF_APP_CONFIG` only after confirming it contains the device configuration and companion images for your working E8 setup. If that setup uses a different application JSON, set the variable to that file instead. The programming lesson creates separate writer and live-application configurations from this template and preserves its other boot entries. + +The onboard USB-to-UART bridge uses SW4 to select SEUART for SETOOLS ISP or UART4 for the HP application console. To capture application logs, disconnect board power, select UART4, and reconnect PRG USB. The application console runs at 115200 baud, 8N1. Return SW4 to SEUART before using SETOOLS ISP again; J-Link SWD uses the separate debug connection. + +Before proceeding, confirm that your source includes `zephyr/samples/mobilesam-ethosu`, your SDK contains the MT9M114 selfie overlay, and your programming tools can discover the E8. + +## What you've accomplished + +You have selected the matching sample source and vendor SDKs, initialized the Zephyr workspace, and configured the programming tools for E8 HP. Next, export the four-prompt model and inspect its quantization and delegation artifacts. diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/prepare_mram_programming.py b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/prepare_mram_programming.py new file mode 100644 index 0000000000..eb6c7656c0 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/prepare_mram_programming.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python3 +"""Prepare and verify files offline for the tested E8 A0 HP SWD procedure.""" + +import argparse +import hashlib +import json +from pathlib import Path +import re +import shutil + + +APP_START = 0x80000000 +APP_END = 0x80580000 +HP_ADDRESS = 0x80200000 +SECTOR_SIZE = 0x4000 + + +def one_match(pattern, text, label): + matches = re.findall(pattern, text, re.MULTILINE) + if len(matches) != 1: + raise ValueError(f"Expected exactly one {label}; found {len(matches)}") + return matches[0] + + +def file_info(path, address): + path = path.resolve(strict=True) + data = path.read_bytes() + size = len(data) + end = address + size + sector_start = address // SECTOR_SIZE * SECTOR_SIZE + sector_end = (end + SECTOR_SIZE - 1) // SECTOR_SIZE * SECTOR_SIZE + if not size or address % 16 or size % 16: + raise ValueError(f"{path.name}: address and nonzero size must be 16-byte aligned") + if not APP_START <= sector_start < sector_end <= APP_END: + raise ValueError(f"{path.name}: programming envelope exceeds E8 application MRAM") + return { + "source": str(path), + "address": hex(address), + "size": size, + "end_exclusive": hex(end), + "sector_start": hex(sector_start), + "sector_end_exclusive": hex(sector_end), + "sha256": hashlib.sha256(data).hexdigest(), + } + + +def prepare(args): + config = args.config.read_text() + board_target = one_match(r'^CONFIG_BOARD_TARGET="([^"]+)"$', config, "board target") + if (board_target != "alif_e8_dk/ae822fa0e5597xx0/rtss_hp" + or not re.search(r"^CONFIG_SOC_AE822FA0E5597XX0_RTSS_HP=y$", config, re.MULTILINE)): + raise ValueError("This procedure requires an E8 DK AE822FA0 HP build") + values = [ + int(one_match(rf"^{key}=(0x[0-9a-fA-F]+|[0-9]+)$", config, key), 0) + for key in ("CONFIG_FLASH_BASE_ADDRESS", "CONFIG_FLASH_LOAD_OFFSET") + ] + address = sum(values) + if values[0] != APP_START or address != HP_ADDRESS: + raise ValueError("This procedure requires E8 A0 HP firmware linked at 0x80200000") + + package_map = args.package_map.read_text() + before_summary, separator, summary = package_map.partition("APP Package Summary:") + if not separator: + raise ValueError("Missing vendor APP Package Summary") + user_section = before_summary.split("User managed MRAM locations", 1) + if len(user_section) != 2: + raise ValueError("Missing user-managed image section") + user_section = user_section[1].split("APP Package Start Address", 1)[0] + row = one_match( + r"^(0x[0-9a-fA-F]+)\s+(0x[0-9a-fA-F]+)\s+([0-9]+)\s+(\S+)\s+(.+?)\s*$", + user_section, + "user-managed image", + ) + image_address, hex_size, decimal_size, signing, filename = row + if signing != "signed" or filename != args.firmware.name: + raise ValueError("Firmware basename must match the sole signed user-managed image") + if int(image_address, 0) != address or int(hex_size, 0) != int(decimal_size): + raise ValueError("User-managed image address or size disagrees with the build/map") + + package_address = int(one_match( + r"^\s*- APP Package Start Address:\s*(0x[0-9a-fA-F]+)\s*$", + summary, "package start address", + ), 0) + package_size = int(one_match( + r"^\s*- APP Package total size:\s*([0-9]+) bytes\s*$", + summary, "package size", + )) + firmware = file_info(args.firmware, address) + package = file_info(args.package, package_address) + if firmware["size"] != int(decimal_size) or package["size"] != package_size: + raise ValueError("Actual staged file sizes disagree with the vendor map") + if int(package["end_exclusive"], 0) != APP_END: + raise ValueError("Signed package must end exactly at 0x80580000") + if int(firmware["sector_end_exclusive"], 0) > int(package["sector_start"], 0): + raise ValueError("Firmware and package programming envelopes overlap") + if args.firmware.suffix != ".bin" or args.firmware.name.casefold() in { + "apptocpackage.bin", "firmware-readback.bin", "package-readback.bin" + }: + raise ValueError("Firmware must be a .bin image with a non-reserved basename") + + if args.app_config: + hp = json.loads(args.app_config.read_text())["HP_APP"] + if (hp.get("disabled", False) or hp.get("signed") is not True + or hp.get("binary") != filename or hp.get("cpu_id") != "M55_HP" + or int(hp.get("mramAddress", "0"), 0) != address + or hp.get("flags") != ["boot"]): + raise ValueError("HP_APP configuration disagrees with this MRAM boot procedure") + + output = args.output_dir.resolve() + for path in (output, args.firmware.resolve(), args.package.resolve()): + if any(character in str(path) for character in ('"', '\r', '\n')): + raise ValueError("J-Link paths cannot contain quotes or newlines") + if output.exists() and any(output.iterdir()): + raise ValueError("Output directory must be new or empty; preserve previous evidence") + output.mkdir(parents=True, exist_ok=True) + for name, record, snapshot_name in ( + ("firmware", firmware, filename), ("package", package, "AppTocPackage.bin") + ): + snapshot = output / snapshot_name + shutil.copyfile(record["source"], snapshot) + if hashlib.sha256(snapshot.read_bytes()).hexdigest() != record["sha256"]: + raise ValueError(f"{name} changed while preparing its snapshot; start again") + record["file"] = str(snapshot) + record["readback"] = str(output / f"{name}-readback.bin") + shutil.copyfile(args.config, output / "build.config") + shutil.copyfile(args.package_map, output / "app-package-map.txt") + if args.app_config: + shutil.copyfile(args.app_config, output / "app-config.json") + + records = (firmware, package) + (output / "01-program.jlink").write_text("H\nSleep 1000\n" + "".join( + f'LoadFile "{item["file"]}" {item["address"]} noreset\n' for item in records + ) + "Exit\n") + (output / "02-readback-fresh-process.jlink").write_text("H\n" + "".join( + f'SaveBin "{item["readback"]}" {item["address"]} {hex(item["size"])}\n' + for item in records + ) + "Exit\n") + (output / "03-reset-probe-only.jlink").write_text( + "R0\nSleep 200\nR1\nSleep 2500\nExit\n" + ) + manifest = { + "schema_version": 2, + "device": "AE822FA0E5597LS0_M55_HP", + "image_boot_device": "AE822FA0E5597LS0_M55_HP", + "programming_core": args.programming_core, + "programming_device": f"AE822FA0E5597LS0_M55_{args.programming_core}", + "board_target": board_target, + "interface": "SWD", + "speed_khz": 4000, + "application_mram": {"start": hex(APP_START), "end_exclusive": hex(APP_END)}, + "observed_sector_size": SECTOR_SIZE, + "firmware": firmware, + "package": package, + "procedure": [ + f"Run 01 in a named {args.programming_core} connection; close Commander afterwards.", + f"Run 02 in a NEW named {args.programming_core} connection with no preceding downloads; close it.", + "Run offline verify and require both physical readbacks to match.", + "Run 03 in a NEW probe-only process with AutoConnect 0; never connect a CPU there.", + "Open a NEW named HP connection after the pin pulse to check actual boot.", + ], + "scope": "Offline range/name/hash checks; vendor-generated signatures are not authenticated here.", + } + (output / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n") + print(f"Prepared {output / 'manifest.json'}; no debugger was launched") + + +def verify(args): + manifest = json.loads(args.manifest.read_text()) + results = {} + for name in ("firmware", "package"): + item = manifest[name] + for path in (Path(item["file"]), Path(item["readback"])): + data = path.read_bytes() + if len(data) != item["size"] or hashlib.sha256(data).hexdigest() != item["sha256"]: + raise ValueError(f"{name}: size/SHA-256 mismatch in {path}") + results[name] = {"readback": item["readback"], "sha256": item["sha256"], "match": True} + print(json.dumps(results, indent=2)) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest="command", required=True) + setup = commands.add_parser("prepare", help="Validate inputs and emit command fragments offline") + for flag in ("config", "firmware", "package", "package-map", "output-dir"): + setup.add_argument(f"--{flag}", type=Path, required=True) + setup.add_argument("--app-config", type=Path, help="Optional matching vendor HP_APP JSON") + setup.add_argument( + "--programming-core", choices=("HP", "HE"), default="HP", + help="Core used for MRAM debug access; the signed application still boots HP", + ) + check = commands.add_parser("verify", help="Compare fresh target readbacks with prepared hashes") + check.add_argument("--manifest", type=Path, required=True) + args = parser.parse_args() + try: + if args.command == "prepare": + prepare(args) + else: + verify(args) + except (OSError, ValueError, KeyError, TypeError) as error: + parser.error(str(error)) + + +if __name__ == "__main__": + main() diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/program-the-board.md b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/program-the-board.md new file mode 100644 index 0000000000..30f01a6ea1 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/program-the-board.md @@ -0,0 +1,259 @@ +--- +title: Program the external model and final firmware +description: Preserve the E8 boot configuration, verify the model in OSPI flash, and install its matching live-camera firmware. +weight: 6 +layout: learningpathall +--- + +## Use the tested A0 programming route + +This lesson uses J-Link SWD to install two MRAM payloads: the HP firmware and its signed Alif application table-of-contents package (AToC). The board-side writer then programs the separate model in external OSPI flash at `0xC0000000`. + +Writer installation, fresh MRAM readback, reset-pin boot, and the complete public-model transfer were exercised on the original E8 A0 board. A replacement A0 board also passed writer installation and fresh readback through its HE debug connection, then booted the HP writer. The default camera-and-UART firmware passed fresh payload readbacks and two independent reset-pin runtime checks; the validation lesson gives their scope. This procedure targets an HP image for `AE822FA0E5597LS0`; other silicon revisions need separate validation. + +Retain data you need from application MRAM and the model's OSPI region before replacing them. Close other debugger sessions. SW4 selects the UART bridge, not SWD: use UART4 for the HP console or SEUART for SETOOLS ISP. Installed SETOOLS rejects `Interface=jtag`, and the pinned `west flash` runner uses ISP; neither command selects the SWD procedure below. + +## Build the writer and download the preparation helper + +Download the [OSPI writer source](../ospi-writer.zip) and [MRAM preparation helper](../prepare_mram_programming.py), saving both in `$ALIF_WORK`. Check the helper's SHA256 with `shasum -a 256`; it must be: + +```output +3aab6ca07ae2249609bcfc342c4aa2d17560d3fe99441f6cab2fef8c5bdd5614 +``` + +```bash +unzip "$ALIF_WORK/ospi-writer.zip" -d "$ALIF_WORK" +export RECOVERY_SOURCE="$ALIF_WORK/ospi-writer" +export MRAM_HELPER="$ALIF_WORK/prepare_mram_programming.py" +west build \ + -d "$MOBILE_SAM_OUTPUT/build-ospi-writer" \ + -p always \ + -b alif_e8_dk/ae822fa0e5597xx0/rtss_hp \ + "$RECOVERY_SOURCE" -- \ + -DEXTRA_DTC_OVERLAY_FILE="$SDK_ALIF/snippets/ospi-flash/e8_ospi1.overlay" +``` + +The writer stages 64 KiB chunks in RAM, checks their CRC, writes OSPI through the board's flash driver, then reads flash back and compares bytes. The generic J-Link OSPI loader failed with the original large live model, so use this writer for model transfer. + +## Select the probe and back up application MRAM + +List attached probes, then replace the serial placeholder with your intended board's numeric J-Link serial number. This is not the `/dev/cu.usbmodem...` UART path. The executable wrapper passes that serial to every J-Link invocation, including the writer's Expect script: + +```bash +export JLINK_EXECUTABLE="${JLINK_EXECUTABLE:-$JLINK_COMMANDER}" +printf 'ShowEmuList USB\nExit\n' > "$MOBILE_SAM_OUTPUT/list-probes.jlink" +"$JLINK_EXECUTABLE" -AutoConnect 0 -NoGui 1 \ + -CommandFile "$MOBILE_SAM_OUTPUT/list-probes.jlink" +export JLINK_SERIAL=REPLACE_WITH_YOUR_PROBE_SERIAL +cat > "$MOBILE_SAM_OUTPUT/jlink-selected" <<'SH' +#!/bin/sh +case "$JLINK_SERIAL" in + ''|*[!0-9]*) echo 'Set the numeric JLINK_SERIAL' >&2; exit 2;; +esac +exec "$JLINK_EXECUTABLE" -USB "$JLINK_SERIAL" "$@" +SH +chmod +x "$MOBILE_SAM_OUTPUT/jlink-selected" +export JLINK_COMMANDER="$MOBILE_SAM_OUTPUT/jlink-selected" +``` + +Keep the wrapper and its exported variables active for the rest of this Learning Path. `JLINK_EXECUTABLE` must name the actual SEGGER binary, never the wrapper. After switching boards, select that board's serial again. The writer script hardcodes the tested A0 HP device; changing `ALIF_DEVICE` does not retarget this writer script. + +Start with HP for MRAM access. Check that the selected core can read application MRAM before preparing or programming any payload: + +```bash +export ALIF_MRAM_CORE=HP +export ALIF_MRAM_DEVICE=AE822FA0E5597LS0_M55_HP +printf 'H\nMem32 0x80000000 4\nG\nExit\n' \ + > "$MOBILE_SAM_OUTPUT/check-mram-access.jlink" +"$JLINK_COMMANDER" -Device "$ALIF_MRAM_DEVICE" -If SWD -Speed 4000 \ + -AutoConnect 1 -NoGui 1 -ExitOnError 1 \ + -CommandFile "$MOBILE_SAM_OUTPUT/check-mram-access.jlink" +``` + +If HP reports no AHB ROM and cannot connect, try the tested HE fallback by setting `ALIF_MRAM_CORE=HE` and `ALIF_MRAM_DEVICE=AE822FA0E5597LS0_M55_HE`, then rerun the access check. Continue only if that fresh HE connection reads MRAM successfully. Keep `ALIF_DEVICE=AE822FA0E5597LS0_M55_HP`: the signed image and writer boot on HP regardless of which core provides MRAM access. Stop if neither core is readable. + +Before the first download, save the A0 application-MRAM region `[0x80000000, 0x80580000)` in a fresh process: + +```bash +( +set -e +test ! -e "$MOBILE_SAM_OUTPUT/application-mram-before.bin" +cat > "$MOBILE_SAM_OUTPUT/backup-mram.jlink" <&1 | tee "$MOBILE_SAM_OUTPUT/$name-gen-toc.log" + ) || return + python "$MRAM_HELPER" prepare \ + --config "$MOBILE_SAM_OUTPUT/$build_name/zephyr/.config" \ + --firmware "$ALIF_SE_TOOLS_DIR/build/images/$name.bin" \ + --package "$ALIF_SE_TOOLS_DIR/build/AppTocPackage.bin" \ + --package-map "$ALIF_SE_TOOLS_DIR/build/app-package-map.txt" \ + --app-config "$MOBILE_SAM_OUTPUT/$name.json" \ + --programming-core "$ALIF_MRAM_CORE" \ + --output-dir "$run" || return + export MRAM_RUN="$run" +} +prepare_mram_package ospi-writer-hp build-ospi-writer +``` + +Require successful generation and helper checks. Review `$MRAM_RUN/manifest.json` and `app-package-map.txt`. The helper derives the signed package's destination and size from **APP Package Summary**, checks actual sizes and hashes, requires 16-byte alignment, and bounds both payloads and their 16 KiB programming envelopes below system MRAM at `0x80580000` without overlap. It accepts the tested single user-managed signed HP image layout; stop if your package has a different layout. + +The tested writer package began at `0x8057c210`, contained 15,856 bytes, and ended at `0x80580000`. These are generated package values, not addresses to copy into another build. The AToC header at `0x8057ff50` is inside the package and is **not** its load address. Companion images, configuration, and certificates are already included in the signed package; `AppTocPackage.bin.sign` is not a third MRAM payload. The helper checks layout and bytes; it does not authenticate vendor signatures. + +## Program, independently read back, and boot the writer + +Define the programming function below and run it for the prepared writer. Each Commander invocation is a separate process. Downloads use `noreset`; readback occurs in a new named connection to the chosen core with no preceding downloads, so it cannot reuse the earlier process's flash-download cache: + +```bash +program_prepared_mram() { + local script device + test -n "${MRAM_RUN:-}" && test -s "$MRAM_RUN/manifest.json" || return 1 + device=$(python -c 'import json,sys; print(json.load(open(sys.argv[1]))["programming_device"])' \ + "$MRAM_RUN/manifest.json") || return + for script in 01-program.jlink 02-readback-fresh-process.jlink; do + "$JLINK_COMMANDER" -Device "$device" -If SWD -Speed 4000 \ + -AutoConnect 1 -NoGui 1 -ExitOnError 1 \ + -CommandFile "$MRAM_RUN/$script" || return + done + python "$MRAM_HELPER" verify --manifest "$MRAM_RUN/manifest.json" || return + "$JLINK_COMMANDER" -AutoConnect 0 -NoGui 1 -ExitOnError 1 \ + -CommandFile "$MRAM_RUN/03-reset-probe-only.jlink" +} +program_prepared_mram +``` + +Require both fresh readback files to match their prepared size and SHA256. This checks the two payloads; it does not compare the rest of MRAM with your backup. + +The final process is **probe-only**: `R0`, 200 ms, `R1`, then 2,500 ms. It must not connect a CPU or issue generic `r`. On the original A0 board, generic `r` reset the HP subsystem and resumed the old image; the reset-pin pulse let the Security Enclave boot the newly signed package. See [SEGGER Commander commands](https://kb.segger.com/J-Link_Commander) for `R0`/`R1` and `LoadFile ... noreset`. + +Check the writer in another fresh named HP connection: + +```bash +printf 'H\nRegs\nMem32 0x20080000 10\nG\nExit\n' \ + > "$MOBILE_SAM_OUTPUT/check-writer.jlink" +"$JLINK_COMMANDER" -Device "$ALIF_DEVICE" -If SWD -Speed 4000 \ + -AutoConnect 1 -NoGui 1 -ExitOnError 1 \ + -CommandFile "$MOBILE_SAM_OUTPUT/check-writer.jlink" +``` + +Require mailbox magic `4F535049`, version `00000001`, ready status `52454144`, and zero result before transferring. The mailbox starts at `0x20080000`; its staging buffer starts at `0x20081000`. These addresses belong to this writer. A successful debugger connection alone does not establish readiness. + +## Transfer and verify the model + +Keep the live application stopped while the writer owns OSPI. The transfer script uses `JLINK_COMMANDER` and `/usr/bin/python3` for CRC32 calculation: + +```bash +set -o pipefail +expect "$RECOVERY_SOURCE/flash_model.expect" \ + "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.bin" \ + 2>&1 | tee "$MOBILE_SAM_OUTPUT/model-programming.log" +``` + +The initial erase can be silent for **six minutes or longer**: this model spans 3,252 separate 4 KiB sectors, and progress appears after erase completes. Leave the process running until it reports completion or a specific error. The host polling allowance is at least ten minutes plus J-Link overhead; a silent erase alone is not a timeout. + +The public model's successful transfer ends with: + +```output +Model programmed and verified: 13317584 bytes +``` + +Require the complete file size and no CRC, flash-readback, mailbox, or timeout errors. The script then leaves HP halted. Its chunk CRC and board-side flash readback establish model-transfer integrity. Use the normal erase for a fresh transfer; `--skip-erase` is only for recovery after a confirmed completed erase, not an interrupted transfer of unknown state. + +## Install the matching live application + +Regenerate the live application's own signed package and repeat the same independent readback and reset-pin boot sequence: + +```bash +prepare_mram_package mobilesam-live-hp build-mobilesam-live && +program_prepared_mram +cd "$EXECUTORCH_ROOT" +``` + +Require helper preparation and both readbacks to pass before accepting the install. Do not reuse the writer's package or addresses for the live firmware. This signs `build-mobilesam-live`, the default build with its dashboard disabled. Check camera capture, inference, and advancing UART frame reports in the next lesson; completed programming does not establish those runtime results. + +To capture the console, disconnect power, select **UART4 on SW4**, and reconnect PRG USB. Use 115200 baud, 8N1. Select SEUART again only when using SETOOLS ISP. When only firmware changes and the PTE is unchanged, model flash can be retained. When the PTE changes, program the complete new model and rebuild firmware with its matching metadata and size. + +## What you've accomplished + +You have preserved the boot configuration, installed the writer through SWD with fresh MRAM readbacks, programmed and verified the model, and installed its matching camera-and-UART firmware. Next, validate advancing capture and inference on the board. diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/troubleshoot.md b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/troubleshoot.md new file mode 100644 index 0000000000..a8f521427b --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/troubleshoot.md @@ -0,0 +1,79 @@ +--- +title: Diagnose flashing, memory, and camera failures +description: Diagnose model programming, tensor contracts, memory capacity, and camera routing on the Alif E8. +weight: 8 +layout: learningpathall +--- + +## Find the last stage that completed + +Start with the last confirmed checkpoint: model transfer, model load, camera initialization, capture, inference, or advancing UART frames. Identify the build first: the default application uses camera/NPU/UART, while the optional dashboard adds the display and shared-route lifecycle. + +| Symptom | Likely cause or relevant behavior | Next check | +| --- | --- | --- | +| Export stops at the mask-quality check | Calibration input and candidate selection affect FP32/quantized agreement. | Inspect the generated masks and metrics, correct the calibration input, and require the existing quality gate to pass. | +| Neither the J-Link probe nor a USB modem serial port appears | The host has not enumerated the programming/debug USB connection. | Follow the USB visibility checks below before retrying SETOOLS. | +| SETOOLS cannot open the programming port | An attached terminal can own the port. | Close the terminal before programming. | +| SETOOLS times out while the USB bridge shows application boot or frame messages | SW4 is routing the bridge to UART4 rather than the Security Enclave programming UART. | Disconnect board power, select SEUART on SW4, reconnect PRG USB, and retry the toolkit. Use UART4 afterward for the application console. | +| Generic J-Link OSPI programming fails | The large final live model needed the custom board-side writer. | Verify the writer mailbox and complete all block readbacks before flashing the final application. | +| Model transfer prints only the initial erase message for six minutes or longer | The writer erases 3,252 sectors before printing transfer progress. | Keep the process running until completion or a specific error; do not interrupt merely because progress is silent. | +| A successful MRAM download still boots the previous image | A generic HP reset can leave the Security Enclave's boot selection unchanged. | Require fresh-process payload readbacks, then run the separate probe-only `R0`/`R1` reset-pin sequence from the programming lesson. | +| Same-process MRAM readback matches after a download | Commander can return its flash-download cache. | Repeat `SaveBin` in a new process with no preceding downloads and compare both payloads with the helper. | +| Flash operations conflict with a running application | An application using memory-mapped OSPI can hold the controller. | Run the idle or recovery-writer firmware before modifying the model region. | +| Model load stalls or fails | Earlier variants exposed inappropriate model verification/startup behavior. | Check actual programmed bytes, compiled size, readable mapped address, and matching source/metadata. | +| CMake rejects CPU operators | The configure step finds `aten::` or `dim_order_ops::` entries in the PTE operator list. | Re-export from the supplied source with `--quantized-io` and inspect delegation; preserve both model outputs. | +| Firmware rejects input type, byte count, or output tensors | The model interface does not match the sample and generated metadata. | Run the export lesson's PTE and JSON checks, rebuild firmware with the matching pair, and program the complete matching PTE. | +| Scratch-capacity configuration error | The compiled PTE requires more scratch than the configured arena. | Compare the extracted requirement with the configured pool; inspect linker capacity before increasing it. | +| Planned-buffer or method allocation fails | Scratch capacity and method/planned memory are separate budgets. | Inspect the failed allocation and the method pool rather than enlarging only fast scratch. | +| Sensor identification returns `0x0000` or error 19 | Sensor power/reset/sleep and camera/display routing were not yet correct. | Reuse the pinned MT9M114 selfie overlay and explicit startup sequence. | +| MT9M114 is detected as `0x2481` but capture still fails | Detection established the sensor's presence; it did not establish streaming. | Check deferred initialization, RAW10 input format, planar RGB output, routing, and first-stream stabilization. | +| No camera frames after initialization | GPIO14.2 selects the external MIPI route. | The default build must keep the camera selected after initialization. Verify the headless configuration, pinned camera overlay, and complete startup log. | +| First frame takes several seconds | The final code waits seven seconds before starting the sensor stream. | Wait for startup to complete before diagnosing a persistent capture failure. | +| Display stays blank in the default build | The dashboard is disabled intentionally. | Require `Camera + NPU mode; on-board dashboard disabled` and advancing UART frames. Use the optional dashboard build only when investigating its known DSI limitation. | +| DSI messages appear in the default camera-and-UART run | A dashboard-enabled or older firmware may be installed. | Check the exact ELF/bin and fresh readbacks. Require `CONFIG_MOBILESAM_DISPLAY`, `CONFIG_DISPLAY`, and `CONFIG_MIPI_DSI` to be disabled in the default build. | +| Optional dashboard reports DSI ACK or packet errors | The shared camera/display link lifecycle remains unresolved. | Retain the full UART log and exact artifacts. Global frame-ACK disabling left the physical display blank in a prior experiment. The dashboard is an experimental extension, not an error-free supported output path. | +| Strict PT2E comparison fails while integer TOSA matches | Quantized-host floating-point arithmetic and integer lowering have different rounding behavior. | Retain both verdicts and exact input/output tensors. Use the validation lesson's integer execution contract; do not relabel or relax the failed original PT2E limits. | +| Selected-mask count is zero or lower than expected | Fixed prompts, quality filtering, area filtering, and duplicate suppression determine the selected subset. | Inspect retained input/masks and prompt coverage before changing model precision. UART counts alone do not establish scene accuracy. | + +The GPIO14.2 MIPI selector is separate from SW4's UART routing. The default firmware keeps the camera selected; changing SW4 cannot repair its camera path. + +## Check a missing programming USB connection + +If both the J-Link probe and the macOS `/dev/cu.usbmodem...` serial device are absent, check the host connection before retrying programming: + +- Use **PRG USB**, the USB-C connector on the underside near the board corner. MCU USB can supply power without providing this programming/debug connection. +- Check for the green power LED near the E1 device and SW4. Illumination confirms power, not USB data connectivity. +- Verify **SW3 (JLOB) is EN**. This is separate from **SW4**, which must select SEUART for SETOOLS. Compare the labels with Figure 1 of the [official Alif DK-E8 User Guide (AUGD0023)](https://alifsemi.com/download/AUGD0023). Disconnect board power before adjusting the configuration. +- Test a known USB data cable connected directly to the Mac to isolate a cable, adapter, or hub problem. + +Require the probe for the SWD programming lesson and the serial device when using SETOOLS ISP or the application console. SW4 selects the UART signal routed to the bridge; a missing USB connection requires checking the steps above rather than repeatedly changing the UART selection. + +## Retain the final camera configuration + +The code chooses the MT9M114 selfie sensor, performs its board-specific power/reset/sleep sequence, initializes the sensor late, configures RAW10 capture and planar RGB output, and waits before streaming. In the default build, dequeuing and requeueing do not change the MIPI selector. The camera receiver starts before the upstream sensor, and a completed buffer is returned only after the capture pipeline stops. + +Use the supplied sample and pinned SDK overlay together. A successful chip-ID probe confirms only the sensor connection; capture still requires the complete initialization and routing sequence. + +## Keep future ports consistent + +Treat a dependency or source refresh as a new configuration to validate. The separate public FVP example has a different model interface and newer Arm backend passes; its successful execution does not validate this board application. + +The supplied patch selects released PyTorch and TorchAO packages by default. Keep its installer and source checkout together when recreating the environment, and record the resolved versions with your validation results. Re-export the model and rebuild firmware together after changing dependencies. + +Zero or unavailable counters in fast FVP mode do not prove CPU fallback. Inspect delegation and completed execution separately. Compiler cycle estimates, timed simulator counters, and physical-device inference duration describe different measurements; retain that distinction when comparing changes. + +## What you've accomplished + +You can check model-transfer integrity, tensor contracts, camera initialization, and advancing NPU execution separately. Keep the default camera-and-UART evidence distinct from the optional dashboard's DSI errors and the failed strict PT2E comparison. + +## Extend the application + +Choose one change for your next experiment: + +- Change the prompt grid, re-export, and compare mask coverage on representative scenes +- Change input resolution and repeat the quality, scratch-capacity, linker, and target-output checks +- Measure capture, preprocessing, inference, mask selection, and UART reporting separately +- Investigate the optional dashboard's DSI lifecycle in a separate build while preserving the working camera-and-UART firmware +- Refresh dependencies or vendor revisions in a separate workspace and repeat export, build, model readback, and live execution + +Keep the resolved dependencies, model and firmware hashes, programming log, memory report, and output comparisons with each run. Record the scene, prompt grid, and timing boundary so you can compare results after a change. diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/understand-the-system.md b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/understand-the-system.md new file mode 100644 index 0000000000..eb3e261ad3 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/understand-the-system.md @@ -0,0 +1,47 @@ +--- +title: Understand the camera and NPU segmentation pipeline +description: Trace live camera frames through MobileSAM and mask selection, with UART reporting and an optional dashboard. +weight: 2 +layout: learningpathall +--- + +## Run camera inference on the E8 + +You'll deploy MobileSAM on an Alif Ensemble E8 Development Kit to capture camera frames, run segmentation, and report advancing frames and selected-mask counts through UART4. The prompts are fixed during export; the camera image changes at runtime. You'll export the model, build the application, and program model data and firmware separately. + +Zephyr and ExecuTorch run on the Cortex-M55 High-Performance (HP) core, with MobileSAM computation delegated to the Ethos-U85-256 NPU. An MT9M114 selfie camera supplies images. The default build leaves the display and DSI host disabled and keeps the external MIPI selector on the camera after initialization. A blank display is expected in this mode. + +{{% notice Important %}} +The default camera-and-UART firmware passed two independent reset-pin runs on E8 A0: 9 frames in a 35-second observation and 40 frames in a 120-second observation, with no reported errors. The optional dashboard has known DSI errors. The validation lesson separates this runtime result, integer TOSA agreement, and the failed original strict PT2E comparison; none establishes general scene accuracy. +{{% /notice %}} + +## Follow one frame through the application + +The model uses a 384 × 384 image, four independent positive prompts, three candidate masks per prompt, and int8 input and outputs. The public-image export is 13,317,584 bytes in external Octal SPI (OSPI) flash, mapped at `0xC0000000`. Its `.pte` is stored separately from firmware. + +1. **Capture and prepare the image.** The camera pipeline writes planar RGB into the input buffer. The CPU normalizes each channel using the exported pixel mean and standard deviation, then applies the exported scale and zero point to produce a `[1, 3, 384, 384]` int8 tensor. Raw camera bytes and quantized inputs have different meanings, even though both occupy one byte per value. +2. **Execute MobileSAM.** The image encoder produces embeddings, and the mask decoder combines them with prompt embeddings prepared during export. Quantization uses A8W8 globally and A16W8 in the image encoder's attention modules. The external tensor interface remains int8. +3. **Select distinct masks.** Positive prompts sit at `(96, 96)`, `(288, 96)`, `(96, 288)`, and `(288, 288)`. Each requests three candidates. The CPU chooses the highest-quality candidate per point, requires quality at least `0.8`, and rejects masks covering less than roughly 0.5% or more than 85% of the mask image. It sorts survivors by quality and removes proposals whose intersection over union with an already selected mask exceeds `0.75`. At most four masks survive; they have no semantic object categories. +4. **Report and capture again.** UART prints the advancing frame number, time around `method.execute()`, and selected-mask count. The application requeues the camera buffer without switching the MIPI route to the display. + +## Keep export and firmware inputs together + +Keep the `.pte` and exporter JSON together. CMake generates `model_metadata.h` from the JSON. Image dimensions, preprocessing, quantization parameters, prompt count, and quality threshold form the firmware contract. Prompt embeddings are constants in the PTE; editing JSON coordinates cannot move them. Re-export to change prompts or resolution, regenerate firmware metadata, and program the matching model. + +The matching source is `zephyr/samples/mobilesam-ethosu`, including `src/main.cpp`, `src/camera.cpp`, and the optional `src/display.cpp`. Use its supplied exporter throughout this Learning Path. The separate FVP example uses a 448 × 448 image, floating-point input/output, and one prompt. + +## Treat the dashboard as an optional extension + +The build lesson provides a separate dashboard configuration. It adds a 480 × 800 display with the camera image and colored masks, and alternates the shared MIPI route between camera and display. The tested dashboard advanced frames and visibly updated, but continued reporting DSI acknowledgment and packet errors. It is an experimental extension with a known unresolved link lifecycle. + +The default camera-and-UART build avoids that display path. It still captures images, executes the same PTE, selects masks, and reports progress. UART counts establish liveness; they do not show whether the masks match objects in a scene. The validation lesson separates liveness, exact integer execution, and image accuracy. + +## Interpret inference time + +The reported milliseconds cover `method.execute()` only. Capture, normalization, quantization, mask selection, UART printing, and buffer requeue are outside that interval. Measure the complete loop separately before reporting camera-to-result latency. Results from the optional dashboard must identify that configuration. + +## What you've learned and what's next + +You know how camera data reaches the NPU, why prompt coordinates are fixed, and which model and metadata files must agree. Before continuing, explain why int8 camera input still needs preprocessing and why four prompts can yield fewer than four selected masks. + +Next, prepare the matching source and tools and check the board's programming connection. diff --git a/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/validate-live-segmentation.md b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/validate-live-segmentation.md new file mode 100644 index 0000000000..3ffde16e72 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/validate-live-segmentation.md @@ -0,0 +1,123 @@ +--- +title: Validate the live camera and NPU loop +description: Check advancing UART frames, camera and NPU errors, and the limits of numerical and scene-accuracy evidence. +weight: 7 +layout: learningpathall +--- + +## Establish each stage of execution + +The default camera-and-UART application leaves its display disabled. After the verified MRAM install, capture UART4 before a probe-only reset-pin boot. The sample waits seven seconds before its first camera stream, so allow startup to complete. + +{{% notice Important %}} +The final default build passed two independent reset-pin runs on E8 A0 with no reported errors: 9 advancing frames during a 35-second observation and 40 during a 120-second observation. The optional dashboard retains known DSI acknowledgment and packet errors. The original strict PT2E comparison failed and remains recorded separately from integer TOSA agreement. +{{% /notice %}} + +Check these stages in order: + +1. OSPI is ready and the model opens at the configured address and size. +2. The method loads and exposes the expected int8 image input and outputs. +3. The MT9M114 initializes and the camera stream produces a frame. +4. `method.execute()` returns successfully and mask selection completes. +5. UART reports strictly increasing frame numbers and the buffer is requeued for the next capture. +6. The complete log contains no camera, CSI, NPU, or execution failure for the recorded run. + +A model-load message or one frame alone is insufficient. The default build prints `Camera + NPU mode; on-board dashboard disabled`; it should not initialize a DSI host or switch the MIPI route to the display. A blank screen is expected. + +## Capture UART before boot + +With board power disconnected, select **UART4 on SW4**, then reconnect PRG USB. The onboard bridge carries the HP console at 115200 baud, 8N1. Find the serial device and capture the complete output: + +```bash +python -m serial.tools.list_ports +export ALIF_CONSOLE=/dev/cu.replace-with-your-application-uart +python -u -m serial.tools.miniterm "$ALIF_CONSOLE" 115200 --raw \ + 2>&1 | tee "$MOBILE_SAM_OUTPUT/mobilesam-uart.log" +``` + +With capture running, use the programming terminal and its selected-probe wrapper to run the verified live package's `03-reset-probe-only.jlink`. That script pulses the reset pin without attaching a CPU. Keep startup and all error messages in one log. Exit capture with **Ctrl+]** after the observation period. + +Return SW4 to SEUART only for SETOOLS ISP. J-Link SWD works with SW4 left on UART4. Disconnect power before changing the switch. The source prints messages similar to: + +```output +Camera + NPU mode; on-board dashboard disabled +Camera ready: MT9M114 to planar RGB 384x384 +Frame 9: 1338 ms, 1 objects +``` + +The frame line above illustrates the format; record the values from your build. Require at least three strictly increasing frame numbers, then retain a longer uninterrupted run to check for later failures. A selected-mask count of zero is valid when no proposal passes the quality, area, and overlap rules. Counts are selected prompt regions, not semantic classes. + +## Inspect the recorded default-build result + +The verified firmware used the raw build hash `3be06219380e7d2444d0b18bef8ebaa42284f401aed2d8e3e5d720c50edc4a36` and the padded programming hash `6d782da186b607a815f7b4d3e4bed8dd0bb6bc1d6ca8c1a630fed4120472e25a`. Fresh debugger processes read back both the staged firmware and signed AToC package exactly before reset-pin boot. + +| Independent reset-pin run | UART observation | Completed frames | Reported execution time | Reported errors | +| --- | ---: | ---: | ---: | ---: | +| 1 | 35 seconds | 1–9 | 1338 ms per frame | 0 | +| 2 | 120 seconds | 1–40 | 1338 ms per frame | 0 | + +All 49 frame numbers advanced within their own boot epoch. The complete logs contained no CSI, DSI, NPU, or execution error reports. These are bounded observations on the tested A0 board, not an indefinite reliability or cold-power-cycle test. The milliseconds cover `method.execute()`; the observation windows are not measurements of complete-loop latency. + +The same default firmware was also paused after frame 23 completed inference and mask selection, before the next camera requeue. Its exact 442,368-byte int8 input was supplied to the retained post-lowering TOSA graph. All 110,592 mask codes and 12 quality codes matched the NPU outputs exactly. Input and outputs were stable across the capture stops, and the debugger-visible planned outputs agreed with the NPU scratch outputs. This is separate evidence from the uninterrupted UART observations above. + +## Inspect diagnostics only when needed + +UART is the primary liveness evidence. For developer diagnosis, the sample exposes `camera_diagnostics`: entries 19, 20, and 21 hold the frame number, inference milliseconds, and selected-mask count. Resolve its address from the exact flashed ELF; the default build need not contain GDB type information. + +```bash +( +set -e +diagnostic_symbol=$("$ZEPHYR_SDK_INSTALL_DIR/arm-zephyr-eabi/bin/arm-zephyr-eabi-nm" \ + --defined-only "$MOBILE_SAM_OUTPUT/build-mobilesam-live/zephyr/zephyr.elf" | \ + awk '$3 == "camera_diagnostics" {print $1}') +test -n "$diagnostic_symbol" +diagnostic_address=$((0x${diagnostic_symbol} + 19 * 4)) +printf 'H\nMem32 0x%x 3\nG\nExit\n' "$diagnostic_address" \ + > "$MOBILE_SAM_OUTPUT/read-camera.jlink" +) +``` + +Close other J-Link sessions, then use the selected-probe wrapper from the programming lesson: + +```bash +"$JLINK_COMMANDER" -Device "$ALIF_DEVICE" -If SWD -Speed 4000 \ + -AutoConnect 1 -NoGui 1 -ExitOnError 1 \ + -CommandFile "$MOBILE_SAM_OUTPUT/read-camera.jlink" +``` + +The script briefly halts, reads three words, and resumes. It does not reset or program the board. Interpret the words as hexadecimal frame count, milliseconds, and mask count. The address changes with the ELF, and cached CPU writes can make a debugger view stale. A long halt during capture can cause a later camera timeout. + +Full tensor capture is a separate developer check. Stop only at a verified post-inference point with capture already complete, derive addresses and sizes from that exact ELF and model schema, and preserve the input with both outputs. The default build excludes `display_result`; a breakpoint from the dashboard's display function is invalid here. Do not reuse old PCs, tensor addresses, or framebuffers. + +The reported milliseconds time `method.execute()` only. Capture, normalization, quantization, mask selection, UART printing, and buffer requeue are excluded; measure those separately before reporting complete-loop latency. + +## Separate output agreement from scene accuracy + +Check image quality against FP32 and integer execution against the retained TOSA graph. Use the same preprocessing, prompt coordinates, and int8 input for each comparison. A PT2E quantized-host reference and an integer TOSA reference can produce different results: PT2E uses floating-point operations between quantizers, while TOSA specifies integer convolution and fixed-point rescaling. The [TOSA rescale specification](https://www.mlplatform.org/tosa/tosa_spec_1_0_1.html#_rescale) defines the `SINGLE_ROUND` arithmetic used by this export. + +The public dog image, final default-build frame 23, and an earlier dashboard-frame comparison produced these results for the same public PTE: + +| Check | Measured result | Outcome | +| --- | --- | --- | +| Public dog: FP32 versus PT2E | Selected IoU 0.975632; pixel agreement 0.991916 | Original image-quality gate passed | +| Public dog: FP32 versus retained TOSA | Selected IoU 0.974156; pixel agreement 0.991428 | Same gate passed unchanged | +| Final camera-and-UART build, frame 23: NPU versus retained TOSA | All 110,592 mask codes and 12 quality codes identical; zero maximum and mean code error | Bit-exact for this input | +| Earlier dashboard frame: NPU versus PT2E | Mask error: maximum 13 codes, mean 2.08688; quality error: maximum 37 codes | Original strict protocol failed | + +Both public-image checks reused the pinned dog image, preprocessing, prompts, and exporter metric function. The unchanged gate requires selected mean IoU at least 0.90 **or** pixel agreement at least 0.96, on prompts with selected quality at least 0.90. The evaluated prompts are zero-based indices 2 and 3. These results establish image quality for that image only. + +The final camera comparison used ELF SHA256 `a378d4acd6e52d597303f90d4494e4bbcdf43e91b8d74911a4f5a4dcd8841834` and the raw/staged firmware identities recorded above. Frame 23's int8 input has SHA256 `4fffc6e9c91ae0041cb532354df491bc7e3b34ebb5189ed630662236412116c5`. The matching outputs establish integer execution agreement for this input, not all camera scenes. Target model readback matched the public PTE, and captured planned outputs matched the NPU scratch bytes after a controlled cache clean. After the debugger resumed and disconnected, a separate 20-second observation recorded seven further frames, 53–59, with no reported errors and no intervening reset. + +The earlier strict PT2E protocol remains failed: its limits were maximum/mean mask error of 2/0.25 codes and maximum quality error of 2 codes. Two prompts also missed its 0.995 pixel-agreement limit, although candidate choices and quality decisions matched. This is an earlier captured input, not a new strict PT2E test of frame 23. Keep that failed verdict with the successful final TOSA comparison. + +The first localized numerical difference was four one-code discrepancies among 1,179,648 first-convolution outputs. Independent integer convolution, fixed-point rescaling, and float64 rounding agreed with TOSA; PT2E FP32 arithmetic crossed four rounding boundaries. This explains the first difference, rather than all final output drift. Preserve each comparison’s ELF, model, metadata, and input identities. A successful final TOSA comparison does not change the failed original strict PT2E verdict. + +Advancing UART frames establish progress, not pixel accuracy. An optional dashboard overlay also does not prove pixel accuracy. To evaluate accuracy, retain representative frames and compare masks with a reference or labels using the same preprocessing and prompt coordinates. Counted masks are prompt-selected regions, not semantic object classes, and the four-point grid can miss objects between prompts. + +Use the programming chapter's per-chunk CRC and complete flash-readback verification as the model byte-integrity evidence. Model loading and completed inference are separate runtime checks; neither substitutes for verification of all programmed bytes. + +Before finishing, identify the evidence you have for each checkpoint and record your source revision, model hash, scene, prompt grid, and timing boundary with the result. + +## What you've accomplished + +You can distinguish camera/NPU progress, model-transfer integrity, integer execution agreement, and scene accuracy. Record the default-build evidence with its exact artifacts, retain the failed strict PT2E verdict, and treat the dashboard as an optional extension with unresolved DSI errors.