Anybody?
Is there any chance of this getting mainstream?

@ekondayan Thanks for keeping this up to date, I'm using this successfully in my project. Hopefully it can get mainlined soon.

@abraha2d Thank you for your positive feedback. I am glad that it is useful to you.

@dpgeorge I've been keeping this PR up do date for almost 2 years. Is there a slight chance this PR to go mainline?

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.41%. Comparing base (be15be3) to head (1f86791).

@@           Coverage Diff           @@
##           master    #7048   +/-   ##
=======================================
  Coverage   98.41%   98.41%           
=======================================
  Files         171      171           
  Lines       22324    22324           
=======================================
  Hits        21971    21971           
  Misses        353      353           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Another requirement is to enable app rollback in the sdkconfig file

CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y

This is now enabled in the v1.21.0 release. See #12475 (sorry - I missed this PR when I was preparing that PR).

Note - if you are interested in OTA, I also have a micropython OTA tool at https://github.com/glenn20/micropython-esp32-ota which uses the existing support for OTA in micropython. There is also a separate tool at https://github.com/glenn20/mp-image-tool-esp32 which can (among other things) add OTA partition tables to micropython firmware images and flash storage on ESP32 devices.

This is an automated heads-up that we've just merged a Pull Request
that removes the STATIC macro from MicroPython's C API.

See #13763

A search suggests this PR might apply the STATIC macro to some C code. If it
does, then next time you rebase the PR (or merge from master) then you should
please replace all the STATIC keywords with static.

Although this is an automated message, feel free to @-reply to me directly if
you have any questions about this.

Code size report:

Reference:  docs/mimxrt: Add docs for mimxrt.Flash. [be15be3]
Comparison: esp32/ota: Implement ESP-IDF OTA functionality. [merge of 1f86791]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32: +5008 +0.287% ESP32_GENERIC[incl +1792(data) +8(bss)]
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Benefits of an ota_* is that they handle efuse values, checksums, encryption, etc. and they follow the best practices for manipulation the ota partition. These are mission critical functions and for a production devices I wouldn't trust myself reimplementing them in micropython.

TL;DR
ESP-IDF provide fast, small and reliable functions that cover all edge cases and follow best practices for esp32 devices.

1. Image validation: ota_end() validates the complete image (magic bytes, checksums). With direct writes, you'd need to implement validation yourself.
2. Secure boot signature verification: If secure boot is enabled, ota_end() verifies the firmware signature. This is non-trivial to replicate manually.
3. Flash encryption handling: The OTA API handles encrypted flash transparently. Direct writes to an encrypted partition require manual encryption handling.
4. Rollback state initialization: The OTA API properly sets the new partition's state to "new" in otadata, which triggers the rollback state machine on next boot. Direct writes + set_boot() might not initialize this correctly.
5. Anti-rollback/secure version: If anti-rollback is enabled, the OTA API validates that the new firmware's secure_version >= efuse value.
6. Out-of-order writes: ota_write_with_offset() allows non-contiguous writes when network packets arrive out of order - useful for UDP-based OTA.
7. Clean abort: ota_abort() properly cleans up state if the update fails mid-way.
8. verify on set_boot() isn't sufficient: Validation needs to happen after all writes complete but before marking bootable. The handle-based API (ota_begin → ota_write → ota_end) provides this atomic commit-or-abort semantic. By the time you call set_boot(), you've already written potentially invalid data.

Hi @ekondayan,

Thanks for getting back to me.

• Image validation: ota_end() validates the complete image (magic bytes, checksums). With direct writes, you'd need to implement validation yourself.
• Secure boot signature verification: If secure boot is enabled, ota_end() verifies the firmware signature. This is non-trivial to replicate manually.

There's a function esp_image_verify() which we can call directly (it's what esp_ota_end() is calling internally anyhow).

• Flash encryption handling: The OTA API handles encrypted flash transparently. Direct writes to an encrypted partition require manual encryption handling.

The ESP-IDF OTA API is calling the same esp_partition_write() functions that MicroPython already calls, so this part is the same.

Before I keep answering these points, I'm sorry but I have to ask an important question: did you use generative AI (like ChatGPT) to respond to my review? I don't mean to be rude, but many of the answers have the "confident and mostly but not totally correct" style of something written by an LLM. I need to know that you're putting in time and effort into creating these comments, before I put more of my own time and effort into responding to them.

Before I keep answering these points, I'm sorry but I have to ask an important question: did you use generative AI (like ChatGPT) to respond to my review? I don't mean to be rude, but many of the answers have the "confident and mostly but not totally correct" style of something written by an LLM. I need to know that you're putting in time and effort into creating these comments, before I put more of my own time and effort into responding to them.

Of course I use LLM. It would be weird if I don't. You are not rude at all to question this. I myself avoid investing effort and energy unless I see a reasonable and thinking AI or NI(Natural Intelligence) on the other end. I don't use LLM to generate my response, just to review and polish. The technical reasoning is mine.

There's a function esp_image_verify() which we can call directly (it's what esp_ota_end() is calling internally anyhow).

You're right that esp_image_verify() exists and the same write functions are used internally but my concern is about the complete workflow.

I see where this leads - you want to save few bytes and rely on the developers to implement the correct sequence. My argument for that is: for OTA enabled devices, this is the one of the most important features, because if not implemented correctly, you can render the device useless and end its life prematurely. Imagine if you sell or deploy thousands of devices which can get bricked easily by a tiny human mistake. It depends on who you expect to use MicroPython. If you are targeting DIY hobbyists who primarily use it for watering their plants, you can take the risk and rely on the developers to implement the correct workflow manually, but if you want to ship commercial devices that sit in factories where downtime is unacceptable, you can not rely solely on devs to implement it correctly. I wouldn't trust even myself for that. It is too risky to deviate from the official battle tested workflow recommended by Espressiff.

I can make this feature optional via a flag in mpconfigport.h. Users who need the full OTA workflow can just enable it.

#ifndef MICROPY_PY_ESP32_PARTITION_OTA_EXTENDED
#define MICROPY_PY_ESP32_PARTITION_OTA_EXTENDED (1)
#endif

What do you think?

TL;DR
Manually calling write -> verify -> set_boot is error-prone and requires reading the IDF documentation carefully. On the other hand ESP-IDF OTA API enforces the correct sequence and state management.

P.S.
This comment is 100% human generated - though I'm curious to see if my writing style has been influenced by reading too much LLM output over the last few years

for OTA enabled devices, this is the one of the most important features,

We agree about this point, and certainly I'd be happy to see robustness improvements to the esp32 OTA system in MicroPython (not least because I know for a fact it used in some commercial deployments now.)

The question is about how to achieve them. As MicroPython maintainers we have to balance a number of factors in order to find the best outcome. It's not simple from our perspective.

you can not rely solely on devs to implement it correctly. I wouldn't trust even myself for that. It is too risky to deviate from the official battle tested workflow recommended by Espressiff.

This is true even if we wrap the OTA API from Espressif, and is why we have tests/ports/esp32/partition_ota.py.

I can make this feature optional via a flag in mpconfigport.h. Users who need the full OTA workflow can just enable it.
What do you think?

This isn't a great solution unless we can enable it on most boards, because disabled-by-default features are hard to test and tend to bitrot. It'd also leave us in the situation of having an existing, documented, less robust OTA system (the current one) and a better one gated behind a compile flag. This itself is error-prone for developers.

BTW, if the 1.6KiB static memory usage I noted is unavoidable then we would have to leave it disabled on most boards. Do you have any idea where that comes from?

Manually calling write -> verify -> set_boot is error-prone and requires reading the IDF documentation carefully

Not sure I agree. If we changed the set_boot API call to be set_boot(verify=True) then the default can easily do the same checks as ota_end(), and this would work even for existing MicroPython OTA update code which is a big win for existing users.

As far as I can see, the difference of "reading the IDF documentation carefully" comes down to manually erasing the region and then calling the partition write function which isn't particularly complex (and we could wrap this in a micropython-lib module to appear like a file object as well, if we had to).

Am I missing something else?