A11/T8015 APCIe: tunables and decoded bring-up sequence from a D21 (toward Wi-Fi) #24

I've been trying to get Wi-Fi up on an iPhone 8 Plus (iPhone10,2, D21AP) running the hoolock branch at 6831bc70, and along the way I collected a fair amount of T8015 APCIe data that's probably more useful to you than it is sitting on my disk. Posting it here rather than letting it rot.

Short version: T8015 looks close enough to T8010 that pauli1go's pcie-apple-h9p should be able to absorb it as a match-data variant rather than needing its own driver. The port-side register block appears to be identical. What differs is the RC register layout, the common/PHY init, and the per-port enable handshake.

Caveat up front: none of this has been tested on hardware and I haven't written any driver code. The tunables and DT properties below are read off a running device, so I'm confident in those. The register sequences are static analysis of the iOS 16.7.16 kext, so treat them as a decent hypothesis, not as fact.

Reading the live ADT from a booted Linux

This is probably the most immediately reusable bit, and I wish I'd realised it sooner. m1n1 already publishes the ADT to Linux as an MTD phram device, so no special boot is needed:

$ cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00004000 00004000 "m1n1_stage2.log"
mtd1: 00034000 00004000 "adt"

$ cat /dev/mtd1 > adt.bin
$ PYTHONPATH=m1n1/proxyclient python3 -m m1n1.adt adt.bin > adt.txt

That gets you the live tree, i.e. everything iBoot fills in at runtime. That matters here because the IPSW DeviceTree template contains none of the APCIe tunables — they only exist in the live tree. I spent a while confused about that.

The tunable properties serialise as 24-byte records of {u32 offset, u32 size, u64 mask, u64 value}, all with size == 4.

Tunables (D21, iOS 16.7.16)

/arm-io/apcie apcie-phy-tunables, against the PHY region:

{ 0x0000, 0x00003000, 0x00002000 },
{ 0x0814, 0x00000001, 0x00000001 },
{ 0x08dc, 0x00000001, 0x00000001 },
{ 0x08ec, 0x00000001, 0x00000001 },
{ 0x08fc, 0x0000000f, 0x0000000f },
{ 0x090c, 0x00000007, 0x00000000 },
{ 0x091c, 0x0000001f, 0x00000006 },
{ 0x092c, 0x000000ff, 0x00000000 },
{ 0x0934, 0x00000007, 0x00000003 },
{ 0x0814, 0x00000001, 0x00000001 },
{ 0x08dc, 0x00000001, 0x00000000 },
{ 0x08ec, 0x00000001, 0x00000000 },
{ 0x08fc, 0x0000000f, 0x00000000 },
{ 0x090c, 0x00000007, 0x00000000 },
{ 0x091c, 0x0000001f, 0x00000000 },
{ 0x092c, 0x000000ff, 0x00000000 },
{ 0x0934, 0x00000007, 0x00000003 },

Note the repeated offsets — the second pass mostly clears what the first pass set. I've reproduced them in order rather than deduplicating, since I assume the ordering is load-bearing.

/arm-io/apcie apcie-common-tunables, against the RC region. Offsets 0x4000 * (port + 1) are the per-port RC blocks, so this one table covers the common block plus all four ports:

{ 0x00008, 0x7f7f7f7f, 0x00000000 },
{ 0x0000c, 0x00073f3f, 0x00043f00 },
{ 0x00010, 0x00000700, 0x00000000 },
{ 0x00018, 0x00ffffff, 0x000c0800 },
{ 0x0001c, 0x00001fff, 0x0000092c },
{ 0x00028, 0x000000ff, 0x00000009 },
{ 0x04000, 0x31100010, 0x21000000 },
{ 0x04008, 0x00000707, 0x00000000 },
{ 0x0400c, 0x00073f3f, 0x00043f00 },
{ 0x04014, 0x00000111, 0x00000010 },
{ 0x04120, 0x0333003f, 0x0111000f },
{ 0x04130, 0x000000ff, 0x0000000f },
{ 0x04138, 0x000001ff, 0x000000f2 },

...then 0x08000, 0x0c000 and 0x10000 repeat the 0x04000 group verbatim for ports 1-3.

Per-port apcie-config-tunables (identical on pci-bridge2 and pci-bridge3), against the port register block:

{ 0x0090, 0x00ffffff, 0x0003a980 },
{ 0x0130, 0x0000000f, 0x00000007 },
{ 0x0134, 0x00000001, 0x00000001 },
{ 0x0138, 0x00007f7f, 0x00000000 },
{ 0x013c, 0x00000003, 0x00000002 },
{ 0x0140, 0x8073ffff, 0x00703d09 },

Per-port pcie-rc-tunables, against the root port's config space:

{ 0x0098, 0x0000000f, 0x00000000 },
{ 0x0164, 0x00f8ff1f, 0x0000001f },
{ 0x0890, 0x00000201, 0x00000200 },
{ 0x08e0, 0x00000005, 0x00000005 },
{ 0x0890, 0x00000201, 0x00000200 },

These are close to the T8010 values in h9p_rc_common_tunables / h9p_config_tunables / h9p_port_tunables but not identical (e.g. 0x0018 is 0x000c0800 here vs 0x000c0960, 0x0090 is 24-bit here vs 8-bit), so they do need their own tables.

Region layout

The ADT apcie reg list, in order, with the mapping I derived from the kext (the index getters are port * 2 + 1 and port * 2 + 2, which pins it down):

So ECAM and the port bases are unchanged from T8010. The RC block grew from 0x8000 to 0x50000, and PHY / PHY-IP moved out of 0x600008000 / 0x60a000000.

RC offsets, T8010 → T8015:

New on T8015, with no T8010 equivalent I could find: a 0x4128 bit4 / 0x4004 bit0 handshake, and 0x4000 bit20 (muxed aux clock gating).

Other DT-relevant ADT properties: port IRQs are AIC 340/343/346/349, msi-vector-offset 352 with #msi-vectors 32, msi-address 0xbffff000 (same doorbell as T8010), #ports 4, lane-cfg 0 (all ports x1), perst-to-refclk-delay-us 60, t-refclk-to-perst 100. pci-bridge2 is baseband at max link speed 1, pci-bridge3 is Wi-Fi at max link speed 2. I haven't tried to translate the ranges property into DT form — the raw values are two entries, {flags 0x43000000, bus 0x620000000, parent 0x620000000, size 0x20000000} and {flags 0x02000000, bus 0x40000000, parent 0x640000000, size 0x40000000}.

Decoded sequences

From AppleT8015PCIe in the iOS 16.7.16 kernelcache. Addresses are unslid VAs in that specific build and will differ elsewhere; I'm including them only so the work is checkable. The class exposes RC accessors at vtable +0x740 (read) and +0x748 (write), and the helpers below are all (port, bool enable):

where P = port << 14.

Common init:

1. RC[0x04] |= bit4
2. apply apcie-phy-tunables to the PHY region
3. RC[0x34] = 1
4. poll RC[0x24] bit0, 10 µs steps
5. PHY-IP[0x74] |= 0x6
6. delay 100 µs
7. PHY-IP[0x134] |= 0x10
8. apply apcie-common-tunables to the RC region
9. RC[0x30] bit0 = 0

Note there's no equivalent of T8010's PHY portmask wait (phy + 0x000c) and no per-lane PHY-IP programming like apple_h9p_program_phy_port() — on T8015 that's all folded into the tunable table plus those two PHY-IP writes. That's the biggest structural difference I found.

enablePortHardware:

1. +0x7e0(1)
2. +0x7f0(1) — waits RC[0x24] bit4
3. delay 100 µs
4. +0x7f8(appclk-auto-dis) — from the ADT bool of that name
5. +0x7d8(0)
6. poll RC[0x24] bit0, 1 µs steps
7. +0x7e8(1)
8. if muxed-auxclk-auto-dis: +0x800(1)
9. delay
10. +0x7e0(0)
11. wait for pipe clock: poll port[0x208] bit2 in 5 ms steps until it reads set twice consecutively, 250 ms timeout. This is the one that logs Timed out waiting for core_clk_gated.
12. +0x7d0(1), conditional on a port flag I haven't identified
13. apply apcie-config-tunables to the port block
14. _initializeRootComplex (below)
15. set link rate: RC[0x40020 + port * 0x4000] = 3

_initializeRootComplex applies pcie-rc-tunables to the root port config space and then touches a set of registers that are recognisably Synopsys DesignWare port logic — 0x710 PORT_LINK_CTRL, 0x80c GEN2_CTRL, 0x8bc MISC_CONTROL_1 (toggling DBI_RO_WR_EN around the write), 0x154, 0x890, 0x8e0. For gen3 ports it additionally writes 0x154 = 0x44444444 (EQ presets), 0x890 |= 0x400, 0x8e0 |= 1, and sets bit17 of 0x80c. So the T8015 RC is a DesignWare core, which I assume is also true on T8010 and may explain some of the magic offsets there.

disablePortHardware, for completeness: +0x7e0(1), delay perst-to-refclk-delay-us, +0x7d8(1), +0x7f0(0), +0x7e8(0).

The port side matches T8010

This is the part that makes me think a match-data variant is enough. Everything after enablePortHardware lives in the shared AppleEmbeddedPCIE base class, and the order is exactly what apple_h9p_setup_port() already does:

1. enablePortHardware()
2. port[0x104] = IRQ mask, port[0x100] = 0x9fd000 to clear status
3. port[0x140] bit31 — H9P_PORT_ENABLE_APPLE
4. port[0x124] = (log2(msi_vectors) << 4) | 1, i.e. 0x31 for 8 vectors — H9P_PORT_PWRCTL_INIT; port[0x128] = MSI base — H9P_PORT_MSIVECBASE
5. wait for the endpoint to assert CLKREQ#
6. deassert PERST#
7. port[0x80] bit0 to enable LTSSM, then wait for link up

The only difference I spotted is the pre-link IRQ status clear value: 0x009fd000 on T8015 vs H9P_PORT_IRQSTAT_PRE_LINK = 0x00ffd000 on T8010.

DART

dart-apcie3 (Wi-Fi) is at 0x604008000 size 0x4000, compatible dart,t8015, AIC IRQ 350, 4K pages, sids = 1, stt-idx-width = 10, manual-availability = 1. dart-apcie2 (baseband) sits alongside it. The DMA window works out to base 0x80000000 size 0x3ffff000, which is exactly the apple,dma-range that t8010.dtsi uses for pcie0_dart2.

t8015.dtsi currently has no DART nodes at all. AppleT8015DART is built from the same AppleHxDART source tree as the A7-A10 DARTs, which makes me suspect apple,s5l8960x-dart just works, but I haven't verified the register offsets against apple_dart_hw_s5l8960x yet.

One thing worth flagging from Orchard's NOTIFY.md: hoolock at 6831bc70 reportedly no longer reads apple,dma-range and expects the window in the five-cell specifier instead. If that's still the case, new DART nodes will want the newer form.

Wi-Fi endpoint

For whoever gets to this stage. From the live ADT's wlan node:

• BCM4357 rev B1. Firmware lives in the IPSW at System/Library/DriverExtensions/com.apple.DriverKit-AppleBCMWLAN.dext/Firmware/C-4357__s-B1/.
• module-instance = pegasuse, so pegasuse.trx, pegasuse.clmb, pegasuse.txcb, plus NVRAM P-pegasuse_M-GINS_V-{u,m}__m-{5.1,5.3,6.1}.txt. There's also a pegasusk set and a pegasuse-JP. Orchard's "Select Apple NVRAM by OTP" patch looks directly relevant since there's no antenna SKU property here.
• function-reg_on = pKW4('gP0e', 0) — REG_ON is an SMC GPIO key, not an AP GPIO. macsmc-gpio currently probes with Failed to locate of_node on this device, so that needs a DT node before the power sequence can work.
• Port 3 GPIOs: PERST 0xa7, CLKREQ 0xa3, device_wake 0x28. Host wake is interrupt 16 on the AOP pinctrl.
• There's a wifi-calibration-msf blob in the live ADT that presumably needs handing to brcmfmac the way m1n1 does brcm,cal-blob on Macs.
• The node's compatible is wlan-pcie-uart,bcm4357 — Bluetooth shares the module over UART.
• brcmfmac has no 4357 entries at all today: no BRCMF_FW_ENTRY, no BRCMF_FW_CLM_DEF, no PCIe device ID. I don't know the device ID yet since I've never had the link train.

One hazard worth knowing about

On T8015 the NVMe node takes power-domains = <&ps_ans2>, <&ps_pcie> with power-domain-names = "ans", "apcie0". Storage isn't behind APCIe here, but it shares ps_pcie. So a PCIe driver that power-cycles or resets that domain will take the root filesystem down with it. Anyone testing this should do it from a RAM-only rootfs, or at least with the real root mounted read-only.

Relatedly, iPhOmarchy reports that enabling apcie port 3 plus the DARTs with no driver present killed the console at around t+62.8 s, so there's something there that doesn't take kindly to being half-initialised.

What I can offer

I have a D21 that's a dedicated test device — no data on it, already wiped once, happy to reflash. It's tethered (checkm8 via palera1n) so reboots cost me a DFU entry but nothing else, and I have a serial console plus a USB-NCM link to it. If a t8015 variant of pcie-apple-h9p or a DART node lands somewhere I can fetch, I'll build it and report back with logs.

Happy to dig further into the kext if specific questions come up — I have a working headless Ghidra setup against it. I haven't published the kernelcache or any disassembly, and I'm not going to.