Repository navigation
Log each connect's setup and every failed exchange and disconnect; don't mark unsent commands unacknowledged - #209
Conversation
…ation When the pod doesn't answer within the read timeout, readMessagePacket cancelled the connection straight from the PeripheralManager queue. The idle disconnect was moved off that queue in a715d9a because it raced CoreBluetooth's teardown and wedged the next connect; this call was missed. It also left commandConnectInFlight set, so the heartbeat probe didn't re-arm after the disconnect. Route it through disconnectOnDemand. Log the teardown and each peripheral configuration (duration, or the error naming the step that timed out) as device events. A failed configuration is ignored and the session's command then fails with notReady; the nil disconnects in #199 are this teardown, and these lines show where the time after a connect goes.
OverviewI'm not competent to review this and I'm not sure how to test this solution. I asked Claude instead. Here's the response: Claude ReviewThe fix is sound, but it changes keep-alive behavior in a way I'd like confirmed before approving. What checks out
Main concern: keep-alive no longer reconnects after this teardown (non-eager pods)Clearing if shouldHoldConnection && commandConnectInFlight {
// "a deliberate background/idle disconnect clears commandConnectInFlight first, so it won't reconnect"
connectForCommand(peripheral)Before this PR, an unresponsive-pod teardown in the foreground, or in the background with Pod Keep Alive, reconnected right away. Now it takes the idle path (fault scan + probe), and the link stays down until the next command connects on demand. Eager-connect pods aren't affected, because they reconnect earlier in Until now Suggestion: either clear the flag only when Minor
TestingThere are no new tests, and the PR notes it hasn't been run against a pod. Before merging, please test on a pod with Pod Keep Alive enabled and a forced read timeout, to confirm the reconnect behavior above. Recommendation: request changes, or at least get an answer on the keep-alive reconnect. The rest looks good. |
…s held Routing the teardown through disconnectOnDemand cleared commandConnectInFlight unconditionally, so with the app open (or a background Pod Keep Alive mode) the drop no longer took the keep-alive reconnect path and the link stayed down until the next command. Clear the flag only when the connection isn't held: a held link reconnects as before, and an idle one re-arms the probe. Also drop the comments saying a cancel from the PeripheralManager queue raced CoreBluetooth and wedged the next connect. There's no evidence for it, and the stuck-in-.connecting symptom is the InPlay link-layer wedge the eager watchdog handles. The calls still go through managerQueue because the connect-on-demand state lives there.
A cancel issued off managerQueue is suspected, without evidence, of wedging the next connect. Record each cancel's time and whether it ran on managerQueue, and when a connect sticks in .connecting (the eager watchdog fires, or an on-demand connect times out) log a [stuckConnect] device event with how long before it the last cancel ran and from which queue. The forceful reconnect in PeripheralManager.reconnect still cancels from the PeripheralManager queue, so field reports can compare the two.
|
Thanks Marion. The keep-alive concern was right, and the pod simulator confirmed it. I added a pod simulator control that keeps the link up but leaves a message unanswered, then ran the teardown with the app open and the connection held:
1f60442 takes your first suggestion. The teardown now goes through I also took out the claim that cancelling from the PeripheralManager queue "raced CoreBluetooth's teardown and wedged the next connect". It came from an older comment, I couldn't find any evidence for it, and PeripheralManager calls CoreBluetooth from its own queue on every command anyway. 109e834 adds |
Claude reviewRe-review (049152c → 109e834)The keep-alive concern is fixed. The new cancel-provenance logging has an ordering bug on the eager-watchdog path, and the PR description is now out of date. Resolved
Bug: the watchdog
|
…esponsive teardown in didDisconnect logStuckConnect read lastCancel asynchronously, so on the eager watchdog (already on managerQueue) the read ran after the watchdog's own cancel and always reported it. It now reads synchronously when called on managerQueue. disconnectUnresponsive no longer decides whether to keep commandConnectInFlight when it cancels. It marks the teardown, and didDisconnect clears the flag if the link isn't held at that moment. This closes the window where the app goes to the background between the cancel and didDisconnect while the failing command session is still counted: the flag stayed set and the probe didn't re-arm until the next command. Tag the unresponsive-pod and configuration device events.
|
Thanks Marion. 325c254 makes these changes:
I reran the pod-simulator test on 325c254. The held case behaves as before: the teardown completes with |
Re-review: PR #209 (4 commits, head
|
The [stuckConnect] lines only existed to test whether a cancel from the PeripheralManager queue wedges the next connect. Nothing in the field reports points that way, so drop the bookkeeping and the comments that still asserted it. The managerQueue helpers stay because the connect-on-demand state lives there.
…er written sendMessagePacket marks a message as possibly received before writing its last packet. On Omnipod 5 a short message is a single packet, so a missing characteristic after a failed discovery threw notReady before anything was written, and the command was still recorded as unacknowledged. notReady is only thrown before a write is issued, so treat it as unsent: the session gets a certain failure and no pending command to resolve.
…d disconnect The device log couldn't say why the first exchange after a connect failed in #199. Setup and login errors were swallowed, and a disconnect didn't say who closed the link or what was in progress. - [connect] configured in Ns, with each step's time (services, characteristics, write size, hello, notifications, login), or configuration failed after Ns, with the error and the failing step. Login errors now reach this line instead of being logged only to os_log. Setup skipped on a fresh connection is logged too. - [send] and [read] on a failed exchange: the error, the stage (CTS, packet i/n, data seq N, ...), whether the message was sent, what was done about it, the peripheral and central state, whether the characteristics are present, and the time since connect. - [disconnect] on every disconnect: the error domain and code, or for our own cancels the reason (idle, unresponsive pod, background, watchdog, ...), with the time since connect, the phase (connect, setup, session name, idle) and the setup outcome. - The retry after a dropped link says how setup went. This replaces the [connectOnDemand] no response from pod line.
Test✅ Test phone behaved nominally while locked - not a wedging configuration Configuration:LoopWorkspace main v3.14.11, commit 5efba88
Test DetailsThis test was performed on 2026-11-04 After building, Pod Expired is left over from last test with this phone
Activity in rPi terminal Return to the office and check rPi terminal Phone is working in the background. However, this is not a "wedging" configuration. Saved dataHarvest the Xcode debug log and Loop Report. Plan for next test (next comment)
|
Begin Data CollectionConfiguration:LoopWorkspace main v3.14.11, commit 5efba88
Test DetailsThis test was performed on 2026-11-04 Phone has the rPi DASH pod still connected from the previous test
Check rPi terminal and it had quit. Last command was at: Restart the rPi
Xcode fired a heartbeat at 10:26:41
Loop and rPi seem to be behaving nominally Post this comment now. Modification to mock up wedging configurationMake this modification to OmnipodKit OmnipodKit % git diff |
Data CollectionI configured to collect data faking a wedging scenario (hardware not wedging but code told that it was)
14:33 Unlock phone
|
|
We have various users on Trio reporting issues that could be linked to the issue this PR aims to solve. |
|
I did another review with Claude. This time I made sure all information in the conversation was provided.
Excerpt from Claude Review: Recommendation
Section 1: Review summary: OmnipodKit PR #209 (7 commits, head
|
marionbarker
left a comment
There was a problem hiding this comment.
Giving an approval to remove block for merging.
Based on last Claude review - this does fix some issues.
- Warning - I am echoing Claude but do not necessarily understand the review, passing it on for experts to decide
Not sure if we want to include changes based on Claude review posted in this comment.
- A pod-initiated drop during a command still leaves commandConnectInFlight set, so the heartbeat probe never re-arms.
- Logging inconsistency reported
A pod that hangs up during a command, with the connection not held, left commandConnectInFlight set: readMessagePacket saw the link already down and skipped disconnectUnresponsive, so handleDisconnect took the idle path with the flag still set and didn't arm the probe, and the idle disconnect that would clear it needs a connected peripheral. With a network CGM in the background nothing else issues a command, so the heartbeat stopped. handleDisconnect's decision now lives in DisconnectPolicy, which has no CoreBluetooth dependency and is unit tested. A drop outside a command session clears the flag. A drop during one leaves it to the session, which may retry its connect, and when the last session ends with an unheld link down, the flag is cleared and the probe armed. That replaces unresponsiveTeardownPending, and keeps a cancelled probe's disconnect, which can arrive after a command connect has started, from clearing the flag under that connect. Two device-log fixes: a connect that timed out was logged as a setup failure at services, and a link iOS restored at launch was reported as not connected.
|
@marionbarker, thanks. The open issue is real, and 0d361d6 fixes it along the lines you suggested, with the session gate. What changed
Why the gate matters. My first version cleared the flag on any drop when the link wasn't held, and the simulator run broke it. When a command preempts the probe, the cancelled probe's Tests
Simulator. LoopKit/pod#33 adds I ran next-dev plus this PR in the iOS Simulator, with the app in the background, Pod Keep Alive set to "when open" and a mock CGM. Here is a hang-up during a status read: The third line is the late probe disconnect, landing during the command's connect. Hang-ups after answering were handled too, and the probes and heartbeats kept going through every run. What the simulator didn't show. I couldn't get the stall itself on c48aa64 end to end. After a status read, OmnipodKit issues a heartbeat if the last one was more than 2 minutes ago. Loop's cycle then opens another session, and that session's idle disconnect clears the flag. To stall, the pod has to drop the link after the cycle's last session and before the 4 s idle disconnect. The unit tests cover that case. Running this also turned up an ImpossiBLE bug in the same ordering: the late disconnect orphaned the next connect. It's fixed in mickeyl/ImpossiBLE#10. You'll need it to run this in the simulator. |
Claude reviewPR #209 re-review — new commit
|
Relates to #199.
In the #199 traces, the first exchange after a fresh connect fails, and the pod hangs up 6–8 s after the connect (CBError 7, a remote close). There are two shapes:
notReadyabout 2 s after connecting. Service or characteristic discovery timed out (2 s), so the pod's characteristics were never found and the session sent its command anyway. Nothing was written, but the command was logged as "Unacknowledged".incorrectResponse5 s after the send, thenPod disconnected … nil. The pod accepted the command at the Bluetooth level and never replied. Thenildisconnect is OmnipodKit's own teardown after the read timeout.The device log couldn't say which setup step failed, whether login succeeded, or who closed the link. This PR makes that visible and fixes the two code problems the traces show.
Changes
[connect] configured in Ns: services …, characteristics …, write size …, hello …, notifications …, login …, or[connect] configuration failed after Ns: <error>; <steps so far>. Login errors used to be swallowed incompleteConfiguration. They now reach this line, so a connect whose login failed isn't reported as configured. A fresh connection that skips setup is logged too.[send] <error> at <stage>, sent / not sent / may have been received; peripheral …, central …, characteristics present/missing; <time since connect>, during <phase>, setup <outcome>.[read] <error> at <RTS | data seq N | SUCCESS write | …> after Ns, disconnecting / link already down / sent NACK; <same context>. This replaces the earlier[connectOnDemand] no response from podline.[disconnect]on every disconnect. It gives the error domain and code, or for our own cancels the reason (idle, unresponsive pod, background, eager watchdog, probe wake, …), with the time since connect, the phase and the setup outcome.link dropped after connecting) now says how setup went.sendMessagePacketmarks a message as possibly received before writing its last packet, and an Omnipod 5 command is a single packet.notReadyis only thrown before a write is issued, so it now counts as unsent. The session gets a certain failure and no pending command to resolve.BluetoothManager.disconnectUnresponsive, onmanagerQueue, where the connect-on-demand state lives. When the link is held, the resultingdidDisconnectreconnects, as before.handleDisconnect's decision is nowDisconnectPolicy, which doesn't depend on CoreBluetooth and is unit tested.commandConnectInFlightand arms the probe.readMessagePacketfinds the link already down. Before, the flag stayed set and the background heartbeat stopped.didDisconnectfrom clearing the flag while a command connect is under way.[connect] not connected, setup not run, not as a services failure. A link iOS restored at launch is reported as "after restore", not "not connected".[stuckConnect]logging to test it. Neither the Omnipod 5: Repeated "Peripheral Not Ready" errors after successful Bluetooth connection cause failed loops #199 traces nor the simulator tests point that way, so that logging is removed again, along with the remaining comments. The stuck-in-.connectingsymptom is the InPlay link-layer wedge, which the eager watchdog handles.On next-dev, a session is already refused when login hasn't completed on the connection (#201). That gate isn't on
dev. There, the session still runs after a failed setup, and the unsent fix above applies.Testing
failNextConnects: discovery unanswered, pod hangs up at 7 s with CBError 7):ignoreNextMessages), app open:DisconnectPolicyTests: 13 tests. With c48aa64's policy put back, 4 fail. OmniTests: 202 run, 0 failures.hangUpOnNextMessages), app in the background with Pod Keep Alive "when open":[send]failure from a failed setup: on next-dev, Refuse a pod session without a handshake; decode the LTK and BLE handle separately #201 refuses the session first, and I didn't builddev