How do you verify that channels behave the same after migrating off Mirth 4.5.2?

[jgere9]

I'm spending a few weeks researching how teams actually verify HL7 v2 interface migrations - Mirth Connect 4.5.2 to OIE (and to BridgeLink or the commercial 4.7) - and I'd rather ask people who have done it than guess from release notes.

The question is narrow: after you move channels to the new engine, how do you convince yourself (and whoever signs off) that they behave the same?

Specifically, I'd love to hear:

1. Do you run old and new side by side for a period, or cut over and watch?
2. If you compare output, what exactly do you compare - the encoded message, specific segments/fields, ACKs? And what do you do about fields that legitimately differ (timestamps, control IDs)?
3. What is not covered by comparing messages - routing decisions, error and retry paths, database writes, external calls - and how do you handle those?
4. Roughly how long does the verification step take relative to the migration itself?
5. Has anything ever slipped through and shown up in production? What was it?

Why I'm asking: "the channel imported fine" and "the channel behaves the same" are different claims, and between Rhino/E4X details, code template libraries and connector-level differences there seem to be plenty of places for the second one to quietly fail. I can't find a shared practice for demonstrating it, which either means everyone has their own, or that it mostly isn't done.

I'm not selling anything and I'm not looking for work. If there's interest I'll post a summary back into this thread once I've collected the answers here and from a handful of conversations elsewhere.

[firstmind-biodream]

Disclosure first: I build Perfuse, an open-source engine that imports Mirth channels, so verifying "behaves the same" is something we've had to make concrete. This is the practice we settled on. Most of it applies whichever engine you move to.

1. Side by side, then cut over one channel at a time. First an offline pass: export a few thousand inbound/outbound pairs per channel from the old engine's message store and replay the inbound messages through the new engine. After that, run the new engine in shadow on live traffic, receiving everything and delivering nowhere, for a period that includes a weekend and a month end. Those are when the odd message types and batch jobs turn up. Cut over one channel, keep the old one ready, and only then move the next.

2. What to compare. Compare the encoded outbound message byte for byte, paired by MSH-10 (pairing in order breaks the first time either engine drops or retries one). Then split the result into four counts instead of one pass rate:

• identical;
• cosmetic only (trailing empty fields, final segment terminator);
• a field value differs;
• the new engine could not process the message at all.

Group the "differs" findings by field path, and for each one note whether it's systematic (the same old→new value pair every time, e.g. PID-8 Male→M, which is one mapping decision) or whether the content genuinely varies. Fields that legitimately differ, such as MSH-7 and MSH-10 when the engine regenerates them, go on an explicit ignore list that is written into the sign-off, not stripped silently. Compare the ACKs separately: ACK code, MSA-2 and the timing of the ACK (on receipt or after delivery), not just the fact that one was sent.

3. What message comparison does not cover. Write these down per channel, because none of them shows up in an output diff:

• Filtered-vs-errored outcomes: a message the old engine dropped on purpose has to be filtered in the new one too, not failed.
• Retry and queue behaviour when the destination is down, including ordering after recovery.
• Side effects: database writes, HTTP calls, files written. Diff the destination tables or files rather than the messages.
• Anything a script does through Java classes or code templates. Those are the places an import can succeed while the behaviour differs. List every external call a script makes and test each one directly.
• Alerts. A channel that used to page someone on a NAK should still page them.

4. Time. In our experience the shadow period dominates, because it's calendar time (a month end), not effort. The offline pairs pass takes days per busy channel, mostly spent deciding about the systematic findings.

5. What slips through. Exact counts in the verdict matter: "49,993 of 50,000" makes people look at the seven, while "over 99%" doesn't. I don't have a first-hand Mirth production incident to share, so I'll leave that question to people who have one. For what it's worth, the things our own comparisons kept catching were value mappings (codes vs. display text), empty vs. absent fields, and messages one side refused to parse at all.

Perfuse's version of steps 1 and 2 is perfuse compare old-out/ new-out/ -ignore MSH-7,MSH-10, with the method written up in its manual. The method doesn't depend on that tool, though, and I'd be interested in your summary when it's done.