Last year, in my IPFIX 25 Years Anniversary post, I wrote that if we were to design IPFIX from scratch today, we would change a few things. This brand new draft, draft-claise-opsawg-ipfix-ordered-ie, is exactly one of those “few things” — now written up as an actual specification, with Paul Aitken, Holger Keller, and Yao Liu.
It fixes a small gap that has been hiding in plain sight in RFC 7011 since 2013. Small gap, but one that gets more and more relevant these days — the moment you start exporting SRv6 or tunnelling information, where the same Information Element repeats once per encapsulation layer, it bites. The draft makes that case explicitly, pointing at SRv6 packets with two or three nested IPv6 headers as a prime example.
And it is not only SRv6. VXLAN, GRE, MAC-in-MAC, plain IP-in-IP — header stacking is everywhere in modern networks, and in each of these the same Information Element repeats per layer, so without an ordering guarantee the Collector cannot tell the transport layer from the tenant layer. This is the use case that the complementary draft-liu-opsawg-ipfix-muti-layer from Yao Liu and Taoran Zhou brought back into focus — and which pushed us to tackle the broader, underlying problem. It also sits right next to the SRv6 flow-monitoring work we described in RFC 9487.
The problem: “in order” was never an IPFIX promise
IPFIX allows the same Information Element to appear several times in a Template Record. The classic example is an IPv4-in-IPv4 packet where you export the sourceIPv4Address IPFIX Information Element (IPFIX IE) twice: once for the outer header, once for the inner one. Here is exactly what Section 8 of RFC 7011 says about that:
If an Information Element is required more than once in a Template, the different occurrences of this Information Element SHOULD follow the logical order of their treatments by the Metering Process.
SHOULD. Not MUST. And here is the part that really bites: even when an Exporter does the right thing and puts the fields in order, there is no field on the wire that tells the Collector “this ordering is guaranteed.” Worse, Sections 4.2 and 4.3 of the very same RFC 7011 explicitly state that the order of Information Elements in a Template is not guaranteed.
So put yourself in the Collector’s seat. You receive two sourceIPv6Address IPFIX IE values from an SRv6 packet. Which one is the outer header? Which is the inner? You simply cannot know for sure. You’re guessing — and guessing is not a great foundation for network analytics.
Remember, IPFIX is fundamentally an export protocol. RFC 7011 specifies the Exporting Process and the Collecting Process in detail, but it deliberately says almost nothing normative about the Metering Process. A real end-to-end ordering guarantee therefore needs all three to play along: the Metering Process records observations in order, the Exporting Process preserves and signals that order, and the Collecting Process reads the n-th occurrence as the n-th observation. Today, the middle link — the signal — is simply missing.
The proof is in the registry: Information Elements proliferation
If you want proof that this gap costs us, just open the IANA IPFIX registry and look at the MPLS label stack. Instead of one Information Element used ten times, we defined ten numbered ones:
mplsTopLabelStackSection (IPFIX IE 70), mplsLabelStackSection2 (IPFIX IE 71), all the way down to mplsLabelStackSection10 (IPFIX IE 79).
Ten identifiers for what is conceptually a single repeated field. And it has three nasty side effects:
- it burns ten IE identifiers where one would do;
- it artificially caps the exportable label stack at ten entries (not that we can picture a network with a stack of 11 labels);
- it doesn’t generalize, so every new stacked structure needs its own fresh batch of numbered IEs.
And it doesn’t stop at the label stack. There’s a whole cluster of top-of-stack-only IEs (mplsTopLabelType (IPFIX IE 46), mplsTopLabelIPv4Address (IPFIX IE 47), mplsTopLabelPrefixLength (IPFIX IE 91), mplsTopLabelIPv6Address (IPFIX IE 140), …) that exist only because we had no way to say “the attribute of the label at position N.” The same story repeats with IEEE 802.1ad Q-in-Q, where dot1qVlanId (IPFIX IE 243) and dot1qCustomerVlanId (IPFIX IE 245) are nothing more than positional workarounds for “outer VLAN” and “inner VLAN.”
We’ve been paying the registry tax for over a decade because the IPFIX Exporter couldn’t say to the Collector: “these fields are observed, encoded, and sent in order, and I promise it”.
The solution: two new Set IDs
We looked at several candidate solutions in the draft (and yes, we rejected most of them in writing — that’s part of the fun of an IETF document):
- Only change SHOULD to MUST in RFC 7011? Rejected. It constrains the Exporter but still gives the Collector no signal, and it would contradict the “order is not guaranteed” statements elsewhere in 7011.
- Reuse RFC 6313 structured data with the “ordered” semantic? Rejected. The RFC 6313 “ordered” semantic is an export ordering guarantee only — it never claims the order reflects the Metering Process observation order. Close, but not the guarantee we need.
- Bump IPFIX to version 11? Rejected. A Collector that sees an unknown version discards the entire message — every existing Collector would reject everything until upgraded. Disproportionate to the problem.
The recommended answer is actually simple: two new Set IDs.
| Set ID | Name |
|---|---|
| 2 | Template Set (existing) |
| 3 | Options Template Set (existing) |
| 4 | Ordered Template Set (new) |
| 5 | Ordered Options Template Set (new) |
Set IDs 4 and 5 use exactly the same wire format as 2 and 3. The Set ID is the sole distinguishing element: it tells the Collector that the ordering of repeated Information Elements in the corresponding Data Records is guaranteed to reflect the Metering Process observation order. The Collector then interprets the n-th occurrence as the n-th observation — outermost to innermost for encapsulation, top to bottom for label stacks.
Why we like this approach:
- It’s an explicit, per-Set signal. No in-band marker to parse, no out-of-band correlation table to maintain.
- It’s additive and backwards compatible. Set IDs 2 and 3 are untouched. RFC 7011 already reserved Set IDs 4–255 for exactly this kind of extension, and every prior IPFIX extension (including RFC 6313) shipped without a version bump.
- It reuses existing code. If you already parse Set ID 2 and 3, you support 4 and 5 by reusing the same parser and adding the ordering semantic.
- It generalizes. Any stacked or layered structure benefits, with no new position-specific IEs.
What this ordered export specification doesn’t solve
The ten positional MPLS IEs had a quiet virtue: because each one names its own stack position (mplsLabelStackSection2, mplsLabelStackSection3, and so on), a Collector that receives position 2 and position 4 but nothing in between knows immediately that label 3 is missing. The Ordered Set mechanism specified in the draft does not give you that. It guarantees only that the n-th occurrence is the n-th observation, contiguous from the top — so if an Exporter drops a label in the middle of the stack, the Collector just sees a shorter run and has no way to tell a gap happened. The draft is upfront about this: the guarantee covers exported occurrences only, and for sparse or partial export the explicit layer-marking of draft-liu-opsawg-ipfix-muti-layer is the better fit — complementary to, not replaced by, ordered export.
What’s next
This is a fresh -00 individual draft, but we believe it is quite complete already. Our intention is to bring it to OPSAWG — that target is our plan, not yet a working-group decision — and, as always at the IETF, the real test is rough consensus and running code. We’d love to hear from implementers: Exporters, Collectors, and mediator folks. If you maintain a Collector, the cheapest possible review question is: what does your code do today when it meets a Set ID it doesn’t recognise?
After 25 years, IPFIX is still picking up small, sharp improvements like this one. Tells you something about the protocol — and about the community around it. See you at the IETF 126.