Internet Engineering Task Force (IETF) P. Aitken Internet-Draft Brocade Communications Systems, Inc. Intended status: Standards Track March 1, 2017 Expires: September 2, 2017 Utilizing Pre-defined Templates with IPFIX draft-aitken-ipfix-pre-defined-templates-00 Abstract This document specifies a way to pre-define well-known IPFIX Templates which can be pre-shared with Collectors, thus avoiding the need for Exporters to send those Templates to Collectors. This saves export bandwidth and reduces Collector complexity. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on September 2, 2017. Copyright Notice Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Aitken Expires September 2, 2017 [Page 1] Internet-Draft IPFIX pre-defined Templates March 2017 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. Set Header Format . . . . . . . . . . . . . . . . . . . . . . 3 5. Template Record Format . . . . . . . . . . . . . . . . . . . 5 6. Data Record Format . . . . . . . . . . . . . . . . . . . . . 6 7. Options Template Record Format . . . . . . . . . . . . . . . 7 8. Options Data Record Format . . . . . . . . . . . . . . . . . 9 9. The Exporter's Side . . . . . . . . . . . . . . . . . . . . . 10 10. The Collector's Side . . . . . . . . . . . . . . . . . . . . 10 11. Template Management . . . . . . . . . . . . . . . . . . . . . 11 11.1. Template Withdrawal . . . . . . . . . . . . . . . . . . 11 12. Security Considerations . . . . . . . . . . . . . . . . . . . 11 13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 11 14.1. Normative References . . . . . . . . . . . . . . . . . . 11 14.2. Informative References . . . . . . . . . . . . . . . . . 12 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 12 1. Introduction IPFIX Exporters send Templates containing an ordered sequence of pairs which specify the data to be communicated from the IPFIX Exporter to an IPFIX Collector. Templates are useful when the IPFIX Exporter is configurable in the field, such that the data it will export cannot be known beforehand. However, many devices are deployed with fixed configurations, particularly IoT devices, sensors, and smart meters. Nevertheless, the IPFIX Protocol [RFC7011] requires such devices to export a Template defining the data to be Exported - which is wasteful since the data format is fixed and well-known ahead of time, and each device exports exactly the same data. Additional export bandwidth is consumed by the Template export, and the IPFIX Collector will receive the same Template from multiple devices. The bandwidth overhead is particularly inefficient in the case of short-lived export streams, where the Exporter creates a new export session containing a Template and a few Data Records. In the worst case, every Data Record is preceded by the corresponding Template. This inefficiency can easily be avoided if the Template is well-known before the device is even deployed in the field. This document defines a method of pre-sharing the Template with the Collector ahead of time, together with enhancements to the IPFIX Aitken Expires September 2, 2017 [Page 2] Internet-Draft IPFIX pre-defined Templates March 2017 protocol to allow data-only export, i.e. export which contains only IPFIX Data Records. This saves both export bandwidth and storage at the Collector, since only a single Template need be stored for many devices. This is an appropriate method to use when a device exports a fixed IPFIX Template, or selects Templates from a fixed set of IPFIX Templates; when the Templates are well known beforehand; when a large number of devices share the same IPFIX Template(s); and when a single Data Set is exported in each IPFIX Message. This is not an appropriate method to use when the Templates that a device will export are not known in advance; if the Templates to be exported are not shared by many other devices; or if multiple Data Sets are exported per IPFIX Message. In those cases, use regular IPFIX [RFC7011]. 2. Terminology IPFIX-specific terminology (Information Element, Template, Template Record, Options Template Record, Template Set, Collector, Exporter, Data Record, etc.) used in this document is defined in Section 2 of [RFC7011]. As in [RFC7011], these IPFIX-specific terms have the first letter of a word capitalized. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 3. Methodology Templates are not exported directly from the Exporting Device, but are provided to the Collector ahead of time by out-of-band means. Templates may be provided in a number of ways and in a number of formats. e.g. they may be provided with the Exporting Device, or downloaded from the manufacturer's website. They may be in a pcap or XML file, or in [RFC5655] format. Alternatively, the Collector may be provided with a URL or set of URLs from which Templates may be downloaded on-demand. URLs may be provided per Enterprise. 4. Set Header Format Existing IPFIX Sets [RFC7011] contain a 16-bit Set ID. When exporting Data Records, this contains a Template ID valued between 256 and 65535. Values below 256 are reserved. This range is not Aitken Expires September 2, 2017 [Page 3] Internet-Draft IPFIX pre-defined Templates March 2017 sufficiently distinct to allow multiple entities to uniquely pre- define their own Templates, and a central registry would be required to control the allocation of IDs. However sufficient distinction can be achieved by combining the Template ID with a Private Enterprise Number [IANA-PEN]. This allows each enterprise to define more than 65,000 Templates of their own without reference to a third-party registry. The Private Enterprise Number could be included in the IPFIX Message Header or in the IPFIX Set Header. Including it in the IPFIX Message Header requires a new Version number, effectively creating a new protocol. Whereas retaining the existing IPFIX Version and Message Header, but including the Private Enterprise Number in the Set Header incurs the same small overhead (i.e., 4 octets) per Data Set while retaining interoperability with existing IPFIX Collectors. This method would be less optimal if multiple Data Sets were exported per Message. However this method targets optimisation of single Data Set exports. Pre-defined export Sets contain the regular IPFIX Set ID and Length per [RFC7011], together with a 32-bit Private Enterprise Number [IANA-PEN] as shown in Figure 1 below. Compare this with Figure I in [RFC7011]. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Private Enterprise Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 1: Pre-defined Set Header Format Where: Set ID Is identical to the [RFC7011] definition. Length Is identical to the [RFC7011] definition. Private Enterprise Number Aitken Expires September 2, 2017 [Page 4] Internet-Draft IPFIX pre-defined Templates March 2017 A Private Enterprise Number [IANA-PEN] identifying the Enterprise which defined the template with the given Set ID. 5. Template Record Format The pre-defined IPFIX Template format is identical to the regular IPFIX Template format [RFC7011], except that the Set Header contains a Private Enterprise Number [IANA-PEN] as described in Section 4 above. The pre-defined Record format is shown in Figure 2 below. A new IPFIX Set ID TBD1 is used to distinguish pre-defined Templates from regular IPFIX Templates per [RFC7011]. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = TBD1 | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Private Enterprise Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID | Field Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |E| Information Element ident. | Field Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 2: Pre-defined IPFIX Template Where: Set ID = TBD1 Is identical to the [RFC7011] definition. The new IPFIX Set ID value of TBD1 indicates that the following data defines a pre-defined Template. Length Is identical to the [RFC7011] definition. Private Enterprise Number A Private Enterprise Number [IANA-PEN] identifying the Enterprise which defined the template. Aitken Expires September 2, 2017 [Page 5] Internet-Draft IPFIX pre-defined Templates March 2017 The combination of Private Enterprise Number and Template ID uniquely identifies the Template. Template ID Is identical to the [RFC7011] definition. Field Count Is identical to the [RFC7011] definition. Information Element Identifier Is identical to the [RFC7011] definition. Field Length Is identical to the [RFC7011] definition. Note that the Template may contain enterprise-specific Information Elements per [RFC7011]. It is NOT REQUIRED that these Information Elements use the same Private Enterprise Number as used in the Set Header. 6. Data Record Format Data Records follow the IPFIX format specified in [RFC7011], except that they also contain the Private Enterprise Number [IANA-PEN] as described in Section 4 above. The Data Record format is shown in Figure 3 below. Compare this with Figure Q in [RFC7011]. Notice that since the Private Enterprise Number is part of the Set Header, it appears only once nomatter how many Data Records are exported. Aitken Expires September 2, 2017 [Page 6] Internet-Draft IPFIX pre-defined Templates March 2017 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = Template ID | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Private Enterprise Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 1 - Field Value 1 | Record 1 - Field Value 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 1 - Field Value 3 | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 2 - Field Value 1 | Record 2 - Field Value 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 2 - Field Value 3 | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 3 - Field Value 1 | Record 3 - Field Value 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 3 - Field Value 3 | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Padding (optional) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 3: Data Record for a pre-defined IPFIX Template 7. Options Template Record Format The pre-defined IPFIX Options Template format is identical to the regular IPFIX Options Template format [RFC7011], except that the Set Header contains an Private Enterprise Number [IANA-PEN] as described in Section 4 above. The pre-defined Options Record format is shown in Figure 4 below. Compare this with Figure O in [RFC7011]. A new IPFIX Set ID TBD2 is used to distinguish pre-defined Options Templates from regular IPFIX Options Templates per [RFC7011]. Aitken Expires September 2, 2017 [Page 7] Internet-Draft IPFIX pre-defined Templates March 2017 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = TBD2 | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Private Enterprise Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID | Field Count = N + M | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field Count = N |0| Scope 1 Infor. Element id. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length |0| Scope 2 Infor. Element id. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 2 Field Length | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... |1| Scope N Infor. Element id. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope N Field Length | Scope N Enterprise Number ... +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ... Scope N Enterprise Number |1| Option 1 Infor. Element id. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Option 1 Field Length | Option 1 Enterprise Number ... +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ... Option 1 Enterprise Number | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... |0| Option M Infor. Element id. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Option M Field Length | Padding (optional) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4: Pre-defined IPFIX Options Template Where: Set ID = TBD2 Is identical to the [RFC7011] definition. The new IPFIX Set ID value of TBD2 indicates that the following data defines a pre-defined Options Template. Length Is identical to the [RFC7011] definition. Private Enterprise Number Aitken Expires September 2, 2017 [Page 8] Internet-Draft IPFIX pre-defined Templates March 2017 A Private Enterprise Number [IANA-PEN] identifying the Enterprise which defined the template. The combination of Private Enterprise Number and Template ID uniquely identifies the Options Template. Template ID Is identical to the [RFC7011] definition. Field Count Is identical to the [RFC7011] definition. Scope Field Count Is identical to the [RFC7011] definition. Information Element Identifier Is identical to the [RFC7011] definition. Field Length Is identical to the [RFC7011] definition. Note that the Options Template may contain enterprise-specific Information Elements per [RFC7011]. It is NOT REQUIRED that these Information Elements use the same Private Enterprise Number as used in the Set Header. 8. Options Data Record Format Options Data Records follow the IPFIX format specified in [RFC7011], except that they also contain the Private Enterprise Number [IANA-PEN] as described in Section 4 above. The Data Record format is shown in Figure 5 below. Notice that since the Private Enterprise Number is part of the Set Header, it appears only once nomatter how many Data Records are exported. Aitken Expires September 2, 2017 [Page 9] Internet-Draft IPFIX pre-defined Templates March 2017 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = Options Template ID | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Private Enterprise Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 1 - Scope Field Value 1| Record 1 - Scope Field Value 2| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Record 1 - Scope Field Value N| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 1 - Field Value 1 | Record 1 - Field Value 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Record 1 - Field Value M | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 2 - Scope Field Value 1| Record 2 - Scope Field Value 2| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Record 2 - Scope Field Value N| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 2 - Field Value 1 | Record 2 - Field Value 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Record 2 - Field Value M | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Padding (optional) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 5: Data Record for a pre-defined IPFIX Options Template 9. The Exporter's Side Exporters MUST NOT send pre-defined Templates to Collectors. Exporters MAY export Data Records using a mixture of pre-defined Templates and regular IPFIX Templates. Note that regular IPFIX Template IDs and pre-defined Template IDs share a single Template ID number space. It is RECOMMENDED that Exporter manufacturers reserve a block of Template IDs for pre- defined Templates, perhaps preferring lower Template IDs for regular IPFIX Templates and higher Template IDs for pre-defined Templates. 10. The Collector's Side Collectors MUST NOT expect pre-defined Templates to be sent from Exporters. Aitken Expires September 2, 2017 [Page 10] Internet-Draft IPFIX pre-defined Templates March 2017 If a Collector receives a pre-defined Template from an Exporter, it MUST ignore that Template and MAY raise an error. If the pre-defined Template is already known to the Collector and does not match the Template received from the Exporter, then the Collector MUST shut down the connection, raise an error, and MUST NOT attempt to decode the associated Data Records. If a Collector does not have the required pre-defined Template when the corresponding Data Records are received, it MAY store those Data Records for future decoding. The Collector MAY raise a warning or an error as required. A Collector MUST NOT assume that a Template is pre-defined because it is missing from the export stream. 11. Template Management Once a pre-defined Template ID has been published, the format of that Template MUST NOT be changed since this would invalidate the export from all devices currently using that Template. Although a pre-defined Template MAY be obsoleted, its Template ID can never be reused. 11.1. Template Withdrawal Pre-defined Templates cannot be withdrawn. Exporters MUST NOT send an IPFIX Template Withdrawal message [RFC7011] for a pre-defined Template. A Collector which receives an IPFIX Template Withdrawal Message for a pre-defined Template MUST ignore that Message and MAY raise an error. 12. Security Considerations For this extension to the IPFIX protocol, the same security considerations as for the IPFIX protocol apply [RFC7011]. 13. IANA Considerations IANA MUST define Set IDs TBD1 and TBD2 in the [IPFIX-Sets] registry. 14. References 14.1. Normative References [IANA-PEN] IANA, "Private Enterprise Numbers", . Aitken Expires September 2, 2017 [Page 11] Internet-Draft IPFIX pre-defined Templates March 2017 [IPFIX-Sets] IANA, "IPFIX Set IDs registry", . [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ RFC2119, March 1997, . [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information", STD 77, RFC 7011, DOI 10.17487/RFC7011, September 2013, . 14.2. Informative References [RFC5655] Trammell, B., Boschi, E., Mark, L., Zseby, T., and A. Wagner, "Specification of the IP Flow Information Export (IPFIX) File Format", RFC 5655, DOI 10.17487/RFC5655, October 2009, . Author's Address Paul Aitken Brocade Communications Systems, Inc. 19a Canning Street, Level 3 Edinburgh, Scotland EH3 8EG United Kingdom Phone: +44 203 005 0731 Email: paitken@brocade.com Aitken Expires September 2, 2017 [Page 12]