Network Working Group M. Cheng Internet-Draft Y. Bao Intended status: Standards Track BII Group Holdings Ltd. Expires: June 24, 2017 December 21, 2016 SDN Switch Benchmark Test Cases Summary draft-cheng-sdn-switchtest-00 Abstract This document summarized the test cases for SDN switches, and introduced the test methodologies for each case of them. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on June 24, 2017. Copyright Notice Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Cheng & Bao Expires June 24, 2017 [Page 1] Internet-Draft SDN Switch Benchmark Test Cases December 2016 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.1. Case1 . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.2. Case2 . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.3. Case3 . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.4. Case4 . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.5. Case5 . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.6. Case6 . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.7. Case7 . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.8. Case8 . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.9. Case9 . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.10. Case10 . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.11. Case11 . . . . . . . . . . . . . . . . . . . . . . . . . 8 3. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 9 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 9 1. Introduction This document listed 11 test cases for SDN switch. 2. Test Cases 2.1. Case1 Test Case Name:Flow entry installing time Test Purpose:Measure how much time does the switch under test test to valid a flow entry. Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Add a flow entry to table 0 with a match field of TCP five-tuple and an action OFPAT_OUTPUT to an output port 4. Wait enough time(e.g. 5 seconds) for switch to valid the flow entry 5. Send a matched TCP packet to the data plane, record this time as Tt 6. Receive the packet at the output port, record this time as Tr Cheng & Bao Expires June 24, 2017 [Page 2] Internet-Draft SDN Switch Benchmark Test Cases December 2016 7. The result time Tresult = Tr - Tt 8. Iterate this test by a pre-configured number(less than the known maximum flow entry number) of flow entries installed already or the maximum number of flow entries 2.2. Case2 Test Case Name:Flow table look up time Test Purpose:Measure how much time does the switch under test take to look up flow table for a matched packet Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Add a flow entry to table 0 with a match field of TCP five-tuple and an action OFPAT_OUTPUT to an output port 4. Wait enough time(e.g. 5 seconds) for switch to valid the flow entry 5. Send a matched TCP packet to the data plane, record this time as Tt 6. Receive the packet at the output port, record this time as Tr 7. The result time Tresult = Tr - Tt 8. Iterate this test by a pre-configured number(less than the known maximum flow entry number) of flow entries installed already or the maximum number of flow entries 2.3. Case3 Test Case Name:Packet in message performance of table-miss Test Purpose:Measures the time sending packet_in message after a table-miss flow entry is triggered Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of the switch under test Cheng & Bao Expires June 24, 2017 [Page 3] Internet-Draft SDN Switch Benchmark Test Cases December 2016 3. Add a table-miss flow entry matching all wildcards, and a default action OFPAT_OUTPUT to port OFPP_CONTROLLER 4. Wait enough time(e.g. 5 seconds) for switch to valid the flow entry 5. Send a packet to data plane, record this time as Tt 6. Receive Packet_in messages and record this time as Tr 7. The result time Tresult = Tr - Tt 8. Iterate this test with different datapath packet throughput 2.4. Case4 Test Case Name:Packet in message performance of none table-miss Test Purpose:Measures the time sending packet_in message after a normal flow entry is triggered Test Methodology: 1. Setup connection between switch under test and test tool 2. remove all flow entries of switch 3. add a flow entry to table 0 with matching in_port and an action OFPAT_OUTPUT to port OFPP_CONTROLLER 4. Wait enough time(e.g. 5 seconds) for switch to valid the flow entry 5. send a packet to the matched data plane port, record this time as Tt 6. receive Packet_in messages and record this time as Tr 7. The result time Tresult = Tr - Tt 8. Iterate this test with different datapath packet throughput 2.5. Case5 Test Case Name:Flow entry deleting time Test Purpose:Measure how much time does the switch under test take to delete a flow entry. Cheng & Bao Expires June 24, 2017 [Page 4] Internet-Draft SDN Switch Benchmark Test Cases December 2016 Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Add a flow entry to table 0 with a match field of TCP five-tuple and an action OFPAT_OUTPUT to an output port 4. Keep sending TCP packets to dataplane ingress port 5. Wait enough time(e.g. 3 seconds) for switch until packets are forwarded to the output port 6. Send a flow delete message, delete all flow entries, record this time as Tt 7. wait until no packets received at the output port, record the time of the last one received packet as Tr 8. The result time Tresult = Tr - Tt 9. Iterate this test for 100 flow entries and 1000 flow entries installed already or the maximum number(if it is less than 1000) of flow entries 2.6. Case6 Test Case Name:Flow entry modifing time Test Purpose:Measure how much time does the switch under test take to modify a flow entry. Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Add a flow entry to table 0 with a match field of TCP five-tuple and an action OFPAT_OUTPUT to an output port 4. Keep sending TCP packets to dataplane ingress port 5. Wait enough time(e.g. 3 seconds) for switch until packets are forwarded to the output port Cheng & Bao Expires June 24, 2017 [Page 5] Internet-Draft SDN Switch Benchmark Test Cases December 2016 6. Send a flow modify message, output the packets to another port, record this time as Tt 7. wait until packets received at the ohter output port, record the time of the first received packet as Tr 8. The result time Tresult = Tr - Tt 9. Iterate this test for 100 flow entries and 1000 flow entries installed already or the maximum number(if it is less than 1000) of flow entries 2.7. Case7 Test Case Name:Table capacity Test Purpose:Measure the maximum number of flow entries of a certain table of a switch Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Send an empty body ofp_multipart_request with type OFPMP_TABLE_FEATURES, to get the maximum flow entries of a specified table 4. Keep sending flow_add messages with different priorities, actions can be an output action to an output port 5. Wait until ofp_error_msg is received, with type OFPET_FLOW_MOD_FAILED, and error code OFPFMFC_TABLE_FULL 6. Record the number of flow entries sent before the ofp_error_msg is triggered as the test result 2.8. Case8 Test Case Name:Packet out message throughput/latency time Test Purpose:Measure the Packet_out message processing throughput and latency of the switch under test Test Methodology: 1. Setup connection between switch under test and test tool Cheng & Bao Expires June 24, 2017 [Page 6] Internet-Draft SDN Switch Benchmark Test Cases December 2016 2. Send packet_out messages with a certain length of packet in its data field and action OFPAT_OUTPUT to an output port of the switch under test, record the time as Tt 3. Receive the packet contains in the data field, record the time as Tr 4. The result Tresult = Tr - Tt 5. Iterate this test for different Packet_out throughput and data length 2.9. Case9 Test Case Name:Flow entry multipart_reply message test Test Purpose:Measure flow statistics update time of a switch Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Add a flow entry to table 0 with a match field of TCP five-tuple and an action OFPAT_OUTPUT to an output port 4. Keep sending an OFPT_MULTIPART_REQUEST message with type of OFPMP_FLOW to request statistics of the flow added in step 3 5. Send a packet matches the flow added in step 3 to switch from dataplane, record the time as Tt 6. Wait enough time(e.g. 3 seconds) for switch until flow statistics are updated, record the time as Tr 7. The result Tresult = Tr - Tt 8. Iterate this test for different flows 2.10. Case10 Test Case Name:Port multipart_reply message test Test Purpose:Measure port statistics update time of a switch Test Methodology: Cheng & Bao Expires June 24, 2017 [Page 7] Internet-Draft SDN Switch Benchmark Test Cases December 2016 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Send an OFPT_MULTIPART_REQUEST message with type of OFPMP_PORT_STATS to request statistics of one port, record the rx_packets and rx_bytes 4. Keep sending the OFPT_MULTIPART_REQUEST message of step 3. 5. Send a packet to switch from dataplane, record the time as Tt 6. Wait enough time(e.g. 3 seconds) for switch until port statistics are updated, record the time as Tr 7. The result Tresult = Tr - Tt 8. Iterate this test for sending different amount and kind of packets to switch 2.11. Case11 Test Case Name:Idle time out timer performance test Test Purpose:Test the accuracy of idle timeout of flow entries Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Install a flow entry with idle timeout 5s and set the OFPFF_SEND_FLOW_REM flag 4. Send a matching packet to ingress port within idle timeout and record the time as Ts 5. Wait until the switch send an OFP_FLOW_REMOVED message to the test tool, record the time as Te 6. The result time, Tresult = Te - Ts 7. Compare Tresult with the hard timeout(5s), the error percentage equals (Tresult - 5s)/5s * 100% 8. Iterate this test with different number of installed flow entries Cheng & Bao Expires June 24, 2017 [Page 8] Internet-Draft SDN Switch Benchmark Test Cases December 2016 3. Acknowledgements Funding for the RFC Editor function is currently provided by BII Group. Authors' Addresses Mike Cheng BII Group Holdings Ltd. Beijing P. R. China Email: mikecheng@biigroup.com Yaming Bao BII Group Holdings Ltd. Beijing P. R. China Email: ymbao@biigroup.cn Cheng & Bao Expires June 24, 2017 [Page 9]