SDN Switch Benchmark Test Cases Summary

Test Case Name:Flow entry installing time Test Purpose:Measure how much time does the switch under test test to valid a flow entry. Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Add a flow entry to table 0 with a match field of TCP five-tuple and an action OFPAT_OUTPUT to an output port 4. Wait enough time(e.g. 5 seconds) for switch to valid the flow entry 5. Send a matched TCP packet to the data plane, record this time as Tt 6. Receive the packet at the output port, record this time as Tr 7. The result time Tresult = Tr - Tt 8. Iterate this test by a pre-configured number(less than the known maximum flow entry number) of flow entries installed already or the maximum number of flow entries

Test Case Name:Flow table look up time Test Purpose:Measure how much time does the switch under test take to look up flow table for a matched packet Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Add a flow entry to table 0 with a match field of TCP five-tuple and an action OFPAT_OUTPUT to an output port 4. Wait enough time(e.g. 5 seconds) for switch to valid the flow entry 5. Send a matched TCP packet to the data plane, record this time as Tt 6. Receive the packet at the output port, record this time as Tr 7. The result time Tresult = Tr - Tt 8. Iterate this test by a pre-configured number(less than the known maximum flow entry number) of flow entries installed already or the maximum number of flow entries

Test Case Name:Packet in message performance of table-miss Test Purpose:Measures the time sending packet_in message after a table-miss flow entry is triggered Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of the switch under test 3. Add a table-miss flow entry matching all wildcards, and a default action OFPAT_OUTPUT to port OFPP_CONTROLLER 4. Wait enough time(e.g. 5 seconds) for switch to valid the flow entry 5. Send a packet to data plane, record this time as Tt 6. Receive Packet_in messages and record this time as Tr 7. The result time Tresult = Tr - Tt 8. Iterate this test with different datapath packet throughput

Test Case Name:Packet in message performance of none table-miss Test Purpose:Measures the time sending packet_in message after a normal flow entry is triggered Test Methodology: 1. Setup connection between switch under test and test tool 2. remove all flow entries of switch 3. add a flow entry to table 0 with matching in_port and an action OFPAT_OUTPUT to port OFPP_CONTROLLER 4. Wait enough time(e.g. 5 seconds) for switch to valid the flow entry 5. send a packet to the matched data plane port, record this time as Tt 6. receive Packet_in messages and record this time as Tr 7. The result time Tresult = Tr - Tt 8. Iterate this test with different datapath packet throughput

Test Case Name:Flow entry deleting time Test Purpose:Measure how much time does the switch under test take to delete a flow entry. Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Add a flow entry to table 0 with a match field of TCP five-tuple and an action OFPAT_OUTPUT to an output port 4. Keep sending TCP packets to dataplane ingress port 5. Wait enough time(e.g. 3 seconds) for switch until packets are forwarded to the output port 6. Send a flow delete message, delete all flow entries, record this time as Tt 7. wait until no packets received at the output port, record the time of the last one received packet as Tr 8. The result time Tresult = Tr - Tt 9. Iterate this test for 100 flow entries and 1000 flow entries installed already or the maximum number(if it is less than 1000) of flow entries

Test Case Name:Flow entry modifing time Test Purpose:Measure how much time does the switch under test take to modify a flow entry. Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Add a flow entry to table 0 with a match field of TCP five-tuple and an action OFPAT_OUTPUT to an output port 4. Keep sending TCP packets to dataplane ingress port 5. Wait enough time(e.g. 3 seconds) for switch until packets are forwarded to the output port 6. Send a flow modify message, output the packets to another port, record this time as Tt 7. wait until packets received at the ohter output port, record the time of the first received packet as Tr 8. The result time Tresult = Tr - Tt 9. Iterate this test for 100 flow entries and 1000 flow entries installed already or the maximum number(if it is less than 1000) of flow entries

Test Case Name:Table capacity Test Purpose:Measure the maximum number of flow entries of a certain table of a switch Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Send an empty body ofp_multipart_request with type OFPMP_TABLE_FEATURES, to get the maximum flow entries of a specified table 4. Keep sending flow_add messages with different priorities, actions can be an output action to an output port 5. Wait until ofp_error_msg is received, with type OFPET_FLOW_MOD_FAILED, and error code OFPFMFC_TABLE_FULL 6. Record the number of flow entries sent before the ofp_error_msg is triggered as the test result

Test Case Name:Packet out message throughput/latency time Test Purpose:Measure the Packet_out message processing throughput and latency of the switch under test Test Methodology: 1. Setup connection between switch under test and test tool 2. Send packet_out messages with a certain length of packet in its data field and action OFPAT_OUTPUT to an output port of the switch under test, record the time as Tt 3. Receive the packet contains in the data field, record the time as Tr 4. The result Tresult = Tr - Tt 5. Iterate this test for different Packet_out throughput and data length

Test Case Name:Flow entry multipart_reply message test Test Purpose:Measure flow statistics update time of a switch Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Add a flow entry to table 0 with a match field of TCP five-tuple and an action OFPAT_OUTPUT to an output port 4. Keep sending an OFPT_MULTIPART_REQUEST message with type of OFPMP_FLOW to request statistics of the flow added in step 3 5. Send a packet matches the flow added in step 3 to switch from dataplane, record the time as Tt 6. Wait enough time(e.g. 3 seconds) for switch until flow statistics are updated, record the time as Tr 7. The result Tresult = Tr - Tt 8. Iterate this test for different flows

Test Case Name:Port multipart_reply message test Test Purpose:Measure port statistics update time of a switch Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Send an OFPT_MULTIPART_REQUEST message with type of OFPMP_PORT_STATS to request statistics of one port, record the rx_packets and rx_bytes 4. Keep sending the OFPT_MULTIPART_REQUEST message of step 3. 5. Send a packet to switch from dataplane, record the time as Tt 6. Wait enough time(e.g. 3 seconds) for switch until port statistics are updated, record the time as Tr 7. The result Tresult = Tr - Tt 8. Iterate this test for sending different amount and kind of packets to switch

Test Case Name:Idle time out timer performance test Test Purpose:Test the accuracy of idle timeout of flow entries Test Methodology: 1. Setup connection between switch under test and test tool 2. Remove all flow entries of switch 3. Install a flow entry with idle timeout 5s and set the OFPFF_SEND_FLOW_REM flag 4. Send a matching packet to ingress port within idle timeout and record the time as Ts 5. Wait until the switch send an OFP_FLOW_REMOVED message to the test tool, record the time as Te 6. The result time, Tresult = Te - Ts 7. Compare Tresult with the hard timeout(5s), the error percentage equals (Tresult - 5s)/5s * 100% 8. Iterate this test with different number of installed flow entries