Download to read offline
Uploaded byVikram G Hosakote
Network OS Code Coverage demo using Bullseye tool
Vikram Hosakote gave a presentation on using the Bullseye code coverage tool to generate code coverage numbers in Cisco NXOS. Bullseye is used to capture coverage of C and C++ code and provide a ratio of tested vs total lines of code. The presentation covered building a Bullseye NXOS image, running tests to generate coverage files, processing the files on a Linux server, and viewing coverage reports in Bullseye's GUI or merged across devices. Automation ideas and integration with eARMS testing were also discussed.
More Related Content
![[OpenStack Day in Korea 2015] Track 1-6 - 갈라파고스의 이구아나, 인프라에 오픈소스를 올리다. 그래서 보이...](https://cdn.slidesharecdn.com/ss_thumbnails/16-150213044916-conversion-gate02-thumbnail.jpg?width=640&height=640&fit=bounds)
![[2015-11월 정기 세미나]K8s on openstack](https://cdn.slidesharecdn.com/ss_thumbnails/k8sonopenstack-151127132857-lva1-app6891-thumbnail.jpg?width=640&height=640&fit=bounds)
![[OpenInfra Days Korea 2018] (Track 3) - CephFS with OpenStack Manila based on...](https://cdn.slidesharecdn.com/ss_thumbnails/36openinfra-2018-naver-cephfs-180704055415-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Network OS Code Coverage demo using Bullseye tool
Network OS Code Coverage demo using Bullseye tool
- 1. Cisco Confidential© 2010Cisco and/or its affiliates. All rights reserved. 1 Network OS Bullseye code coverage demo Vikram Hosakote vhosakot@cisco.com June 21, 2012 Software Engineer – NOSTG
- 2. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 2 • What is Bullseye ? • Building a Bullseye NXOS image • Generating Bullseye code-coverage numbers in NXOS • Post-processing on NXOS Linux build server • Merging coverage reports • Viewing coverage-numbers in the Bullseye GUI • Demo • Automation ideas, eARMS integration ideas, HA testing like SSO and ISSU • Bullseye help • Q&A
- 3. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 3 • A tool to capture coverage of C and C++ code. • Used to generate code-coverage numbers in NXOS at Cisco. • What is code coverage ? The ratio of number of lines of code tested to number of lines of code developed. A number less than 100. Example – If LDP is developed with 100 lines of code, and just 50 lines tested, the code coverage is 50 / 100 = 50%
- 4. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 4 • Talk to component DEs and find out • The component name that needs Bullseye instrumentation (e.g. BGP, MPLS, LDP) • The directory structure of code and binaries (e.g. build/x86s/gdb/ldp/sup, routing- sw/mpls/ldp/nxos/cradle/) • New filenames, function names, and new lines of code developed (e.g. ldp_feature.c, ldp_cmi.h, ldp_cmi_show_neighbor_handler()). The new lines of code depend on the function name, component, and the feature. • If code-coverage is needed on supervisor, or line card, or both ? • Access to NXOS Linux build server • The exact RELEASE TAG in the build tree (e.g. REL_5_2_5_S30) • Use this link to build the Bullseye NXOS image using ACME, gmake, and isanadd CLIs http://zed.cisco.com/confluence/display/DC3SW/NX-OS+Quality+Initiative
- 5. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 5 • Load the Bullseye NXOS images – both kickstart and system • Run the tests • Code coverage file bootflash:code_coverage/<component>.cov is created both on active and standby • Copy this coverage file from both active and standby to the NXOS Linux build server via FTP / TFTP • Do post-processing (next slide) and generate coverage-numbers • In eARMS, submit Bullseye images like regular NXOS images
- 6. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 6 • Install Bullseye on the NXOS Linux build server by doing source /auto/andatc/independent/shellrc-files/current/rc/.cshrc.build setenv PATH "${PATH}:/auto/andatc/linux/bullseye/7.5.41/bin/" setenv BULLSEYE_VERSION "7.5.41" module add bullseye ; which covbr • Use these Bullseye CLIs with the coverage file <component>.cov for post-processing covsrc (coverage by source filename) covdir (coverage by source file directory) covfn (function coverage, whether a function was invoked or not) • Bullseye CLIs to generate coverage Reports – covhtml or –html option (generate HTML report) -c option to generate output as comma separated values (CSV)
- 7. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 7 NXOS-Linux-Build-Server# covsrc -f ldp.cov Source Function Coverage C/D Coverage --------------------------------- --------------------- --------------------- ...p/nxos/cradle/ldp_cltmgr_pss.c 21 / 29 = 72% 35 / 94 = 37% ...pls/ldp/nxos/cradle/ldp_main.c 13 / 17 = 76% 71 / 111 = 63% ...s/ldp/nxos/cradle/ldp_cltmgr.c 45 / 57 = 78% 123 / 268 = 45% ...mpls/ldp/nxos/cradle/ldp_mts.c 47 / 57 = 82% 173 / 312 = 55% ...mpls/ldp/nxos/cradle/ldp_cdb.c 21 / 25 = 84% 61 / 129 = 47% ...s/ldp/nxos/cradle/ldp_sysmgr.c 34 / 40 = 85% 84 / 148 = 56% ...mpls/ldp/nxos/cradle/ldp_pss.c 22 / 25 = 88% 61 / 106 = 57% ...p/nxos/cradle/ldp_utils_nxos.c 25 / 26 = 96% 52 / 87 = 59% ...os/cradle/ldp_cltmgr_pss_tlv.c 3 / 3 = 100% 14 / 27 = 51% .../nxos/cradle/ldp_cdb_pss_tlv.c 3 / 3 = 100% 22 / 36 = 61% .../ldp/nxos/cradle/ldp_feature.c 11 / 11 = 100% 43 / 68 = 63% ...cmn/include_lcs/ldpx_mem_api.h 1 / 1 = 100% 0 / 0 ...nxos/libldpclt/ldpc_cmn_mmsg.h 1 / 1 = 100% 0 / 0
- 8. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 8 NXOS-Linux-Build-Server# covdir -f ldp.cov Directory Function Coverage C/D Coverage --------------------------------- --------------------- --------------------- ...-bxb/Bullseye/include/routing/ 3 / 141 = 2% 14 / 366 = 3% ...g-sw/mpls/ldp/cmn/include_lcs/ 2 / 27 = 7% 0 / 10 = 0% ...ng-sw/mpls/ldp/nxos/libldpclt/ 7 / 12 = 58% 14 / 47 = 29% ...uting-sw/mpls/ldp/nxos/cradle/ 388 / 617 = 62% 1197 / 3349 = 35%
- 9. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 9 NXOS-Linux-Build-Server# covfn -f ldp.cov Function Source Line FnCov C/D Coverage -------- --------------------------------------- ----- --------------------- ldp...me ...mpls/ldp/nxos/cradle/ldp_cmi.c 3699 1 / 1 8 / 10 = 80% ldp...it ...mpls/ldp/nxos/cradle/ldp_pss.c 472 1 / 1 8 / 10 = 80% ldp...rs ...s/ldp/nxos/cradle/ldp_sysmgr.c 465 1 / 1 8 / 10 = 80% ldp...ng ...mpls/ldp/nxos/cradle/ldp_cdb.c 55 1 / 1 4 / 5 = 80% ldp...it ...pls/ldp/nxos/cradle/ldp_main.c 201 1 / 1 10 / 12 = 83% ldp...sz .../nxos/cradle/ldp_cdb_pss_tlv.c 490 1 / 1 5 / 6 = 83% ldp...ey ...s/ldp/nxos/cradle/ldp_cltmgr.c 295 1 / 1 5 / 6 = 83% ldp...it ...p/nxos/cradle/ldp_utils_nxos.c 259 1 / 1 5 / 6 = 83% ldp...et ...mpls/ldp/nxos/cradle/ldp_mts.c 1637 1 / 1 12 / 14 = 85% ldp...ad ...mpls/ldp/nxos/cradle/ldp_mts.c 1850 1 / 1 14 / 16 = 87% ldp...ad ...mpls/ldp/nxos/cradle/ldp_mts.c 1992 1 / 1 23 / 26 = 88% ldp...ch ...mpls/ldp/nxos/cradle/ldp_mts.c 1351 1 / 1 9 / 10 = 90% ldp...en ...mpls/ldp/nxos/cradle/ldp_pss.c 32 1 / 1 9 / 10 = 90%
- 10. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 10 NXOS-Linux-Build-Server# covfn -f ldp.cov --html > ldp_covfn.html
- 11. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 11 NXOS-Linux-Build-Server# covfn -f ldp.cov -c > ldp_covfn_csv "ldp_base_init","/ws/vhosakot-bxb/Bullseye/routing- sw/mpls/ldp/nxos/cradle/ldp_main.c",201,1,10,12,83% "ldp_cdb_pss_union_key_get_tlv_sz","/ws/vhosakot-bxb/Bullseye/routing- sw/mpls/ldp/nxos/cradle/ldp_cdb_pss_tlv.c",490,1,5,6,83% "ldp_cm_fsb_lookup_by_clt_key","/ws/vhosakot-bxb/Bullseye/routing- sw/mpls/ldp/nxos/cradle/ldp_cltmgr.c",295,1,5,6,83% "ldp_thread_info_init","/ws/vhosakot-bxb/Bullseye/routing- sw/mpls/ldp/nxos/cradle/ldp_utils_nxos.c",259,1,5,6,83% "ldp_mts_msg_options_set","/ws/vhosakot-bxb/Bullseye/routing- sw/mpls/ldp/nxos/cradle/ldp_mts.c",1637,1,12,14,85% "ldp_mts_sysmgr_thread","/ws/vhosakot-bxb/Bullseye/routing- sw/mpls/ldp/nxos/cradle/ldp_mts.c",1850,1,14,16,87% "ldp_mts_ctrl_thread","/ws/vhosakot-bxb/Bullseye/routing- sw/mpls/ldp/nxos/cradle/ldp_mts.c",1992,1,23,26,88% "ldp_dispatch","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_mts.c",1351,1,9,10,90% "ldp_pss_uri_open","/ws/vhosakot-bxb/Bullseye/routing- sw/mpls/ldp/nxos/cradle/ldp_pss.c",32,1,9,10,90%
- 12. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 12 • Bullseye CLI covmerge is used for merging multiple coverage files (*.cov) from multiple N7K routers. • Example to merge coverage files from the active and standby of two N7Ks covmerge -c –f merged_ldp_coverage.cov N7K1_active.cov N7K1_standby.cov N7K2_active.cov N7K2_standby.cov • If • A function has 4 decision paths (say 1,2,3 and 4), one N7K runs tests to cover paths 1 and 4, and another N7K runs tests to cover paths 1 and 2, • then • The merged coverage file from both the N7Ks will show 75 % coverage (1,2,4). Path 3 will be shown "uncovered"
- 13. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 13 /auto/andatc/linux/bullseye/7.5.41/bin/CoverageBrowser ldp.cov &
- 14. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 14
- 15. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 15 # Inside package CodeCoverage proc get_bullseye_coverage_data { {Bullseye_Build_Server_IP “101.0.0.102”} {FTP_username “rcpuser”} {FTP_Password “123rcp!”} Component_Name} { # Refer to flowchart on right USAGE package require CodeCoverage get_bullseye_coverage_data “ldp”
- 16. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 16 • Read variable in job file – set BULLSEYE_COVERAGE_NEEDED “yes” OR set BULLSEYE_COVERAGE_NEEDED $env(EARMS_CODE_COVERAGE) • eARMS enhancement similar to CFLOW (CSCtw78100) - • In eARMS, submit Bullseye images like regular NXOS images
- 17. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 17 • Unlike CFLOW, Bullseye coverage data will NOT get erased and will remain intact through • Router reload • Stateful Switchover (SSO) • In Service Software Upgrade (ISSU) • In Service Software Downgrade (ISSD)
- 18. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 18 • http://www.bullseye.com/help/index.html • http://www.bullseye.com/ • http://zed.cisco.com/confluence/display/DC3SW/NX-OS+Quality+Initiative • https://and-sw-web.cisco.com/twiki/bin/view/Main/WebHome • ACME commands - https://and-sw-web.cisco.com/twiki/bin/view/ACME/ACMECommands • http://www.bullseye.com/contact.html • and-sw-build@cisco.com • Vikram Hosakote (vhosakot@cisco.com)
- 19. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 19 • Webex recording of Bullseye demo to group.bstoner - https://cisco.webex.com/ciscosales/lsr.php?AT=pb&SP=MC&rID=61685767&rKey=0ec1c04f55cf4a23 • Webex recording of Bullseye demo to nostg-nxos-pi-test and nxos-pi-ft-core-team – https://cisco.webex.com/ciscosales/lsr.php?AT=pb&SP=MC&rID=61860282&rKey=8c23197cd63a9a18 • Webex recording of Bullseye demo to ASG-core and STEP teams – https://cisco.webex.com/ciscosales/lsr.php?AT=pb&SP=MC&rID=61903277&rKey=0a391701dee0c555
- 20. © 2010 Ciscoand/or its affiliates. All rights reserved. Cisco Confidential 20
- 21.