Chimbuko: Performance Analysis
Performance analysis C/C++ library
This library is a part of Chimbuko, a workflow-level scalable performance trace analysis tool.
Funded by the Exascale Computing Project (ECP), U.S. Department of Energy
Related Github repositories
Introduction
Installation and Usage
- Installation and Usage
- Installation
- Instrumenting an application with Tau
- Running an application under Chimbuko
- Online Analysis
- Running on Summit
- Running on Slurm-based systems
- Running with the CXI network provider on Frontier/Crusher
- Scaling to large job sizes
- Integration with TAU’s monitoring plugin
- Online analysis of an MPI application with a non-MPI installation of Chimbuko (advanced)
- Online analysis of a non-MPI application with a non-MPI installation of Chimbuko (advanced)
- Launching Chimbuko’s components together through a single script (advanced, experimental)
- Examples
- Online Analysis
- Analysing the results of a run with Chimbuko
IO Schema
Appendix
API For Developers
- Full API Listing
- API
- AD
- chimbuko
- ADAnomalyProvenance
chimbukochimbuko::moduleschimbuko::modules::performance_analysischimbuko::modules::performance_analysis::ADAnomalyProvenanceADAnomalyProvenance()getProvenanceEntries()getEventProvenance()setWindowSize()setMinimumAnomalyTime()linkPerf()linkAlgorithmParams()linkMonitoring()linkMetadata()linkEventManager()getStackInformation()getWindowCounters()getGPUeventInfo()getExecutionWindow()getNodeState()getHostname()getAlgorithmParams()m_perfm_event_manm_metadatam_window_sizem_monitoringm_algo_paramsm_min_anom_timem_normalevents
- ADCounter
- ADDefine
- ADEvent
CommStack_tCommStackMap_p_tCounterStack_tCounterStackMap_p_tCallList_tCallListIterator_tCallListMap_p_tCallStack_tCallStackMap_p_tExecDataMap_tchimbuko::modules::performance_analysis::EventInfochimbuko::modules::performance_analysis::ADEventIDmapchimbuko::modules::performance_analysis::ADEventADEvent()~ADEvent()linkEventType()linkFuncMap()linkCounterMap()linkGPUthreadMap()getFuncMap()getEventType()getCounterMap()getExecDataMap()getCallListMap()getCallListMap()getCallData()getCallWindowStartEnd()getCallStack()clear()addEvent()addFunc()addComm()addCounter()addCall()trimCallList()getCallListSize()purgeCallList()show_status()getUnmatchCorrelationIDevents()ignoreCorrelationIDsForFunction()checkAndMatchCorrelationID()stackProtectGC()stackUnProtectGC()m_funcMapm_eventTypem_counterMapm_eidx_func_entrym_eidx_func_exitm_eidx_comm_sendm_eidx_comm_recvm_cidx_corr_idm_gpu_thread_Mapm_commStackm_counterStackm_callStackm_callListm_execDataMapm_callIDMapm_unmatchedCorrelationIDm_ignoreCorrelationIDm_stackLockedUnlabeledm_verbosechimbuko::modules::performance_analysis::ADEvent::purgeReport
- ADglobalFunctionIndexMap
- ADio
- ADLocalCounterStatistics
chimbuko::modules::performance_analysis::ADLocalCounterStatisticsADLocalCounterStatistics()ADLocalCounterStatistics()gatherStatistics()updateGlobalStatistics()linkPerf()getStats()get_json()serialize()serialize_cerealpb()deserialize_cerealpb()net_serialize()net_deserialize()setStats()getProgramIndex()setProgramIndex()getIOstep()setIOstep()operator==()operator!=()m_program_idxm_stepm_which_counterm_statsm_perfupdateGlobalStatistics()
- ADLocalFuncStatistics
chimbuko::modules::performance_analysis::ADLocalFuncStatisticsADLocalFuncStatistics()ADLocalFuncStatistics()gatherStatistics()gatherAnomalies()updateGlobalStatistics()get_json()linkPerf()getAnomalyData()setAnomalyData()getFuncStats()setFuncStats()serialize()serialize_cerealpb()deserialize_cerealpb()net_serialize()net_deserialize()operator==()operator!=()m_anom_datam_funcstatsm_perfupdateGlobalStatistics()
- ADMetadataParser
- ADNetClient
chimbuko::ADNetClientchimbuko::ADZMQNetClientchimbuko::ADLocalNetClientchimbuko::ADThreadNetClientADThreadNetClient()enqueue_action()connect_ps()disconnect_ps()send_and_receive()async_send()async_send_supported()setRecvTimeout()stopWorkerThread()getNwork()~ADThreadNetClient()send_and_receive()getWorkItem()run()workermcvqueuem_is_runningchimbuko::ADThreadNetClient::ClientAction
- ADNormalEventProvenance
- ADOutlier
chimbuko::ADOutlierADOutlier()~ADOutlier()use_ps()linkNetworkClient()run()linkPerf()get_global_parameters()setGlobalParameters()setGlobalModelSyncFrequency()getAlgorithmName()set_algorithm()sync_param()updateGlobalModel()m_use_psm_net_clientm_paramm_local_paramm_sync_call_countm_global_model_sync_freqm_rankm_perfchimbuko::ADOutlier::AlgoParams
chimbuko::ADOutlierSSTDchimbuko::ADOutlierHBOSchimbuko::ADOutlierCOPOD
- ADParser
chimbuko::modules::performance_analysis::ADParserADParser()~ADParser()linkNetClient()linkPerf()getFuncMap()getEventType()getCounterMap()getStatus()getCurrentStep()beginStep()endStep()setBeginStepTimeout()update_attributes()fetchFuncData()fetchCommData()fetchCounterData()getFuncData()getNumFuncData()getCommData()getNumCommData()getCounterData()getNumCounterData()getNewMetaData()getEvents()addFuncData()addCounterData()addCommData()setFuncDataCapacity()setCommDataCapacity()setCounterDataCapacity()setFuncMap()setEventTypeMap()setCounterMap()getGlobalFunctionIndex()getCorrelationIDcounterIdx()setDataRankOverride()checkEventOrder()validateEvent()createAndValidateEvent()m_adm_iom_readerm_inputFilem_engineTypem_beginstep_timeoutm_statusm_openedm_attr_oncem_current_stepm_rankm_program_idxm_metadata_seenm_new_metadatam_funcMapm_eventTypem_counterMapm_correlation_id_cidm_timer_event_countm_event_timestampsm_comm_countm_comm_timestampsm_counter_countm_counter_timestampsm_global_func_idx_mapm_perfm_data_rank_override
- ADProvenanceDBclient
- ADProvenanceDBengine
- AnomalyData
operator==()operator!=()chimbuko::modules::performance_analysis::AnomalyDataAnomalyData()AnomalyData()set()get_app()get_rank()get_step()get_min_ts()get_max_ts()get_n_anomalies()get_outlier_score_stats()set_min_ts()set_max_ts()incr_n_anomalies()add_outlier_score()get_json()serialize()serialize_cerealpb()deserialize_cerealpb()net_serialize()net_deserialize()m_appm_rankm_stepm_min_timestampm_max_timestampm_n_anomaliesm_outlier_scoresoperator==operator!=
- ExecData
operator<()operator>()chimbuko::modules::performance_analysis::Event_tchimbuko::modules::performance_analysis::CommData_tchimbuko::modules::performance_analysis::CounterData_tchimbuko::modules::performance_analysis::ExecData_tExecData_t()ExecData_t()ExecData_t()~ExecData_t()get_id()get_funcname()get_pid()get_tid()get_rid()get_fid()get_entry()get_exit()get_runtime()get_inclusive()get_exclusive()get_label()get_parent()get_messages()get_counters()get_n_message()get_n_children()get_n_counter()get_outlier_score()get_outlier_severity()set_label()set_id()set_outlier_score()set_parent()set_funcname()update_exit()update_exit()update_exclusive()inc_n_children()add_message()add_counter()is_same()get_json()can_delete()register_reference()deregister_reference()reference_count()set_GPU_correlationID_partner()has_GPU_correlationID_partner()n_GPU_correlationID_partner()get_GPU_correlationID_partner()m_idm_funcnamem_pidm_tidm_ridm_fidm_entrym_exitm_runtimem_exclusivem_labelm_scorem_parentm_n_childrenm_n_messagesm_messagesm_countersm_referencesm_gpu_correlation_id_partner
chimbuko::modules::performance_analysis::MetaData_t
- utils
- Anomaly Detection Algorithm Parameters
- ParamInterface
- CopodParam
chimbuko::CopodFuncParamchimbuko::CopodParamCopodParam()~CopodParam()clear()copy()find()get_copodstats()set_copodstats()size()serialize()update()assign()show()assign()operator[]()update()update()update_and_return()get_algorithm_params()get_all_algorithm_params()serialize_json()deserialize_json()serialize_cerealpb()deserialize_cerealpb()m_copodstats
- HbosParam
chimbuko::HbosFuncParamchimbuko::HbosParamHbosParam()~HbosParam()copy()clear()find()get_hbosstats()set_hbosstats()size()serialize()update()assign()show()assign()operator[]()update()update()update_and_return()get_algorithm_params()get_all_algorithm_params()getMaxBins()setMaxBins()generate_histogram()serialize_json()deserialize_json()operator==()operator!=()serialize_cerealpb()deserialize_cerealpb()update_internal()m_hbosstatsm_maxbins
- SstdParam
chimbuko::SstdParamSstdParam()~SstdParam()clear()size()serialize()update()assign()show()update()update()update()update_and_return()update_and_return()assign()operator[]()get_runstats()get_algorithm_params()get_all_algorithm_params()serialize_json()deserialize_json()serialize_cerealpb()deserialize_cerealpb()access_runstats()m_runstats
- Parameter Server
- AnomalyStat
- global_anomaly_stats
- global_counter_stats
- PSglobalFunctionIndexMap
- PSProvenanceDBclient
- PSstatSender
- Network
- NetInterface
- MPINet
- ZMQNet
chimbuko::ZMQNetStatusZMQNet()~ZMQNet()init()finalize()run()stop()name()setMaxMsgPerPollCycle()setIOthreads()setPort()setAutoShutdown()setTimeOut()getStatus()send()recv()init_thread_pool()recvAndSend()m_contextm_n_requestsm_threadsm_perfm_perf_thrm_thr_mutexm_max_pollcyc_msgm_io_threadsm_mutexm_clientsm_client_has_connectedm_portm_autoshutdownm_statusm_poll_timeoutm_remote_stop_cmd
- ZMQMENet
- Message
MessageTypeBuiltinMessageKindMessageCmdtoString()toString()toString()chimbuko::Message
- Utils
- ADIOS2parseUtils
- Anomalies
- barrier
- commandLineParser
_WRP_P_0_WRP_P_1_WRP_P_2_WRP_P_3_WRP_P_4_WRP_P_5_WRP_P_GET_MACROWRAP_PARENTaddOptionalCommandLineArgaddOptionalCommandLineArgWithDefaultaddOptionalCommandLineArgDefaultHelpStringaddOptionalCommandLineArgWithFlagaddOptionalCommandLineArgMultiValueaddOptionalCommandLineArgOptArgaddOptionalCommandLineArgOptArgWithDefaultaddMandatoryCommandLineArgaddMandatoryCommandLineArgDefaultHelpStringchimbuko::get_member_ptr_class< R T::* >chimbuko::get_member_ptr_data_type< R T::* >chimbuko::optionalCommandLineArgBasechimbuko::optionalCommandLineArgchimbuko::optionalCommandLineArgWithFlagchimbuko::optionalCommandLineArgMultiValue_parsechimbuko::optionalCommandLineArgMultiValue_parse< MemberType >chimbuko::optionalCommandLineArgMultiValuechimbuko::mandatoryCommandLineArgBasechimbuko::mandatoryCommandLineArgchimbuko::commandLineParser
- DispatchQueue
- error
- hash
- map
- memutils
- mtQueue
- PerfStats
- RunMetric
- RunStats
operator+()operator==()operator!=()static_mean()static_std()chimbuko::RunStatsRunStats()~RunStats()clear()serialize()serialize_cerealpb()deserialize_cerealpb()net_serialize()net_deserialize()push()count()minimum()maximum()accumulate()mean()variance()stddev()skewness()kurtosis()set_do_accumulate()get_do_accumulate()get_json()get_stat_values()operator+=()equiv()set_eta()set_rho()set_count()m_countm_etam_rhom_taum_phim_minm_maxm_accm_do_accumulateoperator+operator==operator!=chimbuko::RunStats::RunStatsValues
- string
- threadPool
- time
- verbose
- AD
- API