Academia.eduAcademia.edu

Cross Entropy

description1,052 papers
group15 followers
lightbulbAbout this topic
Cross entropy is a measure from the field of information theory that quantifies the difference between two probability distributions. It is commonly used in machine learning to evaluate the performance of classification models by calculating the average number of bits needed to encode events from one distribution using the optimal code for another distribution.
lightbulbAbout this topic
Cross entropy is a measure from the field of information theory that quantifies the difference between two probability distributions. It is commonly used in machine learning to evaluate the performance of classification models by calculating the average number of bits needed to encode events from one distribution using the optimal code for another distribution.

Key research themes

1. How are different entropy variants mathematically defined and related for time-series and complex systems analysis?

This research area aims to clarify the foundations and relationships of diverse entropy measures, especially those applied to time-series, continuous variables, and complex systems. It addresses conceptual confusions due to multiple entropy variants, establishing mathematical definitions, interrelations, and applicability to select appropriate entropy forms for specific data types and scientific fields.

Key finding: This paper systematically reviews and constructs an inclusive 'Entropy Universe' of various entropy definitions applied to time-series, specifying their mathematical formulations, origins, and mutual relationships. It... Read more
Key finding: It provides tight analytical bounds on the differential entropy for mixture distributions, revealing how entropy concavity deficits depend on component separations via total variation distance. This work extends classical... Read more
Key finding: Introduces logical entropy as a fundamental measure of information based on distinctions or differences within partitions, and demonstrates a precise non-linear transformation connecting logical entropy to Shannon entropy.... Read more
Key finding: Proposes a fundamentally discrete approach for estimating entropy in continuous state spaces by defining macrostates through sample quantiles and computing the Shannon entropy over these partitions. This geometric partition... Read more
Key finding: Examines entropy from a Bayesian perspective, highlighting entropy’s subjective and anthropomorphic aspects in thermodynamics. Revisits classical formulations by Linhart with Bayesian inference foundations, emphasizing the... Read more

2. What are the physical interpretations and thermodynamic implications of entropy, including irreversibility and entropy generation?

This theme investigates entropy’s physical meaning within thermodynamics, focusing on irreversibility, entropy generation, and the arrow of time. It tackles how macroscopic entropy changes emerge from microscopic interactions, the role of quanta, and links entropy to fundamental thermodynamic laws, energy distribution, and system-environment interactions, clarifying longstanding conceptual and philosophical issues about entropy's nature and conservation.

Key finding: Develops an analytical thermodynamic framework relating entropy generation to quantum transfers and irreversibility in system-environment interactions. Establishes that entropy generation physically arises from quanta... Read more
Key finding: Clarifies that thermodynamic entropy fundamentally quantifies thermal energy redistribution at absolute temperature and is always generated due to irreversible processes. It differentiates thermal entropy from general... Read more
Key finding: Critically evaluates commonly used metaphors for entropy (such as disorder or missing information), arguing that entropy is better understood as 'spreading' in space, time, and energy. Demonstrates how entropy’s distinctive... Read more
Key finding: Proposes that entropy decrease corresponds to the introduction of symmetry ('ordering') in a system, equating disorder with absence of symmetry. Demonstrates mathematically that symmetry imposition reduces entropy in binary... Read more

3. How does non-additive or generalized entropy frameworks (e.g., Tsallis, κ-entropy) relate to physical systems, and what are their mathematical composition properties?

This line of research focuses on generalized, non-extensive entropies that depart from the classical Boltzmann-Gibbs additive form. It addresses their interpretation, mathematical composition laws for independent subsystems, applications to finite heat bath effects, particle production processes, and connections to complex system behaviors, thereby expanding entropy’s applicability to non-equilibrium and complex systems.

Key finding: Derives the explicit composition rule for κ-entropy in statistically independent systems, showing that joint κ-entropy depends not only on the individual κ-entropies but also on entropic functionals evaluated at scaled... Read more
Key finding: Analyzes Tsallis entropy in multiparticle production phenomena, revealing a duality between non-extensivity parameters derived from entropy definitions and from observed distributions. Discusses implications for non-additive... Read more
Key finding: Shows that deviations from classical logarithmic entropy additivity can be physically interpreted as finite heat bath capacities and temperature fluctuations. It employs formal group theory to derive generalized composition... Read more
Key finding: Investigates mathematical properties of Tsallis entropy, including its scale and stochastic order preservation under aging processes, behavior in dynamical forms, and applicability to coherent and mixed system lifetimes.... Read more

All papers in Cross Entropy

A threshold-based algorithm is presented for the extraction of the pectoral muscle edge in mediolateral oblique view mammograms. The minimum cross-entropy thresholding algorithm is applied to local areas around the pectoral muscle to... more
Credal predictors are epistemic-uncertainty-aware models that produce a convex set of probabilistic predictions. They provide a principled framework for quantifying predictive epistemic uncertainty (EU) and have been shown to improve... more
Informational Approach (IA) to morphology processing has shaped significant body of psycholinguistic research in Serbian and some other languages. However, its central claim that word processing variability is fully accounted for by the... more
One of the most often-used stopping criteria is the cross-entropy stopping criterion (CESC). The CESC can stop turbo decoder iterations early by calculating mutual information improvements while maintaining bit error rate (BER)... more
Face recognition is one of the most widely publicized feature in the devices today and hence represents an important problem that should be studied with the utmost priority. As per the recent trends, the Convolutional Neural Network (CNN)... more
and treatment are crucial factors in preventing extensive retina damage and even vision loss. In today's practice, DR is diagnosed based on the examination of retina's fundus images by experienced medical practitioners based on the... more
An algorithm for quickly determining the presence of bacteria based on their intrinsic fluorescence signals has been developed. Applications of this algorithm are discussed.
An algorithm for quickly determining the presence of bacteria based on their intrinsic fluorescence signals has been developed. Applications of this algorithm are discussed.
This paper investigates a new heterogeneous method that dynamically sets the migration period of a distributed Genetic Algorithm (dGA). Each island GA of this multipopulation technique self-adapts the period for exchanging information... more
Handwritten Character Recognition (HCR) remains a fundamental yet challenging problem in pattern recognition due to variations in writing styles, distortions, noise, and inter-class similarity. This study proposes an intelligent AI-based... more
Brain tumor segmentation is a critical and challenging task in medical image analysis, essential for enhancing patient outcomes through timely detection and treatment. Manual segmentation is both laborious and subjective, highlighting the... more
We present a novel non-parametric unsupervised segmentation algorithm based on Region Competition ; but implemented within a Level Sets framework . The key novelty of the algorithm is that it can solve N ≥ 2 class segmentation problems... more
本論文は、AI支援研究における方法論的フレームワークである「プロンプト指向型研究オペレーティングシステム(Research OS)」の構造を、ソフトウェアアーキテクチャ可視化手法であるPlantUML C4モデルを用いて体系的に分析する。C4モデルの4階層(Context/Container/Component/Code)により、Research... more
Generalized zero shot learning (GZSL) is still a technical challenge of deep learning as it has to recognize both source and target classes without data from target classes. To preserve the semantic relation between source and target... more
В роботі запропоновано метод класифікації. Він заснований на комбінованому алгоритмі негативної селекції, який був спочатку розроблений для задач бінарної класифікації. Точність розробленого алгоритму була перевірена експериментальним... more
This paper proposes a data-driven approximate Bayesian computation framework for parameter estimation and uncertainty quantification of epidemic models, which incorporates two novelties: (i) the identification of the initial conditions by... more
This paper investigates various combinations of preprocessing methods (attribute selection, normalization, resampling, and imputation) and evaluates their impact on the performance of decision tree models for predicting customer churn.... more
Modern machine learning, fueled by large datasets and complex models, faces a critical tension. The statistical principles underpinning learning (generalization, efficiency, robustness) often clash with the computational realities of... more
National accounts statistics are the result of the integration of several data sources. At present, the Italian national accounts use household surveys data for estimating labour units only, not for estimating the monetary aggregates. In... more
This report is an output from a project funded by the UK Department for International Development (DFID) under the UK provision of technical assistance to developing countries. The views expressed are not necessarily those of the... more
Segmenting biomarkers in medical images is crucial for various biotech applications. Despite advances, Transformer and CNN based methods often struggle with variations in staining and morphology, limiting feature extraction. In medical... more
Whole-body pose estimation is a challenging task that requires simultaneous prediction of keypoints for the body, hands, face, and feet. Whole-body pose estimation aims to predict fine-grained pose information for the human body,... more
Driver drowsiness is a major factor that contributing to road accidents. Several researches are ongoing to detect driver drowsiness, but they suffer from the complexity and cost of the models. This paper introduces a hybrid artificial... more
This article presents a compression-based adaptive algorithm for Chinese Pinyin input. There are many different input methods for Chinese character text and the phonetic Pinyin input method is the one most commonly used. Compression by... more
A realistic analysis of media access control protocols for wireless applications must incorporate the affects of physical layer phenomena such as Rayleigh fading, shadowing and other processes that cause signal attenuation. In this work... more
“Neutrosophic Sets and Systems” (NSS) Vol. 76 (2025) presents the latest scholarly contributions to neutrosophy and its interdisciplinary extensions. The issue showcases original research on neutrosophic set theory, neutrosophic logic,... more
In this paper we analyze the performance of three algorithms for soft evidential update, in which a probability distribution represented by a Bayesian network is modified to a new distribution constrained by given marginals, and closest... more
Contrastive losses yield state-of-the-art performance for person re-identification, face verification and few shot learning. They have recently outperformed the cross-entropy loss on classification at the ImageNet scale and outperformed... more
Contrastive losses yield state-of-the-art performance for person re-identification, face verification and few shot learning. They have recently outperformed the cross-entropy loss on classification at the ImageNet scale and outperformed... more
This work proposes real time fixed yoga pose estimation and correction system. The pose estimation approaches, including MoveNet and OpenPose, are likely to suffer from low-quality keypoint localization when there are complex body poses... more
Information theory provides principled models for different machine learning problems, such as clustering, dimensionality reduction, classification, and many others. However, using the definitions of information theoretic quantities... more
Out-of-distribution (OOD) detection is a critical aspect of deploying robust and reliable machine learning models. This paper investigates the efficacy of self-supervised pretraining using Momentum Contrast (MoCo) for OOD detection. We... more
Skin cancer represents one of the most common malignancies globally, making early detection crucial for effective treatment and improved patient outcomes. While dermatologists typically rely on dermoscopy and clinical examinations for... more
In this paper, we consider the local fractional decomposition method, variational iteration method, and differential transform method for analytic treatment of linear and nonlinear local fractional differential equations, homogeneous or... more
Download research papers for free!