Skip to main content

Fuzzy cross-entropy

Abstract

This paper deals with the divergence of fuzzy variables from a priori one. Within the framework of credibility theory, a fuzzy cross-entropy is defined to measure the divergence, and some mathematical properties are investigated. Furthermore, a minimum cross-entropy principle is proposed, which tells us that out of all membership functions satisfying given moment constraints, we should choose the one that is closest to the given a priori membership function.

Introduction

Fuzzy entropy provides a quantitative measure of the uncertainty associated with each fuzzy variable. Since Zadeh [1] introduced the fuzzy entropy as a weighted shannon entropy, researchers gave several definitions from different angles, such as De Luca and Termini [2], Yager [3], Kaufmann [4], Kosko [5], Pal and Pal [6]. The above definitions characterize the uncertainty resulting primarily from the linguistic vagueness rather than resulting from information deficiency and vanish when the fuzzy variable is an equipossible one. However, Liu [7] suggested that a fuzzy entropy should meet at least the following three basic requirements: the entropy of a crisp number is zero; the entropy of an equipossible fuzzy variable is maximum; and the entropy is applicable not only to finite and infinite cases but also to discrete and continuous cases. In order to meet these requirements, within the framework of credibility theory, Li and Liu [8] provided a new definition of fuzzy entropy to characterize the uncertainty resulting from information deficiency which is caused by the impossibility to predict the specified value that a fuzzy variable takes. Based on this definition, Li and Liu [9] proposed the fuzzy maximum entropy principle and proved some maximum entropy theorems.

This paper is devoted to formulate a fuzzy cross-entropy characterized by credibility measure. For this purpose, we organize this paper as follows. The ‘Preliminaries’ section recalls some useful definitions and properties about credibility theory. The ‘Fuzzy cross-entropy’ section defines the fuzzy cross-entropy and studies some useful properties. In the ‘Minimum cross-entropy principle’ section, the minimum cross-entropy principle is proposed. At the end of this paper, a brief summary is given.

Preliminaries

Credibility theory [10] is a branch of mathematics for studying the behavior of fuzzy phenomena. Let Θ be a nonempty set, and let be the power set of Θ. Each element A of is called an event. In 2002, Liu and Liu [11] presented a credibility measure Cr{A} to express the chance that event A occurs. Furthermore, Li and Liu [12] proved that a set function is a credibility measure if and only if it satisfies the following axioms:

Axiom 1. (Normality) Cr{Θ}=1;

Axiom 2. (Monotonicity) Cr{A}≤Cr{B} whenever A⊂B;

Axiom 3. (Self-duality) Cr is self-dual, i.e., Cr{A}+Cr{A c}=1 for any event A;

Axiom 4. (Countable subadditivity) Cr{∪ i A i }= supiCr{A i } for any events {A i } with supiCr{A i }<0.5.

If Cr is a credibility measure, the triplet (Θ,,Cr) is called a credibility space. A fuzzy variable is defined as a function from a credibility space (Θ,,Cr) to the set of real numbers. Let ξ be a fuzzy variable. Then, its membership function is derived from the credibility measure by:

$$\mu(x)=\left(2\text{Cr}\{\xi=x\}\right)\wedge 1, \ \forall x\in\Re. $$

Conversely, if ξ is a fuzzy variable with membership function μ, then, for any set B⊆ℜ, we have:

$$\text{Cr}\{\xi\in B\}=\frac{1}{2}\left(\sup\limits_{\small\small x\in B}\mu(x)+1-\sup\limits_{\small x\in B^{c}}\mu(x)\right). $$

This formula is also called the credibility inversion theorem.

Definition 2.1.

Let ξ be a fuzzy variable taking values in {x 1,x 2,⋯,x n } (Li and Liu [8]). Then, its fuzzy entropy is defined as:

$$H[\!\xi]=\sum\limits_{i=1}^{n}S(\text{Cr}\{\xi=x_{i}\}) $$

where S(t)=−t lnt−(1−t) ln(1−t).

Fuzzy entropy is used to quantify the uncertainty associated to fuzzy variables.

Theorem 2.1.

Let ξ be a fuzzy variable taking values in {x 1,x 2,⋯,x n } (Li and Liu [8]). Then, we have:

$$0\leq H[\!\xi]\leq n\ln2. $$

Especially, H[ ξ] attains its minimum value 0 if and only if ξ is a crisp number, and H[ ξ] attains its maximum value n ln2 if and only if ξ is an equipossible fuzzy variable.

Definition 2.2.

Let ξ be a continuous fuzzy variable (Li and Liu [8]). Then, its fuzzy entropy is defined as:

$$H[\!\xi]=\int_{-\infty}^{+\infty}S(\text{Cr}\{\xi=x\})\mathrm{d}x. $$

Theorem 2.2.

Let ξ be a continuous fuzzy variable taking values in [a,b] (Li and Liu [8]). Then, we have:

$$0\leq H[\!\xi]\leq (b-a)\ln2. $$

Especially, H[ ξ] attains its minimum value if and only if ξ is a crisp number, and H[ ξ] attains its maximum value if and only if ξ is an equipossible fuzzy variable.

In 2007, Li and Liu [9] proposed a fuzzy maximum entropy principle, which tells us that out of all the membership functions satisfying the given constraints, we should select the one that maximizes the entropy.

Fuzzy cross-entropy

In this section, we define a fuzzy cross-entropy for quantifying the divergence of fuzzy variables from an a priori one. The relation between fuzzy entropy and fuzzy cross-entropy is also discussed.

Definition 3.1.

Let ξ and η be two discrete fuzzy variables taking values in {x 1,x 2,⋯,x n }. Then, the fuzzy cross-entropy of ξ from η is defined as:

$$D[\!\xi;\eta]=\sum\limits_{i=1}^{n}T\left(\text{Cr}\{\xi=x_{i}\},\text{Cr}\{\eta=x_{i}\}\right) $$

where T(s,t)=s ln(s/t)+(1−s) ln((1−s)/(1−t)).

It is easy to prove that D[ ξ;η] is permutationally symmetric, i.e., the value does not change if the outcomes are labeled differently.

Definition 3.2.

Let ξ and η be two continuous fuzzy variables taking values in [a,b]. Then, the cross-entropy of ξ from η is defined as:

$$D[\!\xi;\eta]={\int_{a}^{b}}T\left(\text{Cr}\{\xi=x\},\text{Cr}\{\eta=x\}\right)\mathrm{d}x. $$

Let μ and ν be the membership functions of continuous fuzzy variables ξ and η, respectively. Since Cr{ξ=x}=μ(x)/2 and Cr{η=x}=ν(x)/2, the cross-entropy of ξ from η can be rewritten as:

$$D[\!\xi;\eta]={\int_{a}^{b}}\mu(x)/2\ln\left(\mu(x)/\nu(x)\right) +\left(1-\mu(x)/2\right)\ln\left((2-\mu(x))/(2-\nu(x))\right)\mathrm{d}x.$$

Remark 3.1.

It is easy to extend the concept of cross-entropy to fuzzy vectors. If ξ=(ξ 1,ξ 2,⋯,ξ m ) and η=(η 1,η 2,⋯,η m ) are discrete, we have:

$$\begin{array}{@{}rcl@{}} D[\!\boldsymbol{\xi}, \boldsymbol{\eta}]=\!\!\!\!&&\sum\limits_{i_{1}=1}^{n_{1}}\sum\limits_{i_{2}=1}^{n_{2}}\cdots\sum\limits_{i_{m}=1}^{n_{m}}T(\text{Cr}\{\xi_{1}=x_{i_{1}},\xi_{2}=x_{i_{2}},\cdots,\xi_{m}=x_{i_{m}}\},\\ &&\text{Cr}\{\eta_{1}=x_{i_{1}},\eta_{2}=x_{i_{2}},\cdots,\eta_{m}=x_{i_{m}}\}). \end{array} $$

If ξ and η are continuous variables, we have:

$$\begin{array}{@{}rcl@{}} D[\!\boldsymbol{\xi}, \boldsymbol{\eta}]\,=\!\!\!&&\int_{a_{1}}^{b_{1}}\int_{a_{2}}^{b_{2}}\cdots\int_{a_{m}}^{b_{m}}T(\text{Cr}\{\xi_{1}=x_{1},\xi_{2}=x_{2},\cdots,\xi_{m}=x_{m}\},\\ &&\text{Cr}\{\eta_{1}=x_{1},\eta_{2}=x_{2},\cdots,\eta_{m}=x_{m}\})\mathrm{d}x_{1}\mathrm{d}x_{2}\cdots\mathrm{d}x_{m}. \end{array} $$

Remark 3.2.

It is clear that T(s,t) is a function from [ 0,1]×[ 0,1] to [ 0,+∞). Please also mention that:

$$T(s,0)=\left\{\begin{array}{cc} 0, &\text{if} \ s=0\\ +\infty, &\text{if} \ s>0, \end{array} \right. \ \ \ \ \ \ \ \ \ \ T(s,1)= \left\{ \begin{array}{cc} 0, &\text{if} \ s=1\\ +\infty, &\text{if} \ s<1. \end{array} \right. $$

In addition, it is easy to prove that:

$$\;\,\frac{\partial T}{\partial s}=\ln\left(\frac{s}{t}\right)-\ln\left(\frac{1-s}{1-t}\right),\ \frac{\partial T}{\partial t}=\frac{t-s}{t(1-t)},$$
$$\frac{\partial^{2} T}{\partial s^{2}}=\frac{1}{s(1-s)}, \ \frac{\partial^{2} T}{\partial s\partial t}=\frac{\partial^{2} T}{\partial t\partial s}=-\frac{1}{t(1-t)},\ \frac{\partial^{2} T}{\partial t^{2}}=\frac{s}{t^{2}}+\frac{1-s}{(1-t)^{2}}. $$

Then, the following properties about T(s,t) can be easily proved: (a) T(s,t) is strictly convex with respect to (s,t) and attains its minimum value zero on the line s=t; and (b) for any 0≤s≤1 and 0≤t≤1, we have T(s,t)=T(1−s,1−t).

Theorem 3.1.

For any fuzzy variables ξ and η, we have D[ ξ;η]≥0, and the equality holds if and only if ξ and η have the same membership function.

Proof.

Let μ and ν be the membership functions of discrete fuzzy variables ξ and η, respectively. Since T(s,t) is strictly convex about (s,t) and attains its minimum value zero on the line s=t, we have T(Cr{ξ=x i },Cr{η=x i })≥0 for all i, which implies that:

$$D[\!\xi;\eta]=\sum\limits_{i=1}^{\infty}T(\text{Cr}\{\xi=x_{i}\},\text{Cr}\{\eta=x_{i}\})\geq 0. $$

Furthermore, for any 0≤s ∗≤1, the unique minimum point of T(s ∗,t) is t=s ∗. Thus, we have D[ ξ;η]=0 if and only if T(Cr{ξ=x i },Cr{η=x i })=0, that is:

$$\mu(x_{i})=(2\text{Cr}\{\xi=x_{i}\})\wedge 1=(2\text{Cr}\{\eta=x_{i}\})\wedge1=\nu(x_{i}) $$

for all i=1,2,⋯,n. If ξ and η are continuous fuzzy variables, the theorem can be proved in a similar way. The proof is complete. □

Theorem 3.2.

Let τ be the equipossible fuzzy variable with membership function ν(x i )=1 for all i=1,2,⋯,n. Then, for any discrete fuzzy variable ξ taking values in {x 1,x 2,⋯,x n }, we have:

$$D[\!\xi,\tau]=n\ln2-H[\!\xi]. $$

Proof.

According to the credibility inversion theorem, it is easy to prove that Cr{τ=x i }=0.5 for all i=1,2,⋯,n. It follows from the definition of cross-entropy that D[ ξ,τ] is:

$$\begin{array}{@{}rcl@{}} &&\sum\limits_{i=1}^{n}\text{Cr}\{\xi=x_{i}\}\ln(2\text{Cr}\{\xi=x_{i}\})+\left(1-\text{Cr}\{\xi=x_{i}\}\right)\ln(2-2\text{Cr}\{\xi=x_{i}\})\\ =&&\sum\limits_{i=1}^{n}\ln2+\text{Cr}\{\xi=x_{i}\}\ln\text{Cr}\{\xi=x_{i}\}+\left(1-\text{Cr}\{\xi=x_{i}\}\right)\ln(1-\text{Cr}\{\xi=x_{i}\})\\ =&&n\ln2-H[\!\xi]. \end{array} $$

The proof is complete. â–¡

Theorem 3.3.

Let τ be the equipossible fuzzy variable with membership function ν(x)=1 for all x∈[ a,b]. Then, for any continuous fuzzy variable ξ taking values in [ a,b], we have:

$$D[\!\xi,\tau]=(b-a)\ln2-H[\!\xi]. $$

Proof.

It follows from the definition of cross-entropy that D[ ξ,τ] is:

$$\begin{array}{@{}rcl@{}} &&{\int_{a}^{b}}(\mu(x)/2)\ln\mu(x)+\left(1-\mu(x)/2\right)\ln(2-\mu(x))\mathrm{d}x\\[0.2cm] =&&{\int_{a}^{b}}\ln2+\left((\mu(x)/2)\ln\left(\mu(x)/2\right)+\left(1-\mu(x)/2\right)\ln\left(1-\mu(x)/2\right)\right)\mathrm{d}x\\[0.2cm] =&&(b-a)\ln2-H[\!\xi]. \end{array} $$

The proof is complete. â–¡

Minimum cross-entropy principle

In many real problems, the membership function of a fuzzy variable is unavailable except some partial information, for example, moment constraints, which may be based on observations. In this case, the maximum entropy principle (Li and Liu [9]) tells us that out of all the membership functions satisfying given constraints, choose the one that has maximum entropy. However, there may be another type of information, for example, a priori membership function, which may be based on intuition or experience with the problem. If both the a priori membership function and the moment constraints are given, which membership function should we choose? The following minimum cross-entropy principle tells us that out of all membership functions satisfying given moment constraints, choose the one that is closest to the given a priori membership function.

There is nothing mysterious about this principle. It is just based on common sense. Our membership function must be consistent with observations or given information, and if there are many membership functions consistent with the given information, we must choose the one that is nearest to our intuition and experience. On the other hand, if we have no a priori experience or intuition to guide us, we choose the membership function that is nearest to the equipossible one. In this sense, if the a priori membership function is not prescribed and the fuzzy variable is simple (bounded for continuous case), the maximum entropy principle and minimum cross-entropy principle are consistent because:

$$D[\!\xi;\upsilon]=\max_{\eta}D[\!\eta;\upsilon]-H[\!\xi] $$

where Ï… is the equipossible fuzzy variable.

Conclusion

Based on credibility measure, a definition of cross-entropy was proposed in this paper to measure the divergence of fuzzy variables from a priori one, and some properties were investigated. Furthermore, a minimum cross-entropy principle was proposed as an important entropy optimization principle.

References

  1. Zadeh, LA: Probability measures of fuzzy events. J. Math. Anal. Appl. 23, 421–427 (1968).

    Article  MATH  MathSciNet  Google Scholar 

  2. De Luca, A, Termini, S: A definition of nonprobabilistic entropy in the setting of fuzzy sets theory. Inf. Control. 20, 301–312 (1972).

    Article  MATH  MathSciNet  Google Scholar 

  3. Yager, RR: On measures of fuzziness and negation, part I: membership in the unit interval. Int. J. General Syst. 5, 221–229 (1979).

    Article  MATH  MathSciNet  Google Scholar 

  4. Kaufmann, A: Introduction to the Theory of Fuzzy Subsets. Academic Press, New York (1975).

    MATH  Google Scholar 

  5. Kosko, B: Fuzzy entropy and conditioning. Inf. Sci. 40, 165–174 (1986).

    Article  MATH  MathSciNet  Google Scholar 

  6. Pal, NR, Pal, SK: Higher order fuzzy entropy and hybrid entropy of a set. Inf. Sci. 61, 211–231 (1992).

    Article  MATH  Google Scholar 

  7. Liu, B: A survey of entropy of fuzzy variables. J. Uncertain Syst. 1(1), 4–13 (2007).

    Google Scholar 

  8. Li, P, Liu, B: Entropy of credibility distributions for fuzzy variables. IEEE Trans. Fuzzy Syst. 16(1), 123–129 (2008).

    Article  Google Scholar 

  9. Li, X, Liu, B: Maximum entropy principle for fuzzy variables. Int. J. Uncertainty Fuzziness Knowledge-Based Syst. 15(Supp 2), 40–48 (2007).

    Google Scholar 

  10. Liu, B: Uncertainty Theory. Springer-Verlag, Berlin (2004).

    Book  MATH  Google Scholar 

  11. Liu, B, Liu, YK: Expected value of fuzzy variable and fuzzy expected value models. IEEE Trans. Fuzzy Syst. 10(4), 445–450 (2002).

    Article  Google Scholar 

  12. Li, X, Liu, B: A sufficient and necessary condition for credibility measures. Int. J. Uncertainty Fuzziness Knowledge-Based Syst. 14(5), 527–535 (2006).

    Article  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 71371027) and Program for New Century Excellent Talents in University under Grant No. NCET-13-0649.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xiang Li.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X. Fuzzy cross-entropy. J. Uncertain. Anal. Appl. 3, 2 (2015). https://doi.org/10.1186/s40467-015-0029-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40467-015-0029-5

Keywords