# Fuzzy cross-entropy

## Abstract

This paper deals with the divergence of fuzzy variables from a priori one. Within the framework of credibility theory, a fuzzy cross-entropy is defined to measure the divergence, and some mathematical properties are investigated. Furthermore, a minimum cross-entropy principle is proposed, which tells us that out of all membership functions satisfying given moment constraints, we should choose the one that is closest to the given a priori membership function.

## Introduction

Fuzzy entropy provides a quantitative measure of the uncertainty associated with each fuzzy variable. Since Zadeh [1] introduced the fuzzy entropy as a weighted shannon entropy, researchers gave several definitions from different angles, such as De Luca and Termini [2], Yager [3], Kaufmann [4], Kosko [5], Pal and Pal [6]. The above definitions characterize the uncertainty resulting primarily from the linguistic vagueness rather than resulting from information deficiency and vanish when the fuzzy variable is an equipossible one. However, Liu [7] suggested that a fuzzy entropy should meet at least the following three basic requirements: the entropy of a crisp number is zero; the entropy of an equipossible fuzzy variable is maximum; and the entropy is applicable not only to finite and infinite cases but also to discrete and continuous cases. In order to meet these requirements, within the framework of credibility theory, Li and Liu [8] provided a new definition of fuzzy entropy to characterize the uncertainty resulting from information deficiency which is caused by the impossibility to predict the specified value that a fuzzy variable takes. Based on this definition, Li and Liu [9] proposed the fuzzy maximum entropy principle and proved some maximum entropy theorems.

This paper is devoted to formulate a fuzzy cross-entropy characterized by credibility measure. For this purpose, we organize this paper as follows. The â€˜Preliminariesâ€™ section recalls some useful definitions and properties about credibility theory. The â€˜Fuzzy cross-entropyâ€™ section defines the fuzzy cross-entropy and studies some useful properties. In the â€˜Minimum cross-entropy principleâ€™ section, the minimum cross-entropy principle is proposed. At the end of this paper, a brief summary is given.

## Preliminaries

Credibility theory [10] is a branch of mathematics for studying the behavior of fuzzy phenomena. Let Î˜ be a nonempty set, and let be the power set of Î˜. Each element A of is called an event. In 2002, Liu and Liu [11] presented a credibility measure Cr{A} to express the chance that event A occurs. Furthermore, Li and Liu [12] proved that a set function is a credibility measure if and only if it satisfies the following axioms:

Axiom 1. (Normality) Cr{Î˜}=1;

Axiom 2. (Monotonicity) Cr{A}â‰¤Cr{B} whenever AâŠ‚B;

Axiom 3. (Self-duality) Cr is self-dual, i.e., Cr{A}+Cr{A c}=1 for any event A;

Axiom 4. (Countable subadditivity) Cr{âˆª i A i }= supiCr{A i } for any events {A i } with supiCr{A i }<0.5.

If Cr is a credibility measure, the triplet (Î˜,,Cr) is called a credibility space. A fuzzy variable is defined as a function from a credibility space (Î˜,,Cr) to the set of real numbers. Let Î¾ be a fuzzy variable. Then, its membership function is derived from the credibility measure by:

$$\mu(x)=\left(2\text{Cr}\{\xi=x\}\right)\wedge 1, \ \forall x\in\Re.$$

Conversely, if Î¾ is a fuzzy variable with membership function Î¼, then, for any set BâŠ†â„œ, we have:

$$\text{Cr}\{\xi\in B\}=\frac{1}{2}\left(\sup\limits_{\small\small x\in B}\mu(x)+1-\sup\limits_{\small x\in B^{c}}\mu(x)\right).$$

This formula is also called the credibility inversion theorem.

### Definition2.1.

Let Î¾ be a fuzzy variable taking values in {x 1,x 2,â‹¯,x n } (Li and Liu [8]). Then, its fuzzy entropy is defined as:

$$H[\!\xi]=\sum\limits_{i=1}^{n}S(\text{Cr}\{\xi=x_{i}\})$$

where S(t)=âˆ’t lntâˆ’(1âˆ’t) ln(1âˆ’t).

Fuzzy entropy is used to quantify the uncertainty associated to fuzzy variables.

### Theorem2.1.

Let Î¾ be a fuzzy variable taking values in {x 1,x 2,â‹¯,x n } (Li and Liu [8]). Then, we have:

$$0\leq H[\!\xi]\leq n\ln2.$$

Especially, H[ Î¾] attains its minimum value 0 if and only if Î¾ is a crisp number, and H[ Î¾] attains its maximum value n ln2 if and only if Î¾ is an equipossible fuzzy variable.

### Definition2.2.

Let Î¾ be a continuous fuzzy variable (Li and Liu [8]). Then, its fuzzy entropy is defined as:

$$H[\!\xi]=\int_{-\infty}^{+\infty}S(\text{Cr}\{\xi=x\})\mathrm{d}x.$$

### Theorem2.2.

Let Î¾ be a continuous fuzzy variable taking values in [a,b] (Li and Liu [8]). Then, we have:

$$0\leq H[\!\xi]\leq (b-a)\ln2.$$

Especially, H[ Î¾] attains its minimum value if and only if Î¾ is a crisp number, and H[ Î¾] attains its maximum value if and only if Î¾ is an equipossible fuzzy variable.

In 2007, Li and Liu [9] proposed a fuzzy maximum entropy principle, which tells us that out of all the membership functions satisfying the given constraints, we should select the one that maximizes the entropy.

## Fuzzy cross-entropy

In this section, we define a fuzzy cross-entropy for quantifying the divergence of fuzzy variables from an a priori one. The relation between fuzzy entropy and fuzzy cross-entropy is also discussed.

### Definition3.1.

Let Î¾ and Î· be two discrete fuzzy variables taking values in {x 1,x 2,â‹¯,x n }. Then, the fuzzy cross-entropy of Î¾ from Î· is defined as:

$$D[\!\xi;\eta]=\sum\limits_{i=1}^{n}T\left(\text{Cr}\{\xi=x_{i}\},\text{Cr}\{\eta=x_{i}\}\right)$$

where T(s,t)=s ln(s/t)+(1âˆ’s) ln((1âˆ’s)/(1âˆ’t)).

It is easy to prove that D[ Î¾;Î·] is permutationally symmetric, i.e., the value does not change if the outcomes are labeled differently.

### Definition3.2.

Let Î¾ and Î· be two continuous fuzzy variables taking values in [a,b]. Then, the cross-entropy of Î¾ from Î· is defined as:

$$D[\!\xi;\eta]={\int_{a}^{b}}T\left(\text{Cr}\{\xi=x\},\text{Cr}\{\eta=x\}\right)\mathrm{d}x.$$

Let Î¼ and Î½ be the membership functions of continuous fuzzy variables Î¾ and Î·, respectively. Since Cr{Î¾=x}=Î¼(x)/2 and Cr{Î·=x}=Î½(x)/2, the cross-entropy of Î¾ from Î· can be rewritten as:

$$D[\!\xi;\eta]={\int_{a}^{b}}\mu(x)/2\ln\left(\mu(x)/\nu(x)\right) +\left(1-\mu(x)/2\right)\ln\left((2-\mu(x))/(2-\nu(x))\right)\mathrm{d}x.$$

### Remark3.1.

It is easy to extend the concept of cross-entropy to fuzzy vectors. If Î¾=(Î¾ 1,Î¾ 2,â‹¯,Î¾ m ) and Î·=(Î· 1,Î· 2,â‹¯,Î· m ) are discrete, we have:

$$\begin{array}{@{}rcl@{}} D[\!\boldsymbol{\xi}, \boldsymbol{\eta}]=\!\!\!\!&&\sum\limits_{i_{1}=1}^{n_{1}}\sum\limits_{i_{2}=1}^{n_{2}}\cdots\sum\limits_{i_{m}=1}^{n_{m}}T(\text{Cr}\{\xi_{1}=x_{i_{1}},\xi_{2}=x_{i_{2}},\cdots,\xi_{m}=x_{i_{m}}\},\\ &&\text{Cr}\{\eta_{1}=x_{i_{1}},\eta_{2}=x_{i_{2}},\cdots,\eta_{m}=x_{i_{m}}\}). \end{array}$$

If Î¾ and Î· are continuous variables, we have:

$$\begin{array}{@{}rcl@{}} D[\!\boldsymbol{\xi}, \boldsymbol{\eta}]\,=\!\!\!&&\int_{a_{1}}^{b_{1}}\int_{a_{2}}^{b_{2}}\cdots\int_{a_{m}}^{b_{m}}T(\text{Cr}\{\xi_{1}=x_{1},\xi_{2}=x_{2},\cdots,\xi_{m}=x_{m}\},\\ &&\text{Cr}\{\eta_{1}=x_{1},\eta_{2}=x_{2},\cdots,\eta_{m}=x_{m}\})\mathrm{d}x_{1}\mathrm{d}x_{2}\cdots\mathrm{d}x_{m}. \end{array}$$

### Remark3.2.

It is clear that T(s,t) is a function from [ 0,1]Ã—[ 0,1] to [ 0,+âˆž). Please also mention that:

$$T(s,0)=\left\{\begin{array}{cc} 0, &\text{if} \ s=0\\ +\infty, &\text{if} \ s>0, \end{array} \right. \ \ \ \ \ \ \ \ \ \ T(s,1)= \left\{ \begin{array}{cc} 0, &\text{if} \ s=1\\ +\infty, &\text{if} \ s<1. \end{array} \right.$$

In addition, it is easy to prove that:

$$\;\,\frac{\partial T}{\partial s}=\ln\left(\frac{s}{t}\right)-\ln\left(\frac{1-s}{1-t}\right),\ \frac{\partial T}{\partial t}=\frac{t-s}{t(1-t)},$$
$$\frac{\partial^{2} T}{\partial s^{2}}=\frac{1}{s(1-s)}, \ \frac{\partial^{2} T}{\partial s\partial t}=\frac{\partial^{2} T}{\partial t\partial s}=-\frac{1}{t(1-t)},\ \frac{\partial^{2} T}{\partial t^{2}}=\frac{s}{t^{2}}+\frac{1-s}{(1-t)^{2}}.$$

Then, the following properties about T(s,t) can be easily proved: (a) T(s,t) is strictly convex with respect to (s,t) and attains its minimum value zero on the line s=t; and (b) for any 0â‰¤sâ‰¤1 and 0â‰¤tâ‰¤1, we have T(s,t)=T(1âˆ’s,1âˆ’t).

### Theorem3.1.

For any fuzzy variables Î¾ and Î·, we have D[ Î¾;Î·]â‰¥0, and the equality holds if and only if Î¾ and Î· have the same membership function.

### Proof.

Let Î¼ and Î½ be the membership functions of discrete fuzzy variables Î¾ and Î·, respectively. Since T(s,t) is strictly convex about (s,t) and attains its minimum value zero on the line s=t, we have T(Cr{Î¾=x i },Cr{Î·=x i })â‰¥0 for all i, which implies that:

$$D[\!\xi;\eta]=\sum\limits_{i=1}^{\infty}T(\text{Cr}\{\xi=x_{i}\},\text{Cr}\{\eta=x_{i}\})\geq 0.$$

Furthermore, for any 0â‰¤s âˆ—â‰¤1, the unique minimum point of T(s âˆ—,t) is t=s âˆ—. Thus, we have D[ Î¾;Î·]=0 if and only if T(Cr{Î¾=x i },Cr{Î·=x i })=0, that is:

$$\mu(x_{i})=(2\text{Cr}\{\xi=x_{i}\})\wedge 1=(2\text{Cr}\{\eta=x_{i}\})\wedge1=\nu(x_{i})$$

for all i=1,2,â‹¯,n. If Î¾ and Î· are continuous fuzzy variables, the theorem can be proved in a similar way. The proof is complete. â–¡

### Theorem3.2.

Let Ï„ be the equipossible fuzzy variable with membership function Î½(x i )=1 for all i=1,2,â‹¯,n. Then, for any discrete fuzzy variable Î¾ taking values in {x 1,x 2,â‹¯,x n }, we have:

$$D[\!\xi,\tau]=n\ln2-H[\!\xi].$$

### Proof.

According to the credibility inversion theorem, it is easy to prove that Cr{Ï„=x i }=0.5 for all i=1,2,â‹¯,n. It follows from the definition of cross-entropy that D[ Î¾,Ï„] is:

$$\begin{array}{@{}rcl@{}} &&\sum\limits_{i=1}^{n}\text{Cr}\{\xi=x_{i}\}\ln(2\text{Cr}\{\xi=x_{i}\})+\left(1-\text{Cr}\{\xi=x_{i}\}\right)\ln(2-2\text{Cr}\{\xi=x_{i}\})\\ =&&\sum\limits_{i=1}^{n}\ln2+\text{Cr}\{\xi=x_{i}\}\ln\text{Cr}\{\xi=x_{i}\}+\left(1-\text{Cr}\{\xi=x_{i}\}\right)\ln(1-\text{Cr}\{\xi=x_{i}\})\\ =&&n\ln2-H[\!\xi]. \end{array}$$

The proof is complete. â–¡

### Theorem3.3.

Let Ï„ be the equipossible fuzzy variable with membership function Î½(x)=1 for all xâˆˆ[ a,b]. Then, for any continuous fuzzy variable Î¾ taking values in [ a,b], we have:

$$D[\!\xi,\tau]=(b-a)\ln2-H[\!\xi].$$

### Proof.

It follows from the definition of cross-entropy that D[ Î¾,Ï„] is:

$$\begin{array}{@{}rcl@{}} &&{\int_{a}^{b}}(\mu(x)/2)\ln\mu(x)+\left(1-\mu(x)/2\right)\ln(2-\mu(x))\mathrm{d}x\\[0.2cm] =&&{\int_{a}^{b}}\ln2+\left((\mu(x)/2)\ln\left(\mu(x)/2\right)+\left(1-\mu(x)/2\right)\ln\left(1-\mu(x)/2\right)\right)\mathrm{d}x\\[0.2cm] =&&(b-a)\ln2-H[\!\xi]. \end{array}$$

The proof is complete. â–¡

## Minimum cross-entropy principle

In many real problems, the membership function of a fuzzy variable is unavailable except some partial information, for example, moment constraints, which may be based on observations. In this case, the maximum entropy principle (Li and Liu [9]) tells us that out of all the membership functions satisfying given constraints, choose the one that has maximum entropy. However, there may be another type of information, for example, a priori membership function, which may be based on intuition or experience with the problem. If both the a priori membership function and the moment constraints are given, which membership function should we choose? The following minimum cross-entropy principle tells us that out of all membership functions satisfying given moment constraints, choose the one that is closest to the given a priori membership function.

There is nothing mysterious about this principle. It is just based on common sense. Our membership function must be consistent with observations or given information, and if there are many membership functions consistent with the given information, we must choose the one that is nearest to our intuition and experience. On the other hand, if we have no a priori experience or intuition to guide us, we choose the membership function that is nearest to the equipossible one. In this sense, if the a priori membership function is not prescribed and the fuzzy variable is simple (bounded for continuous case), the maximum entropy principle and minimum cross-entropy principle are consistent because:

$$D[\!\xi;\upsilon]=\max_{\eta}D[\!\eta;\upsilon]-H[\!\xi]$$

where Ï… is the equipossible fuzzy variable.

## Conclusion

Based on credibility measure, a definition of cross-entropy was proposed in this paper to measure the divergence of fuzzy variables from a priori one, and some properties were investigated. Furthermore, a minimum cross-entropy principle was proposed as an important entropy optimization principle.

## References

1. Zadeh, LA: Probability measures of fuzzy events. J. Math. Anal. Appl. 23, 421â€“427 (1968).

2. De Luca, A, Termini, S: A definition of nonprobabilistic entropy in the setting of fuzzy sets theory. Inf. Control. 20, 301â€“312 (1972).

3. Yager, RR: On measures of fuzziness and negation, part I: membership in the unit interval. Int. J. General Syst. 5, 221â€“229 (1979).

4. Kaufmann, A: Introduction to the Theory of Fuzzy Subsets. Academic Press, New York (1975).

5. Kosko, B: Fuzzy entropy and conditioning. Inf. Sci. 40, 165â€“174 (1986).

6. Pal, NR, Pal, SK: Higher order fuzzy entropy and hybrid entropy of a set. Inf. Sci. 61, 211â€“231 (1992).

7. Liu, B: A survey of entropy of fuzzy variables. J. Uncertain Syst. 1(1), 4â€“13 (2007).

8. Li, P, Liu, B: Entropy of credibility distributions for fuzzy variables. IEEE Trans. Fuzzy Syst. 16(1), 123â€“129 (2008).

9. Li, X, Liu, B: Maximum entropy principle for fuzzy variables. Int. J. Uncertainty Fuzziness Knowledge-Based Syst. 15(Supp 2), 40â€“48 (2007).

10. Liu, B: Uncertainty Theory. Springer-Verlag, Berlin (2004).

11. Liu, B, Liu, YK: Expected value of fuzzy variable and fuzzy expected value models. IEEE Trans. Fuzzy Syst. 10(4), 445â€“450 (2002).

12. Li, X, Liu, B: A sufficient and necessary condition for credibility measures. Int. J. Uncertainty Fuzziness Knowledge-Based Syst. 14(5), 527â€“535 (2006).

## Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 71371027) and Program for New Century Excellent Talents in University under Grant No. NCET-13-0649.

## Author information

Authors

### Corresponding author

Correspondence to Xiang Li.

## Rights and permissions

Reprints and permissions

Li, X. Fuzzy cross-entropy. J. Uncertain. Anal. Appl. 3, 2 (2015). https://doi.org/10.1186/s40467-015-0029-5