New Approach to Fisher Information

Author
Mehdi Shams
University of Kashan
Abstract
Fisher information is a measure of the information inside a random variable about an unknown parameter. Mutual information shows the dependence between two variables and relative entropy shows the difference between the two probability distributions. In this paper, Fisher information is generalized for mutual information and relative entropy and the monotonicity properties of Fisher information are examined. Then concepts such as information correlation and information correlation coefficient are introduced. Finally, it is shown that Shannon differential entropy, which measures the behavior of a random variable, and conditional Fisher information are used to determine the probability of estimation error.
Keywords
Shannon differential entropy
Fisher information
Markov chains
CramerRao inequality
Basu, D. ( 1955), On Statistics Independent of a Complete Sufficient Statistic, Sankhya, 15, 377–380.

Bercher, J. F. (2013), Some properties of generalized Fisher information in the context of nonextensive thermostatistics, Physica A: Statistical Mechanics and its

Applications, 392, 3140–3154.

Bobkov, S. G., Chistyakov, G. P. and Gotze, F. (2014), Fisher information and the central limit theorem, Probab. Theory Relat. Fields, 159, 1–59.

Boekee, D. E. (1974), Generalized Fisher information with application to estimation problems, IFAC Proceedings, 10, 75–82.

Bouchon, B. and Pessoa, F. (1985), Logarithmic information of degree q linked with an extension of Fisher’s information, Kybernetika, 21, 346–359.

Cianchi, A., Lutwak, E., Yang, D. and Zhang, G. (2013), A unified approach to Cramér–Rao inequalities, IEEE Trans. Inf. Theory, 60, 643–650.

Cohen, M. (1968), The Fisher Information and Convexity, IEEE Trans. Inf. Theory, 14, 591–592.

Cover, T. and Thomas, J. (2006), Elements of Information Theory, John Wiley and Sons, Inc.: Hoboken, NJ, USA.

Fisher, R. (1925), Theory of Statistical Estimation, Proc. Camb. Philos. Soc. 22, 700–725.

Flego, S., Olivares, F., Plastino, A. and Casas, M. (2011), Extreme Fisher Information, Non-Equilibrium Thermodynamics and Reciprocity Relations, Entropy, 13, 184–194.

Frieden, B.R. (2004), Science from Fisher Information: A Unification, Cambridge University Press: Cambridge, UK.

Frieden, B.R. and Gatenby, R.A. (2013), Principle of maximum Fisher information from Hardy’s axioms applied to statistical systems, Phys. Rev. E, 88, 1–6.

Frieden, B.R. and Petri, M. (2012), Motion-dependent levels of order in a relativistic universe, Phys. Rev. E, 86, 1–5.

Frieden, B.R., Plastino, A., Plastino, A.R. and Soffer, B.H. (1999), Fisher-Based Thermodynamics: Its Legendre Transform and Concavity Properties, Phys. Rev. E, 60, 48–53.

Frieden, B.R., Plastino, A., Plastino, A.R. and Soffer, B.H. (2002), Non-equilibrium thermodynamics and Fisher information: An illustrative example, Phys. Lett. A, 304, 73–78.

Guo, D. (2009), Relative Entropy and Score Function: New Information-Estimation Relationships through Arbitrary Additive Perturbation, In Proceedings of the IEEE International Symposium on Information Theory, Seoul, Korea, 28 June–3 July 2009; 814–818.

Hirata, M., Nemoto, A. and Yoshida, H. (2012), An integral representation of the relative entropy, Entropy, 14, 1469–1477.

Hogg, R.V. and Craig, A.T. (1995), Introduction to Mathematical Statistics, Prentice Hall: Upper Saddle River, NJ, USA.

Kullback, S. (1968), Information Theory and Statistics, Dover Publications Inc.: Mineola, NY, USA.

Kullback, S. and Leibler, R.A. (1951), On Information and Sufficiency, Ann. Math. Stat., 22, 79–86.

Langley, R. S. (2013), Probability Functionals for Self-Consistent and Invariant Inference: Entropy and Fisher Information, IEEE Trans. Inf. Theory, 59, 4397–4407.

Lehmann, E. L. and Casella, G. (1998), Theory of Point Estimation, 2nd Edition, Springer-Verlage, New York.

Luo, S. (2002), Maximum Shannon entropy, minimum Fisher information, and an elementary game, Found. Phys, 32, 1757–1772.

Lv, S. (2017), General Fisher information matrices of a random vector, Advances in Applied Mathematics, 89, 18–40.

Ly, A., Marsman, M., Verhagen, J., Grasman, R. P. and Wagenmakers, E.J. (2002), A tutorial on Fisher information, Journal of Mathematical Psychology, 80, 40–55.

Otto, F. and Villani, C. (2000), Generalization of an Inequality by Talagrand and Links with the Logarithmic Sobolev Inequality, J. Funct. Anal., 173, 361–400.

Park, C. and Basu, A. (2003), The generalized Kullback-Leibler divergence and robust inference, Journal of Statistical Computation and Simulation, 73, 311–332.

Pascanu, R. and Bengio, Y. (2014), Revisiting Natural Gradient for Deep Networks, Cornell University Library: Ithaca, NY, USA, 1–18.

Plastino, A.R. and Plastino, A. (1996), Symmetries of the Fokke-rPlanck equation and the Fisher-Frieden arrow of time, Phys. Rev. E, 54, 4423–4426.

SánchezMoreno, P., Zarzo, A. and Dehesa, J.S. (2012), Jensen divergence based on Fisher’s information, J. Phys. A: Math. Theor, 45, 125–305.

Shannon, C. A. (1948), Mathematical Theory of Communication, Bell Syst. Tech. J., 27, 379–423.

Tsallis, (1998), C. Generalized entropy-based criterion for consistent testing, Phys. Rev. , 58, 1442–1445.

Venkatesan, R.C. and Plastino, A. (2014), Legendre transform structure and extremal properties of the relative Fisher information, Phys. Lett. A, 378, 1341–1345.

Zamir, R. A. ( 1998), Proof of the Fisher Information Inequality Via a Data Processing Argument, IEEE Trans. Inf. Theory, 44, 1246–1250.

Zegers, P., Fuentes, A. and Alarcon, C. (2013), Relative Entropy Derivative Bounds, Entropy, 15, 2861–2873.