| GENERAL |
|
|
|
|
Exploring Explicit Coarse-Grained Structure in Artificial Neural Networks |
| Xi-Ci Yang1, Z. Y. Xie2*, and Xiao-Tao Yang1* |
1College of Power and Energy Engineering, Harbin Engineering University, Harbin 150001, China
2Department of Physics, Renmin University of China, Beijing 100872, China
|
|
| Cite this article: |
|
Xi-Ci Yang, Z. Y. Xie, and Xiao-Tao Yang 2023 Chin. Phys. Lett. 40 020501 |
|
|
|
|
Abstract We propose to employ a hierarchical coarse-grained structure in artificial neural networks explicitly to improve the interpretability without degrading performance. The idea has been applied in two situations. One is a neural network called TaylorNet, which aims to approximate the general mapping from input data to output result in terms of Taylor series directly, without resorting to any magic nonlinear activations. The other is a new setup for data distillation, which can perform multi-level abstraction of the input dataset and generate new data that possesses the relevant features of the original dataset and can be used as references for classification. In both the cases, the coarse-grained structure plays an important role in simplifying the network and improving both the interpretability and efficiency. The validity has been demonstrated on MNIST and CIFAR-10 datasets. Further improvement and some open questions related are also discussed.
|
|
|
Received: 05 November 2022
Published: 17 January 2023
|
|
| PACS: |
05.10.Cc
|
(Renormalization group methods)
|
| |
07.05.Mh
|
(Neural networks, fuzzy logic, artificial intelligence)
|
| |
05.10.-a
|
(Computational methods in statistical physics and nonlinear dynamics)
|
| |
31.15.aq
|
(Strongly correlated electron systems: generalized tight-binding method)
|
|
|
|
|
|
| [1] | Lei N, Luo Z, Yau S T, and Gu D X 2018 arXiv:1805.10451 [cs.LG] |
| [2] | Lei N, Su K, Yau S T, and Gu D X 2019 Comput. Aided Geom. Des. 68 1 |
| [3] | Fawzi A, Balog M, Huang A, Hubert T, Paredes B R, Barekatain M, Novikov A, Ruiz F J R, Schrittwieser J, Swirszcz G, Silver D, Hassabis D, and Kohli P 2022 Nature 610 47 |
| [4] | Gao X and Duan L M 2017 Nat. Commun. 8 662 |
| [5] | Carrasquilla J and Melko R G 2017 Nat. Phys. 13 431 |
| [6] | Wu D, Wang L, and Zhang P 2019 Phys. Rev. Lett. 122 080602 |
| [7] | Carleo G, Cirac I, Cranmer K, Daudet L, Schuld M, Tishby N, Vogt-Maranto L, and Zdeborová L 2019 Rev. Mod. Phys. 91 045002 |
| [8] | Bedolla E, Padierna L C, and Priego R C 2021 J. Phys.: Condens. Matter 33 053001 |
| [9] | Font B, Weymouth G, Nguyen V T, and Tutty O R 2021 J. Comput. Phys. 434 110199 |
| [10] | Di Sante D, Medvidovic M, Toschi A, Sangiovanni G, Franchini C, Sengupta A M, and Millis A J 2022 Phys. Rev. Lett. 129 136402 |
| [11] | Helmstaedter M, Briggman K L, Turaga S C, Jain V, Seung H S, and DenkHelmstaedter W 2013 Nature 500 168 |
| [12] | Webb S 2018 Nature 554 555 |
| [13] | Dauparas J, Anishchenko I, Bennett N, Ragotte H B R J, Milles L F, Wicky B I M, Courbet A, deHaas R J, Bethel N, Leung P J Y, Huddy T F, Pellock S, Tischer D, Chan F, Koepnick B, Nguyen H, Kang A, Sankaran B, Bera A K, King N P, and Baker D 2022 Science 378 49 |
| [14] | Stanev V, Oses C, Kusne A G, Rodriguez E, Paglione J, Curtarolo S, and Takeuchi I 2018 npj Comput. Mater. 4 29 |
| [15] | Wang A Y T, Murdock R J, Kauwe S K, Oliynyk A O, Gurlo A, Brgoch J, Persson K A, and Sparks T D 2020 Chem. Mater. 32 4954 |
| [16] | Batra R, Song L, and Ramprasad R 2021 Nat. Rev. Mater. 6 655 |
| [17] | LeCun Y, Bengio Y, and Hinton G 2015 Nature 521 436 |
| [18] | Goodfellow I, Bengio Y, and Courville A 2016 Deep Learning (Cambridge: MIT Press) |
| [19] | Smolensky P 1986 Information Processing in Dynamical Systems: Foundations of Harmony Theory (Cambridge: MIT Press) |
| [20] | Bourlard H and Kamp Y 1988 Biol. Cybern. 59 291 |
| [21] | Lecun Y, Bottou L, Bengio Y, and Haffner P 1998 Proc. IEEE 86 2278 |
| [22] | Bengio Y and Bengio S 2000 Advances in Neural Information Processing Systems 12 400 |
| [23] | Carleo G and Troyer M 2017 Science 355 602 |
| [24] | Cai Z and Liu J 2018 Phys. Rev. B 97 035116 |
| [25] | Liang X, Liu W Y, Lin P Z, Guo G C, Zhang Y S, He L X 2018 Phys. Rev. B 98 104426 |
| [26] | Nomura Y and Imada M 2021 Phys. Rev. X 11 031034 |
| [27] | Pfau D, Spencer J S, Matthews A G D G, and Foulkes W M C 2020 Phys. Rev. Res. 2 033429 |
| [28] | Hermann J, Schatzle Z, and Noe F 2020 Nat. Chem. 12 891 |
| [29] | Zeng S M, Zhao Y C, Li G, Wang R R, Wang X M, and Ni J 2019 npj Comput. Mater. 5 84 |
| [30] | Konno T, Kurokawa H, Nabeshima F, Sakishita Y, Ogawa R, Hosako I, and Maeda A 2021 Phys. Rev. B 103 014509 |
| [31] | van Nieuwenburg E P L, Liu Y H, and Huber S D 2017 Nat. Phys. 13 435 |
| [32] | Wetzel S J 2017 Phys. Rev. E 96 022140 |
| [33] | Valiant L G 1984 Commun. ACM 27 1134 |
| [34] | Bahri Y, Kadmon J, Pennington J, Schoenholz S S, Dickstein J S, and Ganguli S 2020 Annu. Rev. Condens. Matter Phys. 11 501 |
| [35] | Hibat-Allah M, Inack E M, Wiersema R, Melko R G, and Carrasquilla J 2021 Nat. Mach. Intell. 3 952 |
| [36] | Karniadakis G E, Kevrekidis I G, Lu L, Perdikaris P, Wang S, and Yang L 2021 Nat. Rev. Phys. 3 422 |
| [37] | Stoudenmire E and Schwab D J 2016 arXiv:1605.05775 [stat.ML] |
| [38] | Glasser I, Pancotti N, and Cirac J I 2020 IEEE Access 8 68169 |
| [39] | Cheng S, Wang L, and Zhang P 2021 Phys. Rev. B 103 125117 |
| [40] | Han Z Y, Wang J, Fan H, Wang L, and Zhang P 2018 Phys. Rev. X 8 031012 |
| [41] | Cheng S, Wang L, Xiang T, and Zhang P 2019 Phys. Rev. B 99 155131 |
| [42] | Vieijra T, Vanderstraeten L, and Verstraete F 2022 arXiv:2202.08177 [quant-ph] |
| [43] | Liu D, Ran S J, Wittek P, Peng C, Garcia R B, Su G, and Lewenstein M 2019 New J. Phys. 21 073059 |
| [44] | Gao Z F, Cheng S, He R Q, Xie Z Y, Zhao H H, Lu Z Y, and Xiang T 2020 Phys. Rev. Res. 2 023300 |
| [45] | Žunkovič B 2022 Quantum Mach. Intell. 4 21 |
| [46] | Saremi S and Sejnowski T J 2013 Proc. Natl. Acad. Sci. USA 110 3071 |
| [47] | Beny C 2013 arXiv:1301.3124 [quant-ph] |
| [48] | Lin H W, Tegmark M, and Rolnick D 2017 J. Stat. Phys. 168 1223 |
| [49] | Mehta P and Schwab D J 2014 arXiv:1410.3831 [stat.ML] Schwab D J and Mehta P 2016 arXiv:1609.03541 [cond-mat.dis-nn] |
| [50] | Koch-Janusz M and Ringel Z 2018 Nat. Phys. 14 578 |
| [51] | Li S H and Wang L 2018 Phys. Rev. Lett. 121 260601 |
| [52] | De Koch L M, De Koch R M, and Cheng L 2020 IEEE Access 8 106487 |
| [53] | Cardy J 1996 Scaling and Renormalization in Statistical Physics (Cambridge: Cambridge University Press) |
| Kardar M 2007 Statistical Physics of Fields (Cambridge: Cambridge University Press) |
| [54] | Levin M and Nave C P 2007 Phys. Rev. Lett. 99 120601 |
| [55] | Xie Z Y, Jiang H C, Chen Q N, Weng Z Y, and Xiang T 2009 Phys. Rev. Lett. 103 160601 |
| Zhao H H, Xie Z Y, Chen Q N, Wei Z C, Cai J W, and Xiang T 2010 Phys. Rev. B 81 174411 |
| [56] | Xie Z Y, Chen J, Qin M P, Zhu J W, Yang L P, and Xiang T 2012 Phys. Rev. B 86 045139 |
| [57] | Efrati E, Wang Z, Kolan A, Kadanoff L P 2014 Rev. Mod. Phys. 86 647 |
| [58] | Meurice Y, Sakai R, and Yockey J U 2022 Rev. Mod. Phys. 94 025005 |
| [59] | Chen B B, Gao Y, Guo Y B, Liu Y Z, Zhao H H, Liao H J, Wang L, Xiang T, Li W, and Xie Z Y 2020 Phys. Rev. B 101 220409(R) |
| [60] | Hinton G, Vinyals O, and Dean J 2015 arXiv:1503.02531 [stat.ML] |
| [61] | Wang T, Zhu J Y, Torralba A, and Efros A A 2020 arXiv:1811.10959v3 [cs.LG] |
| [62] | Zhao B, Mopuri K R, and Bilen H 2021 arXiv:2006.05929 [cs.CV] |
| [63] | Zhao B and Bilen H 2022 arXiv:2110.04181v3 [cs.LG] |
| [64] | The official website of MNIST is available at http://yann.lecun.com/exdb/mnist |
| [65] | The official website of CIFAR-10 is available at https://www.cs.toronto.edu/ kriz/cifar.html |
| [66] | Gell-Mann M and Low F E 1954 Phys. Rev. 95 1300 |
| [67] | Kadanoff L P 1966 Phys. Phys. Fiz. 2 263 |
| [68] | Wilson K G 1971 Phys. Rev. B 4 3174 |
| Wilson K G 1971 Phys. Rev. B 4 3184 |
| [69] | Wilson K G 1975 Rev. Mod. Phys. 47 773 |
| [70] | Kadanoff L P 1975 Phys. Rev. Lett. 34 1005 |
| [71] | Swendsen R H 1979 Phys. Rev. Lett. 42 859 |
| [72] | White S R 1992 Phys. Rev. Lett. 69 2863 |
| [73] | Wetterich C 1993 Phys. Lett. B 301 90 |
| [74] | Verstraete F, García-Ripoll J J, and Cirac J I 2004 Phys. Rev. Lett. 93 207204 |
| [75] | Pirvu B, Murg V, Cirac J I, and Verstraete F 2010 New J. Phys. 12 025012 |
| [76] | Bray J W and Chui S T 1979 Phys. Rev. B 19 4876 |
| [77] | Pan C Y and Chen X Y 1987 Phys. Rev. B 36 8600 |
| [78] | Kovarik M D 1990 Phys. Rev. B 41 6889 |
| [79] | Yang L P, Liu Y Z, Zou H Y, Xie Z Y, and Meurice Y 2016 Phys. Rev. E 93 012138 |
| [80] | Chen B B, Chen L, Chen Z, Li W, and Weichselbaum A 2018 Phys. Rev. X 8 031082 |
| [81] | Hornik K, Stinchcombe M, and White H 1989 Neural Networks 2 359 |
| [82] | Cybenko G 1989 Math. Control Signals Syst. 2 303 |
| [83] | He K M, Zhang X Y, Ren S Q, and Sun J 2016 Proc. IEEE Conference Computer Vision Pattern Recognition (Las Vegas, USA, 26 June–1 July 2016) pp 770–778 |
| [84] | Huang G, Liu Z, van der Maaten L, and Weinberge K Q 2017 Proc. IEEE Conference Computer Vision Pattern Recognition (Hawaii USA, 21–26 July 2017) pp 4700–4708 |
| [85] | Vaswani A, Shazeer N, Parmer N, Uszkoreit J, Jones L, Gomez A N, Kaiser L, and Polosukhin I 2017 arXiv:1706.03762 [cs.CL] |
| [86] | Press W H, Teukolsky S A, Vetterling W T, and Flannery B P 2007 Numerical Recipes: The Art of Scientific Computing (Cambridge: Cambridge University Press) |
| [87] | Tolstikhin I, Houlsby N, Kolesnikov A, Beyer L, Zhai X, Unterthiner T, Yung J, Steiner A, Keysers D, Uszkoreit J, Lucic M, and Dosovitskiy A 2021 arXiv:2105.01601 [cs.CV] |
| [88] | Luo W, Li Y, Urtasun R, and Zemel R 2016 Advances in Neural Information Processing Systems (Barcelona, Spain, 5–8 December 2016) |
| [89] | Wang P, Chen P, Yuan Y, Liu D, Huang Z, Hou X, and Cottrell G 2018 IEEE Winter Conference on Applications of Computer Vision (Nevada, USA, 12–15 March 2018) pp 1451–1460 |
| [90] | Deng J, Guo J, Xue N, and Zafeiriou S 2019 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (Long Beach, USA, 16–20 June 2019) pp 4690–4699 |
| [91] | Chen X, Ma Q, and Alkharobi T 2009 IEEE International Conference on Computer Science and Information Technology (Beijing, China 8–11 August 2009) pp 291–294 |
| [92] | Montavon G, Lapuschkin S, Binder A, Samek W, and Muller K R 2017 Pattern Recognit. 65 211 |
| [93] | Tong Y J, Xiong S Y, He X Z, Pan G H, and Zhu B 2021 J. Comput. Phys. 437 110325 |
| [94] | Rao C, Sun H, and Liu Y 2021 arXiv:2106.04781 [cs.LG] |
| [95] | Rendle S 2010 IEEE International Conference on Data Mining (Sydney, Australia, 13–17 December 2010) pp 995–1000 |
| [96] | Meng Y M, Zhang J, Zhang P, Gao C, and Ran S J 2021 arXiv:2012.11841v2 [cs.LG] |
| [97] | Novikov A, Trofimov M, and Oseledets I 2017 arXiv: 1605.03795v3 [stat.ML] |
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
