Scientific Publications
Speeding‐up pruning for Artificial Neural Networks: Introducing Accelerated Iterative Magnitude Pruning
Abstract: In recent years, Artificial Neural Networks (ANNs) pruning has become the focal point of many researches, due to the extreme overparametrization of such models. This has urged the scientific world to investigate methods for the simplification of the structure of weights in ANNs, mainly in an effort to reduce time for both training and inference. Frankle and Carbin [1], and later Renda, Frankle, and Carbin [2] introduced and refined an iterative pruning method which is able to effectively prune the network of a great portion of its parameters with little to no loss in performance. On the downside, this method requires a large amount of time for its application, since, for each iteration, the network has to be trained for (almost) the same amount of epochs of the unpruned network. In this work, we show that, for a limited setting, if targeting high overall sparsity rates, this time can be effectively reduced for each iteration, save for the last one, by more than 50%, while yielding a final product (i.e., final pruned network) whose performance is comparable to the ANN obtained using the existing method. Authors: Marco Zullich, Eric Medvet; Felice Andrea Pellegrino; Alessio Ansuini Journal: 2020 25th International Conference on Pattern Recognition (ICPR) Publication date: 05/05/2021 Consult the publication
SOMEnv: An R package for mining environmental monitoring datasets by Self‐Organizing Map and k‐means algorithms with a graphical user interface
Abstract: The Self-Organizing Map (SOM) algorithm belongs to the family of artificial neural networks. It is an unsupervised method that requires no a priori knowledge regarding experimental data classification. Further, it can deal with large datasets and non-linear problems, providing powerful visualization features for outcome exploration on 2D maps. For environmental pollution assessments other unsupervised techniques are widely used, such as principal component and hierarchical cluster analyses, but their application for mining large datasets and properly visualizing the results is limited, making them difficult to use for handling of large datasets obtained by high frequency environmental monitoring. This study presents an R package (SOMEnv) that allows non-expert users to elaborate by SOM algorithm environmental variables (pollutants and/or chemical physical properties) recorded with high frequency for a long monitoring period. Additionally, SOMEnv can also be used for elaborating small datasets derived from uneven sampling. All the calculations and outcome visualizations can be done using a graphical user interface (GUI), meaning that experience in R software coding is not necessary, and only a basic knowledge regarding the employed algorithm is needed to interpret the results. The benefits of the SOMEnv package are that (i) both the software environment and tool are freely available; (ii) it is able to handle large datasets; (iii) it provides heuristic rules for SOM initialization; (iv) it has a built-in GUI for performing calculations and visualizing the results. Moreover, it comes with a wide range of visualizations, several of which are dedicated to high frequency data monitoring. An example of application is presented. The package is freely available on the Comprehensive R Archive Network (CRAN) repository. Authors Sabina Licen, Marco Franzon, Tommaso Rodani, Pierluigi Barbieri Journal Microchemical Journal Publication date 31/12/2020 Consult the pubblication
Investigating Similarity Metrics for Convolutional Neural Networks in the Case of Unstructured Pruning
Abstract: Deep Neural Networks (DNNs) are essential tools of modern science and technology. The current lack of explainability of their inner workings and of principled ways to tame their architectural complexity triggered a lot of research in recent years. There is hope that, by making sense of representations in their hidden layers, we could collect insights on how to reduce model complexity—without performance degradation—by pruning useless connections. It is natural then to ask the following question: how similar are representations in pruned and unpruned models? Even small insights could help in finding principled ways to design good lightweight models, enabling significant savings of computation, memory, time and energy. In this work, we investigate empirically this problem on a wide spectrum of similarity measures, network architectures and datasets. We find that the results depend critically on the similarity measure used and we discuss briefly the origin of these differences, concluding that further investigations are required in order to make substantial advances. Authors: Alessio Ansuini, Eric Medvet, Felice Andrea Pellegrino, Marco Zullich Journal: International Conference on Pattern Recognition Applications and Methods (ICPRAM) Publication date: 23/12/2020 Consult the publication
Hierarchical nucleation in deep neural networks
Abstract Deep convolutional networks (DCNs) learn meaningful representations where data that share the same abstract characteristics are positioned closer and closer. Understanding these representations and how they are generated is of unquestioned practical and theoretical interest. In this work we study the evolution of the probability density of the ImageNet dataset across the hidden layers in some stateof-the-art DCNs. We find that the initial layers generate a unimodal probability density getting rid of any structure irrelevant for classification. In subsequent layers density peaks arise in a hierarchical fashion that mirrors the semantic hierarchy of the concepts. Density peaks corresponding to single categories appear only close to the output and via a very sharp transition which resembles the nucleation process of a heterogeneous liquid. This process leaves a footprint in the probability density of the output layer where the topography of the peaks allows reconstructing the semantic relationships of the categories. Authors Diego Doimo, Aldo Glielmo, Alessio Ansuini, Alessandro Laio Journal NIPS’20: Proceedings of the 34th International Conference on Neural Information Processing Systems Publication date 06/12/2020 Consult the publication