Skip to content

Commit 04254c2

Browse files
Demoren1Demoren1
committed
add draft of introduction
1 parent 4761dbc commit 04254c2

File tree

5 files changed

+211
-46
lines changed

5 files changed

+211
-46
lines changed

‎linkreview.md

Lines changed: 25 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,14 @@
11
# other NAS
2-
| Title | Year | Source | Description |
3-
| :---- | ---: | :----- | :---------- |
2+
| Title | Year | Source | Description |
3+
| :----------------------------------------------------- | ---: | :--------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------- |
4+
| Neural Architecture Search with Reinforcement Learning | 2014 | [paper](https://arxiv.org/abs/1611.01578) | Поиск архитектуры сети с использованием обучения с подкреплением с основой на RNN |
5+
| Handbook of Evolutionary Computation | 1997 | [paper](https://www.taylorfrancis.com/books/edit/10.1201/9780367802486/handbook-evolutionary-computation-fogel-michalewicz-thomas-baeck) | --- |
46

57

68
# NES
7-
| Title | Year | Source | Description |
8-
| :---- | ---: | :----- | :---------- |
9+
| Title | Year | Source | Description |
10+
| :------------------------------------------------------------------ | ---: | :------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------- |
11+
| Neural Ensemble Search for Uncertainty Estimation and Dataset Shift | 2021 | [paper](https://proceedings.neurips.cc/paper_files/paper/2021/file/41a6fd31aa2e75c3c6d427db3d17e Представлены два метода построения ансамбля нейронных моделей в случае сдвига в данных, также есть подробный обзор статей посвященных NES ы ы ы ы ы ы ы ы |
912

1013
# ENAS
1114
| Title | Year | Source | Description |
@@ -17,22 +20,24 @@
1720

1821
# Surrogate assisted ENAS
1922

20-
| Title | Year | Source | Description |
21-
| :------------------------------------------------------------------------------------------------------------------------------------------ | ---: | :--------------------------------------------------------------------------------- | :---------- |
22-
| Similarity surrogate-assisted evolutionary neural architecture search with dual encoding strategy | 2024 | [paper](https://www.aimspress.com/aimspress-data/era/2024/2/PDF/era-32-02-050.pdf) | - |
23-
| Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor | 2019 | [paper](https://ieeexplore.ieee.org/abstract/document/8744404) | - |
24-
| A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions | 2021 | [paper](https://arxiv.org/pdf/2006.02903) | - |
25-
| NSgGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search | 2020 | [paper](https://arxiv.org/pdf/2007.10396) | - |
26-
| MoSegNAS: Surrogate-assisted Multi-objective Neural Architecture Search for Real-time Semantic Segmentation | 2022 | [paper](https://arxiv.org/pdf/2208.06820) | - |
27-
| Convolutional Neural Networks based Lung Nodule Classification: A Surrogate-Assisted Evolutionary Algorithm for Hyperparameter Optimization | 2015 | [paper](https://shiruipan.github.io/publication/tevc-21-zhang/tevc-21-zhang.pdf) | - |
28-
| A Survey on Evolutionary Neural Architecture Search | 2022 | [paper](https://arxiv.org/pdf/2008.10937) | - |
23+
| Title | Year | Source | Description |
24+
| :------------------------------------------------------------------------------------------------------------------------------------------ | ---: | :--------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------- |
25+
| Similarity surrogate-assisted evolutionary neural architecture search with dual encoding strategy | 2024 | [paper](https://www.aimspress.com/aimspress-data/era/2024/2/PDF/era-32-02-050.pdf) | Представлен алгоритм, оценивающий схожесть архитектуры, но только для одной функции |
26+
| Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor | 2019 | [paper](https://ieeexplore.ieee.org/abstract/document/8744404) | - |
27+
| A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions | 2021 | [paper](https://arxiv.org/pdf/2006.02903) | - |
28+
| NSgGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search | 2020 | [paper](https://arxiv.org/pdf/2007.10396) | - |
29+
| MoSegNAS: Surrogate-assisted Multi-objective Neural Architecture Search for Real-time Semantic Segmentation | 2022 | [paper](https://arxiv.org/pdf/2208.06820) | - |
30+
| Convolutional Neural Networks based Lung Nodule Classification: A Surrogate-Assisted Evolutionary Algorithm for Hyperparameter Optimization | 2015 | [paper](https://shiruipan.github.io/publication/tevc-21-zhang/tevc-21-zhang.pdf) | - |
31+
| A Survey on Evolutionary Neural Architecture Search | 2022 | [paper](https://arxiv.org/pdf/2008.10937) | Обзор основных алгоритмов поиска архитектур нейронных сетей. |
2932

3033
# Others
3134

32-
| Title | Year | Source | Description |
33-
| :----------------------------------------------------------------------------------------- | ---: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------- |
34-
| DARTS: Differentiable Architecture Search | 2020 | [paper](https://arxiv.org/abs/1806.09055) | - |
35-
| Brain tumor detection using CNN, AlexNet & GoogLeNet ensembling learning approaches | 2023 | [paper](https://www.researchgate.net/publication/369308467_Brain_tumor_detection_using_CNN_AlexNet_GoogLeNet_ensembling_learning_approaches) | - |
36-
| Combining global and local surrogate models to accelerate evolutionary optimization | 2006 | [paper](https://www.researchgate.net/publication/3421747_Combining_global_and_local_surrogate_models_to_accelerate_evolutionary_optimization_IEEE_Trans_Syst_Man_Cybern_Part_C_Appl_Rev) | - |
37-
| A Density-Based Algorithm for Discovering Clusters<br>in Large Spatial Databaseswith Noise | 1996 | [paper](https://cdn.aaai.org/KDD/1996/KDD96-037.pdf) | DBSCAN original paper |
38-
| Adam: A Method for Stochastic Optimization | 2014 | [paper](https://arxiv.org/abs/1412.6980) | Adam original paper |
35+
| Title | Year | Source | Description |
36+
| :--------------------------------------------------------------------------------------------- | ---: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------- |
37+
| DARTS: Differentiable Architecture Search | 2020 | [paper](https://arxiv.org/abs/1806.09055) | - |
38+
| Brain tumor detection using CNN, AlexNet & GoogLeNet ensembling learning approaches | 2023 | [paper](https://www.researchgate.net/publication/369308467_Brain_tumor_detection_using_CNN_AlexNet_GoogLeNet_ensembling_learning_approaches) | - |
39+
| Combining global and local surrogate models to accelerate evolutionary optimization | 2006 | [paper](https://www.researchgate.net/publication/3421747_Combining_global_and_local_surrogate_models_to_accelerate_evolutionary_optimization_IEEE_Trans_Syst_Man_Cybern_Part_C_Appl_Rev) | - |
40+
| A Density-Based Algorithm for Discovering Clusters<br>in Large Spatial Databaseswith Noise | 1996 | [paper](https://cdn.aaai.org/KDD/1996/KDD96-037.pdf) | DBSCAN original paper |
41+
| Adam: A Method for Stochastic Optimization | 2014 | [paper](https://arxiv.org/abs/1412.6980) | Adam original paper |
42+
| Ensemble Classification and Regression-Recent Developments, Applications and Future Directions | 2016 | [paper](https://www.researchgate.net/profile/Le-Zhang-61/publication/290476291_Ensemble_Classification_and_Regression-Recent_Developments_Applications_and_Future_Directions_Review_Article/links/5c0a1b8fa6fdcc494fdf7e43/Ensemble-Classification-and-Regression-Recent-Developments-Applications-and-Future-Directions-Review-Article.pdf) | Рассказывается о преимуществах ансамблей над одиночными моделями при различном использовании. |
43+
| Semi-Supervised Classification with Graph Convolutional Networks | 2016 | [paper](https://arxiv.org/abs/1609.02907) | Оригинальная статья посвященная GCN |

‎paper/m1p-1.pdf

-92.4 KB
Binary file not shown.

‎paper/main.tex

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
\usepackage[numbers,square,sort&compress]{natbib}
1010
\usepackage{algorithm}
1111
\usepackage[utf8]{inputenc} % allow utf-8 input
12-
% \usepackage[english]{babel} % локализация и переносы
12+
% \usepackage[english, russian]{babel} % локализация и переносы
1313
% \usepackage[russian]{babel} % локализация и переносы
1414
%\usepackage[T1]{fontenc} % use 8-bit T1 fonts
1515
\usepackage{booktabs} % professional-quality tables
@@ -104,7 +104,19 @@
104104

105105
\section{Introduction}
106106

107+
Neural network ensembles often demonstrate better generalization ability compared to single models, especially in classification and regression tasks \cite{E_Ren_2016, Hansen1990}. However, the key factor for a successful ensemble is not only the number of models but also their architectural diversity and ability to complement each other. Selecting an optimal architecture for even a single model is a challenging task, particularly when considering data-specific constraints and computational limitations \cite{B_Swarup_2023}.
107108

109+
One approach to automating ensemble construction is Neural Ensemble Search (NES) \cite{Zaidi2021}, which aims to find the optimal combination of neural networks. NES, in turn, relies on Neural Architecture Search (NAS) methods, which are extensively studied and applied to search for individual neural network architectures \cite{Zoph2017, Baeck2018, Liu2023}. Unlike traditional NAS, which focuses on finding a single model, NES is designed to efficiently combine multiple networks into an ensemble.
110+
111+
Modern NAS methods widely use surrogate functions to estimate architecture quality without requiring full model training \cite{Lu2022, Lu2020}. These functions significantly reduce computational costs, which is particularly important when searching for an optimal ensemble. For example, in \cite{Lu2022}, evolutionary algorithms were proposed in combination with surrogate models.
112+
113+
In this work, we propose a method for constructing neural network ensembles using a surrogate function that accounts for both model classification accuracy and architectural diversity. Diversity is crucial because ensembles consisting of similar models often fail to provide a significant performance gain. To achieve this, we encode architectures and their predictions on the CIFAR-10 dataset into a latent space \cite{S_Xue_2024}. Based on the encoded dataset, we train a Graph Convolutional Network (GCN) \cite{Kipf2017}. We claim that ensembles constructed in this manner achieve higher accuracy compared to one-shot models, such as DARTS \cite{Liu2018}, or single models.
114+
115+
Main Contributions:
116+
117+
1) We adapt surrogate functions for ensemble construction, taking into account both predictive performance and architectural diversity.
118+
119+
2) We propose a method for encoding the DARTS search space into a representation suitable for training a Graph Convolutional Network (GCN), where graph nodes correspond to operations within the network.
108120

109121
\bibliographystyle{unsrtnat}
110122

‎paper/references.bib

Lines changed: 0 additions & 25 deletions
This file was deleted.

0 commit comments

Comments
 (0)