FeaSel-Net

FeaSel-Net is a python package that enables feature selection algorithms embedded in a neural network, originally described in the 2022 paper by Fischer er al. "FeaSel-Net: A Recursive Feature Selection Callback in Neural Networks (DOI: 10.3390/make4040049)". The algorithm was only capable to be used one-dimensional datasets. This work extends the package to work with high-dimensional datasets with some optimization.

The core algorithm combines a leave-one-out cross-validation (LOOCV) approach for feature selection with recursive pruning, allowing the model to iteratively remove less relevant features while maintaining classification performance. To ensure compatibility with high-dimensional data, modifications were made to the data processing pipeline, masking operations, and the LinearPass layer. Additionally, improvements were introduced to enhance computational efficiency and integrate seamlessly with TensorFlow’s tf.data API, enabling efficient handling of large-scale datasets.

The package is an extension for the Keras and Tensorflow libraries. Please see the links for further information on their software packages and to get a grasp of neural networks in general and the Callback construct used for FeaSel-Net.

Currently, this package only supports relative imports. To use it, drag and drop the feasel_test folder into your working directory. Then, follow the example below to import and use the package.

To start using the package there are two important imports that need to be made:

from feasel_test.nn.tfcustom.layers import LinearPass
from feasel_test.nn.tfcustom.callbacks22 import FeatureSelection

or

from feasel_test.nn.tfcustom.layers import LinearPass
from feasel_test.nn.tfcustom.callbacks import FeatureSelection

if the training is being applied to a one-dimensional dataset.

These two imports enable the use of the LinearPass layer and the feature selection callback during training. Before implementation, the data must be processed into TensorFlow Datasets, and the user must define an appropriate input pipeline. Next, the model should be built, preferably using TensorFlow’s Functional API, where the LinearPass layer is added at the model’s input.

def cnn(input_shape, output_shape):
  inputs = x = tf.keras.Input(input_shape)
  x = LinearPass(name="Linear")(x)
  ...

Finally, the feature selection callback must be instantiated using the FeatureSelection class with its corresponding parameters and added to the model during training.

cb = FeatureSelection(evaluation_data=eval_ds,
                      layer_name="Linear",
                      n_classes=12,
                      samples=51,
                      evaluation_data_shape=(480, 640, 100),
                      evaluation_data_labels=labels_eval,
                      val_thresh=0.7,
                      callback={'eval_type': 'accuracy',
                                'n_features': 3,
                                'd_min': 10,
                                'n_samples': None,
                                'thresh': 0.8,
                                'pruning_type': 'exp',
                                'scale': True,
                                'pruning_rate': 0.3,
                                'remove_outliers': True,
                                'accelerate': True})

...

history = model.fit(train_ds,
                    epochs=250,
                    steps_per_epoch=len(data_paths)//batch_size,
                    shuffle=False,
                    validation_data=val_ds,
                    callbacks=[cb])

For more detailed example please refer to [vgg_wandb.ipynb].

1. Data Preparation

   • Convert raw data into TensorFlow Datasets.
   • Optional: Apply preprocessing steps, such as normalization and augmentation.
   • Ensure compatibility with FeaSel-Net by structuring the dataset correctly.

2. Model Definition

   • Define the neural network architecture using TensorFlow/Keras.
   • Integrate the LinearPass layer at the model’s input to enable feature selection.
   • Configure additional layers according to the desired architecture (e.g., CNN, ResNet, or VGG-like models).

3. Feature Selection Callback Initialization

   • Instantiate the FeatureSelection callback with appropriate parameters.
   • Specify evaluation data, pruning criteria, and thresholds.
   • Ensure that the callback is properly linked to the model training process.

4. Training the Model

   • Train the model with the FeatureSelection callback enabled.
   • Monitor performance metrics, including validation accuracy and loss.
   • Observe the feature pruning process as the model iteratively refines its feature set.

5. Optimization and Fine-Tuning

   • Adjust pruning parameters to optimize performance.
   • Experiment with different architectures and hyperparameters.
   • Ensure that the final model achieves a balance between accuracy and efficiency.

• Introducing new ungreedy algorithms for the feature selection process, e.g. genetic algorithm, simulated annealing...