BiSHop

An end-to-end framework for deep tabular learning which leverages a sparse Hopfield model with adaptable sparsity, enhanced by column-wise and row-wise modules.

Functions

class NumEmb(torch.nn.Module)

Numerical embedding layer for tabular data using quantile-based binning.

Parameters:

  • n_num (int) - Number of numerical features.

  • emb_dim (int) - Embedding dimension (number of bins).

Methods:

  • get_bins(self, data, identifier=’num’) - Compute quantile bins from data.

  • forward(self, x) - Forward pass through numerical embedding.

  • _to(self, device) - Move model to specified device.

Input Shape:

(batch_size, n_num)

Output Shape:

(batch_size, n_num, emb_dim)

class FullEmbDropout(torch.nn.Module)

Full embedding dropout layer.

Parameters:

  • dropout (float, optional, Default is 0.1) - Dropout rate.

Input:

  • X (torch.Tensor) - Input tensor.

Output:

  • torch.Tensor - Tensor with dropout applied.

def _trunc_normal_(x, mean=0., std=1.)

Truncated normal initialization approximation.

Parameters:

  • x (Tensor) - Tensor to initialize.

  • mean (float, optional, Default is 0.) - Mean of normal distribution.

  • std (float, optional, Default is 1.) - Standard deviation.

Returns:

  • Tensor - Initialized tensor.

class _Embedding(torch.nn.Embedding)

Embedding layer with truncated normal initialization.

Parameters:

  • ni (int) - Number of input features.

  • nf (int) - Number of output features.

  • std (float, optional, Default is 0.01) - Standard deviation for initialization.

class CatEmb(torch.nn.Module)

Categorical embedding layer with sharing options.

Parameters:

  • n_cat (int) - Number of categorical features.

  • emb_dim (int) - Embedding dimension.

  • n_class (list) - List of number of classes for each categorical feature.

  • share (bool, optional, Default is True) - Whether to share embeddings.

  • share_add (bool, optional, Default is False) - Whether to add shared embeddings.

  • share_div (int, optional, Default is 8) - Division factor for shared embeddings.

  • full_dropout (bool, optional, Default is False) - Whether to use full dropout.

  • emb_dropout (float, optional, Default is 0.1) - Embedding dropout rate.

Input:

  • x (torch.Tensor) - Categorical feature tensor.

Output:

  • torch.Tensor - Embedded categorical features.

class PatchEmb(torch.nn.Module)

Patch embedding layer for transformer architectures.

Parameters:

  • patch_dim (int) - Patch dimension.

  • d_model (int) - Model dimension.

Input:

  • x (torch.Tensor) - Input tensor.

Output:

  • torch.Tensor - Patch embedded tensor.

Referencses:

Xu, C., Huang, Y.-C., Hu, J. Y.-C., Li, W., Gilani, A., Goan, H.-S., & Liu, H. (2024). BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model. In Proceedings of the 41st International Conference on Machine Learning (ICML). https://arxiv.org/abs/2404.03830