AttentionSmithy code co-pilot?

[csmova]

Hi,

I really like this package and am excited to use it! I'm more of an analytical chemist and am not too familiar with the technical coding details of implementing a transformer. Having this 'lego block' approach to building the models seems useful. I've seen transformers being used a lot for proteomics data, and I have a bit of intuition for the machine learning side of such applications, but actually building a transformer seems daunting.

I wanted to just quickly see what the simplest possible proteomics transformer might look like using AttentionSmithy, before I dove deeper into your repo. For example, would I still be needing to code up multiple .py files and hundreds of lines of code? Or would it just be a couple dozen?

Your code is really well structured, so I was thinking maybe ChatGPT could act as 'interactive documentation', where I could chat with it about how to implement your package. I used GitInjest to upload your repo to ChatGPT and had it suggest a basic proteomics transformer model for me. I'm posting as this might be helpful for others. You can view my full chat here

I also uploaded the .txt file from GitInjest, which is attached here, but I'm not sure how much this helped since ChatGPT had the github link anyway.
xomicsdatascience-attentionsmithy.txt

As you should be able to see in the chat link, I asked ChatGPT for some starter code for an example proteomics project. It seemed to grasp your project structure well. Here is the output, can you tell if these a sensible starting point of using AttentionSmithy?:

---

Building a Transformer for Proteomics
To use this package for proteomics, we'll need to adapt the input handling so that it works with protein sequences or numerical peptide features instead of text. Below is a step-by-step approach:

Step 1: Define a Simple Transformer Model
We'll create a transformer that takes in amino acid sequences (or other proteomics-related features) and processes them through an encoder.

import torch
import torch.nn as nn
from attention_smithy.components.Encoder import Encoder
from attention_smithy.components.EncoderLayer import EncoderLayer
from attention_smithy.components.FeedForwardNetwork import FeedForwardNetwork
from attention_smithy.components.MultiheadAttention import MultiheadAttention
from attention_smithy.numeric_embeddings.LearnedEmbedding import LearnedPositionEmbedding
from attention_smithy.attention.StandardAttentionMethod import StandardAttentionMethod

class ProteomicsTransformer(nn.Module):
    def __init__(self, input_dim, embed_dim, num_heads, ff_hidden_dim, num_layers, dropout=0.1):
        super(ProteomicsTransformer, self).__init__()

        # Numeric embedding layer for peptides/protein sequences
        self.embedding = LearnedPositionEmbedding(input_dim, embed_dim)

        # Create multi-head attention and feedforward layers
        self.attention = MultiheadAttention(embed_dim, num_heads, StandardAttentionMethod(dropout))
        self.ffn = FeedForwardNetwork(embed_dim, ff_hidden_dim, "relu", dropout)

        # Create multiple encoder layers
        self.encoder_layers = nn.ModuleList([
            EncoderLayer(embed_dim, self.attention, self.ffn, dropout)
            for _ in range(num_layers)
        ])

        # Final encoder block
        self.encoder = Encoder(self.encoder_layers)

        # Output layer for classification or regression
        self.output_layer = nn.Linear(embed_dim, 1)  # Adjust for classification

    def forward(self, x):
        x = self.embedding(x)  # Convert input to embeddings
        x = self.encoder(x, src_padding_mask=None)  # Pass through transformer encoder
        x = self.output_layer(x.mean(dim=1))  # Aggregate output
        return x

Step 2: Instantiate the Model

# Define model hyperparameters
input_dim = 20  # Example: 20 amino acids
embed_dim = 128  # Size of embeddings
num_heads = 8  # Number of attention heads
ff_hidden_dim = 256  # Hidden layer size in feedforward network
num_layers = 4  # Number of encoder layers

# Initialize model
model = ProteomicsTransformer(input_dim, embed_dim, num_heads, ff_hidden_dim, num_layers)

# Example input: batch of 10 peptide sequences (each sequence has 50 amino acid tokens)
dummy_input = torch.randint(0, input_dim, (10, 50))  # Random amino acid sequences

# Forward pass
output = model(dummy_input)
print(output.shape)  # Expected output: (10, 1)

---

Is it really this simple (assuming I had real data formatted correctly)? If so, building a custom GPT specifically for AttentionSmity might be worth it (think like a code co-pilot, but tailored with docuemntation and instructions to make AttentionSmithy implementations and codebase Q&A really simple). If this seems promising, perhaps we could work together to do this.

Happy to hear thoughts.

[CCranney]

I love this idea, thank you for sharing @csmova! This example is really close, but not quite ready - the learned embedding in particular can work, but takes the place of tokenized embeddings, which I don't think is compatible with the end goal of the model.

I'm going to work on having a few basic examples (likely simple encoder-only, decoder-only, and encoder-decoder examples) that could be used as reference for something like GitInjest so it can automate creating models like this. That should provide enough breadth of examples that an LLM should be able to implement a given model. Then making a code co-pilot for any interested parties would be a cinch.