In a typical machine learning(ML) work flow, we program the feature production, training, and inference. We do that mostly using frameworks to write high-level program and not have to manage the low level details required for ML or deep learning (DL). The pytorch or tensorflow (frameworks) calls the cuda if available and the operations are now performed on the GPU. DL models have achieved state-of-the-art (SOTA) performance in multiple-domains due to their hierarchial structure of parametric as well as non-parametric layers. Therefore, CUDA has to descide on how to perform the operations. Libraries like cuBLAS, cuDNN, or PyTorch’s built-in kernels are highly optimized for common operations (matrix multiply, convolutions, etc.). But if our applications have specialized algorithms, unique data layouts, and non-standard precision or formats, then CUDA might not perform well. Therefore, you write CUDA program for faster execution.
However, CUDA programming is very manual and tedious. It works on the principle of Scalar Program, Blocked threads. This means we have to define what each thread does and manage it. It is a low-level programming method. Therefore, Triton was developed to make the specialized algorithms faster and CUDA programming a little less tedious and manual. Triton is a high-level CUDA programming method. It works on the principle of Blocked Program, Scalar Threads. This means that instead of managing each thread we manage a group of threads instead. And Trition handles the actual operation based on our memory flow and our data flow and chooses the optimum way to perform the given task/operation. Making it faster for the specialized use cases.
Therefore, Triton has gained popularity and is helping researchers and developers with cuda programming.
Bridge the gap between the idealistic and the reality of what can be built.
What are the steps that is required to build it?
not clear on this part
System Design step
Core architecture and design choices
How are we going to build it?
What the book covers:
Chapter 1: Understanding
The chapter covers the following topics:
Understanding the LLM Twin concept
Planning the MVP of the LLM Twin product.
Building ML systems with feature/training/inference pipelines
Designing the system architecture of the LLM Twin
The key of the LLM Twin stands in the following:
What data we collect
How we preprocess the data
How we feed the data into the LLM
How we chain multiple prompts for the desired results
How we evaluate the generated content
We have to consider how to do the following (MLOps):
Ingest, clean, and validate fresh data
Training versus inference setups
Compute and serve features in the right environment
Serve the model in a cost-effective way
Version, track, and share the datasets and models
Monitor your infrastructure and models
Deploy the model on a scalable infrastructure
Automate the deployments and training
In every software architecture, Database->Business Logic->UI. And, any layer can be as complex as required. But for ML, what do we require? Well, that is the FTI architecture. Feature->Training->Inference.
To conclude, the most important thing you must remember about the FTI pipelines is their interface:
The feature pipeline takes in data and outputs the features and labels saved to the feature store.
The training pipeline queries the features store for features and labels and outputs a model to the model registry.
The inference pipeline uses the features from the feature store and the model from the model registry to make predictions.
Requirements of the ML system from a purely technical perspective:
Data
collect
standardize
clean the raw data
create instruct database for fine-tuning an LLM
chunk and embed the cleaned data. Store the vectorized data into a vector DB for RAG.
Training
Fine-tune LLMs of various sizes
Fine-tune on instruction datasets of multiple sizes.
Switch between LLM types
Track and compare experiments.
Test potential production LLM candidates before deploying them.
Automatically start the training when new instruction datasets are available.
Inference
A REST API interface for clients to interact with the LLM
Access to the vector DB in real time for RAG.
Inference with LLMs of various sizes
Autoscaling based on user requests
Automatically deploy the LLMs that pass the evaluation step
LLMOPs
Instruction dataset versioning, lineage, and reusability
Model versioning, lineage, and reusability
Experiment tracking
Continuous training, continuous integration, and continuous delivery (CT/CI/CD)
Prompt and system monitoring
Chapter2: Tooling and Installation
The chapter covers:
Python ecosystem and project installation
MLOps and LLMOps tooling
Databases for storing unstructured and vector data
Preparing for AWS
Any Python project needs three fundamental tools: the Python interpreter, dependency management, and a task execution tool.
Poetry is one of the most popular dependency and virtual environment managers within the Python ecosystem.
An orchestrator is a system that automates, schedules, and coordinates all your ML pipelines. It ensures that each pipeline—such as data ingestion, preprocessing, model training, and deployment—executes in the correct order and handles dependencies efficiently.
ZenML is one such orchestrator.
It orchestrates by pipelines and steps. They are just python functions. Where steps are called in pipeline functions. Modular code should be written for this.
ZenML transforms any step output into artifacts.
Any file produced during the ML lifecycle.
Experiment Tracker:
Training ML models is an entirely iterative and experimental process. Therefore, an experiment tracker is required.
CometML is one that helps us in this aspect.
Prompt monitoring
you cannot use standard logging tools as prompts are complex and unstructured chains.
Optik is simple to use prompt monitoring compared to other prompt monitoring tools.
MongoDB, NoSQL dataset.
Qdrant, vector database.
For our MVP, AWS, it’s the perfect option as it provides robust features for everything we need, such as S3 (object storage), ECR (container registry), and SageMaker (compute for training and inference).
Chapter 3: Data Engineering
In this chapter, we will study the following topics:
Designing the LLM Twin’s data collection pipeline
Implementing the LLM Twin’s data collection pipeline
Gathering raw data into the data warehouse
Collect and curate the dataset
From raw data, Extract -> Transform -> Load into MongoDB. (ETL)
crawling
standardizing data
load into data warehouse
Chapter 4: RAG Feature Pipeline
Retrieval-augmented generation (RAG)
Chapter teaches you what RAG is and how to implement it.
The main sections of this chapter are:
Understanding RAG
An overview of advanced RAG
Exploring the LLM Twin’s RAG feature pipeline architecture
Implementing the LLM Twin’s RAG feature pipeline
Chapter 5: Supervised Fine-Tuning
SFT refines the model’s capabilities (here model refers to pre-trained model that can predict the new sequence) learning to predict instruction-answer pair.
Makes the general ability of pre-trained LLMs of understanding language to specific application, or in this case more conversational.
In this chapter, the authors cover the following topics:
Creating a high-quality instruction dataset
SFT techniques
Implementing fine-tuning in practice
Chapter 6: Fine-Tuning with Preference Alignment
SFT cannot address a human’s preference of how a conversation should be, therefore we use preference alignment, specifically the Direct Preference Optimization (DPO).
Authors cover the following topics in this chapter:
Understanding preference datasets
How to create our own preference dataset
Direct preference optimization (DPO)
Implementing DPO in practice to align our model
Chapter 7: Evaluating LLMs
no unified approach to measuring a model’s performance but there are patterns and recipes that we can adapt to specific use cases.
The chapter covers:
Model evaluation
RAG evaluation
Evaluating TwinLlama-3.1-8B
Chapter 8: Inference Optimization
Some tasks like document generation take hours and some tasks like code completion take a small amount of time, this is why optimization of the inference is quite important. The things that are optimized are the latency (the speed of the generation of the first token), throughput (number of tokens generated per second), and memory footprint of the LLM.
The chapter covers:
Model optimization strategies
Model parallelism
Model quantization
Chapter 9: RAG Inference Pipeline
Where the magic happens for the RAG system.
The chapter covers the following topics:
Understanding the LLM Twin’s RAG inference pipeline
Exploring the LLM Twin’s advanced RAG techniques
Implementing the LLM Twin’s RAG inference pipeline
Chapter 10: Inference Pipeline Deployment
The chapter covers:
Criteria for choosing deployment types
Understanding inference deployment types
Monolithic versus microservices architecture in model serving
Exploring the LLM Twin’s inference pipeline deployment strategy
Deploying the LLM Twin service
Autoscaling capabilities to handle spikes in usage
Chapter 11: MLOps and LLMOps
This chapter covers:
The path to LLMOps: Understanding its roots in DevOps and MLOps
should be able to change the shape of the array dynamically.
should be able to add/delete element fast
should be able to insert/delete a element in the middle.
since we need to make this as efficient as possible. Let’s try what would we have done if we had to invent it for ourself.
First, we take the functionality of it and try to simplify it as much as possible.
Here, let’s take only the ‘adding dynamically’ part.
Adding Dynamically
say we have an fixed array of 4 elements. Then, how can we make it such that we can add an element to it.
Here, we know that we need to describe on a fixed space required for our task before hand to utilize a memory (refer to how memory works).
Alternative #1: Make an array of 5 element then copy all the data to the new array.
Here, using this we can make an dynamic array. However, it is very expensive to do this for huge amount of data.
For example, for an 1M length array, we need to perform around 90 billion copies.
Here lets assume we are continuously adding element to the array. So, for 5th element we need 5 copying operation. For the 6th element, we need to first create a new array of size 6 and copy the 6 elements. So here our total operations is 5+6. For the 7th element, it is 5+6+7. In big O notation it is $O(N^2)$.
Alternative #2: Making a new array of size of the fixed array + 8 (say).
Here, it will reduce a lot of copying operations, however it is still of $O(N^2)$ complexity.
Alternative #3: Making the new array the double size of the array.
Here, the number of copying operation needed for an array of size N is always N. So it’s complexity is $O(N)$.
This is very cool problem, so if you are math savvy then take out a piece of paper and do the math, it is quite fun to think about this problem. Find how this is $O(N)$.
This is how programming languages define dynamic arrays.
For deletion, say if the filled element is less that the half of the size then we reduce the size of array by half. Here, this means our memory usage has also been optimized.
Similarly, with this way we can also easily perform insertions and deletion from the middle or front of the array with high speed .
The foundation of a successful company and the most important thing for a business is it’s business strategy. Business strategy is the game plan you employ to create a successful business.
To learn how to create a great business plan is to study the business plan of the multi-billion dollar companies.
And the most efficient way to study other business plan, is to use the “Porter’s Five Forces”
1.1. Porter’s Five Forces
The five forces are:
Competition: How competitive is your market?
Substitute : How many substitute are there in the market?
New Entry: How susceptible is your product to a new entry.
Buyer's Power: How much the buyer have power over you?
Supply Chain : How much of the supply chain you control?
To understand this, John Ha, the maker of the video takes the example of Apple.
Apple has huge competition, in the phone market, the laptop market, the PC market and other thing in the name of Samsung, Microsoft, etc. Additionally, there are many substitute to apple products. Here, from the first two we can clearly analyze that Apple should have a hard time. However, they do not. In fact, they thrive in this environment. But why?
The answer is their Ecosystem-lock. Once you are in the apple eco-system, it is very hard to get out of it. Therefore, even if a new product is launched, the tendency of the apple user to choose a different company is pretty low. This means that Apple is not at all susceptible to new products. It does not matter if your first apple device is an iPhone, a MacBook, or an iPad, it is most likely the first apple product of many that you are going to own. And that is because of the ecosystem that Apple has built.
In addition to this, due to this ecosystem people tend to buy Apple. For example if there is a group in iMessage and an android user is in the group, it would cost money to send the person a text instead of using Wi-fi. Therefore, people have a social pressure to buy Apple. Then, once you have bought one Apple product, due to their superior inter-device functionality, it will most likely not be your last Apple device. So buyer’s power over Apple is less.
Apple also has a really good supply chain, of which it has a very good control on. They tend to make most of the things they use, therefore the price can be controlled. Also, if another company tries to compete with you, due to your superior supply chain, it would be really hard for another company to compete.
These are the reason why Apple is the first trillion dollar public company (first company was PetroChina).
According to the author, ML is a collection of tools and techniques that transforms data into decisions.
Basically, ML is about 2 things:
Classifying things (Classification) and
Quantifying Predictions (Regression).
Comparing ML Methods:
To choose, which methods to use for your application, we can just compare the prediction of the method/model with the actual outcomes. This is called evaluation of a model and the metrics used are called evaluation metrics.
For this, we first fit the model to the training data.
Then make predictions based on the trained model.
Then we evaluate the predictions made on test set with the actual outcome.
We can do this for different model/methods and based on the evaluation metrics we can select a suitable method for our application.
Here, just because a machine learning methods fits the training data well, it doesn’t mean it will perform well with the Testing Data.
Fit Train Data well but poor predictions = Overfitting
Doesn’t fit train data well = Underfitting.
Independent and Dependent Variables
variable: value of which vary from record to record.
Say that we have two variables, ‘height’ and ‘weight’. And let us also say that height prediction depends on weight of a person, then here, the ‘height’ is a dependent variable, and ‘weight’ is an independent variable, as this variable used to predict a dependent variable.
Here, the independent variables are also called features.
Discrete and Continuous Data
discrete data: countable values. only takes specific values.
continuous data: measurable values under a particular pre-defined range.
Chapter 2: Cross Validation
From Chapter 1 we learned, that we train the model on ‘train set’ and evaluate the model on ‘test set’.
But how do we decide on what data points to choose for ‘test set’ or the ‘train set’.
The answer is cross validation.
Say you have 10 data points. And let us say that we have chosen a 80/20 train-test split. This means that we are going to assign 8 points randomly to train set and the rest of the 2 data point for test set. Here the 2 points chosen will not be used in test set for the next cross validation. So, for second we choose another 2 data points for test set and remaining for train set. We can do this 5 times, since our total data points is 10 and we have chosen a 80/20 train-test split.
Therefore, cross validation is a way solving the problem of not knowing which points are the best for testing by using them all in an iterative way.
You can also think of it like make 5 groups. And each time using one group as the ‘test set’ and remaining as the ‘train set’.
The number of iterations/groups are also called folds. Therefore, this is an example of 5-fold cross validation.
But why can’t we use all the data as ‘train set’.
Because, only way to determine if a model has been overfit or not is to evaluate on new data.
Reusing same data points for training and testing is called Data Leakage.
The main advantage of cross-validation is that it is a proper measure of how good a model has performed, instead of relying on chance for train-test split. Here, if test set is by chance easy, then the model will be interpreted as better than it actually is.
When we have a lot of data, 10-Fold Cross Validation is commonly used.
Another commonly used cross validation is Leave-One-Out.
used all but one point for training, and the remaining point for testing.
iterate until every single point has been tested.
we usually use this for small dataset.
Commonly, sometimes a particular model performs better in some iteration and another model can perform better in other iteration. In such case we use Statistics to decide the better model.
Chapter 3: Fundamental Concepts in Statistics!!!
Main Idea of Statistics:
Statistics provide us a set of tools to quantify the variation that we find in everyday life.
For example, the number of fries you get in a bucket is not always the same. But say that we track it. Then from statistics, we can predict how many fires will we have tomorrow. And how confident can we be in that prediction can also be determined.
Here, say that you predict a positive result, but are not confident, then you will look for alternative approach.
We know to make a prediction, we need to understand the trend of data.
And histogram is a good way visualizing the trend of data.
divide the range into number of bins.
and stack the element based on the frequency of element that fall into a bin.
Here, the question to think of when making a histogram, is the number of bins you should use.
A Naive Bayes algorithm makes prediction using histogram.
Calculating probability:
Probability of occurrence of something is the total number of occurrence divided by the number of observations made.
Here, the more number of observation we have more confident we can be of our predictions.
But, we know that collecting more samples is expensive both monetarily and time-wise.
We can solve this problem using Probability Distribution.
My initial thoughts and what I would like to get out of this?
What makes it difficult for learning tabular-data for deep-learning algorithms or even Neural Networks?
What is the correct way for benchmarking?
Notes on the paper
There are benchmarks for Deep Learning, however not much for tabular-data.
The superiority of GBTs over NNs is explained by specific features of tabular data:
irregular patterns in the target function,
uninformative features, and
non rotationally-invariant data where linear combinations of features misrepresent the information.
the paper defines
a standard set of 45 datasets from varied domains with clear characteristics of tabular data and
a benchmarking methodology accounting for
fitting models and
finding good hyperparameters.
results show that tree-based models remain SOTA on medium-sized dta (~10K samples)
Inductive biases that lead to better performance for tree-based models for tabular data
NNs are biased to overly smooth solutions
To test the effectiveness of learning smooth functions, the authors smoothed the train set using Gaussian Kernel.
Smoothing degrades the performance of the decision tress but does not effect the performance of the NNs.
NN’s are biased towards low-frequency functions. What does this mean?
regularization and careful optimization does help NNs learn irregular patters/functions.
Periodic embedding of PLE might help learn the high-frequency part of the target function.
This also explain why ExU activation is better for tabular deep learning.
Now the question is why does the NNs fail to learn irregular patterns? And why does PLE help NNs learn better. Does this make the train set more regular?
Uninformative features affect more MLP-like NNs.
MLP like structure have harder time to filter out uninformative features as compared to GBTs.
What does uninformative features mean?
Features that does not provide meaningful or useful information to help make predictions or gain insights from the data.
For GBT, even if we remove half of the features(informative as well as uninformative), the performance does not degrade as much.
However, for NNs (Resnet, FT-Transformer) removing features negatively affect the performance of the model.
Therefore, they are less robust to uninformative features.
MLP like structure’s inherent rotational invariance prevents it from easily isolating and ignoring uninformative features when they are mixed through linear transformations.
However the, weak learners in GBTs, recursively partition the feature space by making splits based on individual feature value. Therefore they are not rotationally invariant, and therefore can easily filter out uninformative features.
Here, FT-Transformer requires a embedding layer due to its use of Attention mechanism. But this embedding transports the feature into different embedding space, breaking the rotational invariant bias of a MLP like architecture.
Data are non invariant by rotation, so should be learning procedures.
Why are MLPs much more hindered by uninformative features, compared to other models?
Random has not lead to much difference in performance of ResNet, and leads to slight degradation in FT-Transformer. But hugely disrupts the performance of GBTs.
This suggests that rotation invariance is not desirable: similarly to vision [Krizhevsky et al., 2012],
We note that a promising avenue for further research would be to find other ways to break rotation invariance which might be less computationally costly than embeddings.
Challenges to build tabular-specific NNs as per the authors
be robust to uninformative features,
preserve the orientation of the data, and
be able to easily learn irregular functions.
Deep Learning architectures have been crafted to create inductive bias matching invariance and spatial dependencies of data.
This means that the inherent assumptions of the model, like what structure does the model expect the input data to have help match with the invariance of the model (for example., how CNN has translational invariance i.e., it will detect an object wherever the object is placed. It does have some other factor but now lets only consider object that is less than the size of the window of CNN).
Benchmark
the code provides 45 datasets split across different settings
medium/large
with/without categorical features
accounts for hyperparameter tuning cost.
But how does it account for it?
Hyperparameter tuning leads to uncontrolled variance on a benchmark Bouthillier et al., 2021, especially with a small budget of model evaluations.
Here at different configuration of hyperparameter, there are different scores. A model can achieve the best score at 3rd trial or may be cannot get the best score until 300th trial. This depends on the configuration, therefore there is an variance in result and this is uncontrollable as we do not know which configuration yields the best result.
design a benchmarking procedure that jointly samples the variance of hyperparameter tuning and explores increasingly high budgets of model. evaluations.
choosing dataset.
what is “inter-sample” attention
preprocessing dataset?
Raw comparison of DL and tree based models.
Explanations of why Tree-based models work better than NNs.
BENCHMARKING
test
Questions that arose while reading the paper
What are the characteristics that let the authors to select a particular dataset?
The characteristics are as follows:
Heterogeneous Columns: Columns should correspond to features of different nature. Not signal or image.
Not high dimensional: Only dataset with a d/n ratio below 1/10. Note: I am not sure what d/n means in this context.
Dataset cannot have too little information.
Dataset cannot be stream-like or time series.
Should be real-world data.
Dataset cannot have features<4 or samples<3000.
Dataset cannot be too easy.
The authors use a different scoring system. But does it account for different Bayes rate for different dataset, that is the real question.
they remove a dataset if a default Logistic Regression (or Linear Regression for regression) reach a score whose relative difference with the score of both a default Resnet (from Gorishniy et al. 2021) and a default HistGradientBoosting model (from scikit learn) is below 5%.
Dataset should be non-deterministic. That means removing dataset where the target is a deterministic function of the data.
The benchmarking is created to make learning task as homogeneous as possible. Therefore challanges of tabular-data that requires a seperate analysis has been omitted. (Here, the question is, is it ommited form analysis or has it been ommited from the benchmarking? )
Only used Medim-sized training set for the analysis.
Remove all the missing data.
Balanced classes.
categorical features with more than 20 items are removed.
Numerical features with less that 10 uniques values are also removed.
Does tree-based models still remain SOTA for small- and large-sized data?
How do you account for
How can a model be robust to uninformative features?
Can NN models preserve the orientation of the data?
This article introduces the TabR model, a retrieval-augmented model designed for tabular data. It is part of a series on tabular deep learning using the Mambular library, which started with an introduction to using an MLP for these tasks.
Architecture Overview
TabR is a retrieval-augmented tabular deep learning method that leverages context from rest of the dataset/database to enrich the representation of the target object, producing more accurate and up-to-date responses. It uses related data points to enhance the prediction. The TabR model consists of three main components: the encoder module, the retrieval module, and the predictor module. The architecture of the TabR model is illustrated in the figure below:
The model is a feed-forward network with a retrieval component located in the residual branch. First, a target object and its candidates for retrieval are encoded with the same encoder E. Then, the retrieval module R enriches the target object’s representation by retrieving and processing relevant objects from the candidates. Finally, predictor P makes a prediction. The bold path highlights the structure of the feed-forward retrieval-free model before the addition of the retrieval module R.
Model Fitting
Now that we have outlined the TabR model, let’s move on to model fitting. The dataset and packages are publicly available, so everything can be copied and run locally or in a Google Colab notebook, provided the necessary packages are installed. We will start by installing the mambular package, loading the dataset, and fitting TabR. Subsequently, we will compare these results with those obtained in earlier articles of this series.
Install Mambular
pip install mambular
pip install delu
pip install faiss-cpu # faiss-gpu for gpu
Prepare the Data
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import StandardScaler
# Load California Housing dataset
data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target
# Drop NAs
X = X.dropna()
y = y[X.index]
# Standard normalize features and target
y = StandardScaler().fit_transform(y.values.reshape(-1, 1)).ravel()
# Train-test-validation split
X_train, X_temp, y_train, y_temp = train_test_split(
X, y, test_size=0.5, random_state=42
)
X_val, X_test, y_val, y_test = train_test_split(
X_temp, y_temp, test_size=0.5, random_state=42
)
Train TabR with Mambular
from mambular.models import TabRRegressor
model = TabRRegressor()
model.fit(X_train, y_train, max_epochs=200)
preds = model.predict(X_test)
model.evaluate(X_test, y_test)
Mean Squared Error on Test Set: 0.1877
Compared to MLP-PLR and MLP-PLE, it is a comparable performance. However, compared to XGBoost, it is not a good fit. Let’s try the TabR with PLE as numerical pre-processing as already used in the FT Transformer article.
Again, compared to XGBoost, this approach does not seem to be a good fit. Therefore, let’s try with an alternative numerical preprocessing method — let’s try MinMax scaling.
The Mean Squared Error (MSE) on the test set is 0.1573, making this our best-performing approach to date, even outperforming deep learning models as well as, tree-based methods like XGBoost.
Below we have summarized the results from all articles so far. Try playing around with some more parameters and improve performance even more. Throughout this series, we will add the results of each introduced method to this table:
