Ankit Dahal - AI/ML Engineer

Current Focus

Learning Reasoning, reinforcement learning, RLVR, reward modeling

Recently finished: Instruction tuning with LoRA, quantization experiments, IFEval benchmarking

Projects

Language Model Pretraining

Pretraining

End-to-end pretraining of a 416M parameter GPT/Llama-style transformer on FineWeb-Edu (10B tokens). Trained on 8xA100 GPUs using Distributed Data Parallel with custom training loops and memory optimizations.

                            416M
                            Parameters
                        

                            10B
                            Tokens
                        

                            8x
                            A100 GPUs
                        

PyTorch, DDP Custom training loop Chinchilla scaling law RoPE, RMSNorm, SwiGLU, GQA/MHA, KV cache Flash attention

Code Model Checkpoint

Llama 3.1 8B Instruction Tuning

Fine-tuning

LoRA fine-tuning of Llama 3.1 8B base model for instruction following. Extensive quantization experiments (4-bit, 8-bit, BF16) with IFEval and tinyMMLU benchmarking. Achieved 52% improvement in instruction following (200 → 305/834 on IFEval) with ~$10 of compute.

                            +52%
                            IFEval Improvement
                        

                            8B
                            Parameters
                        

                            ~$10
                            Compute Cost
                        

LoRA Unsloth Quantization IFEval W&B

Code

Legal RAG System

RAG / Agents

Production-grade agentic RAG system for legal documents. Full pipeline from data sourcing through retrieval, reranking, and agent orchestration. Hybrid search combining dense embeddings (bge-m3) with sparse retrieval (BM25).

bge-m3 ChromaDB Elasticsearch bge-reranker Gemini-2.5 Docker GKE

Embeddings: bge-m3, gemini-embedding-001

Indexing: ChromaDB (HNSW), Elasticsearch (BM25)

Retrieval: Hybrid search with RRF, convex combination

Agents: Conversational + search agents, planning, self-triage

Training: 560M parameter model fine-tuning

Architecture Technical Notes

Full code available on request

Transformer from Scratch

Fundamentals

Clean PyTorch implementation of "Attention Is All You Need" for deep understanding of transformer mechanics. Extended with modern architectural improvements used in current LLMs.

PyTorch Original sinusoidal positional encoding SwiGLU/ReLU MHA LayerNorm, RMSNorm

Components: encoder/decoder blocks, multi-head attention, positional encoding, layer normalization

Code

LLM VRAM Calculator

Tooling

Comprehensive tool for estimating GPU memory requirements for LLM training and inference. Supports dense transformers and MoE architectures with detailed breakdowns of weights, gradients, optimizer states, activations, and KV cache.

Gradio HuggingFace MoE Mixed Precision

Code

Open to Opportunities

Looking to join teams building interesting AI systems.

GitHub LinkedIn