|
Google Papers Blog |
| 12/2017 |
Attention Is All You Need (Transformers) |
| 10/2018 |
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding |
| 10/2019 |
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5) |
| 11/2019 |
Fast Transformer Decoding: One Write-Head is All You Need |
| 02/2020 |
GLU Variants Improve Transformer |
| 03/2020 |
Talking-Heads Attention |
| 05/2020 |
Conformer: Convolution-augmented Transformer for Speech Recognition |
| 09/2020 |
Efficient Transformers: A Survey |
| 12/2020 |
RealFormer: Transformer Likes Residual Attention |
| 01/2021 |
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity |
| 09/2021 |
Finetuned Language Models Are Zero-Shot Learners (Flan) |
| 09/2021 |
Primer: Searching for Efficient Transformers for Language Modeling |
| 11/2021 |
Sparse is Enough in Scaling Transformers |
| 12/2021 |
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 01/2022 |
LaMDA: Language Models for Dialog Applications |
| 01/2022 |
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models |
| 04/2022 |
PaLM: Scaling Language Modeling with Pathways |
| 10/2022 |
Scaling Instruction-Finetuned Language Models (Flan-Palm) |
| 10/2022 |
Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models |
| 10/2022 |
Large Language Models Can Self-Improve |
| 11/2022 |
Efficiently Scaling Transformer Inference |
| 11/2022 |
Fast Inference from Transformers via Speculative Decoding |
| 02/2023 |
Symbolic Discovery of Optimization Algorithms (Lion) |
| 03/2023 |
PaLM-E: An Embodied Multimodal Language Model |
| 04/2023 |
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference |
| 05/2023 |
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes |
| 05/2023 |
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction |
| 05/2023 |
PaLM 2 Technical Report |
| 05/2023 |
Symbol tuning improves in-context learning in language models |
| 05/2023 |
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models |
| 05/2023 |
Towards Expert-Level Medical Question Answering with Large Language Models (Med-Palm 2) |
| 05/2023 |
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining |
| 05/2023 |
How Does Generative Retrieval Scale to Millions of Passages? |
| 05/2023 |
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoint |
| 05/2023 |
Small Language Models Improve Giants by Rewriting Their Outputs |
| 06/2023 |
StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners |
| 06/2023 |
AudioPaLM: A Large Language Model That Can Speak and Listen |
| 06/2023 |
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting |
| 07/2023 |
HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models |
| 09/2023 |
Uncovering mesa-optimization algorithms in Transformers |
| 10/2023 |
Think before you speak: Training Language Models With Pause Tokens |
| 10/2023 |
SpecTr: Fast Speculative Decoding via Optimal Transport |
| 11/2023 |
UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs |
| 11/2023 |
Automatic Engineering of Long Prompts |
|
|
|
OpenAI Papers Blog |
| 07/2017 |
Proximal Policy Optimization Algorithms |
| 04/2019 |
Generating Long Sequences with Sparse Transformers |
| 01/2020 |
Scaling Laws for Neural Language Models |
| 05/2020 |
Language Models are Few-Shot Learners (GPT-3) |
| 01/2022 |
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets |
| 03/2022 |
Training language models to follow instructions with human feedback (InstructGPT) |
| 07/2022 |
Efficient Training of Language Models to Fill in the Middle |
| 03/2023 |
GPT-4 Technical Report |
| 04/2023 |
Consistency Models |
| 05/2023 |
Let's Verify Step by Step |
| 10/2023 |
Improving Image Generation with Better Captions (DALL·E 3) |
|
|
|
Deepmind (Google Deepmind as of 4/2023) Papers Blog |
| 10/2019 |
Stabilizing Transformers for Reinforcement Learning |
| 12/2021 |
Scaling Language Models: Methods, Analysis & Insights from Training Gopher |
| 12/2021 |
Improving language models by retrieving from trillions of tokens (RETRO) |
| 02/2022 |
Competition-Level Code Generation with AlphaCode |
| 02/2022 |
Unified Scaling Laws for Routed Language Models |
| 03/2022 |
Training Compute-Optimal Large Language Models (Chinchilla) |
| 04/2022 |
Flamingo: a Visual Language Model for Few-Shot Learning |
| 05/2022 |
A Generalist Agent (GATO) |
| 07/2022 |
Formal Algorithms for Transformers |
| 02/2023 |
Accelerating Large Language Model Decoding with Speculative Sampling |
| 05/2023 |
Tree of Thoughts: Deliberate Problem Solving with Large Language Models |
| 05/2023 |
Block-State Transformer |
| 05/2023 |
Randomized Positional Encodings Boost Length Generalization of Transformers |
| 08/2023 |
From Sparse to Soft Mixtures of Experts |
| 09/2023 |
Large Language Models as Optimizers |
| 09/2023 |
MADLAD-400: A Multilingual And Document-Level Large Audited Dataset (MT Model) |
| 09/2023 |
Scaling Laws for Sparsely-Connected Foundation Models |
| 09/2023 |
Language Modeling Is Compression |
| 09/2023 |
Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution |
| 10/2023 |
Large Language Models as Analogical Reasoners |
| 10/2023 |
Controlled Decoding from Language Models |
| 10/2023 |
A General Theoretical Paradigm to Understand Learning from Human Preferences |
|
|
|
Meta (Facebook AI Research) Papers Blog |
| 04/2019 |
fairseq: A Fast, Extensible Toolkit for Sequence Modeling |
| 07/2019 |
Augmenting Self-attention with Persistent Memory |
| 11/2019 |
Improving Transformer Models by Reordering their Sublayers |
| 08/2021 |
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation |
| 05/2022 |
OPT: Open Pre-trained Transformer Language Models |
| 07/2022 |
Beyond neural scaling laws: beating power law scaling via data pruning |
| 11/2022 |
Galactica: A Large Language Model for Science |
| 01/2023 |
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA) |
| 02/2023 |
LLaMA: Open and Efficient Foundation Language Models |
| 02/2023 |
Toolformer: Language Models Can Teach Themselves to Use Tools |
| 03/2023 |
Scaling Expert Language Models with Unsupervised Domain Discovery |
| 03/2023 |
SemDeDup: Data-efficient learning at web-scale through semantic deduplication |
| 04/2023 |
Segment Anything (SAM) |
| 04/2023 |
A Cookbook of Self-Supervised Learning |
| 05/2023 |
Learning to Reason and Memorize with Self-Notes |
| 05/2023 |
ImageBind: One Embedding Space To Bind Them All |
| 05/2023 |
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers |
| 05/2023 |
LIMA: Less Is More for Alignment |
| 05/2023 |
Scaling Speech Technology to 1,000+ Languages |
| 05/2023 |
READ: Recurrent Adaptation of Large Transformers |
| 05/2023 |
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models |
| 06/2023 |
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles |
| 06/2023 |
Simple and Controllable Music Generation (MusicGen) |
| 06/2023 |
Improving Open Language Models by Learning from Organic Interactions (BlenderBot 3x) |
| 06/2023 |
Extending Context Window of Large Language Models via Positional Interpolation |
| 06/2023 |
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale |
| 07/2023 |
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning (CM3leon) |
| 07/2023 |
Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 08/2023 |
SeamlessM4T—Massively Multilingual & Multimodal Machine Translation |
| 08/2023 |
D4: Improving LLM Pretraining via Document De-Duplication and Diversification |
| 08/2023 |
Code Llama: Open Foundation Models for Code |
| 08/2023 |
Nougat: Neural Optical Understanding for Academic Documents |
| 09/2023 |
Contrastive Decoding Improves Reasoning in Large Language Models |
| 09/2023 |
Effective Long-Context Scaling of Foundation Models |
| 09/2023 |
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model |
| 09/2023 |
Vision Transformers Need Registers |
| 10/2023 |
RA-DIT: Retrieval-Augmented Dual Instruction Tuning |
| 10/2023 |
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation |
| 10/2023 |
Generative Pre-training for Speech with Flow Matching |
| 11/2023 |
Emu Edit: Precise Image Editing via Recognition and Generation Tasks |
|
|
|
Microsoft Papers Blog |
| 12/2015 |
Deep Residual Learning for Image Recognition |
| 05/2021 |
EL-Attention: Memory Efficient Lossless Attention for Generation |
| 01/2022 |
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale |
| 03/2022 |
DeepNet: Scaling Transformers to 1,000 Layers |
| 12/2022 |
A Length-Extrapolatable Transformer |
| 01/2023 |
Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases |
| 02/2023 |
Language Is Not All You Need: Aligning Perception with Language Models (Kosmos-1) |
| 03/2023 |
Sparks of Artificial General Intelligence: Early experiments with GPT-4 |
| 03/2023 |
TaskMatrix. AI: Completing Tasks by Connecting Foundation Models with Millions of APIs |
| 04/2023 |
Instruction Tuning with GPT-4 |
| 04/2023 |
Inference with Reference: Lossless Acceleration of Large Language Models |
| 04/2023 |
Low-code LLM: Visual Programming over LLMs |
| 04/2023 |
WizardLM: Empowering Large Language Models to Follow Complex Instructions |
| 04/2023 |
MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks |
| 04/2023 |
ResiDual: Transformer with Dual Residual Connections |
| 05/2023 |
Code Execution with Pre-trained Language Models |
| 05/2023 |
Small Models are Valuable Plug-ins for Large Language Models |
| 05/2023 |
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing |
| 06/2023 |
Orca: Progressive Learning from Complex Explanation Traces of GPT-4 |
| 06/2023 |
Augmenting Language Models with Long-Term Memory |
| 06/2023 |
WizardCoder: Empowering Code Large Language Models with Evol-Instruct |
| 06/2023 |
Textbooks Are All You Need (phi-1) |
| 07/2023 |
In-context Autoencoder for Context Compression in a Large Language Model |
| 07/2023 |
Retentive Network: A Successor to Transformer for Large Language Models |
| 08/2023 |
Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference |
| 09/2023 |
Efficient RLHF: Reducing the Memory Usage of PPO |
| 09/2023 |
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models |
| 09/2023 |
Textbooks Are All You Need II (phi-1.5) |
| 09/2023 |
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training |
| 09/2023 |
A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models |
| 09/2023 |
Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models |
| 10/2023 |
Sparse Backpropagation for MoE Training |
| 10/2023 |
Nugget 2D: Dynamic Contextual Compression for Scaling Decoder-only Language Models |
| 10/2023 |
Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness |
| 10/2023 |
Augmented Embeddings for Custom Retrievals |
| 10/2023 |
Guiding Language Model Reasoning with Planning Tokens |
| 10/2023 |
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V |
| 10/2023 |
CodeFusion: A Pre-trained Diffusion Model for Code Generation |
| 10/2023 |
LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery |
| 10/2023 |
FP8-LM: Training FP8 Large Language Models |
| 11/2023 |
Orca 2: Teaching Small Language Models How to Reason |
|
|
|
Hazy Research (Stanford) Papers Blog |
| 10/2021 |
Efficiently Modeling Long Sequences with Structured State Spaces (S4) |
| 04/2022 |
Monarch: Expressive Structured Matrices for Efficient and Accurate Training |
| 05/2022 |
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness |
| 12/2022 |
Hungry Hungry Hippos: Towards Language Modeling with State Space Models |
| 02/2023 |
Simple Hardware-Efficient Long Convolutions for Sequence Modeling |
| 02/2023 |
Hyena Hierarchy: Towards Larger Convolutional Language Models |
| 06/2023 |
TART: A plug-and-play Transformer module for task-agnostic reasoning |
| 07/2023 |
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning |
| 11/2023 |
FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores |
|
|
|
THUDM (Tsinghua University) Papers Github |
| 10/2022 |
GLM-130B: An Open Bilingual Pre-Trained Model |
| 03/2023 |
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X |
| 04/2023 |
DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task |
| 06/2023 |
WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences |
| 09/2023 |
GPT Can Solve Mathematical Problems Without a Calculator (MathGLM) |
| 10/2023 |
AgentTuning: Enabling Generalized Agent Abilities for LLMs (AgentLM) |
| 11/2023 |
CogVLM: Visual Expert for Pretrained Language Models |
|
|
|
Open Models |
| 06/2021 |
GPT-J-6B: 6B JAX-Based Transformer |
| 09/2021 |
Pythia: A Customizable Hardware Prefetching Framework Using Online Reinforcement Learning |
| 03/2022 |
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis |
| 04/2022 |
GPT-NeoX-20B: An Open-Source Autoregressive Language Model |
| 11/2022 |
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model |
| 04/2023 |
Visual Instruction Tuning (LLaVA) |
| 05/2023 |
StarCoder: May the source be with you! |
| 05/2023 |
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages |
| 05/2023 |
Otter: A Multi-Modal Model with In-Context Instruction Tuning |
| 05/2023 |
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning |
| 05/2023 |
CodeT5+: Open Code Large Language Models for Code Understanding and Generation |
| 05/2023 |
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities |
| 05/2023 |
RWKV: Reinventing RNNs for the Transformer Era |
| 05/2023 |
Lion: Adversarial Distillation of Closed-Source Large Language Model |
| 05/2023 |
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training |
| 06/2023 |
Segment Anything in High Quality |
| 06/2023 |
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding |
| 06/2023 |
High-Fidelity Audio Compression with Improved RVQGAN |
| 06/2023 |
StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models |
| 06/2023 |
Anticipatory Music Transformer |
| 06/2023 |
RepoFusion: Training Code Models to Understand Your Repository |
| 06/2023 |
MPT-30B: Raising the bar for open-source foundation models |
| 06/2023 |
Vec2Vec: A Compact Neural Network Approach for Transforming Text Embeddings with High Fidelity |
| 06/2023 |
ViNT: A Foundation Model for Visual Navigation |
| 06/2023 |
How Long Can Open-Source LLMs Truly Promise on Context Length? (LongChat) |
| 07/2023 |
Hierarchical Open-vocabulary Universal Image Segmentation |
| 07/2023 |
Focused Transformer: Contrastive Training for Context Scaling (LongLLaMA |
| 07/2023 |
Rhythm Modeling for Voice Conversion (Urhythmic) |
| 07/2023 |
Scaling TransNormer to 175 Billion Parameters |
| 08/2023 |
Separate Anything You Describe |
| 08/2023 |
StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data |
| 09/2023 |
RADIO: Reference-Agnostic Dubbing Video Synthesis |
| 09/2023 |
Matcha-TTS: A fast TTS architecture with conditional flow matching |
| 09/2023 |
DreamLLM: Synergistic Multimodal Comprehension and Creation |
| 09/2023 |
Baichuan 2: Open Large-scale Language Models |
| 09/2023 |
Qwen Technical Report |
| 09/2023 |
Mistral 7B |
| 10/2023 |
MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning |
| 10/2023 |
Improved Baselines with Visual Instruction Tuning (LLaVA 1.5) |
| 10/2023 |
LLark: A Multimodal Foundation Model for Music |
| 10/2023 |
SALMONN: Towards Generic Hearing Abilities for Large Language Models |
| 10/2023 |
Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents |
| 11/2023 |
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models |
| 11/2023 |
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition |
| 11/2023 |
YUAN 2.0: A Large Language Model with Localized Filtering-based Attention |
|
|
|
Various |
| 09/2014 |
Neural Machine Translation by Jointly Learning to Align and Translate |
| 06/2019 |
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View |
| 10/2019 |
Root Mean Square Layer Normalization |
| 10/2019 |
Transformers without Tears: Improving the Normalization of Self-Attention |
| 12/2019 |
Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection |
| 02/2020 |
On Layer Normalization in the Transformer Architecture |
| 04/2020 |
Longformer: The Long-Document Transformer |
| 06/2020 |
Memory Transformer |
| 07/2020 |
Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexity |
| 12/2020 |
ERNIE-Doc: A Retrospective Long-Document Modeling Transformer |
| 01/2021 |
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks |
| 03/2021 |
The Low-Rank Simplicity Bias in Deep Networks |
| 04/2021 |
RoFormer: Enhanced Transformer with Rotary Position Embedding |
| 06/2021 |
LoRA: Low-Rank Adaptation of Large Language Models |
| 07/2023 |
CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention |
| 03/2022 |
Memorizing Transformers |
| 04/2022 |
UL2: Unifying Language Learning Paradigms |
| 05/2022 |
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (IA3) |
| 06/2022 |
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models |
| 07/2022 |
Language Models (Mostly) Know What They Know |
| 08/2022 |
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale |
| 09/2022 |
Petals: Collaborative Inference and Fine-tuning of Large Models |
| 10/2022 |
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers |
| 10/2022 |
Truncation Sampling as Language Model Desmoothing |
| 10/2022 |
DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation |
| 11/2022 |
An Algorithm for Routing Vectors in Sequences |
| 12/2022 |
Self-Instruct: Aligning Language Model with Self Generated Instructions |
| 12/2022 |
Parallel Context Windows Improve In-Context Learning of Large Language Models |
| 12/2022 |
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor |
| 12/2022 |
Pretraining Without Attention |
| 12/2022 |
The case for 4-bit precision: k-bit Inference Scaling Laws |
| 12/2022 |
Prompting Is Programming: A Query Language for Large Language Models |
| 01/2023 |
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient |
| 01/2023 |
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot |
| 01/2023 |
Memory Augmented Large Language Models are Computationally Universal |
| 01/2023 |
Progress measures for grokking via mechanistic interpretability |
| 02/2023 |
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models |
| 02/2023 |
The Wisdom of Hindsight Makes Language Models Better Instruction Followers |
| 02/2023 |
End-to-End Deep Learning Framework for Real-Time Inertial Attitude Estimation using 6DoF IMU |
| 02/2023 |
The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation |
| 03/2023 |
COLT5: Faster Long-Range Transformers with Conditional Computation |
| 03/2023 |
High-throughput Generative Inference of Large Language Models with a Single GPU |
| 03/2023 |
Meet in the Middle: A New Pre-training Paradigm |
| 03/2023 |
Reflexion: an autonomous agent with dynamic memory and self-reflection |
| 03/2023 |
Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning |
| 03/2023 |
FP8 versus INT8 for efficient deep learning inference |
| 03/2023 |
Self-Refine: Iterative Refinement with Self-Feedback |
| 04/2023 |
RPTQ: Reorder-based Post-training Quantization for Large Language Models |
| 04/2023 |
REFINER: Reasoning Feedback on Intermediate Representations |
| 04/2023 |
Generative Agents: Interactive Simulacra of Human Behavior |
| 04/2023 |
Compressed Regression over Adaptive Networks |
| 04/2023 |
A Cheaper and Better Diffusion Language Model with Soft-Masked Noise |
| 04/2023 |
RRHF: Rank Responses to Align Language Models with Human Feedback without tears |
| 04/2023 |
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society |
| 04/2023 |
Automatic Gradient Descent: Deep Learning without Hyperparameters |
| 04/2023 |
SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models |
| 04/2023 |
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study |
| 04/2023 |
Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling |
| 04/2023 |
Scaling Transformer to 1M tokens and beyond with RMT |
| 04/2023 |
Answering Questions by Meta-Reasoning over Multiple Chains of Thought |
| 04/2023 |
Towards Multi-Modal DBMSs for Seamless Querying of Texts and Tables |
| 04/2023 |
We're Afraid Language Models Aren't Modeling Ambiguity |
| 04/2023 |
The Internal State of an LLM Knows When its Lying |
| 04/2023 |
Search-in-the-Chain: Towards the Accurate, Credible and Traceable Content Generation for Complex Knowledge-intensive Tasks |
| 05/2023 |
Towards Unbiased Training in Federated Open-world Semi-supervised Learning |
| 05/2023 |
Unlimiformer: Long-Range Transformers with Unlimited Length Input |
| 05/2023 |
FreeLM: Fine-Tuning-Free Language Model |
| 05/2023 |
Cuttlefish: Low-rank Model Training without All The Tuning |
| 05/2023 |
AttentionViz: A Global View of Transformer Attention |
| 05/2023 |
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models |
| 05/2023 |
A Frustratingly Easy Improvement for Position Embeddings via Random Padding |
| 05/2023 |
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision |
| 05/2023 |
Explanation-based Finetuning Makes Models More Robust to Spurious Cues |
| 05/2023 |
An automatically discovered chain-of-thought prompt generalizes to novel models and datasets |
| 05/2023 |
Recommender Systems with Generative Retrieval |
| 05/2023 |
Fast Distributed Inference Serving for Large Language Models |
| 05/2023 |
Chain-of-Dictionary Prompting Elicits Translation in Large Language Models |
| 05/2023 |
Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach |
| 05/2023 |
Active Retrieval Augmented Generation |
| 05/2023 |
Scalable Coupling of Deep Learning with Logical Reasoning |
| 05/2023 |
Interpretability at Scale: Identifying Causal Mechanisms in Alpaca |
| 05/2023 |
StructGPT: A General Framework for Large Language Model to Reason over Structured Data |
| 05/2023 |
Pre-Training to Learn in Context |
| 05/2023 |
ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings |
| 05/2023 |
Accelerating Transformer Inference for Translation via Parallel Decoding |
| 05/2023 |
Cooperation Is All You Need |
| 05/2023 |
PTQD: Accurate Post-Training Quantization for Diffusion Models |
| 05/2023 |
LLM-Pruner: On the Structural Pruning of Large Language Models |
| 05/2023 |
SelfzCoT: a Self-Prompt Zero-shot CoT from Semantic-level to Code-level for a Better Utilization of LLMs |
| 05/2023 |
QLoRA: Efficient Finetuning of Quantized LLMs |
| 05/2023 |
"According to ..." Prompting Language Models Improves Quoting from Pre-Training Data |
| 05/2023 |
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training |
| 05/2023 |
Landmark Attention: Random-Access Infinite Context Length for Transformers |
| 05/2023 |
Scaling Data-Constrained Language Models |
| 05/2023 |
Fine-Tuning Language Models with Just Forward Passes |
| 05/2023 |
Intriguing Properties of Quantization at Scale |
| 05/2023 |
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time |
| 05/2023 |
Blockwise Parallel Transformer for Long Context Large Models |
| 05/2023 |
The Impact of Positional Encoding on Length Generalization in Transformers |
| 05/2023 |
Adapting Language Models to Compress Contexts |
| 05/2023 |
Direct Preference Optimization: Your Language Model is Secretly a Reward Model |
| 06/2023 |
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration |
| 06/2023 |
Faster Causal Attention Over Large Sequences Through Sparse Flash Attention |
| 06/2023 |
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training |
| 06/2023 |
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression |
| 06/2023 |
Fine-Tuning Language Models with Advantage-Induced Policy Alignment |
| 06/2023 |
Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards |
| 06/2023 |
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model |
| 06/2023 |
Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memories |
| 06/2023 |
Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion |
| 06/2023 |
Word sense extension |
| 06/2023 |
Mitigating Transformer Overconfidence via Lipschitz Regularization |
| 06/2023 |
Recurrent Attention Networks for Long-text Modeling |
| 06/2023 |
One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning |
| 06/2023 |
SqueezeLLM: Dense-and-Sparse Quantization |
| 06/2023 |
Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training |
| 06/2023 |
Propagating Knowledge Updates to LMs Through Distillation |
| 06/2023 |
Full Parameter Fine-tuning for Large Language Models with Limited Resources |
| 06/2023 |
A Simple and Effective Pruning Approach for Large Language Models |
| 06/2023 |
InRank: Incremental Low-Rank Learning |
| 06/2023 |
Evaluating the Zero-shot Robustness of Instruction-tuned Language Models |
| 06/2023 |
Learning to Generate Better Than Your LLM (RLGF) |
| 06/2023 |
Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing |
| 06/2023 |
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Model |
| 06/2023 |
FLuRKA: Fast fused Low-Rank & Kernel Attention |
| 06/2023 |
Stay on topic with Classifier-Free Guidance |
| 07/2023 |
AutoST: Training-free Neural Architecture Search for Spiking Transformers |
| 07/2023 |
Single Sequence Prediction over Reasoning Graphs for Multi-hop QA |
| 07/2023 |
Shifting Attention to Relevance: Towards the Uncertainty Estimation of Large Language Models |
| 07/2023 |
Facing off World Model Backbones: RNNs, Transformers, and S4 |
| 07/2023 |
Improving Retrieval-Augmented Large Language Models via Data Importance Learning |
| 07/2023 |
Teaching Arithmetic to Small Transformers |
| 07/2023 |
QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models |
| 07/2023 |
Stack More Layers Differently: High-Rank Training Through Low-Rank Updates |
| 07/2023 |
Copy Is All You Need (CoG) |
| 07/2023 |
Multi-Method Self-Training: Improving Code Generation With Text, And Vice Versa |
| 07/2023 |
Divide & Bind Your Attention for Improved Generative Semantic Nursing |
| 07/2023 |
Challenges and Applications of Large Language Models |
| 07/2023 |
Soft Prompt Tuning for Augmenting Dense Retrieval with Large Language Models |
| 07/2023 |
QuIP: 2-Bit Quantization of Large Language Models With Guarantees |
| 07/2023 |
CoRe Optimizer: An All-in-One Solution for Machine Learning |
| 07/2023 |
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time |
| 08/2023 |
ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation |
| 08/2023 |
EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models |
| 08/2023 |
Activation Addition: Steering Language Models Without Optimization |
| 08/2023 |
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models |
| 08/2023 |
Accelerating LLM Inference with Staged Speculative Decoding |
| 08/2023 |
YaRN: Efficient Context Window Extension of Large Language Models |
| 08/2023 |
LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models |
| 09/2023 |
Making Large Language Models Better Reasoners with Alignment |
| 09/2023 |
Data-Juicer: A One-Stop Data Processing System for Large Language Models |
| 09/2023 |
Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices |
| 09/2023 |
SLiMe: Segment Like Me |
| 09/2023 |
Norm Tweaking: High-performance Low-bit Quantization of Large Language Models |
| 09/2023 |
When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale |
| 09/2023 |
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs |
| 09/2023 |
Efficient Memory Management for Large Language Model Serving with PagedAttention |
| 09/2023 |
Cure the headache of Transformers via Collinear Constrained Attention |
| 09/2023 |
Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity |
| 09/2023 |
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models |
| 09/2023 |
MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation |
| 09/2023 |
Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models |
| 09/2023 |
Improving Code Generation by Dynamic Temperature Sampling |
| 09/2023 |
Efficient Streaming Language Models with Attention Sinks |
| 10/2023 |
DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models |
| 10/2023 |
GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length |
| 10/2023 |
Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models |
| 10/2023 |
Elephant Neural Networks: Born to Be a Continual Learner |
| 10/2023 |
Ring Attention with Blockwise Transformers for Near-Infinite Context |
| 10/2023 |
Retrieval meets Long Context Large Language Models |
| 10/2023 |
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines |
| 10/2023 |
LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers |
| 10/2023 |
Amortizing intractable inference in large language models (GFlowNet Tuning) |
| 10/2023 |
SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF |
| 10/2023 |
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity |
| 10/2023 |
Let Models Speak Ciphers: Multiagent Debate through Embeddings |
| 10/2023 |
InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining |
| 10/2023 |
CacheGen: Fast Context Loading for Language Model Applications |
| 10/2023 |
MatFormer: Nested Transformer for Elastic Inference |
| 10/2023 |
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models |
| 10/2023 |
Towards End-to-end 4-Bit Inference on Generative Large Language Models (QUIK) |
| 10/2023 |
Microscaling Data Formats for Deep Learning |
| 10/2023 |
xVal: A Continuous Number Encoding for Large Language Models |
| 10/2023 |
An Emulator for Fine-Tuning Large Language Models using Small Language Models |
| 10/2023 |
Frozen Transformers in Language Models Are Effective Visual Encoder Layers |
| 10/2023 |
LoBaSS: Gauging Learnability in Supervised Fine-tuning Data |
| 10/2023 |
Quality-Diversity through AI Feedback |
| 10/2023 |
DoGE: Domain Reweighting with Generalization Estimation |
| 10/2023 |
E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity |
| 10/2023 |
Mixture of Tokens: Efficient LLMs through Cross-Example Aggregation |
| 10/2023 |
Personas as a Way to Model Truthfulness in Language Models |
| 10/2023 |
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving |
| 11/2023 |
AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models |
| 11/2023 |
FlashDecoding++: Faster Large Language Model Inference on GPUs |
| 11/2023 |
Divergent Token Metrics: Measuring degradation to prune away LLM components -- and optimize quantization |
| 11/2023 |
Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs |
| 11/2023 |
REST: Retrieval-Based Speculative Decoding |
| 11/2023 |
DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines |
| 11/2023 |
Token-level Adaptation of LoRA Adapters for Downstream Task Generalization |
| 11/2023 |
Exponentially Faster Language Modelling |
| 11/2023 |
MultiLoRA: Democratizing LoRA for Better Multi-Task Learning |
| 11/2023 |
LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning |
| 11/2023 |
Token Recycling for Efficient Sequential Inference with Vision Transformers |
| 11/2023 |
Enabling Fast 2-bit LLM on GPUs: Memory Alignment, Sparse Outlier, and Asynchronous Dequantization |
|
|
|
Articles |
| 03/2019 |
Rich Sutton - The Bitter Lesson |
| 06/2022 |
Yann LeCun - A Path Towards Autonomous Machine Intelligence |
| 01/2023 |
Lilian Weng - The Transformer Family Version 2.0 |
| 01/2023 |
Lilian Weng - Large Transformer Model Inference Optimization |
| 03/2023 |
Stanford - Alpaca: A Strong, Replicable Instruction-Following Model |
| 05/2023 |
OpenAI - Language models can explain neurons in language models |
| 05/2023 |
Alex Turner - Steering GPT-2-XL by adding an activation vector |
| 06/2023 |
YyWang - Do We Really Need the KVCache for All Large Language Models |
| 06/2023 |
kaiokendev - Extending Context is Hard…but not Impossible |
| 06/2023 |
bloc97 - NTK-Aware Scaled RoPE |
| 07/2023 |
oobabooga - A direct comparison between llama.cpp, AutoGPTQ, ExLlama, and transformers perplexities |
| 07/2023 |
Jianlin Su - Carrying the beta position to the end (better NTK RoPe method) |
| 09/2023 |
FasterDecoding - Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads |
| 10/2023 |
Tri Dao - Flash-Decoding for Long-Context Inference |
| 10/2023 |
Evan Armstrong - Human-Sourced, AI-Augmented: a promising solution for open source conversational data |
| 11/2023 |
LMSYS - Break the Sequential Dependency of LLM Inference Using Lookahead Decoding |