Updated on 2025-01-11

Author: lxaw

papers/list.json

@@ -1,11 +1,20 @@
 [
+  {
+    "title": "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads",
+    "author": "Tianle Cai et al",
+    "year": "2024",
+    "topic": "speculative decoding, drafting, llm",
+    "venue": "ICML",
+    "description": "This paper presents Medusa which augments LLM inference by adding extra decoding heads to predict multiple subsequent tokens in parallel. They also introduce a form of tree-based attention to process candidates. Through the Medusa heads, they obtain probability predictions for the subsequent K+1 tokens. These predictions enable them to create length-K+1 continuations as the candidates. In order to process multiple cnadidates concurrently, they structure their attention such that only tokens from the same continuation are regarded as historical data.For instance, they have in Figure 2 an example where the first Medusa head and generates some top two predictions while the second medusa head generates a top three for each of the top two from the first head. Instead of filling the entire attention mask, they only consider the mask from these 2*3 = 6 tokens, plus the standard identity line.",
+    "link": "https://arxiv.org/pdf/2401.10774"
+  },
   {
     "title": "Recurrent Drafter for Fast Speculative Decoding in Large Language Models",
     "author": "Yunfei Cheng et al",
     "year": "2024",
     "topic": "speculative decoding, drafting, llm",
     "venue": "Arxiv",
-    "description": "This paper indroduces ReDrafter (Recurrent Drafter) that uses an RNN as the draft model and conditions on the LLM's hidden states. They use a beam search to explore the candidate seqeunces and then apply a dynamic tree attention alg to remove duplicated prefixes among the candidates to improve the speedup. They also train via knowledge distillation from LLMs to improve the alignment of the draft model's predictions with those of the LLM.",
+    "description": "This paper introduces ReDrafter (Recurrent Drafter) that uses an RNN as the draft model and conditions on the LLM's hidden states. They use a beam search to explore the candidate seqeunces and then apply a dynamic tree attention alg to remove duplicated prefixes among the candidates to improve the speedup. They also train via knowledge distillation from LLMs to improve the alignment of the draft model's predictions with those of the LLM.",
     "link": "https://arxiv.org/pdf/2403.09919"
   },
   {

papers_read.html

@@ -16,10 +16,10 @@ <h1>Here's where I keep a list of papers I have read.</h1>
         I typically use this to organize papers I found interesting. Please feel free to do whatever you want with it. Note that this is not every single paper I have ever read, just a collection of ones that I remember to put down.
     </p>
     <p id="paperCount">
-        So far, we have read 205 papers. Let's keep it up!
+        So far, we have read 206 papers. Let's keep it up!
     </p> 
     <small id="searchCount">
-        Your search returned 205 papers. Nice! 
+        Your search returned 206 papers. Nice! 
     </small>
 
     <div class="search-inputs">
@@ -46,13 +46,23 @@ <h1>Here's where I keep a list of papers I have read.</h1>
         </thead>
         <tbody>
 
+            <tr>
+                <td>Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads</td>
+                <td>Tianle Cai et al</td>
+                <td>2024</td>
+                <td>speculative decoding, drafting, llm</td>
+                <td>ICML</td>
+                <td>This paper presents Medusa which augments LLM inference by adding extra decoding heads to predict multiple subsequent tokens in parallel. They also introduce a form of tree-based attention to process candidates. Through the Medusa heads, they obtain probability predictions for the subsequent K+1 tokens. These predictions enable them to create length-K+1 continuations as the candidates. In order to process multiple cnadidates concurrently, they structure their attention such that only tokens from the same continuation are regarded as historical data.For instance, they have in Figure 2 an example where the first Medusa head and generates some top two predictions while the second medusa head generates a top three for each of the top two from the first head. Instead of filling the entire attention mask, they only consider the mask from these 2*3 = 6 tokens, plus the standard identity line.</td>
+                <td><a href="https://arxiv.org/pdf/2401.10774" target="_blank">Link</a></td>
+            </tr>
+        
             <tr>
                 <td>Recurrent Drafter for Fast Speculative Decoding in Large Language Models</td>
                 <td>Yunfei Cheng et al</td>
                 <td>2024</td>
                 <td>speculative decoding, drafting, llm</td>
                 <td>Arxiv</td>
-                <td>This paper indroduces ReDrafter (Recurrent Drafter) that uses an RNN as the draft model and conditions on the LLM&#x27;s hidden states. They use a beam search to explore the candidate seqeunces and then apply a dynamic tree attention alg to remove duplicated prefixes among the candidates to improve the speedup. They also train via knowledge distillation from LLMs to improve the alignment of the draft model&#x27;s predictions with those of the LLM.</td>
+                <td>This paper introduces ReDrafter (Recurrent Drafter) that uses an RNN as the draft model and conditions on the LLM&#x27;s hidden states. They use a beam search to explore the candidate seqeunces and then apply a dynamic tree attention alg to remove duplicated prefixes among the candidates to improve the speedup. They also train via knowledge distillation from LLMs to improve the alignment of the draft model&#x27;s predictions with those of the LLM.</td>
                 <td><a href="https://arxiv.org/pdf/2403.09919" target="_blank">Link</a></td>
             </tr>