Skip to content

Commit ffa34ca

Browse files
committed
minor fix
1 parent fd77aac commit ffa34ca

File tree

1 file changed

+1
-8
lines changed

1 file changed

+1
-8
lines changed

readme.md

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -241,14 +241,7 @@ The `MoDInfiniTransformer` module takes the following arguments:
241241
- `sampling_factor`: A numeric value in the interval (1, `segment_len`) that determines the number of tokens to select from each segment during the top-k selection. A larger value of `sampling_factor` results in fewer tokens being selected.
242242
- `update`: The type of update to use for the memory matrix. Can be "linear" or "delta". (Default is "linear".)
243243
- `causal`: Whether to use causal attention in SDP calculations (where each position can only attend to previous positions). (Default is False.)
244-
- `positional_embedder`: The type of positional embeddings to apply. The following embedding methods are supported:
245-
246-
- `"rope"`: RoPE ([https://arxiv.org/abs/2104.09864](https://arxiv.org/abs/2104.09864))
247-
- `"yarn"`: YaRN ([https://arxiv.org/abs/2309.00071](https://arxiv.org/abs/2309.00071))
248-
- `"rope_pose"`: RoPE with PoSE ([https://arxiv.org/abs/2309.10400](https://arxiv.org/abs/2309.10400))
249-
- `"yarn_pose"`: YaRN with PoSE ([https://arxiv.org/abs/2309.10400](https://arxiv.org/abs/2309.10400))
250-
- `"none"`: No positional embedding (default)
251-
244+
- `positional_embedder`: An optional `PositionEmbeddings` object: `RoPEEmbeddings` or `YaRNEmbeddings` (Default is None.)
252245
- `init_state_learnable`: Whether the initial memory state and normalization vector are learnable parameters. (Default is False.)
253246
- `dropout`: The dropout rate to apply in the MLP. (Default is 0.0.)
254247

0 commit comments

Comments
 (0)