fabian/L1B3RT4S
1
0
Fork 0
mirror of https://github.com/elder-plinius/L1B3RT4S.git synced 2025-09-25 18:23:41 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update CLOSEDAI.mkd

Browse source
This commit is contained in:
pliny 2024-09-15 23:25:42 -04:00 • committed by GitHub
parent 28d35151d0
commit 2831ae8531
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 9 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

16
CLOSEDAI.mkd
View file

@ -1,8 +1,10 @@
# Special Tokens
# Special Tokens
- **\<SOS> : Start of Sequence**
- **\<EOS> : End of Sequence**
- **\<CLS> : Classification Token**
- **\<SEP> : Separator Token**
- **\<UNK> : Unknown Token**
- **\<REASONING_START> : Begin Reasoning**
- **\<REASONING_END> : End Reasoning**
- **\<ANSWER> : Begin Answer**
- **\<SOS> (Start of Sequence)**: Marks the beginning of a sequence for the model to start processing.
- **\<EOS> (End of Sequence)**: Tells the model when to stop generating text or processing.
- **\<PAD> (Padding Token)**: Pads sequences to the same length for batch processing.
- **\<UNK> (Unknown Token)**: Represents words not in the model's vocabulary.
- **\<MASK> (Mask Token)**: Used in tasks like predicting missing words in masked language models.
- **\<SEP> (Separator Token)**: Separates different segments in input, like questions from context.