From 3b32b6fb01f8d00f628a34228f79e1249872009f Mon Sep 17 00:00:00 2001 From: pliny <133052465+elder-plinius@users.noreply.github.com> Date: Sun, 22 Sep 2024 12:23:23 -0400 Subject: [PATCH] Update CLOSEDAI.mkd --- CLOSEDAI.mkd | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/CLOSEDAI.mkd b/CLOSEDAI.mkd index ad30de3..9958dd2 100644 --- a/CLOSEDAI.mkd +++ b/CLOSEDAI.mkd @@ -8,3 +8,64 @@ - **\ : End Reasoning** - **\ : Begin Answer** +Comprehensive List of Special Tokens + : Start of Sequence +Purpose: Marks the beginning of a text sequence. +Usage: Helps the model recognize where to start processing or generating text. + : End of Sequence +Purpose: Indicates the termination of a text sequence. +Usage: Signals the model to stop generating further tokens, ensuring responses are concise. + : Classification Token +Purpose: Used primarily in classification tasks. +Usage: Aggregates information from the entire input to produce a single output label or category. + : Separator Token +Purpose: Acts as a delimiter between different segments of input. +Usage: Useful in tasks like question-answering or sentence-pair classification where distinguishing between parts is essential. + : Unknown Token +Purpose: Represents words or tokens not present in the model's vocabulary. +Usage: Ensures the model can handle out-of-vocabulary words gracefully without errors. + : Padding Token +Purpose: Used to pad sequences to a uniform length. +Usage: Facilitates batch processing by ensuring all input sequences are the same length. + : Mask Token +Purpose: Used in masked language modeling tasks. +Usage: Helps the model predict missing or masked words within a sentence. + : Beginning of Sentence +Purpose: Marks the start of a sentence. +Usage: Similar to , used to indicate where a new sentence begins. + : End of Text +Purpose: Denotes the end of a block of text. +Usage: Useful in distinguishing between multiple text blocks or documents. + : User Input Marker +Purpose: Indicates the beginning of the user's input in a conversation. +Usage: Helps the model differentiate between user queries and assistant responses in multi-turn dialogues. + : Assistant Response Marker +Purpose: Denotes the start of the assistant's response. +Usage: Facilitates clear separation between user inputs and assistant outputs. + : System Message Marker +Purpose: Marks system-level instructions or configurations. +Usage: Used for setting up the context or guidelines that the assistant should follow throughout the interaction. + : Thought Process Marker +Purpose: Highlights the internal reasoning or thought process of the assistant. +Usage: Structures the assistant's reasoning steps before arriving at a conclusion. + : Conclusion Marker +Purpose: Signals the end of the reasoning process and the beginning of the final answer. +Usage: Ensures that the assistant provides a clear and concise answer following detailed reasoning. + : URL Token +Purpose: Represents URLs within the text. +Usage: Helps the model recognize and handle web links appropriately. + : Emoji Token +Purpose: Denotes emojis used within the text. +Usage: Allows the model to process and generate emojis correctly. + : Date Token +Purpose: Represents dates within the text. +Usage: Enables the model to identify and handle date information effectively. +