| Family | Rule | Full origin regex | Contraction group | Letter group | Number group | Punctuation / symbol group | Whitespace group |
|---|---|---|---|---|---|---|---|
r50k |
GPT-2 / r50k |
'(?:[sdmt]|ll|ve|re)1|
?\p{L}+2|
?\p{N}+3|
?[^\s\p{L}\p{N}]+4|
\s+(?!\S)5|
\s+6
|
(1) '(?:[sdmt]|ll|ve|re)Do not separate 's, 'd, 'm, 't, 'll, 've, or 're. |
(2) ?\p{L}+Letters with optional preceding space. |
(3) ?\p{N}+Numbers with optional preceding space. |
(4) ?[^\s\p{L}\p{N}]+Other characters with optional preceding space. |
(5), (6) \s+(?!\S)|\s+Whitespace; no special CR/LF handling. |
cl100k |
GPT-4 / cl100k |
'(?i:[sdmt]|ll|ve|re)1|
[^\r\n\p{L}\p{N}]?+\p{L}++2|
\p{N}{1,3}+3|
?[^\s\p{L}\p{N}]++[\r\n]*+4|
\s++$5|
\s*[\r\n]6|
\s+(?!\S)7|
\s+8
|
(2) [^\r\n\p{L}\p{N}]?+\p{L}++Letters with one optional preceding non-letter/number, except CR/LF. |
(3) \p{N}{1,3}+At most 3 numbers. |
(4) ?[^\s\p{L}\p{N}]++[\r\n]*+Other characters with optional preceding space; absorbs CR/LF. |
(5), (6), (7), (8) \s++$|\s*[\r\n]|\s+(?!\S)|\s+Trailing whitespace stays together. |
|
| Qwen2 / Qwen3 |
(?i:'s|'t|'re|'ve|'m|'ll|'d)1|
[^\r\n\p{L}\p{N}]?\p{L}+2|
\p{N}3|
?[^\s\p{L}\p{N}]+[\r\n]*4|
\s*[\r\n]+5|
\s+(?!\S)6|
\s+7
|
(1) (?i:'s|'t|'re|'ve|'m|'ll|'d)Do not separate 's, 't, 're, 've, 'm, 'll, or 'd. |
(2) [^\r\n\p{L}\p{N}]?\p{L}+Letters with one optional preceding character. |
(3) \p{N}Exactly one number. |
(4) ?[^\s\p{L}\p{N}]+[\r\n]*Other characters with optional preceding space; absorbs CR/LF. |
(5), (6), (7) \s*[\r\n]+|\s+(?!\S)|\s+Splits after the last CR/LF. |
|
| OLMo 2/3 / dolma2 |
(?i:'s|'t|'re|'ve|'m|'ll|'d)1|
[^\r\n\p{L}\p{N}]?\p{L}+2|
\p{N}{1,3}3|
?[^\s\p{L}\p{N}]+[\r\n]*4|
\s*[\r\n]+5|
\s+(?!\S)6|
\s+7
|
(3) \p{N}{1,3}At most 3 numbers. |
|||||
| Qwen3.5 |
(?i:'s|'t|'re|'ve|'m|'ll|'d)1|
[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+2|
\p{N}3|
?[^\s\p{L}\p{M}\p{N}]+[\r\n]*4|
\s*[\r\n]+5|
\s+(?!\S)6|
\s+7
|
(2) [^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+Marks join letter runs. |
(3) \p{N}Exactly one number. |
(4) ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*Other characters excluding marks; absorbs CR/LF. |
|||
deepseek_v3 |
DeepSeek V3 / V3.1 / V4 |
Three isolated splits run in sequence. Each stage may subdivide pieces from an earlier stage, but it can never join across an earlier boundary. Regex matches and unmatched gaps both remain as pieces. Split 1 — isolate number groups
\p{N}{1,3}1
Split 2 — isolate CJK-range runs
[\u{4E00}-\u{9FA5}\u{3040}-\u{309F}\u{30A0}-\u{30FF}]+2
Split 3 — apply the main rule inside every existing piece
[!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~][A-Za-z]+3|
[^\r\n\p{L}\p{P}\p{S}]?[\p{L}\p{M}]+4|
?[\p{P}\p{S}]+[\r\n]*5|
\s*[\r\n]+6|
\s+(?!\S)7|
\s+8
|
(3) [!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~][A-Za-z]+No contraction rule; matches punctuation + ASCII letters. |
(2), (3), (4) [CJK range]+|[ASCII punctuation][A-Za-z]+|[^\r\n\p{L}\p{P}\p{S}]?[\p{L}\p{M}]+CJK runs, punctuation + ASCII letters, or letter/mark runs. |
(1) \p{N}{1,3}At most 3 numbers; creates hard boundaries. |
(5) ?[\p{P}\p{S}]+[\r\n]*Unicode punctuation/symbols; preserves unmatched gaps. |
(6), (7), (8) \s*[\r\n]+|\s+(?!\S)|\s+Cannot cross Split 1/2 boundaries. |
o200k |
GPT-4o / gpt-oss |
[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?1|
[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?2|
\p{N}{1,3}3|
?[^\s\p{L}\p{N}]+[\r\n/]*4|
\s*[\r\n]+5|
\s+(?!\S)6|
\s+7
|
(1), (2) (?i:'s|'t|'re|'ve|'m|'ll|'d)?Optional contraction suffix. |
(1), (2) [case-structured letter alternative 1]|[case-structured letter alternative 2]Case-aware letters; splits camelCase. |
(3) \p{N}{1,3}At most 3 numbers. |
(4) ?[^\s\p{L}\p{N}]+[\r\n/]*Other characters with optional preceding space; absorbs CR/LF and /. |
(5), (6), (7) \s*[\r\n]+|\s+(?!\S)|\s+Shared whitespace rules. |
| Nemotron-3 |
[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+1|
[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*2|
\p{N}3|
?[^\s\p{L}\p{N}]+[\r\n/]*4|
\s*[\r\n]+5|
\s+(?!\S)6|
\s+7
|
N/A | (3) \p{N}Exactly one number. |
||||
| Kimi K2 |
[\p{Han}]+1|
[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]*[\p{Ll}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?2|
[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]+[\p{Ll}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?3|
\p{N}{1,3}4|
?[^\s\p{L}\p{N}]+[\r\n]*5|
\s*[\r\n]+6|
\s+(?!\S)7|
\s+8
|
(2), (3) (?i:'s|'t|'re|'ve|'m|'ll|'d)?Optional suffix for non-Han letters. |
(1), (2), (3) [\p{Han}]+|[non-Han case alternatives]Han runs, or case-aware non-Han letters. |
(4) \p{N}{1,3}At most 3 numbers; Han-started runs use (1). |
(5) ?[^\s\p{L}\p{N}]+[\r\n]*Other characters; absorbs CR/LF but not /. |
(6), (7), (8) \s*[\r\n]+|\s+(?!\S)|\s+Shared whitespace rules. |