Tokenizer

class langchain_text_splitters.base.Tokenizer(

chunk_overlap: int,

tokens_per_chunk: int,

decode: Callable[[List[int]], str],

encode: Callable[[str], List[int]],

)

Tokenizer data class.

Attributes

Methods

__init__(chunk_overlap, tokens_per_chunk, ...)

Parameters:

• chunk_overlap (int)

• tokens_per_chunk (int)

• decode (Callable[[List[int]], str])

• encode (Callable[[str], List[int]])

__init__(

chunk_overlap: int,

tokens_per_chunk: int,

decode: Callable[[List[int]], str],

encode: Callable[[str], List[int]],

) → None

Parameters:

• chunk_overlap (int)

• tokens_per_chunk (int)

• decode (Callable[[List[int]], str])

• encode (Callable[[str], List[int]])

Return type:

None