fiftyone.utils.clip.tokenizer.SimpleTokenizer

class documentation

class SimpleTokenizer(object): (source)

Constructor: SimpleTokenizer(bpe_path)

View In Hierarchy

Undocumented

Method __init__ Undocumented
Method bpe Undocumented
Method decode Undocumented
Method encode Undocumented
Instance Variable bpe_ranks Undocumented
Instance Variable byte_decoder Undocumented
Instance Variable byte_encoder Undocumented
Instance Variable cache Undocumented
Instance Variable decoder Undocumented
Instance Variable encoder Undocumented
Instance Variable pat Undocumented
def __init__(self, bpe_path: str = default_bpe()): (source)

Undocumented

def bpe(self, token): (source)

Undocumented

def decode(self, tokens): (source)

Undocumented

def encode(self, text): (source)

Undocumented

bpe_ranks = (source)

Undocumented

byte_decoder = (source)

Undocumented

byte_encoder = (source)

Undocumented

cache: dict[str, str] = (source)

Undocumented

decoder = (source)

Undocumented

encoder = (source)

Undocumented

pat = (source)

Undocumented