Breaking News: Grepper is joining You.com. Read the official announcement!

Couldn't instantiate the backend tokenizer from one of

Hung Truong Ngoc answered on December 6, 2022 Popularity 7/10 Helpfulness 9/10

answer Couldn't instantiate the backend tokenizer from one of

related ValueError: Couldn't instantiate the backend tokenizer from one of:

Couldn't instantiate the backend tokenizer from one of

Comment

Install sentencepiece
Use_fast = False like this: tokenizer = AutoTokenizer.from_pretrained(“XXXXX”, use_fast=False)

Popularity 7/10 Helpfulness 9/10 Language python

Source: Grepper

Tags: backend python

Link to this answer
Share Copy Link

Contributed on Dec 06 2022

Hung Truong Ngoc

0 Answers Avg Quality 2/10

Closely Related Answers

ValueError: Couldn't instantiate the backend tokenizer from one of:

Comment

from transformers import AutoTokenizer

# Replace 'tokenizer_name' with the actual name of the tokenizer you are using
tokenizer_name = "example_tokenizer"

try:
    tokenizer = AutoTokenizer.from_pretrained(tokenizer_name, use_fast=True)
    # Continue with any further processing using the tokenizer
    
    # Example usage:
    text = "Hello, how are you?"
    tokens = tokenizer.tokenize(text)
    
    # Rest of your code
    
except ValueError as e:
    print("Error: Failed to instantiate the backend tokenizer.")
    print(f"Details: {str(e)}")
    # Handle the error or display appropriate instructions to the user

Popularity 3/10 Helpfulness 1/10 Language python

Source: Grepper

Link to this answer
Share Copy Link

Contributed on Sep 12 2023

YOU.com

0 Answers Avg Quality 2/10

Continue with Google

By continuing, I agree that I have read and agree to Greppers's Terms of Service and Privacy Policy.

Couldn't instantiate the backend tokenizer from one of

Contents

More Related Answers

Couldn't instantiate the backend tokenizer from one of

Closely Related Answers

ValueError: Couldn't instantiate the backend tokenizer from one of:

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.