transformers ：bert1

from transformers import pipeline


from transformers import AutoTokenizer


tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')

# encoded_input = tokenizer("Hello, I'm a single sentence!")
# encoded_input = tokenizer("你好我是你爸爸")

batch_sentences = ["Hello I'm a single sentence",
                   "And another sentence",
                   "And the very very last one"]
encoded_inputs = tokenizer(batch_sentences)
print(encoded_inputs)

{'input_ids': [[101, 8667, 146, 112, 182, 170, 1423, 5650, 102], [101, 1262, 1330, 5650, 102], [101, 1262, 1103, 1304, 1304, 1314, 1141, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1]]}

注意这里tokenizer在encode之后返回了三个部分：

• input_ids: 相当于是分词之后每个token转变成了一个id
• token_type_ids: 我们知道BERT模型允许我们传入两个sequence。而这个token_type_id就表示当前的token究竟是第一个sequence还是第二个sequence
• attention_mask: 表示当前的位置是真正的token还是只是padding而已。