MindSpore框架 加载文本数据集 示例

代码原地址：

https://www.mindspore.cn/tutorial/training/zh-CN/r1.2/use/load_dataset_text.html

=======================================================

完整代码：

import os

os.system("rm -f ./datasets/tokenizer.txt")

if not os.path.exists('./datasets'):
    os.mkdir('./datasets')
file_handle=open('./datasets/tokenizer.txt',mode='w')
file_handle.write('Welcome to Beijing 
北京欢迎您！ 
我喜欢English! 
')
file_handle.close()





import mindspore.dataset as ds
import mindspore.dataset.text as text

DATA_FILE = './datasets/tokenizer.txt'
dataset = ds.TextFileDataset(DATA_FILE, shuffle=False)

ds.config.set_seed(58)
dataset = dataset.shuffle(buffer_size=3)
for data in dataset.create_dict_iterator(output_numpy=True):
    print(text.to_str(data['text']))


print('='*30)


replace_op1 = text.RegexReplace("Beijing", "Shanghai")
replace_op2 = text.RegexReplace("北京", "上海")
dataset = dataset.map(operations=replace_op1)
dataset = dataset.map(operations=replace_op2)
for data in dataset.create_dict_iterator(output_numpy=True):###need to mark
    print(text.to_str(data['text']))


print('='*30)


tokenizer = text.WhitespaceTokenizer()

dataset = dataset.map(operations=tokenizer)

for data in dataset.create_dict_iterator(num_epochs=1,output_numpy=True):
    print(text.to_str(data['text']).tolist())

运行结果：

============================================================================

需要注意的一点是，如果将

dataset.create_dict_iterator(output_numpy=True)  改为

dataset.create_dict_iterator()

则会报错：

修改后的代码：

import os

os.system("rm -f ./datasets/tokenizer.txt")

if not os.path.exists('./datasets'):
    os.mkdir('./datasets')
file_handle=open('./datasets/tokenizer.txt',mode='w')
file_handle.write('Welcome to Beijing 
北京欢迎您！ 
我喜欢English! 
')
file_handle.close()





import mindspore.dataset as ds
import mindspore.dataset.text as text

DATA_FILE = './datasets/tokenizer.txt'
dataset = ds.TextFileDataset(DATA_FILE, shuffle=False)

ds.config.set_seed(58)
dataset = dataset.shuffle(buffer_size=3)
for data in dataset.create_dict_iterator(output_numpy=True):
    print(text.to_str(data['text']))


print('='*30)


replace_op1 = text.RegexReplace("Beijing", "Shanghai")
replace_op2 = text.RegexReplace("北京", "上海")
dataset = dataset.map(operations=replace_op1)
dataset = dataset.map(operations=replace_op2)
for data in dataset.create_dict_iterator():###need to mark
    print(text.to_str(data['text']))


print('='*30)


tokenizer = text.WhitespaceTokenizer()

dataset = dataset.map(operations=tokenizer)

for data in dataset.create_dict_iterator(num_epochs=1,output_numpy=True):
    print(text.to_str(data['text']).tolist())

报错信息：

WARNING: 'ControlDepend' is deprecated from version 1.1 and will be removed in a future version, use 'Depend' instead.
[WARNING] ME(5047:140238748528768,MainProcess):2021-07-11-02:12:43.597.916 [mindspore/ops/operations/array_ops.py:2302] WARN_DEPRECATED: The usage of Pack is deprecated. Please use Stack.
我喜欢English!
Welcome to Beijing
北京欢迎您！
==============================
Traceback (most recent call last):
  File "/tmp/pycharm_project_753/x.py", line 34, in <module>
    for data in dataset.create_dict_iterator():###need to mark
  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/dataset/engine/iterators.py", line 125, in __next__
    data = self._get_next()
  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/dataset/engine/iterators.py", line 169, in _get_next
    return {k: self._transform_tensor(t) for k, t in self._iterator.GetNextAsMap().items()}
  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/dataset/engine/iterators.py", line 169, in <dictcomp>
    return {k: self._transform_tensor(t) for k, t in self._iterator.GetNextAsMap().items()}
  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/dataset/engine/iterators.py", line 84, in <lambda>
    self._transform_tensor = lambda t: Tensor(t.as_array())
  File "/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/common/tensor.py", line 74, in __init__
    raise TypeError(f"For Tensor, the input_data is a numpy array, "
TypeError: For Tensor, the input_data is a numpy array, but it's data type is not in supported list: ['int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', 'uint32', 'uint64', 'float16', 'float32', 'float64', 'bool_'].

进程已结束，退出代码为 1

其原因就是如果设置output_numpy=True， 
那么输出的就是numpy类型数据，由于输入的是numpy类型数据，那么在内部进行数据处理时不对数据类型进行转换。

而如果设置output_numpy=False  (默认设置)， 
那么输出的就是Tensor类型数据，由于输入的是numpy类型数据，那么在内部进行数据处理时就需要对数据类型进行转换。

根据报错信息：
TypeError: For Tensor, the input_data is a numpy array, but it's data type is not in supported list: ['int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', 'uint32', 'uint64', 'float16', 'float32', 'float64', 'bool_'].

我们可以知道如果内部需要对数据类型转换的话，那么输入数据必须是以下类型：

['int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', 'uint32', 'uint64', 'float16', 'float32', 'float64', 'bool_'].

或者是可以转换为这些类型的数据，而我们调用 dataset.create_dict_iterator 时其内部输入的数据是 字符型（str）, 因此无法转换从而报错。






===============================================================




不过这里不得不吐槽以下MindSpore框架的报错信息写的真是很需要猜，不然真是看不懂，要是没有些经验的话这种报错信息也是难以懂的。