基本思路：
每个评论取前200个单词。然后生成词汇表，利用词汇index标注评论（对 每条评论的前200个单词编号而已），然后使用LSTM做正负评论检测。
代码解读见【【【评论】】】！embedding层本质上是word2vec！！！在进行数据降维，但是不是所有的LSTM都需要这个，比如在图像检测mnist时候，就没有这层！
import tensorflow as tf from tensorflow.contrib.learn.python import learn from sklearn import metrics from sklearn.model_selection import train_test_split import numpy as np from sklearn.naive_bayes import GaussianNB import os from sklearn.feature_extraction.text import CountVectorizer from tensorflow.contrib.layers.python.layers import encoders from sklearn import svm import tflearn from tflearn.data_utils import to_categorical, pad_sequences from tflearn.datasets import imdb MAX_DOCUMENT_LENGTH = 200 EMBEDDING_SIZE = 50 n_words=0 def load_one_file(filename): x="" with open(filename) as f: for line in f: x+=line return x def load_files(rootdir,label): list = os.listdir(rootdir) x=[] y=[] for i in range(0, len(list)): path = os.path.join(rootdir, list[i]) if os.path.isfile(path): print "Load file %s" % path y.append(label) x.append(load_one_file(path)) return x,y def load_data(): x=[] y=[] x1,y1=load_files("../data/movie-review-data/review_polarity/txt_sentoken/pos/",0) x2,y2=load_files("../data/movie-review-data/review_polarity/txt_sentoken/neg/", 1) x=x1+x2 y=y1+y2 return x,y def do_rnn(trainX, testX, trainY, testY): global n_words # Data preprocessing # Sequence padding print "GET n_words embedding %d" % n_words trainX = pad_sequences(trainX, maxlen=MAX_DOCUMENT_LENGTH, value=0.) testX = pad_sequences(testX, maxlen=MAX_DOCUMENT_LENGTH, value=0.) # Converting labels to binary vectors trainY = to_categorical(trainY, nb_classes=2) testY = to_categorical(testY, nb_classes=2)
print trainX[:10]
print testX[:10]
# Network building net = tflearn.input_data([None, MAX_DOCUMENT_LENGTH]) net = tflearn.embedding(net, input_dim=n_words, output_dim=128) net = tflearn.lstm(net, 128, dropout=0.8) net = tflearn.fully_connected(net, 2, activation='softmax') net = tflearn.regression(net, optimizer='adam', learning_rate=0.001, loss='categorical_crossentropy') # Training model = tflearn.DNN(net, tensorboard_verbose=3) model.fit(trainX, trainY, validation_set=(testX, testY), show_metric=True, batch_size=32,run_id="maidou") def do_NB(x_train, x_test, y_train, y_test): gnb = GaussianNB() y_predict = gnb.fit(x_train, y_train).predict(x_test) score = metrics.accuracy_score(y_test, y_predict) print('NB Accuracy: {0:f}'.format(score)) def main(unused_argv): global n_words x,y=load_data() x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.4, random_state=0) vp = learn.preprocessing.VocabularyProcessor(max_document_length=MAX_DOCUMENT_LENGTH, min_frequency=1) vp.fit(x) x_train = np.array(list(vp.transform(x_train))) x_test = np.array(list(vp.transform(x_test))) n_words=len(vp.vocabulary_) print('Total words: %d' % n_words) do_NB(x_train, x_test, y_train, y_test) do_rnn(x_train, x_test, y_train, y_test) if __name__ == '__main__': tf.app.run()
负面的示例评论：
plot : two teen couples go to a church party , drink and then drive . they get into an accident . one of the guys dies , but his girlfriend continues to see him in her life , and has nightmares . what's the deal ? watch the movie and " sorta " find out . . . critique : a mind-fuck movie for the teen generation that touches on a very cool idea , but presents it in a very bad package . which is what makes this review an even harder one to write , since i generally applaud films which attempt to break the mold , mess with your head and such ( lost highway & memento ) , but there are good and bad ways of making all types of films , and these folks just didn't snag this one correctly . they seem to have taken this pretty neat concept , but executed it terribly . so what are the problems with the movie ? well , its main problem is that it's simply too jumbled . it starts off " normal " but then downshifts into this " fantasy " world in which you , as an audience member , have no idea what's going on . there are dreams , there are characters coming back from the dead , there are others who look like the dead , there are strange apparitions , there are disappearances , there are a looooot of chase scenes , there are tons of weird things that happen , and most of it is simply not explained . now i personally don't mind trying to unravel a film every now and then , but when all it does is give me the same clue over and over again , i get kind of fed up after a while , which is this film's biggest problem . it's obviously got this big secret to hide , but it seems to want to hide it completely until its final five minutes . and do they make things entertaining , thrilling or even engaging , in the meantime ? not really . the sad part is that the arrow and i both dig on flicks like this , so we actually figured most of it out by the half-way point , so all of the strangeness after that did start to make a little bit of sense , but it still didn't the make the film all that more entertaining . i guess the bottom line with movies like this is that you should always make sure that the audience is " into it " even before they are given the secret password to enter your world of understanding . i mean , showing melissa sagemiller running away from visions for about 20 minutes throughout the movie is just plain lazy ! ! okay , we get it . . . there are people chasing her and we don't know who they are . do we really need to see it over and over again ? how about giving us different scenes offering further insight into all of the strangeness going down in the movie ? apparently , the studio took this film away from its director and chopped it up themselves , and it shows . there might've been a pretty decent teen mind-fuck movie in here somewhere , but i guess " the suits " decided that turning it into a music video with little edge , would make more sense . the actors are pretty good for the most part , although wes bentley just seemed to be playing the exact same character that he did in american beauty , only in a new neighborhood . but my biggest kudos go out to sagemiller , who holds her own throughout the entire film , and actually has you feeling her character's unraveling . overall , the film doesn't stick because it doesn't entertain , it's confusing , it rarely excites and it feels pretty redundant for most of its runtime , despite a pretty cool ending and explanation to all of the craziness that came before it . oh , and by the way , this is not a horror or teen slasher flick . . . it's just packaged to look that way because someone is apparently assuming that the genre is still hot with the kids . it also wrapped production two years ago and has been sitting on the shelves ever since . whatever . . . skip it !
正面的：
films adapted from comic books have had plenty of success , whether they're about superheroes ( batman , superman , spawn ) , or geared toward kids ( casper ) or the arthouse crowd ( ghost world ) , but there's never really been a comic book like from hell before . for starters , it was created by alan moore ( and eddie campbell ) , who brought the medium to a whole new level in the mid '80s with a 12-part series called the watchmen . to say moore and campbell thoroughly researched the subject of jack the ripper would be like saying michael jackson is starting to look a little odd . the book ( or " graphic novel , " if you will ) is over 500 pages long and includes nearly 30 more that consist of nothing but footnotes . in other words , don't dismiss this film because of its source . if you can get past the whole comic book thing , you might find another stumbling block in from hell's directors , albert and allen hughes . getting the hughes brothers to direct this seems almost as ludicrous as casting carrot top in , well , anything , but riddle me this : who better to direct a film that's set in the ghetto and features really violent street crime than the mad geniuses behind menace ii society ? the ghetto in question is , of course , whitechapel in 1888 london's east end . it's a filthy , sooty place where the whores ( called " unfortunates " ) are starting to get a little nervous about this mysterious psychopath who has been carving through their profession with surgical precision . when the first stiff turns up , copper peter godley ( robbie coltrane , the world is not enough ) calls in inspector frederick abberline ( johnny depp , blow ) to crack the case . abberline , a widower , has prophetic dreams he unsuccessfully tries to quell with copious amounts of absinthe and opium . upon arriving in whitechapel , he befriends an unfortunate named mary kelly ( heather graham , say it isn't so ) and proceeds to investigate the horribly gruesome crimes that even the police surgeon can't stomach . i don't think anyone needs to be briefed on jack the ripper , so i won't go into the particulars here , other than to say moore and campbell have a unique and interesting theory about both the identity of the killer and the reasons he chooses to slay . in the comic , they don't bother cloaking the identity of the ripper , but screenwriters terry hayes ( vertical limit ) and rafael yglesias ( les mis ? rables ) do a good job of keeping him hidden from viewers until the very end . it's funny to watch the locals blindly point the finger of blame at jews and indians because , after all , an englishman could never be capable of committing such ghastly acts . and from hell's ending had me whistling the stonecutters song from the simpsons for days ( " who holds back the electric car/who made steve guttenberg a star ? " ) . don't worry - it'll all make sense when you see it . now onto from hell's appearance : it's certainly dark and bleak enough , and it's surprising to see how much more it looks like a tim burton film than planet of the apes did ( at times , it seems like sleepy hollow 2 ) . the print i saw wasn't completely finished ( both color and music had not been finalized , so no comments about marilyn manson ) , but cinematographer peter deming ( don't say a word ) ably captures the dreariness of victorian-era london and helped make the flashy killing scenes remind me of the crazy flashbacks in twin peaks , even though the violence in the film pales in comparison to that in the black-and-white comic . oscar winner martin childs' ( shakespeare in love ) production design turns the original prague surroundings into one creepy place . even the acting in from hell is solid , with the dreamy depp turning in a typically strong performance and deftly handling a british accent . ians holm ( joe gould's secret ) and richardson ( 102 dalmatians ) log in great supporting roles , but the big surprise here is graham . i cringed the first time she opened her mouth , imagining her attempt at an irish accent , but it actually wasn't half bad . the film , however , is all good . 2 : 00 - r for strong violence/gore , sexuality , language and drug content
pad后和category后的数据示例：
padded and cated data:
trainX 3条: [[ 1299 6 1 1596 26 354 155 1 62 101 537 252 5 22 2048 516 4 140 252 119 19 1 147 226 16 56 19 2 435 37 2 77 648 15 1 164 222 22 389 12 93 39 19392 235 16 189 83 1299 6 2 1426 453 7 976 1375 97 1 67 6 2928 42 2489 58 1 251 225 3 36 4 133 120 305 138 8 4730 244 1274 70 1018 4 49 14539 2290 947 3881 22772 1594 1296 67 1812 11 2663 9397 7513 3133 2 1619 8232 307 16958 4 2015 1329 1 16813 3571 4 869 3376 5 1019 41 7 518 33 598 7 1 1600 4 15406 1473 29 2 77 199 812 15956 21 33 1841 315 1852 371 5280 27 468 2663 343 2 334 11397 1619 5 1562 47 19 0 3 4239 11 100 10 234 219 10 0 0 8 30 4 220 144 1 414 4 3226 11120 3161 92 299 366 725 1010 27520 5 3343 76 7 1 1205 12 12549 1121 4 44 1 2195 9938 6 23 0 12 2663 6858 5 1425 19 2 1378] [ 1361 1 1647 4 1 4974 130 26 11041 1126 130 1232 1 57 26 7 269 641 5 205 3325 1053 3 5152 6318 622 2 5999 4 911 223 14 3772 5166 15739 6635 2036 633 1 2146 778 2697 327 9589 8311 3 3031 19 36 1 4974 8164 28 1 3103 4276 6344 27 618 2 4266 5 1 4203 1427 1199 1083 7 150 192 1 2294 3 15520 185 52 6 2 3689 572 4 6431 15520 6635 6 130 1232 2 5020 778 503 12 36 2805 4 1538 9333 4795 1518 4 25 405 1539 17927 6489 1427 6646 34 17491 13 3501 99 1232 5309 17 90 2 4074 1232 32 68 13660 162 5 2 7412 258 83 4 405 460 11 8238 12857 18618 3890 922 3915 3 146 32 5 488 10 2125 9736 5 2 2217 16298 3915 81 2529 48 1232 996 4 54 1053 522 18 157 9 410 24 25 4 23045 348 24 1535 35 1689 1 5410 1232 23995 3 4 220 9 340 41 1053 6 1391 18618 9608 16865 1232 24 272 6 681 7 0 100 20 109 642] [ 83 59 25 11208 9 371 3442 7 2 546 181 29 176 158 13 546 25133 3 13 1554 4819 20 25356 12 36 46 5311 1 1075 4 3442 169 31 134 5 75 11 1 98 44 104 22 6759 12 2 13377 235 1397 4 1 1948 826 26697 371 3442 1605 13 260 1364 12771 4462 7 2 429 1340 29 1 164 63 1142 7782 4587 1599 6 7 1 1758 1217 12 2 541 8661 1142 168 10363 541 3 9 588 33 5 826 37 1 546 4553 4 36 140 300 93 97 361 168 2 8661 28 2 1988 508 3 102 6 2524 5 7651 100 516 1180 20 4837 11 13791 5 1 67 8 115 245 529 391 109 2 821 78 578 198 715 5 103 1218 95 5 1 415 662 337 415 1605 337 1415 3 10 2 571 6 2311 2812 3 10809 3442 3 1599 245 2349 5 4402 87 4339 3 18 1422 6642 12 2 11316 8790 5 819 46 116 266 193 1599 32 1585 5 141 85 7 546 487 144 18 1537 3442 124 41 4 13]]
trainY 3条:
[[ 1. 0.] [ 0. 1.] [ 0. 1.]]
其中，MAX_DOCUMENT_LENGTH = 200，由于每个文档都进行了剪切。超过200的就直接截断文本，不再计算了！！！因为：
tf.contrib.learn.preprocessing.VocabularyProcessor (max_document_length, min_frequency=0, vocabulary=None, tokenizer_fn=None)参数：
max_document_length: 文档的最大长度。如果文本的长度大于最大长度，那么它会被剪切，反之则用0填充。
min_frequency: 词频的最小值，出现次数小于最小词频则不会被收录到词表中。
vocabulary: CategoricalVocabulary 对象。
tokenizer_fn：分词函数
代码：
from tensorflow.contrib import learn
import numpy as np
max_document_length = 4
x_text =[
'i love you',
'me too'
]
vocab_processor = learn.preprocessing.VocabularyProcessor(max_document_length)
vocab_processor.fit(x_text)
print next(vocab_processor.transform(['i me too'])).tolist()
x = np.array(list(vocab_processor.fit_transform(x_text)))
print x
[1, 4, 5, 0]
[[1 2 3 0]
[4 5 0 0]]