编程Tips集锦

以下是自己编程的一些小贴士，记录，总结提高自己。

1.python中集合类型的查找，尽量用dict or set类型。

dict和set类型，在python内部的实现都是使用hash映射，查找的时间复杂度是O(1)，比任何的查找算法都高效。

当在程序中使用到>1K次的查询，就应该开始考虑使用dict或set类型来进行数据的组织。

#coding:utf-8
from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
import string
import operator
import datetime

commonWords = ["the", "be", "and", "of", "a", "in", "to", "have", "it", "i", "that", "for", "you", "he", "with", "on", "do", "say", "this", "they", "is", "an", "at", "but","we", "his", "from", "that", "not", "by", "she", "or", "as", "what", "go", "their","can", "who", "get", "if", "would", "her", "all", "my", "make", "about", "know", "will","as", "up", "one", "time", "has", "been", "there", "year", "so", "think", "when", "which", "them", "some", "me", "people", "take", "out", "into", "just", "see", "him", "your", "come", "could", "now", "than", "like", "other", "how", "then", "its", "our", "two", "more", "these", "want", "way", "look", "first", "also", "new", "because", "day", "more", "use", "no", "man", "find", "here", "thing", "give", "many", "well"]
#若不注释，则为set类型，跑一遍程序，对比一下，则知优劣！
#commonWords = set(commonWords)

def isCommon(word):
    global commonWords
    if word in commonWords:
        return True
    return False


def cleanText(input):
    input = re.sub('
+', " ", input).lower()
    input = re.sub('[[0-9]*]', "", input)
    input = re.sub(' +', " ", input)
    input = re.sub("u.s.", "us", input)
    input = bytes(input, "UTF-8")
    input = input.decode("ascii", "ignore")
    return input

def cleanInput(input):
    input = cleanText(input)
    cleanInput = []
    input = input.split(' ')
    for item in input:
        item = item.strip(string.punctuation)
        if len(item) > 1 or (item.lower() == 'a' or item.lower() == 'i'):
            cleanInput.append(item)

    cleanContent = []
    for word in cleanInput:
        if not isCommon(word):
            cleanContent.append(word)
    return cleanContent

def getNgrams(input, n):
    input = cleanInput(input)
    output = {}
    for i in range(len(input)-n+1):
        ngramTemp = " ".join(input[i:i+n])
        if ngramTemp not in output:
            output[ngramTemp] = 0
        output[ngramTemp] += 1
    return output

def getFirstSentenceContaining(ngram, content):
    #print(ngram)
    sentences = content.split(".")
    for sentence in sentences:
        if ngram in sentence:
            return sentence
    return ""

content = str(urlopen("http://pythonscraping.com/files/inaugurationSpeech.txt").read(), 'utf-8')

print('Use the set as the format of common words.')
print('Begin:',datetime.datetime.now())
for i in range(50):
    ngrams = getNgrams(content, 2)
    sortedNGrams = sorted(ngrams.items(), key = operator.itemgetter(1), reverse = True)
print('End:',datetime.datetime.now())
print(sortedNGrams)

2.python往数据库插入数据

在插入数据之前，记得先进行一次查询，查看数据是否已经在数据库中。

一可以使程序更健壮，二也可顺便避免二次查询。

3.数据库在建表的时候，最后有索引

最近需要往数据库中插入上百万级的数据，十万级以后之后，数据库变得极慢，磁盘读写也是爆满！

后来，发现查询次数太多，重新建表，顺便加入索引。特别是unique index，我猜背后的实现机制是hash映射。

加入索引之后的数据库，大大减轻了磁盘的负担，查询速度几乎恒定，不过数据库的增大还是降低了读写的速度（实属情理之中）。

3.python字符串中转义字符的处理

python中 所占位为4位，不是通常的8位。