沉淀，再出发：结合案例看python

沉淀，再出发：结合案例看python

一、前言

  关于python，如果不经过大型程序开发的洗礼，我们很难说自己已经懂得了python了，因此，我们需要通过稍微结构化的编程来学习python。

二、一个案例

   首先我们看一下需要具备的前提知识。

   2.1、新建pandas表格

man_num = 100
women_num = 100
pd.DataFrame( [['w'+str(i) for i in random.sample(range(1,women_num+1),women_num)] for j in range(man_num)],
                        index = ['m'+str(i) for i in range(1,man_num+1)],
                        columns = ['level'+str(i) for i in range(1,women_num+1)]
             )

    通过上述的方式，我们创建出了一个表格：

   其中random的sample方法，我们可以从下面的例子中理解，在上面就是挑选自身的一种置换来填充每一个单元，for j in range(man_num)其实就是重复多少行的意思：

import random

list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
for i in range(3):
    slice = random.sample(list, 5)  # 从list中随机获取5个元素，作为一个片断返回
    print(slice)
    print(list, '
')  # 原有序列并没有改变

      2.2、reset_index()和to_csv()

    def to_csv(self, path=None, index=True, sep=",", na_rep='',
               float_format=None, header=False, index_label=None,
               mode='w', encoding=None, compression=None, date_format=None,
               decimal='.'):
        """
        Write Series to a comma-separated values (csv) file
        Parameters
        ----------
        path : string or file handle, default None
            File path or object, if None is provided the result is returned as
            a string.
        na_rep : string, default ''
            Missing data representation
        float_format : string, default None
            Format string for floating point numbers
        header : boolean, default False
            Write out series name
        index : boolean, default True
            Write row names (index)
        index_label : string or sequence, default None
            Column label for index column(s) if desired. If None is given, and
            `header` and `index` are True, then the index names are used. A
            sequence should be given if the DataFrame uses MultiIndex.
        mode : Python write mode, default 'w'
        sep : character, default ","
            Field delimiter for the output file.
        encoding : string, optional
            a string representing the encoding to use if the contents are
            non-ascii, for python versions prior to 3
        compression : string, optional
            A string representing the compression to use in the output file.
            Allowed values are 'gzip', 'bz2', 'zip', 'xz'. This input is only
            used when the first argument is a filename.
        date_format: string, default None
            Format string for datetime objects.
        decimal: string, default '.'
            Character recognized as decimal separator. E.g. use ',' for
            European data
        """
        from pandas.core.frame import DataFrame
        df = DataFrame(self)
        # result is only a string if no path provided, otherwise None
        result = df.to_csv(path, index=index, sep=sep, na_rep=na_rep,
                           float_format=float_format, header=header,
                           index_label=index_label, mode=mode,
                           encoding=encoding, compression=compression,
                           date_format=date_format, decimal=decimal)
        if path is None:
           return result

     2.3、stack()和unstack()

     在用pandas进行数据重排时，经常用到stack和unstack两个函数。stack的意思是堆叠，堆积，unstack即“不要堆叠”。常见的数据的层次化结构有两种，一种是表格，一种是“花括号”，即下面这样的l两种形式：

 

       表格在行列方向上均有索引（类似于DataFrame），花括号结构只有“列方向”上的索引（类似于层次化的Series），结构更加偏向于堆叠（Series-stack，方便记忆）。stack函数会将数据从”表格结构“变成”花括号结构“，即将其行索引变成列索引，反之，unstack函数将数据从”花括号结构“变成”表格结构“，即要将其中一层的列索引变成行索引。

import numpy as np
import pandas as pd
from pandas import Series,DataFrame
data=DataFrame(np.arange(6).reshape((2,3)),index=pd.Index(['street1','street2']),columns=pd.Index(['one','two','three']))
print(data)
print('-----------------------------------------
')
data2=data.stack()
data3=data2.unstack()
print(data2)
print('-----------------------------------------
')
print(data3)

   2.4、DataFrame上的concat()

    2.5、案例

   有一座城市，当地风俗是，想结婚的男子必须先向心仪的女子求婚，而女子则需要等待求婚。
   牧师每年会邀请人数相同的适婚男女参与一次集体相亲。一次相亲活动可能有很多轮，男子会首先向自己最爱的女子求婚，女子则会在所有的追求者中选择她的最爱；
     如果男子被拒绝，下一轮会向他第二喜欢的女子求婚；上一轮已经订婚的女子如果得到她更爱的人的求婚，则会毫不留情地抛弃未婚夫，和更爱的人在一起。
     被抛弃的男子需要重新参与求婚。如此反复，等大家都订婚，就举办集体婚礼。    
    假设：
    1)参加求婚的男女数量保持一致
    2)每个男子都按喜爱程度对女子进行排序，比如最爱a，其次爱b，再次爱c
    3)每个女子也同样给每个男子排序 

    2.6、安装 pyecharts和pyecharts_snapshot

    2.7、创造样例

import pandas as pd
import random

from configuration import MAN_NUM,WOMAN_NUM


def create_sample():
    man_num = MAN_NUM
    women_num = WOMAN_NUM

    #设置男女生喜好样本
    print('==============================生成样本数据==============================')
    man = pd.DataFrame( [['w'+str(i) for i in random.sample(range(1,women_num+1),women_num)] 
                          for i in range(man_num)],
                        index = ['m'+str(i) for i in range(1,man_num+1)],
                        columns = ['level'+str(i) for i in range(1,women_num+1)]
                        )

    women = pd.DataFrame( [['m'+str(i) for i in random.sample(range(1,man_num+1),man_num)] 
                          for i in range(women_num)],
                        index = ['w'+str(i) for i in range(1,women_num+1)],
                        columns = ['level'+str(i) for i in range(1,man_num+1)]
                        )
    return (man,women)

if __name__ == '__main__':
    create_sample(man,women)

   2.8、生成映射表

import pandas as pd

#设置姻缘关系表
def create_mapping_table(man,women):

    man_ismapping = pd.DataFrame({
            'man_id':man.index,
            'target':'n',
            'love_level':0,
            'range':0
            }).set_index('man_id')

    women_ismapping = pd.DataFrame({
            'women_id':women.index,
            'target':'n',
            'love_level':0,
            'range':0
            }).set_index('women_id')
    return (man_ismapping,women_ismapping)

if __name__ == '__main__':
    create_mapping_table(man,women)

    2.9、创建目录，完成初始化

from configuration import  TEST_NUM
import os
from scr.create_sample import create_sample
from scr.create_mapping_table import create_mapping_table
import pandas as pd

def loop_script():
    test_num = TEST_NUM
    if not os.path.exists('./data'):
        os.makedirs('./data')

    for i in range(1,test_num+1):
        print('==============================开始创建测试文件夹{}=============================='.format(i))
        path = './data/test' + str(i)
        if not os.path.exists(path):
            os.makedirs(path)
        sample_data = create_sample()
        man = sample_data[0]
        women = sample_data[1]
        man.reset_index().to_csv(path+'/'+'man_sample.csv', index=0)
        women.reset_index().to_csv(path+'/'+'woman_sample.csv', index=0)
        print('==============================样本数据生成成功==============================')
        print('==============================创建婚姻关系表==============================')
        man = pd.read_csv(path+'/'+'man_sample.csv').set_index('index')
        women = pd.read_csv(path+'/'+'woman_sample.csv').set_index('index')
        mapping_data = create_mapping_table(man, women)
        man_ismapping = mapping_data[0]
        women_ismapping = mapping_data[1]
        man_ismapping.reset_index().to_csv(path+'/'+'man_ismapping.csv', index=0)
        women_ismapping.reset_index().to_csv(path+'/'+'women_ismapping.csv', index=0)
        print('==============================婚姻关系表创建完成==============================')

   2.10、业务逻辑计算

import pandas as pd
from configuration import  TEST_NUM

def calculation():
    test_num =  TEST_NUM
    for i in range(1,test_num+1):
        path = './data/test' + str(i)
        man = pd.read_csv(path + '/' + 'man_sample.csv').set_index('index')
        women = pd.read_csv(path + '/' + 'woman_sample.csv').set_index('index')
        man_ismapping = pd.read_csv(path + '/' + 'man_ismapping.csv').set_index('man_id')
        women_ismapping = pd.read_csv(path + '/' + 'women_ismapping.csv').set_index('women_id')
        print('==============================测试集{}模拟开始=============================='.format(i))
        print('==============================开始模拟求婚过程==============================')
        level_num = 0
        while man_ismapping['love_level'].min() == 0:
            level_num += 1
            print('==============================开始第{}天婚姻配对=============================='.format(level_num))
            u_mapping_man = man_ismapping[man_ismapping.target == 'n'].index.tolist()

            if level_num < 2:
                level_col = 'level' + str(level_num)
                man_choose = man[man.index.isin(u_mapping_man)][level_col].to_frame().reset_index()
                man_choose.columns = ['man_id', 'women_id']
                man_choose['range'] = 1
            else:
                m_id = u_mapping_man
                l = []
                for man_id in m_id:
                    col_n = int(man_ismapping[man_ismapping.index == man_id].range[0])
                    level_col = 'level' + str(col_n + 1)
                    women_id = man[man.index == man_id][level_col][0]
                    rg = col_n + 1
                    l.append([man_id, women_id, rg])
                man_choose = pd.DataFrame(l, columns=['man_id', 'women_id', 'range'])

            for r in range(0, len(man_choose)):
                relationship = man_choose[man_choose.index == r]
                m = [i for i in relationship['man_id']][0]
                w = [i for i in relationship['women_id']][0]
                find = women[women.index == w].unstack().reset_index()
                find.columns = ['level', 'women_id', 'man_id']
                find = int([i for i in find[find['man_id'] == m]['level']][0].split('level')[1])
                o_love_level = [i for i in women_ismapping[women_ismapping.index == w]['love_level']][0]
                rg = [i for i in relationship['range']][0]
                if o_love_level == 0:
                    women_ismapping.loc[w, 'love_level'] = find
                    women_ismapping.loc[w, 'target'] = m
                    women_ismapping.loc[w, 'range'] = level_num
                    man_ismapping.loc[m, 'love_level'] = rg
                    man_ismapping.loc[m, 'target'] = w
                    man_ismapping.loc[m, 'range'] = rg
                elif o_love_level > find:
                    m_o = women_ismapping.loc[w, 'target']
                    man_ismapping.loc[m_o, 'love_level'] = 0
                    man_ismapping.loc[m_o, 'target'] = 'n'
                    man_ismapping.loc[m, 'love_level'] = rg
                    man_ismapping.loc[m, 'target'] = w
                    man_ismapping.loc[m, 'range'] = rg
                    women_ismapping.loc[w, 'love_level'] = find
                    women_ismapping.loc[w, 'target'] = m
                    women_ismapping.loc[w, 'range'] = level_num
                else:
                    man_ismapping.loc[m, 'range'] = rg
                    pass

        print('==============================婚姻配对完成==============================')
        print('共进行了{}次牵线搭桥，在第{}天举办集体婚礼。'.format(level_num, level_num + 1))

        print('==============================导出配对明细表==============================')
        man_love_level_mean = man_ismapping.love_level.mean()
        women_love_level_mean = women_ismapping.love_level.mean()
        detail = [[level_num,level_num,man_love_level_mean,women_love_level_mean]]
        pd.DataFrame(detail,columns=['match_range','party_time','man_love_level_mean','women_love_level_mean'])
                .to_csv(path+'/'+'test_detail.csv', index=0)
        man_ismapping.reset_index().to_csv(path+'/'+'man_match_table.csv', index=0)
        women_ismapping.reset_index().to_csv(path+'/'+'woman_match_table.csv', index=0)
        print('==============================导出完毕==============================')

if __name__ == '__main__':
    calculation()

   2.11、绘制图表

import pandas as pd
from configuration import  TEST_NUM
from pyecharts import Bar
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings("ignore")

def drawing():
    test_num =  TEST_NUM
    l_table = []
    for i in range(1,test_num+1):
        path = './data/test' + str(i)
        man_match_table = pd.read_csv(path + '/' + 'man_match_table.csv').set_index('man_id')
        woman_match_table = pd.read_csv(path + '/' + 'woman_match_table.csv').set_index('women_id')

        man_match_table = man_match_table.groupby('love_level').count()['range'].sort_values(ascending=False)
        woman_match_table = woman_match_table.groupby('love_level').count()['range'].sort_values(ascending=False)

        man_attr = man_match_table.index.tolist()
        man_v = man_match_table.values.tolist()
        bar_man = Bar('男生匹配对象喜爱程度分布',width=900,height=500)
        bar_man.add('频数',man_attr,man_v,mark_line=["max"],label_color = ['#9932CC'])
        bar_man.render(path + '/' + '男生匹配对象喜爱程度分布.html')

        women_attr = woman_match_table.index.tolist()
        women_v = woman_match_table.values.tolist()
        bar_women = Bar('女生匹配对象喜爱程度分布',width=900,height=500)
        bar_women.add('频数',women_attr,women_v,mark_line=["max"],label_color = ['#FF3030'])
        bar_women.render(path + '/' + '女生匹配对象喜爱程度分布.html')

        detail = pd.read_csv(path + '/' + 'test_detail.csv')
        l_table.append(detail)
        detail_table = pd.concat(l_table)
        s_match_range = detail_table['match_range']
        s_man_love_level_mean = detail_table['man_love_level_mean']
        women_love_level_mean = detail_table['women_love_level_mean']
    fig = plt.figure(figsize=(10, 6), facecolor='gray')
    ax1 = fig.add_subplot(2, 2, 1)
    ax1.hist(s_match_range,bins = 20,
                        histtype = 'bar',
                        align = 'mid',
                        orientation = 'vertical',
                        alpha=0.5,
                        normed =False
                        )
    plt.grid(True)
    ax2 = fig.add_subplot(2, 2, 2)
    ax2.hist(s_man_love_level_mean, bins=20,
                         histtype='bar',
                         align='mid',
                         orientation='vertical',
                         alpha=0.5,
                         normed=False
                         )
    plt.grid(True)
    ax3 = fig.add_subplot(2, 2, 3)
    ax3.hist(women_love_level_mean, bins=20,
                         histtype='bar',
                         align='mid',
                         orientation='vertical',
                         alpha=0.5,
                         normed=False
                         )
    plt.grid(True)
    plt.savefig('./data/detail_graph.png',dpi=400)
    plt.show()
    print('==========================制图完成==========================')

     2.12、配置文件和程序入口

#基本参数配置

#MAN_NUM:男生样本数量
#WOMAN_NUM：女生样本数量
#TEST_NUM：测试次数


MAN_NUM = 100
WOMAN_NUM = 100
TEST_NUM = 50

from scr.loop_script import loop_script
from scr.calculation import calculation
from scr.drawing import drawing

if __name__ == '__main__':
    loop_script()
    calculation()
    drawing()
    print('========================================任务结束========================================')

三、总结

      通过一个案例，我们更好的理解了python的相关语法和应用，以及结合其他工具的强大的绘图能力和表达能力，对语言的高度浓缩和精炼等等。

  参考资料：https://mp.weixin.qq.com/s?__biz=MzAxMjUyNDQ5OA==&mid=2653557526&idx=1&sn=fa6afe368f3395afc80758f88c81868f&chksm=806e3dabb719b4bd452fe6f2389a56966a65e8e542d34ca475fc8061ace1a591f8ce27be6e8e&mpshare=1&scene=23&srcid=1016fiPYJuCJ4b33lW5V0jpx#rd