大数据背景下互联网用户行为分析

要求：

编一个程序就是要能捕捉后一个上网期间做的事情，比如他浏览百度页面3分钟，然后浏览新浪6分钟，然后下面还浏览了其他网页等等，能用程序捕捉到他所有的上网行为

准备工作：
查看需要可能用到的包：

pynput.mouse：包含控制和监控鼠标或者触摸板的类。

pynput.keyboard：包含控制和监控键盘的类。

　　

鼠标事件监听器是一个线程，所有的回调函数都会在独立的线程中运行。

调用pynput.mouse.Listener.stop，发起StopException异常，或者回调函数中返回False都会停止事件的监听。

对鼠标的操作：

#!/usr/bin/env python3
#-*- coding:utf-8 -*-
'''
Administrator 
2018/8/16 
'''

from pynput.mouse import Button, Controller
import time

mouse = Controller()
print(mouse.position)
time.sleep(3)
print('The current pointer position is {0}'.format(mouse.position))


#set pointer positon
mouse.position = (277, 645)
print('now we have moved it to {0}'.format(mouse.position))

#鼠标移动（x,y）个距离
mouse.move(5, -5)
print(mouse.position)

mouse.press(Button.left)
mouse.release(Button.left)

#Double click
mouse.click(Button.left, 1)

#scroll two  steps down
mouse.scroll(0, 500)

对鼠标行为的监控：

from pynput import mouse

def on_move(x, y):
    print('Pointer moved to {0}'.format(
        (x, y)))

def on_click(x, y, button, pressed):
    print('{0} at {1}'.format(
        'Pressed' if pressed else 'Released',
        (x, y)))
    if not pressed:
        # Stop listener
        return False

def on_scroll(x, y, dx, dy):
    print('Scrolled {0} at {1}'.format(
        'down' if dy < 0 else 'up',
        (x, y)))

# Collect events until released
with mouse.Listener(
        on_move=on_move,
        on_click=on_click,
        on_scroll=on_scroll) as listener:
    listener.join()

处理鼠标监听器错误

from pynput import mouse

class MyException(Exception): pass

def on_click(x, y, button, pressed):
    if button == mouse.Button.left:
        raise MyException(button)

# Collect events until released
with mouse.Listener(
        on_click=on_click) as listener:
    try:
        listener.join()
    except MyException as e:
        print('{0} was clicked'.format(e.args[0]))

键盘事件监听器是一个线程，所有的回调函数都会在独立的线程中运行。

调用pynput.keyboard.Listener.stop，发起StopException异常，或者回调函数中返回False都会停止事件的监听。

传递给回调函数的key参数是一个pynput.keyboard.Key类的实例。当特殊按键和普通按键一起按下时，数字字母按键的值会被放置在pynput.keyboard.KeyCode类的实例中，对于不知道的按键会返回None。

Controlling the keyboard 控制键盘

from pynput.keyboard import Key, Controller

keyboard = Controller()

# Press and release space
keyboard.press(Key.space)
keyboard.release(Key.space)

# Type a lower case A; this will work even if no key on the
# physical keyboard is labelled 'A'
keyboard.press('a')
keyboard.release('a')

# Type two upper case As
keyboard.press('A')
keyboard.release('A')
with keyboard.pressed(Key.shift):
    keyboard.press('a')
    keyboard.release('a')

# Type 'Hello World' using the shortcut type method
keyboard.type('Hello World')

Monitoring the keyboard 监控键盘

from pynput import keyboard

def on_press(key):
    try:
        print('alphanumeric key {0} pressed'.format(
            key.char))
    except AttributeError:
        print('special key {0} pressed'.format(
            key))

def on_release(key):
    print('{0} released'.format(
        key))
    if key == keyboard.Key.esc:
        # Stop listener
        return False

# Collect events until released
with keyboard.Listener(
        on_press=on_press,
        on_release=on_release) as listener:
    listener.join()

处理键盘监听器错误

from pynput import keyboard

class MyException(Exception): pass

def on_press(key):
    if key == keyboard.Key.esc:
        raise MyException(key)

# Collect events until released
with keyboard.Listener(
        on_press=on_press) as listener:
    try:
        listener.join()
    except MyException as e:
        print('{0} was pressed'.format(e.args[0]))

利用python实现查看浏览器历史记录

#coding:utf8
'''
Created on 2018年8月16日

@author: Administrator
'''
#统计浏览器访问历史记录
#se://version/ 用于查看浏览器文件存储地址


import os  
import sqlite3  
import operator  
from collections import OrderedDict  
import matplotlib.pyplot as plt  

def parse(url):  
    try:  
        parsed_url_components = url.split('//')  
        sublevel_split = parsed_url_components[1].split('/', 1)  
        domain =sublevel_split[0].replace("www.", "")  
        return domain
    except IndexError:  
        print('URL format error!') 

def analyze(results):  
    prompt =input("[.] Type <c> to print or <p> to plot
[>] ")

    if prompt == "c":
        with open('./history.txt','w') as f:
            for site, count in sites_count_sorted.items():
                f.write(site+'	'+str(count)+'
')
    elif prompt == "p":
        key=[]
        value=[]
        for k,v in results.items():
            key.append(k)
            value.append(v)
        n=25
        X=range(n)
        Y=value[:n]
        plt.bar(X,Y,align='edge')
        plt.xticks(rotation=45)  
        plt.xticks(X,key[:n])
        for x,y in zip(X,Y):
            plt.text(x+0.4, y+0.05,y, ha='center', va= 'bottom')
        plt.show()
    else:  
        print("[.] Uh?")  
        quit()  

if __name__=='__main__':
    #path to user's history database (Chrome)  
    data_path=r"D:360浏览器360se6User DataDefault"
    files=os.listdir(data_path)
    #"D:360浏览器360se6User DataDefaultHistory"
    history_db = os.path.join(data_path, 'History')
    print(history_db)

    #querying the db  
    c = sqlite3.connect(history_db)  
    cursor = c.cursor()  
    select_statement = "SELECT urls.url, urls.visit_count FROM urls, visits WHERE urls.id = visits.url;"  
    cursor.execute(select_statement)  

    results = cursor.fetchall() #tuple  

    sites_count = {} #dict makes iterations easier :D  

    for url, count in results:  
        url = parse(url)  
        if url in sites_count:  
            sites_count[url] += 1  
        else:  
            sites_count[url] = 1  

    sites_count_sorted = OrderedDict(sorted(sites_count.items(), key=operator.itemgetter(1), reverse=True))  

    analyze (sites_count_sorted)  

实现原理，找到浏览器浏览历史的保存的SQLit数据文件。利用代码读取数据，并对数据进行处理，加工

#-*- utf-8 -*-


# 已经可以在这个上面编写了

# 统计浏览器访问历史记录
# se://version/ 用于查看浏览器文件存储地址
# matplotlib  这是用来显示数据的包   一般都是科学计算用的。估计你用的会多点。你可以好好学习一下  还有一个包 numpy
#

import os  # 访问系统的包
import sqlite3  # 链接数据库文件胡包
import operator
from collections import OrderedDict
import matplotlib.pyplot as plt
import re

def parse(url):
    try:
        parsed_url_components = url.split('//')
        sublevel_split = parsed_url_components[1].split('/', 1)
        domain = sublevel_split[0].replace("www.", "")
        return domain
    except IndexError:
        print('URL format error!')
def filter_data(url):
    try:
        parsed_url_components = url.split('//')
        sublevel_split = parsed_url_components[1].split('/', 1)
        data=re.search('w+.(com|cn|net|tw|la|io|org|cc|info|cm|us|tv|club|co|in)',sublevel_split[0])
        if data:
            return data.group()
        else:
            yuming_count.add(sublevel_split[0])
            return "ok"
    except IndexError:
        print('URL format error!')

def analyze(results):
    prompt = input("[.] Type <c> to print or <p> to plot
[>] ")

    if prompt == "c":
        with open('./history.txt', 'w') as f:
            for site, count in sites_count_sorted.items():
                f.write(site + '	' + str(count) + '
')
    elif prompt == "p":
        key = []
        value = []
        for k, v in results.items():
            key.append(k)
            value.append(v)
        n = 25
        X = range(n)
        Y = value[:n]
        plt.bar( Y,X, align='edge')
        plt.xticks(rotation=45)
        plt.xticks(X, key[:n])
        for x, y in zip(X, Y):
            plt.text(x + 0.4, y + 0.05, y, ha='center', va='bottom')
        plt.show()
    else:
        print("[.] Uh?")
        quit()
def analyze2(results):
    print("我一看就知道你要打印折线图")
    key=[]
    value=[]
    for k,v in results.items():
        key.append(k)
        value.append(v)
    n = 20
    X = key[:n]
    Y = value[:n]

    plt.plot(Y,X,label="number count")
    plt.xticks(rotation=45)
    plt.xlabel('numbers')
    plt.ylabel('webname')
    plt.title('number count')
    plt.show()

if __name__ == '__main__':
    # 先运行起来吧
    #  这是查看360的。

    history_db = r"C:UsersAdministratorDesktopHistory"
    # files=os.listdir(data_path)

    # history_db = os.path.join(data_path, 'History')
    # print(history_db)

    # querying the db
    c = sqlite3.connect(history_db)
    cursor = c.cursor()
    select_statement = "SELECT urls.url, urls.visit_count FROM urls, visits WHERE urls.id= visits.url;"
    cursor.execute(select_statement)

    results = cursor.fetchall() # tuple

    sites_count = {}  # dict makes iterations easier :D
    yuming_count=set()#创建一个空的集合，用来收集已经存在国际域名
    for url, count in results:
        url= filter_data(url)
        if url in sites_count:
            sites_count[url] += 1
        else:
            sites_count[url] = 1
    print(yuming_count)
    # print(sites_count)
    del sites_count["ok"]
    sites_count_sorted = OrderedDict(sorted(sites_count.items(), key=operator.itemgetter(1), reverse=True))
    #
    # # analyze(sites_count_sorted)
    analyze2(sites_count_sorted)

---

利用python编程实现对浏览器接口的监听，实时查看用户访问的网站和数据

---

 

https://blog.csdn.net/xuanhun521/article/details/51779292 

Python黑客编程3网络数据监听和过滤

课程的实验环境如下：

•      操作系统：window7

•      编程工具：pycharm IDE

•      Python版本：3.6.4

•      涉及到的主要python模块：pypcap，dpkt,scapy，scapy-http

---

https://blog.csdn.net/sinat_22659313/article/details/53420492

大数据背景下互联网用户行为分析