团队项目开发冲刺日（三）

　　今日一整天没有学习团队项目相关知识，仅仅读了这篇博客：https://www.cnblogs.com/andzhang/p/6075814.html。

　　完成了昨天老师布置的词云任务，下面附上相关代码：

python:爬取，存入数据库

import re
import requests
from bs4 import BeautifulSoup
import pymysql

num = 0;
def insertdata(data,n):
    conn=pymysql.connect("localhost","root","Inazuma","paqu",charset='utf8')
    cur=conn.cursor()
    sql="INSERT INTO lunwen(Timu,Zhaiyao,Lianjie) VALUES(%s,%s,%s)"
    try:
        cur.execute(sql,data)
        conn.commit()
        print(n)
    except:
        conn.rollback()
        print("ERROR")
    conn.close()

#网址，这个网站提供了CVPR论文的题目，链接信息，还有pdf格式的论文全文
r = requests.get('http://openaccess.thecvf.com/ICCV2019.py')
soup = BeautifulSoup(r.text,'lxml')
#简单的按条件寻找标签
for item in soup.find_all('a',href=re.compile('content_ICCV_2019/html/')):
    num=num+1
    timu = item.string
    lianjie = item['href']
    #这里拿出的'href'没有http那段开头，想再读取链接进入网址，我在前面拼了上去，这个网址里可以获取摘要信息
    ra = requests.get('http://openaccess.thecvf.com/'+item['href'])
    soupa = BeautifulSoup(ra.text, 'lxml')
    diva = soupa.find(attrs={"id": "abstract"})
    #根据给出的BUG信息，网址有的论文点进去发生了Not Found错误，需要加个判断语句
    if(diva == None):
        zhaiyao=-1
    else:
        zhaiyao=diva.string
    insertdata((timu,zhaiyao,lianjie),num)

java代码：读库，根据关键字分类，界面显示：

后台servlet+dao层：

CVPRServlet，读库，统计分类，由于使用Eclipse创建Servlet文件都是统一样式，上面的就不粘了，只粘doPost：

protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        // TODO Auto-generated method stub
        request.setCharacterEncoding("utf-8");
        response.setContentType("text/html;charset=UTF-8");
        Map<String,Integer>sortMap=CloudDao.getallmax();
        JSONArray json=new JSONArray();
        int k=0;
        for(Map.Entry<String, Integer>entry:sortMap.entrySet()) {
            JSONObject job=new JSONObject();
            job.put("Kname", entry.getKey());
            job.put("Tvalue", entry.getValue());
            if(!entry.getKey().equals("for")||entry.getKey().equals("and")||entry.getKey().equals("With")||entry.getKey().equals("of")
                ||entry.getKey().equals("in")||entry.getKey().equals("From")||entry.getKey().equals("A")||entry.getKey().equals("to")
                ||entry.getKey().equals("a")||entry.getKey().equals("the")||entry.getKey().equals("by")) {
                json.add(job);
                k++;
            }
            if(k==30) {
                break;
            }
        }
        response.getWriter().write(json.toString());
    }

读取的Map在这里；

package Dao;

import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.Map;

import util.DBUtil;

public class CloudDao {
    public static Map<String, Integer>getallmax(){
        String sql="select * from lunwen";
        Map<String, Integer>map=new HashMap<String, Integer>();
        Connection conn=null;
        Statement state=null;
        ResultSet res=null;
        conn=DBUtil.getConn();
        try {
            state=conn.createStatement();
            res=state.executeQuery(sql);
            while(res.next()) {
                String timu=res.getString("Timu");
                String[]keyword=timu.split(" ");
                for(int i=0; i<keyword.length; i++) {
                    if(map.get(keyword[i])==null) {
                        map.put(keyword[i], 0);
                    }else {
                        map.replace(keyword[i], map.get(keyword[i])+1);
                    }
                }
            }
        }catch(Exception e) {
            e.printStackTrace();
        }
        DBUtil.close(res, state, conn);
        Map<String, Integer>sorted=new LinkedHashMap<>();
        map.entrySet().stream()
           .sorted(Map.Entry.<String,Integer>comparingByValue().reversed())
           .forEachOrdered(x->sorted.put(x.getKey(), x.getValue()));
        return sorted;
    }
}

先将库中的论文名提取出来，分开（是一段句子），按单词作为Key，统计出现个数，排序，传到CVPRServlet，最后再取前30个。

findServlet：点击词云某词时显示所有包含这个词的论文题目的题目以及链接，同样只粘贴doPost：

protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        // TODO Auto-generated method stub
        request.setCharacterEncoding("utf-8");
        response.setContentType("text/html;charset=UTF-8");
        String word=request.getParameter("word");
        response.getWriter().write(TitleDao.searchTitle(word));
    }

dao层代码：按词模糊查询，返回json数组：

package Dao;

import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;

import net.sf.json.JSONArray;
import net.sf.json.JSONObject;
import util.DBUtil;

public class TitleDao {
    public static String searchTitle(String word) {
        JSONArray jsonarrayC=new JSONArray();
        Connection con=DBUtil.getConn();
        Statement state=null;
        //读取数据库
        String sql="select Timu from lunwen where Timu like '%"+word+"%'";
        String timustr="";
        ResultSet res=null;
        try {
            JSONObject jsonobC=new JSONObject();
            state=con.createStatement();
            res=state.executeQuery(sql);
            while(res.next()) {
                timustr=timustr+res.getString("Timu")+"，";
            }
            res.close();
            String str[]=timustr.split("，");
            for(int i=0;i<str.length;i++) {
                sql="select Lianjie from lunwen where Timu='"+str[i]+"'";
                res=state.executeQuery(sql);
                res.next();
                String lianjie=res.getString("Lianjie");
                String Rlianjie="http://openaccess.thecvf.com/"+lianjie;
                jsonobC.put("Title", str[i]);
                jsonobC.put("Link", Rlianjie);
                res.close();
                //写入JSONArray数组
                jsonarrayC.add(jsonobC);
            }
        }catch(Exception e) {
            e.printStackTrace();
        }
        //类型转换，返回
        return jsonarrayC.toString();
    }
}

最后是界面代码，参考博客：https://www.cnblogs.com/xiaofengzai/p/12702136.html

<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>词云</title>
<script type="text/javascript" src="js/jquery-1.11.0.min.js"></script>
<script src="js/echarts.js"></script>
<script src="js/echarts-wordcloud.js"></script>
</head>
<body>
<div id='show'style="display:block;overflow:auto;700px;height:1000px;float:left"></div>
<div id='main'style="700px;height:1000px;float:left"></div>
<script type="text/javascript">
$(function(){
    echartsCloud();
});

function eConsole(param){
    if(typeof param.seriesIndex == 'undefined'){
        return;
    }
    if(param.type=='click'){
        var word=param.name;
        var htmltext="<table style='text-align:center'><caption style='text-align:center'>论文题目与链接</caption>"
        $.post(
                'findServlet',
                {'word':word},
                function(result){
                    json=JSON.parse(result);
                    for(var i=0;i<json.length;i++){
                        htmltext+="<tr><td><a target='_blank'href='"+json[i].Link+"'>"+json[i].Title+"</a></td></tr>"
                    }
                    htmltext+="</table>"
                    $("#show").html(htmltext);
                }
                )
    }
}
    function echartsCloud(){
        $.ajax({
            url:"CVPRServlet",
            type:"post",
            datatype:"json",
            async:true,
            success:function(data){
                var json=JSON.parse(data);
                var mydata=[];
                size=json.length;
                for(i=0;i<size;i++){
                    mydata.push({name:json[i].Kname,value:json[i].Tvalue});
                }
                var myChart=echarts.init(document.getElementById('main'));
                var ecConfig=echarts.config;
                myChart.on('click',eConsole);
                var option={
                    title:{
                        text:'词云'
                    },
                    tooltip:{
                        show:true
                    },
                    series:[{
                        type:'wordCloud',
                        shape:'smooth',
                        gridSize:8,
                        size:['50%','50%'],
                        rotationRange:[-45,0,45,90],
                        textStyle:{
                            normal:{
                                fontFamily:'微软雅黑',
                                color:function(){
                                    return 'rgb('+Math.round(Math.random()*255)+','
                                            +Math.round(Math.random()*255)+','
                                            +Math.round(Math.random()*255)+')'
                                }
                            },
                            emphasis:{
                                shadowBlur:5,
                                shadowColor:'#333'
                            }
                        },
                        left:'center',
                        top:'center',
                        right:null,
                        bottom:null,
                        '100%',
                        height:'100%',
                        data:mydata
                    }]
                };
                myChart.setOption(option);
            }
        });
    }
</script>
</body>
</html>

接下来看看效果：

 明天任务：补上昨天预定的计划，接下来应该没有其他代码作业了。