Java使用代理服务器

HTTP协议是基于TCP协议的,TCP协议在Java中的体现就是套接字.在了解HTTP协议的基础上,完全可以通过TCP来实现一套HTTP库,这个库可以发起网络请求和接受网络请求.只要能用URLConnection能实现的事情,用Socket同样能够实现.

代理是"代理服务器",我不直接向百度发起请求,而是向代理服务器发起请求,然后代理服务器代替我向百度发起请求.这样一来,我的IP地址就不会暴露.在编写爬虫时,同一个IP爬多了就会被要求输入验证码,这时就可以通过代理来不停地换IP从而规避验证码.

本文提供四种方法,第一种方法通过设置环境变量,第二种方法通过URLConnection,第三种方法通过nio中的SocketChannel,第四种方法通过Socket.

public class TestProxy {
    static String host = "112.126.65.26";
    static int port = 12345;
    static String url = "http://1212.ip138.com/ic.asp";
    public static void main(String[] args) throws Exception {
        four();
    }
    static void one() throws MalformedURLException, IOException {
        // 没有这句话是不行的
        System.setProperty("http.proxySet", "true");
        System.setProperty("http.proxyHost", host);
        System.setProperty("http.proxyPort", port + "");
        URLConnection connection = new URL(url).openConnection();
        show(connection.getInputStream());
    }
    static void two() throws MalformedURLException, IOException {
        SocketAddress addr = new InetSocketAddress(host, port);
        Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);
        // 下面这个网址会告诉你你的ip地址
        URLConnection connection = new URL(url).openConnection(proxy);
        show(connection.getInputStream());
    }
    // 使用socket也是一样
    static void three() throws IOException {
        SocketChannel sc = SocketChannel
                .open(new InetSocketAddress(host, port));
        sc.write(Charset.forName("utf8")
                .encode("GET " + url + " HTTP/1.1

"));
        ByteBuffer buffer = ByteBuffer.allocate(1024);
        while (sc.read(buffer) != -1) {
            buffer.flip();
            System.out.println(Charset.forName("utf8").decode(buffer));
            buffer.clear();
        }
        sc.close();
    }
    static void four() throws IOException {
        // 以下地址是代理服务器的地址
        Socket socket = new Socket(host, port);
        // 写与的内容就是遵循HTTP请求协议格式的内容，请求百度
        socket.getOutputStream().write(
                new String("GET " + url + " HTTP/1.1

").getBytes());
        show(socket.getInputStream());
        socket.close();
    }
    static void show(InputStream in) throws IOException {
        Scanner cin = new Scanner(in);
        StringBuilder builder = new StringBuilder();
        while (cin.hasNext()) {
            builder.append(cin.nextLine());
        }
        cin.close();
        Pattern pattern = Pattern
                .compile("\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}");
        Matcher matcher = pattern.matcher(builder.toString());
        matcher.find();
        System.out.println(matcher.group());
    }
}