1.Avro介绍
- Avro是Hadoop中的一个子项目
- Avro是基于二进制传输高性能的中间件【Hbase和Hive的客户端与服务端的数据传输也采用该工具】
- Avro可以将数据进行序列化,适用远程和本地大批量数据的交互
- Avro可以支持对定义的数据结构(Schema)进行动态加载,提高性能
2.Avro特点
- 提供了丰富的数据结构类型,8种基本数据类型以及6种复杂数据类型
- 快速可压缩的二进制形式
- 提供容器文件用于持久化数据
- 远程过程调用RPC框架
入门demo
1.创建maven工程,导入pom依赖
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.blb</groupId> <artifactId>Avro</artifactId> <version>1.0-SNAPSHOT</version> <name>Avro</name> <!-- FIXME change it to the project's website --> <url>http://www.example.com</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.source>1.7</maven.compiler.source> <maven.compiler.target>1.7</maven.compiler.target> </properties> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.11</version> <scope>test</scope> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.avro/avro --> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> <version>1.8.2</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.avro/avro-tools --> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro-tools</artifactId> <version>1.8.2</version> </dependency> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro-maven-plugin</artifactId> <version>1.8.2</version> </dependency> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro-compiler</artifactId> <version>1.8.2</version> </dependency> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro-ipc</artifactId> <version>1.8.2</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.avro</groupId> <artifactId>avro-maven-plugin</artifactId> <version>1.8.2</version> <executions> <execution> <phase>generate-sources</phase> <goals> <goal>schema</goal> </goals> <configuration> <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory> <outputDirectory>${project.basedir}/src/main/java/</outputDirectory> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.8</source> <target>1.8</target> </configuration> </plugin> </plugins> <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) --> <plugins> <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle --> <plugin> <artifactId>maven-clean-plugin</artifactId> <version>3.1.0</version> </plugin> <!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging --> <plugin> <artifactId>maven-resources-plugin</artifactId> <version>3.0.2</version> </plugin> <plugin> <artifactId>maven-compiler-plugin</artifactId> <version>3.8.0</version> </plugin> <plugin> <artifactId>maven-surefire-plugin</artifactId> <version>2.22.1</version> </plugin> <plugin> <artifactId>maven-jar-plugin</artifactId> <version>3.0.2</version> </plugin> <plugin> <artifactId>maven-install-plugin</artifactId> <version>2.5.2</version> </plugin> <plugin> <artifactId>maven-deploy-plugin</artifactId> <version>2.8.2</version> </plugin> <!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle --> <plugin> <artifactId>maven-site-plugin</artifactId> <version>3.7.1</version> </plugin> <plugin> <artifactId>maven-project-info-reports-plugin</artifactId> <version>3.0.0</version> </plugin> </plugins> </pluginManagement> </build> </project>
2.在指定的目录下创建User.avsc文件
{ "namespace": "com.blb", "type": "record", "name": "User", "fields": [ {"name": "name","type": "string"}, {"name": "id","type": "int"}, {"name": "salary","type": "int"}, {"name": "age","type": "int"}, {"name": "address","type": "string"} ] }
3.使用avro-maven插件为avsc文件生成Java类
4.序列化示例
/** * 序列化测试 */ @Test public void write(){ // 初始化User对象 User user = new User("张三",1,6500,25,"云南"); User user1 = new User("李四",2,7000,20,"湖北"); DatumWriter<User> dw = new SpecificDatumWriter<>(User.class); DataFileWriter<User> dfw = new DataFileWriter<>(dw); //创建底层的文件输出通道 //schma - 序列化类的模式 //path - 文件路径 try { dfw.create(user.getSchema(),new File("G://hadoop//Avro//src//test//user.txt")); dfw.append(user); dfw.append(user1); dfw.close(); } catch (IOException e) { System.out.println("找不到指定文件"); } }
5.反序列化测试
/** * 反序列化测试 */ @Test public void read(){ DatumReader<User> dr = new SpecificDatumReader<>(User.class); try { DataFileReader dfr = new DataFileReader<User>(new File("G://hadoop//Avro//src//test//user.txt"), dr); // 通过迭代器的方式,迭代出对象数据 while(dfr.hasNext()){ System.out.println(dfr.next()); } } catch (IOException e) { System.out.println("找不到指定文件"); } }