The above diagram consists of different components. Let’s take a look at each of these components in detail.
DrillBit: Apache Drill consists of a Daemon service called the DrillBit.
It is responsible for accepting requests from the client, processing queries, and returning results to the client.
There is no master-slave concept in DrillBit.
SQL Parser: The SQL parser parses all the incoming queries based on the open source framework called Calcite.
Logical Plan: A Logical plan describes the abstract data flow of a query.
Once a query is parsed into a logical plan, the Drill optimizer determines the most efficient execution plan using a variety of rule-based and cost-based techniques, translating the logical plan into a physical plan.
Optimizer: Apache Drill uses various database optimizations such as rule based/cost based, as well as other optimization rules exposed by the storage engine to re-write and split the query.
The output of the optimizer is a distributed physical query plan.
Optimization in Drill is pluggable so you can provide rules for optimization at various parts of the query execution.
Physical Plan: A Physical plan is also called as the execution plan.
It represents the most efficient and fastest way to execute the query across the different nodes in the cluster.
The physical plan is a DAG (directed acyclic graph) of physical operators, and each parent-child relationship implies how data flows through the graph.
Storage Engine interface: A Storage plugin interfaces in Drill represent the abstractions that Drill uses to interact with the data sources.
The plugins are extensible, allowing you to write new plugins for any additional data sources.
The following image shows a DrillBit query execution diagram:
The above diagram involves the following steps –
The Drill client issues a query. Any Drillbit in the cluster can accept queries from clients.
A Drillbit then parses the query, optimizes it, and generates an optimized distributed query plan for fast and efficient execution.
The Drillbit that accepts the initial query becomes the Foreman (driving Drillbit) for the request. It gets a list of available Drillbit nodes in the cluster from ZooKeeper.
The foreman gets a list of available Drillbit nodes in the cluster from ZooKeeper and schedules the execution of query fragments on individual nodes according to the execution plan.
The individual nodes finish their execution and return data to the foreman.
The foreman finally returns the results back to the client.
4.Apache Drill – 安装
……
5. Apache Drill – SQL Operations
Apache Drill is an open-source SQL-On-Everything engine.
It allows SQL queries to be executed on any kind of data source, ranging from a simple CSV file to an advanced SQL and NoSQL database servers
To execute a query in a Drill shell, open your terminal move to the Drill installed directory and then type the following command.
$ bin/drill-embedded
Then you will see the response as shown in the following program.
0: jdbc:drill:zk=local>
Now you can execute your queries.
Otherwise you can run your queries through web console application to the url of http://localhost:8047.
If you need any help info type the following command.
$ 0: jdbc:drill:zk=local> !help
Primitive Data Types
Apache Drill supports the following list of data types.
Date, Time and Timestamp
Apache Drill supports time functions in the range from 1971 to 2037. The processing logic of data types can be easily tested by “VALUES()” statement. The following query returns date, time and timestamp for the given values.
Query:
0: jdbc:drill:zk=local> select DATE '2016-04-07',TIME '12:12:23',TIMESTAMP '2016-04-07 12:12:23' from (values(1));
Result:
+-------------+-----------+------------------------+
| EXPR$0 | EXPR$1 | EXPR$2 |
+-------------+-----------+------------------------+
| 2016-04-07 | 12:12:23 | 2016-04-07 12:12:23.0 |
+-------------+-----------+------------------------+
Interval
The INTERVALYEAR and INTERVALDAY internal types represent a period of time.
The INTERVALYEAR type specifies values from a year to a month.
The INTERVALDAY type specifies values from a day to seconds.
Query
0: jdbc:drill:zk=local> select timestamp '2016-04-07 12:45:50' + interval '10' year(2) from (values(1));
Result:
+------------------------+
| EXPR$0 |
+------------------------+
| 2026-04-07 12:45:50.0 |
+———————————————————————-+
1 row selected (0.251 seconds)
In the above query, INTERVAL keyword followed by 10 adds 10 years to the timestamp.
The 2 in parentheses in YEAR(2) specifies the precision of the year interval, 2 digits in this case to support the ten interval.
INTERVALDAY Query
0: jdbc:drill:zk=local> select timestamp '2016-04-07 12:45:52' + interval '1' day(1) from (values(1));
Result:
+------------------------+
| EXPR$0 |
+------------------------+
| 2016-04-08 12:45:52.0 |
+———————————————————————-+
Here INTERVAL ‘1’ indicates that two days will be added from that specified day.
Operators
The following operators are used in Apache Drill to perform the desired operations.
Drill Scalar Functions
Apache Drill scalar functions supports Math and Trig functions. Most scalar functions use data types such as INTEGER, BIGINT, FLOAT and DOUBLE.
Math Functions
The following table describes the list of “Math functions” in Apache Drill.
Now let’s run queries for the scalar functions.
The Drill scalar functions can be easily tested by the values() statement, otherwise you can also use the select statement.
ABS(x)
The output of this function type is the same as the same input type.
Query:
0: jdbc:drill:zk=local> values(ABS(1.899));
or
0: jdbc:drill:zk=local> select ABS(1.899) from (values(1));
The result will be as shown in the following program:
+---------+
| EXPR$0 |
+---------+
| 1.899 |
+————————-+
CBRT(x)
This cubic root returns the output type as a float type.
Query:
0: jdbc:drill:zk=local> values(CBRT(125));
Result:
+---------+
| EXPR$0 |
+---------+
| 5.0 |
+---------+
CEIL(x)
The following program shows the query for this function:
0: jdbc:drill:zk=local> values(ceil(4.6));
Result:
+---------+
| EXPR$0 |
+---------+
| 5.0 |
+---------+
The output is returned as the largest following value.
……
Trig Functions
Apache Drill supports the following trig functions and these functions’ return type is a floating point value.
sin(x)
The following program shows the query for this function:
0: jdbc:drill:zk=local> values(sin(45));
Result:
+---------------------+
| EXPR$0 |
+---------------------+
| 0.8509035245341184 |
+---------------------+
Here the sin 45 value is returned as the output.
cosh(x)
The following program shows the query for this function:
0: jdbc:drill:zk=local> values(cosh(90));
Result:
+-----------------------+
| EXPR$0 |
+-----------------------+
| 6.102016471589204E38 |
+-----------------------+
The output result is a hyperbolic cosine value for the angle 90.
……
Data Type Conversion
In Apache Drill, you can cast or convert data to the required type for moving data from one data source to another.
Drill also supports the following functions for casting and converting data types:
CAST( x AS y)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select cast('3' as double) from (values(1));
Result:
+———————-+
| EXPR$0 |
+———————-+
| 3.0 |
+———————-+
Here the input value integer 3 is casting as double 3.0.
CONVERT_FROM()
The following program shows the query for this function:
0: jdbc:drill:zk=local> Select CONVERT_FROM ('{x:1, y:2}' ,'JSON') AS Convertion from (values(1));
+--------------------+
| Conversion |
+--------------------+
| {"x":1,"y":2} |
+--------------------+
The above query converts varchar data to JSON format.
Similarly, you can use other data types to Drill supported data format.
For naming columns, you can use the alias method.
CONVERT_TO()
The following program shows the query for this function:
0: jdbc:drill:zk=local> select CONVERT_TO(2,'int') as conversion from (values(1));
Result:
+—————————————+
| conversion |
+————————————-+
| 00000040 |
+————————————-+
Here the output is returned as the hexadecimal value for 2.
Date - Time Functions
Apache Drill supports time functions based on the Gregorian calendar and in the range from 1971 to 2037. The following table describes the list of Date/Time functions.
Age( x, [,y] ) function
The following program shows the query for this function:
0: jdbc:drill:zk=local> select age('2000-04-13') from (values(1));
Result:
+-----------------+
| EXPR$0 |
+-----------------+
| P195M4DT66600S |
+—————————————————+
The output result is the interval limit from the specified year to midnight of the current day.
CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP Function
The following program shows the query for this function:
0: jdbc:drill:zk=local> select CURRENT_DATE,CURRENT_TIME,CURRENT_TIMESTAMP from (values(1));
Result:
+---------------+---------------+--------------------------+
| current_date | current_time | current_timestamp |
+---------------+---------------+--------------------------+
| 2016-04-07 | 11:50:34.384 | 2016-04-24 11:50:34.384 |
+———————+———————+——————————————————————————————————————————+
The output returns current date, time, and timestamp for the day.
LOCALTIME, LOCALTIMESTAMP Functions
The following program shows the query for this function:
0: jdbc:drill:zk=local> select LOCALTIME,LOCALTIMESTAMP from (values(1));
Result:
+---------------+--------------------------+
| LOCALTIME | LOCALTIMESTAMP |
+---------------+--------------------------+
| 15:17:46.333 | 2016-04-24 15:17:46.333 |
+---------------+--------------------------+
NOW(),TIMEOFDAY()
The following program shows the query for this function:
0: jdbc:drill:zk=local> select NOW(),TIMEOFDAY() from (values(1));
Result:
+--------------------------+---------------------------------------+
| EXPR$0 | EXPR$1 |
+--------------------------+---------------------------------------+
| 2016-04-24 15:19:23.975 | 2016-04-24 15:19:24.243 Asia/Kolkata |
+--------------------------+---------------------------------------+
Here, TIMEOFDAY() returns the result for UTC time zone.
DATE_ADD(x, integer)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select DATE_ADD('2016-04-07',3) FROM (VALUES(1));
Result:
+—————————-+
| EXPR$0
+——————————+
| 2016-04-10
+——————————+
From this result, 3 days will be added.
DATE_ADD(x, interval)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select date_add(‘2016-04-07',6-2) from (values(1));
Result:
+———————————+
| EXPR$0
+——————————-+
| 2016-04-11
+——————————-+
Here the interval limit 6-2 gives the result as 4, then the result 4 will be added to the given date.
DATE_SUB(x,y)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select date_sub('2016-04-07',2) from (values(1));
Result:
+————————————-+
| EXPR$0 |
+————————————-+
| 2016-04-05 |
+————————————-+
The output indicates 2 days subtracted from the specified day.
EXTRACT( x from y)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select EXTRACT(SECOND FROM TIME '12:20:40') from(values(1));
Result:
+————————-+
| EXPR$0
+————————-+
| 40.0
+————————-+
The seconds extracted from the given time.
String Manipulation Function
Apache Drill supports the following list of string functions.
BYTE_SUBSTR(x,y [, z ] )
The following program shows the query for this function:
0: jdbc:drill:zk=local> select BYTE_SUBSTR('Drill',1,2) from (values(1));
Result:
+———————-+
| EXPR$0
+———————-+
| 4472
+———————-+
The above query returns a binary format of the substring position of the string Drill.
……
INITCAP(x)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select initcap('drill') from (values(1));
Result:
+————————+
| EXPR$0
+————————+
| Drill
+————————+
The Initcap function returns the result as the first character of the string becomes capitalized.
……
LPAD(x,y [ , z ])
The following program shows the query for this function:
0: jdbc:drill:zk=local> select lpad('drill',2,4) from (values(1));
Result:
+---------+
| EXPR$0 |
+---------+
| dr |
+---------+
Left pad the value of the given string “drill” from the position of 2 to 4, so the result will be just dr.
RPAD(x,y,z)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select rpad('tutorialspoint',2,4) from (values(1));
Result:
+———————+
| EXPR$0
+——————-+
| tu
+——————-+
Right pad the value of the given string.
RTRIM(x)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select rtrim('tutorialspoint','point') from (values(1));
Result:
+—————————-+
| EXPR$0
+—————————-+
| tutorials
+——————————+
Right trimming the characters form the two given strings.
LTRIM(x)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select ltrim('tutorialspoint','point') from (values(1));
Result:
+——————————————-+
| EXPR$0
+——————————————-+
| utorialspoint
+———————————————+
Left trimming the character from the given string.
REGEXP_REPLACE(x,y,x)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select regexp_replace('new','e','o') from (values(1));
Result:
+————————-+
| EXPR$0 |
+————————-+
| now |
+————————-+
Here the given string new is replaced as now.
……
POSITION( x IN y)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select position('o' in 'tutorialspoint') from (values(1));
Result:
+——————-+
| EXPR$0
+———————+
| 4
+———————+
……
SUBSTR(x,y,z)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select substr('tutorialspint',4,7) from (values(1));
Result:
+----------+
| EXPR$0 |
+----------+
| orialsp |
+----------+
Null Handling Function
Apache Drill supports the following list of null handling functions.
COALESCE(x, y [ , y ]...)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select coalesce(3,1,7) from (values(1));
Result:
+---------+
| EXPR$0 |
+---------+
| 3 |
+————————-+
Here first arg is non null, so it returns the value 3.
NULLIF(x,y)
The following program shows the query for this function:
0: jdbc:drill:zk=local> select NULLIF(3,3) from (values(1));
Result:
+---------+
| EXPR$0 |
+---------+
| null |
+---------+
Here both the arguments are same, so the result is NULL.