Slow Query Basics: Optimize Data Access
Analyze a poorly performing query in two steps:
- Find out whether your application is retrieving more data than you need. That usually means it’s accessing too many rows, but it might also be accessing too many columns.
- Find out whether the MySQL Server is analyzing more rows than it needs.
Are You Asking the Database for Data You Don’t Need?
Here are some typical mistakes:
- Fetching more rows than needed
- Fetching all columns from a multitable join
- Fetching all columns
Is MySQL Examining Too Much Data?
In MySQL, the simplest query cost metrics are:
- Execution time
- Number of rows examined
- Number of rows returned
All these metrics are logged in the slow query log, so looking at the slow query log is one of the best ways to find queries that examine too much data.
- Execution time
- Rows examined and rows returned
- Rows examined and access types
The access method(s) appear in the type column in EXPLAIN’s output. The access types range from a full table scan to index scans, range scans, unique index lookups, and constants. Each of these is faster than the one before it, because it requires reading less data.
In general, MySQL can apply a WHERE clause in three ways, from best to worst:
- Apply the conditions to the index lookup operation to eliminate nonmatching rows. This happens at the storage engine layer.
- Using the covering index (“Using index” in the Extra column) to avoid row accesses, and filter out nonmatching rows after retrieving each result from the index. This happens at the server layer, but it doesn’t reading rows from the table.
- Retrieving rows from the table, then filter nonmatching rows (“Using where” in the Extra column). This happens at the server layer and requires the server to read rows from the table before it can filter them.
Ways to Restructure Queries
Complex Queries Versus Many Queries
MySQL was designed to handle connecting and disconnecting very efficiently and to respond to small and simple queries very quickly.
Chopping Up a Query
Join Decomposition
Many high-performance web sites use join decomposition. You can decompose a join by running multiple single queries instead of a multitable join, and then performing the join in the application.
- Caching can be more efficient. Many applications cache “objects” that map directly to tables
- For MyISAM tables, performing one query per table uses table locks more efficiently: the queries will lock the tables individually and relatively briefly, instead of locking them all for a longer time.
- Doing joins in the application makes it easier to scale the database by placing tables on different servers.
- The queries themselves can be more efficient.
- You can reduce redundant row accesses.
- To some extent, you can view this technique as manually implementing a hash join instead of the nested loops algorithm MySQL uses to execute a join.
Summary: When Application Joins May Be More Efficient
- You cache and reuse a lot of data from earlier queries
- You use multiple MyISAM tables
- You distribute data across multiple servers
- You replace joins with IN() lists on large tables
- A join refers to the same table multiple times
Query Execution Basics
The MySQL Client/Server Protocol
The client sends a query to the server as a single packet of data. This is why the max_packet_size configuration variable is important if you have large queries.
Query states
Each MySQL connection, or thread, has a state that shows what it is doing at any given time. There are several ways to view these states, but the easiest is to use the SHOW FULL PROCESSLIST command (the states appear in the Command column)
- Sleep: The thread is waiting for a new query from the client
- Query: The thread is either executing the query or sending the result back to the client
- Locked: The thread is waiting for a table lock to be granted at the server level.
- Analyzing and statistics: The thread is checking storage engine statistics and optimizing the query.
- Copying to tmp table [on disk]
- Sorting result
- Sending data: This can mean several things: the thread might be sending data between stages of the query, generating the result set, or returning the result set to the client.
The Query Cache
Case sensitive hash lookup.
If MySQL does find a match in the query cache, it must check privileges before returning the cached query. This is possible without parsing the query, because MySQL stores table information with the cached query.
The Query Optimization Process
The parser and the preprocessor
The query optimizer
MySQL uses a cost-based optimizer. The unit of cost is a single random 4K data page read. You can see how expensive the optimizer estimated a query to be by running the query, then inspecting the last_query_cost session variable:
The optimizer does not include the effects of any type of caching in its estimates – it assumes every read will result in a disk I/O operation.
There are two basic types of optimizations, which we call static and dynamic. Static optimizations can be performed simply by inspecting the parse tree. Static optimizations are independent of values, such as the value of a constant in a WHERE clause. They can be performed once and will always be valid, even then the query is reexecuted with different values. In contrast, dynamic optimizations are based on context and can depend on many factors, such as which value is in a WHERE clause or how many rows are in an index.
IN() List comparisions
In many database servers, IN() is just a synonym for multiple OR clauses, because the two are logically equivalent. Not so in MySQL, which sorts the values in the IN() list and uses a fast binary search to see whether a value is in the list. This is O(log n) in the size of the list, whereas an equivalent series of OR clauses is O(n) in the size of the list(i.e. much slower for large list)
Table and index statistics
MySQL’s join execution strategy
MySQL considers every query a join – not just every query that matches rows from two tables, but every query, period (including subqueries, and even a SELECT against a single table).
MySQL treats every join as a nested-loop join.
MySQL executes UNION queries with temporary tables,and it rewrites all RIGHT OUTER JOIN queries to equivalent LEFT OUTER JION.
MySQL doesn’t support FULL OUTER JOIN
The execution plan
If you execute EXPLAIN EXTENDED on a query, followed by SHOW WARNINGS, you’ll see the reconstructed query.
The join optimizer
The most important part of the MySQL query optimizer is the join optimizer, which decides the best order of the execution for multitable queries.
STRAIGHT_JOIN
Sort optimizer
It can do sort in memory or on disk, but it always calls this process a filesort, even if it doesn’t actually use a file.
There are two filesort algorithms:
- Two passes (old)
Reads row pointers and ORDER BY columns, sorts them, and then scans the sorted list and rereads the rows for output.
- One pass (new)
Reads all the columns needed for the query, sorts them by the ORDER BY columns, and then scans the sorted list and output the specified columns.
MySQL allocates a fixed-size record for each tuple it will sort, these records are large enough to hold the largest possible tuple, including the full length of each VARCHAR column. Also, if you’re using UTF-8, MySQL allocates 3 bytes for each character.
The Query Execution Engine
Retuning Results to the Client
Limitations of the MySQL Query Optimizer
Correlated Subqueries
When a correlated subquery is good
UNION limitations
MySQL sometimes can’t “push down” conditions from the outside of a UNION to the inside, where they could be used to limit results or enable additional optimizations.
Index merge optimizations
Index merge algorithms let MySQL use more than one index per table in a query.
There are 3 variations on the algorithm: union for OR conditions, intersection for AND conditions, and unions of intersections for combinations of the two.
Equality propagation
Parallel execution
MySQL can’t execute a single query in parallel on many CPUs.
Hash joins
MySQL can’t do true hash joins now.
Loose index scans
MySQL has historically been unable to do loose index scans, which scan noncontiguous ranges of an index. MySQL’s index scans generally require a defined start point and a defined end point in the index, even if only a few noncontiguous rows in the middle are really desired for the query. MySQL will scan the entire range of rows within these end points.
Beginning in MySQL 5.0, loose index scans are possible in certain limited circumstances, such as queries that find maximum and minimum values in a grouped query
MIN() and MAX()
==>
SELECT and UPDATE on the same table
MySQL doesn’t let you SELECT from a table while simultaneous running an UPDATE on it.
Optimizer Specific Types of Queries
Optimizing COUNT() Queries
COUNT() counts values and rows. A value is a non-NULL expression (NULL is the absence of a value).
When you want to know the number of rows in the result, you should always use COUNT(*). This communicates your intention clearly and avoids poor performance.
==>
Optimizing JOIN Queries
- Make sure there are indexes on the columns in the ON or USING clauses. In general, you need to add indexes only on the second table in the join order, unless they’re needed for some other reason.
- Try to ensure that any GROUP BY or ORDER BY expression refers only to columns from a single table, so MySQL can try to use an index for that operation.
-
Optimizing Subqueries
You should usually prefer a join where possible.
Optimizing GROUP BY and DISTINCT
MySQL has two kinds of GROUP BY strategies when it can’t use an index: it can use a temporary table or a filesort to perform the grouping. You can force the optimizer to choose one method or the other with the SQL_BIG_RESULT and SQL_SMALL_RESULT optimizer hints.
MySQL automatically orders grouped queries by the columns in the GROUP BY clause, unless you specify an ORDER BY clause explicitly. If you don’t care about the order and you see this causing a filesort, you can use ORDER BY NULL to skip the automatic sort. You can also add an optional DESC or ASC keyword right after the GROUP BY clause to order the result in the desired direction by the clause’s columns.
Optimizing LIMIT and OFFSET
One simple technique to improve efficiency is to do the offset on a covering index, rather than the full rows. You can then join the result to the full row and retrieve the additional columns you need. This can be much more efficient.
SELECT film_id, description FROM film ORDER BY title LIMIT 50, 5;
==> (if the table is very large, this query is better written as follows:)
SELECT film_id, description FROM film
INNER JOIN (select film_id FROM film ORDER BY title LIMIT 50, 5) as lim USING(film_id);
If you really need to optimize pagination systems, you should probably use precomputed summaries.
Optimizing SQL_CALC_FOUND_ROWS
Optimizing UNION
MySQL always executes UNION queries by creating a temporary table and filling it with the UNION results. You might have to help the optimizer by manually “pushing down” WHERE, LIMIT, ORDER BY, and other conditions. MySQL always places results into a temporary table and then reads them out again, even when it’s not really necessary.
Query Optimizer Hints
- HIGH_PRIORITY and LOW_PRIORITY
These hints are effective on storage engines with table-level locking, but you should never need them on InnoDB or other engines with fine-grained locking and concurrency control. Be careful when using then on MyISAM, because they can disable concurrent insert and greatly reduce performance.
- DELAYED
- STRAIGHT_JOIN
- SQL_SMALL_RESULT and SQL_BIG_RESULT
SQL_SMALL_RESULT tells the optimizer that the result set will be small and can be put into indexed temporary tables to avoid sorting for the grouping, whereas SQL_BIG_RESULT indicates that the result will be large and that it will be better to use temporary tables on disk with sorting.
- SQL_BUFFER_RESULT
This hint tells the optimizer to put the results into a temporary table and release table locks as soon as possible.
- SQL_CACHE and SQL_NO_CACHE
- SQL_CACL_FOUND_ROWS
This hint tells MySQL to calculate a full result set when there’s a LIMIT clause, even though it returns only LIMIT ROWS. You can retrieve the total number of rows it found via FOUND_ROWS()
- FOR UDPATE and LOCK IN SHARE MODE
When using these hints with InnoDB, be aware that they may disable some optimizations, such as covering indexes.
InnoDB can’t lock rows exclusively without accessing the primary key, which is where the row version information is stored.
- USE INDEX, IGNORE INDEX, and FORCE INDEX
User-Defined Variables
Where executes before the SELECT, that’s why there are two records returned.
The query returns every row in the table, because the ORDER BY added a filesort and the WHERE is evaluated before the filesort.
(The @rownum := @rownum + 1 in the SELECT clause is executed at last, thus the value in the column cnt is in order)
The solution to this problem is to assign and read in the same stage of query execution.
Note that the @rownum is 7 after the query!
(Though the returned row count is 1, the @rownum := @rownum + 1 is executed for every row in the table as MySQL doesn’t know when to stop evaluating @rownum := @rownum + 1!)
Note the column cnt is 7 this time as we add “Order by” clause. Thus, the select will be “executed” after the sort operation; the @rownum will be the last value.
