第9章?企業(yè)級(jí)調(diào)優(yōu)

9.1 Fetch抓取

Fetch抓取是指，Hive中對(duì)某些情況的查詢(xún)可以不必使用MapReduce計(jì)算。例如：SELECT * FROM employees;在這種情況下，Hive可以簡(jiǎn)單地讀取employee對(duì)應(yīng)的存儲(chǔ)目錄下的文件，然后輸出查詢(xún)結(jié)果到控制臺(tái)。

在hive-default.xml.template文件中hive.fetch.task.conversion默認(rèn)是more，老版本hive默認(rèn)是minimal，該屬性修改為more以后，在全局查找、字段查找、limit查找等都不走mapreduce。

<property> ????<name>hive.fetch.task.conversion</name> ????<value>more</value> ????<description> ??????Expects one of [none, minimal, more]. ??????Some select queries can be converted to single FETCH task minimizing latency. ??????Currently the query should be single sourced not having any subquery and should not have ??????any aggregations or distincts (which incurs RS), lateral views and joins. ??????0. none : disable hive.fetch.task.conversion ??????1. minimal : SELECT STAR, FILTER on partition columns, LIMIT only ??????2. more ?: SELECT, FILTER, LIMIT only (support TABLESAMPLE and virtual columns) ????</description> ??</property>

案例實(shí)操：

1）把hive.fetch.task.conversion設(shè)置成none，然后執(zhí)行查詢(xún)語(yǔ)句，都會(huì)執(zhí)行mapreduce程序。

hive (default)> set hive.fetch.task.conversion=none;

hive (default)> select * from emp;

hive (default)> select ename from emp;

hive (default)> select ename from emp limit 3;

2）把hive.fetch.task.conversion設(shè)置成more，然后執(zhí)行查詢(xún)語(yǔ)句，如下查詢(xún)方式都不會(huì)執(zhí)行mapreduce程序。

hive (default)> set hive.fetch.task.conversion=more;

hive (default)> select * from emp;

hive (default)> select ename from emp;

hive (default)> select ename from emp limit 3;