Hive table with multiple SerDe
up vote
0
down vote
favorite
We have one HIVE table that is partitioned by date. It has currently Sequence file format, I want to convert it into Parquet Table.
Is it possible that we have new Partition with Parquet Serde, and older with Sequence format, so that I don't need to backfill it?
hive
asked Jan 31 at 18:07
rajnish
489514
add a comment |
up vote
0
down vote
favorite
We have one HIVE table that is partitioned by date. It has currently Sequence file format, I want to convert it into Parquet Table.
Is it possible that we have new Partition with Parquet Serde, and older with Sequence format, so that I don't need to backfill it?
hive
asked Jan 31 at 18:07
rajnish
489514
Why not make a separate table?CREATE TABLE t2 LIKE t STORED AS PARQUET?
– cricket_007
Jan 31 at 22:43
@cricket_007 But then I need to backfill it, by converting Sequence files to Parquet files ( For 2-3 year of history data) . Also it will be different tablename that could break pipeline (that could be fixed by multiple ways)
– rajnish
Feb 1 at 16:44
You cannot mix serdes. It's a table level setting, not partition level
– cricket_007
Feb 1 at 18:39
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
We have one HIVE table that is partitioned by date. It has currently Sequence file format, I want to convert it into Parquet Table.
Is it possible that we have new Partition with Parquet Serde, and older with Sequence format, so that I don't need to backfill it?
hive
asked Jan 31 at 18:07
rajnish
489514
We have one HIVE table that is partitioned by date. It has currently Sequence file format, I want to convert it into Parquet Table.
Is it possible that we have new Partition with Parquet Serde, and older with Sequence format, so that I don't need to backfill it?
hive
hive
asked Jan 31 at 18:07
rajnish
489514
asked Jan 31 at 18:07
rajnish
489514
asked Jan 31 at 18:07
rajnish
489514
asked Jan 31 at 18:07
rajnish
489514
asked Jan 31 at 18:07
rajnish
489514
489514
Why not make a separate table?CREATE TABLE t2 LIKE t STORED AS PARQUET?
– cricket_007
Jan 31 at 22:43
@cricket_007 But then I need to backfill it, by converting Sequence files to Parquet files ( For 2-3 year of history data) . Also it will be different tablename that could break pipeline (that could be fixed by multiple ways)
– rajnish
Feb 1 at 16:44
You cannot mix serdes. It's a table level setting, not partition level
– cricket_007
Feb 1 at 18:39
add a comment |
Why not make a separate table?CREATE TABLE t2 LIKE t STORED AS PARQUET?
– cricket_007
Jan 31 at 22:43
@cricket_007 But then I need to backfill it, by converting Sequence files to Parquet files ( For 2-3 year of history data) . Also it will be different tablename that could break pipeline (that could be fixed by multiple ways)
– rajnish
Feb 1 at 16:44
You cannot mix serdes. It's a table level setting, not partition level
– cricket_007
Feb 1 at 18:39
Why not make a separate table?
CREATE TABLE t2 LIKE t STORED AS PARQUET?– cricket_007
Jan 31 at 22:43
Why not make a separate table?
CREATE TABLE t2 LIKE t STORED AS PARQUET?– cricket_007
Jan 31 at 22:43
@cricket_007 But then I need to backfill it, by converting Sequence files to Parquet files ( For 2-3 year of history data) . Also it will be different tablename that could break pipeline (that could be fixed by multiple ways)
– rajnish
Feb 1 at 16:44
@cricket_007 But then I need to backfill it, by converting Sequence files to Parquet files ( For 2-3 year of history data) . Also it will be different tablename that could break pipeline (that could be fixed by multiple ways)
– rajnish
Feb 1 at 16:44
You cannot mix serdes. It's a table level setting, not partition level
– cricket_007
Feb 1 at 18:39
You cannot mix serdes. It's a table level setting, not partition level
– cricket_007
Feb 1 at 18:39
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
create a external empty table with default serde(LazySimpleSerDe) and default stored(textfile).
add partition.
alter partition set fileformat(or set serde).
Hive LanguageManual DDL
CREATE EXTERNAL TABLE test(ip string, localTime string )
PARTITIONED BY (partition__hive__ STRING) location '/tmp/table/empty';
alter table test add partition (partition__hive__='p_0') location 'hdfs://hdfsTest/hive/table/test/2018/11/21/08';
alter table test partition (partition__hive__='p_0') SET FILEFORMAT parquet;
alter table test add partition (partition__hive__='p_1') location 'hdfs://hdfsTest/hive/table/test/2018/11/21/09';
alter table test partition (partition__hive__='p_1') SET SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
answered Nov 21 at 9:03
Tianwang Li
11
Maybe you can explain with few sentences
– vahdet
Nov 21 at 9:07
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
create a external empty table with default serde(LazySimpleSerDe) and default stored(textfile).
add partition.
alter partition set fileformat(or set serde).
Hive LanguageManual DDL
CREATE EXTERNAL TABLE test(ip string, localTime string )
PARTITIONED BY (partition__hive__ STRING) location '/tmp/table/empty';
alter table test add partition (partition__hive__='p_0') location 'hdfs://hdfsTest/hive/table/test/2018/11/21/08';
alter table test partition (partition__hive__='p_0') SET FILEFORMAT parquet;
alter table test add partition (partition__hive__='p_1') location 'hdfs://hdfsTest/hive/table/test/2018/11/21/09';
alter table test partition (partition__hive__='p_1') SET SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
answered Nov 21 at 9:03
Tianwang Li
11
Maybe you can explain with few sentences
– vahdet
Nov 21 at 9:07
add a comment |
up vote
0
down vote
create a external empty table with default serde(LazySimpleSerDe) and default stored(textfile).
add partition.
alter partition set fileformat(or set serde).
Hive LanguageManual DDL
CREATE EXTERNAL TABLE test(ip string, localTime string )
PARTITIONED BY (partition__hive__ STRING) location '/tmp/table/empty';
alter table test add partition (partition__hive__='p_0') location 'hdfs://hdfsTest/hive/table/test/2018/11/21/08';
alter table test partition (partition__hive__='p_0') SET FILEFORMAT parquet;
alter table test add partition (partition__hive__='p_1') location 'hdfs://hdfsTest/hive/table/test/2018/11/21/09';
alter table test partition (partition__hive__='p_1') SET SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
answered Nov 21 at 9:03
Tianwang Li
11
Maybe you can explain with few sentences
– vahdet
Nov 21 at 9:07
add a comment |
up vote
0
down vote
up vote
0
down vote
create a external empty table with default serde(LazySimpleSerDe) and default stored(textfile).
add partition.
alter partition set fileformat(or set serde).
Hive LanguageManual DDL
CREATE EXTERNAL TABLE test(ip string, localTime string )
PARTITIONED BY (partition__hive__ STRING) location '/tmp/table/empty';
alter table test add partition (partition__hive__='p_0') location 'hdfs://hdfsTest/hive/table/test/2018/11/21/08';
alter table test partition (partition__hive__='p_0') SET FILEFORMAT parquet;
alter table test add partition (partition__hive__='p_1') location 'hdfs://hdfsTest/hive/table/test/2018/11/21/09';
alter table test partition (partition__hive__='p_1') SET SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
answered Nov 21 at 9:03
Tianwang Li
11
create a external empty table with default serde(LazySimpleSerDe) and default stored(textfile).
add partition.
alter partition set fileformat(or set serde).
Hive LanguageManual DDL
CREATE EXTERNAL TABLE test(ip string, localTime string )
PARTITIONED BY (partition__hive__ STRING) location '/tmp/table/empty';
alter table test add partition (partition__hive__='p_0') location 'hdfs://hdfsTest/hive/table/test/2018/11/21/08';
alter table test partition (partition__hive__='p_0') SET FILEFORMAT parquet;
alter table test add partition (partition__hive__='p_1') location 'hdfs://hdfsTest/hive/table/test/2018/11/21/09';
alter table test partition (partition__hive__='p_1') SET SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
answered Nov 21 at 9:03
Tianwang Li
11
edited Nov 22 at 1:37
answered Nov 21 at 9:03
Tianwang Li
11
answered Nov 21 at 9:03
Tianwang Li
11
answered Nov 21 at 9:03
Tianwang Li
11
11
Maybe you can explain with few sentences
– vahdet
Nov 21 at 9:07
add a comment |
Maybe you can explain with few sentences
– vahdet
Nov 21 at 9:07
Maybe you can explain with few sentences
– vahdet
Nov 21 at 9:07
Maybe you can explain with few sentences
– vahdet
Nov 21 at 9:07
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f48548723%2fhive-table-with-multiple-serde%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Why not make a separate table?
CREATE TABLE t2 LIKE t STORED AS PARQUET?– cricket_007
Jan 31 at 22:43
@cricket_007 But then I need to backfill it, by converting Sequence files to Parquet files ( For 2-3 year of history data) . Also it will be different tablename that could break pipeline (that could be fixed by multiple ways)
– rajnish
Feb 1 at 16:44
You cannot mix serdes. It's a table level setting, not partition level
– cricket_007
Feb 1 at 18:39