How to fetch Kafka source connector schema based on connector name
up vote
1
down vote
favorite
I am using Confluent JDBC Kafka connector to publish messages into topic. The source connector will send data to topic along with schema on each poll. I want to retrieve this schema.
Is it possible? How? Can anyone suggest me
My intention is to create a KSQL stream or table based on schema build by Kafka connector on poll.
apache-kafka apache-kafka-connect confluent confluent-schema-registry ksql
edited Nov 22 at 17:53
cricket_007
77.6k1142107
asked Nov 22 at 6:55
Achaius
1,837134286
add a comment |
up vote
1
down vote
favorite
I am using Confluent JDBC Kafka connector to publish messages into topic. The source connector will send data to topic along with schema on each poll. I want to retrieve this schema.
Is it possible? How? Can anyone suggest me
My intention is to create a KSQL stream or table based on schema build by Kafka connector on poll.
apache-kafka apache-kafka-connect confluent confluent-schema-registry ksql
edited Nov 22 at 17:53
cricket_007
77.6k1142107
asked Nov 22 at 6:55
Achaius
1,837134286
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I am using Confluent JDBC Kafka connector to publish messages into topic. The source connector will send data to topic along with schema on each poll. I want to retrieve this schema.
Is it possible? How? Can anyone suggest me
My intention is to create a KSQL stream or table based on schema build by Kafka connector on poll.
apache-kafka apache-kafka-connect confluent confluent-schema-registry ksql
edited Nov 22 at 17:53
cricket_007
77.6k1142107
asked Nov 22 at 6:55
Achaius
1,837134286
I am using Confluent JDBC Kafka connector to publish messages into topic. The source connector will send data to topic along with schema on each poll. I want to retrieve this schema.
Is it possible? How? Can anyone suggest me
My intention is to create a KSQL stream or table based on schema build by Kafka connector on poll.
apache-kafka apache-kafka-connect confluent confluent-schema-registry ksql
apache-kafka apache-kafka-connect confluent confluent-schema-registry ksql
edited Nov 22 at 17:53
cricket_007
77.6k1142107
asked Nov 22 at 6:55
Achaius
1,837134286
edited Nov 22 at 17:53
cricket_007
77.6k1142107
asked Nov 22 at 6:55
Achaius
1,837134286
edited Nov 22 at 17:53
cricket_007
77.6k1142107
edited Nov 22 at 17:53
cricket_007
77.6k1142107
edited Nov 22 at 17:53
cricket_007
77.6k1142107
77.6k1142107
asked Nov 22 at 6:55
Achaius
1,837134286
asked Nov 22 at 6:55
Achaius
1,837134286
asked Nov 22 at 6:55
Achaius
1,837134286
1,837134286
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
2
down vote
accepted
The best way to do this is to use Avro, in which the schema is stored separately and automatically used by Kafka Connect and KSQL.
You can use Avro by configuring Kafka Connect to use the AvroConverter. In your Kafka Connect worker configuration set:
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://schema-registry:8081
(Update schema-registry to the hostname of where your Schema Registry is running)
From there, in KSQL you just use
CREATE STREAM my_stream WITH (KAFKA_TOPIC='source_topic', VALUE_FORMAT='AVRO');
You don't need to specify the schema itself here, because KSQL fetches it from the Schema Registry.
You can read more about Converters and serialisers here.
Disclaimer: I work for Confluent, and wrote the referenced blog post.
answered Nov 22 at 13:18
Robin Moffatt
5,8181228
I saw your post. But I have 2 questions here. 1. I am using confluent JDBC connector which sends records in delimited format. Is it still work? 2. What happened if schema was updated. Is KSQL able to refer the updated schema?
– Achaius
Nov 23 at 5:59
The answer for first questtion is yes, but confirm it. Kindly respond for the 2nd question
– Achaius
Nov 23 at 6:22
1
1. The JDBC Connector uses the Kafka Connect API, which supports multiple converters, including JSON, Avro, etc. So it's not the case that "JDBC connector sends records in delimited format". It is configurable the format in which it sends the data.
– Robin Moffatt
Nov 23 at 9:01
2
2. KSQL should pull the latest version of the schema when you create a new stream against the topic, yes.
– Robin Moffatt
Nov 23 at 9:01
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
accepted
The best way to do this is to use Avro, in which the schema is stored separately and automatically used by Kafka Connect and KSQL.
You can use Avro by configuring Kafka Connect to use the AvroConverter. In your Kafka Connect worker configuration set:
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://schema-registry:8081
(Update schema-registry to the hostname of where your Schema Registry is running)
From there, in KSQL you just use
CREATE STREAM my_stream WITH (KAFKA_TOPIC='source_topic', VALUE_FORMAT='AVRO');
You don't need to specify the schema itself here, because KSQL fetches it from the Schema Registry.
You can read more about Converters and serialisers here.
Disclaimer: I work for Confluent, and wrote the referenced blog post.
answered Nov 22 at 13:18
Robin Moffatt
5,8181228
I saw your post. But I have 2 questions here. 1. I am using confluent JDBC connector which sends records in delimited format. Is it still work? 2. What happened if schema was updated. Is KSQL able to refer the updated schema?
– Achaius
Nov 23 at 5:59
The answer for first questtion is yes, but confirm it. Kindly respond for the 2nd question
– Achaius
Nov 23 at 6:22
1
1. The JDBC Connector uses the Kafka Connect API, which supports multiple converters, including JSON, Avro, etc. So it's not the case that "JDBC connector sends records in delimited format". It is configurable the format in which it sends the data.
– Robin Moffatt
Nov 23 at 9:01
2
2. KSQL should pull the latest version of the schema when you create a new stream against the topic, yes.
– Robin Moffatt
Nov 23 at 9:01
add a comment |
up vote
2
down vote
accepted
The best way to do this is to use Avro, in which the schema is stored separately and automatically used by Kafka Connect and KSQL.
You can use Avro by configuring Kafka Connect to use the AvroConverter. In your Kafka Connect worker configuration set:
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://schema-registry:8081
(Update schema-registry to the hostname of where your Schema Registry is running)
From there, in KSQL you just use
CREATE STREAM my_stream WITH (KAFKA_TOPIC='source_topic', VALUE_FORMAT='AVRO');
You don't need to specify the schema itself here, because KSQL fetches it from the Schema Registry.
You can read more about Converters and serialisers here.
Disclaimer: I work for Confluent, and wrote the referenced blog post.
answered Nov 22 at 13:18
Robin Moffatt
5,8181228
I saw your post. But I have 2 questions here. 1. I am using confluent JDBC connector which sends records in delimited format. Is it still work? 2. What happened if schema was updated. Is KSQL able to refer the updated schema?
– Achaius
Nov 23 at 5:59
The answer for first questtion is yes, but confirm it. Kindly respond for the 2nd question
– Achaius
Nov 23 at 6:22
1
1. The JDBC Connector uses the Kafka Connect API, which supports multiple converters, including JSON, Avro, etc. So it's not the case that "JDBC connector sends records in delimited format". It is configurable the format in which it sends the data.
– Robin Moffatt
Nov 23 at 9:01
2
2. KSQL should pull the latest version of the schema when you create a new stream against the topic, yes.
– Robin Moffatt
Nov 23 at 9:01
add a comment |
up vote
2
down vote
accepted
up vote
2
down vote
accepted
The best way to do this is to use Avro, in which the schema is stored separately and automatically used by Kafka Connect and KSQL.
You can use Avro by configuring Kafka Connect to use the AvroConverter. In your Kafka Connect worker configuration set:
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://schema-registry:8081
(Update schema-registry to the hostname of where your Schema Registry is running)
From there, in KSQL you just use
CREATE STREAM my_stream WITH (KAFKA_TOPIC='source_topic', VALUE_FORMAT='AVRO');
You don't need to specify the schema itself here, because KSQL fetches it from the Schema Registry.
You can read more about Converters and serialisers here.
Disclaimer: I work for Confluent, and wrote the referenced blog post.
answered Nov 22 at 13:18
Robin Moffatt
5,8181228
The best way to do this is to use Avro, in which the schema is stored separately and automatically used by Kafka Connect and KSQL.
You can use Avro by configuring Kafka Connect to use the AvroConverter. In your Kafka Connect worker configuration set:
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://schema-registry:8081
(Update schema-registry to the hostname of where your Schema Registry is running)
From there, in KSQL you just use
CREATE STREAM my_stream WITH (KAFKA_TOPIC='source_topic', VALUE_FORMAT='AVRO');
You don't need to specify the schema itself here, because KSQL fetches it from the Schema Registry.
You can read more about Converters and serialisers here.
Disclaimer: I work for Confluent, and wrote the referenced blog post.
answered Nov 22 at 13:18
Robin Moffatt
5,8181228
answered Nov 22 at 13:18
Robin Moffatt
5,8181228
answered Nov 22 at 13:18
Robin Moffatt
5,8181228
answered Nov 22 at 13:18
Robin Moffatt
5,8181228
5,8181228
I saw your post. But I have 2 questions here. 1. I am using confluent JDBC connector which sends records in delimited format. Is it still work? 2. What happened if schema was updated. Is KSQL able to refer the updated schema?
– Achaius
Nov 23 at 5:59
The answer for first questtion is yes, but confirm it. Kindly respond for the 2nd question
– Achaius
Nov 23 at 6:22
1
1. The JDBC Connector uses the Kafka Connect API, which supports multiple converters, including JSON, Avro, etc. So it's not the case that "JDBC connector sends records in delimited format". It is configurable the format in which it sends the data.
– Robin Moffatt
Nov 23 at 9:01
2
2. KSQL should pull the latest version of the schema when you create a new stream against the topic, yes.
– Robin Moffatt
Nov 23 at 9:01
add a comment |
I saw your post. But I have 2 questions here. 1. I am using confluent JDBC connector which sends records in delimited format. Is it still work? 2. What happened if schema was updated. Is KSQL able to refer the updated schema?
– Achaius
Nov 23 at 5:59
The answer for first questtion is yes, but confirm it. Kindly respond for the 2nd question
– Achaius
Nov 23 at 6:22
1
1. The JDBC Connector uses the Kafka Connect API, which supports multiple converters, including JSON, Avro, etc. So it's not the case that "JDBC connector sends records in delimited format". It is configurable the format in which it sends the data.
– Robin Moffatt
Nov 23 at 9:01
2
2. KSQL should pull the latest version of the schema when you create a new stream against the topic, yes.
– Robin Moffatt
Nov 23 at 9:01
I saw your post. But I have 2 questions here. 1. I am using confluent JDBC connector which sends records in delimited format. Is it still work? 2. What happened if schema was updated. Is KSQL able to refer the updated schema?
– Achaius
Nov 23 at 5:59
I saw your post. But I have 2 questions here. 1. I am using confluent JDBC connector which sends records in delimited format. Is it still work? 2. What happened if schema was updated. Is KSQL able to refer the updated schema?
– Achaius
Nov 23 at 5:59
The answer for first questtion is yes, but confirm it. Kindly respond for the 2nd question
– Achaius
Nov 23 at 6:22
The answer for first questtion is yes, but confirm it. Kindly respond for the 2nd question
– Achaius
Nov 23 at 6:22
1
1
1. The JDBC Connector uses the Kafka Connect API, which supports multiple converters, including JSON, Avro, etc. So it's not the case that "JDBC connector sends records in delimited format". It is configurable the format in which it sends the data.
– Robin Moffatt
Nov 23 at 9:01
1. The JDBC Connector uses the Kafka Connect API, which supports multiple converters, including JSON, Avro, etc. So it's not the case that "JDBC connector sends records in delimited format". It is configurable the format in which it sends the data.
– Robin Moffatt
Nov 23 at 9:01
2
2
2. KSQL should pull the latest version of the schema when you create a new stream against the topic, yes.
– Robin Moffatt
Nov 23 at 9:01
2. KSQL should pull the latest version of the schema when you create a new stream against the topic, yes.
– Robin Moffatt
Nov 23 at 9:01
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53425385%2fhow-to-fetch-kafka-source-connector-schema-based-on-connector-name%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
