Put MQ record to Kinesis | Python Lambda function
up vote
0
down vote
favorite
I have python lambda function, which will get messages from AWS MQ and with those messages I need to put them into Kinesis.
I followed this link but that did not help much. So Rob helped me to get through this.
My Final achievement is - I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to do to my below code?
Code:
import time
import boto3
import stomp
import json
kinesis_client = boto3.client('kinesis')
class Listener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)
def on_message(self, headers, message):
print('received a message %s (%s)'%(message, headers))
kinesis_client.put_record(
StreamName='',
Data=b'bytes',
PartitionKey='1'
)
def lambda_handler(event, context):
conn = stomp.Connection(host_and_ports=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
lst = Listener()
conn.set_listener('Listener', lst)
conn.set_ssl(for_hosts=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
conn.start()
print('CONNECTION Started')
conn.connect(login='test_mq', passcode='',wait=True)
print('CONNECTION established')
conn.subscribe(destination='/queue/', id=1, ack='auto', headers={'transformation' : 'jms-map-json'})
print('CONNECTION Subscribed')
time.sleep(10)
conn.disconnect()
return
python python-3.x python-2.7 amazon-web-services amazon-kinesis
|
show 1 more comment
up vote
0
down vote
favorite
I have python lambda function, which will get messages from AWS MQ and with those messages I need to put them into Kinesis.
I followed this link but that did not help much. So Rob helped me to get through this.
My Final achievement is - I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to do to my below code?
Code:
import time
import boto3
import stomp
import json
kinesis_client = boto3.client('kinesis')
class Listener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)
def on_message(self, headers, message):
print('received a message %s (%s)'%(message, headers))
kinesis_client.put_record(
StreamName='',
Data=b'bytes',
PartitionKey='1'
)
def lambda_handler(event, context):
conn = stomp.Connection(host_and_ports=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
lst = Listener()
conn.set_listener('Listener', lst)
conn.set_ssl(for_hosts=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
conn.start()
print('CONNECTION Started')
conn.connect(login='test_mq', passcode='',wait=True)
print('CONNECTION established')
conn.subscribe(destination='/queue/', id=1, ack='auto', headers={'transformation' : 'jms-map-json'})
print('CONNECTION Subscribed')
time.sleep(10)
conn.disconnect()
return
python python-3.x python-2.7 amazon-web-services amazon-kinesis
Don't you want to put the recordon_message
instead ofon_error
?
– Rob Bricheno
Nov 22 at 15:36
@RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
– Tinku
Nov 22 at 15:47
It looks like you need to give your lambda function permission to put records to your kinesis stream.
– Rob Bricheno
Nov 22 at 15:48
@RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
– Tinku
Nov 22 at 16:14
@RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
– Tinku
Nov 22 at 17:02
|
show 1 more comment
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have python lambda function, which will get messages from AWS MQ and with those messages I need to put them into Kinesis.
I followed this link but that did not help much. So Rob helped me to get through this.
My Final achievement is - I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to do to my below code?
Code:
import time
import boto3
import stomp
import json
kinesis_client = boto3.client('kinesis')
class Listener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)
def on_message(self, headers, message):
print('received a message %s (%s)'%(message, headers))
kinesis_client.put_record(
StreamName='',
Data=b'bytes',
PartitionKey='1'
)
def lambda_handler(event, context):
conn = stomp.Connection(host_and_ports=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
lst = Listener()
conn.set_listener('Listener', lst)
conn.set_ssl(for_hosts=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
conn.start()
print('CONNECTION Started')
conn.connect(login='test_mq', passcode='',wait=True)
print('CONNECTION established')
conn.subscribe(destination='/queue/', id=1, ack='auto', headers={'transformation' : 'jms-map-json'})
print('CONNECTION Subscribed')
time.sleep(10)
conn.disconnect()
return
python python-3.x python-2.7 amazon-web-services amazon-kinesis
I have python lambda function, which will get messages from AWS MQ and with those messages I need to put them into Kinesis.
I followed this link but that did not help much. So Rob helped me to get through this.
My Final achievement is - I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to do to my below code?
Code:
import time
import boto3
import stomp
import json
kinesis_client = boto3.client('kinesis')
class Listener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)
def on_message(self, headers, message):
print('received a message %s (%s)'%(message, headers))
kinesis_client.put_record(
StreamName='',
Data=b'bytes',
PartitionKey='1'
)
def lambda_handler(event, context):
conn = stomp.Connection(host_and_ports=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
lst = Listener()
conn.set_listener('Listener', lst)
conn.set_ssl(for_hosts=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
conn.start()
print('CONNECTION Started')
conn.connect(login='test_mq', passcode='',wait=True)
print('CONNECTION established')
conn.subscribe(destination='/queue/', id=1, ack='auto', headers={'transformation' : 'jms-map-json'})
print('CONNECTION Subscribed')
time.sleep(10)
conn.disconnect()
return
python python-3.x python-2.7 amazon-web-services amazon-kinesis
python python-3.x python-2.7 amazon-web-services amazon-kinesis
edited Nov 23 at 11:43
asked Nov 22 at 15:15
Tinku
337
337
Don't you want to put the recordon_message
instead ofon_error
?
– Rob Bricheno
Nov 22 at 15:36
@RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
– Tinku
Nov 22 at 15:47
It looks like you need to give your lambda function permission to put records to your kinesis stream.
– Rob Bricheno
Nov 22 at 15:48
@RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
– Tinku
Nov 22 at 16:14
@RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
– Tinku
Nov 22 at 17:02
|
show 1 more comment
Don't you want to put the recordon_message
instead ofon_error
?
– Rob Bricheno
Nov 22 at 15:36
@RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
– Tinku
Nov 22 at 15:47
It looks like you need to give your lambda function permission to put records to your kinesis stream.
– Rob Bricheno
Nov 22 at 15:48
@RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
– Tinku
Nov 22 at 16:14
@RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
– Tinku
Nov 22 at 17:02
Don't you want to put the record
on_message
instead of on_error
?– Rob Bricheno
Nov 22 at 15:36
Don't you want to put the record
on_message
instead of on_error
?– Rob Bricheno
Nov 22 at 15:36
@RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
– Tinku
Nov 22 at 15:47
@RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
– Tinku
Nov 22 at 15:47
It looks like you need to give your lambda function permission to put records to your kinesis stream.
– Rob Bricheno
Nov 22 at 15:48
It looks like you need to give your lambda function permission to put records to your kinesis stream.
– Rob Bricheno
Nov 22 at 15:48
@RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
– Tinku
Nov 22 at 16:14
@RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
– Tinku
Nov 22 at 16:14
@RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
– Tinku
Nov 22 at 17:02
@RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
– Tinku
Nov 22 at 17:02
|
show 1 more comment
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53433902%2fput-mq-record-to-kinesis-python-lambda-function%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Don't you want to put the record
on_message
instead ofon_error
?– Rob Bricheno
Nov 22 at 15:36
@RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
– Tinku
Nov 22 at 15:47
It looks like you need to give your lambda function permission to put records to your kinesis stream.
– Rob Bricheno
Nov 22 at 15:48
@RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
– Tinku
Nov 22 at 16:14
@RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
– Tinku
Nov 22 at 17:02