Put MQ record to Kinesis | Python Lambda function











up vote
0
down vote

favorite












I have python lambda function, which will get messages from AWS MQ and with those messages I need to put them into Kinesis.
I followed this link but that did not help much. So Rob helped me to get through this.



My Final achievement is - I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to do to my below code?



Code:



import time
import boto3
import stomp
import json

kinesis_client = boto3.client('kinesis')


class Listener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)

def on_message(self, headers, message):
print('received a message %s (%s)'%(message, headers))
kinesis_client.put_record(
StreamName='',
Data=b'bytes',
PartitionKey='1'
)
def lambda_handler(event, context):
conn = stomp.Connection(host_and_ports=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
lst = Listener()
conn.set_listener('Listener', lst)
conn.set_ssl(for_hosts=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
conn.start()
print('CONNECTION Started')
conn.connect(login='test_mq', passcode='',wait=True)
print('CONNECTION established')
conn.subscribe(destination='/queue/', id=1, ack='auto', headers={'transformation' : 'jms-map-json'})
print('CONNECTION Subscribed')
time.sleep(10)
conn.disconnect()
return









share|improve this question
























  • Don't you want to put the record on_message instead of on_error ?
    – Rob Bricheno
    Nov 22 at 15:36










  • @RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
    – Tinku
    Nov 22 at 15:47










  • It looks like you need to give your lambda function permission to put records to your kinesis stream.
    – Rob Bricheno
    Nov 22 at 15:48










  • @RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
    – Tinku
    Nov 22 at 16:14










  • @RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
    – Tinku
    Nov 22 at 17:02















up vote
0
down vote

favorite












I have python lambda function, which will get messages from AWS MQ and with those messages I need to put them into Kinesis.
I followed this link but that did not help much. So Rob helped me to get through this.



My Final achievement is - I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to do to my below code?



Code:



import time
import boto3
import stomp
import json

kinesis_client = boto3.client('kinesis')


class Listener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)

def on_message(self, headers, message):
print('received a message %s (%s)'%(message, headers))
kinesis_client.put_record(
StreamName='',
Data=b'bytes',
PartitionKey='1'
)
def lambda_handler(event, context):
conn = stomp.Connection(host_and_ports=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
lst = Listener()
conn.set_listener('Listener', lst)
conn.set_ssl(for_hosts=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
conn.start()
print('CONNECTION Started')
conn.connect(login='test_mq', passcode='',wait=True)
print('CONNECTION established')
conn.subscribe(destination='/queue/', id=1, ack='auto', headers={'transformation' : 'jms-map-json'})
print('CONNECTION Subscribed')
time.sleep(10)
conn.disconnect()
return









share|improve this question
























  • Don't you want to put the record on_message instead of on_error ?
    – Rob Bricheno
    Nov 22 at 15:36










  • @RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
    – Tinku
    Nov 22 at 15:47










  • It looks like you need to give your lambda function permission to put records to your kinesis stream.
    – Rob Bricheno
    Nov 22 at 15:48










  • @RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
    – Tinku
    Nov 22 at 16:14










  • @RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
    – Tinku
    Nov 22 at 17:02













up vote
0
down vote

favorite









up vote
0
down vote

favorite











I have python lambda function, which will get messages from AWS MQ and with those messages I need to put them into Kinesis.
I followed this link but that did not help much. So Rob helped me to get through this.



My Final achievement is - I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to do to my below code?



Code:



import time
import boto3
import stomp
import json

kinesis_client = boto3.client('kinesis')


class Listener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)

def on_message(self, headers, message):
print('received a message %s (%s)'%(message, headers))
kinesis_client.put_record(
StreamName='',
Data=b'bytes',
PartitionKey='1'
)
def lambda_handler(event, context):
conn = stomp.Connection(host_and_ports=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
lst = Listener()
conn.set_listener('Listener', lst)
conn.set_ssl(for_hosts=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
conn.start()
print('CONNECTION Started')
conn.connect(login='test_mq', passcode='',wait=True)
print('CONNECTION established')
conn.subscribe(destination='/queue/', id=1, ack='auto', headers={'transformation' : 'jms-map-json'})
print('CONNECTION Subscribed')
time.sleep(10)
conn.disconnect()
return









share|improve this question















I have python lambda function, which will get messages from AWS MQ and with those messages I need to put them into Kinesis.
I followed this link but that did not help much. So Rob helped me to get through this.



My Final achievement is - I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to do to my below code?



Code:



import time
import boto3
import stomp
import json

kinesis_client = boto3.client('kinesis')


class Listener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)

def on_message(self, headers, message):
print('received a message %s (%s)'%(message, headers))
kinesis_client.put_record(
StreamName='',
Data=b'bytes',
PartitionKey='1'
)
def lambda_handler(event, context):
conn = stomp.Connection(host_and_ports=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
lst = Listener()
conn.set_listener('Listener', lst)
conn.set_ssl(for_hosts=[('b-4714-4441-8166-47aae158281a-1.mq.eu-central-1.amazonaws.com', 61614)])
conn.start()
print('CONNECTION Started')
conn.connect(login='test_mq', passcode='',wait=True)
print('CONNECTION established')
conn.subscribe(destination='/queue/', id=1, ack='auto', headers={'transformation' : 'jms-map-json'})
print('CONNECTION Subscribed')
time.sleep(10)
conn.disconnect()
return






python python-3.x python-2.7 amazon-web-services amazon-kinesis






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 23 at 11:43

























asked Nov 22 at 15:15









Tinku

337




337












  • Don't you want to put the record on_message instead of on_error ?
    – Rob Bricheno
    Nov 22 at 15:36










  • @RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
    – Tinku
    Nov 22 at 15:47










  • It looks like you need to give your lambda function permission to put records to your kinesis stream.
    – Rob Bricheno
    Nov 22 at 15:48










  • @RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
    – Tinku
    Nov 22 at 16:14










  • @RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
    – Tinku
    Nov 22 at 17:02


















  • Don't you want to put the record on_message instead of on_error ?
    – Rob Bricheno
    Nov 22 at 15:36










  • @RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
    – Tinku
    Nov 22 at 15:47










  • It looks like you need to give your lambda function permission to put records to your kinesis stream.
    – Rob Bricheno
    Nov 22 at 15:48










  • @RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
    – Tinku
    Nov 22 at 16:14










  • @RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
    – Tinku
    Nov 22 at 17:02
















Don't you want to put the record on_message instead of on_error ?
– Rob Bricheno
Nov 22 at 15:36




Don't you want to put the record on_message instead of on_error ?
– Rob Bricheno
Nov 22 at 15:36












@RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
– Tinku
Nov 22 at 15:47




@RobBricheno My bad accidentally copied .. Corrected it now. Captured new logs as well.
– Tinku
Nov 22 at 15:47












It looks like you need to give your lambda function permission to put records to your kinesis stream.
– Rob Bricheno
Nov 22 at 15:48




It looks like you need to give your lambda function permission to put records to your kinesis stream.
– Rob Bricheno
Nov 22 at 15:48












@RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
– Tinku
Nov 22 at 16:14




@RobBricheno- I gave AmazonKinesisFullAccess to my role. Now error message is gone:) But not able view the message in Kinesis GUI:'(
– Tinku
Nov 22 at 16:14












@RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
– Tinku
Nov 22 at 17:02




@RobBricheno-- I have my Kinesis in Different AWS account with Role name, ARN, StreamARN,Stream Name and Endpoint as well as the Partition Key. What changes do I need to more do to my below code. Please suggest
– Tinku
Nov 22 at 17:02

















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53433902%2fput-mq-record-to-kinesis-python-lambda-function%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown






























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes
















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53433902%2fput-mq-record-to-kinesis-python-lambda-function%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Trompette piccolo

Slow SSRS Report in dynamic grouping and multiple parameters

Simon Yates (cyclisme)