Generate Random Amazon Products
Bit of an odd question, but I am looking to get around 10,000 random amazon products for a school thesis and I am having trouble figuring out how to go about getting them. There are a few websites which seem to have this down:
www.randomking.com
www.randasin.com
But because these sites don't list their methodology I can't really use their items directly because I have no guarantee it's true random instead of some database. I'm wondering if any of you may know how to go about doing this or if you might have knowledge on how those guys did that?
Any help is greatly appreciated – I've spent too long banging my head against a wall already.
Thanks!
python web-scraping beautifulsoup amazon amazon-product-api
asked Nov 23 '18 at 5:57
Matthew Sciamanna
214
add a comment |
Bit of an odd question, but I am looking to get around 10,000 random amazon products for a school thesis and I am having trouble figuring out how to go about getting them. There are a few websites which seem to have this down:
www.randomking.com
www.randasin.com
But because these sites don't list their methodology I can't really use their items directly because I have no guarantee it's true random instead of some database. I'm wondering if any of you may know how to go about doing this or if you might have knowledge on how those guys did that?
Any help is greatly appreciated – I've spent too long banging my head against a wall already.
Thanks!
python web-scraping beautifulsoup amazon amazon-product-api
asked Nov 23 '18 at 5:57
Matthew Sciamanna
214
1
You should look into working with scrapy. Also, not many people are just going to code something for you. Give it a try to figure out what you need to do and when you run into a problem look it up or post it here for some help. Hope that helps..
– Kamikaze_goldfish
Nov 23 '18 at 22:55
Oh no I know I'm not expecting any code just a big picture idea if anyone's ever done something similar. I imagine it's not uncommon to want to research an online marketplace with some semblance of exogeneity you know?
– Matthew Sciamanna
Nov 23 '18 at 23:42
Gotcha.. I mean if I were going to do something like this I would go to amazon and start to scrape the pages for all the different departments, then the subcategories of each. After I have all the pages in one list, I would then navigate to them one by one putting all the item absolute urls in a dictionary {'item-name':'url'} and when that has over 10000 items then just use the info however you want. Scrapy would be able to accomplish this the fastest of all.
– Kamikaze_goldfish
Nov 24 '18 at 0:10
add a comment |
Bit of an odd question, but I am looking to get around 10,000 random amazon products for a school thesis and I am having trouble figuring out how to go about getting them. There are a few websites which seem to have this down:
www.randomking.com
www.randasin.com
But because these sites don't list their methodology I can't really use their items directly because I have no guarantee it's true random instead of some database. I'm wondering if any of you may know how to go about doing this or if you might have knowledge on how those guys did that?
Any help is greatly appreciated – I've spent too long banging my head against a wall already.
Thanks!
python web-scraping beautifulsoup amazon amazon-product-api
asked Nov 23 '18 at 5:57
Matthew Sciamanna
214
Bit of an odd question, but I am looking to get around 10,000 random amazon products for a school thesis and I am having trouble figuring out how to go about getting them. There are a few websites which seem to have this down:
www.randomking.com
www.randasin.com
But because these sites don't list their methodology I can't really use their items directly because I have no guarantee it's true random instead of some database. I'm wondering if any of you may know how to go about doing this or if you might have knowledge on how those guys did that?
Any help is greatly appreciated – I've spent too long banging my head against a wall already.
Thanks!
python web-scraping beautifulsoup amazon amazon-product-api
python web-scraping beautifulsoup amazon amazon-product-api
asked Nov 23 '18 at 5:57
Matthew Sciamanna
214
asked Nov 23 '18 at 5:57
Matthew Sciamanna
214
asked Nov 23 '18 at 5:57
Matthew Sciamanna
214
asked Nov 23 '18 at 5:57
Matthew Sciamanna
214
asked Nov 23 '18 at 5:57
Matthew Sciamanna
214
214
1
You should look into working with scrapy. Also, not many people are just going to code something for you. Give it a try to figure out what you need to do and when you run into a problem look it up or post it here for some help. Hope that helps..
– Kamikaze_goldfish
Nov 23 '18 at 22:55
Oh no I know I'm not expecting any code just a big picture idea if anyone's ever done something similar. I imagine it's not uncommon to want to research an online marketplace with some semblance of exogeneity you know?
– Matthew Sciamanna
Nov 23 '18 at 23:42
Gotcha.. I mean if I were going to do something like this I would go to amazon and start to scrape the pages for all the different departments, then the subcategories of each. After I have all the pages in one list, I would then navigate to them one by one putting all the item absolute urls in a dictionary {'item-name':'url'} and when that has over 10000 items then just use the info however you want. Scrapy would be able to accomplish this the fastest of all.
– Kamikaze_goldfish
Nov 24 '18 at 0:10
add a comment |
1
You should look into working with scrapy. Also, not many people are just going to code something for you. Give it a try to figure out what you need to do and when you run into a problem look it up or post it here for some help. Hope that helps..
– Kamikaze_goldfish
Nov 23 '18 at 22:55
Oh no I know I'm not expecting any code just a big picture idea if anyone's ever done something similar. I imagine it's not uncommon to want to research an online marketplace with some semblance of exogeneity you know?
– Matthew Sciamanna
Nov 23 '18 at 23:42
Gotcha.. I mean if I were going to do something like this I would go to amazon and start to scrape the pages for all the different departments, then the subcategories of each. After I have all the pages in one list, I would then navigate to them one by one putting all the item absolute urls in a dictionary {'item-name':'url'} and when that has over 10000 items then just use the info however you want. Scrapy would be able to accomplish this the fastest of all.
– Kamikaze_goldfish
Nov 24 '18 at 0:10
1
1
You should look into working with scrapy. Also, not many people are just going to code something for you. Give it a try to figure out what you need to do and when you run into a problem look it up or post it here for some help. Hope that helps..
– Kamikaze_goldfish
Nov 23 '18 at 22:55
You should look into working with scrapy. Also, not many people are just going to code something for you. Give it a try to figure out what you need to do and when you run into a problem look it up or post it here for some help. Hope that helps..
– Kamikaze_goldfish
Nov 23 '18 at 22:55
Oh no I know I'm not expecting any code just a big picture idea if anyone's ever done something similar. I imagine it's not uncommon to want to research an online marketplace with some semblance of exogeneity you know?
– Matthew Sciamanna
Nov 23 '18 at 23:42
Oh no I know I'm not expecting any code just a big picture idea if anyone's ever done something similar. I imagine it's not uncommon to want to research an online marketplace with some semblance of exogeneity you know?
– Matthew Sciamanna
Nov 23 '18 at 23:42
Gotcha.. I mean if I were going to do something like this I would go to amazon and start to scrape the pages for all the different departments, then the subcategories of each. After I have all the pages in one list, I would then navigate to them one by one putting all the item absolute urls in a dictionary {'item-name':'url'} and when that has over 10000 items then just use the info however you want. Scrapy would be able to accomplish this the fastest of all.
– Kamikaze_goldfish
Nov 24 '18 at 0:10
Gotcha.. I mean if I were going to do something like this I would go to amazon and start to scrape the pages for all the different departments, then the subcategories of each. After I have all the pages in one list, I would then navigate to them one by one putting all the item absolute urls in a dictionary {'item-name':'url'} and when that has over 10000 items then just use the info however you want. Scrapy would be able to accomplish this the fastest of all.
– Kamikaze_goldfish
Nov 24 '18 at 0:10
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53441318%2fgenerate-random-amazon-products%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53441318%2fgenerate-random-amazon-products%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
You should look into working with scrapy. Also, not many people are just going to code something for you. Give it a try to figure out what you need to do and when you run into a problem look it up or post it here for some help. Hope that helps..
– Kamikaze_goldfish
Nov 23 '18 at 22:55
Oh no I know I'm not expecting any code just a big picture idea if anyone's ever done something similar. I imagine it's not uncommon to want to research an online marketplace with some semblance of exogeneity you know?
– Matthew Sciamanna
Nov 23 '18 at 23:42
Gotcha.. I mean if I were going to do something like this I would go to amazon and start to scrape the pages for all the different departments, then the subcategories of each. After I have all the pages in one list, I would then navigate to them one by one putting all the item absolute urls in a dictionary {'item-name':'url'} and when that has over 10000 items then just use the info however you want. Scrapy would be able to accomplish this the fastest of all.
– Kamikaze_goldfish
Nov 24 '18 at 0:10