{"id":96,"date":"2019-01-19T22:07:39","date_gmt":"2019-01-19T22:07:39","guid":{"rendered":"http:\/\/api.intelligentonlinetools.com\/diy\/?p=96"},"modified":"2019-01-22T01:54:11","modified_gmt":"2019-01-22T01:54:11","slug":"reddit","status":"publish","type":"post","link":"http:\/\/api.intelligentonlinetools.com\/diy\/reddit\/","title":{"rendered":"How to Get Submission and Comments with Python Reddit API Wrapper – PRAW."},"content":{"rendered":"

\"\"<\/p>\n

According to Alexa [1] people spent more time on Reddit than on Facebook, Instagramm or Youtube. Users use Reddit to post questions, share content or ideas and discuss topics. So it is very interesting to extract automatically text data from this web service. We will look how to do this with PRAW<\/b> – The Python Reddit API Wrapper.[2]<\/p>\n

The example of how to get API key and use python PRAW API can be found at How to scrape reddit with python<\/a> It is however is not adding all comments, that might be attached to submission. Comments can have important information so I decided to build the python script with PRAW API that is modified from above link for adding comments and few minor things.<\/p>\n

To get comments we first need to obtain a submission<\/b> object.
\nWith a submission object we can then like below:<\/p>\n

\r\ncomment_body = ""\r\n    for comment in submission.comments.list():\r\n        print(comment.body)\r\n        comment_body =  comment_body + comment.body + "\\n"\r\n<\/pre>\n
If we wanted to output only the body of the top level comments in the thread we could do:<\/p>\n
\r\nfor top_level_comment in submission.comments:\r\n    print(top_level_comment.body)\r\n<\/pre>\n
Here is the full python script<\/b> of API example that can get Reddit information including comments. Note that as we only downloading data and not changing anything, we do not need user name and password. But in case you modifying data on reddit, you would need include login information too.<\/p>\n
\r\n"""\r\nTo install module:\r\npip install praw\r\n"""\r\n\r\nimport praw\r\nimport pandas as pd\r\nfrom datetime import datetime\r\n\r\n\r\nreddit = praw.Reddit(client_id='xxxxxxxx', \\\r\n                     client_secret='xxxxxxxx', \\\r\n                     user_agent='personal use script')   \r\n                     ##username='YOUR_REDDIT_USER_NAME', \\\r\n                     ###password='YOUR_REDDIT_LOGIN_PASSWORD')\r\n\r\n\r\ndef get_yyyy_mm_dd_from_utc(dt):\r\n    date = datetime.utcfromtimestamp(dt)\r\n   \r\n    return str(date.year) + "-" + str(date.month) + "-" + str(date.day)\r\n\r\n\r\n\r\nsubreddit = reddit.subreddit('learnmachinelearning')\r\n\r\n\r\ntop_subreddit = subreddit.top(limit=998)\r\n\r\ntopics_dict = { "title":[], "score":[], "id":[], "url":[], \\\r\n                "comms_num": [], "created": [],  "body":[], "z_comments":[]}\r\n\r\n\r\nfor submission in top_subreddit:\r\n    \r\n    # https:\/\/www.reddit.com\/r\/redditdev\/comments\/46g9ao\/using_praw_to_call_reddit_api_need_help\/\r\n    topics_dict["title"].append(submission.title)\r\n    topics_dict["score"].append(submission.score)\r\n    topics_dict["id"].append(submission.id)\r\n    topics_dict["url"].append(submission.url)\r\n    topics_dict["comms_num"].append(submission.num_comments)\r\n    topics_dict["created"].append(get_yyyy_mm_dd_from_utc(submission.created))\r\n    topics_dict["body"].append(submission.selftext)\r\n    \r\n   \r\n  \r\n    all_comments = submission.comments.list()\r\n    print (all_comments)\r\n  \r\n    \r\n  \r\n    # https:\/\/praw.readthedocs.io\/en\/latest\/tutorials\/comments.html\r\n    submission.comments.replace_more(limit=None)\r\n    comment_body = ""\r\n    for comment in submission.comments.list():\r\n        print(comment.body)\r\n        comment_body =  comment_body + comment.body + "\\n"\r\n    topics_dict["z_comments"].append (comment_body)\r\n\r\ntopics_data = pd.DataFrame(topics_dict)\r\n\r\ntopics_data.to_csv('Reddit_data.csv', index=False) \r\n<\/pre>\n
References<\/strong>
\n1. The top 500 sites on the web<\/a>
\n2. PRAW<\/a>
\n3. How to scrape reddit with python<\/a>
\n4. Tutorials<\/a>
\n5. Webscraping Reddit\u200a\u2014\u200aPython Reddit API Wrapper (PRAW) Tutorial for Windows<\/a>  <\/p>\n","protected":false},"excerpt":{"rendered":"
According to Alexa [1] people spent more time on Reddit than on Facebook, Instagramm or Youtube. Users use Reddit to post questions, share content or ideas and discuss topics. So it is very interesting to extract automatically text data from this web service. We will look how to do this with PRAW – The Python … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_mi_skip_tracking":false},"categories":[4],"tags":[5,25,24,23],"aioseo_notices":[],"yoast_head":"\nHow to Get Submission and Comments with Python Reddit API Wrapper - PRAW. - Using Web API with Python<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Get Submission and Comments with Python Reddit API Wrapper - PRAW. - Using Web API with Python\" \/>\n<meta property=\"og:description\" content=\"According to Alexa [1] people spent more time on Reddit than on Facebook, Instagramm or Youtube. Users use Reddit to post questions, share content or ideas and discuss topics. So it is very interesting to extract automatically text data from this web service. We will look how to do this with PRAW – The Python ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/\" \/>\n<meta property=\"og:site_name\" content=\"Using Web API with Python\" \/>\n<meta property=\"article:published_time\" content=\"2019-01-19T22:07:39+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-01-22T01:54:11+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/api.intelligentonlinetools.com\/diy\/wp-content\/uploads\/2019\/01\/reddit-1007072_640-e1548119638346.jpg\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/\",\"url\":\"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/\",\"name\":\"How to Get Submission and Comments with Python Reddit API Wrapper - PRAW. - Using Web API with Python\",\"isPartOf\":{\"@id\":\"http:\/\/api.intelligentonlinetools.com\/diy\/#website\"},\"datePublished\":\"2019-01-19T22:07:39+00:00\",\"dateModified\":\"2019-01-22T01:54:11+00:00\",\"author\":{\"@id\":\"http:\/\/api.intelligentonlinetools.com\/diy\/#\/schema\/person\/6788923b31e0ea6b3bfa2c079adb1332\"},\"breadcrumb\":{\"@id\":\"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/api.intelligentonlinetools.com\/diy\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Get Submission and Comments with Python Reddit API Wrapper – PRAW.\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/api.intelligentonlinetools.com\/diy\/#website\",\"url\":\"http:\/\/api.intelligentonlinetools.com\/diy\/\",\"name\":\"Using Web API with Python\",\"description\":\"Using Web API with Python\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/api.intelligentonlinetools.com\/diy\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/api.intelligentonlinetools.com\/diy\/#\/schema\/person\/6788923b31e0ea6b3bfa2c079adb1332\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/api.intelligentonlinetools.com\/diy\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"},\"url\":\"http:\/\/api.intelligentonlinetools.com\/diy\/author\/owygs156\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Get Submission and Comments with Python Reddit API Wrapper - PRAW. - Using Web API with Python","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/","og_locale":"en_US","og_type":"article","og_title":"How to Get Submission and Comments with Python Reddit API Wrapper - PRAW. - Using Web API with Python","og_description":"According to Alexa [1] people spent more time on Reddit than on Facebook, Instagramm or Youtube. Users use Reddit to post questions, share content or ideas and discuss topics. So it is very interesting to extract automatically text data from this web service. We will look how to do this with PRAW – The Python ... Read more","og_url":"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/","og_site_name":"Using Web API with Python","article_published_time":"2019-01-19T22:07:39+00:00","article_modified_time":"2019-01-22T01:54:11+00:00","og_image":[{"url":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-content\/uploads\/2019\/01\/reddit-1007072_640-e1548119638346.jpg"}],"author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/","url":"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/","name":"How to Get Submission and Comments with Python Reddit API Wrapper - PRAW. - Using Web API with Python","isPartOf":{"@id":"http:\/\/api.intelligentonlinetools.com\/diy\/#website"},"datePublished":"2019-01-19T22:07:39+00:00","dateModified":"2019-01-22T01:54:11+00:00","author":{"@id":"http:\/\/api.intelligentonlinetools.com\/diy\/#\/schema\/person\/6788923b31e0ea6b3bfa2c079adb1332"},"breadcrumb":{"@id":"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/api.intelligentonlinetools.com\/diy\/reddit\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/api.intelligentonlinetools.com\/diy\/"},{"@type":"ListItem","position":2,"name":"How to Get Submission and Comments with Python Reddit API Wrapper – PRAW."}]},{"@type":"WebSite","@id":"http:\/\/api.intelligentonlinetools.com\/diy\/#website","url":"http:\/\/api.intelligentonlinetools.com\/diy\/","name":"Using Web API with Python","description":"Using Web API with Python","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/api.intelligentonlinetools.com\/diy\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/api.intelligentonlinetools.com\/diy\/#\/schema\/person\/6788923b31e0ea6b3bfa2c079adb1332","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/api.intelligentonlinetools.com\/diy\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"},"url":"http:\/\/api.intelligentonlinetools.com\/diy\/author\/owygs156\/"}]}},"_links":{"self":[{"href":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-json\/wp\/v2\/posts\/96"}],"collection":[{"href":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-json\/wp\/v2\/comments?post=96"}],"version-history":[{"count":13,"href":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-json\/wp\/v2\/posts\/96\/revisions"}],"predecessor-version":[{"id":113,"href":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-json\/wp\/v2\/posts\/96\/revisions\/113"}],"wp:attachment":[{"href":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-json\/wp\/v2\/media?parent=96"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-json\/wp\/v2\/categories?post=96"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/api.intelligentonlinetools.com\/diy\/wp-json\/wp\/v2\/tags?post=96"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}