Charles Karpati | Meetup's API docs

That sounds like us!

To begin. The directions say:

Redirect using the following url in a browser:

And

Once you visit that link:

• "Meetup will ask the user to login if they are not already logged in."

• "If the user has previously authorized access for the provided client_id, Meetup will immediately redirect the user back to the redirect_uri with success query parameters."

__Response Parameters__

• code - A string that can only be used once to request an access token

• state - An opaque string that you may provide in the initial request

Here is an example URL:

https://secure.meetup.com/oauth2/authorize?client_id=pfrvbpt8fpsf50v27o110ltjab&response_type=code&redirect_uri=https://dataworksmd.org/

Clicking on that link will take you to an authorization page.

In the past, we had been getting: "invalid_request" errors. Meaning: "The request was malformed or missing parameters".

Naturally, the query failed when trying here for us because we didn't know the redirect_uri.

There exists more info in original documentation about all this.

Now the docs say:

With all this being done...

Make a post request with the following format using the key given to you in the previous response.

As apposed to a get request, this will not be something you can do by entering a url into an address bar.

Perform POST request for an access token using this:

import requests
 postparameters = {
     'client_id':  'pfrvbpt8fpsf50v27o110ltjab',
     'client_secret': '5hjhr7d87fpv3krkp44f32g24o', 
     'grant_type': 'authorization_code', 
     'redirect_uri': 'https://dataworksmd.org/', 
     'code': 'ee628ebb6ad82caa03d01640c26b6fb5'
   }
 
 url = 'https://secure.meetup.com/oauth2/access' 
 
 x = requests.post(url, data = postparameters)

Lets try it

x

SUCCESS!!

Lets look at the content

x.text

Did it look like whats below? A successfull submisson should look like this:

We got back JSON encoded in a string. Lets get that refresh_token because it's what we need amongst the other credentials to perform our query.

import json
 
 json.loads(x.text)['access_token']

Otherwise, you'll get an error message. More information in the docs.

Once completed, API Queries can be made using the access_token and the KEY from before.

Skip down to the 'Querying Data' section if you made it down here.

Here are a bunch of queries we could make.

Docs, Template: https://api.meetup.com/pro/:urlname/members

!curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/pro/:DataWorks/members

API Query 5 - MEMBER INFORMATION Through Their ID

!curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/DataWorks/members?page=256827702

API Query 6 - Which Cities Are The Members From?

!curl -H "Authorization: Bearer 775d4b6775de1fb108acd0401e319987" https://api.meetup.com/DataWorks/members?page=20

Save Those Queries!

We can use append > filename.json to the end of our Terminal commands to save the output.

!curl -H "Authorization: Bearer 603eb12d73e95a67e057dda7414cc0ca" https://api.meetup.com/DataWorks > DataWorks.json
 !curl -H "Authorization: Bearer 603eb12d73e95a67e057dda7414cc0ca" https://api.meetup.com/DataWorks/members/?page=20000 > DataWorks_members.json
 # !curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/members/256827702?page=20 > members.json
 # !curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/DataWorks/events?page=80&has_ended=0 > DataWorks_events.json
 # !curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/pro/:DataWorks/members > pro_DataWorks_members.json

Colabs has this special feature I will use that lets me store the output of a terminal command (deonted by the !) into a python variable.

In the following instance, I use the cat terminal command to read the contents of each file and store them into variables as a textual 'string'.

From there, much like before, we can open the files with python to view their contents.

import json
 # 
 # Here I mix 
 dataWorksResp = !cat DataWorks.json
 dataWorksJsonObject = json.loads(dataWorksResp[0])
 print(dataWorksJsonObject)
 
 dataWorksMembersResp = !cat DataWorks_members.json
 dataWorksMembJsonObject = json.loads(dataWorksMembersResp[0])
 print(dataWorksMembJsonObject)

It's possible to perform data science with JSON.

dataWorksJsonObject['id']

But for now lets convert it to tabularized form: a CSV

the dataWorksCSV only has one record since we only requested data on one 'group' -> DataWorks

Colabs pre-formats the output for us.

dataWorksCSV

You'll see that the json_normalize function flattened the hierarchical nesting of the json object.

Here are the new column names for this flattened JSON Object.

dataWorksCSV.columns

pd.set_option('display.max_colwidth', None)
 dataWorksMembersCSV.head(12)

Lets take a peek at some information?

norun = """
 from datetime import datetime as dt
 # The `[0]` retrieves the value of the members column of our dataset at the first (and only) index (row).
 print('Number of members: ', dataWorksCSV['members'][0])
 
 # The datetime column is a bit tricky to work with
 created = dataWorksCSV['created'][0]
 print('created: ', dt.fromtimestamp( int(str(created)[:-3]) ).strftime('%Y-%m-%d %H:%M:%S') )
 """

[link text](https:// [link text](https:// link text))Alright.. So we have 2311[link text](https:// link text) members and the group was created mid 2018?

What can we determine about our members?

import pandas as pd

A lot of these columns don't have much information. The status column has only 1 unique value 'active', for instance.

dataWorksMembersCSV['country'].value_counts()[:20]

dataWorksMembersCSV['state'].value_counts()[:20]

dataWorksMembersCSV['state'].value_counts()[1:8].plot()

dataWorksMembersCSV['city'].value_counts()[:30]

created - The time this member joined the Group, represented as milliseconds since the epoch

updated - The last time this member edited their Group profile, represented as milliseconds since the epoch

visited - The last time this member visited the Group, represented as milliseconds since the epoch

df.to_csv('meetup_api_member_data.csv')

we have previously created a dataset 'meetup_api_member_data.csv' by running through step 3-6.

We wont have to do those parts again. Just re-upload the file now.

from google.colab import files
 import io
 import pandas as pd
 uploaded = files.upload()
 for fn in uploaded.keys():
   print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn])))
   df = pd.read_csv(io.BytesIO(uploaded[fn]))
   dataWorksMembersCSV = df.copy()

df.head()

memberdata = df.copy()

df['state'].value_counts()[:6]

ndf=df[ df['joined'] == df['group_profile.created']]

ndf.head()

ndf.to_csv('JoiningDateMatchCreationDateOfMember.csv')

df['state'].value_counts()

df['state'].value_counts()[:8].plot()

!mkdir resp
 %cd resp

ls

First we download data for every single record.

ls

import time
 for index, row in df[684:10000].iterrows():
   print( str(index)+'-'+row['name'].replace(" ","-") )
   url = "https://api.meetup.com/DataWorks/members/"+str(row['id'])+"?fields=memberships&page=20"
   txt = str(index)+'-'+row['name'].replace(" ","-") + ".json"
   print( url, txt )
   t = !curl -o {txt} -H "Authorization: Bearer 6992ff864f9e5f165ff42df94c2bd733" {url}
   time.sleep(5)

Then we flatten it

ls

This is a test to see if it works

import json
 from pandas.io.json import json_normalize
 # 
 # Here I mix 
 txt = !cat 100-Gary-Mann.json
 txt = json.loads(txt[0])
 memberships = False
 # print(txt['memberships']['member'][0])
 try:
   x = pd.DataFrame.from_dict( json_normalize(txt['memberships']['member']) ) 
   x['userId'] = txt['id']
   x['userName'] = txt['name']
   x['userCity'] = txt['city']
   memberships = x.copy() 
   memberships.rename(columns={"status	visited": "visited"})
   memberships = memberships[['userId', 'userName', 'userCity', 'group.localized_location', 'group.urlname', 'group.status', 'group.join_mode', 'group.members', 'group.who',
                              "visited", 'created', 'updated', 'group.id']]
   display(memberships.head())
   # df.reset_index(inplace=True)
 except: print('fail')

memberships.drop(memberships.index[:], inplace=True)
 memberships
 # df = memberships.copy()

This will do the flattening for all records

cd resp

import json
 from pandas.io.json import json_normalize
 import os
 
 directory = os.fsencode('./')
     
 for file in os.listdir(directory):
     filename = os.fsdecode(file)
     # 
     # Here I mix 
     txt = !cat {filename}
     try:
       txt = json.loads(txt[0])
     except: txt = ''
     mems = df.copy() 
     try: 
       x = pd.DataFrame.from_dict( json_normalize(txt['memberships']['member']) ) 
       x['userId'] = txt['id']
       x['userName'] = txt['name']
       x['userCity'] = txt['city']
       # print(x.columns)
       mems = x.copy() 
       mems.rename(columns={"status	visited": "visited"})
       mems = mems[['userId', 'userName', 'userCity', 'group.localized_location', 'group.urlname', 'group.status', 'group.join_mode', 'group.members', 'group.who',
                                 "visited", 'created', 'updated', 'group.id']]
       memberships = pd.concat([memberships, mems], sort=False)
       print('good', filename)
     except: print('fail', filename)
 memberships.reset_index(drop=True, inplace=True)

memberships

cd ../

memberships.to_csv('members_joined_groups.csv')

from google.colab import files
 import io
 import pandas as pd
 uploaded = files.upload()
 for fn in uploaded.keys():
   print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn])))
   memberships = pd.read_csv(io.BytesIO(uploaded[fn]))

memberships['visited'] = memberships.apply(lambda x: dt.fromtimestamp( int(str(x['visited'])[:-3]) ).strftime('%Y-%m-%d'), axis=1, result_type='expand')
 memberships['created'] = memberships.apply(lambda x: dt.fromtimestamp( int(str(x['created'])[:-3]) ).strftime('%Y-%m-%d'), axis=1, result_type='expand')
 memberships['updated'] = memberships.apply(lambda x: dt.fromtimestamp( int(str(x['updated'])[:-3]) ).strftime('%Y-%m-%d'), axis=1, result_type='expand')

memberships

memberships['group.who'].value_counts()

memberships[['group.urlname', 'group.who', 'group.members']].value_counts()

memberships['group.members'].value_counts()

memberships['group.urlname'].value_counts()