Home Labs Nb 2 Html Tests 211 Web Scraper Scooter Exploration Meetup Wordclouds Apis Looking At Data Satellltes

Don't Look! I'm changing!

URL Copied

Meetup API

BinderBinderBinderOpen Source Love svg3

NPM LicenseActiveGitHub last commit

GitHub starsGitHub watchersGitHub forksGitHub followers

Content found herein may be directly quoted from the above links documentation

✖️ Quick Notes

Our Checklist

Before you can use OAuth 2 for user authorization, you need to either register a new OAuth undefined (giving you a undefined) or add (another) undefined to an existing consumer.

We need from each undefined we will need it's:

  • ✔️ Public facing undefined that can be used with the redirect url to get you to the meetups login/ sign up/ authorize api page.

  • ✔️ Hidden undefined (also refered to as a undefined).

  • ✔️ registered undefined for a given client will be used to validate future oauth2 requests.

There are 3 ways to 'flow' through the Meetup OAuth process.

  • Server Flow - The path we'd prefer to take. 2 steps. The documentation uses the phrase undefined for this section

  • Server Flow with User Credentials - requires a username, password for a pro user account holder. Not our prefered route

  • Implicit Flow - This flow is suitable for JavaScript based browser clients.

Critical_Question: What does it mean if I login and am not the pro account holder of api key?

Critical Answer: The second flow requires a Pro members account user/pass to be passed in the request. The former only requires that the 'user' making the request be registered with meetup and authorize the api access using the api key. Regardless, You will need to be a Pro member to access the pro API features.

Critical Proof: We tried this using the console (logged in using our own meetup accounts) and were denied access but Jason was able to get it working. Perhaps we are doing it wrong? Or more likely, the console doesn't work for us non pro-users full stop, I didn't see a place to put the api key or redirect uri so I don't think the console permits 'does' that part.

👉 Server Flow

1. 👉 Requesting Authorization

That sounds like us!

To begin. The directions say:

Redirect using the following url in a browser:

undefined

And

Once you visit that link:

  • "Meetup will ask the user to login if they are not already logged in."

  • "If the user has previously authorized access for the provided client_id, Meetup will immediately redirect the user back to the redirect_uri with success query parameters."

__Response Parameters__

  • code - A string that can only be used once to request an access token

  • state - An opaque string that you may provide in the initial request

Here is an example URL:

https://secure.meetup.com/oauth2/authorize?client_id=pfrvbpt8fpsf50v27o110ltjab&response_type=code&redirect_uri=https://dataworksmd.org/

Clicking on that link will take you to an authorization page.

image.png

image.png

In the past, we had been getting: "invalid_request" errors. Meaning: "The request was malformed or missing parameters".

Naturally, the query failed when trying here for us because we didn't know the redirect_uri.

There exists more info in original documentation about all this.

2. 👉 Requesting Access Token

Now the docs say:

With all this being done...

Make a post request with the following format using the key given to you in the previous response.

As apposed to a get request, this will not be something you can do by entering a url into an address bar.

undefined

Perform POST request for an access token using this:

import requests
 postparameters = {
     'client_id':  'pfrvbpt8fpsf50v27o110ltjab',
     'client_secret': '5hjhr7d87fpv3krkp44f32g24o', 
     'grant_type': 'authorization_code', 
     'redirect_uri': 'https://dataworksmd.org/', 
     'code': 'ee628ebb6ad82caa03d01640c26b6fb5'
   }
 
 url = 'https://secure.meetup.com/oauth2/access' 
 
 x = requests.post(url, data = postparameters)

Lets try it

x

SUCCESS!!

Lets look at the content

x.text

Did it look like whats below? A successfull submisson should look like this:

undefined

We got back JSON encoded in a string. Lets get that undefined because it's what we need amongst the other credentials to perform our query.

import json
 
 json.loads(x.text)['access_token']

Otherwise, you'll get an error message. More information in the docs.

Once completed, API Queries can be made using the access_token and the KEY from before.

Skip down to the 'Querying Data' section if you made it down here.

Here are a bunch of queries we could make.

3. ✖️ Making Queries Examples

API Query 1 - SELF
undefined
!curl -H "Authorization: Bearer 327ce972bbf7e2839a5323702ee91f78" https://api.meetup.com/members/self/

Docs, Template: undefined

!curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/self/groups?page=20
API Query 2 - GROUPS

Docs, Docs.P2, Template: undefined

"created": 1530147506000, "members": 2307,

"next_event": { "id": "278394107", "yes_rsvp_count": 26,

"name": "Online: Introducing Datawave - Scalable Data Ingest and Query",

!curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/DataWorks
API Query 3a - GROUP Members

Docs, Docs.P2, Template: undefined

!curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/DataWorks/members?page=20

Docs, Template: undefined

!curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/DataWorks/members/256827702?page=20

Docs, Template: undefined

!curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/members/256827702?page=20
API Query 3b - GROUP Events

Docs, Template: undefined

!curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/DataWorks/events?page=80&has_ended=0
API Query 4 - PRO

Docs.P2 -> # events attended join_time last_access_time

4. ✖️ Saving our Members List

Docs, Template: undefined

!curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/pro/:DataWorks/members

API Query 5 - MEMBER INFORMATION Through Their ID

undefined
!curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/DataWorks/members?page=256827702

API Query 6 - Which Cities Are The Members From?

!curl -H "Authorization: Bearer 775d4b6775de1fb108acd0401e319987" https://api.meetup.com/DataWorks/members?page=20
Save Those Queries!

We can use append undefined to the end of our Terminal commands to save the output.

!curl -H "Authorization: Bearer 603eb12d73e95a67e057dda7414cc0ca" https://api.meetup.com/DataWorks > DataWorks.json
 !curl -H "Authorization: Bearer 603eb12d73e95a67e057dda7414cc0ca" https://api.meetup.com/DataWorks/members/?page=20000 > DataWorks_members.json
 # !curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/members/256827702?page=20 > members.json
 # !curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/DataWorks/events?page=80&has_ended=0 > DataWorks_events.json
 # !curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/pro/:DataWorks/members > pro_DataWorks_members.json

5. ✖️ Convert the Response's JSON Data to a Tabular excel-like Form.

Colabs has this special feature I will use that lets me store the output of a terminal command (deonted by the undefined) into a python variable.

In the following instance, I use the undefined terminal command to read the contents of each file and store them into variables as a textual 'string'.

From there, much like before, we can open the files with python to view their contents.

import json
 # 
 # Here I mix 
 dataWorksResp = !cat DataWorks.json
 dataWorksJsonObject = json.loads(dataWorksResp[0])
 print(dataWorksJsonObject)
 
 dataWorksMembersResp = !cat DataWorks_members.json
 dataWorksMembJsonObject = json.loads(dataWorksMembersResp[0])
 print(dataWorksMembJsonObject)

It's possible to perform data science with JSON.

dataWorksJsonObject['id']

But for now lets convert it to tabularized form: a CSV

the dataWorksCSV only has one record since we only requested data on one 'group' -> DataWorks

Colabs pre-formats the output for us.

dataWorksCSV

You'll see that the undefined function flattened the hierarchical nesting of the json object.

Here are the new column names for this flattened JSON Object.

dataWorksCSV.columns

6. ✖️ Format the Columns (date-time) and do Preliminary Data Exploration. Then Save It.

pd.set_option('display.max_colwidth', None)
 dataWorksMembersCSV.head(12)

Lets take a peek at some information?

norun = """
 from datetime import datetime as dt
 # The `[0]` retrieves the value of the members column of our dataset at the first (and only) index (row).
 print('Number of members: ', dataWorksCSV['members'][0])
 
 # The datetime column is a bit tricky to work with
 created = dataWorksCSV['created'][0]
 print('created: ', dt.fromtimestamp( int(str(created)[:-3]) ).strftime('%Y-%m-%d %H:%M:%S') )
 """

[link text](https:// [link text](https:// link text))Alright.. So we have 2311[link text](https:// link text) members and the group was created mid 2018?

What can we determine about our members?

import pandas as pd

A lot of these columns don't have much information. The status column has only 1 unique value 'active', for instance.

dataWorksMembersCSV['country'].value_counts()[:20]
dataWorksMembersCSV['state'].value_counts()[:20]
dataWorksMembersCSV['state'].value_counts()[1:8].plot()
dataWorksMembersCSV['city'].value_counts()[:30]

created - The time this member joined the Group, represented as milliseconds since the epoch

updated - The last time this member edited their Group profile, represented as milliseconds since the epoch

visited - The last time this member visited the Group, represented as milliseconds since the epoch

df.to_csv('meetup_api_member_data.csv')

7. 👉 Reuploading the Cleaned-Up Members List from step 6.

we have previously created a dataset 'meetup_api_member_data.csv' by running through step 3-6.

We wont have to do those parts again. Just re-upload the file now.

from google.colab import files
 import io
 import pandas as pd
 uploaded = files.upload()
 for fn in uploaded.keys():
   print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn])))
   df = pd.read_csv(io.BytesIO(uploaded[fn]))
   dataWorksMembersCSV = df.copy()
df.head()
memberdata = df.copy()
df['state'].value_counts()[:6]
ndf=df[ df['joined'] == df['group_profile.created']]
ndf.head()
ndf.to_csv('JoiningDateMatchCreationDateOfMember.csv')
df['state'].value_counts()
df['state'].value_counts()[:8].plot()

8. 👉 Use Member ID's from the Members-List to get every Members Full Info.

!mkdir resp
 %cd resp
ls

First we download data for every single record.

ls
import time
 for index, row in df[684:10000].iterrows():
   print( str(index)+'-'+row['name'].replace(" ","-") )
   url = "https://api.meetup.com/DataWorks/members/"+str(row['id'])+"?fields=memberships&page=20"
   txt = str(index)+'-'+row['name'].replace(" ","-") + ".json"
   print( url, txt )
   t = !curl -o {txt} -H "Authorization: Bearer 6992ff864f9e5f165ff42df94c2bd733" {url}
   time.sleep(5)

Then we flatten it

ls

This is a test to see if it works

import json
 from pandas.io.json import json_normalize
 # 
 # Here I mix 
 txt = !cat 100-Gary-Mann.json
 txt = json.loads(txt[0])
 memberships = False
 # print(txt['memberships']['member'][0])
 try:
   x = pd.DataFrame.from_dict( json_normalize(txt['memberships']['member']) ) 
   x['userId'] = txt['id']
   x['userName'] = txt['name']
   x['userCity'] = txt['city']
   memberships = x.copy() 
   memberships.rename(columns={"status	visited": "visited"})
   memberships = memberships[['userId', 'userName', 'userCity', 'group.localized_location', 'group.urlname', 'group.status', 'group.join_mode', 'group.members', 'group.who',
                              "visited", 'created', 'updated', 'group.id']]
   display(memberships.head())
   # df.reset_index(inplace=True)
 except: print('fail')
memberships.drop(memberships.index[:], inplace=True)
 memberships
 # df = memberships.copy()

This will do the flattening for all records

cd resp
import json
 from pandas.io.json import json_normalize
 import os
 
 directory = os.fsencode('./')
     
 for file in os.listdir(directory):
     filename = os.fsdecode(file)
     # 
     # Here I mix 
     txt = !cat {filename}
     try:
       txt = json.loads(txt[0])
     except: txt = ''
     mems = df.copy() 
     try: 
       x = pd.DataFrame.from_dict( json_normalize(txt['memberships']['member']) ) 
       x['userId'] = txt['id']
       x['userName'] = txt['name']
       x['userCity'] = txt['city']
       # print(x.columns)
       mems = x.copy() 
       mems.rename(columns={"status	visited": "visited"})
       mems = mems[['userId', 'userName', 'userCity', 'group.localized_location', 'group.urlname', 'group.status', 'group.join_mode', 'group.members', 'group.who',
                                 "visited", 'created', 'updated', 'group.id']]
       memberships = pd.concat([memberships, mems], sort=False)
       print('good', filename)
     except: print('fail', filename)
 memberships.reset_index(drop=True, inplace=True)
memberships
cd ../
memberships.to_csv('members_joined_groups.csv')

9. 👉 Look into Membership data

from google.colab import files
 import io
 import pandas as pd
 uploaded = files.upload()
 for fn in uploaded.keys():
   print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn])))
   memberships = pd.read_csv(io.BytesIO(uploaded[fn]))
memberships['visited'] = memberships.apply(lambda x: dt.fromtimestamp( int(str(x['visited'])[:-3]) ).strftime('%Y-%m-%d'), axis=1, result_type='expand')
 memberships['created'] = memberships.apply(lambda x: dt.fromtimestamp( int(str(x['created'])[:-3]) ).strftime('%Y-%m-%d'), axis=1, result_type='expand')
 memberships['updated'] = memberships.apply(lambda x: dt.fromtimestamp( int(str(x['updated'])[:-3]) ).strftime('%Y-%m-%d'), axis=1, result_type='expand')
memberships
memberships['group.who'].value_counts()
memberships[['group.urlname', 'group.who', 'group.members']].value_counts()
memberships['group.members'].value_counts()
memberships['group.urlname'].value_counts()

❌ Server Flow with User Credentials

This aint us

❌ Implicit Flow

Yeah, this'll work too. But the documentation is all of 100 words or something.

And a really awkwardly worded 100 words at that:

undefined

Which really just means:

undefined

I'm sure that this will all make sense when we get to it..?

The rest of the documentation in this section just state that when a user visits your website and a request for data is sent via the useres browser, the user will be redirected to meetup and be asked to register or login and check a authorization check box or two.

Awesome. Lets get to work..?

Javascript Fetch Request

Or not.

I mean. We really tried everything. This one could still work, though!

You may need to allow cors to run it. Heres a chrome extension for that.

Btw. The code here is just a copy of the html file I used to test it (Spoiler: Invalid credentials).

undefined

❌ Python Package: Meetup-API

There are few to no resources other than the official documentation on the web to help inform us.

But this python library existed and I figured maybe it'd be the easy way out!

Here's the code I was using to test it. You can run it if you like...?

(Spoiler: This library is depricated as of 2 years ago).

(Spoiler P2: I tried manually patching the library to no avail )

!pip install meetup-api
import meetup.api
 client = meetup.api.Client('')
 type(client)
 group_info = client.GetGroup({'urlname': 'DataWorks'})
 type(group_info)
 group_info.__dict__.keys()
 group_info.id
 group_info.name
 group_info.link