Content found herein may be directly quoted from the above links documentation
✖️ Quick Notes
Our Checklist
Before you can use OAuth 2 for user authorization, you need to either register a new OAuth consumer (giving you a client_id) or add (another) redirect_uri to an existing consumer.
We need from each consumer we will need it's:
- ✔️ Public facing
client_idthat can be used with the redirect url to get you to the meetups login/ sign up/ authorize api page. - ✔️ Hidden
client_secret(also refered to as aconsumer_secret). - ✔️ registered
redirect_urifor a given client will be used to validate future oauth2 requests.
There are 3 ways to 'flow' through the Meetup OAuth process.
Server Flow - The path we'd prefer to take. 2 steps. The documentation uses the phrase
Redirect your userfor this sectionServer Flow with User Credentials - requires a username, password for a pro user account holder. Not our prefered route
Implicit Flow - This flow is suitable for JavaScript based browser clients.
Critical_Question: What does it mean if I login and am not the pro account holder of api key?
Critical Answer: The second flow requires a Pro members account user/pass to be passed in the request. The former only requires that the 'user' making the request be registered with meetup and authorize the api access using the api key. Regardless, You will need to be a Pro member to access the pro API features.
Critical Proof: We tried this using the console (logged in using our own meetup accounts) and were denied access but Jason was able to get it working. Perhaps we are doing it wrong? Or more likely, the console doesn't work for us non pro-users full stop, I didn't see a place to put the api key or redirect uri so I don't think the console permits 'does' that part.
👉 Server Flow
1. 👉 Requesting Authorization
That sounds like us!
To begin. The directions say:
Redirect using the following url in a browser:
https://secure.meetup.com/oauth2/authorize ?client_id=YOUR_CONSUMER_KEY &response_type=code &redirect_uri=YOUR_CONSUMER_REDIRECT_URI
And
Once you visit that link:
"Meetup will ask the user to login if they are not already logged in."
"If the user has previously authorized access for the provided client_id, Meetup will immediately redirect the user back to the redirect_uri with success query parameters."
__Response Parameters__
- code - A string that can only be used once to request an access token
- state - An opaque string that you may provide in the initial request
Here is an example URL:
Clicking on that link will take you to an authorization page.
In the past, we had been getting: "invalid_request" errors. Meaning: "The request was malformed or missing parameters".
Naturally, the query failed when trying here for us because we didn't know the redirect_uri.
There exists more info in original documentation about all this.
2. 👉 Requesting Access Token
Now the docs say:
With all this being done...
Make a post request with the following format using the key given to you in the previous response.
As apposed to a get request, this will not be something you can do by entering a url into an address bar.
client_id=YOUR_CONSUMER_KEY &client_secret=YOUR_CONSUMER_SECRET &grant_type=authorization_code &redirect_uri=SAME_REDIRECT_URI_USED_FOR_PREVIOUS_STEP &code=CODE_YOU_RECEIVED_FROM_THE_AUTHORIZATION_RESPONSE
Perform POST request for an access token using this:
postparameters = { 'client_id': 'pfrvbpt8fpsf50v27o110ltjab', 'client_secret': '5hjhr7d87fpv3krkp44f32g24o', 'grant_type': 'authorization_code', 'redirect_uri': 'https://dataworksmd.org/', 'code': 'ee628ebb6ad82caa03d01640c26b6fb5' } url = 'https://secure.meetup.com/oauth2/access' x = requests.post(url, data = postparameters)Lets try it
SUCCESS!!
Lets look at the content
Did it look like whats below? A successfull submisson should look like this:
{
"access_token":"ACCESS_TOKEN_TO_STORE",
"token_type":"bearer",
"expires_in":3600,
"refresh_token":"TOKEN_USED_TO_REFRESH_AUTHORIZATION"
}
We got back JSON encoded in a string. Lets get that refresh_token because it's what we need amongst the other credentials to perform our query.
Otherwise, you'll get an error message. More information in the docs.
Once completed, API Queries can be made using the access_token and the KEY from before.
Skip down to the 'Querying Data' section if you made it down here.
Here are a bunch of queries we could make.
3. ✖️ Making Queries Examples
API Query 1 - SELF
Making Authenticated Requests ¶
After successfully obtaining member authorization and an access_token you may now make authenticated API requests using HTTPS by supplying the token value in an Authorization header prefixed with bearer and a space.
curl -H "Authorization: Bearer {access_token}" https://api.meetup.com/members/self/
Docs, Template: https://api.meetup.com/self/groups
API Query 2 - GROUPS
Docs, Docs.P2, Template: https://api.meetup.com/:urlname
"created": 1530147506000, "members": 2307,
"next_event": { "id": "278394107", "yes_rsvp_count": 26,
"name": "Online: Introducing Datawave - Scalable Data Ingest and Query",
API Query 3a - GROUP Members
Docs, Docs.P2, Template: https://api.meetup.com/:urlname/members
Docs, Template: https://api.meetup.com:urlname/members/:member_id
Docs, Template: https://api.meetup.com/members/:member_id
API Query 3b - GROUP Events
Docs, Template: https://api.meetup.com/:urlname/events
API Query 4 - PRO
Docs.P2 -> # events attended join_time last_access_time
4. ✖️ Saving our Members List
Docs, Template: https://api.meetup.com/pro/:urlname/members
API Query 5 - MEMBER INFORMATION Through Their ID
# This is formatted as code
API Query 6 - Which Cities Are The Members From?
Save Those Queries!
We can use append > filename.json to the end of our Terminal commands to save the output.
5. ✖️ Convert the Response's JSON Data to a Tabular excel-like Form.
Colabs has this special feature I will use that lets me store the output of a terminal command (deonted by the !) into a python variable.
In the following instance, I use the cat terminal command to read the contents of each file and store them into variables as a textual 'string'.
From there, much like before, we can open the files with python to view their contents.
# # Here I mix dataWorksResp = !cat DataWorks.json dataWorksJsonObject = json.loads(dataWorksResp[0]) print(dataWorksJsonObject) dataWorksMembersResp = !cat DataWorks_members.json dataWorksMembJsonObject = json.loads(dataWorksMembersResp[0]) print(dataWorksMembJsonObject)It's possible to perform data science with JSON.
But for now lets convert it to tabularized form: a CSV
the dataWorksCSV only has one record since we only requested data on one 'group' -> DataWorks
Colabs pre-formats the output for us.
You'll see that the json_normalize function flattened the hierarchical nesting of the json object.
Here are the new column names for this flattened JSON Object.
6. ✖️ Format the Columns (date-time) and do Preliminary Data Exploration. Then Save It.
dataWorksMembersCSV.head(12)Lets take a peek at some information?
from datetime import datetime as dt # The `[0]` retrieves the value of the members column of our dataset at the first (and only) index (row). print('Number of members: ', dataWorksCSV['members'][0]) # The datetime column is a bit tricky to work with created = dataWorksCSV['created'][0] print('created: ', dt.fromtimestamp( int(str(created)[:-3]) ).strftime('%Y-%m-%d %H:%M:%S') ) """[link text](https:// [link text](https:// link text))Alright.. So we have 2311[link text](https:// link text) members and the group was created mid 2018?
What can we determine about our members?
A lot of these columns don't have much information. The status column has only 1 unique value 'active', for instance.
created - The time this member joined the Group, represented as milliseconds since the epoch
updated - The last time this member edited their Group profile, represented as milliseconds since the epoch
visited - The last time this member visited the Group, represented as milliseconds since the epoch
7. 👉 Reuploading the Cleaned-Up Members List from step 6.
we have previously created a dataset 'meetup_api_member_data.csv' by running through step 3-6.
We wont have to do those parts again. Just re-upload the file now.
import io import pandas as pd uploaded = files.upload() for fn in uploaded.keys(): print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn]))) df = pd.read_csv(io.BytesIO(uploaded[fn])) dataWorksMembersCSV = df.copy()8. 👉 Use Member ID's from the Members-List to get every Members Full Info.
%cd respFirst we download data for every single record.
for index, row in df[684:10000].iterrows(): print( str(index)+'-'+row['name'].replace(" ","-") ) url = "https://api.meetup.com/DataWorks/members/"+str(row['id'])+"?fields=memberships&page=20" txt = str(index)+'-'+row['name'].replace(" ","-") + ".json" print( url, txt ) t = !curl -o {txt} -H "Authorization: Bearer 6992ff864f9e5f165ff42df94c2bd733" {url} time.sleep(5)Then we flatten it
This is a test to see if it works
from pandas.io.json import json_normalize # # Here I mix txt = !cat 100-Gary-Mann.json txt = json.loads(txt[0]) memberships = False # print(txt['memberships']['member'][0]) try: x = pd.DataFrame.from_dict( json_normalize(txt['memberships']['member']) ) x['userId'] = txt['id'] x['userName'] = txt['name'] x['userCity'] = txt['city'] memberships = x.copy() memberships.rename(columns={"status visited": "visited"}) memberships = memberships[['userId', 'userName', 'userCity', 'group.localized_location', 'group.urlname', 'group.status', 'group.join_mode', 'group.members', 'group.who', "visited", 'created', 'updated', 'group.id']] display(memberships.head()) # df.reset_index(inplace=True) except: print('fail') memberships # df = memberships.copy()This will do the flattening for all records
from pandas.io.json import json_normalize import os directory = os.fsencode('./') for file in os.listdir(directory): filename = os.fsdecode(file) # # Here I mix txt = !cat {filename} try: txt = json.loads(txt[0]) except: txt = '' mems = df.copy() try: x = pd.DataFrame.from_dict( json_normalize(txt['memberships']['member']) ) x['userId'] = txt['id'] x['userName'] = txt['name'] x['userCity'] = txt['city'] # print(x.columns) mems = x.copy() mems.rename(columns={"status visited": "visited"}) mems = mems[['userId', 'userName', 'userCity', 'group.localized_location', 'group.urlname', 'group.status', 'group.join_mode', 'group.members', 'group.who', "visited", 'created', 'updated', 'group.id']] memberships = pd.concat([memberships, mems], sort=False) print('good', filename) except: print('fail', filename) memberships.reset_index(drop=True, inplace=True)9. 👉 Look into Membership data
import io import pandas as pd uploaded = files.upload() for fn in uploaded.keys(): print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn]))) memberships = pd.read_csv(io.BytesIO(uploaded[fn])) memberships['created'] = memberships.apply(lambda x: dt.fromtimestamp( int(str(x['created'])[:-3]) ).strftime('%Y-%m-%d'), axis=1, result_type='expand') memberships['updated'] = memberships.apply(lambda x: dt.fromtimestamp( int(str(x['updated'])[:-3]) ).strftime('%Y-%m-%d'), axis=1, result_type='expand')❌ Server Flow with User Credentials
This aint us
❌ Implicit Flow
Yeah, this'll work too. But the documentation is all of 100 words or something.
And a really awkwardly worded 100 words at that:
The response parameters listed in the server flow's success (with the exception of
refresh_token) and failure access token responses will be included in the implicit
authorization's client response appended as a url fragment.
Which really just means:
Because this information is encoded in a url fragment,
it can only be retrieved with client-side browser scripts.
I'm sure that this will all make sense when we get to it..?
The rest of the documentation in this section just state that when a user visits your website and a request for data is sent via the useres browser, the user will be redirected to meetup and be asked to register or login and check a authorization check box or two.
Awesome. Lets get to work..?
Javascript Fetch Request
Or not.
I mean. We really tried everything. This one could still work, though!
You may need to allow cors to run it. Heres a chrome extension for that.
Btw. The code here is just a copy of the html file I used to test it (Spoiler: Invalid credentials).
<script> var url = "https://secure.meetup.com/oauth2/authorize?client_id= &response_type=token&redirect_uri=https://bniajfi.org/"; fetch(url) .then(response => response.json()) .then(data => console.log(data)); </script> <!-- https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch -->
❌ Python Package: Meetup-API
There are few to no resources other than the official documentation on the web to help inform us.
But this python library existed and I figured maybe it'd be the easy way out!
Here's the code I was using to test it. You can run it if you like...?
(Spoiler: This library is depricated as of 2 years ago).
(Spoiler P2: I tried manually patching the library to no avail )
client = meetup.api.Client('') type(client) group_info = client.GetGroup({'urlname': 'DataWorks'}) type(group_info) group_info.__dict__.keys() group_info.id group_info.name group_info.link