Home 00 Datalabs 01 Scooter Explorati... 02 211 Web Scraper 03 Nbdev Create Clea... 04 Wordclouds 05 Amivi Meetup
  Quick Notes
  Server Flow
 Redirect Usi...
   Making Que...
   Saving Our...
   Convert Th...
   Format The...
   Use Member...
   Look Into ...
  Server Flow...
  Implicit Fl...
   Javascript F...
  Python Pack...
06 Courts Mechanical... 07 Nb 2 Html Tests 08 Apis 09 Looking At Data

Don't Look! I'm changing!

URL Copied

Musical loops

BinderBinderBinderOpen Source Love svg3

NPM LicenseActiveGitHub last commit

GitHub starsGitHub watchersGitHub forksGitHub followers

TweetTwitter Follow

Content found herein may be directly quoted from the above links documentation

βœ–οΈ Quick Notes

Our Checklist

Before you can use OAuth 2 for user authorization, you need to either register a new OAuth consumer (giving you a client_id) or add (another) redirect_uri to an existing consumer.

We need from each consumer we will need it's:

There are 3 ways to 'flow' through the Meetup OAuth process.

Critical_Question: What does it mean if I login and am not the pro account holder of api key?

Critical Answer: The second flow requires a Pro members account user/pass to be passed in the request. The former only requires that the 'user' making the request be registered with meetup and authorize the api access using the api key. Regardless, You will need to be a Pro member to access the pro API features.

Critical Proof: We tried this using the console (logged in using our own meetup accounts) and were denied access but Jason was able to get it working. Perhaps we are doing it wrong? Or more likely, the console doesn't work for us non pro-users full stop, I didn't see a place to put the api key or redirect uri so I don't think the console permits 'does' that part.

πŸ‘‰ Server Flow

1. πŸ‘‰ Requesting Authorization

That sounds like us!

To begin. The directions say:

Redirect using the following url in a browser:



Once you visit that link:

__Response Parameters__

Here is an example URL:

Clicking on that link will take you to an authorization page.



In the past, we had been getting: "invalid_request" errors. Meaning: "The request was malformed or missing parameters".

Naturally, the query failed when trying here for us because we didn't know the redirect_uri.

There exists more info in original documentation about all this.

2. πŸ‘‰ Requesting Access Token

Now the docs say:

With all this being done...

Make a post request with the following format using the key given to you in the previous response.

As apposed to a get request, this will not be something you can do by entering a url into an address bar.


Perform POST request for an access token using this:

postparameters = { 'client_id': 'pfrvbpt8fpsf50v27o110ltjab', 'client_secret': '5hjhr7d87fpv3krkp44f32g24o', 'grant_type': 'authorization_code', 'redirect_uri': 'https://dataworksmd.org/', 'code': 'ee628ebb6ad82caa03d01640c26b6fb5' } url = 'https://secure.meetup.com/oauth2/access' x = requests.post(url, data = postparameters)

Lets try it


Lets look at the content

Did it look like whats below? A successfull submisson should look like this:


We got back JSON encoded in a string. Lets get that refresh_token because it's what we need amongst the other credentials to perform our query.


Otherwise, you'll get an error message. More information in the docs.

Once completed, API Queries can be made using the access_token and the KEY from before.

Skip down to the 'Querying Data' section if you made it down here.

Here are a bunch of queries we could make.

3. βœ–οΈ Making Queries Examples

API Query 1 - SELF
    Making Authenticated Requests ΒΆ
 After successfully obtaining member authorization and an access_token you may now make authenticated API requests using HTTPS by supplying the token value in an Authorization header prefixed with bearer and a space.
 curl -H "Authorization: Bearer {access_token}" https://api.meetup.com/members/self/

Docs, Template: https://api.meetup.com/self/groups

API Query 2 - GROUPS

Docs, Docs.P2, Template: https://api.meetup.com/:urlname

"created": 1530147506000, "members": 2307,

"next_event": { "id": "278394107", "yes_rsvp_count": 26,

"name": "Online: Introducing Datawave - Scalable Data Ingest and Query",

API Query 3a - GROUP Members

Docs, Docs.P2, Template: https://api.meetup.com/:urlname/members

Docs, Template: https://api.meetup.com:urlname/members/:member_id

Docs, Template: https://api.meetup.com/members/:member_id

API Query 3b - GROUP Events

Docs, Template: https://api.meetup.com/:urlname/events

API Query 4 - PRO

Docs.P2 -> # events attended join_time last_access_time

4. βœ–οΈ Saving our Members List

Docs, Template: https://api.meetup.com/pro/:urlname/members


# This is formatted as code

API Query 6 - Which Cities Are The Members From?

Save Those Queries!

We can use append > filename.json to the end of our Terminal commands to save the output.

!curl -H "Authorization: Bearer 603eb12d73e95a67e057dda7414cc0ca" https://api.meetup.com/DataWorks/members/?page=20000 > DataWorks_members.json # !curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/members/256827702?page=20 > members.json # !curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/DataWorks/events?page=80&has_ended=0 > DataWorks_events.json # !curl -H "Authorization: Bearer 2ccf6ce11e9f1cf4763a3d385481b21c" https://api.meetup.com/pro/:DataWorks/members > pro_DataWorks_members.json

5. βœ–οΈ Convert the Response's JSON Data to a Tabular excel-like Form.

Colabs has this special feature I will use that lets me store the output of a terminal command (deonted by the !) into a python variable.

In the following instance, I use the cat terminal command to read the contents of each file and store them into variables as a textual 'string'.

From there, much like before, we can open the files with python to view their contents.

# # Here I mix dataWorksResp = !cat DataWorks.json dataWorksJsonObject = json.loads(dataWorksResp[0]) print(dataWorksJsonObject) dataWorksMembersResp = !cat DataWorks_members.json dataWorksMembJsonObject = json.loads(dataWorksMembersResp[0]) print(dataWorksMembJsonObject)

It's possible to perform data science with JSON.

But for now lets convert it to tabularized form: a CSV

the dataWorksCSV only has one record since we only requested data on one 'group' -> DataWorks

Colabs pre-formats the output for us.

You'll see that the json_normalize function flattened the hierarchical nesting of the json object.

Here are the new column names for this flattened JSON Object.

6. βœ–οΈ Format the Columns (date-time) and do Preliminary Data Exploration. Then Save It.


Lets take a peek at some information?

from datetime import datetime as dt # The `[0]` retrieves the value of the members column of our dataset at the first (and only) index (row). print('Number of members: ', dataWorksCSV['members'][0]) # The datetime column is a bit tricky to work with created = dataWorksCSV['created'][0] print('created: ', dt.fromtimestamp( int(str(created)[:-3]) ).strftime('%Y-%m-%d %H:%M:%S') ) """

[link text](https:// [link text](https:// link text))Alright.. So we have 2311[link text](https:// link text) members and the group was created mid 2018?

What can we determine about our members?

A lot of these columns don't have much information. The status column has only 1 unique value 'active', for instance.

created - The time this member joined the Group, represented as milliseconds since the epoch

updated - The last time this member edited their Group profile, represented as milliseconds since the epoch

visited - The last time this member visited the Group, represented as milliseconds since the epoch

7. πŸ‘‰ Reuploading the Cleaned-Up Members List from step 6.

we have previously created a dataset 'meetup_api_member_data.csv' by running through step 3-6.

We wont have to do those parts again. Just re-upload the file now.

import io import pandas as pd uploaded = files.upload() for fn in uploaded.keys(): print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn]))) df = pd.read_csv(io.BytesIO(uploaded[fn])) dataWorksMembersCSV = df.copy()

8. πŸ‘‰ Use Member ID's from the Members-List to get every Members Full Info.

%cd resp

First we download data for every single record.

for index, row in df[684:10000].iterrows(): print( str(index)+'-'+row['name'].replace(" ","-") ) url = "https://api.meetup.com/DataWorks/members/"+str(row['id'])+"?fields=memberships&page=20" txt = str(index)+'-'+row['name'].replace(" ","-") + ".json" print( url, txt ) t = !curl -o {txt} -H "Authorization: Bearer 6992ff864f9e5f165ff42df94c2bd733" {url} time.sleep(5)

Then we flatten it

This is a test to see if it works

from pandas.io.json import json_normalize # # Here I mix txt = !cat 100-Gary-Mann.json txt = json.loads(txt[0]) memberships = False # print(txt['memberships']['member'][0]) try: x = pd.DataFrame.from_dict( json_normalize(txt['memberships']['member']) ) x['userId'] = txt['id'] x['userName'] = txt['name'] x['userCity'] = txt['city'] memberships = x.copy() memberships.rename(columns={"status visited": "visited"}) memberships = memberships[['userId', 'userName', 'userCity', 'group.localized_location', 'group.urlname', 'group.status', 'group.join_mode', 'group.members', 'group.who', "visited", 'created', 'updated', 'group.id']] display(memberships.head()) # df.reset_index(inplace=True) except: print('fail') memberships # df = memberships.copy()

This will do the flattening for all records

from pandas.io.json import json_normalize import os directory = os.fsencode('./') for file in os.listdir(directory): filename = os.fsdecode(file) # # Here I mix txt = !cat {filename} try: txt = json.loads(txt[0]) except: txt = '' mems = df.copy() try: x = pd.DataFrame.from_dict( json_normalize(txt['memberships']['member']) ) x['userId'] = txt['id'] x['userName'] = txt['name'] x['userCity'] = txt['city'] # print(x.columns) mems = x.copy() mems.rename(columns={"status visited": "visited"}) mems = mems[['userId', 'userName', 'userCity', 'group.localized_location', 'group.urlname', 'group.status', 'group.join_mode', 'group.members', 'group.who', "visited", 'created', 'updated', 'group.id']] memberships = pd.concat([memberships, mems], sort=False) print('good', filename) except: print('fail', filename) memberships.reset_index(drop=True, inplace=True)

9. πŸ‘‰ Look into Membership data

import io import pandas as pd uploaded = files.upload() for fn in uploaded.keys(): print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn]))) memberships = pd.read_csv(io.BytesIO(uploaded[fn])) memberships['created'] = memberships.apply(lambda x: dt.fromtimestamp( int(str(x['created'])[:-3]) ).strftime('%Y-%m-%d'), axis=1, result_type='expand') memberships['updated'] = memberships.apply(lambda x: dt.fromtimestamp( int(str(x['updated'])[:-3]) ).strftime('%Y-%m-%d'), axis=1, result_type='expand')

❌ Server Flow with User Credentials

This aint us

❌ Implicit Flow

Yeah, this'll work too. But the documentation is all of 100 words or something.

And a really awkwardly worded 100 words at that:

    The response parameters listed in the server flow's success (with the exception of   
    refresh_token) and failure access token responses will be included in the implicit 
    authorization's client response appended as a url fragment. 

Which really just means:

    Because this information is encoded in a url fragment, 
    it can only be retrieved with client-side browser scripts.

I'm sure that this will all make sense when we get to it..?

The rest of the documentation in this section just state that when a user visits your website and a request for data is sent via the useres browser, the user will be redirected to meetup and be asked to register or login and check a authorization check box or two.

Awesome. Lets get to work..?

Javascript Fetch Request

Or not.

I mean. We really tried everything. This one could still work, though!

You may need to allow cors to run it. Heres a chrome extension for that.

Btw. The code here is just a copy of the html file I used to test it (Spoiler: Invalid credentials).

 var url = "https://secure.meetup.com/oauth2/authorize?client_id= &response_type=token&redirect_uri=https://bniajfi.org/";
   .then(response => response.json())
   .then(data => console.log(data));
 <!-- https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch -->

❌ Python Package: Meetup-API

There are few to no resources other than the official documentation on the web to help inform us.

But this python library existed and I figured maybe it'd be the easy way out!

Here's the code I was using to test it. You can run it if you like...?

(Spoiler: This library is depricated as of 2 years ago).

(Spoiler P2: I tried manually patching the library to no avail )

client = meetup.api.Client('') type(client) group_info = client.GetGroup({'urlname': 'DataWorks'}) type(group_info) group_info.__dict__.keys() group_info.id group_info.name group_info.link