Core Python Programming

Creating Hangouts Chat bots with Python

2018-05-09T17:00:00.000-07:00

NOTE: The code featured here is also available as a video + overview post as part of this developers series from Google.

Introduction

Earlier today at Google I/O, the Hangouts Chat team (including yours truly) delivered a featured talk on the bot framework for Hangouts Chat (talk video here [~40 mins]). If you missed it several months ago, Google launched the new Hangouts Chat application for G Suite customers (but not consumer Gmail users at this time). This next-generation collaboration platform has several key features not available in "classic Hangouts," including the ability to create chat rooms, search, and the ability to attach files from Google Drive. However for developers, the biggest new feature is a bot framework and API allowing developers to create bots that interact with users inside "spaces," i.e., chat rooms or direct messages (DMs).

Before getting started with bots, let's review typical API usage where your app must be authorized for data access, i.e., OAuth2. Once permission is granted, your app calls the API, the API services the request and responds back with the desired results, as illustrated here:

Bot architecture is quite different. With bots, recognize that requests come from users in chat rooms or DMs. Users direct messages towards a bot, then Hangouts Chat relays those requests to the bot. The bot performs all the necessary processing (possibly calling other APIs and services), collates the response, and returns it to Chat, which in turn, displays the results to users. Here's the summarized bot workflow:

A key takeaway from this discussion is that normally in your apps, you call APIs. For bots, Hangouts Chat calls you. For this reason, the service is extremely flexible for developers: you can create bots using pretty much any language (not just Python), using your stack of choice, and hosted on any public or private cloud. The only requirement is that as long as Hangouts Chat can HTTP POST to your bot, you're good to go.

Furthermore, if you think there's something missing in the diagram because you don't see OAuth2 nor API calls, you'd also be correct. Look for both in a future post where I focus on asynchronous responses from Hangouts Chat bots.

Event types & bot processing

So what does the payload look like when your bot receives a call from Hangouts Chat? The first thing your bot needs to do is to determine the type of event received. It can then process the request based on that event type, and finally, return an appropriate JSON payload back to Hangouts Chat to render within the space. There are currently four event types in Hangouts Chat:

ADDED_TO_SPACE
REMOVED_FROM_SPACE
MESSAGE
CARD_CLICKED

The first pair are for when a bot is added to or removed from a space. In the first case, the bot would likely send a welcome message like, "Thanks for adding me to this room," and perhaps give some instructions on how to communicate with the bot. When a bot is removed from a space, it can no longer communicate with chat room participants, so there's no response message sent in this case; the most likely action here would be to log that the bot was removed from the space.

The 3rd message type is likely the most common scenario, where a bot has been added to a room, and now a human user is sending it a request. The other common type would be the last one. Rather than a message, this event occurs when users click on a UI card element. The bot's job here is to invoke the "callback" associated with the card element clicked. A number of things can happen here: the bot can return a text message (or nothing at all), a UI card can be updated, or a new card can be returned.

All of what we described above is realized in the pseudocode below (whether you choose to implement in Python or any other supported language) and helpful in preparing you to review the official documentation:

def process_event(req, rsp):
    event = json.loads(req['body']) # event received
    if event['type'] == 'REMOVED_FROM_SPACE':
        # no response as bot removed from room
        return
    elif event['type'] == 'ADDED_TO_SPACE':
        # bot added to room; send welcome message
        msg = {'text': 'Thanks for adding me to this room!'}
    elif event['type'] == 'MESSAGE':
        # message received during normal operations
        msg = respond_to_message(event['message']['text'])
    elif event['type'] == 'CARD_CLICKED':
        # result from user-click on card UI
        action = event['action']
        msg = respond_to_click(
          action['actionMethodName'], action['parameters'])
    else:
        return
    rsp.send(json.dumps(msg))

Regardless of implementation language, you still need to make the necessary tweaks to run on the hosting platform you choose. Our example uses Google App Engine (GAE) for hosting; below you'll see the obvious similarities between our actual bot app and this pseudocode.

Polling chat room users for votes

For a basic voting bot, we need a vote counter, buttons for users to upvote or downvote, and perhaps some reset or "new vote" button. To implement it, we need to unpack the inbound JSON object, determine the event type, then process each event type with actions like this:

ADDED_TO_SPACE — ignore.. we don't do anything unless asked by users
REMOVED_FROM_SPACE — no action (bot removed)
MESSAGE — start new vote
CARD_CLICKED — process "upvote", "downvote", or "newvote" request

Here's a handler implementation for Python App Engine using its webapp2 micro framework:

class VoteBot(webapp2.RequestHandler):
    def post(self):
        event = json.loads(self.request.body)
        user = event['user']['displayName']
        if event['type'] == 'CARD_CLICKED':
            method = event['action']['actionMethodName']
            if method == 'newvote':
                body = create_message(user)
            else:
                delta = 1 if method == 'upvote' else -1
                vote_count = int(event['action']['parameters'][0]['value']) + delta
                body = create_message(user, vote_count, True)
        elif event['type'] == 'MESSAGE':
            body = create_message(user)
        else: # no response for ADDED_TO_SPACE or REMOVED_FROM_SPACE
            return
        self.response.headers['Content-Type'] = 'application/json'
        self.response.out.write(json.dumps(body))

If you've never written an app using webapp2, don't fret, as this example is easily portable to Flask or other web framework. Why didn't I write it with Flask to begin with? Couple of reasons: 1) the webapp2 framework comes native with App Engine... no need to go download/install it, and 2) because it's native, I don't need to deploy any other files other than bot.py and its app.yaml config file whereas with Flask, you're uploading another ~1300 files in addition this pair. (More on this towards the end of this post.)

Compare this real app to the pseudocode... similar, right? Our bot is a bit more complex in that it renders UI cards, so the create_message() function will need to do more than just return a plain text response. Instead, it must generate and return the JSON markup rendering the card in the Hangouts Chat UI:

def create_message(voter, vote_count=0, should_update=False):
    PARAMETERS = [{'key': 'count', 'value': str(vote_count)}]
    return {
        'actionResponse': {
            'type': 'UPDATE_MESSAGE' if should_update else 'NEW_MESSAGE'
        },
        'cards': [{
            'header': {'title': 'Last vote by %s!' % voter},
            'sections': [{
                'widgets': [{
                    'textParagraph': {'text': '%d votes!' % vote_count}
                }, {
                    'buttons': [{
                        'textButton': {
                            'text': '+1',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'upvote',
                                    'parameters': PARAMETERS,
                                }
                            }
                        }
                    }, {
                        'textButton': {
                            'text': '-1',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'downvote',
                                    'parameters': PARAMETERS,
                                }
                            }
                        }
                    }, {
                        'textButton': {
                            'text': 'NEW',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'newvote',
                                }
                            }
                        }
                    }]
                }]
            }]
        }]
    }

Finishing touches

The last thing App Engine needs is an app.yaml configuration file:

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: bot.app

If you've deployed apps to GAE before, you know you need to create a project in the Google Cloud Developers Console. You also need to have a project for bots, however you can use the same project for both your use of the Hangouts Chat bot framework (link only works for G Suite customers) as well as hosting your App Engine-based bot.

Once you've deployed your bot to GAE, go to the Hangouts Chat API configuration tab, and add the HTTP endpoint for your App Engine-based bot to the Conenctions Settings section:

As you can see, there are several options here for where Hangouts Chat can post messages destined for bots: standard HTTPS bots like our App Engine example, Google Apps Script (a customized JavaScript-in-the-cloud platform with built-in G Suite integration [intro video]), or Google Cloud Pub/Sub. Pub/Sub is the message queue proxy for when your bot is hosted on-premise behind a firewall, requiring you to register a pull subscription to retrieve bot messages from Hangouts Chat.

Once you've published your bot, add the bot to a room or @mention it in a DM. Send it a message, and it returns an interactive vote card like what you see below. Cast some votes or create a new vote to test drive your new bot.

Conclusion

Congrats for creating your first Python bot! Below is the entire bot.py file for your convenience. (At this time, Python App Engine standard only supports Python 2, so if you want to run Python 3 instead, use the Python App Engine flexible environment.) Also don't forget the app.yaml configuration file above.

import json
import webapp2

def create_message(voter, vote_count=0, should_update=False):
    PARAMETERS = [{'key': 'count', 'value': str(vote_count)}]
    return {
        'actionResponse': {
            'type': 'UPDATE_MESSAGE' if should_update else 'NEW_MESSAGE'
        },
        'cards': [{
            'header': {'title': 'Last vote by %s!' % voter},
            'sections': [{
                'widgets': [{
                    'textParagraph': {'text': '%d votes!' % vote_count}
                }, {
                    'buttons': [{
                        'textButton': {
                            'text': '+1',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'upvote',
                                    'parameters': PARAMETERS,
                                }
                            }
                        }
                    }, {
                        'textButton': {
                            'text': '-1',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'downvote',
                                    'parameters': PARAMETERS,
                                }
                            }
                        }
                    }, {
                        'textButton': {
                            'text': 'NEW',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'newvote',
                                }
                            }
                        }
                    }]
                }]
            }]
        }]
    }

class VoteBot(webapp2.RequestHandler):
    def post(self):
        event = json.loads(self.request.body)
        user = event['user']['displayName']
        if event['type'] == 'CARD_CLICKED':
            method = event['action']['actionMethodName']
            if method == 'newvote':
                body = create_message(user)
            else:
                delta = 1 if method == 'upvote' else -1
                vote_count = int(event['action']['parameters'][0]['value']) + delta
                body = create_message(user, vote_count, True)
        elif event['type'] == 'MESSAGE':
            body = create_message(user)
        else: # no response for ADDED_TO_SPACE or REMOVED_FROM_SPACE
            return

        self.response.headers['Content-Type'] = 'application/json'
        self.response.out.write(json.dumps(body))

app = webapp2.WSGIApplication([
    ('/', VoteBot),
], debug=True)

Yep, the entire bot is made up of just these 2 files. While porting to Flask is fairly straightforward and Flask apps are supported by App Engine, Flask itself doesn't come with App Engine (quickstart here), so you'll need to upload all of Flask (in addition to those 2 files) to host a Flask version of your bot on GAE.

Both files are also available from this app's GitHub repo which you can fork. I'm happy to entertain PRs for any bugs you find or ways we can simplify this code even more. Below are some additional resources for learning about Hangouts Chat bots and GAE:

Hangouts Chat bots concepts page
Creating new Hangouts Chat bots guide
Hangouts Chat guide to interactive cards
Other Hangouts Chat sample bots
Official Hangouts Chat developer documentation—everything you need to know and more!
Python App Engine (standard) documentation
Google App Engine product information

Python's a great language to implement bots with because not only can you create something quickly, but choose any platform to host it on, Google App Engine, Amazon EC2/Google Compute Engine, or any cloud that runs Python apps.

Epilogue: code challenge

While usable, this vote bot can be significantly improved. Once you get the basic bot working, I recommend the following enhancements as exercises for the reader:

Support vote topics: users starting new votes must state topic in bot message; use as card header
Add images (like the Node.js version of our vote bot in the docs hosted on Google Cloud Functions)
Track users who have voted (you decide on implementation)
Don't let the vote count to go below zero
Allow downvotes only from users who have at least one upvote
Port your working bot from webapp2 to Flask

Modifying events with the Google Calendar API

2017-06-26T11:36:00.000-07:00

NOTE: The code featured here is also available as a video + overview post as part of this developers series from Google.

Introduction

In an earlier post, I introduced Python developers to adding events to users' calendars using the Google Calendar API. However, while being able to insert events is "interesting," it's only half the picture. If you want to give your users a more complete experience, modifying those events is a must-have. In this post, you'll learn how to modify existing events, and as a bonus, learn how to implement repeating events too.

In order to modify events, we need the full Calendar API scope:

'https://www.googleapis.com/auth/calendar'—read-write access to Calendar

Skipping the OAuth2 boilerplate, once you have valid authorization credentials, create a service endpoint to the Calendar API like this:

GCAL = discovery.build('calendar', 'v3',
    http=creds.authorize(Http()))

Now you can send the API requests using this endpoint.

Using the Google Calendar API

Our sample script requires an existing Google Calendar event, so either create one programmatically with events().insert() & save its ID as we showed you in that earlier post, or use events().list() or events().get() to get the ID of an existing event.

While you can use an offset from GMT/UTC, such as the GMT_OFF variable from the event insert post, today's code sample "upgrades" to a more general IANA timezone solution. For Pacific Time, it's "America/Los_Angeles". The reason for this change is to allow events that survive across Daylight Savings Time shifts. IOW, a dinner at 7PM/1900 stays at 7PM as we cross fall and spring boundaries. This is especially important for events that repeat throughout the year. Yes, we are discussing recurrence in this post too, so it's particularly relevant.

Modifying calendar events

In the other post, the EVENT body constitutes an "event record" containing the information necessary to create a calendar entry—it consists of the event name, start & end times, and invitees. That record is an API resource which you created/accessed with the Calendar API via events().insert(). (What do you think the "R" in "URL" stands for anyway?!?) The Calendar API adheres to RESTful semantics in that the HTTP verbs match the actions you perform against a resource.

In today's scenario, let's assume that dinner from the other post didn't work out, but that you want to reschedule it. Furthermore, not only do you want to make that dinner happen again, but because you're good friends, you've made a commitment to do dinner every other month for the rest of the year, then see where things stand. Now that we know what we want, we have a choice.

There are two ways to modifying existing events in Calendar:

events().patch() (HTTP PATCH)—"patch" 1 or more fields in resource
events().update() (HTTP PUT)—replace/rewrite entire resource

Do you just update that resource with events().patch() or do you replace the entire resource with events().update()? To answer that question, ask yourself, "How many fields am I updating?" In our case, we only want to change the date and make this event repeat, so PATCH is a better solution. If instead, you also wanted to rename the event or switch dinner to another set of friends, you'd then be changing all the fields, so PUT would be a better solution in that case.

Using PATCH means you're just providing the deltas between the original & updated event, so the EVENT body this time reflects just those changes:

TIMEZONE = 'America/Los_Angeles'
EVENT = {
    'start':  {'dateTime': '2017-07-01T19:00:00', 'timeZone': TIMEZONE},
    'end':    {'dateTime': '2017-07-01T22:00:00', 'timeZone': TIMEZONE},
    'recurrence': ['RRULE:FREQ=MONTHLY;INTERVAL=2;UNTIL=20171231']
}

Repeating events

Something you haven't seen before is how to do repeating events. To do this, you need to define what’s known as a recurrence rule ("RRULE"), which answers the question of how often an event repeats. It looks somewhat cryptic but follows the RFC 5545 Internet standard which you can basically decode like this:

FREQ=MONTHLY—event to occur on a monthly basis...
INTERVAL=2—... but every two months (every other month)
UNTIL=20171231—... until this date

There are many ways events can repeat, so I suggest you look at all the examples at the RFC link above.

Finishing touches

Finally, provide the EVENT_ID and call events().patch():

EVENT_ID = YOUR_EVENT_ID_STR_HERE # use your own!
e = GCAL.events().patch(calendarId='primary', eventId=EVENT_ID,
        sendNotifications=True, body=EVENT).execute()

Keep in mind that in real life, your users may be accessing your app from their desktop or mobile devices, so you need to ensure you don't override an earlier change. In this regard, developers should use the If-Match header along with an ETag value to validate unique requests. For more information, check out the conditional modification page in the official docs.

The one remaining thing is to confirm on-screen that the calendar event was updated successfully. We do that by checking the return value—it should be an Event object with all the existing details as well as the modified fields:

print('''\
*** %r event (ID: %s) modified:
    Start: %s
    End:   %s
    Recurring (rule): %s
''' % (e['summary'].encode('utf-8'), e['id'], e['start']['dateTime'],
        e['end']['dateTime'], e['recurrence'][0]))

That's pretty much the entire script save for the OAuth2 boilerplate code we've explored previously. The script is posted below in its entirety, and if you add a valid event ID and run it, depending on the date/times you use, you'll see something like this:

$ python gcal_modify.py
*** 'Dinner with friends' event (ID: YOUR_EVENT_ID_STR_HERE) modified:
    Start: 2017-07-01T19:00:00-07:00
    End:   2017-07-01T22:00:00-07:00
    Recurring (rule): RRULE:FREQ=MONTHLY;UNTIL=20171231;INTERVAL=2

It also works with Python 3 with one slight nit/difference being the "b" prefix on from the event name due to converting from Unicode to bytes:
*** b'Dinner with friends' event...

Conclusion

Now you know how to modify events as well as make them repeat. To complete the example, below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

from __future__ import print_function
from apiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/calendar'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
CAL = build('calendar', 'v3', http=creds.authorize(Http()))

TIMEZONE = 'America/Los_Angeles'
EVENT = {
    'start':  {'dateTime': '2017-07-01T19:00:00', 'timeZone': TIMEZONE},
    'end':    {'dateTime': '2017-07-01T22:00:00', 'timeZone': TIMEZONE},
    'recurrence': ['RRULE:FREQ=MONTHLY;INTERVAL=2;UNTIL=20171231']
}
EVENT_ID = YOUR_EVENT_ID_STR_HERE
e = GCAL.events().patch(calendarId='primary', eventId=EVENT_ID,
        sendNotifications=True, body=EVENT).execute()

print('''\
*** %r event (ID: %s) modified:
    Start: %s
    End:   %s
    Recurring (rule): %s
''' % (e['summary'].encode('utf-8'), e['id'], e['start']['dateTime'],
        e['end']['dateTime'], e['recurrence'][0]))

You can now customize this code for your own needs, for a mobile frontend, a server-side backend, or to access other Google APIs. If you want to learn more about using the Google Calendar API, check out the following resources:

Creating great experiences with email markup—video
Creating events with the Calendar API—video & blog post
Creating events with the Calendar API in Python—blog post
Official Calendar API Python quickstart—list the next 10 events in your calendar
Official Calendar API documentation—everything you need to know and more!
Email markup documentation for Google Calendar

Managing Shared (formerly Team) Drives with Python and the Google Drive API

2017-06-01T10:00:00.005-07:00

2023 UPDATE: We are working to put updated versions of all the code into GitHub... stay tuned. The link will provided in all posts once the code sample(s) is(are) available.

2019 UPDATE: "G Suite" is now called "Google Workspace", "Team Drives" is now known as "Shared Drives", and the corresponding supportsTeamDrives flag has been renamed to supportsAllDrives. Please take note of these changes regarding the post below.

NOTE 1: Team Drives is only available for G Suite Business Standard users or higher. If you're developing an application for Team Drives, you'll need similar access.
NOTE 2: The code featured here is also available as a video + overview post as part of this series.

Introduction

Team Drives is a relatively new feature from the Google Drive team, created to solve some of the issues of a user-centric system in larger organizations. Team Drives are owned by an organization rather than a user and with its use, locations of files and folders won't be a mystery any more. While your users do have to be a G Suite Business (or higher) customer to use Team Drives, the good news for developers is that you won't have to write new apps from scratch or learn a completely different API.

Instead, Team Drives features are accessible through the same Google Drive API you've come to know so well with Python. In this post, we'll demonstrate a sample Python app that performs core features that all developers should be familiar with. By the time you've finished reading this post and the sample app, you should know how to:

Create Team Drives
Add members to Team Drives
Create a folder in Team Drives
Import/upload files to Team Drives folders

Using the Google Drive API

The demo script requires creating files and folders, so you do need full read-write access to Google Drive. The scope you need for that is:

'https://www.googleapis.com/auth/drive' — Full (read-write) access to Google Drive

If you're new to using Google APIs, we recommend reviewing earlier posts & videos covering the setting up projects and the authorization boilerplate so that we can focus on the main app. Once we've authorized our app, assume you have a service endpoint to the API and have assigned it to the DRIVE variable.

Create Team Drives

New Team Drives can be created with DRIVE.teamdrives().create(). Two things are required to create a Team Drive: 1) you should name your Team Drive. To make the create process idempotent, you need to create a unique request ID so that any number of identical calls will still only result in a single Team Drive being created. It's recommended that developers use a language-specific UUID library. For Python developers, that's the uuid module. From the API response, we return the new Team Drive's ID. Check it out:

def create_td(td_name):
    request_id = str(uuid.uuid4())
    body = {'name': td_name}
    return DRIVE.teamdrives().create(body=body,
            requestId=request_id, fields='id').execute().get('id')

Add members to Team Drives

To add members/users to Team Drives, you only need to create a new permission, which can be done with DRIVE.permissions().create(), similar to how you would share a file in regular Drive with another user. The pieces of information you need for this request are the ID of the Team Drive, the new member's email address as well as the desired role... choose from: "organizer", "owner", "writer", "commenter", "reader". Here's the code:

def add_user(td_id, user, role='commenter'):
    body = {'type': 'user', 'role': role, 'emailAddress': user}
    return DRIVE.permissions().create(body=body, fileId=td_id,
            supportsTeamDrives=True, fields='id').execute().get('id')

Some additional notes on permissions: the user can only be bestowed permissions equal to or less than the person/admin running the script... IOW, they cannot grant someone else greater permission than what they have. Also, if a user has a certain role in a Team Drive, they can be granted greater access to individual elements in the Team Drive. Users who are not members of a Team Drive can still be granted access to Team Drive contents on a per-file basis.

Create a folder in Team Drives

Nothing to see here! Yep, creating a folder in Team Drives is identical to creating a folder in regular Drive, with DRIVE.files().create(). The only difference is that you pass in a Team Drive ID rather than regular Drive folder ID. Of course, you also need a folder name too. Here's the code:

def create_td_folder(td_id, folder):
    body = {'name': folder, 'mimeType': FOLDER_MIME, 'parents': [td_id]}
    return DRIVE.files().create(body=body,
            supportsTeamDrives=True, fields='id').execute().get('id')

Import/upload files to Team Drives folders

Uploading files to a Team Drives folder is also identical to to uploading to a normal Drive folder, and also done with DRIVE.files().create(). Importing is slightly different than uploading because you're uploading a file and converting it to a G Suite/Google Apps document format, i.e., uploading CSV as a Google Sheet, or plain text or Microsoft Word® file as Google Docs. In the sample app, we tackle the former:

def import_csv_to_td_folder(folder_id, fn, mimeType):
    body = {'name': fn, 'mimeType': mimeType, 'parents': [folder_id]}
    return DRIVE.files().create(body=body, media_body=fn+'.csv',
            supportsTeamDrives=True, fields='id').execute().get('id')

The secret to importing is the MIMEtype. That tells Drive whether you want conversion to a G Suite/Google Apps format (or not). The same is true for exporting. The import and export MIMEtypes supported by the Google Drive API can be found in my SO answer here.

Driver app

All these functions are great but kind-of useless without being called by a main application, so here we are:

FOLDER_MIME = 'application/vnd.google-apps.folder'
SOURCE_FILE = 'inventory' # on disk as 'inventory.csv'
SHEETS_MIME = 'application/vnd.google-apps.spreadsheet'

td_id = create_td('Corporate shared TD')
print('** Team Drive created')
perm_id = add_user(td_id, 'email@example.com')
print('** User added to Team Drive')
folder_id = create_td_folder(td_id, 'Manufacturing data')
print('** Folder created in Team Drive')
file_id = import_csv_to_td_folder(folder_id, SOURCE_FILE, SHEETS_MIME)
print('** CSV file imported as Google Sheets in Team Drives folder')

The first set of variables represent some MIMEtypes we need to use as well as the CSV file we're uploading to Drive and requesting it be converted to Google Sheets format. Below those definitions are calls to all four functions described above.

Conclusion

If you run the script, you should get output that looks something like this, with each print() representing each API call:

$ python3 td_demo.py
** Team Drive created
** User added to Team Drive
** Folder created in Team Drive
** CSV file imported as Google Sheets in Team Drives folder

When the script has completed, you should have a new Team Drives folder called "Corporate shared TD", and within, a folder named "Manufacturing data" which contains a Google Sheets file called "inventory".

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!)—by using, copying, and/or modifying this code or any other piece of source from this blog, you implicitly agree to its Apache2 license:

from __future__ import print_function
import uuid

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

def create_td(td_name):
    request_id = str(uuid.uuid4()) # random unique UUID string
    body = {'name': td_name}
    return DRIVE.teamdrives().create(body=body,
            requestId=request_id, fields='id').execute().get('id')

def add_user(td_id, user, role='commenter'):
    body = {'type': 'user', 'role': role, 'emailAddress': user}
    return DRIVE.permissions().create(body=body, fileId=td_id,
            supportsTeamDrives=True, fields='id').execute().get('id')

def create_td_folder(td_id, folder):
    body = {'name': folder, 'mimeType': FOLDER_MIME, 'parents': [td_id]}
    return DRIVE.files().create(body=body,
            supportsTeamDrives=True, fields='id').execute().get('id')

def import_csv_to_td_folder(folder_id, fn, mimeType):
    body = {'name': fn, 'mimeType': mimeType, 'parents': [folder_id]}
    return DRIVE.files().create(body=body, media_body=fn+'.csv',
            supportsTeamDrives=True, fields='id').execute().get('id')

FOLDER_MIME = 'application/vnd.google-apps.folder'
SOURCE_FILE = 'inventory' # on disk as 'inventory.csv'... CHANGE!
SHEETS_MIME = 'application/vnd.google-apps.spreadsheet'

td_id = create_td('Corporate shared TD')
print('** Team Drive created')
perm_id = add_user(td_id, 'email@example.com') # CHANGE!
print('** User added to Team Drive')
folder_id = create_td_folder(td_id, 'Manufacturing data')
print('** Folder created in Team Drive')
file_id = import_csv_to_td_folder(folder_id, SOURCE_FILE, SHEETS_MIME)
print('** CSV file imported as Google Sheets in Team Drives folder')

As with our other code samples, you can now customize it to learn more about the API, integrate into other apps for your own needs, for a mobile frontend, sysadmin script, or a server-side backend!

Code challenge

Write a simple application that moves folders (and its files or folders) in regular Drive to Team Drives. Each folder you move should be a corresponding folder in Team Drives. Remember that files in Team Drives can only have one parent, and the same goes for folders.

Adding text & shapes with the Google Slides API

2017-02-22T14:39:00.000-08:00

NOTE: The code featured here is also available as a video + overview post as part of this series.

Introduction

This is the fourth entry highlighting primary use cases of the Google Slides API with Python; check back in the archives to access the first three. Today, we're focused on some of the basics, like adding text to slides. We'll also cover adding shapes, and as a bonus, adding text into shapes!

Using the Google Slides API

The demo script requires creating a new slide deck (and adding a new slide) so you need the read-write scope for Slides:

'https://www.googleapis.com/auth/presentations' — Read-write access to Slides and Slides presentation properties

Create new presentation & get its objects' IDs

A new slide deck can be created with SLIDES.presentations().create()—or alternatively with the Google Drive API which we won't do here. From the API response, we save the new deck's ID along with the IDs of the title and subtitle textboxes on the default title slide:

rsp = SLIDES.presentations().create(
        body={'title': 'Adding text formatting DEMO'}).execute()
deckID = rsp['presentationId']
titleSlide = rsp['slides'][0]     # title slide object IDs
titleID    = titleSlide['pageElements'][0]['objectId']
subtitleID = titleSlide['pageElements'][1]['objectId']

The title slide only has two elements on it, the title and subtitle textboxes, returned in that order, hence why we grab them at indexes 0 and 1 respectively.

Generating our own unique object IDs

In the next steps, we generate our own unique object IDs. We'll first explain what those objects are followed by why you'd want to create your own object IDs rather than letting the API create default IDs for the same objects.

As we've done in previous posts on the Slides API, we create one new slide with the "main point" layout. It has one notable object, a "large-ish" textbox and nothing else. We'll create IDs for the slide itself and another for its textbox. Next, we'll (use the API to) "draw" 3 shapes on this slide, so we'll create IDs for each of those. That's 5 (document-unique) IDs total. Now let's discuss why you'd "roll your own" IDs.

Why and how to generate our own IDs

It's advantageous for all developers to minimize the overall number of calls to Google APIs. While most of services provided through the APIs are free, they'll have some quota to prevent abuse (Slides API quotas page FYI). So how does creating our own IDs help reduce API calls?

Passing in object IDs is optional for "create" calls. Providing your own ID lets you create an object and modify it using additional requests within the same API call to SLIDES.presentations().batchUpdate(). If you don't provide your own object IDs, the API will generate a unique one for you.

Unfortunately, this means that instead of one API call, you'll need one to create the object, likely another to get that object to determine its ID, and yet another to update that object using the ID you just fetched. Separate API calls to create, get, and update means (at least) 3x more than if you provided your own IDs (where you can do create & update with a single API call; no get necessary).

Here are a few things to know when rolling your own IDs:

IDs must start with an alphanumeric character or underscore (matches regex [a-zA-Z0-9_])
Any remaining characters can also include a hyphen or colon (matches regex [a-zA-Z0-9_-:])
The length of the ID must conform to: 5 ≤ len(ID) ≤ 50.
Object IDs must be unique across all objects in a presentation.

You'll somehow need to ensure your IDs are unique or use UUIDs (universally unique identifiers) for which most languages have libraries for. Examples: Java developers can use java.util.UUID.randomUUID().toString() while Python users can import the uuid module plus any extra work to get UUID string values:

import uuid
gen_uuid = lambda : str(uuid.uuid4())  # get random UUID string

Finally, be aware that if an object is modified in the UI, its ID may change. For more information, review the "Working with object IDs" section in the Slides API Overview page.

Back to sample app

All that said, let's go back to the code and generate those 5 random object IDs we promised earlier:

mpSlideID   = gen_uuid() # mainpoint IDs
mpTextboxID = gen_uuid()
smileID     = gen_uuid() # shape IDs
str24ID     = gen_uuid()
arwbxID     = gen_uuid()

With that, we're ready to create the requests array (reqs) to send to the API.

Create "main point" slide

The first request creates the "main point" slide...

reqs = [
    {'createSlide':
        'objectId': mpSlideID,
        'slideLayoutReference': {'predefinedLayout': 'MAIN_POINT'},
        'placeholderIdMappings': [{
            'objectId': mpTextboxID,
            'layoutPlaceholder': {'type': 'TITLE', 'index': 0}
        }],
    }},

...where...

objectID—our generated ID we're assigning to the newly-created slide
slideLayoutReference—new slide layout type ("main point")
placeholderIdMappings—array of IDs ([inner] objectId) for each of the page elements and which object (layoutPlaceholder) they should map or be assigned to

The page elements created on the new slide (depends [obviously] on the layout chosen); "main point" only has the one textbox, hence why placeholderIdMappings only has one element.

Add title slide and main point textbox text

The next requests fill in the title & subtitle in the default title slide and also the textbox on the main point slide.

{'insertText': {'objectId': titleID, 'text': 'Adding text and shapes'}},
{'insertText': {'objectId': subtitleID, 'text': 'via the Google Slides API'}},
{'insertText': {'objectId': mpTextboxID, 'text': 'text & shapes'}},

The first pair use IDs that were generated by the Slides API when the presentation was created while the main point textbox ID was generated by us.

Create three shapes

Above, we created IDs for three shapes, a "smiley face," a 24-point star, and a double arrow box (smileID, str24ID, arwbxID). The request for the first looks like this:

{'createShape': {
    'objectId': smileID,
    'shapeType': 'SMILEY_FACE',
    'elementProperties': {
        "pageObjectId": mpSlideID,
        'size': {
            'height': {'magnitude': 3000000, 'unit': 'EMU'},
            'width':  {'magnitude': 3000000, 'unit': 'EMU'}
        },
        'transform': {
            'unit': 'EMU', 'scaleX': 1.3449, 'scaleY': 1.3031,
            'translateX': 4671925, 'translateY': 450150,
        },
    },
}}

The JSON for the other two shapes are similar, with differences being: the object ID, the shapeType, and the transform. You can see the corresponding requests for the other shapes in the full source code at the bottom of this post, so we won't display them here as the descriptions will be nearly identical.

Size & transform for slide objects

When placing or manipulating objects on slides, key element properties you must provide are the sizes and transforms. These are components you must either use some math to create or derive from pre-existing objects. Resizing, rotating, and similar operations require some basic knowledge of matrix math. Take a look at the Page Elements page in the official docs as well as the Transforms concept guide for more details.

Deriving from pre-existing objects: if you're short on time, don't want to deal with the math, or perhaps thinking something like, "Geez, I just want to draw a smiley face on a slide." One common pattern then, is to bring up the Slides UI, create a blank slide & place your image or draw your shape the way you want, with the size you want, & putting it exactly where you want. For example:

Once you have that desired shape (and size and location), you can use the API (either presentations.get or presentations.pages.get) to read that object's size and transform then drop both of those into your application so the API creates a new shape in the exact way, mirroring what you created in the UI. For the smiley face above, the JSON payload we got back from one of the "get" calls could look something like:

If you scroll back up to the createShape request, you'll see we used those exact values. Note: because the 3 shapes are all in different locations and sizes, expect the corresponding values for each shape to be different.

Bonus: adding text to shapes

Now that you know how to add text and shapes, it's only fitting that we show you how to add text into shapes. The good news is that the technique is no different than adding text to textboxes or even tables. So with the shape IDs, our final set of requests along with the batchUpdate() call looks like this:

    {'insertText': {'objectId': smileID, 'text': 'Put the nose somewhere here!'}},
    {'insertText': {'objectId': str24ID, 'text': 'Count 24 points on this star!'}},
    {'insertText': {'objectId': arwbxID, 'text': "An uber bizarre arrow box!"}},
] # end of 'reqs'
SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute()

Conclusion

If you run the script, you should get output that looks something like this, with each print() representing each API call:

$ python3 slides_shapes_text.py 
** Create new slide deck & set up object IDs
** Create "main point" slide, add text & interesting shapes
DONE

When the script has completed, you should have a new presentation with a title slide and a main point slide with shapes which should look something like this:

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!)—by using, copying, and/or modifying this code or any other piece of source from this blog, you implicitly agree to its Apache2 license:

from __future__ import print_function
import uuid

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

gen_uuid = lambda : str(uuid.uuid4())  # get random UUID string

SCOPES = 'https://www.googleapis.com/auth/presentations',
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
SLIDES = discovery.build('slides', 'v1', http=creds.authorize(Http()))

print('** Create new slide deck & set up object IDs')
rsp = SLIDES.presentations().create(
        body={'title': 'Adding text & shapes DEMO'}).execute()
deckID = rsp['presentationId']
titleSlide  = rsp['slides'][0]      # title slide object IDs
titleID     = titleSlide['pageElements'][0]['objectId']
subtitleID  = titleSlide['pageElements'][1]['objectId']
mpSlideID   = gen_uuid()            # mainpoint IDs
mpTextboxID = gen_uuid()

smileID     = gen_uuid()            # shape IDs

str24ID     = gen_uuid()
arwbxID     = gen_uuid()

print('** Create "main point" slide, add text & interesting shapes')
reqs = [
    # create new "main point" layout slide, giving slide & textbox IDs
    {'createSlide': {
        'objectId': mpSlideID,
        'slideLayoutReference': {'predefinedLayout': 'MAIN_POINT'},
        'placeholderIdMappings': [{
            'objectId': mpTextboxID,
            'layoutPlaceholder': {'type': 'TITLE', 'index': 0}
        }],
    }},
    # add title & subtitle to title slide; add text to main point slide textbox
    {'insertText': {'objectId': titleID,     'text': 'Adding text and shapes'}},
    {'insertText': {'objectId': subtitleID,  'text': 'via the Google Slides API'}},
    {'insertText': {'objectId': mpTextboxID, 'text': 'text & shapes'}},
    # create smiley face
    {'createShape': {
        'objectId': smileID,
        'shapeType': 'SMILEY_FACE',
        'elementProperties': {
            "pageObjectId": mpSlideID,
            'size': {
                'height': {'magnitude': 3000000, 'unit': 'EMU'},
                'width':  {'magnitude': 3000000, 'unit': 'EMU'}
            },
            'transform': {
                'unit': 'EMU', 'scaleX': 1.3449, 'scaleY': 1.3031,
                'translateX': 4671925, 'translateY': 450150,
            },
        },
    }},
    # create 24-point star
    {'createShape': {
        'objectId': str24ID,
        'shapeType': 'STAR_24',
        'elementProperties': {
            "pageObjectId": mpSlideID,
            'size': {
                'height': {'magnitude': 3000000, 'unit': 'EMU'},
                'width':  {'magnitude': 3000000, 'unit': 'EMU'}
            },
            'transform': {
                'unit': 'EMU', 'scaleX': 0.7079, 'scaleY': 0.6204,
                'translateX': 2036175, 'translateY': 237350,
            },
        },
    }},
    # create double left & right arrow w/textbox
    {'createShape': {
        'objectId': arwbxID,
        'shapeType': 'LEFT_RIGHT_ARROW_CALLOUT',
        'elementProperties': {
            "pageObjectId": mpSlideID,
            'size': {
                'height': {'magnitude': 3000000, 'unit': 'EMU'},
                'width':  {'magnitude': 3000000, 'unit': 'EMU'}
            },
            'transform': {
                'unit': 'EMU', 'scaleX': 1.1451, 'scaleY': 0.4539,
                'translateX': 1036825, 'translateY': 3235375,
            },
        },
    }},
    # add text to all 3 shapes
    {'insertText': {'objectId': smileID, 'text': 'Put the nose somewhere here!'}},
    {'insertText': {'objectId': str24ID, 'text': 'Count 24 points on this star!'}},
    {'insertText': {'objectId': arwbxID, 'text': "An uber bizarre arrow box!"}},
]
SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute()
print('DONE')

As with our other code samples, you can now customize it to learn more about the API, integrate into other apps for your own needs, for a mobile frontend, sysadmin script, or a server-side backend!

Code challenge

Create a 2x3 or 3x4 table on a slide and add text to each "cell." This should be a fairly easy exercise, especially if you look at the Table Operations documentation. HINT: you'll be using insertText with just an extra field, cellLocation. EXTRA CREDIT: generalize your solution so that you're grabbing cells from a Google Sheet and "import" them into a table on a slide. HINT: look for the earlier post where we describe how to create slides from spreadsheet data.

Formatting text with the Google Slides API

2016-12-13T22:54:00.000-08:00

NOTE: The code covered in this post are also available in a video walkthrough.

Introduction

If you know something about public speaking, you're aware that the most effective presentations are those which have more images and less text. As a developer of applications that auto-generate slide decks, this is even more critical as you must ensure that your code creates the most compelling presentations possible for your users.

This means that any text featured in those slide decks must be more impactful. To that end, it's important you know how to format any text you do have. That's the exact subject of today's post, showing you how to format text in a variety of ways using Python and the Google Slides API.

The API is fairly new, so if you're unfamiliar with it, check out the launch post and take a peek at the API overview page to acclimate yourself to it first. You can also read related posts (and videos) explaining how to replace text & images with the API or how to generate slides from spreadsheet data. If you're ready-to-go, let's move on!

Using the Google Slides API

The demo script requires creating a new slide deck so you need the read-write scope for Slides:

'https://www.googleapis.com/auth/presentations' — Read-write access to Slides and Slides presentation properties

Create deck & set up new slide for text formatting

A new slide deck can be created with SLIDES.presentations().create()—or alternatively with the Google Drive API which we won't do here. We'll name it, "Slides text formatting DEMO" and save its ID along with the IDs of the title and subtitle textboxes on the auto-created title slide:

DATA = {'title': 'Slides text formatting DEMO'}
rsp = SLIDES.presentations().create(body=DATA).execute()
deckID = rsp['presentationId']
titleSlide = rsp['slides'][0]
titleID = titleSlide['pageElements'][0]['objectId']
subtitleID = titleSlide['pageElements'][1]['objectId']

The title slide only has two elements on it, the title and subtitle textboxes, returned in that order, hence why we grab them at indexes 0 and 1 respectively. Now that we have a deck, let's add a slide that has a single (largish) textbox. The slide layout with that characteristic that works best for our demo is the "main point" template:

While we're at it, let's also add the title & subtitle on the title slide. Here's the snippet that builds and executes all three requests:

print('** Create "main point" layout slide & add titles')
reqs = [
  {'createSlide':
     {'slideLayoutReference': {'predefinedLayout': 'MAIN_POINT'}}},
  {'insertText':
     {'objectId': titleID, 'text': 'Formatting text'}},
  {'insertText':
     {'objectId': subtitleID, 'text': 'via the Google Slides API'}},
]
rsp = SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute().get('replies')
slideID = rsp[0]['createSlide']['objectId']

The requests are sent in the order you see above, and responses come back in the same order. We don't care much about the 'insertText' directives, but we do want to get the ID of the newly-created slide. In the array of 3 returned responses, that slideID comes first.

Why do we need the slide ID? Well, since we're going to be using the one textbox on that slide, the only way to get the ID of that textbox is by doing a presentations().pages().get() call to fetch all the objects on that slide. Since there's only one "page element," the textbox in question, we make that call and save the first (and only) object's ID:

print('** Fetch "main point" slide title (textbox) ID')
rsp = SLIDES.presentations().pages().get(presentationId=deckID,
        pageObjectId=slideID).execute().get('pageElements')
textboxID = rsp[0]['objectId']

Armed with the textbox ID, we're ready to add our text and format it!

Formatting text

The last part of the script starts by inserting seven (short) paragraphs of text—then format different parts of that text (in a variety of ways). Take a look here, then we'll discuss below:

reqs = [
    # add 6 paragraphs
    {'insertText': {
        'text': 'Bold 1\nItal 2\n\tfoo\n\tbar\n\t\tbaz\n\t\tqux\nMono 3',
        'objectId': textboxID,
    }},
    # shrink text from 48pt ("main point" textbox default) to 32pt
    {'updateTextStyle': {
        'objectId': textboxID,
        'style': {'fontSize': {'magnitude': 32, 'unit': 'PT'}},
        'textRange': {'type': 'ALL'},
        'fields': 'fontSize',
    }},
    # change word 1 in para 1 ("Bold") to bold
    {'updateTextStyle': {
        'objectId': textboxID,
        'style': {'bold': True},
        'textRange': {'type': 'FIXED_RANGE', 'startIndex': 0, 'endIndex': 4},
        'fields': 'bold',
    }},
    # change word 1 in para 2 ("Ital") to italics
    {'updateTextStyle': {
        'objectId': textboxID,
        'style': {'italic': True},
        'textRange': {'type': 'FIXED_RANGE', 'startIndex': 7, 'endIndex': 11},
        'fields': 'italic'
    }},
    # change word 1 in para 7 ("Mono") to Courier New
    {'updateTextStyle': {
        'objectId': textboxID,
        'style': {'fontFamily': 'Courier New'},
        'textRange': {'type': 'FIXED_RANGE', 'startIndex': 36, 'endIndex': 40},
        'fields': 'fontFamily'
    }},
    # bulletize everything
    {'createParagraphBullets': {
        'objectId': textboxID,
        'textRange': {'type': 'ALL'},
    }},
]

After the text is inserted, the first operation this code performs is to change the font size of all the text inserted ('ALL' means to format the entire text range) to 32 pt. The main point layout specifies a default font size of 48 pt, so this request shrinks the text so that everything fits and doesn't wrap. The 'fields' parameter specifies that only the 'fontSize' attribute is affected by this command, meaning leave others such as the font type, color, etc., alone.

The next request bolds the first word of the first paragraph. Instead of 'ALL', the exact range for the first word is given. (NOTE: the end index is excluded from the range, so that's why it must be 4 instead of 3, or you're going to lose one character.) In this case, it's the "Bold" word from the first paragraph, "Bold 1". Again, 'fields' is present to indicate that only the font size should be affected by this request while everything else is left alone. The next directive is nearly identical except for italicizing the first word ("Ital") of the second paragraph ("Ital 2").

After this we have a text style request to alter the font of the first word ("Mono") in the last paragraph ("Mono 3") to Courier New. The only other difference is that 'fields' is now 'fontFamily' instead of a flag. Finally, bulletize all paragraphs. Another call to SLIDES.presentations().batchUpdate() and we're done.

Conclusion

If you run the script, you should get output that looks something like this, with each print() representing execution of key parts of the application:

$ python3 slides_format_text.py 
** Create new slide deck
** Create "main point" layout slide & add titles
** Fetch "main point" slide title (textbox) ID
** Insert text & perform various formatting operations
DONE

When the script has completed, you should have a new presentation with these slides:

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!)—by using, copying, and/or modifying this code or any other piece of source from this blog, you implicitly agree to its Apache2 license:

from __future__ import print_function

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/presentations',
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
SLIDES = discovery.build('slides', 'v1', http=creds.authorize(Http()))

print('** Create new slide deck')
DATA = {'title': 'Slides text formatting DEMO'}
rsp = SLIDES.presentations().create(body=DATA).execute()
deckID = rsp['presentationId']
titleSlide = rsp['slides'][0]
titleID = titleSlide['pageElements'][0]['objectId']
subtitleID = titleSlide['pageElements'][1]['objectId']

print('** Create "main point" layout slide & add titles')
reqs = [
    {'createSlide': {'slideLayoutReference': {'predefinedLayout': 'MAIN_POINT'}}},
    {'insertText': {'objectId': titleID, 'text': 'Formatting text'}},
    {'insertText': {'objectId': subtitleID, 'text': 'via the Google Slides API'}},
]
rsp = SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute().get('replies')
slideID = rsp[0]['createSlide']['objectId']

print('** Fetch "main point" slide title (textbox) ID')
rsp = SLIDES.presentations().pages().get(presentationId=deckID,
        pageObjectId=slideID).execute().get('pageElements')
textboxID = rsp[0]['objectId']

print('** Insert text & perform various formatting operations')
reqs = [
    # add 7 paragraphs
    {'insertText': {
        'text': 'Bold 1\nItal 2\n\tfoo\n\tbar\n\t\tbaz\n\t\tqux\nMono 3',
        'objectId': textboxID,
    }},
    # shrink text from 48pt ("main point" textbox default) to 32pt
    {'updateTextStyle': {
        'objectId': textboxID,
        'style': {'fontSize': {'magnitude': 32, 'unit': 'PT'}},
        'textRange': {'type': 'ALL'},
        'fields': 'fontSize',
    }},
    # change word 1 in para 1 ("Bold") to bold
    {'updateTextStyle': {
        'objectId': textboxID,
        'style': {'bold': True},
        'textRange': {'type': 'FIXED_RANGE', 'startIndex': 0, 'endIndex': 4},
        'fields': 'bold',
    }},
    # change word 1 in para 2 ("Ital") to italics
    {'updateTextStyle': {
        'objectId': textboxID,
        'style': {'italic': True},
        'textRange': {'type': 'FIXED_RANGE', 'startIndex': 7, 'endIndex': 11},
        'fields': 'italic'
    }},
    # change word 1 in para 6 ("Mono") to Courier New
    {'updateTextStyle': {
        'objectId': textboxID,
        'style': {'fontFamily': 'Courier New'},
        'textRange': {'type': 'FIXED_RANGE', 'startIndex': 36, 'endIndex': 40},
        'fields': 'fontFamily'
    }},
    # bulletize everything
    {'createParagraphBullets': {
        'objectId': textboxID,
        'textRange': {'type': 'ALL'},
    }},
]
SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute()
print('DONE')

As with our other code samples, you can now customize it to learn more about the API, integrate into other apps for your own needs, for a mobile frontend, sysadmin script, or a server-side backend!

Modifying email signatures with the Gmail API

2016-12-06T15:59:00.000-08:00

NOTE: The content here is also available as a video and overview post, part of this series.

UPDATE (Feb 2017): Tweaked the code sample as the isPrimary flag may be missing from non-primary aliases; also added link above to video.

Introduction

In a previous post, I introduced Python developers to the Gmail API with a tutorial on how to search for threads with a minimum number of messages. Today, we'll explore another part of the API, covering the settings endpoints that were added in mid-2016. What's the big deal? Well, you couldn't use the API to read nor modify user settings before, and now you can!

One example all of us can relate to is your personal email signature. Wouldn't it be great if we could modify it programmatically, say to include some recent news about you (perhaps a Tweet other social post), or maybe some random witty quote? You could then automate it to change once a quarter, or even hourly if you like being truly random!

Using the Gmail API

Our simple Python script won't be sending email nor reading user messages, so the only authorization scope needed is the one that accesses basic user settings (there's another for more sensitive user settings):

https://www.googleapis.com/auth/gmail.settings.basic — Manage basic Gmail user settings

See the documentation for a list of all Gmail API scopes and what each of them mean. Since we've fully covered the authorization boilerplate in earlier posts and videos, including how to connect to the Gmail API, we're going to skip that here and jump right to the action. You can copy the boilerplate from other scripts you've written. Regardless, be sure to create an service endpoint to the API:

GMAIL = discovery.build('gmail', 'v1',

    http=creds.authorize(Http()))

What are "sendAs" email addresses?

First, a quick word about "sendAs" email addresses. Gmail lets you send email from addresses other than your actual Gmail address (considered your primary address). This lets you manage multiple accounts from the same Gmail user interface. (As expected, you need to own or otherwise have access to the alternate email addresses in order to do this.) However, most people only use their primary address, so you may not know about it. You can learn more about sendAs addresses here and here.

Now you may be tempted to use the term "alias," especially because that word was mentioned in those Help pages you just looked at right? However for now, I'd recommend trying to avoid that terminology as it refers to something else in a G Suite/Google Apps context. Can't you see how we already got distracted from the core reason for this post? See, you almost forgot about email signatures already, right? If you stick with "sender addresses" or "sendAs email addresses," there won’t be any confusion.

Using a "Quote of the Day" in your email signature

The Python script we're exploring in this post sets a "Quote of the Day" (or "QotD" for short) as the signature of your primary sendAs address. Where does the QotD come from? Well, it can be as simple (and boring) as this function that returns a hardcoded string:

Cute but not very random right? A better idea is to choose from a number of quotes you have in a relational database w/columns for quotes & authors. Here’s some sample code for data in a SQLite database:

More random, which is cool, but this particular snippet isn't efficient because we’re selecting all rows and then choosing a quote randomly. Obviously there's a better way if a database is your data source. I prefer using a web service instead, coming in the form of a REST API. The code snippet here does just that:

You only need to find a quote-of-the-day service and provide its URL on line 8 that returns a JSON payload. Obviously you'll need a bit more scaffolding if this were a real service, but in this pseudocode example, you can assume that using urllib.{,request.}urlopen() works where the service sends back an empty string upon failure. To play it safe, this snippet falls back to the hardcoded string we saw earlier if the service doesn't return a quote, which comes back as a 2-tuple representing quote and author, respectively.

Setting your new email signature

Now that we're clear on the source for the QotD, we can focus on actually setting it as your new email signature. To do that, we need to get all of your sender (sendAs email) addresses—the goal is only to change your primary addresses (and none of the others if you have any):

addresses = GMAIL.users().settings().sendAs().list(userId='me',
    fields='sendAs(isPrimary,sendAsEmail)').execute().get('sendAs')

As in our other Gmail example, a userId of 'me' indicates the currently-authenticated user. The API will return a number of attributes. If know exactly which ones we want, we can specify them in with the fields attribute so as to control size of the return payload which may contribute to overall latency. In our case, we're requesting just the sendAs.isPrimary flag and sendAs.sendAsEmail, the actual email address string of the sender addresses. What's returned is a Python list consisting of all of your sendAs email addresses, which we cycle through to find the primary address:

for address in addresses:
    if address.get('isPrimary'):
        break

One of your sender addresses must be primary, so unless there's a bug in Gmail, when control of the for loop concludes, address will point to your primary sender address. Now all you have to do is set the signature and confirm to the user:

rsp = GMAIL.users().settings().sendAs().patch(userId='me',
        sendAsEmail=address['sendAsEmail'], body=DATA).execute()
print("Signature changed to '%s'" % rsp['signature'])

If you only have one sender address, there's no need request all the addresses and loop through them looking for the primary address as we did above. In such circumstances, that entire request and loop are extraneous... just pass your email address as the sendAsEmail argument, like this:

rsp = GMAIL.users().settings().sendAs().patch(userId='me',
        sendAsEmail=YOUR_EMAIL_ADDR_HERE, body=DATA).execute()

Conclusion

That's all there is... just 26 lines of code. If we use the static string qotd() function above, your output when running this script will look like this:

$ python gmail_change_sig.py # or python3
Signature changed to '&quot;I heart cats.&quot;  ~anonymous'
$

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!). By using, copying, and/or modifying this code or any other piece of source from this blog, you implicitly agree to its Apache2 license:

from __future__ import print_function

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

import qotd
DATA = {'signature': qotd.qotd()}   # quote source up-to-you!

SCOPES = 'https://www.googleapis.com/auth/gmail.settings.basic'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
GMAIL = discovery.build('gmail', 'v1', http=creds.authorize(Http()))
# this entire block optional if you only have one sender address
addresses = GMAIL.users().settings().sendAs().list(userId='me',
        fields='sendAs(isPrimary,sendAsEmail)').execute().get('sendAs')
for address in addresses:
    if address.get('isPrimary'):
        break
rsp = GMAIL.users().settings().sendAs().patch(userId='me',
        sendAsEmail=address['sendAsEmail'], body=DATA).execute()
print("Signature changed to '%s'" % rsp['signature'])

As with our other code samples, you can now customize for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs.

Code challenge

Want to exercise your newfound knowledge of using the Gmail API's settings endpoints? Write a script that uses the API to manage filters or configure a vacation responder. HINT: take a look at the official Gmail API docs, including the pages specific to filters and vacation settings.

Generating slides from spreadsheet data

2016-11-29T14:33:00.000-08:00

NOTE: The code covered in this post are also available in a video walkthrough.

Introduction

A common use case when you have data in a spreadsheet or database, is to find ways of making that data more visually appealing to others. This is the subject of today's post, where we'll walk through a simple Python script that generates presentation slides based on data in a spreadsheet using both the Google Sheets and Slides APIs.

Specifically, we'll take all spreadsheet cells containing values and create an equivalent table on a slide with that data. The Sheet also features a pre-generated pie chart added from the Explore in Google Sheets feature that we'll import into a blank slide. Not only do we do that, but if the data in the Sheet is updated (meaning the chart is as well), then so can the imported chart image in the presentation. These are just two examples of generating slides from spreadsheet data. The example Sheet we're getting the data from for this script looks like this:

The data in this Sheet originates from the Google Sheets API codelab. In the codelab, this data lives in a SQLite relational database, and in the previous post covering how to migrate SQL data to Google Sheets, we "imported" that data into the Sheet we're using. As mentioned before, the pie chart comes from the Explore feature.

Using the Google Sheets & Slides APIs

The scopes needed for this application are the read-only scope for Sheets (to read the cell contents and the pie chart) and the read-write scope for Slides since we're creating a new presentation:

'https://www.googleapis.com/auth/spreadsheets.readonly' — Read-only access to Google Sheets and properties
'https://www.googleapis.com/auth/presentations' — Read-write access to Slides and Slides presentation properties

If you're new to using Google APIs, we recommend reviewing earlier posts & videos covering the setting up projects and the authorization boilerplate so that we can focus on the main app. Once we've authorized our app, two service endpoints are created, one for each API. The one for Sheets is saved to the SHEETS variable while the one for Slides goes to SLIDES.

Start with Sheets

The first thing to do is to grab all the data we need from the Google Sheet using the Sheets API. You can either supply your own Sheet with your own chart, or you can run the script from the earlier post mentioned earlier to create an identical Sheet as above. In either case, you need to provide the Sheet ID to read from, which is saved to the sheetID variable. Using its ID, we call spreadsheets().values().get() to pull out all the cells (as rows & columns) from the Sheet and save it to orders:

sheetID = '. . .'   # use your own!
orders = SHEETS.spreadsheets().values().get(range='Sheet1',
        spreadsheetId=sheetID).execute().get('values')

The next step is to call spreadsheets().get() to get all the sheets in the Sheet —there's only one, so grab it at index 0. Since this sheet only has one chart, we also use index 0 to get that:

sheet = SHEETS.spreadsheets().get(spreadsheetId=sheetID,
        ranges=['Sheet1']).execute().get('sheets')[0]
chartID = sheet['charts'][0]['chartId']

That's it for Sheets. Everything from here on out takes places in Slides.

Create new Slides presentation

A new slide deck can be created with SLIDES.presentations().create()—or alternatively with the Google Drive API which we won't do here. We'll name it, "Generating slides from spreadsheet data DEMO" and save its (new) ID along with the IDs of the title and subtitle textboxes on the (one) title slide created in the new deck:

DATA = {'title': 'Generating slides from spreadsheet data DEMO'}
rsp = SLIDES.presentations().create(body=DATA).execute()
deckID = rsp['presentationId']
titleSlide = rsp['slides'][0]
titleID = titleSlide['pageElements'][0]['objectId']
subtitleID = titleSlide['pageElements'][1]['objectId']

Create slides for table & chart

A mere title slide doesn't suffice as we need a place for the cell data as well as the pie chart, so we'll create slides for each. While we're at it, we might as well fill in the text for the presentation title and subtitle. These requests are self-explanatory as you can see below in the reqs variable. The SLIDES.presentations().batchUpdate() method is then used to send the four commands to the API. Upon return, save the IDs for both the cell table slide as well as the blank slide for the chart:

reqs = [
  {'createSlide': {'slideLayoutReference': {'predefinedLayout': 'TITLE_ONLY'}}},
  {'createSlide': {'slideLayoutReference': {'predefinedLayout': 'BLANK'}}},
  {'insertText': {'objectId': titleID,    'text': 'Importing Sheets data'}},
  {'insertText': {'objectId': subtitleID, 'text': 'via the Google Slides API'}},
]
rsp = SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute().get('replies')
tableSlideID = rsp[0]['createSlide']['objectId']
chartSlideID = rsp[1]['createSlide']['objectId']

Note the order of the requests. The create-slide requests come first followed by the text inserts. Responses that come back from the API are returned in the same order as they were sent, hence why the cell table slide ID comes back first (index 0) followed by the chart slide ID (index 1). The text inserts don't have any meaningful return values and are thus ignored.

Filling out the table slide

Now let's focus on the table slide. There are two things we need to accomplish. In the previous set of requests, we asked the API to create a "title only" slide, meaning there's (only) a textbox for the slide title. The next snippet of code gets all the page elements on that slide so we can get the ID of that textbox, the only thing on that page:

rsp = SLIDES.presentations().pages().get(presentationId=deckID,
        pageObjectId=tableSlideID).execute().get('pageElements')
textboxID = rsp[0]['objectId']

On this slide, we need to add the cell table for the Sheet data, so a create-table request takes care of that. The required elements in such a call include the ID of the slide the table should go on as well as the total number of rows and columns desired. Fortunately all that are available from tableSlideID and orders saved earlier. Oh, and add a title for this table slide too. Here's the code:

reqs = [
    {'createTable': {
        'elementProperties': {'pageObjectId': tableSlideID},
        'rows': len(orders),
        'columns': len(orders[0])},
    },
    {'insertText': {'objectId': textboxID, 'text': 'Toy orders'}},
]
rsp = SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute().get('replies')
tableID = rsp[0]['createTable']['objectId']

Another call to SLIDES.presentations().batchUpdate() and we're done, saving the ID of the newly-created table. Next, we'll fill in each cell of that table.

Populate table & add chart image

The first set of requests needed now fill in each cell of the table. The most compact way to issue these requests is with a double-for loop list comprehension. The first loops over the rows while the second loops through each column (of each row). Magically, this creates all the text insert requests needed.

reqs = [
    {'insertText': {
        'objectId': tableID,
        'cellLocation': {'rowIndex': i, 'columnIndex': j},
        'text': str(data),
    }} for i, order in enumerate(orders) for j, data in enumerate(order)]

The final request "imports" the chart from the Sheet onto the blank slide whose ID we saved earlier. Note, while the dimensions below seem completely arbitrary, be assured we're using the same size & transform as a blank rectangle we drew on the slide earlier (and read those values from). The alternative would be to use math to come up with your object dimensions. Here is the code we're talking about, followed by the actual call to the API:

reqs.append({'createSheetsChart': {
    'spreadsheetId': sheetID,
    'chartId': chartID,
    'linkingMode': 'LINKED',
    'elementProperties': {
        'pageObjectId': chartSlideID,
        'size': {
            'height': {'magnitude': 7075, 'unit': 'EMU'},
            'width':  {'magnitude': 11450, 'unit': 'EMU'}
        },
        'transform': {
            'scaleX': 696.6157,
            'scaleY': 601.3921,
            'translateX': 583875.04,
            'translateY': 444327.135,
            'unit': 'EMU',
        },
    },
}})
SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute()

Once all the requests have been created, send them to the Slides API then we're done. (In the actual app, you'll see we've sprinkled various print() calls to let the user knows which steps are being executed.

Conclusion

The entire script clocks in at just under 100 lines of code... see below. If you run it, you should get output that looks something like this:

$ python3 slides_table_chart.py
** Fetch Sheets data
** Fetch chart info from Sheets
** Create new slide deck
** Create 2 slides & insert slide deck title+subtitle
** Fetch table slide title (textbox) ID
** Create table & insert table slide title
** Fill table cells & create linked chart to Sheets
DONE

When the script has completed, you should have a new presentation with these 3 slides:

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!). If I were to divide the script into major sections, they would be represented by each of the print() calls above. Here's the complete script—by using, copying, and/or modifying this code or any other piece of source from this blog, you implicitly agree to its Apache2 license:

from __future__ import print_function

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = (
    'https://www.googleapis.com/auth/spreadsheets.readonly',
    'https://www.googleapis.com/auth/presentations',
)
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
HTTP = creds.authorize(Http())
SHEETS = discovery.build('sheets', 'v4', http=HTTP)
SLIDES = discovery.build('slides', 'v1', http=HTTP)

print('** Fetch Sheets data')
sheetID = '. . .'   # use your own!
orders = SHEETS.spreadsheets().values().get(range='Sheet1',
        spreadsheetId=sheetID).execute().get('values')

print('** Fetch chart info from Sheets')
sheet = SHEETS.spreadsheets().get(spreadsheetId=sheetID,
        ranges=['Sheet1']).execute().get('sheets')[0]
chartID = sheet['charts'][0]['chartId']

print('** Create new slide deck')
DATA = {'title': 'Generating slides from spreadsheet data DEMO'}
rsp = SLIDES.presentations().create(body=DATA).execute()
deckID = rsp['presentationId']
titleSlide = rsp['slides'][0]
titleID = titleSlide['pageElements'][0]['objectId']
subtitleID = titleSlide['pageElements'][1]['objectId']

print('** Create 2 slides & insert slide deck title+subtitle')
reqs = [
  {'createSlide': {'slideLayoutReference': {'predefinedLayout': 'TITLE_ONLY'}}},
  {'createSlide': {'slideLayoutReference': {'predefinedLayout': 'BLANK'}}},
  {'insertText': {'objectId': titleID,    'text': 'Importing Sheets data'}},
  {'insertText': {'objectId': subtitleID, 'text': 'via the Google Slides API'}},
]
rsp = SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute().get('replies')
tableSlideID = rsp[0]['createSlide']['objectId']
chartSlideID = rsp[1]['createSlide']['objectId']

print('** Fetch table slide title (textbox) ID')
rsp = SLIDES.presentations().pages().get(presentationId=deckID,
        pageObjectId=tableSlideID).execute().get('pageElements')
textboxID = rsp[0]['objectId']

print('** Create table & insert table slide title')
reqs = [
    {'createTable': {
        'elementProperties': {'pageObjectId': tableSlideID},
        'rows': len(orders),
        'columns': len(orders[0])},
    },
    {'insertText': {'objectId': textboxID, 'text': 'Toy orders'}},
]
rsp = SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute().get('replies')
tableID = rsp[0]['createTable']['objectId']

print('** Fill table cells & create linked chart to Sheets')
reqs = [
    {'insertText': {
        'objectId': tableID,
        'cellLocation': {'rowIndex': i, 'columnIndex': j},
        'text': str(data),
    }} for i, order in enumerate(orders) for j, data in enumerate(order)]

reqs.append({'createSheetsChart': {
    'spreadsheetId': sheetID,
    'chartId': chartID,
    'linkingMode': 'LINKED',
    'elementProperties': {
        'pageObjectId': chartSlideID,
        'size': {
            'height': {'magnitude': 7075, 'unit': 'EMU'},
            'width':  {'magnitude': 11450, 'unit': 'EMU'}
        },
        'transform': {
            'scaleX': 696.6157,
            'scaleY': 601.3921,
            'translateX': 583875.04,
            'translateY': 444327.135,
            'unit': 'EMU',
        },
    },
}})
SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=deckID).execute()
print('DONE')

As with our other code samples, you can now customize it to learn more about the API, integrate into other apps for your own needs, for a mobile frontend, sysadmin script, or a server-side backend!

Code challenge

Given the knowledge you picked up from this post and its code sample, augment the script with another call to the Sheets API that updates the number of toys ordered by one of the customers, then add the corresponding call to the Slides API that refreshes the linked image based on the changes made to the Sheet (and chart). EXTRA CREDIT: Use the Google Drive API to monitor the Sheet so that any updates to toy orders will result in an "automagic" update of the chart image in the Slides presentation.

Replacing text & images with the Google Slides API with Python

2016-11-09T10:00:00.000-08:00

NOTE: The code covered in this post are also available in a video walkthrough however the code here differs slightly, featuring some minor improvements to the code in the video.

Introduction

One of the critical things developers have not been able to do previously was access Google Slides presentations programmatically. To address this "shortfall," the Slides team pre-announced their first API a few months ago at Google I/O 2016—also see full announcement video (40+ mins). In early November, the G Suite product team officially launched the API, finally giving all developers access to build or edit Slides presentations from their applications.

In this post, I'll walk through a simple example featuring an existing Slides presentation template with a single slide. On this slide are placeholders for a presentation name and company logo, as illustrated below:

One of the obvious use cases that will come to mind is to take a presentation template replete with "variables" and placeholders, and auto-generate decks from the same source but created with different data for different customers. For example, here's what a "completed" slide would look like after the proxies have been replaced with "real data:"

Using the Google Slides API

We need to edit/write into a Google Slides presentation, meaning the read-write scope from all Slides API scopes below:

'https://www.googleapis.com/auth/presentations' — Read-write access to Slides and Slides presentation properties
'https://www.googleapis.com/auth/presentations.readonly' — View-only access to Slides presentations and properties
'https://www.googleapis.com/auth/drive' — Full access to users' files on Google Drive

Why is the Google Drive API scope listed above? Well, think of it this way: APIs like the Google Sheets and Slides APIs were created to perform spreadsheet and presentation operations. However, importing/exporting, copying, and sharing are all file-based operations, thus where the Drive API fits in. If you need a review of its scopes, check out the Drive auth scopes page in the docs. Copying a file requires the full Drive API scope, hence why it's listed above. If you're not going to copy any files and only performing actions with the Slides API, you can of course leave it out.

Since we've fully covered the authorization boilerplate fully in earlier posts and videos, we're going to skip that here and jump right to the action.

Getting started

What are we doing in today's code sample? We start with a slide template file that has "variables" or placeholders for a title and an image. The application code will go then replace these proxies with the actual desired text and image, with the goal being that this scaffolding will allow you to automatically generate multiple slide decks but "tweaked" with "real" data that gets substituted into each slide deck.

The title slide template file is TMPFILE, and the image we're using as the company logo is the Google Slides product icon whose filename is stored as the IMG_FILE variable in my Google Drive. Be sure to use your own image and template files! These definitions plus the scopes to be used in this script are defined like this:

IMG_FILE = 'google-slides.png'     # use your own!
TMPLFILE = 'title slide template'  # use your own!
SCOPES = (
    'https://www.googleapis.com/auth/drive',
    'https://www.googleapis.com/auth/presentations',
)

Skipping past most of the OAuth2 boilerplate, let's move ahead to creating the API service endpoints. The Drive API name is (of course) 'drive', currently on 'v3', while the Slides API is 'slides' and 'v1' in the following call to create a signed HTTP client that's shared with a pair of calls to the apiclient.discovery.build() function to create the API service endpoints:

HTTP = creds.authorize(Http())
DRIVE =  discovery.build('drive',  'v3', http=HTTP)
SLIDES = discovery.build('slides', 'v1', http=HTTP)

Copy template file

The first step of the "real" app is to find and copy the template file TMPLFILE. To do this, we'll use DRIVE.files().list() to query for the file, then grab the first match found. Then we'll use DRIVE.files().copy() to copy the file and name it 'Google Slides API template DEMO':

rsp = DRIVE.files().list(q="name='%s'" % TMPLFILE).execute().get('files')[0]
DATA = {'name': 'Google Slides API template DEMO'}
print('** Copying template %r as %r' % (rsp['name'], DATA['name']))
DECK_ID = DRIVE.files().copy(body=DATA, fileId=rsp['id']).execute().get('id')

Find image placeholder

Next, we'll ask the Slides API to get the data on the first (and only) slide in the deck. Specifically, we want the dimensions of the image placeholder. Later on, we will use those properties when replacing it with the company logo, so that it will be automatically resized and centered into the same spot as the image placeholder.
The SLIDES.presentations().get() method is used to read the presentation metadata. Returned is a payload consisting of everything in the presentation, the masters, layouts, and of course, the slides themselves. We only care about the slides, so we get that from the payload. And since there's only one slide, we grab it at index 0. Once we have the slide, we're loop through all of the elements on that page and stop when we find the rectangle (image placeholder):

print('** Get slide objects, search for image placeholder')
slide = SLIDES.presentations().get(presentationId=DECK_ID
       ).execute().get('slides')[0]
obj = None
for obj in slide['pageElements']:
    if obj['shape']['shapeType'] == 'RECTANGLE':
        break

Find image file

At this point, the obj variable points to that rectangle. What are we going to replace it with? The company logo, which we now query for using the Drive API:

print('** Searching for icon file')
rsp = DRIVE.files().list(q="name='%s'" % IMG_FILE).execute().get('files')[0]
print(' - Found image %r' % rsp['name'])
img_url = '%s&access_token=%s' % (
        DRIVE.files().get_media(fileId=rsp['id']).uri, creds.access_token)

The query code is similar to when we searched for the template file earlier. The trickiest thing about this snippet is that we need a full URL that points directly to the company logo. We use the DRIVE.files().get_media() method to create that request but don't execute it. Instead, we dig inside the request object itself and grab the file's URI and merge it with the current access token so what we're left with is a valid URL that the Slides API can use to read the image file and create it in the presentation.

Replace text and image

Back to the Slides API for the final steps: replace the title (text variable) with the desired text, add the company logo with the same size and transform as the image placeholder, and delete the image placeholder as it's no longer needed:

print('** Replacing placeholder text and icon')
reqs = [
    {'replaceAllText': {
        'containsText': {'text': '{{NAME}}'},
        'replaceText': 'Hello World!'
    }},
    {'createImage': {
        'url': img_url,
        'elementProperties': {
            'pageObjectId': slide['objectId'],
            'size': obj['size'],
            'transform': obj['transform'],
        }
    }},
    {'deleteObject': {'objectId': obj['objectId']}},
]
SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=DECK_ID).execute()
print('DONE')

Once all the requests have been created, send them to the Slides API then let the user know everything is done.

Conclusion

That's the entire script, just under 60 lines of code. If you watched the video, you may notice a few minor differences in the code. One is use of the fields parameter in the Slides API calls. They represent the use of field masks, which is a separate topic on its own. As you're learning the API now, it may cause unnecessary confusion, so it's okay to disregard them for now. The other difference is an improvement in the replaceAllText request—the old way in the video is now deprecated, so go with what we've replaced it with in this post.

If your template slide deck and image is in your Google Drive, and you've modified the filenames and run the script, you should get output that looks something like this:

$ python3 slides_template.py
** Copying template 'title slide template' as 'Google Slides API template DEMO'
** Get slide objects, search for image placeholder
** Searching for icon file
 - Found image 'google-slides.png'
** Replacing placeholder text and icon
DONE

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!). If I were to divide the script into major sections, they would be:

Get creds & build API service endpoints
Copy template file
Get image placeholder size & transform (for replacement image later)
Get secure URL for company logo
Build and send Slides API requests to...

Replace slide title variable with "Hello World!"
Create image with secure URL using placeholder size & transform
Delete image placeholder

Here's the complete script—by using, copying, and/or modifying this code or any other piece of source from this blog, you implicitly agree to its Apache2 license:

from __future__ import print_function

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

IMG_FILE = 'google-slides.png'      # use your own!
TMPLFILE = 'title slide template'   # use your own!
SCOPES = (
    'https://www.googleapis.com/auth/drive',
    'https://www.googleapis.com/auth/presentations',
)
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
HTTP = creds.authorize(Http())
DRIVE  = discovery.build('drive',  'v3', http=HTTP)
SLIDES = discovery.build('slides', 'v1', http=HTTP)

rsp = DRIVE.files().list(q="name='%s'" % TMPLFILE).execute().get('files')[0]
DATA = {'name': 'Google Slides API template DEMO'}
print('** Copying template %r as %r' % (rsp['name'], DATA['name']))
DECK_ID = DRIVE.files().copy(body=DATA, fileId=rsp['id']).execute().get('id')

print('** Get slide objects, search for image placeholder')
slide = SLIDES.presentations().get(presentationId=DECK_ID,
        fields='slides').execute().get('slides')[0]
obj = None
for obj in slide['pageElements']:
    if obj['shape']['shapeType'] == 'RECTANGLE':
        break

print('** Searching for icon file')
rsp = DRIVE.files().list(q="name='%s'" % IMG_FILE).execute().get('files')[0]
print(' - Found image %r' % rsp['name'])
img_url = '%s&access_token=%s' % (
        DRIVE.files().get_media(fileId=rsp['id']).uri, creds.access_token)

print('** Replacing placeholder text and icon')
reqs = [
    {'replaceAllText': {
        'containsText': {'text': '{{NAME}}'},
        'replaceText': 'Hello World!'
    }},
    {'createImage': {
        'url': img_url,
        'elementProperties': {
            'pageObjectId': slide['objectId'],
            'size': obj['size'],
            'transform': obj['transform'],
        }
    }},
    {'deleteObject': {'objectId': obj['objectId']}},
]
SLIDES.presentations().batchUpdate(body={'requests': reqs},
        presentationId=DECK_ID).execute()
print('DONE')

As with our other code samples, you can now customize it to learn more about the API, integrate into other apps for your own needs, for a mobile frontend, sysadmin script, or a server-side backend!

Code challenge

Add more slides and/or text variables and modify the script replace them too. EXTRA CREDIT: Change the image-based image placeholder to a text-based image placeholder, say a textbox with the text, "{{COMPANY_LOGO}}" and use the replaceAllShapesWithImage request to perform the image replacement. By making this one change, your code should be simplified from the image-based image replacement solution we used in this post.

Formatting cells in Google Sheets with Python

2016-09-28T12:40:00.001-07:00

Introduction

One of the critical things that developers have not been able to do in previous versions of the Google Sheets API is to format cells... that's a big deal! Anyway, the past is the past, and I choose to look ahead. In my earlier post on the Google Sheets API, I introduced Sheets API v4 with a tutorial on how to transfer data from a SQL database to a Sheet. You'd do that primarily to make database data more presentable rather than deliberately switching to a storage mechanism with weaker querying capabilities. At the very end of that post, I challenged readers to try formatting. If you got stuck, confused, or haven't had a chance yet, today's your lucky day. One caveat is that there's more JavaScript in this post than Python... you've been warned!

Using the Google Sheets API

We need to write (formatting) into a Google Sheet, so we need the same scope as last time, read-write:

'https://www.googleapis.com/auth/spreadsheets' — Read-write access to Sheets and Sheet properties

Since we've fully covered the authorization boilerplate fully in earlier posts and videos, including how to connect to the Sheets API, we're going to skip that here and jump right to the action.

Formatting cells in Google Sheets

The way the API works, in general, is to take one or more commands, and execute them on the Sheet. This comes in the form of individual requests, either to cells, a Sheet, or the entire spreadsheet. A group if requests is organized as a JavaScript array. Each request in the array is represented by a JSON object. Yes, this part of the post may seem like a long exercise in JavaScript, but stay with me here. Continuing... once your array is complete, you send all the requests to the Sheet using the SHEETS.spreadsheets().batchUpdate() command. Here's pseudocode sending 5 commands to the API:

SHEET_ID =  . . .

reqs = {'requests': [
    {'updateSheetProperties':
        . . .
    {'repeatCell':
        . . .
    {'setDataValidation':
        . . .
    {'sortRange':
        . . .
    {'addChart':
        . . .
]}
SHEETS.spreadsheets().batchUpdate(
        spreadsheetId=SHEET_ID, body=reqs).execute()

What we're executing will be similar. The target spreadsheet will be the one you get when you run the code from the previous post, only without the timestamp in its title as it's unnecessary:

Once you've run the earlier script and created a Sheet of your own, be sure to assign it to the SHEET_ID variable. The goal is to send enough formatting commands to arrive at the same spreadsheet but with improved visuals:

Four (4) requests are needed to bring the original Sheet to this state:

Set the top row as "frozen", meaning it doesn't scroll even when the data does
Also bold the first row, as these are the column headers
Format column E as US currency with dollar sign & 2 decimal places
Set data validation for column F, requiring values from a fixed set

Creating Sheets API requests

As mentioned before, each request is represented by a JSON object, cleverly disguised as Python dictionaries in this post, and the entire request array is implemented as a Python list. Let's take a look at what it takes to together the individual requests:

Frozen rows

Frozen rows is a Sheet property, so in order to change it, users must employ the updateSheetProperties command. Specifically, frozenRowCount is a grid property, meaning the field that must be updated is gridProperties.frozenRowCount, set to 1. Here's the Python dict (that gets converted to a JSON object) representing this request:

{'updateSheetProperties': {
    'properties': {'gridProperties': {'frozenRowCount': 1}},
    'fields': 'gridProperties.frozenRowCount',
}},

The properties attribute specifies what is changing and what the new value is. The fields property serves as an attribute "mask." It's how you specify what to alter and what to leave alone when applying a change. In this case, both the properties and fields attributes refer to the same thing: the frozen row count grid property. If you leave out the fields attribute here, sure the frozen row count would be set but all other grid properties would be undone, not such a good side effect. It's okay if it doesn't make a lot of sense yet... there are more examples coming.

Bold formatting

Text formatting, such as bold or italics, is a cell operation. Since we want to apply this formatting to multiple cells, the correct command is, repeatCell. Specifically, what needs to be changed about a cell? A cell's userEnteredFormat.textFormat.bold attribute. This is a simple Boolean value, so we set it to True. The fields masks are as described above... we need to tell the API to explicitly change the just userEnteredFormat.textFormat.bold attribute. Everything else should stay as-is.

The last thing we need is to tell the API which cells in the Sheet should be formatted. For this we have range. This attribute tells the API what Sheet (by ID) and which cells (column and row ranges) in that Sheet to format. Above, you see that we want to bold just one row, row #1. Similarly, there are currently six columns to bold, A-F.

However, like most modern computer systems, the API supports start and end index values beginning with zero... not alphabetic column names nor rows consisting of whole numbers, and the range is exclusive of the end index, meaning it goes up to but does not include the ending row or column. For row 1, this means a starting index of 0 and an ending index of 1. Similarly, columns A-F have start & end index value of 0 and 6, respectively. Visually, here's how you compare traditional row & column values to 0-based index counting:

Here's the dict representing this request:

{'repeatCell': {
    'range': {
        'sheetId': 0,
        'startColumnIndex': 0,
        'endColumnIndex': 6,
        'startRowIndex': 0,
        'endRowIndex': 1
    },
    'cell': {'userEnteredFormat': {'textFormat': {'bold': True}}},
    'fields': 'userEnteredFormat.textFormat.bold',
}},

Before we move on, let's talk about some shortcuts we can make. The ID of the first Sheet created for you is 0. If that's the Sheet you're using, then you can omit passing the Sheet ID. Similarly, the starting row and column indexes default to 0, so you can leave those out too if those are the values to be used. Finally, while an ending column index of 6 works, it won't if more columns are added later. It's best if you just omit the ending index altogether, meaning you want that entire row formatted. All this means that the only thing in the range you need is the ending row index. Instead of the above, your request can be shortened to:

{'repeatCell': {
    'range': {'endRowIndex': 1},
    'cell': {'userEnteredFormat': {'textFormat': {'bold': True}}},
    'fields': 'userEnteredFormat.textFormat.bold',
}},

Range quiz

Okay, now that you know about ranges, take this quiz: assumming the Sheet ID is 0, what are the starting and ending column and row indexes for the four cells highlighted in blue in this Sheet?

If you went with starting and ending column indexes of 3 and 5 and row indexes of 2 and 4, then you're right on the money and ready for more!

Currency formatting

Currency formatting is similar to text formatting, only with numbers, meaning that instead of userEnteredFormat.textFormat, you'd be setting a cell's userEnteredFormat.numberFormat attribute. The command is also repeatCell. Clearly the starting and ending column indexes should be 4 and 5 with a starting row index of 1. But just like the cell bolding we did above, there's no need to restrict ourselves to just the 5 rows of data as more may be coming. Yes, it's best to leave off the ending row index so that the rest of the column is formatted. The only thing you need to learn is how to format cells using US currency, but that's pretty easy to do after a quick look at the docs on formatting numbers:

{'repeatCell': {
    'range': {
        'startRowIndex': 1,
        'startColumnIndex': 4,
        'endColumnIndex': 5,
    },
    'cell': {
        'userEnteredFormat': {
            'numberFormat': {
                'type': 'CURRENCY',
                'pattern': '"$"#,##0.00',
            },
        },
    },
    'fields': 'userEnteredFormat.numberFormat',
}}

More on `fields`

One caveat to our sample app here is that all of the fields mask only have a single value, the one we want to change, but that's not always the case. There may be situations where you want to effect a variety of changes to cells. To see more examples of fields, check out this page in the docs featuring more formatting examples. To learn more about how masking works, check out this page and this one too.

Cell validation

The final formatting request implements cell validation on column F as well as restricting their possible values. The command used here is setDataValidation. The range is similar to that of currency formatting, only for column F, meaning a starting row index of 1, and starting and ending column indexes of 5 and 6, respectively. The rule implements the restriction. Similar to other spreadsheet software, you can restrict cell values in any number of ways, as outlined by the ConditionType documentation page. Ours is to allow for one of three possible values, so the ConditionType is ONE_OF_LIST.

When you restrict cell values, you can choose to allow but flag it (weak enforcement) or disallow any value outside of what you specify (strict enforcement). If you wish to employ strict enforcement, you need to pass in a strict attribute with a True value. The default is weak enforcement, or False. In either case, users entering invalid values will get a default warning message that the input is not allowed. If you prefer a custom message over the default option, you can pass that to the API as the inputMessage attribute. I prefer the system default and elect not to use it here. Here are the 4 combinations of what shows up when you use or don't use inputMessage with strict and weak enforcement:

No inputMessage (default) + weak enforcement

With inputMessage (my custom msg) + weak enforcement

No inputMessage (default) + strict enforcement

With inputMessage (my custom msg) + weak enforcement

The last attribute you can send is showCustomUi. If the showCustomUi flag is set to True, the Sheets user interface will display a small pulldown menu listing the values accepted by the cell. It's a pretty poor user experience without it (because users won’t know what the available choices are), so I recommend you always use it too. With that, this request looks like this:

{'setDataValidation': {
    'range': {
        'startRowIndex': 1,
        'startColumnIndex': 5,
        'endColumnIndex': 6,
    },
    'rule': {
        'condition': {
            'type': 'ONE_OF_LIST',
            'values': [
                {'userEnteredValue': 'PENDING'},
                {'userEnteredValue': 'SHIPPED'},
                {'userEnteredValue': 'DELIVERED'},
            ]
        },
        #'inputMessage': 'Select PENDING, SHIPPED, or DELIVERED',
        #'strict': True,
        'showCustomUi': True,
    },
}}

Since we're not modifying cell attributes, but instead focusing on validation, you'll notice there's no fields mask in these types of requests.

Running our script

Believe it or not, that's the bulk of this application. With the reqs list of these four requests, the last line of code calls the Sheets API exactly like the pseudocode above. Now you can simply run it:

$ python sheets_cell_format.py # or python3
$

There's no output from this script, so you should only expect that your Sheet will be formatted once it has completed. If you bring up the Sheet in the user interface, you should see the changes happening in near real-time:

Conclusion

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/spreadsheets'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_id.json', SCOPES)
    creds = tools.run_flow(flow, store)
SHEETS = discovery.build('sheets', 'v4', http=creds.authorize(Http()))

SHEET_ID = ... # add your Sheet ID here
reqs = {'requests': [
    # frozen row 1
    {'updateSheetProperties': {
        'properties': {'gridProperties': {'frozenRowCount': 1}},
        'fields': 'gridProperties.frozenRowCount',
    }},
    # embolden row 1
    {'repeatCell': {
        'range': {'endRowIndex': 1},
        'cell': {'userEnteredFormat': {'textFormat': {'bold': True}}},
        'fields': 'userEnteredFormat.textFormat.bold',
    }},
    # currency format for column E (E2:E7)
    {'repeatCell': {
        'range': {
            'startRowIndex': 1,
            'endRowIndex': 6,
            'startColumnIndex': 4,
            'endColumnIndex': 5,
        },
        'cell': {
            'userEnteredFormat': {
                'numberFormat': {
                    'type': 'CURRENCY',
                    'pattern': '"$"#,##0.00',
                },
            },
        },
        'fields': 'userEnteredFormat.numberFormat',
    }},
    # validation for column F (F2:F7)
    {'setDataValidation': {
        'range': {
            'startRowIndex': 1,
            'endRowIndex': 6,
            'startColumnIndex': 5,
            'endColumnIndex': 6,
        },
        'rule': {
            'condition': {
                'type': 'ONE_OF_LIST',
                'values': [
                    {'userEnteredValue': 'PENDING'},
                    {'userEnteredValue': 'SHIPPED'},
                    {'userEnteredValue': 'DELIVERED'},
                ]
            },
            #'inputMessage': 'Select PENDING, SHIPPED, or DELIVERED',
            #'strict': True,
            'showCustomUi': True,
        },
    }},
]}

res = SHEETS.spreadsheets().batchUpdate(
        spreadsheetId=SHEET_ID, body=reqs).execute()

As with our other code samples, you can now customize for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs.

Code challenge

Once you fully grasp this sample and are ready for a challenge: Use the API to create a column "G" with a "Total Cost" header in cell G1, set cell G2 with the formula to calculate the cost based on toys ordered & cost in columns D & E then and create an autoFill request to replicate that formula down column G. When you're done, the right-hand side of your Sheet now looks like this:

Here are some steps you can take to achieve this improvement:

Create column G with a "Total Cost" header in cell G1; make sure it's bold too (or do you have to?)
Set cell G2 with formula =MULTIPLY(D2,E2)
Use autoFill command to copy formula from G2 down the entire column (HINT: you only need the range attribute)

You're now well under way to being able to writing useful applications with the Sheets API!

Exporting a Google Sheet spreadsheet as CSV

2016-07-11T11:01:00.000-07:00

Introduction

Today, we'll follow-up to my earlier post on the Google Sheets API and multiple posts (first, second, third) on the Google Drive API by answering one common question: How do you download a Google Sheets spreadsheet as a CSV file? The "FAQ"ness of the question itself as well as various versions of Google APIs has led to many similar StackOverflow questions: one, two, three, four, five, just to list a few. Let's answer this question definitively and walk through a Python code sample that does exactly that. The main assumption is that you have a Google Sheet file in your Google Drive named "inventory".

Choosing the right API

Upon first glance, developers may think the Google Sheets API is the one to use. Unfortunately that isn't the case. The Sheets API is the one to use for spreadsheet-oriented operations, such as inserting data, reading spreadsheet rows, managing individual tab/sheets within a spreadsheet, cell formatting, creating charts, adding pivot tables, etc., It isn't meant to perform file-based requests like exporting a Sheet in CSV (comma-separated values) format. For file-oriented operations with a Google Sheet, you would use the Google Drive API.

Using the Google Drive API

As mentioned earlier, Google Drive features numerous API scopes of authorization. As usual, we always recommend you use the most restrictive scope possible that allows your app to do its work. You'll request fewer permissions from your users (which makes them happier), and it also makes your app more secure, possibly preventing modifying, destroying, or corrupting data, or perhaps inadvertently going over quotas. Since we're only exporting a Google Sheets file from Google Drive, the only scope we need is:

'https://www.googleapis.com/auth/drive.readonly' — Read-only access to file content or metadata

The earlier post I wrote on the Google Drive API featured sample code that exported an uploaded Google Docs file as PDF and download that from Drive. This post will not only feature a change to exporting a Google Sheets file in CSV format, but also demonstrate one additional feature of the Drive API: querying

Since we've fully covered the authorization boilerplate fully in earlier posts and videos, we're going to skip that here and jump right to the action, creating of a service endpoint to Drive. The API name is (of course 'drive', and the current version of the API is 3, so use the string 'v3' in this call to the apiclient.discovey.build() function:

DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

Query and export files from Google Drive

While unnecessary, we'll create a few string constants representing the filename, source and destination file MIME types to make the code easier to understand:

FILENAME = 'inventory'
SRC_MIMETYPE = 'application/vnd.google-apps.spreadsheet'
DST_MIMETYPE = 'text/csv'

In this simple example, we're only going to export one Google Sheets file as CSV, arbitrarily choosing a file named, "inventory." So to perform the query, you need both the filename and its MIME type, "application/vnd.google-apps.spreadsheet". Query components are conjoined with the "and" keyword, so your query string will look like this: q='name="%s" and mimeType="%s"' % (FILENAME, SRC_MIMETYPE).

Since there may be more than one Google Sheets file named 'inventory". we opt for newest one and thus need to sort all matching files in descending order of last modification time then name if "mtime"s are identical via an "order by" clause: orderBy='modifiedTime desc,name'. Here is the complete call to DRIVE.files().list() to issue the query:

files = DRIVE.files().list(
    q='name="%s" and mimeType="%s"' % (FILENAME, SRC_MIMETYPE),
    orderBy='modifiedTime desc,name').execute().get('files', [])

If any files match, the payload will contain a 'files' key, else we default to an empty list and display to the user on the last line that no files were found. Otherwise, grab the first match, the most recently-modified 'inventory' file, create a suitable CSV filename from it, and change all spaces to underscores:

fn = '%s.csv' % os.path.splitext(files[0]['name'].replace(' ', '_'))[0]

The final Drive API call requests an export of 'inventory' as a CSV file, and if successful, the downloaded data is written with the filename above. In either case, the user is notified of success or failure of the export:

data = DRIVE.files().export(fileId=files[0]['id'], mimeType=DST_MIMETYPE).execute()
if data:
    with open(fn, 'wb') as f:
        f.write(data)
    print('DONE')
else:
    print('ERROR (could not download file)')

Note that if downloading as CSV, the Drive API only exports of the first sheet in a Sheets file... you won't get any others. However, it does support 3 other download formats that will get you all the sheets.

If you create a Sheets file named 'inventory', run the script, grant the script access to your Google Drive (via the OAuth2 prompt that pops up in the browser), and then you should get output that looks like this:

$ python drive_sheets_csv_export.py # or python3
Exporting "inventory" as "inventory.csv"... DONE

Conclusion

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!). If I were to divide the script into 4 major sections, they would be:

Get creds & build Google Drive service endpoint
Source and destination file info
Query Google Drive for matching files
Export most recent matching Sheets file as CSV

Here's the code itself:

from __future__ import print_function
import os

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive.readonly'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

FILENAME = 'inventory'
SRC_MIMETYPE = 'application/vnd.google-apps.spreadsheet'
DST_MIMETYPE = 'text/csv'

files = DRIVE.files().list(
    q='name="%s" and mimeType="%s"' % (FILENAME, SRC_MIMETYPE),
    orderBy='modifiedTime desc,name').execute().get('files', [])

if files:
    fn = '%s.csv' % os.path.splitext(files[0]['name'].replace(' ', '_'))[0]
    print('Exporting "%s" as "%s"... ' % (files[0]['name'], fn), end='')
    data = DRIVE.files().export(fileId=files[0]['id'], mimeType=DST_MIMETYPE).execute()
    if data:
        with open(fn, 'wb') as f:
            f.write(data)
        print('DONE')
    else:
        print('ERROR (could not download file)')
else:
    print('!!! ERROR: File not found')

As with our other code samples, you can now customize for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs. Hope this helps answer yet another frequently asked question!

Migrating SQL data to Google Sheets using the new Google Sheets API

2016-06-09T14:00:00.001-07:00

NOTE: The code covered in this post are also available in a video walkthrough.
UPDATE (Sep 2016): Removed use of argparse module & flags (effective as of Feb 2016).

Introduction

In this post, we're going to demonstrate how to use the latest generation Google Sheets API. Launched at Google I/O 2016 (full talk here), the Sheets API v4 can do much more than previous versions, bringing it to near-parity with what you can do with the Google Sheets UI (user interface) on desktop and mobile. Below, I'll walk you through a Python script that reads the rows of a relational database representing customer orders for a toy company and pushes them into a Google Sheet. Other API calls we'll make: one to create new Google Sheets with and another that reads the rows from a Sheet.

Earlier posts demonstrated the structure and "how-to" use Google APIs in general, so more recent posts, including this one, focus on solutions and use of specific APIs. Once you review the earlier material, you're ready to start with authorization scopes then see how to use the API itself.

Google Sheets API authorization & scopes

Previous versions of the Google Sheets API (formerly called the Google Spreadsheets API), were part of a group of "GData APIs" that implemented the Google Data (GData) protocol, an older, less-secure, REST-inspired technology for reading, writing, and modifying information on the web. The new API version falls under the more modern set of Google APIs requiring OAuth2 authorization and whose use is made easier with the Google APIs Client Libraries.

The current API version features a pair of authorization scopes: read-only and read-write. As usual, we always recommend you use the most restrictive scope possible that allows your app to do its work. You'll request fewer permissions from your users (which makes them happier), and it also makes your app more secure, possibly preventing modifying, destroying, or corrupting data, or perhaps inadvertently going over quotas. Since we're creating a Google Sheet and writing data into it, we must use the read-write scope:

'https://www.googleapis.com/auth/spreadsheets' — Read/write access to Sheets and Sheet properties

Using the Google Sheets API

Let's look at some code that reads rows from a SQLite database and creates a Google Sheet with that data. Since we covered the authorization boilerplate fully in earlier posts and videos, we're going straight to creating a Sheets service endpoint. The API string to use is 'sheets' and the version string to use is 'v4' as we call the apiclient.discovey.build() function:

SHEETS = discovery.build('sheets', 'v4', http=creds.authorize(Http()))

With the SHEETS service endpoint in hand, the first thing to do is to create a brand new Google Sheet. Before we use it, one thing to know about the Sheets API is that most calls require a JSON payload representing the data & operations you wish to perform, and you'll see this as you become more familiar with it. For creating new Sheets, it's pretty simple, you don't have to provide anything, in which case you'd pass in an empty (dict as the) body, but a better bare minimum would be a name for the Sheet, so that's what data is for:

data = {'properties': {'title': 'Toy orders [%s]' % time.ctime()}}

Notice that a Sheet's "title" is part of its "properties," and we also happen to add the timestamp as part of its name. With the payload complete, we call the API with the command to create a new Sheet [spreadsheets().create()], passing in data in the (eventual) request body:

res = SHEETS.spreadsheets().create(body=data).execute()

Alternatively, you can use the Google Drive API (v2 or v3) to create a Sheet but would also need to pass in the Google Sheets (file) MIME type:

data = {
    'name': 'Toy orders [%s]' % time.ctime(),
    'mimeType': 'application/vnd.google-apps.spreadsheet',
}
res = DRIVE.files().create(body=data).execute() # insert() for v2

The general rule-of-thumb is that if you're only working with Sheets, you can do all the operations with its API, but if creating files other than Sheets or performing other Drive file or folder operations, you may want to stick with the Drive API. You can also use both or any other Google APIs for more complex applications. We'll stick with just the Sheets API for now. After creating the Sheet, grab and display some useful information to the user:

SHEET_ID = res['spreadsheetId']
print('Created "%s"' % res['properties']['title'])

You may be wondering: Why do I need to create a Sheet and then make a separate API call to add data to it? Why can't I do this all when creating the Sheet? The answer (to this likely FAQ) is you can, but you would need to construct and pass in a JSON payload representing the entire Sheet—meaning all cells and their formatting—a much larger and more complex data structure than just an array of rows. (Don't believe me? Try it yourself!) This is why we have all of the spreadsheets().values() methods... to simplify uploading or downloading of only values to or from a Sheet.

Now let's turn our attention to the simple SQLite database file (db.sqlite) available from the Google Sheets Node.js codelab. The next block of code just connects to the database with the standard library sqlite3 package, grabs all the rows, adds a header row, and filters the last two (timestamp) columns:

FIELDS = ('ID', 'Customer Name', 'Product Code', 'Units Ordered',
        'Unit Price', 'Status', 'Created at', 'Updated at')
cxn = sqlite3.connect('db.sqlite')
cur = cxn.cursor()
rows = cur.execute('SELECT * FROM orders').fetchall()
cxn.close()
rows.insert(0, FIELDS)
data = {'values': [row[:6] for row in rows]}

When you have a payload (array of row data) you want to stick into a Sheet, you simply pass in those values to spreadsheets().values().update() like we do here:

SHEETS.spreadsheets().values().update(spreadsheetId=SHEET_ID,
    range='A1', body=data, valueInputOption='RAW').execute()

The call requires a Sheet's ID and command body as expected, but there are two other fields: the full (or, as in our case, the "upper left" corner of the) range of cells to write to (in A1 notation), and valueInputOption indicates how the data should be interpreted, writing the raw values ("RAW") or interpreting them as if a user were entering them into the UI ("USER_ENTERED"), possibly converting strings & numbers based on the cell formatting.

Reading rows out of a Sheet is even easier, the spreadsheets().values().get() call needing only an ID and a range of cells to read:

print('Wrote data to Sheet:')
rows = SHEETS.spreadsheets().values().get(spreadsheetId=SHEET_ID,
    range='Sheet1').execute().get('values', [])
for row in rows:
    print(row)

The API call returns a dict which has a 'values' key if data is available, otherwise we default to an empty list so the for loop doesn't fail.

If you run the code (entire script below) and grant it permission to manage your Google Sheets (via the OAuth2 prompt that pops up in the browser), the output you get should look like this:

$ python3 sheets-toys.py # or python (2.x)
Created "Toy orders [Thu May 26 18:58:17 2016]" with this data:
['ID', 'Customer Name', 'Product Code', 'Units Ordered', 'Unit Price', 'Status']
['1', "Alice's Antiques", 'FOO-100', '25', '12.5', 'DELIVERED']
['2', "Bob's Brewery", 'FOO-200', '60', '18.75', 'SHIPPED']
['3', "Carol's Car Wash", 'FOO-100', '100', '9.25', 'SHIPPED']
['4', "David's Dog Grooming", 'FOO-250', '15', '29.95', 'PENDING']
['5', "Elizabeth's Eatery", 'FOO-100', '35', '10.95', 'PENDING']

Conclusion

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

'''sheets-toys.py -- Google Sheets API demo
    created Jun 2016 by +Wesley Chun/@wescpy
'''
from __future__ import print_function
import sqlite3
import time

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/spreadsheets'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_id.json', SCOPES)
    creds = tools.run_flow(flow, store)
SHEETS = discovery.build('sheets', 'v4', http=creds.authorize(Http()))

data = {'properties': {'title': 'Toy orders [%s]' % time.ctime()}}
res = SHEETS.spreadsheets().create(body=data).execute()
SHEET_ID = res['spreadsheetId']
print('Created "%s"' % res['properties']['title'])

FIELDS = ('ID', 'Customer Name', 'Product Code', 'Units Ordered',
        'Unit Price', 'Status', 'Created at', 'Updated at')
cxn = sqlite3.connect('db.sqlite')
cur = cxn.cursor()
rows = cur.execute('SELECT * FROM orders').fetchall()
cxn.close()
rows.insert(0, FIELDS)
data = {'values': [row[:6] for row in rows]}

SHEETS.spreadsheets().values().update(spreadsheetId=SHEET_ID,
    range='A1', body=data, valueInputOption='RAW').execute()
print('Wrote data to Sheet:')
rows = SHEETS.spreadsheets().values().get(spreadsheetId=SHEET_ID,
    range='Sheet1').execute().get('values', [])
for row in rows:
    print(row)

You can now customize this code for your own needs, for a mobile frontend, devops script, or a server-side backend, perhaps accessing other Google APIs. If this example is too complex, check the Python quickstart in the docs that way simpler, only reading data out of an existing Sheet. If you know JavaScript and are ready for something more serious, try the Node.js codelab where we got the SQLite database from. That's it... hope you find these code samples useful in helping you get started with the latest Sheets API!

EXTRA CREDIT: Feel free to experiment and try cell formatting or other API features. Challenge yourself as there's a lot more to Sheets than just reading and writing values!

Migrating to Google Drive API v3

2015-12-23T10:00:00.000-08:00

NOTE: The code covered in this and the previous post are also available in a video walkthrough. Mar 2018 UPDATE: Modernized the code a bit, shortening it, and changed to R/W scope because drive.file doesn't work if the file hasn't been created yet. The same fixes were made to the Drive API v2 sample in the preceding blog post.

Introduction

In a blog post last week, we introduced readers to performing uploads and downloads files to/from Google Drive from a simple Python command-line script. In an official Google blog post later that same day, the Google Drive API team announced a new version of the API. Great timing huh? Well, good thing I knew it was coming, so that I could prepare this post for you, which is a primer on how to migrate from the current version of the API (v2) to the new one (v3).

As stated by the Drive team, v2 isn't being deprecated, and there are no new features in v3, thus migration isn't required. The new version is mainly for new apps/integrations as well as developers with v2 apps who wish to take advantage of the improvements. This post is intended for those in the latter group, covering porting existing apps to v3. Ready? Let's go straight to the action.

Migrating from Google Drive API v2 to v3

Most of this post will be just examining all the "diffs" between the v2 code sample from the previous post (renamed from drive_updown.py to drive_updown2.py) and its v3 equivalent (drive_updown3.py). We'll take things step-by-step to provide more details, but let's start with all the diffs first:

--- drive_updown2.py   2018-03-11 21:42:33.000000000 -0700
+++ drive_updown3.py   2018-03-11 21:44:57.000000000 -0700
@@ -11,23 +11,24 @@
 if not creds or creds.invalid:
     flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
     creds = tools.run_flow(flow, store)
-DRIVE = discovery.build('drive', 'v2', http=creds.authorize(Http()))
+DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))
 
 FILES = (
-    ('hello.txt', False),
-    ('hello.txt', True),
+    ('hello.txt', None),
+    ('hello.txt', 'application/vnd.google-apps.document'),
 )
 
-for filename, convert in FILES:
-    metadata = {'title': filename}
-    res = DRIVE.files().insert(convert=convert, body=metadata,
-            media_body=filename, fields='mimeType,exportLinks').execute()
+for filename, mimeType in FILES:
+    metadata = {'name': filename}
+    if mimeType:
+        metadata['mimeType'] = mimeType
+    res = DRIVE.files().create(body=metadata, media_body=filename).execute()
     if res:
         print('Uploaded "%s" (%s)' % (filename, res['mimeType']))
 
 if res:
     MIMETYPE = 'application/pdf'
-    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
+    data = DRIVE.files().export(fileId=res['id'], mimeType=MIMETYPE).execute()
     if data:
         fn = '%s.pdf' % os.path.splitext(filename)[0]
         with open(fn, 'wb') as fh:

We'll start with the building of the service endpoint, with the trivial change of the API version string from 'v2' to 'v3':

-DRIVE = build('drive', 'v2', http=creds.authorize(Http()))
+DRIVE = build('drive', 'v3', http=creds.authorize(Http()))

The next change is the deprecation of the conversion flag. The problem with a Boolean variable is that it limits the possible types of file formats supported. By changing it to a file mimeType instead, the horizons are broadened:

 FILES = (
-    ('hello.txt', False),
-    ('hello.txt', True),
+    ('hello.txt', None),
+    ('hello.txt', 'application/vnd.google-apps.document'),
 )

Your next question will be: "What are the mimeTypes for the supported Google Apps document formats?" The answers can be found at this page in the official docs. This changes the datatype in our array of 2-tuples, so we need to change the loop variable to reflect this... we'll use the mimeType instead of a conversion flag:

-for filename, convert in FILES:
+for filename, mimeType in FILES:

Another change related to deprecating the convert flag is that the mimeType isn't a parameter to the API call. Instead, it's another piece of metadata, so we need to add mimeType to the metadata object.

Related to this is a name change: since a file's name is its name and not its title, it makes more sense to use "name" as the metadata value:

-    metadata = {'title': filename}
+    metadata = {'name': filename}
+    if mimeType:
+        metadata['mimeType'] = mimeType

Why the if statement? Not only did v3 see a change to using mimeTypes, but rather than being a parameter like the conversion flag in v2, the mimeType has been moved into the file's metadata, so if we're doing any conversion, we need to add it to our metadata field (then remove the convert parameter down below).

Next is yet another name change: when creating files on Google Drive, "create()" makes more sense as a method name than "insert()". Reducing the size of payload is another key ingredient of v3. We mentioned in the previous post that insert() returns more than 30 fields in the response payload unless you use the fields parameter to specify exactly which you wish returned. In v3, the default response payload only returns four fields, including all the ones we need in this script, so use of the fields parameter isn't required any more:

-    res = DRIVE.files().insert(convert=convert, body=metadata,
-            media_body=filename, fields='mimeType,exportLinks').execute()
+    res = DRIVE.files().create(body=metadata, media_body=filename).execute()

The final improvement we can demonstrate: users no longer have to make an authorized HTTP GET request with a link to export and download a file in an alternate format like PDF®. Instead, it's now a "normal" API call (to the new "export()" method) with the mimeType as a parameter. The only other parameter you need is the file ID, which comes back as part of the (default) response payload when the create() call was made:

-    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
+    data = DRIVE.files().export(fileId=res['id'], mimeType=MIMETYPE).execute()

That's it! If you run the script, grant the script access to your Google Drive (via the OAuth2 prompt that pops up in the browser), and then you should get output that looks like this:

$ python drive_updown3.py # or python3
Uploaded "hello.txt" (text/plain)
Uploaded "hello.txt" (application/vnd.google-apps.document)
Downloaded "hello.pdf" (application/pdf)

Conclusion

The entire v2 script (drive_updown2.py) was spelled out in full in the previous post, and it hasn't changed since then. Below is the v3 script (drive_updown3.py) for your convenience which runs on both Python 2 and Python 3 (unmodified!):

#!/usr/bin/env python

from __future__ import print_function
import os

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

FILES = (
    ('hello.txt', None),
    ('hello.txt', 'application/vnd.google-apps.document'),
)

for filename, mimeType in FILES:
    metadata = {'name': filename}
    if mimeType:
        metadata['mimeType'] = mimeType
    res = DRIVE.files().create(body=metadata, media_body=filename).execute()
    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

if res:
    MIMETYPE = 'application/pdf'
    data = DRIVE.files().export(fileId=res['id'], mimeType=MIMETYPE).execute()
    if data:
        fn = '%s.pdf' % os.path.splitext(filename)[0]
        with open(fn, 'wb') as fh:
            fh.write(data)
        print('Downloaded "%s" (%s)' % (fn, MIMETYPE))

Just as in the previous post(s), you can now customize this code for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs. Hope we accomplished our goal by pointing out some of the shortcomings that are in v2 and how they were improved in v3! All of the content in this and the previous post are spelled out visually in this video that I created for you.

Google Drive: Uploading & Downloading files with Python

2015-12-14T11:30:00.000-08:00

UPDATE: Since this post was published, the Google Drive team released a newer version of their API. After reading this one, go to the next post to learn about migrating your app from v2 to v3 as well as link to my video which walks through the code samples in both posts.

Introduction

So far in this series of blogposts covering authorized Google APIs, we've used Python to access Google Drive, Gmail, and Google Calendar. Today, we're revisiting Google Drive with a small snippet that uploads plain text files to Drive, with & without conversion to a Google Apps format (Google Docs), then exports & downloads the converted one as PDF®.

Earlier posts demonstrated the structure and "how-to" use Google APIs in general, so more recent posts, including this one, focus on solutions and apps, and use of specific APIs. Once you review the earlier material, you're ready to start with authorization scopes then see how to use the API itself.

Google Drive API Scopes

Google Drive features numerous API scopes of authorization. As usual, we always recommend you use the most restrictive scope possible that allows your app to do its work. You'll request fewer permissions from your users (which makes them happier), and it also makes your app more secure, possibly preventing modifying, destroying, or corrupting data, or perhaps inadvertently going over quotas. Since we need to upload/create files in Google Drive, the minimum scope we need is:

'https://www.googleapis.com/auth/drive' — Read/write access to Drive

Using the Google Drive API

Let's get going with our example today that uploads and downloads a simple plain text file to Drive. The file will be uploaded twice, once as-is, and the second time, converted to a Google Docs document. The last part of the script will request an export of the (uploaded) Google Doc as PDF and download that from Drive.

Since we've fully covered the authorization boilerplate fully in earlier posts and videos, we're going to skip that here and jump right to the action, creating of a service endpoint to Drive. The API name is (of course) 'drive', and the current version of the API is 2, so use the string 'v2' in this call to the apiclient.discovey.build() function:

DRIVE = build('drive', 'v2', http=creds.authorize(Http()))

Let's also create a FILES array object (tuple, list, etc.) which holds 2-tuples of the files to upload. These pairs are made up of a filename and a flag indicating whether or not you wish the file to be converted to a Google Apps format:

FILES = (
    ('hello.txt', False),
    ('hello.txt', True),
)

Since we're uploading a plain text file, a conversion to Apps format means Google Docs. (You can imagine that if it was a CSV file, the target format would be Google Sheets instead.) With the setup complete, let's move on to the code that performs the file uploads.

We'll loop through FILES, cycling through each file-convert flag pair and call the files.insert() method to perform the upload. The four parameters needed are: 1) the conversion flag, 2) the file metadata, which is only the filename (see below), 3) the media_body, which is also the filename but has a different purpose — it specifies where the file content will come from, meaning the file will be opened and its data transferred to the API, and 4), a set of fields you want returned.

for filename, convert in FILES:
    metadata = {'title': filename}
    res = DRIVE.files().insert(convert=convert, body=metadata,
            media_body=filename, fields='mimeType,exportLinks').execute()
    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

It's important to give the fields() parameter because if you don't, more than 30(!) are returned by default from the API. There's no need to waste all that network traffic if all you need are just a couple. In our case, we only want the mimeType, to confirm what the file was saved as, and exportLinks, which we'll explore in a moment. If files are uploaded successfully, the print() lets the user know, and then we move on to the final section of the script.

Before we dig into the last bit of code, it's important to realize that the res variable still contains the result from the second upload, the one where the file is converted to Google Docs. This is important because this is where we need to extract the download link for the format you want (res['exportLinks'][MIMETYPE]). The way to download the file is to make an authorized HTTP GET call, passing in that link. In our case, it's the PDF version. If the download is successful, the data variable will have the payload to write to disk. If all's good, let the user know:

if res:
    MIMETYPE = 'application/pdf'
    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
    if data:
        fn = '%s.pdf' % os.path.splitext(filename)[0]
        with open(fn, 'wb') as fh:
            fh.write(data)
        print('Downloaded "%s" (%s)' % (fn, MIMETYPE))

Final note: this code sample is slightly different from previous posts in two big ways: 1) now that the Google APIs Client Library runs on Python 3, I'll try to produce only code samples for this blog that run unmodified under both 2.x and 3.x interpreters — the primary one-line difference being the import of the print() function, and 2) we're going to incorporate the use of the run_flow() function from oauth2client.tools and only fallback to the deprecated run() function if necessary — more info on this change available in this earlier post.

If you run the script, grant the script access to your Google Drive (via the OAuth2 prompt that pops up in the browser), and then you should get output that looks like this:

$ python drive_updown3.py # or python3
Uploaded "hello.txt" (text/plain)
Uploaded "hello.txt" (application/vnd.google-apps.document)
Downloaded "hello.pdf" (application/pdf)

Conclusion

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

#!/usr/bin/env python

from __future__ import print_function
import os

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v2', http=creds.authorize(Http()))

FILES = (
    ('hello.txt', False),
    ('hello.txt', True),
)

for filename, convert in FILES:
    metadata = {'title': filename}
    res = DRIVE.files().insert(convert=convert, body=metadata,
            media_body=filename, fields='mimeType,exportLinks').execute()
    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

if res:
    MIMETYPE = 'application/pdf'
    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
    if data:
        fn = '%s.pdf' % os.path.splitext(filename)[0]
        with open(fn, 'wb') as fh:
            fh.write(data)
        print('Downloaded "%s" (%s)' % (fn, MIMETYPE))

You can now customize this code for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs. If you want to see another example of using the Drive API, check out this earlier post listing the files in Google Drive and its accompanying video as well as a similar example in the official docs or its equivalent in Java (server-side, Android), iOS (Objective-C, Swift), C#/.NET, PHP, Ruby, JavaScript (client-side, Node.js, Google Apps Script), or Go. That's it... hope you find these code samples useful in helping you get started with the Drive API!

UPDATE: Since this post was published, the Google Drive team released a newer version of their API. Go to the next post to learn about migrating your app from v2 to v3 as well as link to my video which walks through the code samples in both posts.

EXTRA CREDIT: Feel free to experiment and try something else to test your skills and challenge yourself as there's a lot more to Drive than just uploading and downloading files. Experiment with creating folders and manipulate files there, work with a folder of photos and organize them using the image metadata available to you, implement a search engine for your Drive files, etc. There are so many things you can do!

Creating events in Google Calendar from Python

2015-09-09T10:00:00.000-07:00

NOTE: The code covered in this blogpost is also available in a video walkthrough here.

UPDATE (Jan 2016): Tweaked the code to support oauth2client.tools.run_flow() which deprecates oauth2client.tools.run(). You can read more about that change and migration steps here.

Introduction

So far in this series of blogposts covering authorized Google APIs, we've used Python code to access Google Drive and Gmail. Today, we're going to demonstrate the Google Calendar API. While Google Calendar, and calendaring in general, have been around for a long time and are fairly stable, it's somewhat of a mystery as to why so few developers create good calendar integrations, whether it be with Google Calendar, or other systems. We'll try to show it isn't necessarily difficult and hopefully motivate some of you out there to add a calendaring feature in your next mobile or web app.

Earlier posts (link 1, link 2) demonstrated the structure and "how-to" use Google APIs in general, so more recent posts, including this one, focus on solutions and apps, and use of specific APIs. Once you review the earlier material, you're ready to start with authorization scopes then see how to use the API itself.

Google Calendar API Scopes

Below are the Google Calendar API scopes of authorization. There are only a pair (at the time of this writing): read-only and read/write. As usual, use the most restrictive scope you possibly can yet still allowing your app to do its work. This makes your app more secure and may prevent inadvertently going over any quotas, or accessing, destroying, or corrupting data. Also, users are less hesitant to install your app if it asks only for more restricted access to their calendars. However, it's likely that in order to really use the API to its fullest, you will probably have to ask for read-write so that you can add, update, or delete events in their calendars.

'https://www.googleapis.com/auth/calendar.readonly' — Read-only access to calendar
'https://www.googleapis.com/auth/calendar' — Read/write access to calendar

Using the Google Calendar API

We're going to create a sample Python script that inserts a new event into your Google Calendar. Since this requires modifying your calendar, you need the read/write scope above. The API name is 'calendar' which is currently on version 3, so here's the call to apiclient.discovery.build() you'll use:

GCAL = discovery.build('calendar', 'v3',
    http=creds.authorize(Http()))

Note that all lines of code above that is predominantly boilerplate (that was explained in earlier posts and videos). Anyway, we've got an established service endpoint with build(), we need to come up with the data to create a calendar event with, at the very least, an event name plus start and end times.

Timezone or offset required

The API requires either a timezone or a GMT offset, the number of hours your timezone is away from Coordinated Universal Time (UTC, more commonly known as GMT). The format is +/-HH:MM away from UTC. For example, Pacific Daylight Time (PDT, also known as Mountain Standard Time, or MST), is "-07:00," or seven hours behind UTC while Nepal Standard Time (NST [or NPT to avoid confusion with Newfoundland Standard Time]), is "+05:45," or five hours and forty-five minutes ahead of UTC. Also, the offset must be in RFC 3339 format, which implements the specifications of ISO 8601 for the Internet. Timestamps look like the following in the required format: "YYYY-MM-DDTHH:MM:SS±HH:MM". For example, September 15, 2015 at 7 PM PDT is represented by this string: "2015-09-15T19:00:00-07:00".

If you wish to avoid offsets and would rather use timezone names instead, see the next post in this series (link at bottom).

The script in this post uses the PDT timezone, so we set the GMT_OFF variable to "-07:00". The EVENT body will hold the event name, and start and end times suffixed with the GMT offset:

GMT_OFF = '-07:00'    # PDT/MST/GMT-7
EVENT = {
    'summary': 'Dinner with friends',
    'start':   {'dateTime': '2015-09-15T19:00:00%s' % GMT_OFF},
    'end':     {'dateTime': '2015-09-15T22:00:00%s' % GMT_OFF},
}

Use the insert() method of the events() service to add the event. As expected, one required parameter is the ID of the calendar to insert the event into. A special value of 'primary' has been set aside for the currently authenticated user. The other required parameter is the event body. In our request, we also ask the Calendar API to send email notifications to the guests, and that's done by passing in the sendNotifications flag with a True value. Our call to the API looks like this:

e = GCAL.events().insert(calendarId='primary',
    sendNotifications=True, body=EVENT).execute()

The one remaining thing is to confirm that the calendar event was created successfully. We do that by checking the return value — it should be an Event object with all the details we passed in a moment ago:

print('''*** %r event added:
    Start: %s
    End:   %s''' % (e['summary'].encode('utf-8'),
        e['start']['dateTime'], e['end']['dateTime']))

Now, if you really want some proof the event was created, one of the fields that's created is a link to the calendar event. We don't use it in the code, but you can... just use e['htmlLink'].

Regardless, that's pretty much the entire script save for the OAuth2 code that we're so familiar with from previous posts. The script is posted below in its entirety, and if you run it, depending on the date/times you use, you'll see something like this:

$ python gcal_insert.py
*** 'Dinner with friends' event added:
    Start: 2015-09-15T19:00:00-07:00
    End:   2015-09-15T22:00:00-07:00

It also works with Python 3 with one slight nit/difference being the "b" prefix on from the event name due to converting from Unicode to bytes:

*** b'Dinner with friends' event added:

Conclusion

There can be much more to adding a calendar event, such as events that repeat with a recurrence rule, the ability to add attachments for an event, such as a party invitation or a PDF of the show tickets. For more on what you can do when creating events, take a look at the docs for events().insert() as well as the corresponding developer guide. All of the docs for the Google Calendar API can be found here. Also be sure to check out the companion video for this code sample. That's it!

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

from __future__ import print_function
from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/calendar'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
GCAL = discovery.build('calendar', 'v3', http=creds.authorize(Http()))

GMT_OFF = '-07:00'      # PDT/MST/GMT-7
EVENT = {
    'summary': 'Dinner with friends',
    'start':  {'dateTime': '2015-09-15T19:00:00%s' % GMT_OFF},
    'end':    {'dateTime': '2015-09-15T22:00:00%s' % GMT_OFF},
    'attendees': [
        {'email': 'friend1@example.com'},
        {'email': 'friend2@example.com'},
    ],
}

e = GCAL.events().insert(calendarId='primary',
        sendNotifications=True, body=EVENT).execute()

print('''*** %r event added:
    Start: %s
    End:   %s''' % (e['summary'].encode('utf-8'),
        e['start']['dateTime'], e['end']['dateTime']))

You can now customize this code for your own needs, for a mobile frontend, a server-side backend, or to access other Google APIs. If you want to see another example of using the Calendar API (listing the next 10 events in your calendar), check out the Python Quickstart example or its equivalent in Java (server-side, Android), iOS (Objective-C, Swift), C#/.NET, PHP, Ruby, JavaScript (client-side, Node.js), or Go. That's it... hope you find these code samples useful in helping you get started with the Calendar API!

Code challenge

To test your skills and challenge yourself, try creating recurring events (such as when you expect to receive your paycheck), events with attachments, or perhaps editing existing events. UPDATE (Jul 2017): If you're ready for the next step, we cover the first and last of those choices in our follow-up post.

Accessing Gmail from Python (plus BONUS)

2015-08-06T11:08:00.001-07:00

NOTE: The code covered in this blogpost is also available in a video walkthrough here.

UPDATE (Aug 2016): The code has been modernized to use oauth2client.tools.run_flow() instead of the deprecated oauth2client.tools.run(). You can read more about that change here.

Introduction

The last several posts have illustrated how to connect to public/simple and authorized Google APIs. Today, we're going to demonstrate accessing the Gmail (another authorized) API. Yes, you read that correctly... "API." In the old days, you access mail services with standard Internet protocols such as IMAP/POP and SMTP. However, while they are standards, they haven't kept up with modern day email usage and developers' needs that go along with it. In comes the Gmail API which provides CRUD access to email threads and drafts along with messages, search queries, management of labels (like folders), and domain administration features that are an extra concern for enterprise developers.

Earlier posts demonstrate the structure and "how-to" use Google APIs in general, so the most recent posts, including this one, focus on solutions and apps, and use of specific APIs. Once you review the earlier material, you're ready to start with Gmail scopes then see how to use the API itself.

Gmail API Scopes

Below are the Gmail API scopes of authorization. We're listing them in most-to-least restrictive order because that's the order you should consider using them in — use the most restrictive scope you possibly can yet still allowing your app to do its work. This makes your app more secure and may prevent inadvertently going over any quotas, or accessing, destroying, or corrupting data. Also, users are less hesitant to install your app if it asks only for more restricted access to their inboxes.

'https://www.googleapis.com/auth/gmail.readonly' — Read-only access to all resources + metadata
'https://www.googleapis.com/auth/gmail.send' — Send messages only (no inbox read nor modify)
'https://www.googleapis.com/auth/gmail.labels' — Create, read, update, and delete labels only
'https://www.googleapis.com/auth/gmail.insert' — Insert and import messages only
'https://www.googleapis.com/auth/gmail.compose' — Create, read, update, delete, and send email drafts and messages
'https://www.googleapis.com/auth/gmail.modify' — All read/write operations except for immediate & permanent deletion of threads & messages
'https://mail.google.com/' — All read/write operations (use with caution)

Using the Gmail API

We're going to create a sample Python script that goes through your Gmail threads and looks for those which have more than 2 messages, for example, if you're seeking particularly chatty threads on mailing lists you're subscribed to. Since we're only peeking at inbox content, the only scope we'll request is 'gmail.readonly', the most restrictive scope. The API string is 'gmail' which is currently on version 1, so here's the call to apiclient.discovery.build() you'll use:

GMAIL = discovery.build('gmail', 'v1', http=creds.authorize(Http()))

Note that all lines of code above that is predominantly boilerplate (that was explained in earlier posts). Anyway, once you have an established service endpoint with build(), you can use the list() method of the threads service to request the file data. The one required parameter is the user's Gmail address. A special value of 'me' has been set aside for the currently authenticated user.

threads = GMAIL.users().threads().list(userId='me').execute().get('threads', [])

If all goes well, the (JSON) response payload will (not be empty or missing and) contain a sequence of threads that we can loop over. For each thread, we need to fetch more info, so we issue a second API call for that. Specifically, we care about the number of messages in a thread:

for thread in threads:
    tdata = GMAIL.users().threads().get(userId='me', id=thread['id']).execute()
    nmsgs = len(tdata['messages'])

We're seeking only all threads more than 2 (that means at least 3) messages, discarding the rest. If a thread meets that criteria, scan the first message and cycle through the email headers looking for the "Subject" line to display to users, skipping the remaining headers as soon as we find one:

    if nmsgs > 2:
        msg = tdata['messages'][0]['payload']
        subject = ''
        for header in msg['headers']:
            if header['name'] == 'Subject':
                subject = header['value']
                break
        if subject:
            print('%s (%d msgs)' % (subject, nmsgs))

If you're on many mailing lists, this may give you more messages than desired, so feel free to up the threshold from 2 to 50, 100, or whatever makes sense for you. (In that case, you should use a variable.) Regardless, that's pretty much the entire script save for the OAuth2 code that we're so familiar with from previous posts. The script is posted below in its entirety, and if you run it, you'll see an interesting collection of threads... YMMV depending on what messages are in your inbox:

$ python3 gmail_threads.py
[Tutor] About Python Module to Process Bytes (3 msgs)
Core Python book review update (30 msgs)
[Tutor] scratching my head (16 msgs)
[Tutor] for loop for long numbers (10 msgs)
[Tutor] How to show the listbox from sqlite and make it searchable? (4 msgs)
[Tutor] find pickle and retrieve saved data (3 msgs)

BONUS: Python 3!

As of Mar 2015 (formally in Apr 2015 when the docs were updated), support for Python 3 was added to Google APIs Client Library (3.3+)! This update was a long time coming (relevant GitHub thread), and allows Python 3 developers to write code that accesses Google APIs. If you're already running 3.x, you can use its pip command (pip3) to install the Client Library:

$ pip3 install -U google-api-python-client

Because of this, unlike previous blogposts, we're deliberately going to avoid use of the print statement and switch to the print() function instead. If you're still running Python 2, be sure to add the following import so that the code will also run in your 2.x interpreter:

from __future__ import print_function

Conclusion

To find out more about the input parameters as well as all the fields that are in the response, take a look at the docs for threads().list(). For more information on what other operations you can execute with the Gmail API, take a look at the reference docs and check out the companion video for this code sample. That's it!

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

from __future__ import print_function

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/gmail.readonly'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
GMAIL = discovery.build('gmail', 'v1', http=creds.authorize(Http()))

threads = GMAIL.users().threads().list(userId='me').execute().get('threads', [])
for thread in threads:
    tdata = GMAIL.users().threads().get(userId='me', id=thread['id']).execute()
    nmsgs = len(tdata['messages'])

    if nmsgs > 2:
        msg = tdata['messages'][0]['payload']
        subject = ''
        for header in msg['headers']:
            if header['name'] == 'Subject':
                subject = header['value']
                break
        if subject:
            print('%s (%d msgs)' % (subject, nmsgs))

You can now customize this code for your own needs, for a mobile frontend, a server-side backend, or to access other Google APIs. If you want to see another example of using the Gmail API (displaying all your inbox labels), check out the Python Quickstart example in the official docs or its equivalent in Java (server-side, Android), iOS (Objective-C, Swift), C#/.NET, PHP, Ruby, JavaScript (client-side, Node.js), or Go. That's it... hope you find these code samples useful in helping you get started with the Gmail API!

EXTRA CREDIT: To test your skills and challenge yourself, try writing code that allows users to perform a search across their email, or perhaps creating an email draft, adding attachments, then sending them! Note that to prevent spam, there are strict Program Policies that you must abide with... any abuse could rate limit your account or get it shut down. Check out those rules plus other Gmail terms of use here.

Google APIs: migrating from tools.run() to tools.run_flow()

2015-04-07T00:26:00.002-07:00

Got AttributeError? As in: AttributeError: 'module' object has no attribute 'run'? Rename run() to run_flow(), and you'll be good-to-go. TL;DR: This mini-tutorial slash migration guide slash PSA (public service announcement) is aimed at Python developers using the Google APIs Client Library (to access Google APIs from their applications) currently calling oauth2client.tools.run() and likely getting an exception (see Jan 2016 update below), and need to oauth2client.tools.run_flow(), its replacement.

UPDATE (Aug 2016): The flags parameter in run_flow() function became optional in Feb 2016, so tweaked the blogpost to reflect that.

UPDATE (Jun 2016): Revised the code and cleaned up the dialog so there are no longer any instances of using run() function, significantly shortening this post.

UPDATE (Jan 2016): The tools.run() function itself was forcibly removed (without a fallback) in Aug 2015, so if you're using any release on or after that, any such calls from your code will throw an exception (AttributeError: 'module' object has no attribute 'run'). To fix this problem, continue reading.

Prelude

We're going to continue our look at accessing Google APIs from Python. In addition to the previous pair of posts (http://goo.gl/57Gufk and http://goo.gl/cdm3kZ), as part of my day job, I've been working on corresponding video content, some of which are tied specifically to posts on this blog.

In this follow-up, we're going to specifically address the sidebar in the previous post, where we bookmarked an item for future discussion where the future is now: in the oauth2client package, tools.run() has been deprecated by tools.run_flow(). Note you need at least Python 2.7 or 3.3 to use the Google APIs Client Library. (If you didn't even know Python 3 was supported at all, then you need to see this post and this Quora Q&A.)

Replacing tools.run() with tools.run_flow()

Now let's convert the authorized access to Google APIs code from using tools.run() to tools.run_flow(). Here is the old snippet I'm talking about that needs upgrading:

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = # one or more scopes (str or iterable)
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run(flow, store)

SERVICE = discovery.build(API, VERSION, http=creds.authorize(Http()))

If you're using the latest Client Library (as of Feb 2016), all you need to do is change the tools.run() call to tools.run_flow(), as italicized below. Everything else stays exactly the same:

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = # one or more scopes (str or iterable)
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)

If you don't have the latest Client Library, then your update involves the extra steps of adding lines that import argparse and using it to get the flags argument needed by tools.run_flow() plus the actual change from tools.run(); all updates italicized below:

import argparse

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = # one or more scopes (str or iterable)
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flags = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
    flow = client.flow_from_clientsecrets('client_id.json', SCOPES)
    creds = tools.run_flow(flow, store, flags)

SERVICE = discovery.build(API, VERSION, http=creds.authorize(Http()))

Command-line argument processing, or "Why argparse?"

Python has had several modules in the Standard Library that allow developers to process command-line arguments. The original one was getopt which mirrored the getopt() function from C. In Python 2.3, optparse was introduced, featuring more powerful processing capabilities. However, it was deprecated in 2.7 in favor of a similar module, argparse. (To find out more about their similarities, differences and rationale behind developing argparse , see PEP 389 and this argparse docs page.) For the purposes of using Google APIs, you're all set if using Python 2.7 as it's included in the Standard Library. Otherwise Python 2.3-2.6 users can install it with: "pip install -U argparse".

Irregardless of whether you need argparse, once you migrate to either snippet with tools.run_flow(), your application should go back to working the way it had before.

Authorized Google API access from Python (part 2 of 2)

2014-11-06T17:07:00.000-08:00

Listing your files with the Google Drive API

NOTE: You can also watch a video walkthrough of the common code covered in this blogpost here.

UPDATE (Mar 2020): You can build this application line-by-line with our codelab (self-paced, hands-on tutorial) introducing developers to G Suite APIs. The deprecated auth library comment from the previous update below is spelled out in more detail in the green sidebar towards the bottom of step/module 5 (Install the Google APIs Client Library for Python). Also, the code sample is now maintained in a GitHub repo which includes a port to the newer auth libraries so you have both versions to refer to.

UPDATE (Apr 2019): In order to have a closer relationship between the GCP and G Suite worlds of Google Cloud, all G Suite Python code samples have been updated, replacing some of the older G Suite API client libraries with their equivalents from GCP. NOTE: using the newer libraries requires more initial code/effort from the developer thus will seem "less Pythonic." However, we will leave the code sample here with the original client libraries (deprecated but not shutdown yet) to be consistent with the video.

UPDATE (Aug 2016): The code has been modernized to use oauth2client.tools.run_flow() instead of the deprecated oauth2client.tools.run_flow(). You can read more about that change here.

UPDATE (Jun 2016): Updated to Python 2.7 & 3.3+ and Drive API v3.

Introduction

In this final installment of a (currently) two-part series introducing Python developers to building on Google APIs, we'll extend from the simple API example from the first post (part 1) just over a month ago. Those first snippets showed some skeleton code and a short real working sample that demonstrate accessing a public (Google) API with an API key (that queried public Google+ posts). An API key however, does not grant applications access to authorized data.

Authorized data, including user information such as personal files on Google Drive and YouTube playlists, require additional security steps before access is granted. Sharing of and hardcoding credentials such as usernames and passwords is not only insecure, it's also a thing of the past. A more modern approach leverages token exchange, authenticated API calls, and standards such as OAuth2.

In this post, we'll demonstrate how to use Python to access authorized Google APIs using OAuth2, specifically listing the files (and folders) in your Google Drive. In order to better understand the example, we strongly recommend you check out the OAuth2 guides (general OAuth2 info, OAuth2 as it relates to Python and its client library) in the documentation to get started.

The docs describe the OAuth2 flow: making a request for authorized access, having the user grant access to your app, and obtaining a(n access) token with which to sign and make authorized API calls with. The steps you need to take to get started begin nearly the same way as for simple API access. The process diverges when you arrive on the Credentials page when following the steps below.

Google API access

In order to Google API authorized access, follow these instructions (the first three of which are roughly the same for simple API access):

Go to the Google Developers Console and login.

Use your Gmail or Google credentials; create an account if needed

Click "Create a Project" from pulldown under your username (at top)

Enter a Project Name (mutable, human-friendly string only used in the console)
Enter a Project ID (immutable, must be unique and not already taken)

Once project has been created, enable APIs you wish to use

You can toggle on any API(s) that support(s) simple or authorized API access.
For the code example below, we use the Google Drive API.
Other ideas: YouTube Data API, Google Sheets API, etc.
Find more APIs (and version#s which you need) at the OAuth Playground.

Select "Credentials" in left-nav

Click "Create credentials" and select OAuth client ID
In the new dialog, select your application type — we're building a command-line script which is an "Installed application"
In the bottom part of that same dialog, specify the type of installed application; choose "Other" (cmd-line scripts are not web nor mobile)
Click "Create Client ID" to generate your credentials

Finally, click "Download JSON" to save the new credentials to your computer... perhaps choose a shorter name like "client_secret.json" or "client_id.json"

NOTEs: Instructions from the previous blogpost were to get an API key. This time, in the steps above, we're creating and downloading OAuth2 credentials. You can also watch a video walkthrough of this app setup process of getting simple or authorized access credentials in the "DevConsole" here.

Accessing Google APIs from Python

In order to access authorized Google APIs from Python, you still need the Google APIs Client Library for Python, so in this case, do follow those installation instructions from part 1.

We will again use googleapiclient.discovery.build(), which is required to create a service endpoint for interacting with an API, authorized or otherwise. However, for authorized data access, we need additional resources, namely the httplib2 and oauth2client packages. Here are the first five lines of the new boilerplate code for authorized access:

from __future__ import print_function

from googleapiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = # one or more scopes (strings)

SCOPES is a critical variable: it represents the set of scopes of authorization an app wants to obtain (then access) on behalf of user(s). What's does a scope look like?

Each scope is a single character string, specifically a URL. Here are some examples:

'https://www.googleapis.com/auth/plus.me' — access your personal Google+ settings
'https://www.googleapis.com/auth/drive.metadata.readonly' — read-only access your Google Drive file or folder metadata
'https://www.googleapis.com/auth/youtube' — access your YouTube playlists and other personal information

You can request one or more scopes, given as a single space-delimited string of scopes or an iterable (list, generator expression, etc.) of strings. If you were writing an app that accesses both your YouTube playlists as well as your Google+ profile information, your SCOPES variable could be either of the following:
SCOPES = 'https://www.googleapis.com/auth/plus.me https://www.googleapis.com/auth/youtube'

That is space-delimited and made tiny by me so it doesn't wrap in a regular-sized browser window; or it could be an easier-to-read, non-tiny, and non-wrapped tuple:

SCOPES = (
'https://www.googleapis.com/auth/plus.me',
'https://www.googleapis.com/auth/youtube',
)

Our example command-line script will just list the files on your Google Drive, so we only need the read-only Drive metadata scope, meaning our SCOPES variable will be just this:
SCOPES = 'https://www.googleapis.com/auth/drive.metadata.readonly'
The next section of boilerplate represents the security code:

store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)

Once the user has authorized access to their personal data by your app, a special "access token" is given to your app. This precious resource must be stored somewhere local for the app to use. In our case, we'll store it in a file called "storage.json". The lines setting the store and creds variables are attempting to get a valid access token with which to make an authorized API call.

If the credentials are missing or invalid, such as being expired, the authorization flow (using the client secret you downloaded along with a set of requested scopes) must be created (by client.flow_from_clientsecrets()) and executed (by tools.run_flow()) to ensure possession of valid credentials. The client_secret.json file is the credentials file you saved when you clicked "Download JSON" from the DevConsole after you've created your OAuth2 client ID.

If you don't have credentials at all, the user much explicitly grant permission — I'm sure you've all seen the OAuth2 dialog describing the type of access an app is requesting (remember those scopes?). Once the user clicks "Accept" to grant permission, a valid access token is returned and saved into the storage file (because you passed a handle to it when you called tools.run_flow()).

Note: tools.run() deprecated by tools.run_flow()
You may have seen usage of the older tools.run() function, but it has been deprecated by tools.run_flow(). We explain this in more detail in another blogpost specifically geared towards migration.

Once the user grants access and valid credentials are saved, you can create one or more endpoints to the secure service(s) desired with googleapiclient.discovery.build(), just like with simple API access. Its call will look slightly different, mainly that you need to sign your HTTP requests with your credentials rather than passing an API key:

DRIVE = discovery.build(API, VERSION, http=creds.authorize(Http()))

In our example, we're going to list your files and folders in your Google Drive, so for API, use the string 'drive'. The API is currently on version 3 so use 'v3' for VERSION:

DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

If you want to get comfortable with OAuth2, what it's flow is and how it works, we recommend that you experiment at the OAuth Playground. There you can choose from any number of APIs to access and experience first-hand how your app must be authorized to access personal data.

Going back to our working example, once you have an established service endpoint, you can use the list() method of the files service to request the file data:

files = DRIVE.files().list().execute().get('files', [])

If there's any data to read, the response dict will contain an iterable of files that we can loop over (or default to an empty list so the loop doesn't fail), displaying file names and types:

for f in files:
print(f['name'], f['mimeType'])

Conclusion

To find out more about the input parameters as well as all the fields that are in the response, take a look at the docs for files().list(). For more information on what other operations you can execute with the Google Drive API, take a look at the reference docs and check out the companion video for this code sample. Don't forget the codelab and this sample's GitHub repo. That's it!

Below is the entire script for your convenience:

'''
drive_list.py -- Google Drive API demo; maintained at:
    http://github.com/googlecodelabs/gsuite-apis-intro
'''
from __future__ import print_function

from googleapiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive.readonly.metadata'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)

DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))
files = DRIVE.files().list().execute().get('files', [])
for f in files:
    print(f['name'], f['mimeType'])

When you run it, you should see pretty much what you'd expect, a list of file or folder names followed by their MIMEtypes — I named my script drive_list.py:

$ python3 drive_list.py
Google Maps demo application/vnd.google-apps.spreadsheet
Overview of Google APIs - Sep 2014 application/vnd.google-apps.presentation
tiresResearch.xls application/vnd.google-apps.spreadsheet
6451_Core_Python_Schedule.doc application/vnd.google-apps.document
out1.txt application/vnd.google-apps.document
tiresResearch.xls application/vnd.ms-excel
6451_Core_Python_Schedule.doc application/msword
out1.txt text/plain
Maps and Sheets demo application/vnd.google-apps.spreadsheet
ProtoRPC Getting Started Guide application/vnd.google-apps.document
gtaskqueue-1.0.2_public.tar.gz application/x-gzip
Pull Queues application/vnd.google-apps.folder
gtaskqueue-1.0.1_public.tar.gz application/x-gzip
appengine-java-sdk.zip application/zip
taskqueue.py text/x-python-script
Google Apps Security Whitepaper 06/10/2010.pdf application/pdf

Obviously your output will be different, depending on what files are in your Google Drive. But that's it... hope this is useful. You can now customize this code for your own needs and/or to access other Google APIs. Thanks for reading!

EXTRA CREDIT: To test your skills, add functionality to this code that also displays the last modified timestamp, the file (byte)size, and perhaps shave the MIMEtype a bit as it's slightly harder to read in its entirety... perhaps take just the final path element? One last challenge: in the output above, we have both Microsoft Office documents as well as their auto-converted versions for Google Apps... perhaps only show the filename once and have a double-entry for the filetypes!

Simple Google API access from Python (part 1 of 2)

2014-09-20T00:35:00.005-07:00

NOTE: You can also watch a video walkthrough of the common code covered in this blogpost here.

UPDATE (Aug 2016): The code has been modernized to recognize that the Client Library is available for Python 2 or 3.

Introduction

Back in 2012 when I published Core Python Applications Programming, 3rd ed., I
posted about how I integrated Google technologies into the book. The only problem is that I presented very specific code for Google App Engine and Google+ only. I didn't show a generic way how, using pretty much the same boilerplate Python snippet, you can access any number of Google APIs; so here we are.

In this multi-part series, I'll break down the code that allows you to leverage Google APIs to the most basic level (even for Python), so you can customize as necessary for your app, whether it's running as a command-line tool or something server-side in the cloud backending Web or mobile clients. If you've got the book and played around with our Google+ API example, you'll find this code familiar, if not identical — I'll go into more detail here, highlighting the common code for generic API access and then bring in the G+-relevant code later.

We'll start in this first post by demonstrating how to access public or unauthorized data from Google APIs. (The next post will illustrate how to access authorized data from Google APIs.) Regardless of which you use, the corresponding boilerplate code stands alone. In fact, it's probably best if you saved these generic snippets in a library module so you can (re)use the same bits for any number of apps which access any number of modern Google APIs.

Google API access

In order to access Google APIs, follow these instructions:

Go to the Google Developers Console and login.

Use your Gmail or Google credentials; create an account if needed

Click "Create Project" button

Enter a Project Name (mutable, human-friendly string only used in the console)
Enter a Project ID (immutable, must be unique and not already taken)

Once project has been created, click "Enable an API" button

You can toggle on any API(s) that support(s) simple API access (not authorized).
For the code example below, we use the Google+ API.
Other ideas: YouTube Data API, Google Maps API, etc.
Find more APIs (and version#s which you need) at the OAuth Playground.

Select "Credentials" in left-nav under "APIs & auth"

Go to bottom half and click "Create new Key" button
Grab long "API KEY" cryptic string and save to Python script

NOTE: You can also watch a video walkthrough of this app setup process in the "DevConsole" here.

Accessing Google APIs from Python

Now that you're set up, everything else is done on the Python side. To talk to a Google API, you need the Google APIs Client Library for Python, specifically the apiclient.discovery.build() function. Download and install the library in your usual way, for example:

$ pip install -U google-api-python-client # or pip3 for 3.x

NOTE: If you're building a Python App Engine app, you'll need something else, the Google APIs Client Library for Python on Google App Engine. It's similar but has extra goodies (specifically decorators — brief generic intro to those in my previous post) just for cloud developers that must be installed elsewhere. As App Engine developers know, libraries must be in the same location on the filesystem as your source code.

Once everything is installed, make sure that you can import apiclient.discovery:

$ python
Python 2.7.6 (default, Apr 9 2014, 11:48:52)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import apiclient.discovery
>>>

In discovery.py is the build() function, which is what we need to create a service endpoint for interacting with an API. Now craft the following lines of code in your command-line tool, using the shorthand from-import statement instead:

from apiclient import discovery

API_KEY = # copied from project credentials page
SERVICE = discovery.build(API, VERSION, developerKey=API_KEY)

Take the API key you copied from the credentials page and assign to the API_KEY variable as a string. Obviously, embedding an API key in source code isn't something you'd so in practice as it's not secure whatsoever — stick it in a database, key broker, encrypt, or at least have it in a separate byte code (.pyc/.pyo) file that you import — but we'll allow it now solely for illustrative purposes of a simple command-line script.

In our short example we're going to do a simple search for "python" in public Google+ posts, so for the API variable, use the string 'plus'. The API version is currently on version 1 (at the time of this writing), so use 'v1' for VERSION. (Each API will use a different name and version string... again, you can find those in the OAuth Playground or in the docs for the specific API you want to use.) Here's the call once we've filled in those variables:

GPLUS = discovery.build('plus', 'v1', developerKey=API_KEY)

We need a template for the results that come back. There are many fields in a Google+ post, so we're only going to pick three to display... the user name, post timestamp, and a snippet of the post itself:

TMPL = '''
User: %s
Date: %s
Post: %s
'''

Now for the code. Google+ posts are activities (known as "notes;" there are other activities as well). One of the methods you have access to is search(), which lets you query public activities; so that's what we're going to use. Add the following call using the GPLUS service endpoint you already created using the verbs we just described and execute it:

items = GPLUS.activities().search(query='python').execute().get('items', [])

If all goes well, the (JSON) response payload will contain a set of 'items' (else we assign an empty list for the for loop). From there, we'll loop through each matching post, do some minor string manipulation to replace all whitespace characters (including NEWLINEs [ \n ]) with spaces, and display if not blank:

for data in items:
post = ' '.join(data['title'].strip().split())
if post:
print(TMPL % (data['actor']['displayName'],
data['published'], post))

Conclusion

To find out more about the input parameters as well as all the fields that are in the response, take a look at the docs. Below is the entire script missing only the API_KEY which you'll have to fill in yourself.

from __future__ import print_function
from apiclient import discovery

TMPL = '''
    User: %s
    Date: %s
    Post: %s
'''

API_KEY = # copied from project credentials page
GPLUS = discovery.build('plus', 'v1', developerKey=API_KEY)
items = GPLUS.activities().search(query='python').execute().get('items', [])
for data in items:
    post = ' '.join(data['title'].strip().split())
    if post:
        print(TMPL % (data['actor']['displayName'],
                      data['published'], post))

When you run it, you should see pretty much what you'd expect, a few posts on Python, some on Monty Python, and of course, some on the snake — I called my script plus_search.py:

$ python plus_search.py # or python3

    User: Jeff Ward
    Date: 2014-09-20T18:08:23.058Z
    Post: How to make python accessible in the command window.


    User: Fayland Lam
    Date: 2014-09-20T16:40:11.512Z
    Post: Data Engineer http://findmjob.com/job/AB7ZKitA5BGYyW1oAlQ0Fw/Data-Engineer.html #python #hadoop #jobs...


    User: Willy's Emporium LTD
    Date: 2014-09-20T16:19:33.851Z
    Post: MONTY PYTHON QUOTES MUG Take a swig to wash down all that albatross and crunchy frog. Featuring 20 ...


    User: Doddy Pal
    Date: 2014-09-20T15:49:54.405Z
    Post: Classic Monty Python!!!


    User: Sebastian Huskins
    Date: 2014-09-20T15:33:00.707Z
    Post: Made a small python script to get shellcode out of an executable. I found a nice commandlinefu.com oneline...

EXTRA CREDIT: To test your skills, check the docs and add a fourth line to each output which is the URL/link to that specific post, so that you (and your users) can open a browser to it if of interest.

If you want to build on from here, check out the larger app using the Google+ API featured in Chapter 15 of the book — it adds some brains to this basic code where the Google+ posts are sorted by popularity using a "chatter" score. That just about wraps it up this post. Once you're good to go, then you're ready to learn how to perform authorized Google API access in part 2 of this two-part series!

Introduction to Python decorators

2014-07-26T21:28:00.002-07:00

In this post, we're going to give you a user-friendly introduction to Python decorators. (The code works on both Python 2 [2.6 or 2.7 only] and 3 so don't be concerned with your version.) Before jumping into the topic du jour, consider the usefulness of the map() function. You've got a list with some data and want to apply some function [like times2() below] to all its elements and get a new list with the modified data:

def times2(x):
return x * 2

>>> list(map(times2, [0, 1, 2, 3, 4]))
[0, 2, 4, 6, 8]

Yeah yeah, I know that you can do the same thing with a list comprehension or generator expression, but my point was about an independent piece of logic [like times2()] and mapping that function across a data set ([0, 1, 2, 3, 4]) to generate a new data set ([0, 2, 4, 6, 8]). However, since mapping functions like times2()aren't tied to any particular chunk of data, you can reuse them elsewhere with other unrelated (or related) data.

Along similar lines, consider function calls. You have independent functions and methods in classes. Now, think about "mapped" execution across functions. What are things that you can do with functions that don't have much to do with the behavior of the functions themselves? How about logging function calls, timing them, or some other introspective, cross-cutting behavior. Sure you can implement that behavior in each of the functions that you care about such information, however since they're so generic, it would be nice to only write that logging code just once.

Introduced in 2.4, decorators modularize cross-cutting behavior so that developers don't have to implement near duplicates of the same piece of code for each function. Rather, Python gives them the ability to put that logic in one place and use decorators with its at-sign ("@") syntax to "map" that behavior to any function (or method). This compartmentalization of cross-cutting functionality gives Python an aspect-oriented programming flavor.

How do you do this in Python? Let's take a look at a simple example, the logging of function calls. Create a decorator function that takes a function object as its sole argument, and implement the cross-cutting functionality. In logged() below, we're just going to log function calls by making a call to the print() function each time a logged function is called.

def logged(_func):

def _wrapped():
print('Function %r called at: %s' % (

_func.__name__, ctime()))

return _func()
return _wrapped

In logged(), we use the function's name (given by func.__name__) plus a timestamp from time.ctime() to build our output string. Make sure you get the right imports, time.ctime() for sure, and if using Python 2, the print() function:

from __future__ import print_function # 2-3 compatibility
from time import ctime

Now that we have our logged() decorator, how do we use it? On the line above the function which you want to apply the decorator to, place an at-sign in front of the decorator name. That's followed immediately on the next line with the normal function declaration. Here's what it looks like, applied to a boring generic foo() function which just print()s it's been called.

@logged

def foo():

print('foo() called')

When you call foo(), you can see that the decorator logged() is called first, which then calls foo() on your behalf:

$ log_func.py

Function 'foo' called at: Sun Jul 27 04:09:37 2014

foo() called

If you take a closer look at logged() above, the way the decorator works is that the decorated function is "wrapped" so that it is passed as func to the decorator then the newly-wrapped function _wrapped()is (re)assigned as foo(). That's why it now behaves the way it does when you call it.

The entire script:

#!/usr/bin/env python
'log_func.py -- demo of decorators'

from __future__ import print_function # 2-3 compatibility
from time import ctime

def logged(_func):
    def _wrapped():
        print('Function %r called at: %s' % (
              _func.__name__, ctime()))
        return _func()
    return _wrapped

@logged
def foo():
    print('foo() called')

foo()

That was just a simple example to give you an idea of what decorators are. If you dig a little deeper, you'll discover one caveat is that the wrapping isn't perfect. For example, the attributes of foo() are lost, i.e., its name and docstring. If you ask for either, you'll get _wrapped()'s info instead:

>>> print("My name:", foo.__name__) # should be 'foo'!
My name: _wrapped
>>> print("Docstring:", foo.__doc__) # _wrapped's docstring!
Docstring: None

In reality, the "@" syntax is just a shortcut. Here's what you really did, which should explain this behavior:

def foo():

print('foo() called')

foo = logged(foo) # returns _wrapped (and its attributes)

So as you can tell, it's not a complete wrap. A convenience function that ties up these loose ends is functools.wraps(). If you use it and run the same code, you will get foo()'s info. However, if you're not going to use a function's attributes while it's wrapped, it's less important to do this.

There's also support for additional features, such calling decorated functions with parameters, applying more complex decorators, applying multiple levels of decorators, and also class decorators. You can find out more about (function and method) decorators in Chapter 11 of Core Python Programming or live in my upcoming course which starts in just a few days near the San Francisco airport... there are still a few seats left!

Learning Programming

2013-09-04T00:01:00.001-07:00

Two years ago, I wrote a post on "learning Python" to launch this blog dedicated to Python. While useful, it doesn't address beginners' needs as much, so it's time for a revisit. Because Python is such a user-friendly language for beginners, I'm often asked whether Python is the "best first language" for those new to programming. While tempted to respond in the affirmative, my answer really is, "it depends." It depends on your experience, age, level of exposure, etc.

Yes, there are indeed plenty of resources out there, such as courses from online learning brands such as Khan Academy, Udacity, Coursera, Codecademy, CodeSchool, and edX, but most certainly don't come with an instructor, instead relying on live or recorded videos and possibly supplemental study groups, or "cohort learning," as a colleague of mine has branded it. Whatever the mechanism, it's surely better than pure online tutorials or slaving away over a book, neither of which come with instructors either.

Stepping back a bit, before jumping into hardcore C/C++, Java, PHP, Ruby, or Javascript lessons, for learning tools that are used in industry today, there are better stepping stones to get you there. You may be a kid or a professional who either doesn't code much or had done so long ago. You're say that type of user who is "insulted" by the move "left" or "right" commands for controlling a turtle, say, and desire something more complex. The good news is that there are tools out there, more which allow you to venture further without an instructor.

One of them is Scratch, a "jigsaw puzzle"-like programming language created at MIT (or Tynker or Blockly, Scratch-like derivatives). Yes, you will do left, right, up, down, etc., but you'll also get to play audio, video, repeat commands, draw graphics, and make sounds. This tool is great for teaching the young learner, who don't need any of the advanced features but which are available for when they're ready to take the next step. It can be used to teach children the concepts of programming without all the syntax that text-based programming languages feature which may make learning those concepts a burden.

If you wish to proceed, go to the website to get started. They've got videos there as well as projects you can copy. As you can see, you snap together puzzle pieces that teach you coding. Better yet, to get started even more quickly, clone one (or more) of the projects, and "tweak" the code a bit to "do your own thing." In time, you may even develop your own fun applications or real games. Another similar graphical learning tool to consider is Alice from the University of Virginia and now Carnegie-Mellon University.

Once you're comfortable with that type of working environment, there's a similar tool from MIT called App Inventor. Leverage your Scratch skills and start building applications that run on Android devices! There's an emulator, so you don't really need an Android device, but it's certainly more rewarding when you can use an app that you built running on a tablet or phone! (Try a family friend who may have an old device they don't use any more.)

Once you're to move beyond block-like languages, there are 2 good choices (or better yet, do both!). One of which the de facto language of the web: Javascript. Unfortunately, there are so many online tutorials out there, I wouldn't know which to suggest, so looking forward to your comments below. The ones which are the most effective however, have you learning then coding directly into the browser and seeing results immediately, requiring you to write successful Javascript before allowing you to move on.

The thing about Javascript is that code typically only runs within the browser, to control web pages (i.e., "DOM manipulation") and actions you can take on a single page -- it can also be AJAX code that makes an external call to update a page without requiring a page load. Nevertheless, browser-only execution can be somewhat limiting, so there are now 2 additional ways you can use it.

One is to write "server-side" applications via Node.js. This type of Javascript allows you to write code that executes on the remote machine serving your web pages (generally) after you've entered information in a form and clicked submit. For every web page that users see and interact with, there's also got to be code on the server side that does all the work! This code will also end up returning the final HTML that users see in their browsers once the form has been submitted and results returned.

Another place you can use Javascript is in Google's cloud. The tool there is called Google Apps Script. Using Apps Script, you can create applications that interact with various Google Apps, automate repetitive tasks, or write glue code that lets you connect and share data between different Google services. Try some of their tutorials to get started!

The other option besides Javascript is Python. No doubt you already know what it is since you're here. Python's syntax is extremely approachable for beginners and is widely considered "executable pseudocode." That's right, a programming language that doesn't require you have a Computer Science degree to make good use of it! It's also one of those rare languages that can be used by adults in the professional world as well as by kids learning how to code. Sure there are many online learning systems out there, a sampling of which are here:

Codecademy
LearnPython.org
PySchools
Online Python Tutor
LearnStreet (login optional)
CodingBat (login optional)
SingPath (login required)

See if you like any of them or have your new coder friends try them out. However, I think kids (and even adults) learn programming best when they get to write cool games (leveraging the amazing PyGame library). There are several books written just for kids, including "Hello World" which was actually written by an engineer and his son! Along with that book there are two more you should consider:

Hello World! Computer Programming for Kids and Other Beginners (Sande & Sande)
Invent your Own Computer Games with Python (Sweigart)
Python for Kids: A Playful Introduction to Programming (Briggs)

Two of the three books above are in the beginners list I created over a year ago along with two other Python reading lists in this post. (The third book should be added to the list as well.) Those of you who are already programmers probably know which one I would recommend. :-) Seriously though, those reading lists show that I can toot other horns too. :P

Here are other online projects and learning resources, including book websites, that you can also try (many are for kids):

How to Think Like a Computer Scientist (Downey, Elkner, Meyers, Wentworth)
Learning to Program (Gauld)
LiveWires Python course
A Byte of Python (Swaroop)
Instant Hacking: Learning to Program with Python (Hetland)
Snake Wrangling for Kids (Briggs)
Computer Programming is Fun! (Handy)
Guido van Robot
RUR-PLE

In conjunction with a good learning system, book, or project-based learning above, you should also try out one of many free online courses to validate things you've picked up but to also build other knowledge you haven't learned yet. There are a pair from Coursera and one from Udacity:

An Introduction to Interactive Programming in Python (Rice University)
Learn to Program: The Fundamentals (University of Toronto)
Introduction to Computer Science (University of Virginia)

For existing programmers who are still questioning why Python, check out Udacity's motivational blogpost.

That's it! Hopefully I've given you enough resources you can pass along to friends and family members who are intrigued by your passion for computer programming and wish to see what all the excitement is all about. A young man I met on vacation this summer motivated this post... good luck Mitchell! I hope to see the rest of you on the road as well, perhaps at a developers' conference or sitting in one of my upcoming Python courses!

Tuples aren't what you think they're for

2012-05-29T14:14:00.012-07:00

While I'm happy that the number of Python users continues to grow at a rapid pace and that there are many tutorials added each day to support all the newbies, there are a few things that make me cringe when I see them.

One example of this is seeing a Python college textbook (you can tell by its retail price) produced by a big-name publisher (one of the largest in the world which shall remain unnamed) that instructs users (of Python 2), to get user command-line input using the input() function! Clearly, this is a major faux pas, as most Python users know that it's a security risk and that raw_input() should always be used instead (and the main reason why raw_input() replaces and is renamed as input() in Python 3).

Another example is this recent article on lists and tuples. While I find the content useful in teaching new Python developers various useful ways of using slicing, I disagree with the premise that tuples...

along with lists are two of Python's most popular data structures
are mostly immutable but there are workarounds, and
should be used for application data manipulation

I would says lists and dictionaries are the two most popular Python data structures; tuples shouldn't even be in that group. In fact, I would even argue that tuples shouldn't be used to manipulate application data at all, as that wasn't what they were generally created for. (If this was the case, then why not have lists with a read-only flag?)

The main reason why tuples exist is to get data to and from function calls. [UPDATE: two other strong use cases: 1) "constructed" dictionary keys (i would've turned such N-tuples into a delimited string) and from that use comes 2) a data structure with positional semantics, aka indices with implied meaning... both of these view such tuples as an individual entity (made up of multiple components), again, not a data structure for manipulating objects. Named tuples is an related alternative. See the debate in the commentary below.]

Calling a foreign API or 3rd-party function and want to pass in a data structure you know can't be altered? Check. Calling any function where you want to pass in only one data structure (instead of separate variables)? Use "*" and you're good to go. Previously worked with a programming language that only allowed you to return a single value? Tuples are that one object (think of it as a single shopping bag for all your groceries).

All of the manipulations in the post on getting around the immutability are superfluous and not adhering to the best practice of not using tuples as a data structure. I mean, this is not a strict rule. If you're needing a data structure where you're not going to make any modifications and desire slightly better performance, sure a tuple can be used in such cases. This is why in Python 2.6, for the first time "evar," tuples were given methods!

There was never any need for tuples to have methods because they were immutable. "Just use lists," is what we would all say. However, lists had a pair of read-only methods (count() and index()) that led to inefficiencies (and poor practices) where developers used tuples for the reason we just outlined but needed to either get a count on how many times an object appeared in that sequence or wanted to find the index of the first appearance of an object. They would have to convert that tuple to a list, just to call those methods. Starting in 2.6, tuples now have those (and only those) methods to avoid this extra nonsense.

So yes, you can use tuples as user-land data structures in such cases, but that's really it. For manipulation, use lists instead. As stated at the top, I'm generally all for more intro posts and tutorials out there. However, there may be some that don't always impart the best practices out there. Readers should always be alert and question whether there are more "Pythonic" ways of doing things. In this case, tuples should not be one of the "[two] of the most commonly used built-in data types in Python...."

Integrating Google APIs and Technologies

2012-04-06T14:19:00.019-07:00

In 1997, long before my tenure at Google, I became a member of the Python community in helping to create Yahoo!Mail, one of the most popular web-based email systems in the world. There were only two Python books on the market back then, and neither addressed my developer’s need to learn Python quickly and competently, so I had to resort to the online docs. This absence, and consequently my development of class materials for a Python course, inspired me to write Prentice Hall’s bestselling Core Python Programming over a decade ago. Since then, I’ve used Python to work on all kinds of interesting applications, from web-based e-mail to geolocalized product search engines, social media games, antispam/antivirus e-mail appliances, and most interestingly, software for doctors to help them analyze and assess patients with spinal fractures. (Ask me about osteoporosis!)

Today at Google, my work involves advocating our tools and APIs to the global developer community. Now that I've been part of the Google family for the past 2.5 years, I thought it would be fun to integrate some of our technologies into the book. With the just-published 3rd edition, readers will find revised but also brand new material they can use to build real applications with. Some of the Google technologies I've integrated into Core Python Applications Programming include accessing your Gmail, parsing Google News XML feeds, and a complete chapter on cloud computing with Google App Engine. It’s also the first published book to feature code that utilizes the Google+ API. While the book contains a longer example using that API, I want to show you how easy it is to connect to Google+ using Python right now!

The bulk of the work in connecting to Google+ (and other Google APIs) is done by my fellow colleagues who maintain the Google APIs Client Library for Python, easily downloaded with pip or easy_install as "google-api-python-client." With this library, the most difficult step to connect with your API of choice has basically been reduced to a single line... see the fourth line of this short Python 2.x example:

# plus.py (by Wesley Chun under CC-SA3.0 license)
from apiclient import discovery

API_KEY = YOUR_KEY_FROM_CONSOLE_API_ACCESS_PAGE
service = discovery.build("plus", "v1", developerKey=API_KEY)
feed = service.activities().search(query='android').execute()
for record in feed['items']:
post = ' '.join(record['title'].strip().split())
if post:
print '\nFrom:', record['actor']['displayName']
print 'Post:', post
print 'Date:', record['published']

In that one line of code (italicized above), we use the Google APIs Client Library's apiclient.discovery.build() method, passing in: a) the desired API ("plus" for Google+), b) the version (currently "v1"), and c) the API key you obtained from your project's development console in the "Simple API Access" section. This key gives your project access to APIs that do not need to access user data. Once we have a handle to the service, we can execute generic queries on the available data stream.

In this code snippet, we're simply querying for the latest (public) Google+ posts that are related to Android and displaying them on the command-line (code can be easily repurposed into any mobile or web application). Naturally, you need to go through the OAuth flow if you do want access to authenticated data. Give it a try!

If you like the code, dig into Core Python Applications Programming for a longer, more detailed example. Both scripts can be downloaded at the website (the code is part of Chapter 15), and you can get involved in the conversation on the Google+ page. I'm open to all feedback, suggestions, and fixes. You can find me at +wescpy or @wescpy. Looking forward to meeting you at an upcoming Google or Python event or in one of my public courses!

A new PyCon... and a new book (and a new article)!!

2012-03-09T00:02:00.026-08:00

I'm excited about this year's PyCon conference happening this time in the heart of Silicon Valley. There are many firsts, so let's just list a few here (let me know if I'm missing any)!

This is the first PyCon...

ever held in Silicon Valley (although older Python workshops have been hosted here)
that had a cap; yes, we "ended" registration at 1500 people
where we ran out of swag bags; 1800 were ordered... POOF, gone by mid-Saturday
to have sold out! (even though we capped at 1500; didn't stop it from going over 2000)
with an attendance near or exceeding 2200 (2257)
that had to stop accepting sponsorships... at 136!!!
to feature a physical race (not to be confused with a race condition)

Another exciting announcement is that my first 3rd edition Core Python book will be published and debuting at the conference!! It's called Core Python Applications Programming and based on the second part of the original Core Python Programming book. All of the books' individual home pages are now unified at corepython.com. The books also have a shared Google+ page for you to encircle! They're literally "hot off the presses" as they were overnighted by the printer to the publisher's hotel and brought by hand to the conference! (Amazon's not shipping them for another 10 days after that!)

The new book features upgrades and new stuff added to existing chapters as well as brand new chapters on Django, Google App Engine, and text processing with CSV, JSON, and XML. There is even new material on Twitter and Google+ in case you're feeling more social than when the previous edition was published. Those of you asking for that PowerPoint slideshow generator for the past N years, or perhaps an intro to NoSQL/MongoDB? Yep, they're in there too! Finally, I've added not only Python 3 equivalents to many of the code samples, but I also cover some best practices when porting from 2.x to 3.x.

With all of the updates and new material, I'm hoping that this will be one of the most popular places for intermediate Python programmers to go once they've gotten comfortable with the langauge but want to apply their skills to a variety of topics in Python development today. While the coverage doesn't necessarily go particularly deep, the goal is to give programmers a kickstart with a comprehensive introduction.

To help kickoff the new book, I got to thinking about Python books in general, especially the numerous times that people have either asked me or asked in some online forum: "What's a good Python book?" Unlike Python, there's not one right answer for this question, so as part of this exploration, I came up with 3 different book lists for diverse audiences of readers out there. You can find that article at InformIT.

In the meantime, it's back to the drawing board for me as I prepare to work on the 3rd edition of the main part of Core Python. If you've got ideas or suggestions on updating part 1 or wish to participate in the review process, please contact me now! (@wescpy/+wescpy)

ps. For those interested in brushing up on your Python skills, I'll be offering my popular Intro+Intermediate course this summer near the San Francisco airport. Go to cyberwebconsulting.com for more information!

Writing 2.x & 3.x Compatible Code

2011-12-06T01:00:00.000-08:00

While we're at the crossroads transitioning from Python 2 to 3, you may wonder whether it is possible to write code runs without modification under both Python 2 and 3. It seems like a reasonable request, but how would you get started? What breaks the most Python 2 code when executed by a 3.x interpreter?

print vs. print()

If you think like me, you'd say the print statement. That's as good a place to start as any, so let's give it a shot. The tricky part is that in 2.x, it's a statement, thus a keyword or reserved word while in 3.x, it's just a BIF. In other words, because language syntax is involved, you cannot use if statements, and no, Python still doesn't have #ifdef macros!

Let's try just putting parentheses around the arguments to print:

>>> print('Hello World!')

Hello World!

Cool! That works under both Python 2 and Python 3! Are we done? Sorry.

>>> print(10, 20) # Python 2

(10, 20)

You're not going to be as lucky this time as the former is a tuple while in Python 3, you're passing in multiple arguments to print():

>>> print(10, 20) # Python 3

10 20

If you think a bit more, perhaps we can check if print is a keyword. You may recall there is a keyword module which contains a list of keywords. Since print won't be a keyword in 3.x, you may think that it can be as simple as this:

>>> import keyword

>>> 'print' in keyword.kwlist

False

As a smart programmer, you'd probably try it in 2.x expecting a True response. Although you would be correct, you'd still fail for a different reason:

>>> import keyword

>>> if 'print' in keyword.kwlist:

... from __future__ import print_function

...

File "", line 2

SyntaxError: from future imports must occur at the beginning of the file

One solution which works requires you to use a function that has similar capabilities as print. One of them is sys.stdout.write() while another is distutils.log.warn(). For whatever reason, we decided to use the latter in many of this book's chapters. I suppose sys.stderr.write() will also work, if unbuffered output is your thing.

The "Hello World!" example would then look like this:

# Python 2.x

print 'Hello World!'

# Python 3.x

print('Hello World!')

The following line would work in both versions:

# Python 2.x & 3.x compatible

from distutils.log import warn as printf

printf('Hello World!')

That reminds me of why we didn't use sys.stdout.write()... we would need to add a NEWLINE character at the end of the string to match the behavior:

# Python 2.x & 3.x compatible

import sys

sys.stdout.write('Hello World!\n')

The one real problem isn't this little minor annoyance, but that these functions are no true proxy for print or print() for that matter... they only work when you've come up with a single string representing your output. Anything more complex requires you to put in more effort.

Import your way to a solution

In other situations, life is a bit easier, and you can just import the correct solution. In the code below, we want to import the urlopen() function. In Python 2, it lives in urllib and urllib2 (we'll use the latter), and in Python 3, it's been integrated into urllib.request. Your solution which works for both 2.x and 3.x is neat and simple in this case:

try:

from urllib2 import urlopen

except ImportError:

from urllib.request import urlopen

For memory conservation, perhaps you're interested in the iterator (Python 3) version of a well-known built-in like zip(). In Python 2, the iterator version is itertools.izip(). This function is renamed as and replaces zip() in Python 3, and if you insist on this iterator version, your import statement is also fairly straightforward:

try:

from itertools import izip as zip

except ImportError:

pass

One example which isn't as elegant looking is the StringIO class. In Python 2, the pure Python version is in the StringIO module, meaning you access it via StringIO.StringIO. There is also a C version for speed, and that's located at cStringIO.StringIO. Depending on your Python installation, you may prefer cStringIO first and fallback to StringIO if cStringIO is not available.

In Python 3, Unicode is the default string type, but if you're doing any kind of networking, it's likely you'll have to manipulate ASCII/bytes strings instead, so instead of StringIO, you'd want io.BytesIO. In order to get what you want, the import is slightly uglier:

try:

from io import BytesIO as StringIO

except ImportError:

try:

from cStringIO import StringIO

except ImportError:

from StringIO import StringIO

Putting it all together

If you're lucky, these are all the changes you have to make, and the rest of your code is simpler than the setup at the beginning. If you install the imports above of distutils.log.warn() [as printf()], url*.urlopen(), *.StringIO, and a normal import of xml.etree.ElementTree (2.5 and newer), you can write a very short parser to display the top headline stories from the Google News service with just these roughly eight lines of code:

g = urlopen('http://news.google.com/news?topic=h&output=rss')

f = StringIO(g.read())

g.close()

tree = xml.etree.ElementTree.parse(f)

f.close()

for elmt in tree.getiterator():

if elmt.tag == 'title' and not \

elmt.text.startswith('Top Stories'):

printf('- %s' % elmt.text)

This script runs exactly the same under 2.x and 3.x with no changes to the code whatsoever. Of course, if you're using 2.4 and older, you need to download ElementTree separately.

The code snippets in this subsection come from the "Text Processing" chapter of the book, so take a look at the goognewsrss.py file to see the full version in action.

Some will feel that these changes really start to mess up the elegance of your Python source. After all, readbility counts! If you prefer to keep your code cleaner yet still write code that runs under both versions without changes, take a look at the six package.

six is a compatibility library who's primary role is to provide an interface to keep your application code the same while hiding the complexities described in this appendix subsection from the developer. To find out more about six, read this: http://packages.python.org/six

Regardless whether you use a library like six or choose to roll your own, we hoped to show in this short narrative that it is possible to write code that runs under both 2.x & 3.x. The bottom line is that you may have to sacrifice some of the elegance and simplicity of Python, trading it off for true 2 to 3 portability. I'm sure we'll be revisiting this issue for the next few years until the whole world has completed the transition to the next generation.

Learning Python

2011-09-29T12:05:00.000-07:00

Finally here!! Welcome to my new blog dedicated specifically to Python programming. My old blog at LiveJournal is getting a little long in the tooth, plus their spam filtering was lacking... something. However, I'll leave that blogsite up for personal posts, but I'm moving here for Python stuff. In fact, I think I'll "refactor" some of those old posts here from time-to-time.

As you can see, I've named this blog (currently) the same name as my book. This was not done intentionally as some marketing effort to promte the book as I do want to focus on having readers/users understand the core elements of the language. This in turn makes for better Python programmers, thus lowers the stress level in the world a little. Now let's really start the contents of this post, and that means going back to the beginning and learning Python:

If you don't know Python but already code, try the Google Python course first. It is basically the internal 2-day training course scrubbed and externalized for all of you. It jumps in fairly quickly without a lot of explanation. If you learn best in this style, you'll be okay. There are also 7 videos available on the site so you can follow the lectures from both days.

If you're really new to programming, consider taking a beginner course in programming. Sure Python is a great first language to learn coding with, but not all such courses feature it. If you have no time for courses, do the online Python tutorial as well as the SingPath exercises. As far as books go, online-wise you can try Learn Python the Hard Way (book+lessons) or Dive Into Python (book-only).

Since Dive Into Python is written by a co-worker of mine, you can buy a dead-tree version if you wish to support them, or Core Python Programming, to help out someone else you know. :-) The primary difference between these books is that one is a quick dive while the other is a deep dive, as so well described in this Amazon review. Here's another more recent review although it does not shed as much positive light on my colleague's tome. As far as references go, you can support yet a third Googler by buying Python in a Nutshell, or a non-colleague who wrote Python Essential Reference.

If you have children or wish to teach kids how to program, Hello World! Computer Programming for Kids and Other Beginners is a friendly and well-received book written by an engineer and his (then) 8-year old son. It's good to have a kid's perspective as kids (generally) respect other kids who are/were in their same shoes.

The cool thing is that the sky is the limit once you've learned some Python. You can take off in any different direction like Google App Engine, Pyramid, or Django for web or mobile development, SciPy/NumPy for scientific development, SQLAlchemy/SQLObject for database ORMs, Jython for Java development, Win32 for PC development, PyGame for writing games, etc. Testing is something you should always keep in mind, so look into Nose or py.test.

For those of you who already know programming, but want to learn Python as quickly and as in-depth as possible in the shortest amount of time, join me near San Francisco for my upcoming 3-day Python training course running mid-week October 18-20! You need to be proficient programming in another high-level programming language (such as C/C++, Java, PHP, Ruby, etc.). The fee covers for all lectures, labs (3 per day), and everyone gets a copy of my bestseller, Core Python Programming. There is a significant discount to primary/secondary school teachers so ask about that if applicable!

I hope this helps some of you get started! We always welcome new users to the Python community!!