{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Administration: Deploy automatic notifications\n", "\n", "> * ✏️ Needs Configuration\n", "* 🗄️ Administration\n", "* 🔔 Notifications\n", "\n", "To optimize your notebook's utility, consider configuring it to alert someone of a notable event. For example, a publisher could be notified that their web map contains a broken layer URL, or an admin could be notified when certain users haven't logged on in over a month. Notifications may arrive as an email or text message - or they could take the form of a tweet or a Slack post. There are many ways to use native Python libraries to send notifications through external services. This notebook will walk you through the technical details of how to configure these external services for your hosted notebooks and provide you the sample code needed to control these services." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Security\n", "\n", "Before writing code that connects to any external service, it is vital to keep security in mind. Notebooks are commonly shared and code is copied and pasted, sometimes leading to passwords and private keys landing in the wrong hands.\n", "\n", "**Passwords and private keys should NEVER be stored in plaintext in a notebook**. There are many paradigms to securely store passwords and private keys. We will be storing all 'secrets' in a [private CSV item](https://www.esri.com/arcgis-blog/products/arcgis-online/sharing-collaboration/managing-security-and-findability-of-items-with-the-arcgis-sharing-model/) that is only accessible by you and your GIS administrator. \n", "\n", "> __Note__: that although this is more secure than storing your passwords as a plain string in a notebook, this is __NOT__ a totally secure solution—items aren't encrypted in the underlying server where they are stored. Consider storing your secrets in an encrypted files in `/arcgis/home/`.\n", "\n", "Run the below cells to import necessary libraries, connect to our Organization, and create a new private blank CSV item." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# First, import some libraries we'll need, and connect to our GIS\n", "import csv\n", "import os\n", "import smtplib\n", "import requests\n", "import logging\n", "log = logging.getLogger()\n", "\n", "from arcgis.gis import GIS\n", "\n", "gis = GIS(\"home\")" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Secret CSV Item ID = '4fbaa460b3e347f09aea312a6d69b274'\n" ] }, { "data": { "text/html": [ "
\n", "
\n", " \n", " \n", " \n", "
\n", "\n", "
\n", " secrets\n", " \n", "
CSV by arcgis_python\n", "
Last Modified: September 20, 2018\n", "
0 comments, 0 views\n", "
\n", "
\n", " " ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Then, make an empty .csv file with headers\n", "with open('./secrets.csv', \"w\") as f:\n", " f.write(\"secret_key,secret_value\\n\")\n", "\n", "# Publish it as a private item only you and admins can see\n", "secret_csv_item = gis.content.add({'access':'private'}, './secrets.csv')\n", "secret_csv_item.protect()\n", "print(\"Secret CSV Item ID = '{}'\".format(secret_csv_item.id))\n", "secret_csv_item" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that you have this blank CSV item created, copy and paste its item ID from above and make it a string constant for this notebook. Then, let's make a helper function that downloads the CSV and converts it to an easy-to-use `dict`." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "SECRET_CSV_ITEM_ID = '4fbaa460b3e347f09aea312a6d69b274'" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def get_secrets(gis=gis,\n", " secret_csv_item_id = SECRET_CSV_ITEM_ID):\n", " \"\"\"Returns the secrets.csv file as a dict of secret_key : secret_value\"\"\"\n", " item = gis.content.get(secret_csv_item_id)\n", " with open(item.download(), 'r') as local_item_file:\n", " reader = csv.DictReader(local_item_file)\n", " return { rows['secret_key'] : rows['secret_value'] for rows in reader}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we run our function, we should see a blank dictionary (since we haven't added any secrets yet)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "secrets = get_secrets()\n", "secrets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's make a helper function to add a new secret. Each secret will have a secret_key to identify the secret, and the secret itself. Let's add an example 'external_service_api_key' secret." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def add_secret(secret_key, secret_value, gis=gis,\n", " secret_csv_item_id = SECRET_CSV_ITEM_ID):\n", " \"\"\"Appends a new secret to the secrets.csv item\"\"\"\n", " item = gis.content.get(secret_csv_item_id)\n", " with open(item.download(), 'a') as local_item_file:\n", " local_item_file.write(\"{},{}\\n\".format(secret_key, secret_value))\n", " return item.update({}, local_item_file.name)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "add_secret(\"external_service_api_key\", \"hPi2KWmYh6\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now if we call `get_secrets()` again, we should see our secret we just added! `secrets` is a dict, so you can get any secret by specifying the secret_key, like this:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'hPi2KWmYh6'" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "secrets = get_secrets()\n", "secrets[\"external_service_api_key\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**If you ever use this add_secret() function, make sure to delete the cell after calling it.** Leaving the function call in a notebook is as much of a security risk as saving your secret in a plaintext string. Always double check your notebooks for strings, print outputs, and variables that can inadvertently expose secrets.\n", "\n", "The rest of the notebook will assume that you have added the appropriate secrets (passwords, API keys, private URLs, and so on) to the private CSV item you own." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Email Notifications\n", "\n", "Emails might be the gold standard for notifications. Emails are simple and sortable, and everyone can receive them. Emails can only be sent by an external service, so Python must know how to connect to that service. The most common way to do this is to connect to an SMTP Server.\n", "\n", "### SMTP Server\n", "\n", "The builtin [`smtplib`](https://docs.python.org/3/library/smtplib.html#module-smtplib) library makes it easy to connect an external email SMTP server, whether that's a server you manage or a commercial offering (e.g. Gmail).\n", "\n", "The below function provides an outline for sending emails via an external SMTP server. Make sure to update the string values of `smtp_server_url`, `smt_server_port`, `sender`, and `username` to reflect your own SMTP server. Add your SMTP server password to your secrets.csv item with the key `'smtp_email_password'`, so you can access it in a secure way." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "def send_email_smtp(recipients, message,\n", " subject=\"Message from your Notebook\"):\n", " \"\"\"Sends the `message` string to all of the emails in the \n", " `recipients` list using the configured SMTP email server. \n", " \"\"\"\n", " try:\n", " # Set up server and credential variables\n", " smtp_server_url = \"smtp.example.com\"\n", " smtp_server_port = 587\n", " sender = \"your_sender@example.com\"\n", " username = \"your_username\"\n", " password = secrets[\"smtp_email_password\"]\n", "\n", " # Instantiate our server, configure the necessary security\n", " server = smtplib.SMTP(smtp_server_url, smtp_server_port)\n", " server.ehlo()\n", " server.starttls() # Needed if TLS is required w/ SMTP server\n", " server.login(username, password)\n", " except Exception as e:\n", " log.warning(\"Error setting up SMTP server, couldn't send \" +\n", " f\"message to {recipients}\")\n", " raise e\n", "\n", " # For each recipient, construct the message and attempt to send\n", " did_succeed = True\n", " for recipient in recipients:\n", " try:\n", " message_body = '\\r\\n'.join(['To: {}'.format(recipient),\n", " 'From: {}'.format(sender),\n", " 'Subject: {}'.format(subject),\n", " '',\n", " '{}'.format(message)])\n", " message_body = message_body.encode(\"utf-8\")\n", " server.sendmail(sender, [recipient], message_body)\n", " print(f\"SMTP server returned success for sending email \"\\\n", " f\"to {recipient}\")\n", " except Exception as e:\n", " log.warning(f\"Failed sending message to {recipient}\")\n", " log.warning(e)\n", " did_succeed = False\n", " \n", " # Cleanup and return\n", " server.quit()\n", " return did_succeed" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that you have updated the above function with your SMTP server information, run the below cell to send a test email. The function should return `True`, print out no warnings, and send an email to your test recipient! You can use this function anywhere in your code to send out an email." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "send_email_smtp(recipients = ['somebody@example.com'],\n", " message = \"Hello World!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Text Messages via Email\n", "\n", "Many major cellular providers offer a mechanism to send text messages (SMS or MMS) via email. You just need to know what the phone number is, and what cellular provider that phone number is under. For example, if phone number _555-123-4567_ has AT&T as their cellular provider, the below code cell can be run to send that number a text message." ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "send_email_smtp(recipients = ['5551234567@txt.att.net'],\n", " message = \"Hello World!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check your area's cellular providers to see if this functionality is supported. Standard messaging and data rates would apply. The below table outlines some of the common cellular providers' email formats—it is not a complete or authoritative list.\n", "\n", "Cellular Provider | SMS email | MMS email |\n", "---|---|---|\n", "AT&T | number@txt.att.net | number@mms.att.net |\n", "T-Mobile | number@tmomail.net | number@tmomail.net |\n", "Verizon |number@vtext.com | number@vzwpix.com |\n", "Sprint | number@messaging.sprintpcs.com | number@pm.sprint.com |\n", "U.S. Cellular | number@email.uscc.net | number@mms.uscc.net |\n", "Virgin Mobile | number@vmobl.com | number@vmpix.com |\n", "Boost Mobile | number@sms.myboostmobile.com | number@myboostmobile.com | \n", "Cricket | number@sms.cricketwireless.net | number@mms.cricketwireless.net |\n", "Google Fi (Project Fi) | number@msg.fi.google.com | number@msg.fi.google.com |\n", "Tracfone | - | number@mmst5.tracfone.com |\n", "Metro PCS | number@mymetropcs.com | number@mymetropcs.com |\n", "Republic Wireless | number@text.republicwireless.com | - |\n", "Ting | number@message.ting.com | - |\n", "Consumer Cellular | number@mailmymobile.net | - |\n", "C-Spire | number@cspire1.com | - |\n", "Page Plus | number@vtext.com | - | \n", "\n", "Source: https://20somethingfinance.com/how-to-send-text-messages-sms-via-email-for-free/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Zapier\n", "\n", "[Zapier.com](https://zapier.com) is a platform that allows you to automate workflows by connecting external apps together. Zapier is useful in this context because you can trigger _any_ action Zapier supports from Python by calling a Zapier webhook with a POST request via the `requests` library.\n", "\n", "In this example, we will be using Zapier to post a tweet. You aren't limited to just Twitter: Zapier integrates with Slack, Github, Salesforce, Jira, Google Docs, Microsoft products (Office 365, OneNote, Excel), Email services (Gmail, MailChimp), and [1000+ other apps](https://zapier.com/apps). Read more about Zapier to see if you can use it for a notifications system by [visiting their website](https://zapier.com)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Twitter via Zapier (example)\n", "\n", "After making an account with Zapier, your homepage will give you a form asking you \"What Do You Want To Automate Today?\". On the left _\"Connect this app...\"_, select __\"Webhooks by Zapier\"__. On the right _\"with this one!\"_, select the app you want to connect Zapier with. We are choosing Twitter for this example, but you can choose any app.\n", "\n", "After selecting these options, new options will appear. On the left _\"When this happens...\"_, select __\"Catch Hook\"__. On the right _\"then do this!\"_, select __\"Create Tweet\"__. Your form should now look like below. Press \"Make a Zap\" to configure the Webhook and Twitter." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Set up your Webhook\n", "\n", "In the __'Catch Hook'__ section, select this option:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Press \"Set Up Webhook\" to get the URL that you will call to trigger this action. **This URL should be kept as private as you would keep a password**. Add this URL to your secrets.csv item with the key `'zapier_tweet_webhook_url'`, so you access it in a secure way." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Press 'Test This Step' to temporarily activate this Webhook endpoint. Webhooks expect a JSON object to be POSTed to the URL, so we will run the below code cell to send a test JSON dict to the endpoint:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "secrets = get_secrets()\n", "requests.post(secrets['zapier_tweet_webhook_url'],\n", " data={'message' : 'Hello World!'})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Back on your Zapier web page, under the drop-down of 'Hook A,' you should see the JSON object that you just sent. This is how you verify your Webhook is working." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Set up Twitter\n", "\n", "Set your action as 'Create Tweet,' then link your Twitter profile with the one you want to send the tweets in 'Choose Account.' You should be redirected to log in to your Twitter account. After you complete that, your page should look like this:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Press 'Set Up Template' to edit the message you want your tweet to send out. There's an icon on the right edge of the 'Message' area that will drop down to show 'Catch Hook' and the JSON that you previously sent to your webhook test. By pressing that `Message: Hello World!` entry, you will see the message content `Hello World!` in the Twitter Message field. This means that the contents of any 'message' entry sent to your webhook will be used for a tweet. It should look like this:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Feel free to test your Tweet from the Zapier webpage. Once you are satisfied that everything is working, it's time to make your 'Zap' public and ready to use! Press this switch to do that:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that your 'Zap' is public and ready for production, let's make a helper function that will call the Webhook URL with the correct parameters, and check the response for any errors." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "def tweet_via_zapier(message):\n", " \"\"\"Sends the `message` string as a tweet via zapier.\"\"\"\n", " # Do the POST request that sends the tweet\n", " secrets = get_secrets()\n", " response = requests.post(secrets['zapier_tweet_webhook_url'],\n", " data={\"message\" : message})\n", " \n", " # Check that the response looks OK; warn the user if it doesn't\n", " if response.ok:\n", " return True\n", " else:\n", " log.warning(\"Zapier tweet POST returned a bad response:\")\n", " log.warning(response.text)\n", " return False" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, call this function and verify that your Twitter account posted the message! You can use this function anywhere in your code to send out a tweet." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tweet_via_zapier(\"Hello World!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### IFTTT\n", "\n", "https://ifttt.com/ is similar to Zapier in that it lets you automate workflows by connecting apps together. The workflow for setting up an IFTTT app is also similar, in that you will be triggering a Webhook from Python to launch an activity in another application.\n", "- Create an account with IFTTT\n", "- Under your profile icon in the upper right corner, press _'New Applet'_\n", "- For the 'this' event, select the webhook _'Receive a web request'_\n", "- Name it `'triggered_from_python'`\n", "- For the 'that' event, select _'Post a tweet'_\n", "- Link your Twitter account with IFTTT\n", "- For your \"Tweet Text\":\n", " - Delete all of the template message\n", " - Press _'Add Ingredient'_\n", " - Select _'Value1'_\n", " - Your \"Tweet Text\" should just be `{{Value1}}`\n", "- Save the applet. It should look something like this:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- You aren't done yet: view the Webhooks you set up by clicking here: https://ifttt.com/services/maker_webhooks/settings\n", "- Find the 'URL' of the webhook that you set up, and open that URL in a new browser window\n", "- You should see a page that says something like _'Your key is: LEU60FvFsKCqbBRlkPXSlaP3qCrVV7G24BhJFTvfta7'_. Add this API key to your secrets.csv item with the key `'ifttt_webhook_api_key'`, so you can access it in a secure way.\n", "\n", "The below cells will create and test a function to send a tweet using IFTTT. Use this function anywhere in your code for that purpose." ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "def tweet_via_ifttt(message):\n", " \"\"\"Sends the `message` string as a tweet via ifttt.\"\"\"\n", " # Do the POST request that sends the tweet\n", " event_name = \"triggered_from_python\"\n", " api_key = get_secrets()[\"ifttt_webhook_api_key\"]\n", " webhook_url = \"https://maker.ifttt.com/trigger/\" + \\\n", " \"{}/with/key/{}\".format(event_name, api_key)\n", " response = requests.post(webhook_url,\n", " data={\"value1\" : message})\n", " \n", " # Check that the response looks OK; warn the user if it doesn't\n", " if response.ok:\n", " return True\n", " else:\n", " log.warning(\"IFTTT tweet POST returned a bad response:\")\n", " log.warning(response.text)\n", " return False" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tweet_via_ifttt(\"Hello World from IFTTT!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Connecting Directly to Applications\n", "\n", "Services like Zapier and IFTTT act as a proxy, as they offer one entry point that allows you to connect to many different apps. Zapier and IFTTT are great for getting started, but you can gain more flexibility and power by connecting directly to these external apps. It is up to you to research if an external application you want to connect to can be driven through a RESTful API, Webhooks, or other method — there are endless apps to connect to, but the process should always be roughly the same. We will do one final example of connecting directly to the Slack API." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Slack\n", "\n", "Slack, a popular team collaboration platform, offers a [rich API](https://api.slack.com/) for programmatically controlling almost every component of the Slack ecosystem. This example will go through the most basic use case: sending a message to a channel.\n", "\n", "- View the 'Getting started with Incoming Webhooks' page here: https://api.slack.com/incoming-webhooks\n", "- Follow steps 1 and 2 to create a Slack app, link it to a workspace, and enable Webhooks\n", "- Follow step 3 to create a new Webhook that will post a message to a channel\n", "- Authorize your webhook, copy the Webhook URL, and add it to your secrets.csv item with the key `'slack_example_channel_webhook_url'`\n", "\n", "The below cells will create and test a function to send a message to this Slack channel. Use this function anywhere in your code for that purpose." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "def notify_example_channel_slack(message):\n", " webhook_url = get_secrets()[\"slack_example_channel_webhook_url\"]\n", " response = requests.post(webhook_url, json={\"text\" : message})\n", " \n", " # Check that the response looks OK; warn the user if it doesn't\n", " if response.ok:\n", " return True\n", " else:\n", " log.warning(\"Slack POST returned a bad response:\")\n", " log.warning(response.text)\n", " return False" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "notify_example_channel_slack(\"Hello World!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusion\n", "\n", "This notebook walked you through the essential technical details needed to connect your Python notebooks to external notifications systems. Armed with a hefty arsenal of sample code and the knowledge of how to best use it, you are ready to go out and notify the world. \n", "\n", "If you're looking for inspiration of where you can use notifications in your GIS workflows, search for the following notebooks in your samples gallery:\n", "\n", "- Check WebMaps for Broken URLs" ] } ], "metadata": { "esriNotebookRuntime": { "notebookRuntimeName": "ArcGIS Notebook Python 3 Standard", "notebookRuntimeVersion": "10.7.1" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" } }, "nbformat": 4, "nbformat_minor": 2 }