{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Content Management: Check for broken URLs\n",
"\n",
"> * ✏️ Needs Configuration\n",
"* 🗄️ Administration\n",
"* 🔔 Notifications\n",
"\n",
"A WebMap is a collection of layers, basemaps, and other services. These services are hosted on either ArcGIS Online or an external ArcGIS Server, separate from the WebMap itself. If the external service gets deleted, the WebMap will still exist, but it will not function properly.\n",
"\n",
"This notebook will search through all WebMaps in specified groups, and will attempt to connect to each layer URL, basemap URL, etc. If the connection fails for any URL, the owner of the WebMap and other specified users will be notified. This notebook can be used to automatically identify broken WebMaps, and alert the necessary users so they can take the appropriate action."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To get started, import the necessary libraries and connect to our GIS:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import csv\n",
"import smtplib\n",
"import requests\n",
"import logging\n",
"log = logging.getLogger()\n",
"\n",
"from arcgis.gis import GIS\n",
"from arcgis.mapping import WebMap\n",
"\n",
"gis = GIS(\"home\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we will create our function that sends out notifications."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notifications\n",
"\n",
"An important part of this process is notifying the right people when we come across a broken URL. We will achieve this by connecting to an external SMTP server and sending out emails, but you can write _any_ code that connects to _any_ external service to send out notifications. __View the 'Notifications' notebook in the examples gallery for more information__.\n",
"\n",
"__You MUST modify the below cell__ to update `secret_csv_item_id`, `smtp_server_url`, `username`, and any other information needed to connect to your external smtp server. This includes making sure your _secrets.csv_ file item contains the `smtp_email_password` entry."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# Helper function for using your private secrets.csv file\n",
"def get_secrets(gis=gis,\n",
" secret_csv_item_id = ''):\n",
" \"\"\"Returns a dict of { secret_key : secret_value } from the \n",
" secrets.csv item. See the 'Notifications' notebook in the \n",
" examples gallery for more information.\n",
" \"\"\"\n",
" try:\n",
" item = gis.content.get(secret_csv_item_id)\n",
" with open(item.download(), 'r') as local_item_file:\n",
" reader = csv.DictReader(local_item_file)\n",
" return { rows['secret_key'] : rows['secret_value'] \\\n",
" for rows in reader }\n",
" except Exception:\n",
" return {}\n",
"\n",
"SECRETS = get_secrets()\n",
"\n",
"# Helper function to send out emails through an SMTP server\n",
"def send_email_smtp(recipients, message,\n",
" subject=\"Message from your Notebook\"):\n",
" \"\"\"Sends the `message` string to all of the emails in the \n",
" `recipients` list using the configured SMTP email server. \n",
" See the 'Notifications' notebook in the examples gallery\n",
" for more information.\n",
" \"\"\"\n",
" try:\n",
" # Set up server and credential variables\n",
" smtp_server_url = \"smtp.example.com\"\n",
" smtp_server_port = 587\n",
" sender = \"your_sender@example.com\"\n",
" username = \"your_username\"\n",
" password = SECRETS[\"smtp_email_password\"]\n",
"\n",
" # Instantiate our server, configure the necessary security\n",
" server = smtplib.SMTP(smtp_server_url, smtp_server_port)\n",
" server.ehlo()\n",
" server.starttls() # Needed if TLS is required w/ SMTP server\n",
" server.login(username, password)\n",
" except Exception as e:\n",
" log.warning(\"Error setting up SMTP server, couldn't send \" +\n",
" f\"message to {recipients}\")\n",
" raise e\n",
"\n",
" # For each recipient, construct the message and attempt to send\n",
" did_succeed = True\n",
" for recipient in recipients:\n",
" try:\n",
" message_body = '\\r\\n'.join(['To: {}'.format(recipient),\n",
" 'From: {}'.format(sender),\n",
" 'Subject: {}'.format(subject),\n",
" '{}'.format(message)])\n",
" message_body = message_body.encode(\"utf-8\")\n",
" server.sendmail(sender, [recipient], message_body)\n",
" print(f\"SMTP server returned success for sending email \"\\\n",
" f\"to {recipient}\")\n",
" except Exception as e:\n",
" log.warning(f\"Failed sending message to {recipient}\")\n",
" log.warning(e)\n",
" did_succeed = False\n",
" \n",
" # Cleanup and return\n",
" server.quit()\n",
" return did_succeed"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Make sure to independently test that your notifications function works before proceeding forward with this notebook."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configure Behavior\n",
"\n",
"Now that we have a function that sends out notifications, let's configure some variables specific to our Organization that will tell our notebook how we want it to run. With the default `CHECK_ALL_ITEMS` set to `False`, this notebook will check all items in the groups specified in the `CHECK_THESE_GROUPS` list of strings. If you would instead prefer to check all items in your Organization, set `CHECK_ALL_ITEMS` to `True`.\n",
"\n",
"__You MUST modify the below cell__ to add your group names to `CHECK_THESE_GROUPS`, __OR__ you must set `CHECK_ALL_ITEMS` to `True`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Set to `True` you would like to check ALL items in an Org\n",
"CHECK_ALL_ITEMS = False\n",
"# If `CHECK_ALL_ITEMS` is `False`, will check all items in these groups\n",
"CHECK_THESE_GROUPS = ['group_name_1', 'group_name_2']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, let's specify who we want to email when we find broken URLs. If `TRY_EMAIL_WEBMAP_AUTHOR` is set to `True`, the owner of the WebMap will attempt to be notified via email. All the email address strings in the `EMAIL_THESE_ADDITIONAL_PEOPLE` list will also be emailed in the event of any broken URL in any WebMap. Example:\n",
" \n",
"```python\n",
"EMAIL_THESE_ADDITIONAL_PEOPLE = ['somebody@example.com', 'somebodyelse@example.com']\n",
"```\n",
"\n",
"The default behavior is to only email the webmap owner. Modify the below cell to change that default behavior."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# If you try to email the author of the WebMap of the Broken URLs\n",
"TRY_EMAIL_WEBMAP_OWNER = True\n",
"# Email these additional people about any broken URLs\n",
"EMAIL_THESE_ADDITIONAL_PEOPLE = []"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## WebMap Parsing Logic\n",
"\n",
"A core component of this notebook will be testing if different URLs are 'reachable'. We will do this by creating a helper function that uses the `requests` library to make an HTTP GET request to the specified url and assert that is returns a good HTTP status code."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def is_url_reachable(url):\n",
" \"\"\"Returns a bool representing if the URL is reachable\"\"\"\n",
" try:\n",
" response = requests.get(url)\n",
" return response.ok\n",
" except Exception as e:\n",
" return False"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Test out this helper function by passing in strings such as `\"https://arcgis.com\"` or `\"https://arcgis.com/nonexistantpage.html\"`.\n",
"\n",
"Now, we can create a function that will test the URLs of all operational layers and basemap layers of a WebMap. This function will return a tuple of reachable and unreachable layers."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def test_urls_in_webmap(webmap_item):\n",
" \"\"\"Takes in an `Item` class instance of a Web Map Item.\n",
" Tests if all operational layers and basemap layers are\n",
" reachable. Returns a tuple of (reachable, unreachable), \n",
" with each tuple entry being a list of layers/basemaps JSON.\n",
" \"\"\"\n",
" reachable = []\n",
" unreachable = []\n",
" wm = WebMap(webmap_item)\n",
"\n",
" # Concatanate all operational layers and basemap layers to 1 list\n",
" all_layers = list(wm.layers)\n",
" if hasattr(wm.basemap, 'baseMapLayers'):\n",
" all_layers += wm.basemap.baseMapLayers\n",
"\n",
" # Test all of the layers, return the results\n",
" for layer in [layer for layer in all_layers \\\n",
" if hasattr(layer, 'url')]:\n",
" if is_url_reachable(layer.url):\n",
" log.debug(f\" [✓] url {layer.url} reachable\")\n",
" reachable.append(layer)\n",
" else:\n",
" log.debug(f\" [X] url {layer.url} NOT reachable\")\n",
" unreachable.append(layer)\n",
" return (reachable, unreachable)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Test out this function by passing in an `Item` class instance of a Web Map Item.\n",
"\n",
"Now, let's create a helper function that assembles a human readable string with information about the WebMap, what URLs aren't reachable, etc. We will use this when sending out emails."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"def assemble_message(webmap_item, reachable, unreachable):\n",
" \"\"\"Makes a human readable message with the args passed in\"\"\"\n",
" def _assemble_bullet_point(layer, is_reachable):\n",
" \"\"\"Internal function to assemble one bullet point from a layer\"\"\"\n",
" icon = '✅' if is_reachable else '❌'\n",
" return f\"* {icon} {layer.id}\"\n",
"\n",
" # Assemble a string representation of both lists\n",
" reachable_str = \"\\n\".join(_assemble_bullet_point(layer, True) \\\n",
" for layer in reachable)\n",
" unreachable_str = \"\\n\".join(_assemble_bullet_point(layer, False) \\\n",
" for layer in unreachable)\n",
"\n",
" # Asemble the message and return it\n",
" return f\"Webmap ID '{webmap_item.id}' contains unreachable URLs. \"\\\n",
" f\"You can view the webmap here: {webmap_item.homepage}\\n\"\\\n",
" f\"\\n\"\\\n",
" f\"Reachable Layers\\n\"\\\n",
" f\"-----------------\\n\"\\\n",
" f\"{reachable_str}\\n\"\\\n",
" f\"\\n\"\\\n",
" f\"Unreachable Layers\\n\"\\\n",
" f\"-------------------\\n\"\\\n",
" f\"{unreachable_str}\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, let's create the function that will be called whenever we encounter a WebMap with broken URLs. This function will assemble the appropriate people to email, assemble the message/subject to send, then send the message to the recipients."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"def handle_unreachable(webmap_item, reachable, unreachable,\n",
" gis=gis):\n",
" \"\"\"Called whenever we encounter a WebMap with broken URLs. Will \n",
" assemble an appropriate message, and send it out to the previously\n",
" configured emails.\n",
" \"\"\"\n",
" # Get the people we need to email\n",
" recipients = list(EMAIL_THESE_ADDITIONAL_PEOPLE)\n",
" if TRY_EMAIL_WEBMAP_OWNER:\n",
" owner = gis.users.get(webmap_item.owner)\n",
" recipients.append(owner.email)\n",
"\n",
" # Assemble the message and send it\n",
" message = assemble_message(webmap_item, reachable, unreachable)\n",
" subject = f\"WebMap '{webmap_item.id}' contains broken URLs\"\n",
" if send_email_smtp(recipients, message, subject):\n",
" return True\n",
" else:\n",
" log.warning(f\"Error emailing users about WebMap {webmap_item.id}.\")\n",
" return False"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> __Note__: If you have an alternative notification function you want to use, make sure you modify the above cell to include that function."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, let's create a generator function that will `yield` `Item`(s). This notebook can run against all items in an Organization, or all items from certain groups, depending on how you configured the notebook above."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"def get_items_to_check():\n",
" \"\"\"Generator function that will yield Items depending on how you\n",
" configured your notebook. Will either yield every item in an \n",
" organization, or will yield items in specific groups.\n",
" \"\"\"\n",
" if CHECK_ALL_ITEMS:\n",
" for user in gis.users.search():\n",
" for item in user.items(max_items=999999999):\n",
" # For the user's root folder\n",
" yield item\n",
" for folder in user.folders:\n",
" # For all the user's other folders\n",
" for item in user.items(folder, max_items=999999999):\n",
" yield item\n",
" else:\n",
" for group_name in CHECK_THESE_GROUPS:\n",
" group = gis.groups.search(f\"title: {group_name}\")[0]\n",
" for item in group.content():\n",
" yield item"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## main()\n",
"\n",
"Finally, let's create our `main()` function that links together all our previously defined functions to search through groups, test all webmaps for broken URLs, and alert the correct people when we find unreachable URLs."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"def main():\n",
" print(\"Notebook is now running, please wait...\\n-----\")\n",
" for item in get_items_to_check():\n",
" print(f\"\\rChecking item {item.id}\", end=\"\")\n",
" if item.type == \"Web Map\":\n",
" reachable, unreachable = test_urls_in_webmap(item)\n",
" if unreachable:\n",
" print(f\"\\nWebmap {item.id} unreachable. Notifying...\")\n",
" handle_unreachable(item, reachable, unreachable)\n",
" print(\"-----\\nNotebook completed running.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have just defined a `main()` function, but we haven't called it yet. Set up some test WebMaps with broken URLs in the specified groups, call `main()` manually, and assert that the correct people are emailed. Once you are sure that this function is working properly, do the following:\n",
"\n",
"1. __Double check the notebook content__, make sure no secrets are visible in the notebook, delete unused code, refactor, etc.\n",
"2. Place a single call to `main()` in the below cell\n",
"3. Save the notebook\n",
"4. In the 'Kernel' menu, press 'Restart and Run All'\n",
"5. Assert that the whole notebook runs without error, and assert that you are seeing the same correct behavior as before"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Notebook is now running, please wait...\n",
"-----\n",
"Webmap ee28fe0d5b8b46aba7c4b746f7bb180a unreacheable. Notifying...\n",
"SMTP server returned success for sending email to somebody@example.com\n",
"SMTP server returned success for sending email to somebodyelse@example.com\n",
"-----\n",
"Notebook completed running.\n",
"\n"
]
}
],
"source": [
"main()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Congratulations! If all is setup correctly, emails should be sent out for any WebMaps with broken URLs."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Conclusion\n",
"\n",
"This notebook provided you the workflow for checking WebMaps for broken URLs and notifying users via an external service. When configured properly, this notebook can be a powerful administrative tool to help you anticipate problems stemming from broken WebMaps. As the saying goes, \"An ounce of prevention is worth a pound of cure\".\n",
"\n",
"### Related Notebooks\n",
"For related notebooks, search for the following in your samples notebook gallery:\n",
"\n",
"- Notifications\n",
"- Identify Items That Use Insecure URLs"
]
}
],
"metadata": {
"esriNotebookRuntime": {
"notebookRuntimeName": "ArcGIS Notebook Python 3 Standard",
"notebookRuntimeVersion": "10.7.1"
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}