{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Administration: Validate registered datastores\n", "\n", "> * ✏️ Needs Configuration\n", "* 🔒 Requires Administrator Privileges\n", "* 🗄️ Administration\n", "* 🔔 Notifications\n", "\n", "Enterprise installations must have data storage configured in order to publish hosted services. Configured datastores are also used to store layers created as output from certain analysis tools or used as caches for scenes. There are numerous types of data stores that can be [registered](https://enterprise.arcgis.com/en/server/latest/publish-services/windows/overview-register-data-with-arcgis-server.htm) with an Enterprise, including folders, databases, geodatabases, and cloud stores. Data registration is how the ArcGIS Server component of an Enterprise installation knows where to access data used for services." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once the datastores have been configured and registered in the Enterprise, regular validation ensures the health and status of the stores. Checking the status of the datastores often serves as the best initial troubleshooting step if hosted layers cannot be published or accessed." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get started, let's import the necessary libraries and connect to our GIS." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "import csv\n", "import smtplib\n", "import logging\n", "log = logging.getLogger()\n", "\n", "import requests\n", "import pandas as pd\n", "\n", "from arcgis.gis import GIS\n", "\n", "gis = GIS(\"home\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Notifications\n", "\n", "An important part of this process is notifying the right personnel about servers that are not validating. We will achieve this by connecting to an external SMTP server and sending out emails, but you can write _any_ code that connects to _any_ external service to send out notifications. __View the 'Notifications' notebook in the examples gallery for more information__.\n", "\n", "## Security\n", "\n", "Before writing code that connects to any external service, it is vital to keep security in mind. Notebooks are commonly shared, code is copied and pasted, and passwords and private keys can inadvertently end up in the wrong hands.\n", "\n", "**Passwords and private keys should NEVER be stored in plaintext in a notebook**. There are many paradigms to securely store passwords and private keys. We will be storing all 'secrets' in a [private CSV item](https://www.esri.com/arcgis-blog/products/arcgis-online/sharing-collaboration/managing-security-and-findability-of-items-with-the-arcgis-sharing-model/) that is only accessible by you and your GIS administrator. \n", "\n", "> __Note__: that although this is more secure than storing your passwords as a plain string in a notebook, this is __NOT__ a totally secure solution—items aren't encrypted in the underlying server where they are stored. Consider storing your secrets in an encrypted files in `/arcgis/home/`.\n", "\n", "__You MUST modify the below cell__ to update `secret_csv_item_id`, `smtp_server_url`, `username`, and any other information needed to connect to your external smtp server. This includes making sure your _secrets.csv_ file item contains the `smtp_email_password` entry." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "SECRET_CSV_ITEM_ID = \"\"\n", "def get_secrets(gis=gis,\n", " secret_csv_item_id = SECRET_CSV_ITEM_ID):\n", " \"\"\"Returns the secrets.csv file as a dict of secret_key : secret_value\"\"\"\n", " item = gis.content.get(secret_csv_item_id)\n", " with open(item.download(), 'r') as local_item_file:\n", " reader = csv.DictReader(local_item_file)\n", " return { rows['secret_key'] : rows['secret_value'] for rows in reader}\n", "\n", "def send_email_smtp(recipients, message,\n", " subject=\"Message from your Notebook\"):\n", " \"\"\"Sends the `message` string to all of the emails in the \n", " `recipients` using the configured SMTP email server.\"\"\"\n", " # Set up server and credential variables\n", " smtp_server_url = \"smtp.mail.yahoo.com\"\n", " smtp_server_port = 587\n", " sender= \"someone@example.com\"\n", " username = \"someone@example.com\"\n", " secrets = get_secrets()\n", " password = secrets[\"smtp_email_password\"]\n", "\n", " # Instantiate our server, configure the necessary security\n", " server = smtplib.SMTP(smtp_server_url, smtp_server_port)\n", " server.ehlo()\n", " server.starttls() # Needed if TLS is required for the SMTP server\n", " server.login(username, password)\n", "\n", " # For each recipient, construct the message and attempt to send\n", " did_succeed = True\n", " for recipient in recipients:\n", " try:\n", " message_body = '\\r\\n'.join(['To: {}'.format(recipient),\n", " 'From: {}'.format(sender),\n", " 'Subject: {}'.format(subject),\n", " \"\",\n", " '{}'.format(message)])\n", " print(f\"{message_body}\")\n", " server.sendmail(sender, [recipient], message_body)\n", " except Exception as e:\n", " log.warning(\"Failed sending message to {}\".format(recipient))\n", " log.warning(e)\n", " did_succeed = False\n", " \n", " # Cleanup and return\n", " server.quit()\n", " return did_succeed" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Access the Server Manager" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Note:** You must connect as an administrator to perform these steps.\n", "\n", "Next, let's use the `servers` property of the GIS [`admin`](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.admin.html# ) module to instantiate the [`ServerManager`](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.server.html#servermanager). We can access options and resources to help automate many administrative tasks from this object." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "server_mgr = gis.admin.servers\n", "server_mgr" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## List Servers\n", "\n", "The [`list()`](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.server.html#arcgis.gis.server.ServerManager.list) method returns all the servers accessible in the GIS. " ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[,\n", " ]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fed_servers = server_mgr.list()\n", "fed_servers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The list returns us the administrative url for each [`server`](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.server.html#). We can access a number of different properties of the `Server` which allow us to manage our Enterprise." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## List Datastores" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For each server, we'll use the [`datastores`](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.server.html#arcgis.gis.server.Server.datastores) property to first check if the server contains datastores to manage. If so, we'll work with the [`DataStoreManager`](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.server.html#datastoremanager) to print information about each of the individual datastore items that have been [`registered`](http://enterprise.arcgis.com/en/server/latest/publish-services/linux/overview-register-data-with-arcgis-server.htm) with the server." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://datascienceqa.esri.com/server/admin\n", "--------------------------------------------------\n", " Datastore Type Datastore ID Datastore Path\n", " egdb 36395d13-ebd8-4f48-94f6-b711c46e0d63 /enterpriseDatabases/AGSDataStore_ds_sn8a7utk\n", " nosql 9d377967-25a3-44cb-8443-42d532651fca /nosqlDatabases/AGSDataStore_cache_cs_054d0h2\n", "\n", "\n" ] } ], "source": [ "for server in fed_servers:\n", " print(f\"{server.url}\\n{'-'*50}\")\n", " if server.datastores:\n", " print(f\"{' ':2}{'Datastore Type':25}{'Datastore ID':50}{'Datastore Path'}\")\n", " for datastore in server.datastores.list():\n", " print(f\"{' ':3}{datastore.type:25}{datastore.id:50}{datastore.path}\")\n", " else:\n", " print(\"\\tServer has no datastores\")\n", " print(\"\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Validate Datastores" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `datastore` class in the API contains a [`validate()`](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.server.html#arcgis.gis.server.Datastore.validate) function that returns either `True` or `False` when run on a datastore.\n", "\n", "Let's define a helper function to return information on any server that prohibits validation or fails to validate:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def validate_datastores(ds):\n", " if not ds.type == \"egdb\":\n", " return {\"datastore_id\": ds.id,\n", " \"validate_message\": \\\n", " \"{} datastore cannot validate.\".format(ds.type)}\n", " else:\n", " if not ds.validate():\n", " return {\"datastore_id\":ds.id, \n", " \"datastore_type\":ds.type, \n", " \"datastore_path\":ds.path}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **NOTE:** See [Validate data stores](http://enterprise.arcgis.com/en/server/latest/publish-services/linux/registering-your-data-with-arcgis-server-using-manager.htm#ESRI_SECTION1_8E7DC72D829144EEBC71B9F01DCE306D) for more information." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can run a preview of the function results and visualize them in a dataframe:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
datastore_idvalidate_message
09d377967-25a3-44cb-8443-42d532651fcanosql datastore cannot validate.
\n", "
" ], "text/plain": [ " datastore_id validate_message\n", "0 9d377967-25a3-44cb-8443-42d532651fca nosql datastore cannot validate." ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.set_option(\"max_colwidth\", 80)\n", "\n", "for server in fed_servers:\n", " if hasattr(server, \"datastores\"):\n", " for datastore in server.datastores.list():\n", " validate_results = validate_datastores(datastore)\n", " if validate_results:\n", " df = pd.DataFrame([validate_results])\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can use this function to regularly validate the datastores in the Enterprise and send an email notification to the appropriate personnel with messaging regarding any server that fails." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for server in fed_servers:\n", " if server.datastores:\n", " for datastore in server.datastores.list():\n", " if validate_datastores(datastore):\n", " val_results = validate_datastores(datastore)\n", " val_results[\"server\"] = server.url\n", " val_results[\"datastore_id\"] = datastore.id\n", " send_email_smtp(recipients=[\"someone@example.com\"],\n", " message='\\n'.join([k + \" : \" + v for k,v in val_results.items()]),\n", " subject=\"Email from Python API Validate Datastores Script.\")" ] } ], "metadata": { "esriNotebookRuntime": { "notebookRuntimeName": "ArcGIS Notebook Python 3 Standard", "notebookRuntimeVersion": "10.7.1" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.2" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }