Key concepts for geoprocessing services

http://help.arcgis.com/en/arcgisserver/10.0/help/arcgis_server_dotnet_help/index.html#//009300000029000000 A geoprocessing service contains geoprocessingtasks accessible by Web-enabled clients. Tasks are created by publishing geoprocessing model and script tools. There are two ways to create a geoprocessing service in ArcGIS Desktop:

• Publish a geoprocessing toolbox. Each tool in the toolbox becomes a task in the geoprocessing service.
• Publish an ArcMap document containing geoprocessing tool layers. Each tool layer becomes a task in the geoprocessing service.

Geoprocessingservices and their tasks are accessed through the public Internet andprivate intranets and can be used in ArcGIS Desktop, ArcGIS Explorer,and Web applications such as a Web site built using ArcGIS ServerManager. In ArcGIS Desktop, geoprocessing services can be added to theArcToolbox window as a toolbox, and the tasks become tools within thetoolbox. The remainder of this topic is about the key design concepts, rules, and guidelines concerning geoprocessing services and tasks. Crafting a geoprocessing serviceWebclients are lightweight applications—they only know how to send packetsof simple data to a server, such as text, numbers, and uncomplicatedgeographic features. A geoprocessing service takes this simple data;processes it; and returns some meaningful and useful output, such asthe probable evacuation area for a hazardous chemical spill, thepredicted track and strength of a gathering hurricane, a map of landcover within a user-defined watershed, a parcel map with historicaldetails of ownership, or a permit for a parade route through downtown.The possibilities for these services are infinite. Geoprocessingprovides the base of rich and powerful tools from which you craft aservice that processes input data. If you currently use geoprocessing,you may already understand how to create tools using models andscripts. What you may need to learn is how to craft your tool to usethe simplest possible input data to reach the largest possible audienceof clients. Forexample, suppose you have a model that computes an upstream watershedfrom a set of points and extracts polygons within the computedwatershed. The input parameters for this model would be

• A raster digital elevation model (DEM)
• A point feature class
• A polygon feature class

This model is appropriate for ArcGIS Desktop but not for a geoprocessing service. It would need to be modified as follows:

• Insteadof operating in a study area defined by the user, it would work in aspecific study area. Since the model has a known study area andoperates on a specific DEM, the DEM is no longer an input parameter.
• As ageoprocessing service, clients cannot upload feature classes. Instead,the user will digitize points in the client application that representwatershed pour points. The service would snap these points to the DEMusing a precomputed snap distance appropriate for the resolution of theDEM.
• Insteadof extracting polygons from an input feature class, the model wouldoperate on a known set of data, such as land-cover polygons. The outputwould be land-cover polygons within the computed watershed. The inputpolygon is no longer needed.

The modifiedmodel takes simple inputs and answers a specific spatial query: For thestudy area, what is the land cover within the watershed defined by thispoint? The model would be published as a geoprocessing service and usedin a Web site meant for land managers in the study area. This is not to say that you cannot build generic services using ArcGIS Server. The example buffer points serviceis a wholly generic service that buffers any set of point features.ArcGIS Server is flexible, and with the use of some advanced techniquesor custom programming, you can build generic services that processlarge datasets submitted by users. But the majority of services arefocused on specific geographic areas, answer specific spatial queries,and work on lightweight clients. The design of ArcGIS Servergeoprocessing services is driven by the need to build and serve thesetypes of focused services. Geoprocessing service configurationsGeoprocessingservices can be created by publishing two different ArcGIS Desktopresources; a geoprocessing toolbox or an ArcMap document (.mxd)containing tool layers.

• When you publish a toolbox, all tools within the toolbox become geoprocessing tasks within the geoprocessing service.
• When you publish an ArcMap document, all tool layerswithin the map document become geoprocessing tasks within thegeoprocessing service. (Tool layers are created by dragging anddropping tools into the ArcMap table of contents.)
• Whenpublishing an ArcMap document containing tool layers, you can alsospecify that you want the ArcMap document to become a map service thatwill be used to draw the output of tasks. A map service that draws taskoutputs is called a result map service.

These three configurations are illustrated below. Geoprocessing service from a toolboxWhenyou publish a toolbox, all tools within the toolbox becomegeoprocessing tasks. Data output by tasks is transported back to theclient. Geoprocessing services with a source map documentIfyou have used geoprocessing tools in an ArcMap session, you know thattools can often use layers found in the ArcMap table of contents, aswell as data on disk. Inthe same way, your geoprocessing task can use layers found in itssource map document. The source map document, in this case, acts as acontainer of layers. You can make layers in the source map documentinput parameters to your task. In the graphic below, the Data to extract variable is an input parameter that allows the user to choose layers in the source map document. Caution:Geoprocessing taskscan only access layers found in its source map document—they cannotaccess layers found in other map services or in the client application. There areperformance benefits to using layers from a source map document in yourmodel or script processes. The illustration below shows a model thatuses a network dataset, StreetsNetwork, to construct a route analysis layer. The StreetsNetworkvariable can either reference a layer (which it does in this case) or adataset on disk. Opening a network dataset is expensive relative toother kinds of datasets because network datasets contain severaladvanced data structures and tables that must be read and cached. Byusing the layer instead of the dataset, there is a performanceadvantage, because ArcMap opens the dataset once, caches basicproperties of the dataset, and keeps the dataset open. When the modelexecutes, the dataset does not have to be reopened, since the sourcemap document already has it opened—a performance boost. Conversely, ifthe StreetsNetwork variable directly referenced the dataset, the dataset would be opened each time the model executes—a performance degradation. Fornetwork analysis, you always want the network dataset as a layer in thesource map document and to use that layer in model variables. For otherkinds of datasets, such as features and rasters, the performanceadvantage of using layers in the source map document is very slight. Geoprocessing services with a result map serviceGeoprocessingservices can have a result map service to create a digital map image oftask results. Digital maps contain visual representations of geographicdatasets that communicate information. Digital maps are transportedacross the Web as images (such as a .jpg).A map image, byte for byte, contains far more human-interpretableinformation than raw features in a feature class. Map images are alsomanageable—they are easily compressed, they can be tiled intomanageable chunks, and there are established methods for transportingand viewing them across the Web. Map images are created by an ArcGIS Server map service and are the result of publishing an ArcMap document (.mxd).Because of the characteristics of a map image, you may want to createone for the results of your geoprocessing task and transport the imageacross the Web rather than transporting the result dataset or datasets.Geoprocessing services can have a result map service used by ArcGISServer to create map images of your output data. Result map services should be used when

• The result of your task is a (potentially) large dataset.
• Thedata type of your output is unsupported by the client, such as rastersin ArcGIS Explorer. In this case, you use the result map service todisplay the output.
• You want to protect the result of your task by allowing it to only be viewed as a map and not downloaded as a dataset.
• You have complex cartography that needs to be drawn by the result map service and not by the client.

When you useresult map services, it is important to realize that there are twoservices—the geoprocessing service and the result map service. Thesetwo services execute independently of each other. When the taskexecutes, ArcGIS Server executes the geoprocessing task first, thenexecutes the result map service to draw the output of the geoprocessingservice. Because of this execution order, the result map service needsdatasets on disk produced by the geoprocessing service. This means thatthe output of the tasks in the geoprocessing service must be datasetson disk, not layers or in-memory datasets. Basemaps for your geoprocessing serviceFormost geoprocessing tasks, your user will need some sort of basemap touse as a geographic reference. It may be a basemap of roads, populatedplaces, points of interest, or any other features. Yourgeoprocessing service probably relies on a particular basemap to guideyour user when entering locations. For example, your user inputs apoint that must fall within a parcel boundary in a specific city, sothe basemap should show that city's parcel boundaries. Furthermore,your service may only work within a certain study area, as opposed to aservice that works globally. The study area can be thought of as the geoprocessing extent since the service only has knowledge of data within the study area. Note:A common mistake is to use the result map service to display a basemap for your geoprocessing service.For example, if your user identifies a parcel by point-and-click, andyour task draws the identified parcel symbolized by some attribute ofthe parcel, your initial thought would be to use the result map serviceto display the input parcel data (the basemap) as well as theidentified parcel. There are two reasons why you should not use theresult map service as a basemap:

• When aresult map service is added to the application, all layers in the mapservice are available for display. These layers include geoprocessingtool layers used to draw output, layers that may contain sensitivedata, or layers used by your geoprocessing service but that make nosense to the user (like tool layers).
• Basemapsare multiscale and multiresolution. As you zoom in and out, the basemapchanges, showing details at large scales and aggregating details intogeneralizations at small scales (for example, rivers change from linefeatures at small scale to polygon features at large scales).Constructing a multiscale and multiresolution basemap that drawsquickly is not something your result map service needs to be dealingwith—its job is to draw outputs. You need to keep the design andimplementation of basemap map services separate from the design andimplementation of result map services.

Returningto the point-and-click parcel application, you should have a mapservice to display the parcel data (the basemap) and use the result mapservice to return the parcel for display, perhaps color-coded by someattribute. Both map services use the same parcel dataset (there are noissues with this), and you divide the work of displaying a referencebasemap from the work of displaying results. Anotherconsideration in designing map services is the characteristics of theclient. For a Web application, you have full control over what map andgeoprocessing tasks will be available in the application, and resultmap services need not appear as a map layer in the table of contents ofthe Web application. ArcMap and ArcGIS Explorer clients are a bit moreproblematic since, in practice, users can browse to any map orgeoprocessing service and end up with mismatches between the basemapextent and the geoprocessing extent. When publishing services, there isno option where you can specify "when adding this geoprocessingservice, also add these other map services." You can, however,distribute ArcMap documents (.mxd) or ArcGIS Explorer documents (.nmf) that contain the correct services. You can also provide built-in task documentationfor your geoprocessing services and tasks detailing what map servicesare needed. Task documentation is accessible by all clients. Data types and capabilities of the clientArcGISExplorer is a lightweight client application, meaning it has a smallinstallation, unlike ArcGIS Desktop. Web applications are Web sitesaccessed with an Internet browser. Browsers are very lightweight (orthin) clients. Because of the lightweight nature of these clients, thefull range of input and output data typesyou find on ArcGIS Desktop are not available to these clients(otherwise, the client would be heavyweight, like ArcGIS Desktop). Forexample, lightweight clients do not support rasters as input to ageoprocessing task. Note:Sinceprocesses within your published model or script execute on the serverwhere all data types are available, you can use any data type for modelor script processes, as illustrated below. It's only the input andoutput parameter data types that are limited. Any type of data that canbe accessed by the server can be used by your model or scriptprocesses.The following table summarizes key input parameter data types for the three clients. Input parameter data type Supported on ArcGIS Desktop clients? Supported on ArcGIS Explorer client? Supported on Web application clients? Feature Set Yes Yes Yes Record Set Yes Yes Yes Feature Class No (But Feature Class input is supported indirectly with the Feature Set data type.) No No Table No (But Table input is supported indirectly with the Record Set data type.) No No Raster Yes No No Standard types (such as Long, Double, Boolean, Date, String) and Linear Unit (for example, "1000 meters") Yes Yes Yes File (such as a .zip or .xml file) Yes Yes Yes Layer (any type of layer; for example, Feature Layer, Raster Layer, Network Analyst Layer) Only those layers found in result map service or source map document Only those layers found in result map service or source map document Only those layers found in result map service or source map document Input types for geoprocessing servicesAny data type not listed above is either converted to a string data type or not allowed. The topic Input and output data types goes into greater detail about data types for geoprocessing services. Verylikely, your existing models and scripts take feature classes andtables as input, since these are the most common dataset types usedwith geoprocessing. This means these existing models and scripts willhave to be modified before publishing them as geoprocessing tasks. Ifyour model or script takes a feature class input, you can modify it totake a feature set instead. If your model takes a table as input, youcan modify your model to take a record set instead. The following table summarizes the key output parameter data types for the three clients. Output parameter data type Supported on ArcGIS Desktop clients? Supported on ArcGIS Explorer client? Supported on Web application clients? Feature Class Yes Yes Yes Raster Yes No (Can only be displayed in the map through the use of a result map service) No (Can only be displayed in the map through the use of a result map service) Table Yes No. (Services that have a table data type as an output parameter will not be shown in the list of available tasks.) Yes Standard types (such as Long, Double, Boolean, Date, String) and Linear Unit (for example, "1000 meters") Yes (Viewed in the service result found in the Results tab of the ArcToolbox window) Yes (Viewed in the Task Result) Yes File Yes Yes Yes Output dataConcurrent use of data—%scratchworkspace%Ageoprocessing task can be used concurrently (at the same time) byseveral users. For data that is read by your model or script, there areno concurrency issues—concurrent users can all read the same data.However, data created or updated by your service requires that youunderstand issues regarding concurrent writers. Creating new dataGeoprocessing tasks typically create intermediate dataand output data. ArcGIS Server provides a mechanism that insures thatthere are no concurrency issues with intermediate and output data. Whena task is executed, ArcGIS Server creates a unique job folder in a jobsdirectory. This jobs folder contains a folder named scratch, whichfurther contains a file geodatabase named scratch, as illustratedbelow. After creating this folder structure, ArcGIS Server sets thegeoprocessing scratch workspace environment variableto the scratch folder (not the scratch geodatabase, but the folder).Your model and scripts can easily discover and use this scratchworkspace by placing percent signs around the environment variable name(%scratchworkspace%). You must write your intermediate data and output data to either the scratch folder or the scratch geodatabase. There are two methods you can use to ensure that data is written to the scratch folder or geodatabase:

• In ModelBuilder, right-click any intermediate data variable and choose Managed.Caution:Do not set output variables to Managed, only intermediate variables.
• Use variable substitution (%scratchworkspace%) for paths. For example:%scratchworkspace%/templines.shp %scratchworkspace%/scratch.gdb/outWatershed

You can alsowrite intermediate data to memory. Writing data to memory is fasterthan writing to disk. If you write intermediate data to memory, it isnot necessary to make the model variable intermediate or managedbecause the data will be deleted once the task executes. Updating existing dataAny task that updates existing data is of particular concern. Forexample, a task within a service may update an existing table orfeature class, adding new rows or features (using the Append tool, for example), or updating existing attributes (using the Calculate Fieldtool, for example). If the geoprocessing service is configured to allowmultiple concurrent instances (users), and multiple instances arerunning, you may have a collision of multiple instances trying toupdate the same dataset. In these cases, you must limit the number ofinstances to one when configuring the service. This way, ArcGIS Serverwill queue the requests and only one instance at a time will access thedata. Another problem is locking—if you are updating a dataset and thedataset is also a layer in a map service, the map service will put alock on the data, and the update will fail. Do not create map servicesthat display data that is updated by another service. Symbolizing dataYou have two choices for drawing a task's output data:

• The client draws the data.
• The result map service draws the data.

When the clientdraws the output data, two pieces of information are sent to theclient: the data and a layer drawing description. The layer drawingdescription contains the information you specify in the Symbology tab of a layer's Propertiesdialog box. This information includes how to group data (the layersymbology) and what symbols to use (symbol types). Only certain layersymbologies and symbol types are supported by clients. Whena result map service draws the data, the layer symbology and symboltypes found in the corresponding tool layer are used. When using aresult map service, you can use any layer symbology and symbol type,since ArcMap (running on the server) will be drawing the data andtransporting an image of the completed map back to the client. Thecapabilities of the client do not affect how ArcMap draws the result. Learn more about defining output symbology for geoprocessing tasks ValidationIfyou have used geoprocessing tools in ArcGIS Desktop, you have probablyseen how geoprocessing tools validate your inputs. One example ofvalidation is a list of fields changing when an input table changes. Other examples of validation include warnings () and errors (), changing of default values based on input data, and the enabling and disabling of parameters. Note:No validation occurswith geoprocessing tasks. For example, suppose you create a model thathas an input layer parameter and an input field parameter. The list offields shown in the field parameter is dependent on the value of thelayer parameter. In ArcGIS Desktop, if you change the layer, the listof fields changes. This is not the case with a task:

• The list of fields will contain the list at the time you published.
• If the user of your task picks another layer, the list of fields will not refresh.