Jenness Enterprises
Wildlife and Forestry Tools for ArcGIS Pro — Tool Reference
jennessent.com
Tool Reference › GIS Concepts › Projection Distortion

Projection Distortion and Terrain Analysis

GIS Concepts · background · by Jeff Jenness

Slope is a rise divided by a run. The rise comes from the elevations in the DEM; the run is the distance between the center points of two cells on the ground. Every tool that derives something from a DEM — slope, aspect, a hillshade, surface area, ruggedness, the wetness index — needs that distance, and the obvious place to get it is the raster's cell size. On a projected DEM the cell size is a number in the projection's linear unit, meters or feet, and it is tempting to treat it as the distance on the ground. It is not, or not quite. Every map projection stretches or shrinks distances by some amount that changes from place to place, and a cell that is 30 m wide on the map can actually be only 25 m or 8 m wide on the ground. This page explains how large the effect is, which projections suffer from it, and how every tool in this add-in now measures its distances on the spheroid, in any coordinate system, so that the effect drops out.

The short version Nothing to set. Every DEM tool in Wildlife and Forestry Tools measures the distances between cells on the Earth's spheroid, whether the DEM is in latitude and longitude, UTM, State Plane, Albers, Web Mercator or a polar projection. A Web Mercator DEM gives the same slopes, aspects, hillshades and roughness values as the UTM or lat/long DEM it was made from. The run messages of each tool report the projection's scale factor across the raster and how it was handled. Version 1.113.62.1 (October 2026) made this the behavior of every tool; before it, projected DEMs were read with their nominal cell sizes.

What the cell size gets wrong

A map projection's scale factor at a point is the ratio of a distance on the map to the same distance on the ground. Where it is 1 the map is true to scale; elsewhere map distances are too long (scale factor above 1) or too short (below 1). The projections designed for regional work keep it close to 1: a UTM zone runs from 0.9996 on its central meridian to no more than 1.001 at its edges, State Plane zones are built to a tighter tolerance still, and a continental Albers or Lambert conic stays within about a percent of 1 over the region it was designed for (Snyder 1987). A planar method that uses the nominal cell size on those systems is wrong by a few hundredths of a percent to a percent, which no one would notice in a slope map.

Web Mercator is one important exception that matters, because so much data now arrives in it. Mercator's scale factor is 1 at the equator and grows as 1 / cos(latitude): 1.22 at Flagstaff, 1.41 at 45° and 2 at 60°. A “30 m” Web Mercator cell at Flagstaff's latitude is only 24.5 m across on the ground east–west and 24.4 m north–south. Divide a rise by 30 when the true run is 24.5 and every gradient comes out 18 percent too small, and the error grows toward the poles:

LatitudeGround width of a “1000 m” Web Mercator cellA true 30° slope reads as
0°1000 m30.0°
35.2° (Flagstaff)818 m25.3°
45°708 m22.2°
60°501 m16.1°
75°259 m8.5°
Line chart of the slope a planar method reports against latitude on a Web Mercator DEM, for true slopes of 10, 30 and 45 degrees; each curve falls away from its dotted true value as latitude increases, reaching about half the true tangent at 60 degrees
What a planar method reports on a Web Mercator DEM, against latitude, for true slopes of 10°, 30° and 45° (dotted lines). The tangent of the reported slope is the true tangent times the cosine of the latitude, because each cell is that much smaller on the ground than its nominal size. The curves hold for the whole gradient of a 3 × 3 neighborhood, not only its east–west part: Web Mercator shortens a cell by very nearly the same factor east–west and north–south, so the understatement is the same whichever way the slope faces.
Three map panels of the same stretch of the Grand Canyon. The left and middle panels are slope rasters on the same blue-to-red ramp from 0 to 80 degrees, labelled Slope, Geodesic Method and Slope_Grand_NominalPlanar; the middle panel is slightly paler. The right panel, labelled Difference, is a gray-scale raster from -6.5 to 14.5 degrees, dark on the flat canyon floor and along the river, light on the canyon walls, with scattered light and dark cells along the cliff edges.
The understatement on a real DEM: part of the Grand Canyon in Web Mercator at 36°N, where the scale factor is 1.24 and the 34.4 m cells are 27.8 m on the ground. Left: slope from Esri's geodesic method, which the projection cannot affect. Middle: Horn's gradient at the nominal cell size, labelled with its dataset name, on the same 0 to 80° ramp. Every cell in it is too low, by 3.5° on average, but a shift that size is hard to see on an 80° ramp, and the middle panel is only a little paler than the left. Right: the difference, geodesic minus nominal. It is near zero on the flat canyon floor, where a small tangent loses little, climbs to about 6° where the true slope is between 40° and 60°, and falls back to about 4° on the steepest cliffs, where the tangent is so large that dividing it by 1.24 moves the angle less. The scattered light and dark cells along the cliff edges, and the few cells below zero, are not the projection's doing: they are where Esri's geodesic plane fit and Horn's gradient disagree about a cell straddling a cliff, the difference discussed under How this compares with Esri's tools. Against Esri's planar result, which uses the true cell size, the nominal raster is low by 0 to 6.2° with no cell below zero.

Aspect is barely affected by Web Mercator, because the distortion is very nearly the same in every direction there (the two directions differ by less than a percent, which can move an aspect by a fifth of a degree at most), but anything that depends on the magnitude of the gradient is affected: slope, the hillshade's brightness, surface area and its ratio, the vector ruggedness measure, the terrain ruggedness index, the wetness index, and the size in cells of any neighborhood given in meters. A 500 m radius converted with the nominal cell size at 60° north spans half the cells it should.

Geographic (latitude/longitude) DEMs have a different version of the same problem, well known and handled for years by the tools here: a degree of longitude is a different distance at every latitude, so the ground size of a cell changes from row to row of the raster. The hillshade, roughness and TPI tools have always computed each row's true cell dimensions on the spheroid for a geographic DEM, and that per-row treatment is exact there, because on a spheroid of revolution every column is the same.

Where the scale factor changes

How a projection's scale factor varies across a raster depends on the projection's family, and the pattern decides what a correction has to do.

Five square panels shaded by scale factor: horizontal bands for normal cylindrical projections, concentric arcs for conic, vertical bands for transverse cylindrical, concentric circles for azimuthal, and diagonal bands for oblique cylindrical; a regular grid of gold dots overlies each
How the scale factor varies across a raster in five families of projection. The shading is schematic: lighter where the scale factor is smallest, darker where it is largest, with the white lines joining points of equal scale factor. Whether a shade stands for a factor below or above 1 depends on the projection: the factor is 1 on a projection's standard line or lines, below 1 between two standard lines and above 1 beyond them, so the panels show the pattern of change, not the distance from 1. The gold dots are the lattice of cell centers that the tools unproject, described in the next section.

The chart is a sketch. The surest way to see a projection's distortion is to draw something whose true size is known. The figure below takes a grid of circles, each 100 km in radius on the ground, and draws it in four projections, so that every circle takes the size and shape its projection gives that place.

Four map panels of North America, each scattered with yellow circles that are all 100 km in radius on the ground. Web Mercator, top left: the circles grow larger toward the north. Polar stereographic, top right: the circles are smallest around the North Pole and grow away from it. UTM zone 12, bottom left: the circles stay round and equal across the continent, and the map's edges curve away far from the zone. Albers equal area, bottom right: the circles stay the same size but are flattened into ellipses in the far north.
The same grid of geodesic circles, each 100 km in radius on the ground, drawn in four projections. In Web Mercator (top left) the circles stay round but grow steadily northward, twice their equatorial size by 60°N. In polar stereographic (top right) they are smallest near the pole and grow away from it. In UTM zone 12 (bottom left) they stay round and the same size across the continent and distort only far outside the zone, toward the panel's east and west edges. In Albers equal area (bottom right) every circle keeps its area, but far north of the projection's standard parallels the circles are flattened into ellipses.

How the tools measure distances

Rather than correct the cell size by a formula for each projection, the tools go back to the Earth. Each tool takes a lattice of cell centers across the raster — one node every 16 cells or so in each direction, never more than 257 per axis, always including the last row and column — and unprojects each node to latitude and longitude with the coordinate-system library that ArcGIS Pro itself ships (GDAL's osr; see below). The node and its neighboring cell centers are then converted to earth-centered three-dimensional coordinates on the DEM's own spheroid, and the five distances the gradient needs — east-west across the row above, the row itself and the row below, and north-south up and down — are measured there as straight chords, exactly as they always were for a geographic DEM. Every other cell's five distances are then interpolated bilinearly between the four nearest nodes. The spacing stays at 16 cells until the raster's longer side passes 4,096 cells; beyond that it is the longer side divided by 256, rounded up, so a DEM 10,000 cells across gets a node every 40 cells. The lattice therefore never holds more than 257 × 257 nodes, 66,049 cell centers, however large the DEM.

A grid of raster cells shaded by scale factor, with gold lattice nodes every four cells and one cell outlined in red; dashed lines connect the outlined cell to the four nearest nodes, labeled: this cell's spans are bilinear between the four nearest nodes
Each lattice node is unprojected and its five spans to the neighboring cells are measured on the spheroid: east–west across the row above, the node's own row and the row below, and north–south to the cell above and the cell below, the five distances the gradient needs. Every other cell takes its five spans by bilinear interpolation between the four nodes around it. On this sketch the nodes are four cells apart; in the tools they are sixteen or more.
GDAL, and where the tools get it GDAL, the Geospatial Data Abstraction Library, is the open-source library that much of the GIS world relies on to read and write raster and vector data formats and to convert between coordinate systems; QGIS, GRASS GIS and R's terra package are built on it, and it is maintained under the Open Source Geospatial Foundation (gdal.org). Its coordinate-system module, osr (short for OGR Spatial Reference; OGR is the vector half of GDAL), draws on the PROJ library, which holds the definitions of the world's projections and datums. ArcGIS Pro ships GDAL's Python bindings (the osgeo package) in its own Python environment, next to arcpy, numpy and scipy, and that copy is the one the tools use: they read the DEM through it and unproject the lattice nodes with its osr module. Nothing is installed or downloaded; the library is already part of every ArcGIS Pro installation (Pro 3.7 ships GDAL 3.12).

This handles every family in the figure above, because it makes no assumption about how the scale factor varies: azimuthal and oblique projections, whose scale changes along the rows, come out as right as Web Mercator. Scale factors vary smoothly, so the interpolation between nodes is accurate to better than a thousandth of a percent even on polar stereographic data, and the whole lattice for a raster of thousands of cells on a side takes well under a second to build. A geographic DEM needs no lattice and keeps its exact per-row table. If a DEM's coordinate system cannot be unprojected at all — an unusual custom projection, or no projection information — the tool falls back to the nominal cell size and says so in a warning.

The run messages of every tool report what happened, with the projection's name, the size of the lattice and the range of the scale factor across the raster, for example: “Projected DEM (WGS 84 / Pseudo-Mercator; Mercator 1SP projection): true ground distances computed on the spheroid by unprojecting a lattice of 19 x 19 cell centers and interpolating between them. The projection's scale factor ranges from 1.2219 to 1.2284 across the raster, so planar cell sizes (30.00 x 30.00 m) would put gradients and distances off by up to 18.59%.” The metadata of each output raster carries the same information.

The run messages of Extract Slope Values to Points in ArcGIS Pro, with one paragraph highlighted in yellow: Projected DEM (WGS 84 / Pseudo-Mercator; Mercator 1SP projection): true ground distances computed on the spheroid by unprojecting a lattice of 69 x 109 cell centers and interpolating between them. The projection's scale factor ranges from 1.2346 to 1.2442 across the raster (1.2367 east-west and 1.2421 north-south at the center), so planar cell sizes (34.41 x 34.41) would put gradients and distances off by up to 19.63%. Above it a warning says the points and the DEM are on different datums and the points were projected onto the raster.
The run messages of Extract Slope Values to Points on the Web Mercator DEM of the Grand Canyon shown above, with the projected-DEM line highlighted. The lattice here is 69 × 109 cell centers, the scale factor runs from 1.2346 to 1.2442 across the raster, and the nominal 34.41 m cells would have put gradients off by up to 19.63 percent. The line quotes two factors at the raster's center, 1.2367 east–west and 1.2421 north–south, because Web Mercator is not quite conformal: it applies the sphere's Mercator formulas to latitudes measured on the WGS84 ellipsoid (the “auxiliary sphere” in the coordinate system's full name), which stretches a meridian a little more than a parallel, by 0.44 percent at this latitude and by 0.67 percent at most, at the equator. The lattice measures that difference directly in its north–south spans; the closed-form factors at this latitude are 1.2368 and 1.2422. The warning above the highlighted line concerns the points, which are in NAD 1983 while the DEM is in WGS 1984, and has nothing to do with the projection.

One group of quantities still uses a single column: the number of cells a neighborhood radius given in meters converts to, for the roughness, TPI, blur and focal tools. A radius is a whole number of cells per row in any case, and a difference of a percent or two in it, on the azimuthal projections where the single column is an approximation, does not change the result in any way you could see.

Why not one representative column

The shortcut that first suggests itself is to unproject a single column of cell centers, say the middle one, and use its per-row distances for every row — the same per-row table a geographic DEM uses. That is exact for the normal cylindrical family, where the scale factor really is the same along every row, and close enough for conic and transverse systems, whose distortion is small by design. It fails for the azimuthal and oblique families, and not gracefully. On a polar stereographic DEM of Antarctica, a row that passes through the pole has its smallest scale factor at the middle and its largest at the coasts; using the middle value everywhere would correct the coastal cells in the wrong direction, by more than the planar error it set out to remove. The lattice costs little more and has no such failure case, which is why the single column method was rejected.

Aspect and grid north

The distance correction fixes the magnitude of the gradient. Its direction is a separate matter. Aspect is measured on the raster's grid, clockwise from the top of the raster, which is grid north; and on a projected DEM grid north differs from true north by the grid convergence, an angle that depends on the projection and on where the cell sits. It is under a degree across most of a UTM zone, up to about ten degrees at the east and west edges of a continental Albers or Lambert projection of the United States, and on a polar stereographic map it is simply the difference in longitude from the central meridian. Esri's Aspect tool reports aspect from grid north in the same way, and so do the aspect values this add-in's tools derive from a DEM, so that the two agree cell for cell. The values are right for anything that compares cells with one another on the same grid. Where a true-north direction is needed, add the convergence angle at each point; Esri's Surface Parameters tool, which works in geocentric coordinates, reports true-north aspects directly.

How this compares with Esri's tools

Esri's Slope and Aspect tools offer a planar and a geodesic method, and its Surface Parameters tool is geodesic throughout: the geodesic methods convert the cells to earth-centered coordinates and compute on the ellipsoid, so a projection's distortion never enters (Esri, How Slope works; How Surface Parameters works). In tests for this page, Esri's planar Slope in ArcGIS Pro 3.7 also came out within a fifth of a degree of the true slope on Web Mercator and on polar stereographic DEMs, so Pro's planar method evidently allows for projected scale as well; on a lat/long DEM, where Esri's own documentation says to use the geodesic method, its planar method was off by two degrees at Flagstaff's latitude and ten degrees at 75° north. The tools here do not try to reproduce Esri's geocentric arithmetic. They keep the familiar Horn 3 × 3 gradient and the eight-triangle surface area and feed them true ground distances, which gives the same numbers as Esri's planar tools wherever a projection is true to scale and correct numbers where it is not.

On a real DEM the two Esri methods are hard to tell apart. The figure below runs both on the Web Mercator DEM of the Grand Canyon used above, and its caption gives the numbers behind the comparison.

Two maps of the same stretch of the Grand Canyon side by side, each a slope raster in the same blue-to-red color ramp with a legend: Slope, Planar Method, 0 to 80.244 degrees on the left and Slope, Geodesic Method, 0 to 80.0729 degrees on the right; the two pictures are indistinguishable, with the same red cliff bands and blue canyon floors.
Esri's Slope tool run twice on a Web Mercator DEM of part of the Grand Canyon, planar on the left and geodesic on the right, under the same symbology. The two are indistinguishable. On this DEM, whose 34.4 m cells are 27.8 m on the ground, the planar result matches Horn's gradient at the true ground cell size to 0.05° in every cell, while a gradient at the nominal cell size would read 3.5° too low on average. The geodesic result, a plane fitted to the nine cells in earth-centered coordinates, differs from the planar one by 0.07° on average and 0.3° in a typical cell, and by more only along cliff edges, so on Web Mercator the choice between the two is about the estimator, not the projection.

How it was tested

The method was checked against an analytic surface rather than a real DEM, because only an analytic surface has a slope that is known exactly at every point. A plane tilted ten degrees with two Gaussian hills on it was defined as a function of latitude and longitude, written as a DEM in four coordinate systems around Flagstaff (UTM zone 12N, latitude/longitude, Web Mercator and a WGS84 Albers) and in four at 75° north (UTM zone 24N, the NSIDC polar stereographic, Web Mercator and latitude/longitude), and every DEM tool was run on all eight. Slopes from the Horn gradient agreed with the analytic slope to within a hundredth of a degree on every system except Albers, where the raster's own one-percent scale distortion sets the limit at 0.04°; the surface ratio agreed with 1 / cos(slope) to within 0.002 percent; and at the same physical points the slopes from the eight DEMs agreed with one another to within 0.04°. On the UTM DEMs the slope values matched those extracted from Esri's planar Slope raster to 0.006°, which is exactly UTM's 0.9996 scale factor, and the aspects matched to a millionth of a degree.

What to do

Credits and references

Written for version 1.113.62.1 of the add-in, which introduced the lattice method; the diagrams on this page were drawn for it, and the figures from real data are described in the captions above. The scale-factor behavior of the projection families follows Snyder (1987). The per-row treatment of geographic grids was first built into the author's DEM Surface Tools extension for ArcMap in 2009 (Jenness 2013), and has its published counterpart in Florinsky's (1998) derivatives for spheroidal trapezoidal DEMs.

The basemap under the maps in the figures on this page is Esri's National Geographic World Map basemap (sources: National Geographic, Esri, Garmin, HERE, UNEP-WCMC, USGS, NASA, ESA, METI, NRCAN, GEBCO, NOAA, increment P Corp.).