Data Processing

Overview#

Data processing is the critical bridge between field observations and usable survey products. Raw measurements collected in the field -- angles, distances, GNSS vectors, level readings -- must be downloaded, organized, verified, reduced, and corrected before they can be used for mapping, calculations, or legal documents. A disciplined data processing workflow ensures that no observations are lost, all corrections are properly applied, and the final coordinates are traceable back to the original measurements.

The quality of every downstream deliverable -- maps, legal descriptions, reports -- depends entirely on how well the data was processed. Errors introduced or missed during processing propagate through every subsequent step and are often difficult to detect later.

A survey is only as good as its data processing. Field crews collect the raw material; the office transforms it into coordinates you can stake your professional license on.

Downloading Field Data#

From Total Stations

Modern total stations store observations in internal memory or removable media. Downloading typically involves:

• USB cable connection to a desktop or laptop running the manufacturer's software
• SD card or USB drive transfer for instruments with removable storage
• Bluetooth or Wi-Fi for newer instruments supporting wireless transfer

Each manufacturer uses proprietary software for data transfer:

From GNSS Receivers

GNSS data downloading depends on the observation type:

• RTK observations are typically stored in the data collector and downloaded with the collector's software. Coordinates are already computed in the field.
• Static/rapid-static observations produce raw GNSS files (RINEX, or proprietary formats like Trimble T02/T04, Leica MDB) that require post-processing in the office.
• Network RTK (VRS/NTRIP) stores final RTK coordinates in the collector but may also log raw data for quality assurance.

From Data Collectors

Data collectors (field controllers) are the most common source of survey data. They run field software and store observations in project files:

Always download the raw observation file, not just the reduced coordinates. The raw file preserves the original measurements and allows re-processing if errors are discovered later.

Common File Formats#

Raw Data Formats

• RW5 -- Carlson raw data format. A text-based file containing angle/distance observations, GPS vectors, and codes. Widely used because it is human-readable and can be edited in a text editor.
• DC -- Trimble Data Collector format. Contains raw observations from Trimble total stations.
• JOB -- Trimble Access job file. A binary project file containing all observations, point data, and metadata.
• GSI -- Leica's standard observation format. Available in GSI-8 (8-character) and GSI-16 (16-character) versions.
• RINEX -- Receiver Independent Exchange Format. The standard format for raw GNSS observations from any manufacturer.

Processed/Exchange Formats

Point Numbering and Coding

Consistent point numbering and field coding are essential for efficient data processing:

• Block numbering -- Assign point number ranges to different crews, instruments, or project phases (e.g., 1--999 for boundary, 1000--1999 for topo, 5000--5999 for control).
• Feature codes -- Standardized codes that describe what each point represents (e.g., IP for iron pipe, CM for concrete monument, EP for edge of pavement). Many offices maintain a code library that links to CAD symbols and linetypes.
• Linework codes -- Codes that control automatic line drawing in CAD (e.g., begin/end codes, connect codes, curve codes).

Data Reduction#

Data reduction is the process of converting raw field observations into coordinates.

From Angle and Distance Observations

The fundamental sequence for conventional (total station) observations:

1. Reduce slope distance to horizontal distance

   • Apply atmospheric corrections to the EDM distance
   • Compute horizontal distance: $HD = SD \times \cos(\text{zenith angle})$

2. Reduce vertical angle to elevation difference

   • $\Delta h = SD \times \cos(\text{zenith angle}) + HI - HT$
   • Where $HI$ = height of instrument, $HT$ = height of target

3. Apply mean angles -- If multiple sets of angles were turned, compute the mean direction from all sets, rejecting any that exceed the specified tolerance.

4. Compute coordinates from the reduced horizontal distance and direction:

   • $\Delta N = HD \times \cos(\text{azimuth})$
   • $\Delta E = HD \times \sin(\text{azimuth})$

From GNSS Observations

Static GNSS data reduction involves:

1. Import raw RINEX files into post-processing software
2. Download precise ephemerides (rapid or final) from NGS or IGS
3. Process baselines -- Resolve integer ambiguities and compute baseline vectors
4. Perform network adjustment -- Adjust all baselines simultaneously, applying constraints from known control
5. Evaluate results -- Check RMS values, ratio statistics, and residuals

Applying Corrections#

Atmospheric Corrections (EDM)

Electronic distance measurement is affected by atmospheric conditions. The speed of the electromagnetic signal varies with temperature, pressure, and humidity.

Most modern total stations accept temperature and pressure inputs and compute the correction internally. The correction is applied as a parts-per-million (ppm) adjustment:

$D_{\text{corrected}} = D_{\text{measured}} \times (1 + \text{ppm} \times 10^{-6})$

Always record temperature and pressure in the field notes. Even if the instrument applies the correction in real time, the raw values must be documented for quality assurance and potential re-processing.

Curvature and Refraction

For longer lines, the curvature of the Earth and atmospheric refraction affect the observed vertical angle and, consequently, the computed elevation difference. The combined correction for curvature and refraction is:

$h_{cr} = 0.0675 \times \frac{D^2}{R}$

where $D$ is the horizontal distance and $R$ is the mean radius of the Earth (approximately 6,371 km). This correction becomes significant for distances beyond about 300 meters.

Sea Level (Elevation) Correction

Distances measured at elevation must be reduced to the ellipsoid or a datum surface for use in coordinate computations. The sea level correction factor is:

$\text{SF}_{\text{elevation}} = \frac{R}{R + H}$

where $H$ is the mean elevation of the line above the ellipsoid and $R$ is the mean radius of the Earth. At 1,000 meters elevation, the correction is approximately -157 ppm, meaning a measured 1,000-meter line is about 0.157 meters longer than its equivalent at sea level.

Grid Scale Factor

When working in a State Plane or UTM coordinate system, the grid scale factor accounts for the map projection distortion:

$D_{\text{grid}} = D_{\text{ground}} \times \text{SF}_{\text{elevation}} \times \text{SF}_{\text{grid}}$

The combined factor is often called the combined scale factor and must be applied to all distances before computing grid coordinates.

Coordinate Transformations#

Common Transformations

Surveyors frequently need to transform coordinates between systems:

Localization

Localization (or calibration) is the process of computing a best-fit transformation between GNSS-derived coordinates and known local/grid coordinates. This is essential when:

• Working on a project where local coordinates do not match GNSS positions exactly
• Fitting RTK observations to existing ground control
• Establishing a project-specific coordinate system

A typical localization uses 3 or more common points to compute a 2D conformal transformation (4 parameters: 2 translations, 1 rotation, 1 scale) plus a vertical shift or inclined plane.

Point Database Management#

Organizing the Point Database

A well-structured point database is the foundation of efficient office work:

• Unique point numbers -- No duplicates. If two instruments survey the same point, reconcile them to a single coordinate.
• Consistent descriptions -- Use the office code library. Do not allow free-form descriptions that vary by crew.
• Coordinate quality indicators -- Flag points by method (total station, RTK, static GNSS, scaled from plan) so that data quality is always visible.
• Point protection -- Lock finalized control coordinates to prevent accidental overwriting.

Quality Control Checks

Before using processed data for any deliverable:

• Redundant observations -- Compare duplicate measurements of the same point. Differences exceeding tolerance indicate a blunder.
• Closure checks -- Verify that traverses close within acceptable limits before adjustment.
• Control comparison -- Compare reoccupied control points to their published values.
• Visual inspection -- Plot all points and look for obvious outliers, stacked points, or points in impossible locations.
• Code verification -- Confirm that field codes match the features observed. Mis-coded points cause incorrect linework in CAD.

Project File Organization#

Folder Structure

A standardized folder structure ensures that any member of the office can find any file for any project:

PROJECT_NUMBER/
  01_Control/           # Control data, network adjustments
  02_FieldData/         # Raw downloads, field notes
  03_Processing/        # Reduced data, adjustment reports
  04_CAD/               # Drawing files
  05_Maps/              # Final map PDFs
  06_LegalDescriptions/ # Description documents
  07_Reports/           # Narrative reports, certifications
  08_Correspondence/    # Client and agency communications
  09_Reference/         # Record maps, title reports, deeds
  10_Photos/            # Field and aerial photos

File Naming Conventions

Consistent file naming eliminates confusion:

• Include the project number, date, and description in every filename
• Use ISO date format (YYYY-MM-DD) for chronological sorting
• Example: 2024-0150_2024-03-15_Boundary_Topo_RawData.rw5
• Never use spaces in filenames -- use underscores or hyphens

Version Control

• Save dated versions of critical files rather than overwriting
• Keep the original raw data file untouched in a separate folder
• Document any manual edits to processed data in a processing log

Backup and Archiving#

Backup Strategy

Archiving Requirements

Most jurisdictions require survey records to be retained for a minimum period (often 5--10 years; some require permanent retention). The archive should include:

• Original raw field data (unmodified)
• Processing reports and adjustment summaries
• Final coordinates and point databases
• All deliverables (maps, descriptions, reports)
• Relevant correspondence and research materials
• Metadata documenting software versions, datum, and processing parameters

The raw field data is irreplaceable. Everything else can be recomputed from it. Protect raw data above all other files.

Key Takeaways#

• Always download and preserve the raw observation file -- it is the only irreplaceable record of the fieldwork.
• Understand your file formats. Know the native format of every instrument in your fleet and how to extract raw observations from it.
• Apply all required corrections -- atmospheric, curvature and refraction, sea level, and grid scale factor -- before computing final coordinates.
• Data reduction must be traceable. Document every step so that another surveyor can reproduce your results from the raw data.
• Quality control is not optional. Check closures, compare redundant observations, and visually inspect all data before using it.
• Standardize your file organization. A consistent folder structure and naming convention prevents lost files and wasted time.
• Back up early and often. The cost of losing a day of field data dwarfs the cost of any backup system.
• Coordinate transformations require care. Always document the source datum, target datum, and transformation method on every deliverable.

References#

1. Ghilani, C.D. & Wolf, P.R. Elementary Surveying: An Introduction to Geomatics (13th Ed.). Pearson, 2012. Chapters 6, 8, 17.
2. Kavanagh, B.F. & Mastin, T.B. Surveying: Principles and Applications (9th Ed.). Pearson, 2014. Chapters 5, 7.
3. National Geodetic Survey. "OPUS -- Online Positioning User Service." NOAA/NGS. https://geodesy.noaa.gov/OPUS/
4. Carlson Software. "SurvCE/SurvPC User Guide." Carlson Software, Inc.
5. Trimble. "Trimble Access User Guide." Trimble Inc.
6. American Congress on Surveying and Mapping. "Technical Standards and Guidelines for Survey Data Management."