Point Data Sources and Formats

As of 2010, 3D point data from laser scanning (point clouds) are becoming increasingly common.
There are three common cases:
- from an airplane or space (aerial LIDAR)
- terrestrial, from a stationary position, such as a laser-scanning total station
- terrestrial, from a moving vehicle, such as with Topcon's IP-S2 3D Laser and Digital Imagery Mobile Mapping System
The terrestrial data is far more dense, and much more likely to be vertically stacked (not a heightfield)

LIDAR collects massive amounts of 3D point data, i.e. "point clouds".
Sometimes, there is also an attribute (e.g. 0 default 1 ground 2 buildings 3 vegetation)
There is also the intensity of the laser reflection, and the potential of multiple "returns" from a single beam.
The LAS binary format is a fairly standard way to encode these points. LAStools is a collection of utilities to read/write/convert/view LAS files, with source code. There is also the free library libLAS which was based on LAStools.
There are also various proprietary formats, e.g. TerraSolid .BIN format.
LIDAR is inherently expensive and uncommon, because of the unavoidable cost of flying an aircraft and doing all the tracking and registration to produce useful result.
Usually LiDAR is a much, much better source of elevation heightfields than traditional DEMs (from photogrammetry contours converted to raster grids), although there is the danger of picking up too much data, so sophisticated methods are used to filter out "clutter" like trees and buildings.
On the flip side, LIDAR can be the source for automatic recognition of those buildings (and to a lesser degree, vegetation and other other features)
There are products like Titan TLiD which do exactly that:
As of 2011 there is nothing open-source or free, yet, for extracting features from LIDAR

Can be extremely dense (more than 1 point per centimeter!)
With the right equipment and care, can be very accurate: global error on the order of a few cm from a precisely tracked moving platform, or "survey-grade" (mm accuracy) from a total station.
The following snapshots (from Topcon) shows how density decreases with distance from the sensor, and how point clouds can be colorized by correlating an image captured at the same time as the laser:
Although this data is just points - not solid objects or higher-level entities - it can provide a detailed source for either automatic or semi-automatic construction of a real 3D scene.

glob3 is a very ambitious open-source project to create a "3D GIS multiplatform framework", in Java, possibly built on top of WorldWind, which will support both aerial and terrestrial point clouds.
- It is run by Spanish company Igo Software. There is a paper (Feb. 2011) outlining their plans.
- There are some interesting ideas in their approach, like using a kdTree that divides along the largest dimension.
- They are working on a server that specifically serves point cloud data.

The paper Storage, manipulation, and visualization of LiDAR data explains that (as of 2009) traditional database and GIS software's support for point clouds was very weak, with only Oracle 11g having some ability to directly store point clouds, with minimal indexing, and even that was very slow and inefficient.
That has made an opening for point-cloud-specific proprietary software like Pointools, Terrasolid and Quick Terrain Modeler, which can handle the clouds that traditional software (like ESRI) do not.