NASA

NATIONAL AERONAUTICS AND

SPACE ADMINISTRATION

________________________________________________________________

George C. Marshall Space Flight Center

Earth System Science Division

Observing Systems Branch
 

Co-Investigator: Richard J. Blakeslee
Earth System Science Division
NASA/Marshall Space Flight Center
Global Hydrology and Climate Center
Phone (205) 922-5962, Fax (205) 922-5723
e-mail: rich.blakeslee@msfc.nasa.gov

Co-Investigator: Steven J. Goodman
Earth System Science Division
NASA/Marshall Space Flight Center
Global Hydrology and Climate Center
Phone (205) 922-5891, Fax (205) 922-5723
e-mail:steven.goodman@msfc.nasa.gov

Co-Investigator: Douglas M. Mach
Atmospheric Science Department
University of Alabama in Huntsville
Global Hydrology and Climate Center
Phone (205) 922-5830, Fax (205) 922-5723
e-mail: douglas.mach@msfc.nasa.gov

  
1. INTRODUCTION ...........................................................................  4

2. OVERVIEW................................................................................  5
2.1 THE NEED FOR LIGHTNING MEASUREMENTS ON TRMM............................................  5 
2.2 HISTORIC PERSPECTIVE AND HERITAGE .....................................................  7 
2.3 EXPERIMENTAL OBJECTIVES................................................................  8 
2.4 INSTRUMENT CHARACTERISTICS.............................................................  9

3. ALGORITHM DESCRIPTION...................................................................  11
3.1 INTRODUCTION ..........................................................................  11 
3.1.1 Lightning Signal Characteristics ....................................................  11 
3.1.2 LIS Measurement Approach ............................................................  12 
3.1.3 On-Board Signal Processing...........................................................  13 
3.2 DEFINITIONS ............................................................................ 15 
3.2.1 Background ........................................................................... 15 
3.2.2 Event................................................................................  15 
3.2.3 Group................................................................................  16 
3.2.4 Flash................................................................................  16 
3.2.5 Area ................................................................................  17 
3.2.6 Orbit................................................................................. 18 
3.3 LIS DATA DESCRIPTION...................................................................  18 
3.3.1 Orbit Level Data Structures .........................................................  19 
3.3.2 Data Summary (Browse) Structure .....................................................  21 
3.3.3 Background Images (LIS02)............................................................  24 
3.3.4 Area Statistics (LIS06) .............................................................  24 
3.3.5 Flash Statistics (LIS05).............................................................. 25 
3.3.6 Group Statistics (LIS04).............................................................  26 
3.3.7 Event Data (LIS03) ..................................................................  28 
3.3.8 Flash Density (LIS10) ...............................................................  29 
3.3.9 Metadata ............................................................................  30 
3.3.10 Intra-structure Links ..............................................................  30 
3.4 ALGORITHM MATHEMATICAL DESCRIPTION...................................................... 32 
3.4.1 Example Data Processing Sequence ..................................................... 32 
3.4.2 Algorithm Overview ................................................................... 36 
3.4.3 Significant Algorithm Details ........................................................ 36 
3.5 NUMERICAL CONSIDERATIONS ............................................................... 41 
3.5.1 Recursive Code.......................................................................  41 
3.5.2 Use of Integer Arithmetic............................................................. 41 
3.5.3 I/O versus Memory Usage .............................................................. 41 
3.6 SOFTWARE FILTERS........................................................................ 42 
3.6.1 Solar Glint Filter ................................................................... 42 
3.6.2 Radiation Filters..................................................................... 42
3.6.3 Spatial Probability Density Filter ................................................... 43 
3.6.4 Ghost Filter.......................................................................... 44 
3.7 CALIBRATION, VALIDATION, AND QUALITY CONTROL............................................ 44 
3.7.1 Calibration ...........................................................................44 
3.7.2 Validation............................................................................ 46 
3.7.3 Quality Control and Diagnostics....................................................... 47 
3.7.4 Exception Handling.................................................................... 47

4. CONSTRAINTS, LIMITATIONS, AND ASSUMPTIONS ............................................... 49
4.1 TELEMETRY LINK DATA RATE CONSTRAINTS ..................................................  49 
4.2 LIMITATIONS............................................................................  49 
4.2.1 Maximum/Minimum Values ............................................................... 49 
4.2.2 LIS Alignment Criteria................................................................ 50 
4.3 ASSUMPTIONS............................................................................. 50 
4.3.1 Lightning Signal Assumptions ......................................................... 50 
4.3.2 Background Illumination Assumptions .................................................. 50

5. REFERENCES .............................................................................. 51

1. INTRODUCTION

This document is the Lightning Imaging Sensor (LIS) Algorithm Theoretical Basis Document (ATBD). This document defines and describes in detail the algorithm that processes the LIS data into the basic HDF orbit granule data storage structure (a level 1B/2A data product). Section 2 of this document provides a brief overview of the contributions of LIS measurements in TRMM and the EOS Mission to Planet Earth, the heritage of space-based lightning measurements, and the LIS science objectives. The LIS instrument characteristics are also described at the end of section 2. Section 3 discusses the theoretical basis of the LIS data processing algorithm, defines terminology, and provides a detailed description of the operation of the LIS data processing algorithm in generating the HDF orbit granule data product. A LIS data processing scenario is given to more clearly illustrate how the LIS algorithm functions. Finally, section 4 touches on various constraints, limitations, and assumptions associated with the LIS data processing algorithm.

The LIS is an Earth Observing System (EOS) instrument on the Tropical Rainfall Measuring Mission (TRMM) platform designed to acquire and investigate the distribution and variability of total lightning (i.e., cloud-to-ground and intracloud) on a global basis. The addition of LIS on TRMM contributes to many of the goals and objectives of TRMM and EOS. Lightning is an unique indicator of deep convection and represents an important means for detecting deep convection from space without a land-ocean bias. The LIS data will provide the basis for the development of a comprehensive global thunderstorm and lightning climatological data base.

 2. OVERVIEW

 2.1 THE NEED FOR LIGHTNING MEASUREMENTS ON TRMM

Lightning is a global phenomenon that can be used as a measure of some of the variables in global change. Lightning activity is associated with thunderstorms that produce heavy rainfall in many climatic regions and, therefore, can be used to increase our knowledge of the distribution, structure, and variability of thunderstorms at the local, regional, and global scale over the land and ocean. The world's major centers of organized mesoscale convection do not, in general, coincide with the 'maxima' in the thunderstorm day climatology. The orographic and environmental interactions that initiate and sustain long-lived convective activity yield a different, yet important, perspective that is difficult to appreciate when considering only the number of thunderstorm days. This occurs because thunderstorm day statistics tend to be dominated more by the diurnal cycle than by large scale dynamics and give equal weight both to a day in which perhaps only one lightning discharge occurs and to a day in which tens of thousands of discharges occur. For example, up to 25% of the entire annual lightning strikes at a given site within the Central U.S. can be accounted for by the passage of but a single mesoscale convective complex (Goodman and MacGorman, 1986).

Atmospheric teleconnections associated with naturally occurring climate variations such as ENSO (El Nino Southern Oscillation) and anti-ENSO events in the tropical Pacific often result in significant changes in the frequency and movement of storm tracks, precipitation patterns, and cloud cover. These climate variations will also produce changes in lightning activity in both the northern and southern hemispheres. Forest fires are often caused by lightning when the ground is dry, decayed vegetation is present, and surface winds are strong. Doubled CO2 (i.e., global warming) climate simulations suggest a 25% increase in global lightning frequency (Price and Rind, 1990). Does this mean that naturally occurring forest fires (in northern boreal forests, in particular) will be more common in the future? Are global warming simulations of cloud amounts, distributions, and structures, limited by coarse model spatial resolution and relatively simple convective parameterizations, justified by the regional and global observations of lightning? How might sub-cloud evaporation change? An improved, long-term observation capability will help resolve these questions.

Lightning measurements provided by the Lightning Imaging Sensor (LIS) on the Tropical Rainfall Measuring Mission (TRMM) will offer an opportunity to develop combined data algorithms to investigate the electrical, microphysical, and kinematic properties of tropical thunderstorms. It is hypothesized that the type (intracloud versus cloud-to-ground discharges) and frequency of lightning are intimately related to the microphysical (e.g., ice mass, liquid water content) and kinematic properties (e.g., updraft speed) of thunderstorm systems and to the environment (e.g., available buoyant energy). Recent evidence suggests that lightning activity can provide empirical estimates or bound the range of values for some geophysical properties such as the convective rain flux and rain rate, the vertical structure and distribution of storm mass, (convective) latent heating rates, the number and distribution of thunderstorms (Goodman et al., 1988b; Goodman et al., 1989; Buechler and Goodman, 1990; Williams and Rutledge, 1990; Goodman and Christian, 1993; Williams et al., 1992).

The processes that lead to the production of lightning are tightly controlled by updraft activity and the formation of precipitation. Lightning seems to initiate soon after the onset of strong convection, after significant cloud mass and ice have formed in the mixed phase regions of the thunderstorm. Lightning activity tends to track the updraft in both amplitude and phase with rates increasing as the updraft intensifies and decreasing rapidly with cessation of vertical growth. It has been demonstrated that lightning observations from space will clearly delineate the regions of convection embedded within large stratiform cloud systems which are often obscured by cirrus anvils (Goodman and Christian, 1993). Thus the detection of lightning from space specifically identifies those regions that are of paramount importance in the convective rain formation process. This ability to uniquely identify and quantify the convective core regions of storm systems and the existence of a linear relationship between total rain volume and lightning flash rate make LIS an important part of TRMM.

Various methods have been developed to determine convective rainfall amounts remotely using visible, microwave and/or infrared observations from space borne sensors but deficiencies exist in these techniques (Barret and Martin, 1981). One problem is properly delineating the convective areas. Presently, IR thresholding for rainfall estimates is contaminated by signals from cirrus clouds. Combining the IR with VIS data is only useful during the daylight. Using the life history of the cloud to estimate rainfall amounts is hampered by the uncertainty of cloud-top features. Microwave emission and scattering both have problems with their low spatial resolution along with ambiguities between rain water and cloud signals.

A satellite measurement system that senses the amount of lightning (i.e., both intracloud and cloud-to-ground) produced by thunderstorms may overcome some of the drawbacks of current techniques and improve rainfall estimation. For example, the location of active lightning areas could be used to delineate the convective areas used by visible and infrared estimation techniques and indicate the relative amount of precipitation-sized ice (Goodman et al., 1988c). Weinman et al. (1993) propose using continuous observations from long-range sferics networks to augment the sampling gaps of the existing polar orbiting DMSP satellites. In addition, they suggest using the sferics observations to calibrate the rainfall estimates from geostationary infrared imagers (e.g., Arkin and Ardanuy, 1989), which have better temporal sampling than polar orbiting satellites, but have a weaker physical linkage to the associated rainfall, especially over land. We anticipate the lightning observations could calibrate the infrared imagers using methods similar to the combined passive microwave - infrared imager rainfall retrieval schemes (Adler et al., 1993; Kummerow and Giglio, 1992). Another possibility would be to develop a technique for estimating convective rainfall based upon the amount of lightning produced by the storm. Livingston and Krider (1978), Williams (1985), Goodman and MacGorman (1986), Cherna and Stansbury (1986), and others have developed relationships (i.e., empirical algorithms) for lightning rates as a function of storm size, height, and duration.

2.2 HISTORIC PERSPECTIVE AND HERITAGE

Historically, the electrification of thunderstorms and the resulting lightning have been studied on the smallest of scales in an attempt to understand the underlying physical principles that drive the processes. Observation and modeling studies have strongly intertwined cloud electrification, microphysics, and dynamics, resulting in important advances in the fundamental understanding but with obvious need for further clarification of the various process interactions and feedbacks involved. Recent measurements near Darwin, Australia during the monsoon and monsoon 'break' periods have revealed dramatic changes (i.e., factor of 10) in lightning frequency associated with modest variations in surface wet bulb temperature (Williams et al., 1991). Thunderstorm wind fields, precipitation, and lightning are tightly coupled in time and space and lightning cannot occur without the others. That is, strong updrafts and precipitation are necessary (but not sufficient) conditions for the production of lightning. There is now significant effort to quantify those interactions Also, while lightning and thunderstorms occur on local and regional scales, their effects have global consequences. For example, deep convective storms in the tropics are believed to be an important mechanism for heat transport from the surface to the upper troposphere in the Intertropical Convergence Zone (ITCZ) and contribute energy to drive both Walker and Hadley cell circulations.

Lightning has been observed from above clouds using sensors installed on airplanes (e.g., Christian et al., 1983), high altitude balloons (e.g., Holzworth and Chiu, 1982), and satellites (e.g., Turman, 1979). Airplanes are excellent platforms for making close-up, detailed measurements of lightning characteristics, and although aircraft can be vectored to specific regions of a cloud for coordinated studies of storms from above and below, it is not possible to obtain a global view using research aircraft. Nor can a global view be adequately obtained with high altitude balloon observations. Satellites, on the other hand, represent ideal platforms for investigating lightning activity on the global scale.

Over the past 25 years, more than a dozen Earth-orbiting spacecraft have flown instruments that have recorded signals from lightning. The OSO-2 (Vorpahl et al., 1970), OSO-5 (Sparrow and Ney, 1971), and DMSP (e.g., Turman, 1978; Turman and Tettelbach, 1980; Orville and Henderson, 1986) satellites observed lightning with various optical sensors. Lightning sferics have been measured by radio frequency (rf) sensors on the RAE-1 (Herman et al., 1965), ARIEL-3 (Horner and Bent, 1969), and the ISS-b (Kotaki and Katoh, 1983) satellites. The shuttle-based Night/Day Optical Survey of Lightning (NOSL) (Vonnegut et al., 1983) used a small hand-held camera package, while the shuttle-based Mesoscale Lightning Experiment (MLE) used payload bay TV cameras to obtain images of nighttime lightning activity over the Earth. The detection of lightning from some of these satellites was a primary research objective, while for others it was an unanticipated bonus. In general, however, the various instrument packages were unable to provide very precise information on lightning characteristics, spatial extent, and discharge frequency. Also, the location accuracy tended to be poor (hundreds of kilometers) due to the low spatial resolution of the detectors and the detection efficiency of the systems was generally less than 2%, resulting in severe undersampling of the worldwide lightning activity.

During the 1980's, extensive optical and electrical observations of lightning were made from a high altitude NASA U-2 airplane with the primary objective of defining a baseline design criteria for a space sensor capable of mapping lightning discharges from geostationary orbit during day and night with high spatial resolution and high detection efficiency. The results of the U-2 investigations, parametric trade-off studies, and other research (Thomason and Krider, 1982; Norwood, 1983; Eaton et al., 1983; Christian et al., 1984, 1989; Christian and Goodman, 1987; Goodman et al., 1988a) have clearly established the feasibility of making this kind of lightning measurement from space using present state-of-the-art technology. Based on this research, an optical lightning imaging sensor has been developed for deployment in low-Earth orbit.

In the TRMM pre-mission period, we plan to take advantage of the space based lightning observations that will be provided by the launch of the Optical Transient Detector (OTD) in 1994 as a secondary payload on a Pegasus launch vehicle. OTD will be used to address important validation and performance issues associated with the LIS algorithm processing software described in this document The OTD design is based on the LIS instrument for TRMM (Christian et al., 1992) and OTD can be viewed as a LIS prototype

2.3 EXPERIMENTAL OBJECTIVES

The overall investigation objectives of this instrument are to fly a calibrated optical Lightning Imaging Sensor (LIS) in order to acquire and investigate the distribution and variability of total lightning over the Earth, and to increase understanding of underlying and interrelated processes in the Earth/atmosphere system. The LIS will be particularly valuable in providing observations over the data sparse oceans and tropical regions of the world. The proposed lightning measurements provided by the LIS will address primary science objectives which will not be made by any other instruments.

Lightning is one of the responses of the atmosphere to thermodynamic and dynamic forcing and, consequently, contains significant information about the atmosphere. Because of this information content, the LIS will provide unique global data sets, including:

• Detection and Location of Deep Convection Without Land-Ocean Bias
• Estimation of Precipitation Mass in the Mixed Phased Region of Thunderclouds
• Differentiation of Storms with Strong Updrafts from Those with Weak Vertical Motions
• Information on Natural NO x and Other Trace Gas Production in the Atmosphere

2.4 INSTRUMENT CHARACTERISTICS

The LIS will detect and locate lightning with storm scale resolution (i.e., 5-10 km) over a large region of the Earth's surface along the orbital track of the satellite, mark the time of occurrence of the lightning with 2 ms resolution, and measure the radiant energy. The LIS will have a nearly uniform 90% detection efficiency within the field of view of the sensor, and will detect intracloud and cloud-to-ground discharges during the day and night conditions.

The LIS on TRMM will view a total area exceeding 600 km 600 km at the cloud top using a 128 128 charge coupled device (CCD) array. The LIS on TRMM will monitor individual storms and storm systems for 80 s, a period long enough to obtain a measure of the lightning flashing rate in these storms while the storm is in the field of view of the sensor. It will be possible to estimate lightning frequency even for storms with flash rates as low as 1-2 discharges per minute. As a result of the very high temporal resolution of the LIS (i.e., events marked to the nearest 2 ms), the satellite platform will travel a distance of only 8 m in the time between successive readouts of the photodiode array. Table 2.1 gives the performance characteristics of the LIS.

Finally, it is worth stressing that the LIS represents a significant advance over any previous satellite-borne lightning detector. Lightning observations from the LIS can be readily associated with the thunderstorms that produced them, and the detection of even a single discharge is significant and provides important information (e.g., storm location, precipitation estimates, storm height, the presence of ice, lightning frequency, electric current output, etc.). The earlier satellite-borne lightning instruments could provide only a relative global distribution of lightning of a strictly statistical nature which required collecting data for long periods of time before it could be meaningfully interpreted in terms of global distributions. It was not possible to determine the lightning frequency of a particular storm with the earlier instruments, nor, for that matter, to even associate an individual discharge with a particular storm.

 

Table 2.1. Performance Criteria
pixel IFOV (nadir)	5 km
total FOV	80 80 square FOV
wavelength	777.4 nm
threshold	4.7 J m2 sr 1
SNR	6
array size	128 128 pixels
dynamic range	>100
detection efficiency	>90% of all events
false alarm rate	<10% of total events
measurement accuracy	location 1 pixel  intensity 10%   time tag at frame rate
command interface	adjust threshold  record/image   power on/off   self test   safe mode (close/open aperature cover)
weight	20 kg
power	25 watts
telemetry	data rate 6kb/s  format PCM   sample size 12 bits
operating temperature	25 to +40C

 

3. ALGORITHM DESCRIPTION

3.1 INTRODUCTION

 3.1.1 Lightning Signal Characteristics

The occurrence of lightning is accompanied by the sudden release of electrical energy which is converted into rapid heating in the vicinity of the lightning channel, the generation of a shock wave (which rapidly decays into an acoustic wave, i.e., thunder), and electromagnetic radiation ranging from extremely low frequency (ELF) radio waves to x-rays. One of the strongest radiation regions is in the optical wavelengths with peak power typically between 100 to 1000 MW. These optical emissions result from the dissociation, excitation, and subsequent recombination of atmospheric constituents as they respond to the sudden heating in the lightning channel. The heating is so intense (electron temperatures > 20,000 K) that the optical emissions occur primarily at discrete atomic lines with some continuum at shorter wavelengths. Measurements from a NASA U-2 airplane have shown that the strongest emission features in the cloud top optical spectra are produced by the neutral oxygen and neutral nitrogen lines in the near infrared (e.g., the OI(1) line at 777.4 nm and the NI(1) multiplet at 868.3 nm are consistently strong features).

Temporally, the optical lightning signal is comprised of a series of fast rise time, short-duration pulses associated with the energetic discharge processes occurring within the cloud. The individual pulses of cloud-to-ground lightning are generally associated with return strokes and k-changes. The optical pulse widths and rise times are highly variable and are similar for intracloud and cloud-to-ground lightning discharges; however, the interpulse intervals for intracloud flashes tend to be shorter and there are generally significantly more pulses produced during each intracloud flash.

The thundercloud is an optically thick medium and therefore strongly affects the temporal and spatial characteristics of the optical signals produced by lightning which would be observed by a satellite sensor. Although the thundercloud is optically thick, there is very little absorption at optical wavelengths. Hence, the major effect of the cloud on the optical signal is to blur the source geometry and to delay and time-broaden the pulses due to multiple scattering. Extensive measurements from an instrumented NASA U-2 aircraft have shown that the rise time of an optical pulse is typically lengthened 215 s and the pulses' widths tend to be 210 s wider as a result of this multiple scattering. The median pulse rise time and the full-width-at-half-maximum obtained from the analysis of nearly 1300 pulses produced by 79 lightning flashes are 240 s and 370 s, respectively (Christian and Goodman, 1987).

It is important to stress that, while the cloud significantly alters the temporal characteristics of the cloud top optical signals, the cloud does not block these emissions. When viewed from above, the optical lightning signals appear as a diffuse light source radiating from the cloud top. Measurements of the total optical energy radiated from the cloud top are in good agreement with ground-based measurements of cloud-to-ground flashes and support the theory that the cloud acts like a conservative scatterer, i.e., that most of the optical energy escapes the cloud (Christian and Goodman, 1987). Of the 79 discharges referred to above, 90% produced peak radiant energy densities of 4.7 J m2 sr1 or greater. The region of cloud top that is illuminated by a lightning flash depends on where the flash occurred within the cloud, the geometry and physical extent of the flash, and the characteristics of the cloud through which the lightning channel propagated and the radiation scattered. Theory, using Monte Carlo simulations of the radiation transfer of the optical lightning signals, and the NASA U-2 aircraft studies indicate that the diameter of the cloud top illumination associated with a single storm cell will typically be on the order of 10 km. Observations of large storm systems from the shuttle have shown that illuminated regions can exceed 60 km (Goodman and Christian, 1993).

Finally, it should be noted that both intracloud and cloud-to-ground lightning flashes are readily observed from above. Extensive observations with the NASA U-2 aircraft flying over the tops of thunderstorms in coordination with ground-based measurements made under the same storms (Goodman et al., 1988a) have clearly established the viability of optical detection of all lightning. Because the majority of the channel of a cloud-to-ground flash occurs within the cloud, the light emerging from the top of the cloud has undergone a similar scattering process as an intracloud flash (the portion of the channel below cloud base is essentially undetectable from above). Further, since the scattering process dominates the characteristics of the optical signature, the optical pulses from both intracloud and cloud-to-ground flashes are very similar (Guo and Krider, 1982; Thomason and Krider, 1982; Goodman et al., 1988a). We are unable at this time to distinguish between intracloud and cloud-to-ground lightning from the optical signatures alone. While this is a limitation, it is minor because, from a scientific standpoint, it is far more important to determine the total lightning rate than either just the cloud-to-ground or intracloud rate. In fact, optical measurements of lightning from above or below the cloud will detect more return strokes than ground-based detection systems (Goodman and MacGorman, 1986; Mach et al., 1986).

3.1.2 LIS Measurement Approach

The LIS is a conceptually simple device. It images the scene much like a television camera; however, because of the transient nature of lightning, its spectral characteristics, and the difficulty of daytime detection of lightning against the brightly lit cloud background, actual data handling and processing is much different from that required by a simple imager. In order to achieve the performance goals required to meet the scientific objectives, the LIS combines off the shelf components in a unique configuration. A wide field of view lens, combined with a narrow-band interference filter is focused on a small, high speed CCD focal plane. The signal is read out from the focal plane into a real-time event processor for event detection and data compression. The resulting "lightning data only" signal is formatted, queued, and sent to the satellite LAN.

The particular characteristics of the sensor design results from the requirement to detect weak lightning signals during the day. During the day, the background illumination, produced by sunlight reflecting from the tops of clouds, is much brighter than the illumination produced by lightning. Consequently, the daytime lightning signals tend to be buried in the background noise, and the only way to detect lightning during daytime is to implement techniques that increase or maximize the lightning signal relative to this bright background. These techniques take advantage of the significant differences in the temporal, spatial, and spectral characteristics between the lightning signal and the background noise. A combination of four methods are employed by the LIS for this purpose. First, spatial filtering is used which matches the instantaneous field of view (IFOV) of each detector element in the LIS focal plane array to the typical cloud-top area illuminated by a lightning stroke (i.e., 5-10 km). This results in an optimal sampling of the lightning scene relative to the background illumination. Second, spectral filtering is obtained by using a narrow-band interference filter centered on a strong optical emission multiplet (e.g., OI(1) at 777.4 nm) in the lightning spectrum. This method further maximizes the lightning signal relative to the reflected daylight background. Third, the LIS employs temporal filtering which takes advantage of the difference in lightning pulse duration of the order of 400 s versus the background illumination which tends to be constant on the time scale of the order of seconds. In an integrating sensor, such as the LIS, the integration time specifies how long a particular pixel accumulates charge between readouts. The lightning signal-to-noise ratio improves as the integration period approaches the pulse duration. If, however, the integration period becomes too short, the lightning signal tends to be split between successive frames which actually decreases the signal-to-noise ratio. Since the median optical lightning pulse width when viewed from above is 400 s, an integration time of 1 ms is most appropriate to minimize pulse splitting and maximize lightning detectability. Technological limitations require that a 2 ms integration time be used in the LIS instrument design; however, this compromise will not seriously degrade the sensor's performance. Even with the three "filtering" approaches discussed above, the ratio of the background illumination to the lightning signal may still exceed 150 to 1 at the focal plane. Therefore, a fourth technique a modified frame-to-frame background subtraction is implemented to remove the slowly varying background signal from the raw data coming off the LIS focal plane. A more detailed discussion on the measurement approach adopted for the LIS is given in Christian et al. (1989).

The real time event processor generates an estimate of the background scene imaged at each pixel of the focal plane array. This background scene is updated during each frame readout sequence and, at the same time, the background signal is compared with the off-the-focal-plane signal on a pixel-by-pixel basis. When the difference between these signals exceeds a selected threshold, the signal is identified as a lightning event and an event processing sequence is enabled. The implementation of this real-time data processor results in a 105 reduction in data rate requirements while maintaining high detection efficiency for lightning events.

3.1.3 On-Board Signal Processing

The LIS consists of a staring imager optimized to detect and locate lightning. An imaging system, a focal plane assembly, a real-time signal processor and background remover, an event processor and formatter, power supply, and interface electronics are the six major subsystems of the sensor. The imaging system is a simple f/1.6 telescope consisting of a beam expander, an interference filter, and re-imaging optics. The 80 80 full angle LIS field of view is converged to less than 5 at the interference filter in order to minimize wavelength shifts due to non-normal incidence. The focal plane assembly, including the 128 128 element CCD array, preamplifiers, and multiplexers provides subsequent electronics with a serial data stream of sufficient amplitude. As noted earlier, if after the background removal the difference signal for a given pixel exceeds a threshold, that pixel is considered to contain an event. Once the event is identified it is time tagged, location tagged, and passed to platform telemetry via the interface electronics to the satellite LAN.

The LIS processor can extract weak lightning flashes from an intense but slowly evolving background. The daytime background varies with sun angle, clouds, ground albedo, etc., and can reach an excess of 700,000 electrons as compared to lightning signal electrons which may be less than 5000 electrons. When a lightning stroke occurs within a single frame, a small signal is superimposed on top of the essentially constant background signal. The real-time processor continuously averages the output from the focal plane over six frames on a pixel-by-pixel basis in order to generate a background estimate. It then subtracts the average background estimate from the current signal.

The subtracted signal consists of shot noise fluctuating about a zero with occasional peaks due to lightning events. When a peak exceeds the level of the variable threshold, it is considered to be a lightning event and is processed by the rest of the circuit. The threshold must be set sufficiently high that false triggers are kept to a small percent of the total lightning rate. Clearly, the threshold must be higher during daytime when shot noise is dominated by the solar background.

The components of the real-time event processor include a background signal estimator, a background remover, a lightning event thresholder, an event selector, and a signal identifier. Analog/digital hybrid processing is used in a unique way in that it takes advantage of the strengths of each technology in order to provide high processing rates while consuming minimal power. Much of the signal processing is performed in a pipeline fashion that maximizes throughput.

The background estimator (averager) and remover (subtracter) circuits combine to perform the functions of a time domain low pass filter. The signal coming off the focal plane is fed through a buffer and clipping stage in order to ensure that a strong lightning signal does not contaminate the background estimate. This signal is then multiplied by a fractional gain (B) and added to (1-B) times the previous background estimate for the same pixel. The inverse of the fractional gain is equivalent to the number of frames used in generating the background estimate and is analogous to setting the cutoff frequency in conventional frequency domain filters. Too high a fractional gain might permit lightning events to contaminate low background estimates and would increase the processing noise. Too low a fractional gain would not allow the background estimator to respond rapidly enough to changes in background intensity.

The proper operation of the background estimator requires that the background data are clocked through the estimator synchronously with the data being clocked off the focal plane and that the number of discrete storage elements in the background memory is exactly the same as the number of pixels in the focal plane array. When data are properly synchronized, the signal appearing on the output of the delay line during a given clock cycle corresponds spatially to the signal being clocked off the focal plane. That is, it contains a history of what that specific pixel has measured over the last 1/B frames. These two signals are then subtracted using a difference amplifier in order to generate a difference signal. Since the original signal contains either background plus lightning or just background, the subtracted signal will be either a lightning signal, near zero or a false alarm as previously described.

The difference signal is then compared with the threshold level (which may be adaptive). If the signal exceeds the threshold level, a comparator triggers, which enables a switch and passes the lightning signal for further processing. In addition, the comparator output is encoded using a digital multiplexer in order to generate a row address that identifies the specific pixel that detected the lightning event. The digital outputs from the data processor represent the intensity of the lightning event and the location where the lightning occurred. These signals are then forwarded to encoding electronics in which the data are formatted into a digital bit stream and sent to the platform LAN.

The lightning data is compressed in such a way as to minimize the telemetry bandwidth. If two adjacent pixels on the same row are illuminated by a lightning signal, only the outer pixel address is transmitted (both amplitudes are transmitted). It is up to the ground software to decompress the data and add the second adjacent address.

3.2 DEFINITIONS

The basic science data product of LIS is lightning. This product is comprised of several components, including: raw data (level 1-A), background images (level 1-B), events (level 2), groups (level 2), flashes (level 2), areas (level 2), vector data (level 2), browse data (level 3), orbit statistics (level 3), flash density maps (level 3), and metadata. Before we can discuss the details of the various components, we must define each of the underlying data storage classes that drive the algorithm. These data storage classes are backgrounds, events, groups, flashes, areas, and orbits.

3.2.1 Background

A background image is a "snap shot" of the background estimate created by the LIS Real Time Event Processor (RTEP). The background data consists of 12 bit raw count amplitudes at each of the 128x128 pixel locations and the time at which the background image was taken. The background is identified as LIS02 in the Data Products Handbook, Volume 1. The background is transmitted in the data stream along with event data to maintain the average transmission rate. When the transmission of one background is begun, the next background image is captured. New images are sent to the ground as frequently as the event load and transmission rate allow. The full set of data parameters (raw and calculated) associated with a background image will be described in sections 3.3.2.3 and 3.3.3.

 3.2.2 Event

An event is defined as the occurrence of a single pixel exceeding the background threshold during a single frame. In other words, each pixel output from the RTEP produces a separate event. The raw LIS instrument data consists of time, x and y pixel locations, and amplitude of the event. An event is the basic unit of data from the LIS. An event is identified as LIS03 in the Data Products Handbook, Volume 1. The full set of data parameters (raw and calculated) associated with an event will be described in section 3.3.7.

Although an event can be thought of as a single optical pulse due to lightning, it is possible that multiple pulses occurring within the 2 ms integration window may contribute to an event. Therefore, we purposely did not use 'pulse' or 'stroke' (or other similar name) to describe the basic unit of data from the LIS (Note: an event may sometimes not be due to lightning at all. It may be produced by noise in the analog data stream exceeding the background threshold. In that case, the event is a false alarm).

3.2.3 Group

Because a single pixel will almost never correspond to the exact cloud illumination area, a lightning discharge will often illuminate more than one pixel during a single integration time. The result is two or more adjacent events at the same time frame. When these multiple events are adjacent to each other (a side or corner of the events touching), they will be placed in a single group. The formal definition of a group is one or more simultaneous events (i.e., events that occur in the same time integration frame) that register in adjacent (neighboring or diagonal) pixels in the focal plane array. A group may consist of only one event or include many events. The location data for a group will be calculated in earth-based (0.02° latitude/longitude) coordinates. This is done to provide consistent representation in the group/flash/area processing and because the ultimate goal of the analysis to locate lightning on the ground. The exact definition of a group can be tuned in the LIS algorithm as studies of OTD and early LIS data warrant. A group is identified as LIS04 in the Data Products Handbook, Volume 1. The full set of data parameters associated with a group will be described in section 3.3.6.

Although a group may often correspond to a single lightning optical pulse, it is also possible that multiple lightning pulses occurring within the 2 ms integration window may contribute to a group. A false event due to noise at a pixel exceeding the background threshold can also contribute to a group (although noise groups often contain only one event). A discussion of the quality control and error analysis of the LIS data stream and the flagging of suspected data is contained in sections 3.7.3 and 3.7.4

3.2.4 Flash

A lightning flash consists of one to multiple optical pulses which occur in the same storm cell within a specified time and distance. A lightning flash should then correspond to several related groups in a limited area. For the LIS algorithm, we define a flash as a set of groups sequentially separated in time by no more than 333 ms and in space by no more than 0.02° latitude or longitude (the approximate size of a LIS pixel). Note that the temporal and spatial rules can be easily adjusted in the LIS algorithm processing software. We will examine the rules closely during the analysis of OTD and early LIS data to "fine tune" the rules defining a flash. A flash may include as few as one group with a single event or it may consist of many groups, each containing many events. Since there is the possibility that the TRMM satellite will move a significant fraction of a pixel during the time of a flash, spatial characteristics for a flash (and all higher level parameters) are calculated in ground coordinates (i.e., 0.02° latitude and longitude resolution). A flash is identified as LIS05 in the Data Products Handbook, Volume 1. The full set of data parameters associated with a flash will be described in section 3.3.5.

We have used the term flash for this data category because we believe that, as it has been defined above, the resultant 'flash' will generally correspond to the accepted definition of a conventional lightning flash. Note that with LIS data alone, we cannot determine if a flash is a ground or cloud flash. It is possible that future versions of the LIS algorithm may incorporate data from ground flash locating systems to help interpret LIS data as either ground or cloud flashes. We do acknowledge that, on occasion, distinct conventional lightning flashes may result in a single flash being produced by the LIS algorithm (e.g., this might occur in high flashing rate mesoscale convection systems). Other mismatches between algorithm flashes and actual conventional flashes will undoubtedly also occur. Note that there is no absolute time limit to a flash. That is, as long as subsequent groups are produced in an area within the 333 ms time windows, all groups will be assigned to a single flash.

3.2.5 Area

Lightning is produced in thunderstorm cells that have dimensions of about 10 km by 10 km. Many storms, however, are multicellular and may extend over large areas and exist for many hours. In most cases, individual storms will last longer than the LIS will view them. Therefore, we define an area as a near contiguous region on the surface of the earth that has produced lightning (defined as a set of LIS flashes) during a single orbit of the LIS. An area is defined as a set of flashes separated in space by no more than 0.2° latitude or longitude (approximately 4 pixels). The spatial rule can be easily adjusted in the LIS algorithm processing software if necessary after analysis of OTD and/or early LIS data. An area may include many flashes or contain as few as one event (i.e., one flash consisting of one group which in turn consists of one event). There is no interflash or absolute time limit rule being imposed in the area definition since, as noted previously, the LIS viewing time is much shorter than storm life cycle. Although there is no explicit limit to the temporal duration of an area (i.e., as long as there are events/groups/flashes in the region, all will be assigned to the area), the LIS instrument will only view any ground location within its FOV for a maximum of 80 seconds. Therefore, area duration will generally not exceed 80 seconds except possibly for very extensive (and very active) mesoscale storm complexes. An area is identified as LIS06 in the Data Products Handbook, Volume 1. The full set of data parameters associated with an area will be described in section 3.3.4.

The area definition serves as a proxy for a thunderstorm, however, due to the nature of the algorithm and possible spatial and temporal distribution of the data, several storms may be combined into one area. It is also possible for a single thunderstorm to be divided into more than one area. More sophisticated algorithms (with input from external ground-, airborne-, and space-based observing systems) will be needed to more precisely determine 'thunderstorms' in the LIS data.

3.2.6 Orbit

The orbit has been established as the data granule for TRMM. All data from the LIS will be stored and summarized at the orbit granule. The beginning and end times of the granule are as per the TRMM defined orbit, that is, from the time of one descending equatorial crossing to the next. An orbit will include all of the areas, flashes, groups, events, and backgrounds occurring between the start and stop time of the orbit (see sections 3.3.1 to 3.3.3 for a description of the data associated with the orbit). An orbit is identified as LIS07 in the Data Products Handbook, Volume 1.

Since orbits will arbitrarily start and stop at the time of the equatorial crossings, it is possible for flashes and areas to be split between two orbits. This will occur if the flashes or areas were active at the equatorial crossing times. The results of this orbit division will be discussed in section 3.7.4.3.

3.3 LIS DATA DESCRIPTION

The definitions described above can be used along with the EOSDIS Hierarchical Data Format (HDF) to create the data storage structure for the LIS data. The major focus of this section is to define the storage model of the data. It uses HDF Vgroups, Vdatas, Vsets (i.e., sets of Vdata), and Scientific Data Sets (SDS's) of the version 0 EOS HDF Standard Data Format (SDF) [Suresh et al., 1994] document to store the LIS data. The software algorithm has been developed to create this storage model.

The overall HDF file structure is shown in Figure 3.1. Since TRMM defines the data granule as an orbit, all LIS data will be categorized by the associated orbit.

[LIS HDF Data Structure]

The structure describes the data in such a way that a user that with an HDF file reader can read and process the orbit granule data. The actual data are stored in Vdata sets linked to the Vgroups via the HDF Vgroup tag/reference pair links. Indexes are maintained in the Vdatas to link the various Vdatas. In Figure 3.1, the direct HDF tag/reference pair linksare indicated by the black arrows in the figure while the gray arrows indicate internal indexes relating the data. How all the data is linked (either via direct HDF link or by an indirect index link) and how the data storage model relates to the LIS lightning list will be described in more detail in the following sections.

3.3.1 Orbit Level Data Structures

3.3.1.1 Orbit Attributes (LIS07)

Orbit Attributes (part of LIS07) are orbit summary data stored in an HDF Vdata format. The Orbit Attributes data are listed in Table 3.1. The components for the Orbit Attributes are the orbit identification number (orbit ID), the time of the orbit start (start time), the latitude and longitude of the orbit start (start location), the time of the orbit end (end time), and the latitude and longitude of the orbit stop location (end location). The orbit identification number is an int32 uniquely designating the LIS orbit. The times are in TAI double precision seconds. The latitude/longitude locations are in decimal degrees (float32). There is one of these Vdatas per orbit file.
 
 

Table 3.1. Orbit Attributes (LIS07)
Variable Name	Type	Description
orbit ID	int32	The orbit number designation
start time	float64	Orbit start time (TAI93)
start location[2]	float32	The location (latitude/longitude) of the sub-satellite point of the orbit start
end time	float64	Orbit stop time (TAI93)
end location[2]	float32	The location (latitude/longitude) of the sub-satellite point of the orbit stop

 

3.3.1.2 Orbit Summary (LIS07)

The Orbit Summary (part of LIS07) is a Vgroup linking four Vdata sets. The Orbit Summary data are listed in Table 3.2. The first link is to a Vdata (Summary Data) that contains long integer (int32) total counts of areas (areas), flashes (flashes), groups (groups), events (events) and backgrounds (backgrounds) that were found during the orbit. It also has the total integrated (summed) calibrated radiance (total cal radiance, float32) of all events detected during the orbit. The units of the total radiance are J m2 sr 1 . There is one Vdata of this type per orbit file.

The second link in the Orbit Summary Vgroup is to a set containing a variable number of Vdata (flash rates). Each Vdata component contains the flash rate for one area found during the orbit in a float32 value in units of flashes s-1. There is one such Vdata for each area found during the orbit.
 
 

Table 3.2. Orbit Summary (LIS07)
Variable Name	Type	Description
Summary Data (Vdata0)
areas	int32	The number of areas found during the orbit
flashes	int32	The number of flashes found during the orbit
groups	int32	The number of groups found during the orbit
events	int32	The number of events found during the orbit
backgrounds	int32	The number of backgrounds found during the orbit
total cal radiance	float32	The total integrated (summed) radiance measured during the orbit
Flash Rates (Vdata1)
flash rate	float32	The flash rate of each area found during the orbit
Event Rates (Vdata2)
event rate	int16	The event count during each second after the truncated start of orbit time. There are 6100 second slots in the structure. There is one of these structures for each filter output (currently 19).
Ephemeris (Vdata3)
TAI93 time	float64	TAI93 time of the ephemeris kernel
ephemeris kernel[15]	float64	The ephemeris kernel

 

The third link in the Orbit Summary Vgroup is to a set of Vdatas (event rates) associated with the application of data filters. There is one Vdata for each data filter that has been developed (currently there are 19 filters). Each Vdata structure contains the event rates during each second after the truncated start of orbit time. Hence, there are 6100 second slots (int16) in each structure. Note that there are less than 6100 seconds in an TRMM orbit but we have the extra room for orbit start/stop time errors. The time in seconds after truncated start of orbit for each data point in a Vdata structure is simply the index of the data point. If all of the data in a Vdata are set to 0, then the corresponding filter was not applied during that processing run. The identification of each Vdata in this set is in the Metadata.

The last link in the Orbit Summary Vgroup is to a set of Vdatas that contain the general ephemeris kernel storage location. These Vdatas will be indexed by the backgrounds and other lightning data when there is the need for a separate listing of the ephemeris data at a set time. The ephemeris kernel provided in the Vdata allows the user to geo-locate every point in the background if it is needed. An example of the need for this structure is so that we do not have to store the locations (latitude/longitude) for each of the 16,000 pixels in each background image. We separately store the corners and the nadir (in the image attribute Vdata) for those who only need a general knowledge of the background location. Another use of the ephemeris data is for recalculation of the event/group/flash/area locations based on other models of the Earth. The time is a float64 as are the 15 individual kernel components. There will be one of these Vdatas for each second of data from the start of orbit.

3.3.2 Data Summary (Browse) Structure

The Data Summary (Browse) structure is an HDF Vgroup with links to three Vdata sets. These Vdata sets include the LIS Browse Area (LIS09 in the Data Products Handbook, Volume 1), the Vector Statistics (LIS08 in the Data Products Handbook, Volume 1), and the Image Attributes of the background images ( part of LIS02). The Data Summary (Browse) data are listed in Tables 3.3a and 3.3b.

3.3.2.1 Browse Lightning (LIS09)

The first Vdata set is the LIS summary Browse Area data which contains a global summary with a 2.5° latitude/longitude resolution. There will be one Vdata for each 2.5° latitude/longitude grid point on the earth (total 144 x 72 locations). The first parameter (location) is the latitude/longitude of the browse data in degrees. The next set of int16 values (EGFAs) are the counts of events, groups, flashes, and areas found at the earth grid point. The next set of float32 values (EGFAs cal rad) are the summed total radiance's of the events, groups, flashes, and areas found at the earth grid point. The parameter max rate is the maximum flash rate found at the earth grid point. The next parameter (1 event areas) is the total number of single event areas found at that location. Since most areas will have many events associated with them, this variable indicates the possible number of false alarms in the LIS data. The parameter highres loc is the number of full instrument resolution Vdatas associated with this browse resolution Vdata. The next parameter (1st highres) is the index of the first Vector Statistics Vdata associated with the browse resolution Vdata. The Vector Statistics data will be described in the next paragraph (section 3.3.2.2). The parameter bkg locs is the number of background attributes Vdatas associate with this browse Vdata. Since the background images are time-ordered and not location-ordered, it is possible that the backgrounds associated with a browse Vdata may not be sequential in the HDF file so that a background from an adjacent browse Vdata may interrupt the sequential backgrounds associated with the Vdata.
 
 

Table 3.3a. Data Summary (Browse)
Variable Names	Type	Description
Browse Area (Vdata0, LIS09)
location[2]	float32	The latitude/longitude of the browse grid point
EGFAs[4]	int16	Number of events/groups/flashes/areas with centroid at the ground location
EGFAs cal rad[4]	float32	Total (summed) calibrated radiance of the events/groups/flashes/areas with centroids at the ground location
max rate	float32	Maximum flash rate found at the ground location
1 event areas	int16	Number of areas at the ground location with only one event
highres locs	int16	The number of full instrument resolution points associated with this browse location
1st highres	in32	The index of the first full instrument resolution Vdata associated with this browse location
bkg locs	int32	The number of backgrounds associated with this browse location
iloper	int16	The number of background attribute Vdatas not associated with this browse location but within the range of background images
1st bkg	int32	The index of the first background attribute Vdata associated with this browse location
duration	int16	The total time in seconds that this grid point was viewed by LIS.
Vector Statistics (Vdata1, LIS08)
location[2]	float32	The latitude/longitude of the full instrument resolution vector location
EGFA counts[4]	int16	Number of events/groups/flashes/areas with centroid at the ground location
EGFAs cal radiance[4]	float32	Total (summed) calibrated radiance of the events/groups/flashes/areas with centroids at the ground location
max flash rate	float32	Maximum flash rate found at the ground location
single event areas	int16	Number of areas at the ground location with only one event

 

The next parameter (iloper) will flag this occurrence. The details of what is in the background attributes Vdata is in section 3.3.2.3. The next parameter 1st bkg is the index of the first background attributes Vdata associated with this browse Vdata. The last parameter (duration) is the time in seconds that this grid box was viewed by LIS.

 3.3.2.2 Vector Lightning (LIS08)

The second Vdata (Vdata1) set in the Data Summary (Browse) Vgroup contains the full instrument resolution Vector Statistics (LIS08) summary. There will be one of these Vdatas for each 0.02° latitude/longitude grid point on the earth that has some LIS data (events, groups, flashes, areas, or backgrounds) associated with it. The format of this Vdata is very similar to the format of the Browse Area Vdata, only the parameters are at the full instrument resolution and there are no indexes to the background image Vdatas. The Vector Statistics are the "details" of the browse data in the previous Vdata set. The values (EGFAs) are the counts of events, groups, flashes, and areas found at the vector grid point. The next set of values (EGFAs cal rad) are the summed total radiance's of the events, groups, flashes, and areas found at the earth grid point. The parameter max flash rate is the maximum flash rate found at the earth grid point. The next parameter (single event areas) is the total number of single event areas found at that location.

3.3.2.3 Image Attributes (LIS02)

The last Vdata set in the Data Summary (Browse) Vgroup is associated with the background Image Attributes data (LIS02). The first parameter (TAI93) is the TAI time when the background image was taken. The next five parameters (NW location, NE
 
 

Table 3.3b. Data Summary (Browse) cont.
Image Attributes (Vdata2, LIS02)
TAI93	float64	The TAI time of the background
NW location[2]	float32	The latitude/longitude of the upper left hand corner of the background image
NE location[2]	float32	The latitude/longitude of the upper right hand corner of the background image
SW location[2]	float32	The latitude/longitude of the lower left hand corner of the background image
SE location[2]	float32	The latitude/longitude of the lower right hand corner of the background image
centroid location[2]	float32	The latitude/longitude of the center of the background image
image pointer	int32	The HDF record number of the background image
calibration number	int32	The calibration sequence used for this background
ephemeris pointer	int32	Pointer to the ephemeris data closest to the time of the background image.

 

location, SW location, SE location, and centroid location) are the latitude/longitude pairs of the upper left hand corner, upper right hand corner, lower left hand corner, lower right hand corner, and nadir of the background image. The next parameter (image pointer) is the index to the actual background image raw data in the Background Images SDS. The next parameter (calibration number) is the index to the calibration used to produce calibrated background image data. The last parameter is a pointer to the ephemeris data closest to the time of the background image. The ephemeris data can be used to geo-locate all of the pixels of a background image.

 3.3.3 Background Images (LIS02)

The Background Images SDS (Table 3.4, LIS02) is the actual storage location of the background raw count (non-calibrated) image. A background image provides a "snap shot" of the sensor's field of view when the lightning data volume permits the transmission of this image. The data from the background images are transmitted as part of the data stream. It is stored in an SDS format as 128 x 128 resolution. Although the data is stored as an int32, the actual data is only 12 bits. The SDS's are stored in time order as they are transmitted from LIS.
 
 

Table 3.4. Background Images SDS (LIS02)
Variable Name	Type	Description
raw amplitude[128][128]	int32	The pixel by pixel amplitude of the background image

 

3.3.4 Area Statistics (LIS06)

The Area Statistics Vdata (LIS06) set holds the data associated with the various areas recorded during the orbit. The parameters associated with the Area Statistics data are shown in Table 3.5. There will be one such item for each area that was recorded during the orbit. The area creation sequence number is stored in seq. The start time of the area is stored in TAI93 while the lifetime and viewing time of the area are stored in delta and view. The start time is in TAI seconds while the view time and lifetimes are in relative seconds. The lifetime of the area is the time difference between the first event in the area and the last event in the area. The view time is the actual time the centroid of the area was in the field of view of the LIS instrument. The variable s-z-a is the solar zenith angle of the sun at the time and location of the first event in the area. The variable t-o-d is a character representing the time of day at the first event in the area, indicating if the area was recorded during the day (d), night (n) or near sunrise/sunset twilight (t). The variable end status indicates if the area was active when it was written to the HDF file. If an area was active at the end of the orbit, data from the next orbit (in another HDF file) may need to be appended to the area to complete it. It could also indicate that excessive data rates may have terminated an area prematurely (before all possible flashes have been added to this area). In this case, data from other subsequent areas in the same HDF may need to be appended to this area to make it complete. The number of events that are part of the area is recorded in events while the location and size of the area are contained in cent and stdev. The stdev array is the standard deviation of the area centroid and indicates the size (in 0.02° latitude/longitude) of the area. The number of different latitude/longitude locations (Earth grid) that are part of the area (relative size) is recorded in the variable loc count. The total summation of radiance from all of the events in the area is stored in rad. For future use, there are variables orbit id and day that would give the index numbers of any parent structure to this area. Since areas are at the top of the data structure, these variables are set to the orbit ID number and the Julian day of the area. It is here to maintain symmetry in the group/flash/area levels. The number of flashes that make up the area is in children while children seq and children rec are the index of the first flash that forms the area. The children seq is simply an index of the first child flash which (for the first area) starts at 0 and increments for each flash. The children rec is the actual HDF record number of the first children flash.
 
 

Table 3.5. Area Statistics (Vdata, LIS06)
Variable Name	Type	Description
seq	int32	Area creation sequence number
TAI93	float64	TAI start time of the area
delta	float32	Lifetime of the area (in seconds)
view	float32	LIS viewing time of the area (in seconds)
s-z-a	int16	Solar Zenith Angle
d-n-t	character	Day/Night/Twilight
end status	character	Status of area at the end of the orbit
events	int16	Number of events that make up the area
cent[2]	float32	Average latitude/longitude of the area (to 0.02°)
stdev[2]	float32	Latitude/Longitude 'size' of the area (to 0.02°)
loc count	int16	Number of different latitude/longitude pairs that make up the area
rad	float32	The total summation of the event radiance's that make up the area
orbit id	int32	The index of the parent orbit of this area
day	int32	The Julian date of the area
children	int32	The number of flashes that are part of the area
children seq	int32	The creation sequence of the first child flash
children rec	int32	The index of the first child flash

 

3.3.5 Flash Statistics (LIS05)

For the Area/Flash/Group classes, the basic storage structure is the same. The only difference is the Flash and Group classes have parent pointers whereas Areas do not. The Flash Statistics Vdata set (LIS05) holds the data associated with the various flashes recorded during the orbit. The various parameters associated with the Flash Statistics data are shown in Table 3.6. There will be one such item for each flash that was recorded during the orbit. The flash creation sequence number is stored in seq. The start time of the flash is stored in TAI93 while the lifetime and viewing time of the flash are stored in delta and view. The start time is in TAI seconds while the view time and lifetimes are in relative seconds. The lifetime of the flash is the time difference between the first event in the flash and the last event in the flash. The view time is the actual time the centroid of the flash was in the field of view of the LIS instrument. The variable s-z-a is the solar zenith angle of the sun at the time and location of the first event in the flash. The variable t-o-d is a character representing the time of day at the first event in the flash, indicating if the flash was recorded during the day (d), night (n) or near sunrise/sunset (t). The variable end status indicates if the flash was active when it was written to the HDF file. If a flash was active at the end of the orbit, data from the next orbit (in another HDF file) may need to be appended to the flash to complete it. It could also indicate that excessive data rates may have terminated a flash prematurely. In this case, data from other flashes in the same HDF may need to be append to this flash to make it complete. The number of events that are part of the flash is in events while the location and size of the flash are contained in cent and stdev. The stdev array is the standard deviation of the flash centroid and indicate the size (in 0.02° latitude/longitude) of the flash. The number of different latitude/longitude locations (Earth grid) that are part of the flash (relative size) is in the variable loc count. The total summation of radiance from all of the events in the flash is stored in rad. The parent area for the flash is indicated by the variables parent seq and parent rec. The parent seq is the sequence number of the parent area starting at (for the first area in the file) 0. The parent rec is the actual HDF record number of the parent area. The number of groups that make up the flash is in children while children seq and children rec are the index of the first group that forms the flash. The children seq is simply an index of the first child group which (for the first group) starts at 0 and increments for each flash. The children rec is the actual HDF record number of the first child group.

3.3.6 Group Statistics (LIS04)

The Group Statistics Vdata set (LIS04) holds the data associated with the various groups recorded during the orbit. The various parameters associated with the Group Statistics data are shown in Table 3.7. There will be one such item for each group that was recorded during the orbit. The group creation sequence number is stored in seq. The start time of the group is stored in TAI93 while the lifetime and viewing time of the group are stored in delta and view. The start time is in TAI seconds while the view time and lifetimes are in relative seconds. The lifetime of the group is the time difference between the first event in the group and the last event in the group. The view time is the actual time the centroid of the group was in the field of view of the LIS instrument.
 
 

Table 3.6. Flash Statistics (Vdata, LIS05)
Variable Name	Type	Description
seq	int32	Flash creation sequence number
TAI93	float64	TAI start time of the flash
delta	float32	Lifetime of the flash (in seconds)
view	float32	LIS viewing time of the flash (in seconds)
s-z-a	int16	Solar Zenith Angle
d-n-t	character	Day/Night/Twilight
end status	character	Status of flash at the end of the orbit
events	int16	Number of events that make up the flash
cent[2]	float32	Average latitude/longitude of the flash (to 0.02°)
stdev[2]	float32	Latitude/Longitude 'size' of the flash (to 0.02°)
loc count	int16	Number of different latitude/longitude pairs that make up the flash
rad	float32	The total summation of the event radiance's that make up the flash
parent seq	int32	The index of the parent area of this flash
parent rec	int32	The HDF record # of the parent area of this flash
children	int32	The number of groups that are part of the flash
children seq	int32	The creation sequence of the first child group
children rec	int32	The index of the first child group

 

The variable s-z-a is the solar zenith angle of the sun at the time and location of the first event in the group. The variable t-o-d is a character representing the time of day at the first event in the group, indicating if the group was recorded during the day (d), night (n) or near sunrise/sunset (t). The variable end status indicates if the group was active when it was written to the HDF file. It indicates that, due to excessive data rates, the group was terminated prematurely (before all possible events were added to it). In this case, data from other groups in the same HDF may need to be append to this group to make it complete. The number of events that are part of the group is in events while the location and size of the group are contained in cent and stdev. The stdev array is the standard deviation of the group centroid and indicate the size (in 0.02° latitude/longitude) of the group. The number of different latitude/longitude locations (Earth grid) that are part of the group (relative size) is in the variable loc count. The total summation of radiance from all of the events in the group is stored in rad. The parent flash for the group is indicated by the variables parent seq and parent rec. The parent seq is the sequence number of the parent flash starting at (for the first flash in the file) 0. The parent rec is the actual HDF record number of the parent flash. The number of events that make up the group is in children while children seq and children rec are the index of the first event that forms the group. The children seq is simply an index of the first child event which (for the first event) starts at 0 and increments for each event. The children rec is the actual HDF record number of the first child event.
 
 

Table 3.7. Group Statistics (Vdata, LIS04)
Variable Name	Type	Description
seq	int32	Group creation sequence number
TAI93	float64	TAI start time of the group
delta	float32	Lifetime of the group (in seconds)
view	float32	LIS viewing time of the group (in seconds)
s-z-a	int16	Solar Zenith Angle
d-n-t	character	Day/Night/Twilight
end status	character	Status of group at the end of the orbit
events	int16	Number of events that make up the group
cent[2]	float32	Average latitude/longitude of the group (to 0.02°)
stdev[2]	float32	Latitude/Longitude 'size' of the group (to 0.02°)
loc count	int16	Number of different latitude/longitude pairs that make up the group
rad	float32	The total summation of the event radiance's that make up the group
parent seq	int32	The index of the parent flash of this group
parent rec	int32	The HDF record # of the parent flash of this group
children	int32	The number of events that are part of the group
children seq	int32	The creation sequence of the first child event
children rec	int32	The index of the first child event

3.3.7 Event Data (LIS03)

The Event Statistics Vdata set (LIS03) holds the data associated with the various events recorded during the orbit. The various parameters associated with the Event Statistics data are shown in Table 3.8. There will be one such item for each event that was recorded during the orbit. The event HDF record number is stored in event # while the creation sequence number is stored in seq #. The start time of the event is stored in TAI93. The variable s-z-a is the solar zenith angle of the sun at the time and location of the event. The variable t-o-d is a character representing the time of day at the event, indicating if the event was recorded during the day (d), night (n) or near sunrise/sunset (t). The variables x pixel and y pixel store the location of the event in the pixel array. The raw amplitude of the event is stored in raw radiance while cal radiance stores the calibrated radiance of the event total summation of radiance from all of the events in the event is stored in rad. The parent group for the event is indicated by the variables parent seq and parent rec. The parent seq is the sequence number of the parent group starting at (for the first group in the file) 0. The parent rec is the actual HDF record number of the parent group.
 
 

Table 3.8. Event Statistics (Vdata, LIS03)
Variable Name	Type	Description
event #	int32	Event creation number
seq	int32	Event HDF record number
TAI93	float64	TAI start time of the event
s-z-a	int16	Solar Zenith Angle
d-n-t	character	Day/Night/Twilight
x pixel	int16	X pixel value of the event
y pixel	int16	Y pixel value of the event
raw radiance	int16	Event amplitude counts
cal radiance	float32	The calibrated event radiance
parent seq	int32	The index of the parent group of this event
parent rec	int32	The HDF record # of the parent group of this event
QA[4]	character	Four bytes for QA data
location[2]	float32	The Earth latitude/longitude of the event

3.3.8 Flash Density (LIS10)

The Flash Density Vgroup (LIS10 in the Data Products Handbook, Volume 1) holds the sparse array of flash densities recorded during the orbit. The flash density data are recorded in two Vdata sets using the grid patterns of constant latitude/longitude (Vdata0) and constant area (Vdata1) employed in TRMM rainfall product. These Vdata sets are listed in Table 3.9. Both have the same structure. There will be one Vdata for each grid point in the sparse array. Note that there will not be the same number of Vdatas for each set since they are at different resolutions. The first component is location and it stores the location of the grid point in latitude/longitude. The last int32 component is flash count. It is the total number of flashes found at the grid point.

 

Table 3.9. Flash Density (Vgroup, LIS10)
Variable name	type	Description
2.5° x 2.5° grid (Vdata0)
location[2]	float32	Latitude/Longitude of grid point
flash count	int32	Number of flashes in the grid point
500 km x 500 km grid (Vdata1)
	float32	Latitude/Longitude of grid point
flash count	int32	Number of flashes in the grid point

3.3.9 Metadata

The Metadata structure, shown in Table 3.10, is a text based description of the LIS parameters unique to this orbit. There will be as many of these text structures as are needed to describe the unique LIS orbit parameters.
 
 

Table 3.10. Metadata
Variable name	type	Description
metadata	char[80]	Metadata text description

3.3.10 Intra-structure Links

In the Event, Group, Flash, and Area Vgroups/Vdatas described above there are links to the parent and child items. These links are illustrated in Figures 3.2, 3.3, and 3.4. To keep the file overhead to a minimum, HDF tag/reference pair pointers were not used to make the links. Instead, the HDF index of each Vdata or Vgroup was recorded and used as the link. In this way, a user of the data can find the parent and child items that contributed to the item they are currently analyzing by referencing the index of the parent or child item in the HDF file. For example, if a user is looking at a flash, he can easily determine the area that contains the flash. The user can easily find and analyze other flashes in the area. The user could also look at the individual groups and events that contributed to the flash.
 

3.4 ALGORITHM MATHEMATICAL DESCRIPTION

 3.4.1 Example Data Processing Sequence

The purpose of this section is to graphically describe the algorithm that accumulate the individual LIS events into groups, flashes, and areas by "walking through" a typical LIS data scenario. In this illustrative exercise, all times indicated are times after the first event time. Numbers indicate event numbers while small letters represent the groups. The flashes are designated by capital letters and the areas are indicated by Greek letters. Each subsequent section describes how the algorithm processes the events that occurred at that integration time. For the purpose of this demonstration, it is assumed that there were no events prior to the events at time 0 and that the pixel grid is 0.02° wide in latitude and longitude. In general, the latitude/longitude grid in earth-based coordinates and the pixel grid will not be the same size or co-registered. In addition, the times will be time from the start of the orbit.
 

3.4.1.1 Time = 0 ms

The first time integration is shown in Figure 3.5. Three (1,2,3) events occur at this time integration. Since the events are simultaneous and register in adjacent (i.e., neighboring or diagonal) pixels, they are collected into a single group (a). The group is assigned a new parent flash (A) and the new flash is assigned a new parent area ().
 

The next time integration with data is shown in Figure 3.6. At this time (100 ms after the first one), there are three more events (4,5,6). As in the previous case, these three new events are all assigned to a new group (b). These events are not assigned to group a since they occur at a different time. Since group b is within 0.02° of group a

(actually, they touch), and the groups occur within 333 ms of each other, group b is assigned to flash A and therefore, area.
 
 

 3.4.1.3 Time = 350 ms

The next integration time with data is shown in Figure 3.7. The time is 350 ms after the time of the first events, but only 250 ms after the time of the last events. At this

time there are four (7,8,9,10) more events. Events 7 and 8 are adjacent to each other and are assigned to a new group (c). Events 9 and 10 are not adjacent to events 7 and 8, but are adjacent to each other. They are assigned to another new group (d). Since group c is within 333 ms of the last group of flash A (250 ms) and is also within 0.02° of the parts of flash A, group c is assigned to flash A and area . Although group d also occurred within 333 ms of the last group of flash A, it is greater than 0.02° (latitude and longitude) away from any part of flash A so it is assigned to a new flash (B). The parts of flash B (i.e., group d) are greater than 0.2° away from any part of area so flash B is also assigned a new area ().
 

3.4.1.4 Time = 400 ms

Figure 3.8 shows the next integration time with data. The time is 400 ms after the first events and 50 ms after the latest events. Two more events occur (11,12) at this time. These two events are at the same time, but they are not adjacent to each other. They are assigned to two new groups (e for 11 and f for 12). The two new groups are less than 333 ms (50 ms) from the time of the last group of flash B and are within 0.02° (adjacent) of the parts of flash B so the two groups are assigned to flash B and area .

3.4.1.5 Time = 700 ms

The last time with events (for this example) is shown in Figure 3.9. At this time integration, 700 ms after the first events and 300 ms after the last events, there are two new events (13,14). The events are not adjacent, so they are assigned to two new groups (g for 13 and h for 14). Group g overlaps the parts of flash A, however, it has now been 350 ms (greater than 333 ms) since the last group associated with flash A. Therefore, group g is assigned to a new flash (C). Since flash C overlaps the parts of area and since there is no time limit for areas, flash C is assigned to area . Group h is not within 0.02° of any current flash, so it is assigned another new flash (D). Flash D is also not within 0.2° of any currently active area so it is assigned another new area ().
 

3.4.1.6 Summary Data

In the example data processing sequence just described, there were fourteen events, eight groups, four flashes, and three areas. This example shows how the LIS algorithm will convert events into groups, flashes, and areas. Some of the summary data statistics that would be generated from the LIS processing algorithm are shown in Tables 3.12 (areas), 3.13 (flashes), and 3.14 (groups) for this example. During the LIS mission, the start_time is a relative time that will be counted from the beginning of each orbit.
 
 

3.4.2 Algorithm Overview

The overall view of the algorithm that will produce the LIS data sets is illustrated in Figure 3.10. The algorithm flow is very simple. Once all the files have been opened and the data structures have been initialized, the algorithm reads an event from the input queue (this part can be buffered to help system I/O). The event is then located on the Earth (via PGS tools) and calibrated (via a calibration look-up table). Next, the event is assigned a parent group/flash/area based on the time and spatial parameters described previously. The data is written to the appropriate structures, the various statistics are updated and then the algorithm reads another event from the input queue and the process starts over again.

The algorithm will create data summaries and output them into an HDF file along with the data. The algorithm is set up so that all data associated with an area can be written to the HDF files as soon as the area becomes 'inactive'. The data does not have to be written at that time, however. It can be written whenever on-line storage becomes limited. Essentially the algorithm can be tuned to the memory/input output limits/requirements of the system when the algorithm is running. Details of the algorithm processing are described in the next section.
 

3.4.3 Significant Algorithm Details

3.4.3.1 Data Structure Initialization

Figure 3.11 illustrates the on-line data storage initialization. Enough memory is allocated for the EVENT structure so that all events associated with areas can be stored at one time. Since the storage requirements are the same for the GROUP and FLASH structures, both of these structures are stored in the same allocated buffer. This permits the most efficient dynamic allocation of resources between the GROUP and FLASH

structures. Although the requirements for the AREA structure are the same as the GROUP and FLASH ones, the AREA structures are allocated as part of the overall ORBIT structure. In this way time ordering of the areas is accomplished explicitly (the first area is stored in the first AREA structure, the second area is stored in the second AREA structure, etc.). Therefore, when areas are written to the HDF files, they are written in time order of their creation.
 

3.4.3.2 Parent Determination

Figure 3.12 shows the flow of the parent determination algorithm. As each new event is processed, the algorithm determines (from the spatial rules) if the event is part of any existing groups. If it is, the algorithm adds the event to the appropriate group. Since any existing group already has a parent flash and grandparent area, the data from the new event is added to those entities.

If the new event is not identified as part of any existing group, it becomes the basis for a new group. Since this new group does not yet have a parent flash or grandparent area, the same set of code as was used to determine if the event was part of any existing
 

groups is used to determine if this new group is part of any existing flashes. The only difference is unique temporal and spatial limits (specific to flashes) are used. If the new group is found to be part of an existing flash, the data from the new group/event is added to that of the flash and its parent area.

If the new group is not determined to be a part of an existing flash, a new flash is created with the group/event data as its base data. As before with the new group, this new flash does not yet have a parent area. Therefore, the same set of code that was used to determine if the group was part of any existing flashes is now used to determine if this new flash is part of any existing areas. The only difference being the use of unique temporal and spatial limits specific to areas. If the new flash is found to be part of an existing area, the data from the new flash/group/event is added to that of the area. If no current areas are found to be associated with the new flash, a new area is created with the flash/group/event data as its initial data.
 

3.4.3.3 Updating Structure Status

Figure 3.13 shows the operation of this code segment. Basically, the last event time in each active group/flash/area is compared with the current event time. Any groups/flashes/area/ that have exceeded their temporal (or in the case of areas, spatial) limits are marked as 'inactive'. Any new events will not be added to an 'inactive' group. Similarly, new groups will not be added to 'inactive' flashes nor will 'inactive' areas have
 

any new flashes added to them. When a group/flash/area is marked as 'inactive', the code calculates the item's statistics. Also as part of this code segment, the statistics for the pixel, event, and orbit data are updated.

3.4.3.4 HDF Data Output

Figure 3.14 shows the operation of the data output section. Once the output HDF file has been opened for appending (the algorithm allows for multiple writes to the HDF file) the code goes into a set of recursive steps. The code writes the data associated with the first available inactive area to the HDF file. This encompasses writing the area summary data and all flashes (in time order) associated with the area. For each flash written to the HDF file, the code writes the flash summary data and all groups (in time order) associated with the flash. For each group written to the HDF file, the code writes the group summary data and all events (in time order) associated with the group. Once all events of all groups of all the flashes associated with the area are written to the HDF file, the next inactive area is written to the file in the same fashion. Note that once an event/group/flash has been written to the HDF file, its location in the appropriate buffer is freed for the next event/group/flash.

After all inactive areas (and their associated flashes/groups/events) have been written to the HDF file, the code checks to see if it is the end of an orbit. If it is, the orbit summary data (orbit and pixel data) is written to the file and the file closed. If it is not the end of the orbit, the code simply closes the HDF file.

3.5 NUMERICAL CONSIDERATIONS

 3.5.1 Recursive Code

Two major sections of the algorithm (Figures 3.12 and 3.14) heavily use recursive code. Since the use of recursive code precludes the parallelization of that code segment, there needs to be strong reasoning for the use of such code. In the case of the LIS algorithm, the reasoning is due to the common construction of the AREA, FLASH, and GROUP structures. At each of these levels (AREA, FLASH, and GROUP) the processing to determine parent/child relations and to write items out to the HDF file are somewhat complex but identical. The use of recursive code allows us to write a single procedure to analyze all three type of event summations. If an improved algorithm is found to do the processing, only one segment of code has to be changed, thereby eliminating the possible errors in the coding if one or more of the duplicate code segment are not changed, or changed incorrectly.

Testing is also made simpler. Once the code is tested at one level, the only other testing that is needed is to verify that the various levels are implemented identically. This can be accomplished by using the same structures for each level (which has been done). If an error is found in the code segment, it should be very obvious since all three levels will be affected in the same manner. Correction of the code errors will also be much simpler.

Overall, because of the design of the LIS data set output, the use of recursive code makes for smaller compiled modules and more efficient use of memory. Updating and correcting code will also be simplified. Finally, the code becomes easier to understand because once one level of the recursive code and the recursion process is understood, the rest of the code is understood.

3.5.2 Use of Integer Arithmetic

All calculations (with the exception of the calls to PGS calculations) will be done in integer arithmetic. Indeed, nearly all calculations will be between short integers. The major exception is time calculations. For example, the latitude/longitude locations of the pulses will be converted from floating point values to integer values representing 0.05 units; 31.3 will be represented as the integer 626. This use of almost total integer arithmetic should greatly speed the processing of the LIS data stream. Since we are setting our data to only 0.02° accuracy, using integer arithmetic will not increase our error statistics. The use of short integers should also help lower the amount of memory needed to run the program.

 3.5.3 I/O versus Memory Usage

The LIS algorithm processing software can be tuned for nearly any combination of memory and I/O limitations. The minimum amount of data that can be written to the HDF file at one time is the data for a single area. This includes the flashes, groups, and events associated with the area. At the other extreme, all areas recorded during the orbit can be written at one time. In the first case, the minimum amount of on-line memory is used while the maximum number of I/O calls are made. In the second case, the use of on-line memory is maximized while the I/O calls are minimized (only one set of calls at the end of processing). In the LIS algorithm's present form, the user sets the amount memory to allocate to on-line storage and the code then determines when the I/O calls are made. The I/O calls are made when the on-line storage of GROUP and FLASH structures nears capacity. This allows operators to tune the code in near real-time to maximize code efficiency.

3.6 SOFTWARE FILTERS

Intermixed with the "lightning-related events" detected by LIS will be some "false lightning events". These "false lightning events" are not rejected by the real-time event processor (RTEP) because their temporal and spatial characteristics make them too difficult to instantaneously categorize them as either "signal" or "noise". Once the LIS data is transmitted to earth, the events are post-processed using software filters which remove a high percentage of the "noise" events while preserving a high percentage of the "signal" events. This is accomplished by developing algorithms that examine the events for characteristics that indicate whether an event is more likely to be "signal" or "noise". Since noise can come in many different forms, several software filters have been specifically designed for LIS to eliminate or greatly reduce the number of noise events produced by solar glint, radiation hits to the sensor, thermal characteristics of the electronics, as well as hardware bandwidth limitations of the sensor.

An understanding and knowledge of the characteristics of the "signal" and "noise" events comes from experience with the OTD, the flight qualified engineering model to LIS. These filters have been developed and tested using the OTD data.
 

3.6.1 Solar Glint Filter

The solar glint filter is designed to scan through the LIS data sets looking for events that are caused by the sunlight reflecting off of the surface of the earth. Since the location of the satellite relative to the sun and the earth is known at all times, the periods when solar glint is most likely to be observed by the LIS instrument is also known. Any events that occur where glint is likely to be observed are filtered from the LIS data set. Due to the orbit of the TRMM satellite, the false events caused by glint can only occur only once an orbit for a period lasting less than one minute.
 

3.6.2 Radiation Filters

The orbit of the TRMM satellite will take the LIS instrument though some known high radiation locations. Since charged particle collisions with the LIS CCD sensor can create an electrical response that is in many ways similar to light photons hitting the CCD array, it can be very difficult to distinguish a single event caused by radiation from a single event caused by the burst of light produced by a lightning flash. To eliminate as many radiation related events from the data as possible, two types of radiation filters have been designed.
 

3.6.2.1 Streak Filter

The first radiation filter is named the Streak Filter. It is designed to eliminate radiation-related events caused by charged particles impacting the CCD array at acute angles. When charged particles impact the CCD array at low angles, the charged particles skim across multiple pixels in a straight line, causing a "streak" of events to be recorded. Typically, when the trajectory angle of the charged particle is small, the streak is several times longer than it is wide. The result of a single low-angle trajectory can produce in excess of 50 false events. This streak filter examines each 2 ms sample period for streaks, and all neighboring events that form a footprint that is significantly longer than it is wide are assumed to have been produced by radiation and are eliminated from the data set.
 

3.6.2.2 Random Filter

A second radiation filter, the Random Filter, is used to detect and eliminate radiation-related events that are caused by charged particles impacting the CCD array at higher angles. The footprint associated with this type of impact is not much longer than it is wide because the charged particle does not pass across as many pixels, causing fewer events. The number of events resulting from a single collision of this type is usually less than four neighboring events, making this false signal appear very much like a lightning produced event. As a result, it is necessary to use both spatial and temporal event information in order to determine which events are likely to have been caused by lightning rather than radiation. The methodology behind this filter is the assumption that lightning produced events have a different temporal and spatial distribution from radiation-related events. For example, a single lightning flash typically produces multiple optical pulses of light in the same location in a period less than a second, while charged particle collisions with the CCD sensor should be randomly distributed in space and time. By examining the event data in one second intervals, it is possible to assign the events with a probability of being randomly distributed based on the number of events examined and the number of events that occur in the same location (i.e., coincident in pixel space). The radiation filter was specifically designed to statistically eliminate 90% of the false events due to radiation while preserving over 99% of the lightning related events. In addition, the Random Filter also eliminates 90% of the electronic noise, since this type of noise signal also produces events that are spatially distributed in a random fashion.
 

3.6.3 Spatial Probability Density Filter

Although 90% of the radiation-related events will be eliminated from the data by the "random filter", approximately 10% of these events will remain in the data set among the lightning produced events. To eliminate as many of the remaining "false events" as possible, another filter, named the Spatial Probability Density Filter is applied to the data. The radiation-related events remaining in the data exist as discrete groupings, where each grouping is composed of multiple events that are within the one-second examination interval implemented by the Random Filter. Although the radiation-related events that comprise a grouping are not spatially random compared to each other (within the one-second interval), the groupings themselves are spatially random when compared to other groupings occurring over a longer period of several minutes. The Spatial Probability Density Filter tallies the number and probability of events that occur in a 0.15 latitude by 0.15 longitude regions, and if enough high-probability events of a high enough probability occur in this region, then all the events in geo-located to this region are passed by this filter. If there are not enough high-probability events that are geo-located to this 0.15 latitude by 0.15 longitude region, then the events are rejected by the filter. This filter effectively acts as a threshold to remove events that do not occur in sufficient densities to be accepted as a lightning produced event.
 

3.6.4 Ghost Filter

Due to hardware bandwidth limitations of the sensor electronics, as many as three "ghost" events will be recorded for every real event. These ghost events can easily be identified and filtered from real events since the timing and pixel locations of the ghost events are predictable. If the amplitude of the real event detected at a given pixel is sufficiently high, a ghost event will falsely be detected at the next pixel in the CCD pixel read-out sequence. Sometimes, if the amplitude of the real event is extremely high, then a ghost event will falsely be detected at the next 3 pixels in the CCD pixel read-out sequence. The amplitudes of ghost events are predictable and are a function of the amplitude of the first real event in the read-out sequence. In effect, the Ghost Filter eliminates ghost events by checking for events that occur sequentially in the CCD pixel read-out sequence and eliminating events that have amplitudes that are characteristic of ghosts events.

3.7 CALIBRATION, VALIDATION, AND QUALITY CONTROL 

3.7.1 Calibration

LIS calibration can be divided into two general categories. The first of these categories is to obtain an absolute radiometric calibration of the LIS sensor. This calibration is primarily a pre-launch activity, although efforts will be undertaken to determine and monitor how the absolute calibration changes during the lifetime of LIS. The second calibration category is a performance calibration of the LIS that can be fully obtained only after LIS is placed into orbit. Performance calibration is extremely important for the utilization and interpretation of LIS data.
 

3.7.1.1 Pre-launch Calibration (Absolute Radiometric Calibration)

The pre-launch calibration primarily addresses LIS radiometric calibration. The pre-launch calibration activities and procedures are described in detail in the LIS Calibration Procedures Document. These activities include: (1) D.C. uniformity, linearity and false alarm rate tests, (2) field-of-view (FOV) test, (3) A.C. response test, (4) detection efficiency, and (5) spectral test.

The D.C. uniformity and linearity test involves exposing the entire LIS FOV to a steady, isotropic optical source and varying the source amplitude level. The D.C. response for each pixel is fully characterized in this test. In addition, the number of false alarms (i.e., transient events such as those produced by photon shot noise in LIS) is counted for different illumination levels.

In the FOV test set-up, the LIS is illuminated with a highly collimated light source whose azimuth and elevation incidence angles are precisely known relative to the LIS boresight. An Euler angle analysis of LIS output data from this test provides a precise mapping between illuminated pixel and associated light source incidence angles. This test also determines the extremities of the LIS FOV.

The A.C. response test involves illuminating selected LIS pixels with a transient optical pulse. This test provides a very precise radiometric calibration of the selected pixels to transient optical signals as a function of D.C. background illumination. During this test several pixels across the CCD array are individually calibrated and characterized (only one pixel is tested at a time). The transient log-amp response of the Real-Time-Event-Processor (RTEP) of LIS is determined from this test.

The A.C. response test can provide an initial estimate of the LIS detection efficiency when the results of the test are coupled with the energy distribution statistics derived from cloud top optical lightning observations taken during high altitude U-2 aircraft storm overflights. Ground truth calibration/validation studies conducted after the LIS is in orbit will be needed to more precisely determine the detection efficiency (i.e., see section 3.7)

The narrow pass-band filter of LIS is measured using a monochromator as part of the spectral test set-up. Center wavelength and full-width at half power are characterized in the LIS spectral test.

Finally, a lightning simulator is used to exercise the LIS instrument. The simulator employs an acousto-optical modulator and a mirror scanner to externally modulate a laser light signal to generate simulated lightning transients. In this test LIS is illuminated by several thousand simulated lightning transient waveforms. The test provides an end-to-end verification of the operation of LIS subsystems prior to launch. It should be noted that, in the present design, the lightning simulator does not produce signals of sufficient quality and stability to be used for radiometric calibration purposes.
 

3.7.1.2  Post-launch Calibration (Performance Calibration)

Once the LIS is launched and becomes operational the performance of the LIS will be characterized and key performance parameters calibrated. A high priority will be placed on quantifying the LIS performance early in the mission and then monitoring this performance throughout the life of the instrument. Changes in LIS performance during the mission might be an indication of drifts in the radiometric calibration of the instrument. This would be precluded if the natural variability of the LIS performance parameters are found to be high. Initial estimates of the performance parameters will be made during the pre-launch calibration activity (i.e., see section 3.7.1.1).

The key performance parameters are the detection efficiency and the false alarm rate. The detection efficiency is defined as the percentage of lightning flashes occurring in the FOV of the instrument that are detected by the sensor. False alarm rate is defined as the percentage of total detections that are not attributable to lightning. These performance parameters may display significant dependence on the conditions under which the observations are obtained. These conditions including LIS threshold setting, background intensity, observation time (e.g., time of day, time of year), storm characteristics (e.g., continental vs. maritime, large vs. small, developing vs. decaying, high flash rate vs. low flash rate) and geographical location. The effects of these conditions may be very interdependent and the responses nonlinear.

The performance calibration of LIS will be accomplished through intensive ground truth field experiments. From these field experiments we will produce data bases consisting of coincident observations from the LIS with detailed ground based lightning observations at the TRMM ground truth sites in Florida and elsewhere, and augmented with radar/rain gauge networks, geostationary satellites, and other ground based lightning detection systems. In addition, coincident ground truth measurements will be made using the high altitude ER-2 aircraft (simultaneous lightning, infrared, passive microwave, radar observation will also be obtained with the ER-2). The ground truth lightning data that will be collected and analysed includes, but is not limited to, regional ground based lightning networks, long range sferics networks, interferometry, VHF time-of-arrival, optical and electric field sensors and satellites (e.g., OTD).

The correlative or ground-truth observations made in support of the calibration/validation of the LIS instrument and data processing code will be archived in the GHRC , as well as a record of any changes in calibration parameters and/or software used during the life of the instrument.

The coincident data bases that are assembled for the calibration/validation efforts will also support the LIS science activities. For example, these data bases will contribute to the development of combined instrument precipitation algorithms for TRMM that incorporate lightning observations. These data bases will be used to investigate the ratio of in-cloud to cloud-to-ground lightning. The reader is referred to chapter 2 for an overview and discussion of the scientific contributions of LIS.

3.7.2  Validation

Validation is the process of verifying and tuning the performance of both the data processing algorithm described in this document and the LIS hardware. This process will include (1) remotely adjusting threshold settings to maximize detection and minimize false alarm rate, (2) verifying the true amplitude, time of occurrence, and location of lightning event detections, and (3) verifying background image brightness and alignment. The same data bases being assembled and used for LIS performance calibra