Do You Use R? Here's a Tool for Importing Campbell Scientific Data

You may be familiar with R for data processing and analysis. But do you know how to easily import data that is in Campbell Scientific’s TOA5 format into R? In this article, I’ll briefly discuss R and TOA5, and then share a function to create an R dataframe from your TOA5.

R – Statistical Computing and Graphics

R is a powerful, open-source software environment for statistical computing and graphics. It is also popular for its ability to produce publication-quality plots with the necessary mathematical symbols and formulas using UNIX, Windows, and MacOS platforms.

TOA5 File Format

The TOA5 file format is the default format used for data collected from any contemporary Campbell Scientific data loggers. This format is a simple comma-delimited text file that has a .dat extension.

The TOA5 file format includes a four-line ASCII header with information about the data logger you’re using to collect your data. In addition, the header describes the data values with variable names and units of measurement—if they are available.

The four lines of the header are the following:

• The first line contains information about the data logger, such as the serial number and program name.
• The second line is what most people would view as the data header as it contains the names of the variables stored in the table.
• The third line consists of the units for the variables—if they have been defined in the data logger.
• In the fourth line, there is an abbreviation for the processing the data logger performed (for example, sample, average, standard deviation, maximum, minimum, etc.).

How to Import a TOA5 File into R

You can find the complete code for the import function at the end of this section. Simply copy the code into a file or the R command line.  A call to this R function has one required parameter—the file name, as well as one optional parameter—the Return Option (RetOpt):

importCSdata(“filename.dat”, “RetOpt”)

If you just want the data with a simple header, all you need is one line of code like this:

myData ‹- importCSdata(“CR1000_HourlyWeather.dat”)

The optional RetOpt parameter defaults to “data,” as in the line below:

myData ‹- importCSdata(“CR1000_HourlyWeather.dat”, “data”)

When the RetOpt (Return Option) parameter is omitted or is “data,” an R data frame is created with the second line of the raw data file used for the names. The TIMESTAMP is converted to what R recognizes as a date and time stamp so that this information can be used in graphing, time-series analysis, or the aggregation of data based on date or time.

Because there is a lot of good information in the header of the TOA5 file, you may also want to simply import the header as documentation for your process by using the syntax below:

myData.Info ‹- importCSdata(“CR1000_HourlyWeather.dat”, “info”)

The complete code for the import function:

importCSdata ‹- function(filename,RetOpt="data"){
	if(RetOpt=="info"){
		# bring in entire header of CSI TOA5 data file for metadata
		stn.info ‹- scan(file=filename,nlines=4,what=character(),sep="\r")
		return(stn.info)
	} else {
		# second line of header contains variable names
		header ‹- scan(file=filename,skip=1,nlines=1,what=character(),sep=",")
		# bring in data
		stn.data ‹- read.table(file=filename,skip=4,header=FALSE, na.strings=c("NAN"),sep=",")
		names(stn.data) ‹- header
		# add column of R-formatted date/timestamps
		stn.data$TIMESTAMP ‹- as.POSIXlt(strptime(stn.data$TIMESTAMP,"%Y-%m-%d %H:%M:%S"))
		return(stn.data)}
}

You can download this code.

Additional Resources

There are far too many resources available for learning R to list them all, but here are a few for getting started or for quick reference:

• DataCamp: while they do a lot of things, here’s a link to a free tutorial on importing data
  • www.datacamp.com/community/tutorials/r-data-import-tutorial
• Quick-R: lots of basic, easy-to-access info
  • www.statmethods.net/
• Cookbook for R: free online version by a different author than the book with a similar title
  • www.cookbook-r.com/
• Cheat sheets by the folks at RStudio
  • www.rstudio.com/resources/cheatsheets/
• Cookbooks: while these aren’t free, I’ve found the O’Reilly R cookbooks very useful
  • R Cookbook by Paul Teetor
  • R Graphics Cookbook by Winston Chang
• And, of course, the internet search for “how do I ____ in r”

Conclusion

I hope you find this function useful in your data processing and analyses. Feel free to post your comments about your experience, or your suggestions, below.

Happy analyzing!

Credits: The R logo is © 2016 The R Foundation. The logo was used without modification and with permission in accordance with the terms of the Creative Commons Attribution-ShareAlike 4.0 International license (CC-BY-SA 4.0).