R Tutorial

Handling Dates and Times in R

Video Tutorial

  1. Watch Introduction to dates and times

  2. Watch Importing dates

  3. Watch Issues with character dates

  4. Watch Formatting dates

  5. On your own, create some new dates in R and use as.Date() and the date formatting codes to read in different date formats.

  6. Watch Importing Dates with Times

  7. On your own, format the datatime$datetime column using posixlt()

  8. Watch Using Lubridate

Written notes (will vary from videos, though many concepts remain the same)

If you are recording a date, what are the different ways you could write today’s date? If it always the same order of information (month, day, year). Are those pieces of information always given in the same way (always numerical?).

[Instructor: write on board the different ways the students suggest writing today’s date]

What are some issues you see with these formats? How do you think a computer would see these different styles for dates? As the same thing? Or as different things? How would it know what is a day or month or year?

We’re going to explore today what the challenge is for communicating dates and times to computers and learn to use some tools that will make it easier for us to tell the computer what we want in a way that lets both it and us interpret date-times accurately.

First, we need to load three packages. We’ll use the ggplot library for plotting and the lubridate package which has some nice features for working with dates and times.

library(ggplot2)
library(lubridate)
library(dplyr)

We’re going to use some data from a weather station from one of the sites that make up the National Ecological Observatory Network . Let’s get that loaded up and make sure everything is working for everyone.

daily = read.csv("NEON_Harvardforest_date_2001_2006.csv", stringsAsFactors = FALSE)

In RStudio, under the environment tab in the upper right, click on daily to see what we loaded. We see a date column, and an air temperature column. The data I gave you is daily air temperature from 2001 through 2006.

Out of curiosity does anyone have a date that looks different from year-month-day?

[Instructor: the date will get reformatted if a student opens the file then saves in Excel. This is a good time to educate the class on Excel’s date habits. ISO international not a default, but you can force excel to behave by reformatting the column and choosing the year-month-day format]

Let’s see what type of data the computer thinks this date is.

class(daily$date)

The computer thinks this is just character data. These could be names of species, sites, whatever, it will treat dates just like any other character data that we would give it.

To understand what that means, let’s plot our data.

daily = daily |> arrange(date)
ggplot(daily, aes(x=date, y=airt)) + geom_point()

Anything look odd or annoying?

Answer: x axis

What if I told you I took out an entire YEAR of data? I removed 2005 from your data file. Anything wrong now?

When dates are stored as characters, the computer treats them just like any other string of text, whether that’s species names or locations or whatever. It sorts that data character by character.

So if we have a list of dates: [Write on board: 2023-01-25, 2024-01-20, 2026-01-24]

When dates are in this format called ISO, sorting still works, but not for the reasons we assume for date. What if we change to:

[Write on board: 01-25-2023, 01-20-2024, 01-24-2026]

Answer: the dates are now sorted character by character and are out of order from a date perspective but in order from a character perspective

But even in ISO format the computer has no way of knowing that there’s a difference in the spacing between 2023 to 2024 and 2024 to 2025. It’s like expecting it to know that there is a gap of potentially existing names between Henry and Ramon.

Because dates have special rules for how they work, the computer needs to store date information differently and we need to tell it to do that.

lubridate is an r package which makes working with dates and times easy. It contains a suite of functions designed to make importing dates and tinmes easy. All you need to do is know what order the date information is being stored in your data. So let’s look at the data again

[Instructor: switch to the daily data in Environment or head(daily)]

What is the order of our date?

Answer: Year, Month, Day

Then that’s the function we use from lubridate.

daily$date = lubridate::ymd(daily$date)
head(daily$date)

Nothing looks like it changed, but if we examine what type of data the computer thinks this is now:

class(daily$date)

Now let’s plot this:

qplot(x=asdate.date, y=airt, 
      data=daily,
      main="Daily Air Temperature")

Our missing year appears because the computer now knows that December 31 of 2004 is 1 year away from January 1 2006 and that it should space those data points accordingly.

And what happened to the x-axis?

But what if our date is set up differently?

test_date = "8 July 2009"
output = lubridate::dmy(test_date)
output

Once you have converted your date into a date object you can extract bits of the date using special lubridate functions.

lubridate::month(output)

You can also filter your data based on dates and have it behave as you think it should. We’ll take our daily data and filter to just the data before the gap (i.e. all dates before 12/31/2001)

Subsetted = daily |> filter(date > lubridate::ymd("2001-12-31"))
head(Subsetted)

If you have automated data, like from a sensor or weather station, you may also have a time associated with your date. Let’s look at the higher frequency data from the NEON site. This is weather data collected every 15 minutes during 2005.

quarterhour = read.csv("NEON_Harvardforest_datetime.csv", stringsAsFactors = FALSE)
head(quarterhour)

If we have time as well as date information, we simply add that information in the order that it is presented.

quarterhour$datetime = lubridate::ymd_hm(quarterhour$datetime)
class(quarterhour$datetime)

POSIX is the datetime format that includes time related information.

Importing timeseries data into a tsibble

Because dates and times have special properties, many time series analyses
require the data to be loaded into special formats that help the computer recognize and work with the data correctly. The fornat we’re going to work with a lot this semester is called a tsibble. So let’s end today’s tutorial by learning how to load our data into this special type of R dataframe.

To turn your data into a tsibble, you just need two steps: 1) Find your date column 2) Convert date to date object using lubridate 3) convert to tsibble by using as_tsibble() giving it your data and telling it what column your date is in. Let’s try this on our neon daily air temperature.

So let’s do this exactly as you would for the first time.

library(tsibble)
daily = read.csv("NEON_Harvardforest_date_2001_2006.csv", stringsAsFactors = FALSE)

Let’s look at our data. What’s the date format?

Then let’s call the appropriate lubridate function and convert that column.

daily$date = lubridate::ymd(daily$date)

Then we can convert our normal dataframe into a tsibble by using as_tsibble from the tsibble library and telling it what column has our date info using the index argument

ts_daily = tsibble::as_tsibble(daily, index=date)

Then let’s plot to make sure it worked

ggplot(ts_daily, aes(date, airt)) + geom_point()

You can work witha tsibble just like any dataframe.

data_pregap = filter(ts_daily, date < ymd("2005-01-01"))
ggplot(data_pregap, aes(date, airt)) + geom_point()
  • In class exercise

Import the quarterhour data, convert the date time column and import into a tsibble and plot.

quarterhour = read.csv("NEON_Harvardforest_datetime.csv", stringsAsFactors = FALSE)
quarterhour$datetime = lubridate::ymd_hm(quarterhour$datetime)
ts_quarterhour = as_tsibble(quarterhour, index=datetime)
ggplot(ts_quarterhour, aes(datetime, airt)) + geom_point()
  • Homework

Load the Portal data into a tsibble for homework. Data file is available on the assignment page. Load your Rscript into canvas for the assignment.