An overview on dformula

Introduction

dformula allows to easily modify, transform, add and extrapolate using the basic R formula. The operations on the data are the following:

Operation	Function
Add new variables	`add()`
Transform existing variables	`transform()`
Rename existing variables	`rename()`
Selection rows and columns	`select()`
Removing row and column	`remove`

The formula is composed of two part:

$column\_names \sim new\_variables$

the right-hand side shows the names of the columns of the data and the left-hand side the transformation or the new variables to insert in the data.

The I() is used in the right-hand side to indicate the type of transformation of the existing variable. In this function, we can insert logical statement, function implemented in R or user build function.

For example:

$$ var\_name_1 + var\_name_2 \sim I(log(var\_name_1)) + I(var\_name_2 == "something") $$

the two variable $var_name_1$ and $var_name_2$ are transformed in $log(var_name_1)$ or selected to be equal to $"something"$.

In the same fashion of SQL, we have the from argument, the input data, and the as argument, the new name of the variables, after transformation, selection or addition.

The CRAN version can be loaded

library('dformula')

or the development version from GitHub:

remotes::install_github('serafinialessio/dformula')

The data are available in the package will be used in this overview

data("population_data")
pop_data <- population_data

which describes the Population and Area of world countries

str(pop_data)
## 'data.frame':    159 obs. of  3 variables:
##  $ Country   : chr  "China" "India" "United States" "Indonesia" ...
##  $ Population: num  1.44e+09 1.38e+09 3.31e+08 2.74e+08 2.21e+08 ...
##  $ Area      : num  9388211 2973190 9147420 1811570 771 ...

`Adding` variables

The add() function inserts new variables starting from the existing columns in the data.

Suppose we want to calculate population density and attach this to the original dataset

new_pop <- add(from = pop_data, formula = ~ I(Population / Area))
head(new_pop)
##         Country Population       Area        Var.1
## 1         China 1439323776 9388211.00    153.31183
## 2         India 1380004385 2973190.00    464.14941
## 3 United States  331002651 9147420.00     36.18536
## 4     Indonesia  273523615 1811570.00    150.98705
## 5      Pakistan  220892340     770.88 286545.68804
## 6        Brazil  212559417 8358140.00     25.43143

and give a name to this new variable

new_pop <- add(from = pop_data, formula = ~ I(Population / Area), as = "pop_density")
head(new_pop)
##         Country Population       Area  pop_density
## 1         China 1439323776 9388211.00    153.31183
## 2         India 1380004385 2973190.00    464.14941
## 3 United States  331002651 9147420.00     36.18536
## 4     Indonesia  273523615 1811570.00    150.98705
## 5      Pakistan  220892340     770.88 286545.68804
## 6        Brazil  212559417 8358140.00     25.43143

Multiple variable can be added with a single formula

new_pop <- add(from = pop_data, formula = ~ I(Population / Area) + I(log(Area)))
head(new_pop)
##         Country Population       Area        Var.1     Var.2
## 1         China 1439323776 9388211.00    153.31183 16.054965
## 2         India 1380004385 2973190.00    464.14941 14.905146
## 3 United States  331002651 9147420.00     36.18536 16.028982
## 4     Indonesia  273523615 1811570.00    150.98705 14.409704
## 5      Pakistan  220892340     770.88 286545.68804  6.647533
## 6        Brazil  212559417 8358140.00     25.43143 15.938746

and with new names

new_pop <- add(from = pop_data, formula = ~ I(Population / Area) + I(log(Area)), 
               as = c("pop_density", "log_area"))
head(new_pop)
##         Country Population       Area  pop_density  log_area
## 1         China 1439323776 9388211.00    153.31183 16.054965
## 2         India 1380004385 2973190.00    464.14941 14.905146
## 3 United States  331002651 9147420.00     36.18536 16.028982
## 4     Indonesia  273523615 1811570.00    150.98705 14.409704
## 5      Pakistan  220892340     770.88 286545.68804  6.647533
## 6        Brazil  212559417 8358140.00     25.43143 15.938746

If we have one transformation applied to a group of variables, we do not specify the function multiple times

new_pop <- add(from = pop_data, formula = Population + Area ~ log())
head(new_pop)
##         Country Population       Area    Var.1     Var.2
## 1         China 1439323776 9388211.00 21.08744 16.054965
## 2         India 1380004385 2973190.00 21.04535 14.905146
## 3 United States  331002651 9147420.00 19.61764 16.028982
## 4     Indonesia  273523615 1811570.00 19.42690 14.409704
## 5      Pakistan  220892340     770.88 19.21319  6.647533
## 6        Brazil  212559417 8358140.00 19.17473 15.938746

and with new column names

new_pop <- add(from = pop_data, formula = Population + Area ~ log(),
               as = c("log_pop", "log_area"))
head(new_pop)
##         Country Population       Area  log_pop  log_area
## 1         China 1439323776 9388211.00 21.08744 16.054965
## 2         India 1380004385 2973190.00 21.04535 14.905146
## 3 United States  331002651 9147420.00 19.61764 16.028982
## 4     Indonesia  273523615 1811570.00 19.42690 14.409704
## 5      Pakistan  220892340     770.88 19.21319  6.647533
## 6        Brazil  212559417 8358140.00 19.17473 15.938746

Suppose we want to add a numerical id for the countries at the beginning of the dataset, using the position argument

new_pop <- add(from = pop_data, 
               formula = ~ I(1:nrow(new_pop)), 
               position = "left", as = "id")
head(new_pop)
##   id       Country Population       Area
## 1  1         China 1439323776 9388211.00
## 2  2         India 1380004385 2973190.00
## 3  3 United States  331002651 9147420.00
## 4  4     Indonesia  273523615 1811570.00
## 5  5      Pakistan  220892340     770.88
## 6  6        Brazil  212559417 8358140.00

We can also add a constant variable. For example the year of the observation

new_pop <- add(from = pop_data, formula = ~ C("2020"), position = "left")
head(new_pop)
##   Var.1 Var.2 Var.3       Country Population       Area
## 1  2020  2020  2020         China 1439323776 9388211.00
## 2  2020  2020  2020         India 1380004385 2973190.00
## 3  2020  2020  2020 United States  331002651 9147420.00
## 4  2020  2020  2020     Indonesia  273523615 1811570.00
## 5  2020  2020  2020      Pakistan  220892340     770.88
## 6  2020  2020  2020        Brazil  212559417 8358140.00

or both

new_pop <- add(from = pop_data, 
               formula = ~ I(1:nrow(new_pop)) + C("2020"), 
               position = "left", as = c("ids", "year"))
head(new_pop)
##   ids year       Country Population       Area
## 1   1 2020         China 1439323776 9388211.00
## 2   2 2020         India 1380004385 2973190.00
## 3   3 2020 United States  331002651 9147420.00
## 4   4 2020     Indonesia  273523615 1811570.00
## 5   5 2020      Pakistan  220892340     770.88
## 6   6 2020        Brazil  212559417 8358140.00

The C() construct add a constant for all the rows

We can be interested in having a dummy variable, i.e. a variable equal to $1$ if some event happen or $0$ otherwise. For example, we suppose to build a dummy variables with the most populated countries. In this we suppose countries with more than $100$ million of people.

new_pop <- add(from = pop_data, formula =  ~ I(Population > 100000000))
head(new_pop)
##         Country Population       Area Var.1
## 1         China 1439323776 9388211.00     1
## 2         India 1380004385 2973190.00     1
## 3 United States  331002651 9147420.00     1
## 4     Indonesia  273523615 1811570.00     1
## 5      Pakistan  220892340     770.88     1
## 6        Brazil  212559417 8358140.00     1

or two variables one with the most populated countries and the other with the biggest extended countries

new_pop <- add(from = pop_data, 
               formula =  ~ I(Population > 100000000) + I(Area > 8000000))
head(new_pop)
##         Country Population       Area Var.1 Var.2
## 1         China 1439323776 9388211.00     1     1
## 2         India 1380004385 2973190.00     1     0
## 3 United States  331002651 9147420.00     1     1
## 4     Indonesia  273523615 1811570.00     1     0
## 5      Pakistan  220892340     770.88     1     0
## 6        Brazil  212559417 8358140.00     1     1

or a variable indicating the most populated and the biggest countries togheter

new_pop <- add(from = pop_data, 
               formula =  ~ I(Population > 100000000 & Area > 8000000))
head(new_pop)
##         Country Population       Area Var.1
## 1         China 1439323776 9388211.00     1
## 2         India 1380004385 2973190.00     0
## 3 United States  331002651 9147420.00     1
## 4     Indonesia  273523615 1811570.00     0
## 5      Pakistan  220892340     770.88     0
## 6        Brazil  212559417 8358140.00     1

If we want obtain a boolean vector, as an interrogation, setting to TRUE the argument logic_convert the function will return a boolean vector

new_pop <- add(from = pop_data, 
               formula =  ~ I(Population > 100000000), 
               logic_convert = FALSE, as = "most_populated")
head(new_pop)
##         Country Population       Area most_populated
## 1         China 1439323776 9388211.00           TRUE
## 2         India 1380004385 2973190.00           TRUE
## 3 United States  331002651 9147420.00           TRUE
## 4     Indonesia  273523615 1811570.00           TRUE
## 5      Pakistan  220892340     770.88           TRUE
## 6        Brazil  212559417 8358140.00           TRUE

`Transform` variables

The transform() function modifies existing variables in the dataset.

Suppose we want to change the scale on the Population

new_pop <- transform(from = pop_data, 
                     formula =  Population ~ I(Population/10000))
head(new_pop)
##         Country Population       Area
## 1         China  143932.38 9388211.00
## 2         India  138000.44 2973190.00
## 3 United States   33100.27 9147420.00
## 4     Indonesia   27352.36 1811570.00
## 5      Pakistan   22089.23     770.88
## 6        Brazil   21255.94 8358140.00

or we want a logarithmic transformation, renaming the variable

new_pop <- transform(from = pop_data, 
                     formula =  Population ~ I(log(Population)), 
                     as = "log_pop")
head(new_pop)
##                                    
## 1         China 21.08744 9388211.00
## 2         India 21.04535 2973190.00
## 3 United States 19.61764 9147420.00
## 4     Indonesia 19.42690 1811570.00
## 5      Pakistan 19.21319     770.88
## 6        Brazil 19.17473 8358140.00

With a single formula multiple variables can be transformed, as showed before.

new_pop <- transform(from = pop_data, 
                     formula =  Population  + Area~ I(log()))
head(new_pop)
##         Country Population      Area
## 1         China   21.08744 16.054965
## 2         India   21.04535 14.905146
## 3 United States   19.61764 16.028982
## 4     Indonesia   19.42690 14.409704
## 5      Pakistan   19.21319  6.647533
## 6        Brazil   19.17473 15.938746

We can also transformed multiple variables with multiple transformations

new_pop <- transform(from = pop_data, 
                     formula =  Population + Area ~ I(Population > 100000000) + I(log(Area)))
head(new_pop)
##         Country Population      Area
## 1         China          1 16.054965
## 2         India          1 14.905146
## 3 United States          1 16.028982
## 4     Indonesia          1 14.409704
## 5      Pakistan          1  6.647533
## 6        Brazil          1 15.938746

`Rename` variables

The rename() function may be used to change names of existing variables, for example

new_pop <- rename(from = pop_data, formula =  Population  ~ pop )
head(new_pop)
##         Country        pop       Area
## 1         China 1439323776 9388211.00
## 2         India 1380004385 2973190.00
## 3 United States  331002651 9147420.00
## 4     Indonesia  273523615 1811570.00
## 5      Pakistan  220892340     770.88
## 6        Brazil  212559417 8358140.00

or multiple variables

new_pop <- rename(from = pop_data, formula =  Population  + Area ~ pop + area)
head(new_pop)
##         Country        pop       area
## 1         China 1439323776 9388211.00
## 2         India 1380004385 2973190.00
## 3 United States  331002651 9147420.00
## 4     Indonesia  273523615 1811570.00
## 5      Pakistan  220892340     770.88
## 6        Brazil  212559417 8358140.00

`Select` variables and rows

In the same fashion of SQL, the select() function first select the rows, given a statement, and then shows the select variables.

The first part of the formula are the columns to select, as the previous functions, and the right-hand side of the formula, the condition part, will select the rows.

Suppose to want to select only the most populated countries

new_pop <- select(from = pop_data, 
                  formula =  . ~ I(Population > 100000000))
head(new_pop)
##         Country Population       Area
## 1         China 1439323776 9388211.00
## 2         India 1380004385 2973190.00
## 3 United States  331002651 9147420.00
## 4     Indonesia  273523615 1811570.00
## 5      Pakistan  220892340     770.88
## 6        Brazil  212559417 8358140.00

you can also add . to returns all variables instead of nothing.

We want only the name of the most populated countries

new_pop <- select(from = pop_data, 
                  formula =  Country ~ I(Population > 100000000))
head(new_pop)
##         Country
## 1         China
## 2         India
## 3 United States
## 4     Indonesia
## 5      Pakistan
## 6        Brazil

We might be interest in only the most populated and biggest countries

new_pop <- select(from = pop_data, 
               formula = . ~ I(Population > 100000000 & Area > 8000000)) 
head(new_pop)
##         Country Population     Area
## 1         China 1439323776  9388211
## 3 United States  331002651  9147420
## 6        Brazil  212559417  8358140
## 9        Russia  145934462 16376870

or both

new_pop <- select(from = pop_data, 
               formula = ~ I(Population > 100000000 | Area > 8000000)) 
head(new_pop)
##         Country Population       Area
## 1         China 1439323776 9388211.00
## 2         India 1380004385 2973190.00
## 3 United States  331002651 9147420.00
## 4     Indonesia  273523615 1811570.00
## 5      Pakistan  220892340     770.88
## 6        Brazil  212559417 8358140.00

by selecting only the names

new_pop <- select(from = pop_data, 
               formula = Country ~ I(Population > 100000000 | Area > 8000000)) 
head(new_pop)
##         Country
## 1         China
## 2         India
## 3 United States
## 4     Indonesia
## 5      Pakistan
## 6        Brazil

`Remove` variables

The remove() function has the same syntax of select() function, but now the rows and columns will be removed.

new_pop <- remove(from = pop_data, 
                  formula =  Area ~ I(Population > 100000000))
head(new_pop)
##     Country Population
## 15  Vietnam   97338579
## 16    Congo   89561403
## 17   Turkey   84339067
## 18     Iran   83992949
## 19  Germany   83783942
## 20 Thailand   69799978

Handling Missing Values

In all the functions, except for rename, the argument na.remove will remove all the rows with missing values, after adding, transforming or selecting the rows.

The remove function, can be employed to remove all the rows with at least a missing observation,

data("airquality")
dt <- airquality

dt_new <- remove(from = dt,formula = .~., na.remove = TRUE)
head(dt_new)
##   Ozone Solar.R Wind Temp Month Day
## 1    41     190  7.4   67     5   1
## 2    36     118  8.0   72     5   2
## 3    12     149 12.6   74     5   3
## 4    18     313 11.5   62     5   4
## 7    23     299  8.6   65     5   7
## 8    19      99 13.8   59     5   8

If we are interested to focus on the observation with missing values, the na.return = TRUE arguments of select function will return only the incomplete rows after the selection

dt_new <- select(from = dt,formula = ~ I(Temp > 50), na.return = TRUE)
head(dt_new)
##    Ozone Solar.R Wind Temp Month Day
## 5     NA      NA 14.3   56     5   5
## 6     28      NA 14.9   66     5   6
## 10    NA     194  8.6   69     5  10
## 11     7      NA  6.9   74     5  11
## 25    NA      66 16.6   57     5  25
## 26    NA     266 14.9   58     5  26

Introduction

Adding variables

Transform variables

Rename variables