Data Munging in R Programming

Data Munging in detail

Data Munging refers to the process of transforming raw or erroneous data into a clean and usable format. Without data munging—whether done manually by a user or through an automated system—the data is often unsuitable for downstream analysis or consumption. Essentially, data munging involves cleansing and reformatting data manually or using automated tools.

In R Programming, the following methods are commonly used for the data munging process:

• apply() Family
• aggregate()
• dplyr package
• plyr package

Using the apply() Family for Data Munging

The apply() function is one of the foundational functions in R for performing operations on matrices or arrays. Other functions in the same family include lapply(), sapply(), and tapply(). These functions often serve as an alternative to loops, providing a cleaner and more efficient approach to repetitive tasks.

The apply() function is particularly suited for operations on matrices or arrays with homogeneous elements. When applied to other data structures, such as data frames, the function first converts them into a matrix before processing.

Syntax:

apply(X, margin, function)

Parameters:

• X: An array or matrix.
• margin: A value (1 for rows, 2 for columns) indicating where to apply the function.
• function: The operation or function to perform.

Example:

# Example of apply()
matrix_data <- matrix(1:12,
                      nrow = 3,
                      ncol = 4)
matrix_data

result <- apply(matrix_data, 2, sum)
result

Output:

[,1] [,2] [,3] [,4]
[1,]    1    4    7   10
[2,]    2    5    8   11
[3,]    3    6    9   12

[1]  6 15 24 33

The lapply() Function: The lapply() function operates on lists and returns a list of the same length. Unlike apply(), it does not require a margin parameter. The “l” in lapply() signifies that the output is always a list.

Syntax:

lapply(X, func)

Parameters:

• X: A list, vector, or object.
• func: The function to apply.

Example:

# Example of lapply()
fruits <- c("APPLE", "BANANA", "CHERRY", "MANGO")
fruits

lowercase_fruits <- lapply(fruits, tolower)
lowercase_fruits

Output:

[1] "APPLE"   "BANANA"  "CHERRY"  "MANGO"

[[1]]
[1] "apple"

[[2]]
[1] "banana"

[[3]]
[1] "cherry"

[[4]]
[1] "mango"

The sapply() Function: The sapply() function works similarly to lapply(). However, it tries to simplify the output into a vector or matrix if possible.

Example:

# Example of sapply()
fruits <- c("APPLE", "BANANA", "CHERRY", "MANGO")

lowercase_fruits <- sapply(fruits, tolower)
lowercase_fruits

Output:

[1] "apple"  "banana" "cherry" "mango"

The tapply() Function: The tapply() function is used to perform an operation on subsets of data grouped by a factor. It is particularly useful for aggregating data.

Syntax:

tapply(X, index, func = NULL)

Parameters:

• X: A vector or object.
• index: A factor or list of factors for grouping.
• func: The function to apply.

Example:

# Example of tapply()
data(iris)

species_median <- tapply(iris$Sepal.Length,
                         iris$Species,
                         median)
species_median

Output:

setosa versicolor  virginica
5.0        5.9        6.5

Using aggregate() in R

To summarize data by grouping variables and applying a function (e.g., sum, mean).

Syntax:

aggregate(formula, data, function)

Parameters:

• formula: Specifies the variables for grouping.
• data: The dataset for aggregation.
• function: The operation to perform on the grouped data.

Example:

exposures <- aggregate(
  x = assets[c("counterparty.a", "counterparty.b", "counterparty.c")],
  by = assets[c("asset.class", "rating")],
  FUN = function(market.values) { sum(pmax(market.values, 0)) }
)

Using the plyr Package

A versatile package for splitting, applying functions, and combining data.

Key Functions:

• ddply(): Operates on data frames.
• llply(): Operates on lists.

Advantages:

• Simplifies operations with consistent syntax.
• Offers parallel computation and progress bars.

Example with ddply():

library(plyr)
ddply(dfx, .(group, sex), summarize,
      mean = round(mean(age), 2),
      sd = round(sd(age), 2))

Using the dplyr Package

Purpose: Provides a consistent grammar for data manipulation with verbs like arrange, filter, mutate, select, and summarize.

Advantages:

• Fast and efficient backend.
• Easy-to-read pipe (%>%) syntax.

Examples:

• Arrange rows:

starwars %>% arrange(desc(mass))

• Filter rows:

starwars %>% filter(species == "Droid")

• Mutate new variables:

starwars %>% mutate(bmi = mass / ((height / 100) ^ 2)) %>%
            select(name:mass, bmi)

• Summarize grouped data:

starwars %>% group_by(species) %>%
            summarize(n = n(), avg_mass = mean(mass, na.rm = TRUE)) %>%
            filter(n > 1)

Example:

library(dplyr)

# Group by gender, summarise, and filter
starwars %>%
  group_by(gender) %>%
  summarise(
    n = n(),
    avg_height = mean(height, na.rm = TRUE)
  ) %>%
  filter(n > 3)

Output:

Assuming the starwars dataset is unmodified:

gender	n	avg_height
male	60	178.41
female	16	165.56