Filter and Subset Data in R (subset, [ ], dplyr filter)

Filtering Rows: The Logical Mask

To filter a data frame in R, put a condition in the row position of the brackets: df[condition, ]. The condition evaluates to a vector of TRUE/FALSE, one per row, and R keeps the TRUE rows:

scores <- data.frame(
    name  = c("Ana", "Ben", "Chen", "Dana", "Eli"),
    team  = c("red", "blue", "red", "blue", "red"),
    score = c(91, 74, 88, 95, 62)
)

high <- scores[scores$score > 80, ]
print(high)

Read scores[scores$score > 80, ] from the inside out: scores$score > 80 produces TRUE FALSE TRUE TRUE FALSE, and the brackets keep rows 1, 3, and 4. The trailing comma is not optional. Brackets on a data frame take [rows, columns]; leaving the column slot empty means "all columns". Forget the comma and you are indexing columns instead - one of R's most common beginner errors.

Notice the printed row numbers are 1 3 4, not 1 2 3 - filtering keeps the original row labels. Harmless, but it surprises people.

Multiple Conditions: & and | (Not &&)

Combine conditions with & (AND) and | (OR). Each condition gets its own complete comparison:

scores <- data.frame(
    name  = c("Ana", "Ben", "Chen", "Dana", "Eli"),
    team  = c("red", "blue", "red", "blue", "red"),
    score = c(91, 74, 88, 95, 62)
)

print(scores[scores$score > 80 & scores$team == "red", ])
print(scores[scores$score > 90 | scores$score < 70, ])

The trap: R also has && and ||, and they are not interchangeable with & and | here. The double forms work on single values (they exist for if conditions); given whole columns, modern R (4.3+) stops with an error like 'length = 5' in coercion to 'logical(1)', and older R silently compared only the first row - arguably worse. Inside brackets, always the single & and |. The operators doc covers the distinction in full.

%in%: Match Against a Set of Values

When a column should match any of several values, don't chain == with | - use %in%:

orders <- data.frame(
    id     = 1:6,
    status = c("paid", "pending", "paid", "refunded", "shipped", "pending")
)

open <- orders[orders$status %in% c("pending", "shipped"), ]
print(open)

x %in% set returns TRUE wherever x is one of the values in set. It reads better than status == "pending" | status == "shipped", scales to any number of values, and negates cleanly: !(orders$status %in% c("refunded")).

Selecting Columns

The column slot of the brackets takes names or positions:

scores <- data.frame(
    name  = c("Ana", "Ben", "Chen"),
    team  = c("red", "blue", "red"),
    score = c(91, 74, 88)
)

print(scores[, c("name", "score")])            # by name - prefer this
print(scores[scores$team == "red", "score"])   # rows AND columns at once

Two notes. Selecting by name survives column reordering; selecting by position (scores[, c(1, 3)]) breaks the day someone inserts a column. And asking for a single column returns a plain vector, not a data frame - that is why the second print() shows 91 88 with no table around it. When you need to keep the data-frame shape, add drop = FALSE: scores[, "score", drop = FALSE].

subset(): The Friendly Base One-Liner

Base R ships a wrapper that does rows and columns in one readable call - no $, no comma bookkeeping:

scores <- data.frame(
    name  = c("Ana", "Ben", "Chen", "Dana"),
    team  = c("red", "blue", "red", "blue"),
    score = c(91, 74, 88, 95)
)

print(subset(scores, score > 80))
print(subset(scores, team == "red", select = c(name, score)))

Inside subset() you write bare column names (score, not scores$score) - it evaluates them against the data frame. This is called non-standard evaluation, and it comes with a documented caveat: it resolves names at run time in ways that misbehave inside your own functions (a variable named score in your function can be shadowed by a column named score). The rule of thumb, straight from ?subset: great interactively, avoid it in programming - use bracket indexing there.

The NA Trap

A comparison against NA yields NA, not FALSE - and bracket filtering keeps NA-mask rows as rows full of NA:

df <- data.frame(name = c("Ana", "Ben", "Chen"), score = c(91, NA, 62))

print(df[df$score > 80, ])                     # ghost NA row appears
print(df[!is.na(df$score) & df$score > 80, ])  # explicit and clean

The first result contains a junk row of NAs because row 2's condition was neither TRUE nor FALSE. The fix is to require !is.na() explicitly. subset() and dplyr's filter() both drop NA-condition rows silently - convenient, but know that it is happening. The missing values doc covers why NA spreads like this.

The dplyr Way: filter() and select()

The dplyr package splits the job into two verbs - filter() for rows, select() for columns - with bare column names and no df$ repetition (static; the sandbox runs base R only):

library(dplyr)

scores |> filter(score > 80)
scores |> filter(score > 80, team == "red")     # comma = AND
scores |> filter(status %in% c("pending", "shipped"))
scores |> select(name, score)
scores |> filter(team == "red") |> select(name, score)

filter() drops NA-condition rows automatically and always returns a data frame (never a bare vector), which removes both base gotchas above. The trade-off is a package dependency and the same non-standard evaluation caveats as subset() - see the dplyr intro for the full verb tour.

What You Take Away

• df[condition, ] is the fundamental idiom - and the comma is mandatory.
• Combine conditions with single & and |; && errors on vectors.
• %in% beats chained == for matching a set of values.
• Select columns by name, and remember single columns drop to vectors unless drop = FALSE.
• subset() is the pleasant interactive one-liner; brackets are the programmable one.
• Comparisons with NA produce ghost rows in brackets - guard with !is.na().

Next up: adding and transforming columns - df$new <- ..., ifelse(), and dplyr's mutate().