String Tokenizer

Part of the Logic & Flow section of Coddy's Java journey — lesson 48 of 59.

StringTokenizer is used to break a string into tokens (smaller parts) based on a delimiter. By default, it uses whitespace as a delimiter.

Create a StringTokenizer with default delimiter (space):

StringTokenizer st = new StringTokenizer("Hello World Java");

Count and get tokens:

int count = st.countTokens();
// Gets number of tokens
String first = st.nextToken();
// Gets next token

After executing the above code, the variables contain:

count: 3
first: "Hello"

When you call nextToken(), it removes that token from the tokenizer. The countTokens() method returns the number of remaining tokens.

StringTokenizer st = new StringTokenizer("Hello World Java");
System.out.println(st.countTokens());  // Prints: 3

st.nextToken();  // Consumes "Hello"
System.out.println(st.countTokens());  // Prints: 2 (not 3!)

st.nextToken();  // Consumes "World"
System.out.println(st.countTokens());  // Prints: 1

Tip: If you need to know the total token count, call countTokens() before consuming any tokens. Use hasMoreTokens() to check if there are more tokens available while iterating.

Challenge

Easy

Create a method named tokenizeText that takes two arguments:

A String (text) to tokenize
A String (delimiter) to split by

The method should:

Create a StringTokenizer with the given text and delimiter
Return a string containing the token count followed by each token on a new line

Return messages should be:

If text is null: return "Invalid text"
If delimiter is null: use space as delimiter
Format: "Token count: X\nToken: token1\nToken: token2\n..."

Cheat sheet

StringTokenizer breaks a string into tokens (smaller parts) based on a delimiter. By default, it uses whitespace as a delimiter.

Create a StringTokenizer with default delimiter (space):

StringTokenizer st = new StringTokenizer("Hello World Java");

Count and get tokens:

int count = st.countTokens();  // Gets number of tokens
String first = st.nextToken(); // Gets next token

Create StringTokenizer with custom delimiter:

StringTokenizer st = new StringTokenizer(text, delimiter);

When you call nextToken(), it removes that token from the tokenizer. The countTokens() method returns the number of remaining tokens.

StringTokenizer st = new StringTokenizer("Hello World Java");
System.out.println(st.countTokens());  // Prints: 3

st.nextToken();  // Consumes "Hello"
System.out.println(st.countTokens());  // Prints: 2 (not 3!)

st.nextToken();  // Consumes "World"
System.out.println(st.countTokens());  // Prints: 1

Tip: If you need to know the total token count, call countTokens() before consuming any tokens. Use hasMoreTokens() to check if there are more tokens available while iterating.

Try it yourself

import java.util.Scanner;
import java.util.StringTokenizer;

public class Main {
    public static String tokenizeText(String text, String delimiter) {
        // Write your code here
    }
    
    public static void main(String[] args) {
        Scanner scanner = new Scanner(System.in);
        String text = scanner.nextLine();
        String delimiter = scanner.nextLine();
        
        if (text.equals("null")) text = null;
        if (delimiter.equals("null")) delimiter = null;
        
        System.out.println(tokenizeText(text, delimiter));
    }
}