How to split JavaScript strings into sentences, words or graphemes with "Intl.Segmenter"
- Published at
- Updated at
- Reading time
- 4min
I've been reading Axel Rauschmayer's post on the new regular expression flag /v
, which explains a way to split emoji strings into graphemes using Intl
.
I haven't used this Intl
object before. Let's find out what it's about!
Consider you want to split user input into sentences. It looks like a quick split()
task... But there's a lot of nuance in this problem.
Here's a naive approach:
'Hello! How are you?'.split(/[.!?]/);
// ['Hello', ' How are you', '']
Using split()
, you'll lose the defined separators and include all these spaces everywhere. And because it's relying on hardcoded delimiters it's not language-sensitive.
I don't speak Japanese, but how would you try to split the following string into words or sentences?
// I am a cat. My name is Tanuki.
'εΎθΌ©γ―η«γ§γγγεεγ―γγ¬γγ'
Common string methods won't be helpful here, but the Intl
JavaScript API is always good for a surprise!
According to MDN, Intl
allows you to split strings into meaningful parts:
The Intl.Segmenter
object enables locale-sensitive text segmentation, enabling you to get meaningful items (graphemes, words or sentences) from a string.
Define a locale and granularity (sentence
, word
or grapheme
) and throw any string at it to split strings into segments.
const segmenterDe = new Intl.Segmenter('de', {
granularity: 'word'
});
const segmentsDe = segmenterDe.segment('Was geht ab, Freunde?');
Headsup: Firefox doesn't support Intl
at the time of writing. On the server-side, it's supported since Node.js 16.
87 | 87 | 87 | 125 | 125 | 14.1 | 14.1 | 14.0 | 87 |
Play around with a tl;dr demo below. π«΅
But let's look at some Intl
details.
Segmenter.segment
returns an iterable
You might have noticed the Array
call in the example above. Segmenter
doesn't return an array but an iterable. To access all segments, use array spreading, Array
or a for-of loop.
const segmenterDe = new Intl.Segmenter('de', {
granularity: 'sentence'
});
const segmentsDe = segmenterDe.segment('Was geht ab?');
// ----
// Access the segments via array spreading
console.log([...segmentsDe]);
// [
// { segment: 'Was geht ab?', index: 0, input: 'Was geht ab?' }
// ]
// ----
// Access the segments via Array.from
console.log(Array.from(segmentsDe));
// [
// { segment: 'Was geht ab?', index: 0, input: 'Was geht ab?' }
// ]
// ----
// Access the segments via for...of
for (let segment of segmentsDe) {
console.log(segment);
}
// { segment: 'Was geht ab?', index: 0, input: 'Was geht ab?' }
Each segment includes the original string value, the character index in the original and the actual segment string.
To map the segments to their string values, the demo uses the lesser known 2nd argument of Array
β a built-in mapping function.
const segmenterDe = new Intl.Segmenter('de', {
granularity: 'sentence'
});
const segmentsDe = segmenterDe.segment(
'Was geht ab? Ich habe frei.'
);
console.log(Array.from(segmentsDe, s => s.segment));
// [ 'Was geht ab?', 'Ich habe frei.' ]
Word granularity comes with an extra isWordLike
property
If you split a string into words, all segments include spaces and line breaks. Filter them out using the isWordLike
property.
const segmenterDe = new Intl.Segmenter('de', {
granularity: 'word'
});
const segmentsDe = segmenterDe.segment('Was geht ab?');
console.log([...segmentsDe]);
// [
// { segment: 'Was', index: 0, input: 'Was geht ab?', isWordLike: true },
// { segment: ' ', index: 3, input: 'Was geht ab?', isWordLike: false },
// ...
// ]
console.log([...segmentsDe].filter(s => s.isWordLike));
// [
// { segment: 'Was', index: 0, input: 'Was geht ab?', isWordLike: true},
// { segment: 'geht', index: 4, input: 'Was geht ab?', isWordLike: true },
// { segment: 'ab', index: 9, input: 'Was geht ab?', isWordLike: true }
// ]
Note that filtering by isWordLike
removes punctuation such as
, -
, or ?
.
Use Intl.Segmenter
to split emojis
And lastly, here's Axel's example that led me down this rabbit hole. I won't get into Unicode specifics, but if you want to split a string into visual emojis, Intl
is a great help, too.
const emojis = 'π«£π«΅π¨βπ¨βπ¦βπ¦';
// ----
// Split by code units
console.log(emojis.split(''));
// ['\uD83E', '\uDEE3', '\uD83E', '\uDEF5', '\uD83D', '\uDE48']
// ----
// Split by code points
console.log([...emojis]);
// ['π«£', 'π«΅', 'π¨', 'β', 'π¨', 'β', 'π¦', 'β', 'π¦']
// ----
// Split by graphemes
const segmenter = new Intl.Segmenter('en', {
granularity: 'grapheme'
});
const segments = segmenter.segment(emojis);
console.log(Array.from(
segmenter.segment(emojis),
s => s.segment
));
// ['π«£', 'π«΅', 'π¨βπ¨βπ¦βπ¦']
Note that graphemes also include spaces and "normal" characters.
I continue to be amazed by the Intl
feature set. There's always new functionality to discover. Intl
enables fairly easy string splitting that considers locales and keeps the delimiters. π
It's yet another Intl
API to make language-dependent string handling easier! I wonder what I'll discover next!
Join 5.4k readers and learn something new every week with Web Weekly.