Inside a burning building, alarms are going off — and the first responders on-scene still need to be heard and hear their communications devices. Outside, fellow responders are conversing as they stand next to a fire truck with its pump running, as more emergency vehicles arrive with sirens blaring. As public safety agencies start more commonly relying on commercial LTE devices as they perform their duties, how well are those devices likely to perform in noisy emergency scenarios?
It’s a lot more technically challenging to deliver understandable speech in active emergency situations than to deliver good, consumer-grade cell phone audio, according to a new report from Spirent Communications. Although some LTE devices used for public safety purposes can outperform purpose-built public safety devices on audio quality, performance is uneven and there is plenty of room for improvement, the testing company found.
In benchmarking speech intelligibility on LTE devices, Spirent found that “phrase misunderstandings can occur in mission-critical communications scenarios up to almost 50 percent of the time, even when using devices that are marketed specifically for use by the public safety community.”
Spirent tested four devices that support public safety LTE; it did not disclose which devices or their manufacturers.
The company explained that the Mean Opinion Score, the standard by which audio quality is typically evaluated, was originally developed based on input from human listeners. Modern-day MOS algorithms such as POLQA (Perceptual Objective Listening Quality Analysis) are “excellent at evaluating the quality of audio transmission, but they are not sufficient for public safety scenarios,” the company said, because first responders need to communicate in acoustically challenging environments with lots of background noise.
MOS scores are meant to “evaluate the ability of a device and/or a communications channel to accurately reproduce audio. Speech intelligibility, on the other hand, is about the ability of a device to transmit all the critical components of speech in the presence of these background noises so that words and phrases can be easily understood,” Spirent explained.
In order to test speech intelligibility, Spirent created a controlled lab environment in its Maryland Performance Center where it could simulate challenging
acoustic situations from the field. The test subject was a head-and-torso simulator dubbed Horatio, placed in the center of an acoustic isolation chamber. Voice phrases were played out of Horatio’s mouth and captured by a device-under-test, while real-world recordings of background noise were played through speakers placed around the torso in order to create 360-degree sound reproduction, and upstream audio was recorded at the receiving end, the company said.
The testing relied on the ABC-MRT16 algorithm, which was developed by the U.S. National Telecommunications and Information to mimic the human auditory system, and rhyme testing: a sequence of rhyming words with a single consonant difference, such as: sane, name, game, tame, cane, fame.
The company said that the results were “eye-opening.”
“In some typical first responder noisy environments, phrase misunderstanding occurred almost 50% of the time for some devices as compared to less than 10% of the time for others,” Spirent concluded.
Spirent played more than 1,200 phrase samples for each test run, with 300-word samples spoken by four different speakers in a quiet test room and then with different “noise profiles,” including “nightclub,” a K12 fire rescue saw, a fire truck pump panel and two others. The devices were run in both speakerphone mode and non-speakerphone mode, with nearly 10,000 test samples in a full sweep, the company said. Spirent said that it is the first testing company to provide that type of speech intelligibility service, aimed at both helping mobile network operator determine which devices will work best on mission-critical networks, and also providing feedback for device developers and OEMs on how they can improve device audio performance in noisy environments.
No device was perfect, but the top performer was better than the others “by at least a six-to-one margin in certain emulated environments,” Spirent said. But it noted that “great performance in one environment did not necessarily correlate with similar performance in other emulations. … In some cases, consumer-grade devices outperformed devices purpose-built for PS-LTE use.”
“Our testing looked at certain devices through the lens of how well they were able to transmit the essentials of speech while suppressing most background noise, because that’s where the largest performance gaps emerge in public safety applications,” said Saul Einbinder, VP of product marketing for Spirent. “The significant performance discrepancies we uncovered indicate that device makers must further re-evaluate designs and components to assure performance on the mission-critical LTE networks being rolled out around the world.”
“Assuring high-quality speech intelligibility during mission-critical operations is essential,” said Ken Rehbehn, directing analyst at IHS Markit, and himself a former 911 dispatcher and current firefighter/EMT in Montgomery County, MD. “Distorted communications reduces incident command effectiveness and, in the extreme, can lead to career-ending injury or line-of-duty death. Device vendors serving the public safety market have a profound duty to do everything possible to maximize speech intelligibility.”
“While our testing found substantial room for improvement in some cases, the good news is that there is a pathway to better performance,” said Einbinder. “This is absolutely a solvable problem. First responders need to trust that the devices they use for lifesaving measures will perform when it matters most, and we believe pinpointing performance challenges that exist early in the development process will help the entire industry take a major step toward that goal.”
Read the full details of Spirent’s testing here.
