Replacing a station’s heart

Move to PBS’s ACE master control gets mixed reviews

Originally published in Current, Sept. 25, 2006
By Steve Behrens

When Ed Caleca came to PBS a decade ago, he expected to see it through completion of the digital TV transition in ’06 before moving on.

But media switchovers don’t stick to schedule. With PBS in unending, overlapping transitions, Caleca and colleague Andre Mendes are leaving, the network told stations recently. Caleca quit his vice presidency this month, and Mendes, chief technology integration officer, departs this week to be global chief information officer for a not-yet-disclosed international organization. Consultant Eric Wolf will handle some v.p. duties while PBS seeks a successor.

The former v.p. will remain a PBS consultant for months, however, trying to wrap up preparations for the net’s Next Generation Interconnection System (NGIS) as well as loose ends of a smaller project led by Mendes — the Advanced Control Environment.

In the past year ACE has demonstrated in miniature the range of tribulations and triumphs, complexity and surprises, of techno-transitions to come. With varied results, six stations have installed the PBS-coordinated package of hardware and software to replace the master control setups at the heart of their operations. Leaders at five of the stations say their ACE packages work as advertised in most respects by now; some are quite pleased.

Iowa Public Television, the other station, may have the biggest beef about ACE. After installing ACE, the state net still uses its old master-control setup, according to spokeswoman Jennifer Konfrst. Officials otherwise declined to talk. “The network is not at liberty to discuss ACE at this time,” Konfrst says.

PBS Chief Operating Officer Wayne Godwin summarizes ACE much as the station reps do: “I think it’s technically a fine system, the vendors are good, the equipment is good,” but the early installations “were simply not good experiences,” Godwin says. “Our first two to three clients were in my opinion not well served by PBS.”

ACE also disappointed PBS financially. “At this point, we certainly have not maximized our investment,” Godwin says. The network planned to install 25 to 30 sites, Godwin says, but only eight stations have taken the plunge so far.

While PBS and vendors concentrate on installing those systems, and then assessing the results, Godwin says, the net won’t put time into selling more packages.

Even chief engineers who suffered with ACE regard its designer, Mendes, as energetic and visionary. Broadcasting & Cable gave him its Technology Leadership Award last year.

ACE had more than a few attractions. It would bring automation to some stations. It would let them operate unattended, with PBS staffers monitoring transmitters from Virginia. It would let them add multicast channels without hiring more technicians. Major components include: Broadview traffic software, which drives six broadcast schedules and tracks underwriting credits; OmniBus automation that executes the schedule; an Omneon server that stores video; Miranda switching and monitoring; and a Masstech archiving system that fetches programs from storage.

As configured for Philadelphia’s WHYY, ACE cost $1.4 million, plus $400,000 for a video storage library, says Bill Weber, the station’s chief technology officer.

PBS had planned to begin using ACE to run its own operations center—and work out the bugs there—before installing it elsewhere. But some stations wanted to install earlier. Albany’s WMHT, for instance, sought to adopt ACE as it moved to its new building. PBS Hawaii wanted out of an expensive master-control outsourcing deal with the local CBS affiliate.

As things turned out, PBS didn’t start using ACE until this April, after difficult installs at three stations last fall.

WMHT, Albany/Schenectady, N.Y., online in September 2005: “The system’s working far better today than a year ago,” says Anthony Tassarotti, chief technology officer. The design and components are very solid, he adds. “As in most complex systems, the devil’s in the details.”

Iowa Public Television, October: Nearly a year later after installation, technicians from PBS and vendors “probably have several months’ more work” in Iowa, says Godwin.

PBS Hawaii, Nov. 1: “It’s been working great for me, now,” says Stephen Komori, v.p. of content delivery at PBS Hawaii. Major glitches were fixed by February. But he remembers working many 18-hour days at first. Programs put in the digital archives sometimes could not be retrieved, he says. The screen might go black for 10 minutes before PBS learned to monitor broadcasts more reliably.

New Hampshire Public Television, February: “The parts are working, but not all the pieces are talking as well as they might,” comments Brian Shepperd, director of technology. So far, he says, PBS had not delivered on a capability that made ACE a good buy for NHPTV. Though it enables PBS techs to monitor images from remote New Hampshire towers, ACE doesn’t yet allow them to control transmitters remotely or fix problems.

WHYY, Philadelphia, late June: “It’s not perfect,” summarizes Weber, “but we’re confident it’s going to settle in fine.” The station was able to reduce staffing of a shift from two techs to one, he says.

Alabama Public Television, this month: “I was extremely worried when it was our turn,” admits APT’s executive director, Allan Pizzato, but he’s seen no on-air problems in the first week with ACE. “It was a very good process,” he says, helped by excellent trainers from vendors such as Broadview and OmniBus. “PBS has said it made a lot of mistakes,” Pizzato volunteers, “but they also stepped up and made the changes.”

PBS plans to install ACE at two more stations — WITF in Harrisburg, Pa., in November, and WSIU, Carbondale, Ill., in February. A ninth station, KUAC in Fairbanks, Alaska, signed for ACE, but changed plans, PBS said.

Any time you bring a massive system online, you bump into problems you hadn’t expected,” Godwin observes. After planting satellite dishes across the country many years ago, for example, pubTV discovered it would need ways to keep many from continually collecting snow, he recalls.

In an ambitious 1990s project, PBS and partners developed Team Approach software for stations’ membership and fundraising needs. When Godwin ran Cincinnati’s WCET, his staff “went through the birthing pains” of adopting the complex software. “We were a bit surprised we needed a larger staff,” he remembers. “It was far too large for what we were trying to accomplish with it.”

In ACE, unfamiliar digital ways of operating took PBS Hawaii technicians by surprise. When the new video server filled to the brim, they called for help. “Weren’t you guys deleting things?” came the reply. “No one told us,” says Komori. “Being one of the first ones, we weren’t aware of how much training would be necessary in the beginning,” he says.

Weber says WHYY had already learned a key lesson in program storage. When its FM station began using an audio server six years ago, managers adopted techniques to maintain the accuracy of data put into the database so the right programming would come out.

Some broadcast engineers say Mendes and other newcomers from information technology don’t know stations’ needs. Computer folk seldom attempt the high reliability required in 24-hour broadcasting, says Leon Messenie, technology chief at KPBS, San Diego. Banks’ teller machines, for instance, can be rebooted every night, but broadcast systems demand utter reliability.

Planners at PBS and the vendors likewise had limited experience with live pledge breaks, so WMHT can to devise ways to work around the software, Tassarotti says.

PBS and the stations didn’t communicate well while ACE was being developed, says Bruce Jacobs, technology chief at Twin Cities Public Television. PBS neither heard the views of station engineers nor went public with analyses by respected consultants Willis Peligian and S. Merrill Weiss, he says. “Most PTV engineers have heard about these studies and can’t figure out why they have become secret, preventing us from the benefit of what was learned,” Jacobs adds.

Godwin explained that a Weiss report for PBS and CPB was proprietary but highly useful in assessing early ACE installations.