When Workflow Failure Is Not an Option

The annual Java Jazz Festival in Jakarta, Indonesia, is without question a headline event, drawing more than 120,000 attendees from all over the world, as international and local artists perform on 17 stages during this three-day event. Those unable to attend can experience the sights and sounds through live broadcasts on local cable (First Media) and subscription television (BiG TV), as well as live streaming on YouTube, from four of the main stages.

Yet, making sure every one of the performances is captured without a glitch is a tall order. So is managing the plethora of metadata pertaining to each performance. It is also imperative that not a single note of a performance on the four main stages is missed due to a technical or scheduling issue.

“Any failure is not an option,” says Djundi Karjadi, owner of equipment reseller PT Interindo MultiMedia and technical director of the music festival. It is his responsibility to ensure that the live broadcasts of the event and the recordings of the performances are flawless.

Hoping to improve the workflow year upon year, Karjadi added new equipment to the March 2015 festival lineup. In 2014, six TriCaster 8000s, two TriCaster 460s and one TriCaster 410 served as his main production hub. This year, he took a big step forward with three more 8000s, a 32-terabyte ProMAX Studio storage solution, and a 3Play 4800 replay system. Karjadi was able to run three pairs of TriCaster 8000s in redundant mode, covering the four main stages (two of the smaller main stages shared one pair since no more than three main stages are live at any given time). The new configuration provided a back-up system that could be deployed without any delay should one be necessary.

Meanwhile, the ProMAX storage enabled the crew to organize the large number of ISO clips that were generated from the different stages, on different days, from different shows. As a result, the on-site technical team under Karjadi’s direction could record directly to the ProMAX NAS (network-attached storage) in a single, pre-organized volume, making it easier to manage later during remixing. And, the use of 3Play offered the ability to “time-shift” any of the performances for a continuous, uninterrupted broadcast when a show ran late and the next act on a different stage began.

For the broadcast and live stream, the group used dedicated broadcast mixers and real-time mastering. Among the broadcast and recording equipment at the four main halls were an Avid Venue Profile System with 96-channel input and 16-channel output (at stage D2), an Allen & Heath iLive-T112 64-channel input/32-channel output system (at stage D1), dual Yamaha CL5 Systems (at stages A2 and A3), various Venue cards, Mac Pros running Avid Pro Tools, and other hardware. The mastering equipment, all located at the main stage facility, included a 48-channel input/24-channel output Venue Profile System, a number of Venue cards, and a Mac Pro running Pro Tools. In addition, the group used cameras from Sony (dual-format HDC-1700s and HXC-100s) and Panasonic (AK-HC3800G).

Several sets of AJA FiDO-2T SDI/optical fiber mini-converters and AJA FS2 dual-channel frame synchronizers and converters pulled all the video from the stages to the main control room and de-embedded the audio to feed the live mastering system. Two AJA Kumo 3232s were used for routing.

Bigger, Better, More Challenging

The first Java Jazz Festival, held in 2005, boasted 47,500 attendees, with 125 groups and 1,405 artists performing 146 shows on one main stage and several smaller ones throughout the venue. Karjadi, who had worked for the festival owner at his television station, helped with the broadcast production of the inaugural event and those thereafter. As the festival grew in scope, so, too, did Karjadi’s role and the production workflow.

“Each year I try to tackle something new [for the festival], to make it better,” says Karjadi. For instance, at one point he introduced ISO recording on all the cameras, “so we could do a better postproduction process.”

Back in 2008, the crew first experimented with HD production on one stage. Because they didn’t have any HD recorders, they used at the time newly introduced Sony PWM EX3 cameras, literally shuttling the XD-CAM EX memory cards from the cameras to the production rooms, copying the material to local hard disks, then reformatting the memory card before running it back to the cameraman to be re-inserted into the camera. “There were two cards per camera, so we had to be quick before the other card filled,” Karjadi recalls.

For Java Jazz 2009, the group experimented with a CitiDISK HD (120GB) FireWire hard disk that attached to the camera for recording the DVCPro HD clips.

“With these, at least, you can record the whole show on the same drive. But we still had to copy it back to a local hard disk, then re-format it and stick it back into the camera before the start of the next show,” says Karjadi. “Oh, it was fun,” he adds with a laugh.

These options presented more than one logistical nightmare: Not only was it hard to physically navigate through a sea of tightly packed attendees, it was equally difficult to navigate through individual DVCPro files and organize the resulting data used in the editing. There were, of course, other problems with these methods: Sometimes the cameraman forgot to press Record.

Tired of shuttling the hard disks around, in 2010 the group considered using Omneon Spectrum/MediaDecks for the ingests, but the option was too costly. Instead, they rented “a bunch” of Mac Pros from the local Apple resellers and purchased seven Blackmagic DeckLink cards, and put them all in the production room. “At least we had control and could make sure they were recording before the show started,” Karjadi says.

The year 2010 also was when the group began live streaming from the venue, using First Media’s Cisco encoder, a practice that continued for two years. Then, in 2012, Karjadi heard about the TriCaster 855 – a single box that could record eight HD streams into a set of hard disks. “We could insert graphics and do live streaming straight to YouTube servers in Taiwan,” he says.

According to Karjadi, Java Jazz 2013 was the first time they synched the timecode for all the productions, including the multitrack Pro Tools recording with LTC and WordClock. They also added dedicated broadcast, mixing live mastering for the live broadcasts and streaming.

Working In Concert

For this year’s Java Jazz Festival, four of the TriCaster 8000s were located in the main broadcast room. The other two were at the D1 stage, where a fiber connection enabled them to record directly to the ProMAX storage via the network-attached storage server (NAS). Each camera ISO are recorded twice: one to the NAS and one to TriCaster’s local storage.

“For redundancy purposes, the other would take over and the show would go on,” says Karjadi. With the new equipment, the group was able to record up to three stages simultaneously, providing the necessary backup protection Karjadi was looking for.

One of the challenges after the event is organizing the clips from all the various hard drives during the three days of the event. “Adding ProMAX NAS to the workflow really simplified the whole process,” says Karjadi. “Rather than transporting all the individual drives and copying the clips, all the ISO clips are now recorded directly onto the NAS, plus we have a backup copy on the TriCaster 8000 drives.”

To avoid saturating the Gigabit Ethernet connection, they cross-recorded. For example, the D2 stage used a seven-camera production in addition to the program out (PGM). So, one TriCaster was used to record camera one through four to the NAS, while camera five through seven and the PGM were recorded onto the local drives. Meanwhile, the second machine recorded cameras one through four to the local drives, and cameras five through seven and the PGM to the NAS.

“I was using only four streams on each TriCaster,” says Karjadi. “So I could have six TriCasters recording all at the same time onto the NAS; I didn’t go over the limit of the Internet bandwidth, which is 1Gbit. And we had 24 HD streams recording at the same time, and the ProMAX was only at 15 percent utilization.”

According to Karjadi, when he set up the plan and calculated the bandwidth, it all seemed reasonable. “You can design, plan, and calculate bandwidth. But until you actually try it, there’s always a worry,” he says. “But I was extremely pleased with the result. It all worked as expected.”

Play It Again

Another new addition to the pipeline was the 3Play system, which is typically used for instant replay and slow motion in sports, but this year proved its mettle at the musical festival. As Karjadi explains, sometimes an artist or band will finish their scheduled show late due to a delayed sound check or interaction with the audience during the show. This, in turn, creates scheduling issues as the next act begins on another stage. All the while, Karjadi still has to program the channel smoothly without missing a beat.

While this has been an issue in previous years, a reasonable fix, with a piece of equipment from Omneon, was cost-prohibitive. A few months ago, though, the idea came to Karjadi to use the replay system. “If a stage is running long, we just broadcast until [the show] ends and cue up the next show on 3Play and start it when the other one is finished,” Karjadi says. “I don’t have to cut anything.”

Backstage

As for the other equipment, one of the TriCaster 460s acted as a local master controller for output by the First Media/BiG TV Mix HD channel to a separate local TV station, which had permission to air any of the feeds from the four stages. (The station has its own graphics overlays and schedule that can be different from the YouTube live streams.) The other TriCaster 460 was situated in a small studio where interviews by one of the TV stations were conducted. The TriCaster 410, meanwhile, was used to program the promo LED displays spread throughout the venue that advertised various festival info such as schedules, commercials, and so forth.

The group used the TriCasters to handle animated logos, animated store transitions, animated lower thirds using frame buffers, and GFX to insert Twitter comments into the video. They also used TalkShow, a new offering from NewTek for broadcast-quality live Skype interviews to acquire live audience reactions. “I used the graphics, recording, DVR, live streaming – everything but the virtual sets,” Karjadi says, since there was no need for that function. Before the next festival, he plans to learn Character Works for real-time animation so he can integrate that into the production.

Java Jazz occurs once every 12 months, but the festival owner also hosts two other music festivals each year, all of a similar scope. After finding success with the new workflow at Java Jazz 2015, Karjadi plans to adopt these solutions for the next festival and will modify them as necessary. “We always run into things and have to find solutions year after year,” he says. “We solved the scheduling issues with 3Play, the recording challenge using redundant TriCasters. We always come up with something new to do.”

While there may be more robust solutions available, they all carry a hefty price tag. “What we did is economical and does the job really well,” says Karjadi of his new workflow. “I believe I would get the same great result even if I used equipment that costs much more.”

After speaking with Karjadi, it’s apparent that he ended his work at Java Jazz Festival 2015 on a very high note.

FAST FACTS

Java Jazz Festival

• Held in Jakarta, Indonesia
• A three-day show
• One of the largest jazz festivals in the world
• Presented yearly in early March
• Inaugural event held in 2005
• Features Indonesian and international artists
• Initially held at the Jakarta Convention Center (Senayan) but in 2010 moved to the larger Jakarta International Expo (Kemayoran)
• 2015 performers included Christina Perri, Jessie J., Chaka Khan, Incognito, Chris Botti, Alain Caron
• Peter Gontha founded the Java Jazz Festival
• This year’s event featured 16 stages; four major stages are covered by the tech team
• Broadcast live on local cable (First Media) and DTH subscription television (BiG TV)
• Filmed in HD

• Streamed to two HD YouTube channels Tech Specs

• Equipment includes six TriCaster 8000s, two TriCaster 460s, one TriCaster 410, a ProMAX Studio (32TB), 3Play 4800, TalkShow VS-100, Avid Venue Profile System, Allen & Heath iLive-T112 output system, Yamaha CL5 Systems, Venue cards, Mac Pros running Avid Pro Tools, Sony HDC-1700 and HXC-100 cameras, Panasonic AK-HC3800G camera, AJA FiDO-2T and FS2 (converters/synchronizers)
• TriCaster 8000s run in redundant pairs
• Pushed 24 HD streams simultaneously to storage
• TalkShow VS-100
• Setup required 25 crew members (fiber installers, engineers, operators)
• Group processed 48TB of data (including redundancy) Karen Moltenbrey is a technical writer and editor based in New Hampshire. She can be reached at [email protected].

More on Live Production and Streaming

¿Qué TriCaster es el adecuado para usted?

3Play, Broadcast - Traditional, Broadcast - Web, Customer Stories, Live Production, TalkShow, TriCaster,

3Play, AJA FiDO-2T, AJA Kumo, Allen & Heath iLive-T112, Avid Venue Profile System, Java Jazz Festival, Panasonic AK-HC3800G, ProMax NAS, Sony HDC-1700, Sony HXC-100, TriCaster, TriCaster 8000, Yamaha CL5,