Let’s get back to basics. In our first pioneering blogpost it seemed appropriate to start from the beginning: what is automated sports production and what are the technical challenges to solve? This way we hope to help you to navigate your way through the world of automated sports production.
A traditional sports production consists of a (human) camera crew with big cameras and a van with the classic satellite on the roof. This is the kind of setup that is used to produce important sport events like the Champions League or NFL. However, there’s a huge number of matches that are not recorded for the simple reason that traditional productions are too expensive. This is where automated sports production comes in: automated productions don’t require any human intervention and use much less expensive gear. You would simply point a camera to the pitch together with specific software that automatically does the camera work, as if it was a human controlling the camera. The only human steps are: 1) the first installation of the camera and 2) scheduling a match in the software. Having this setup significantly reduces the costs to stream a match, making it much more appealing to a wide audience when compared to a traditional production setup.
If this sounds too futuristic then have a look at the video at the top of this page. How is this possible? Traditional systems rely on rule-based systems to simulate a cameraman. Our software is powered by an end-to-end Artificial Intelligence (AI) algorithm, we exploit the power of AI to its full potential by delivering a system that autonomously extracts cameraman behavior from hours and hours of footage. By doing so we let the AI learn complex patterns that go far beyond the ability of rule-based systems. For this reason we believe AI is the way forward for automated sports production.
The challenges of automated sports production
Mimicking human camera behavior isn’t a trivial task; this comes with many challenges. Furthermore, many practicalities must be solved in order to make it realistic and a user-friendly experience. Thanks to our experience in the field, we’ve been able to come up with 10 essential points that jointly form the 10 pillars of automated sports production.
1. Human-like camera movement
First and foremost: the human touch. In our opinion the production should look and feel human. The camera should follow and zoom in and out on the game, meanwhile doing this with natural movements. This can be regarded as a spectrum with two extremities: on one end there’s a fixed camera that captures the pitch without moving, which would end up in an extremely boring production. Meanwhile on the other end, there’s a camera that is excessively following the game making extreme movements, which would result in seasick viewers. Somewhere on this spectrum lies the perfect balance between the two extremities that make up for human-like camera movements. It might sound simple, but tasks that are simple for humans are often the hardest for computers. We’ve had to put a tremendous amount of effort to replicate human movement, and we’re still not done!
2. High-quality production
The camera setup should consist of affordable gear, meanwhile still having high-quality output. To cover the entire pitch, we often use multiple cameras stacked next to each other. These separate views have to be stitched and dewarped, without leaving any visual artefacts. In other words, we convert multiple camera inputs into one large panoramic output. Furthermore, any form of lens distortion should be removed, which is often found in consumer-grade cameras. Lastly, the output to the viewer should be of high-resolution, e.g., at least Full-HD. This requirement already puts a strain on many networks which calls for efficient video compression.
3. Low latency
Low latency, the time it takes from video to go from the camera to the display (a.k.a. glass-to-glass latency), is often a requirement for many users. Usually, viewers don’t want to watch a ‘live’ match that has a 20-minute delay. The low latency requirement becomes even more crucial for the betting market, which demands a delay of under a few seconds. This latency requirement puts a strain on the entire pipeline, which – for each frame – has to record, simulate the cameraman and stream. This cycle must repeat itself consistently during the entire match in just a few seconds, which in itself is already a technical challenge.
4. Low cost
The entire system should be of low cost to make the system accessible to a wide audience. Obviously, it would be much easier to implement the automatic camera system on a state-of-the-art supercomputer, however, this would significantly increase the costs and as a result the system wouldn’t be accessible to a wide audience. Therefore, it is crucial to implement the camera software on consumer-grade components.
5. Device agnostic
Being device agnostic, i.e., being able to use different cameras or processing units, is important for flexibility. Specially since there is no one-size-fits-all. A top tier football club might require to run in a cloud solution, meanwhile, a small football club without a very stable internet connection should run locally and offline. Similarly, the small football club might settle for a budget camera that meets their needs, while at the same time the top-tier football club might demand a fancier camera with higher resolution. Hence, being device agnostic allows to cater a solution for each unique use case and budget.
6. Robustness
Being robust to external conditions is critical for consistent quality. All kinds of unexpected settings can negatively affect quality. For instance, for outdoor sports the lighting conditions can greatly vary, and the system should be invariant to these changes. Obviously, this issue is less problematic for indoor sports, which mostly have consistent lighting. However, other examples include a safety net which can be installed between the camera and the pitch, the system should also be able to cope with this variable setting. Throughout our journey we’ve seen many examples that attest to the importance of robustness. and the ability to adapt quickly to new variable external conditions.
8. Portability
A portable setup is useful for using one camera setup on different pitches. The crucial part is an easy installation, i.e., having a cumbersome installation process increases the threshold to move a setup from one pitch to another, ultimately limiting the usability of the system.
9. User friendliness
An automated sport production system requires human interaction to schedule recordings. The scheduler – being the only part of the system that requires human input – should be user-friendly and intuitive to use. In addition, the scheduler should also be accessible, i.e., a user should be able to schedule a match from either a phone or a computer.
10. Conclusion
The above 10 points are important and difficult challenges on their own. However, the challenge for automated sports production is to successfully implement them all. The goal is not to only have good quality, but also to make it accessible to everyone and everywhere without compromise.
