Next Stream Ltd. Company Blog

These days IPTV looks pretty much like VoIP did several years ago. No standards, whatsoever. Different manufacturers provide different platforms. Interoperability is nonexistent. And the fact there are several different options, using solutions that intersect with one another makes things ever worse.

First, you have codecs. A great number of those exists. Some you can use for free, for some you must pay royalties. Usually, manufacturers support somewhat proven combinations of audio and video codecs. Most of the time MPEG in different versions is supported. Some IPTV providers, however, use things like RealVideo, WMV, H.263 or VP6/VP7. With those, you must be careful. Not every Set-Top-Box will understand unpopular standards. Test everything before signing anything. It is pretty much the same with audio codecs. MPEG and AAC are generally well supported, but there are exceptions as well.

Next, you are presented with a broad variety of containers. Transport Stream, Program Stream, Windows Media Video, MOV, etc, etc, etc. Again, you may put differently encoded streams in different containers. Some devices understand one type of containers, others will blindly refuse to even look at the content. Anyone dealing with Set-Top-Boxes or Middle-ware should be extremely careful to pick the right combination. You really don’t want to send MOV with MPEG-2 data to a device that “understands” only Transport Streams with MPEG-2 data in them.

To make the matter worse, there is a number of transport protocols you can choose from. Real Time Protocol (RTP) seems very popular. However, keep in mind, that it sometimes comes with RTCP. And your device or middle-ware should be clever enough. Plain UDP is also possible. It is probably the most simple and stable solution. TCP is a valid option too, but its strengths make it somewhat unfit for the duty of real-time streaming protocol. We should also mention the HTTP protocol. For many reasons it is not the right protocol for the task. But it is so easy to implement middle-ware components based on HTTP so many manufacturers and system integrators prefer it. Last but not least, it is the RTSP. Real-Time Streaming Protocol is designed with sole purpose to make streaming real-time data (usually in a separate stream) easy. RTSP is flexible, relatively simple, relatively fast. RTSP is open protocol. It uses others protocols and techniques to describe external streams. And this is somewhat problematic. Some manufacturers expect XML stream descriptions, others expect SDP. From the looks of it, there are no limits, text, binary, you name it, you can use it to describe your streams in a RTSP session. Flexibility does not works for you if you’re stuck with stream descriptions your software is unable to parse.

Last option, we gonna discuss is the “cast” method. There are two option available here: unicast or multicast. Both have pros and cons. Unicast is easier to implement, requires less sophisticated equipment. But it unicast uses lots of network bandwidth. If you have 10 clients and every one of them watches 8 Mbps stream you have 80 Mbps constantly flooding your network. Multicast is more complex. For implementation it requires OS support, modified middle-ware. For deployment it requires “smart” equipment. One that understands IGMP, PIM, etc. But it has one advantage, it doesn’t matter how many clients watch the stream. Required network bandwidth is constant. With one 8 Mbps stream and 100 clients it is still 8 Mbps.

This is all nice. Lots of options, lots of possibilities … until you try to compete in the ever-changing world of IPTV.

Management General, Tech

This was a busy week. Progress is slow, but constant. However, we want to share some thought on appliance development we’re starting at Next Stream Ltd. What should be the main features of an appliance product ? Our Top 3 is:

1. Easy to use
2. Stable
3. Resilient

Appliances must be easy to use. And not only for the technical staff. If you ever need to interact with the device you shouldn’t be expected to know all the technical details on the subject. This is mostly related to user interface. User-Computer(Device) interaction should be straight-forward. Define a goal and let the machine ‘walk the walk’. No one wants to have a full-time job dealing with a bunch of computers constantly needing attention. Simplified user interface with embedded/contextual documentation is a must. Whenever you enter configuration details or click a button, it must be clear what will happen. The contextual help should give you this clarity. Data validation/verification, on the other hand, should prevent you from shooting your leg[s]. So, to make an appliance easy to use:

• minimise number of configuration options
• embed documentation in configuration screens
• always validate input for logical correctness

Don’t let your users make mistakes. It is better to sacrifice configurability than allow malicious actions be taken against your device.

Next on our list, stability. Your device must be stable. It must deliver what’s promised. And it must deliver it under almost all circumstances. Its behaviour must be deterministic. To achieve stability minimise ‘moving parts’. Keep the number of options minimal, keep the number of running components minimal. Test, test, TEST ! Then test some more. We can not stress how important is testing. There are two components that mixed together will give you stability – proper design and testing. So, thing fist, code later. And when coding is finished, test ! Without entering the dark territories of proper tests management, regression testing, fuzz testing, etc, we advise to think about testing procedures during the early stages of development. At least we try to do it this way.

Stable operation is the path to resilience. Once you appliance is stable and predictable enough, you can move to making it resilient. There are many operating systems out there, but only few are really designed to work in any environment. And Linux is not one of those. Nor are most of the other open source projects. Why ? Because they follow design paradigms more than 30 years old, that were meant to accommodate computers of that era. And we all suffer from this. Our every day operating systems are not stable. How can they be resilient ?!

There is work in this direction. QNX, RTEMS, Minix 3 are several examples. New Minix 3 architecture is really impressive. Prof. Tanenbaum and his team came up with the idea of ’state server’. A system process is constantly updated with state information from other processes. When one of those processes crashes, it is automatically restarted and state information is restored in it, so it can continue operation from the moment right before the crash. Which is really a good idea if you want to make your software resilient. The code is out there, so are the papers describing the process in detail. We admire one of the Minix 3 goals, to make the computers reliable like a TV set.

In conclusion we would like to quote a story from Tom Van Vleck (MULTICS developer) about his discussion with Dennis Ritchie (UNIX developer):

… We went to lunch afterward, and I remarked to Dennis that easily half the code I was writing in Multics was error recovery code. He said, “We left all that stuff out. If there’s an error, we have this routine called panic, and when it is called, the machine crashes, and you holler down the hall, ‘Hey, reboot it.’”

30+ years later, it’s the same.

Management General, Tech

We are coming to a success. With some help and lots of work we are hammering problem after problem. We have stable playback on our set-top-boxes. Some modules are still rough around the edges, but the whole system is shaping up. Anyway, we can present Telco/ISP oriented, prototype IPTV system, capable of Multicast and Unicast video distribution. What is more important, flexible enough to adapt to the needs of any Internet Service Providers or Telecoms.

Our IPTV design currently consists of several layers and a number of modules:

• Video Data Acquisition
• Media Library
• User Management
• Load-balanced Content Distribution
• Theme-able User Interface
• HD-enabled Customer Premises Equipment

Basic prototypes of all of the above are present and working. Our work is focused on what I can describe as “commercialisation” of the particular products. We are fixing bugs, adding relevant features. Making devices and appliances user-friendly and intuitive is a main goal. We understand that the packaging sells. We want our customers happy.

But it is not all bells and whistles. We are fighting every day with the lack of standardisation in IPTV. Many protocols exist. Many hardware and software vendors. And all of them usually interpret every protocol in different way. Which is worse, many IPTV related protocols make room for disambiguations. It is usually, up to the implementor to decide how te deal with this. And most implementors do so they favour particular device or manufacturer. Which breaks compatibility, which requires dirty workarounds in the code.

Well, the RFCs give you freedom. They actually let you create unique product. When we speak of IPTV, they even let you break compatibility with proprietary extensions. But please, do know what you are dealing with, and please, do it right. One of my recent findings was that a big IPTV company “extends” particular IPTV related protocol with encapsulation of XML data in the payload. And I can find some logic in this. XML is, de facto, the standard for data representation these days. XML is well understood. XML is wide-spread. Many libraries exist to work with XML content.

What I don’t understand is how you can ship a product that works with malformed XML. And further more, why  would this product refuse to work with correct XML ?!

Please, make all our lives easier. Obey the standards !

Management Advertising, General, Tech

In a previous post we discussed some problems we had with High-definition Set-Top-Box we plan to use for our IPTV work. After several days of work, digging network traffic dumps and patching middle-ware components we can safely say, we fixed this problem. But a new problem emerges. We are currently working on a permanent solution to rework the whole device firmware with the device manufacturer. Great thanks to Michael and John for the cooperation and understanding.

Until we finish that, we plan to release a product targeted at Telcos/ISPs that will take care of acquisition of data for IPTV networks. We would like to provide full, flexible IPTV solution for different setups. Starting from Telcos/ISPs and going “down” to public buildings, advertisement and entertainment installations. Organising appliance requires more than just putting it all together. We want it to have nice look and feel, be intuitive to work with, save you more troubles than it causes. A preliminary design is ready. Some of the prototyping too.

What is more interesting about this appliance is that we will use it to introduce internal project we’ve been working on for some time now. We are talking about secure, centralised configuration solution, scalable enough to handle datacenters. Our goal is to lower the bar for system administration of large number of devices, make it accessible and comfortable. We have several more devices planned down the road that will probably follow the same architectural schematics and depending on the success of this we may even provide stand-alone builds of the configuration system as separate product.

Stay tuned !

Management General, Tech

For IT companies development is substantial, if not the only business. And many of them suffer because of a bad internal organization. If software development is your main business, you must nurture it and protect it until it grows strong. In this post I’ll try to summarize years of good and bad experience working with development companies. I’ll concentrate on Open Source Software solutions and how to use them to your advantage. Required ingredients to build working development infrastructure include:

Electronic Mail: E-mail is the formal communication in your organization. You need scalable and flexible e-mail system. One that support hundreds of aliases, secure communication (TLS), etc. One that has comprehensive web interface. Anti-spam and anti-virus is mandatory. Postfix (MTA) is great Mail Transfer Agent, Spamassassin (Anti-Spam) or DSPAM (Anit-Spam) can handle unwanted messages, ClamAV (Anti-Virus) can deal with harmful code and Cyrus (IMAPS) or Dovecot (IMAPS) can provide IMAP for mail applications. About webmail, I recommend Roundcube.

Source Code Management: Source Code Management (SCM) system is essentially the spine and the nervous system of your development infrastructure. Good SCM will let your developers to concentrate on real work and not revisioning. It will be flexible and non-intrusive. It has to be intuitive, easy to use. It will let you break the ‘client-server’ model. It must has some form of web interface or gui. E-mail Integration is mandatory too. Flexibility is usually underestimated here. In a complex project you may need source code reviews or constant audits. You can let every developer update local source trees from central location, but allow them only commit to a source code reviewer’s repositories. Situation is pretty much the same with branching. Your SCM must provide branching. E-mail notifications are really helpful too. It is often easier to search trough your Inbox rather than trough countless screens of revision diffs. Obvious choices here are Mercurial and Git. I also recommend Subversion to some extent.

Tracking System: Often underestimated, often forgotten, tracking system of some kind is actually helpful. In my opinion there two types of tracking systems, Bug Tracking and Ticket Systems. Doesn’t really matter which one you choose. Integration with SCM, however, does matter. The tighter integration between Tracking System, SCM and E-mail, the better. I strongly recommend Bugzilla or Bugzero for bug tacking and Trac for bug/issue/ticket tracking.

Instant Messaging: Most companies prefer e-mail communications. It is formal, it leaves tracks, it is easy to manage. But people prefer instant messaging. It is easy, it is intuitive, it is usually faster. There is psychological element to this too. When the status of your buddy in ‘Online’ it is harder for your peer to ignore your request. Mail, even E-mail, can be lost or delayed, instant messages are delivered right away, you even get notification for successful delivery or buddy presence. Your instant messaging system must provide several features: secure communication, groups, peer-to-peer communication, peer status notifications, delivery notifications. Jabber is the obvious choice. My favorite is Jabberd2. Ejabberd is also really good server. One alternative to strictly Instant Messaging is IRC. Really viable alternative. It provides everything and once you get to know it, it’s really good, it really fits in your organization. IRCD-Hybrid and InspIRCD may help you here.

Project Management: Being Team Leader or Manager with several different projects, it is hard to keep track of everything. This is what Project Management software is all about. It must provide task assignments, task relations, deadline notifications, reporting. It should help you keep track of your projects without getting in the way. Sometimes the power of Project Management is overestimated. It really doesn’t matter how sophisticated management software is if your developers are lousy. I find dotProject useful.

Documentation: Every project, every process, every workflow is surrounded by tons of documentation. Different revisions, different annotations. different authors. It’s hard to manage it all. Sometimes people change, new people come a board, others leave. To keep the development going you need good documentation and easy ways to manage it. In my opinion, wikis are great for this purpose. The wiki of choice must provide access control, document revisions, annotations, flexible formatting and integration of different file types (at least be able to attach those). Once established wiki can greatly help you. Move all project documentation from hundreds of flying around pages to the wiki. Everything goes there, from coding style and list of project members to overall project architecture. Centralization of documentation will increase productivity. I recommend MediaWiki and TikiWiki.

Given all of the above you need to follow several steps to make your development organization working flawlesly. First of all establish e-mail infrastructure. Create user accounts, using predefined patterns. Create group accounts (aliases) and populate them. Create project accounts and populate them. Then create project repositories in the SCM. Set your SCM to send e-mail notifications on commit to all or a group of developers. Add your projects to your tracking system, again, setup e-mail notifications. Designate who should recieve mails for bug tracking or from tickets. Add your projects to the project management software. Divide each project to tasks and assign those tasks with fixed deadlines. Establish dependencies between tasks. Add project information to the wiki. Instruct every member to put all documentation regarding projects in the wiki. With proper access control this will be available to all related members working on that same project. If you instant messaging system requires additional setup, do it. Send e-mail and/or instant messages to all related peers and the development may start.

From this point on, your development infrastructure requires two things: coordination and backups !

Management General, Tech

Wireless networks (Wi-Fi) surround us for many years. Yet it is surprising how little progress was made. Yes, we have 802.11i, we have 802.11g and 802.11n, but with all the nettops out there, simple wireless infrastructure is not enough. Most service providers use long-range wireless networking for low cost links, sometimes for redundancy. It is hard to cover big areas.

Really ?!

This is what we try to address with one of our projects. We have implemented complete wireless mesh solution and several of its advantages include:

• Unlimited scalability: Unlimited number of nodes. Whenever new area has to be covered, nodes can be added at will.
• Easy to use: One firmware image is enough. Just install our firmware, power the node and that’s it. Every new node will connect to the mesh and start serving as an access point to the potential clients.
• Powerful Monitoring: Every node reports its current network and health status to a centralized monitoring point.
• Powerful NOC Subsystem: You can monitor all your nodes from a single point, check their status and load. Even integrate terrain maps and node location into a specialized GIS.
• Powerful User Management: We provide centralized user management interface with the ability to create multiple user classes and apply restrictions instantly throughout the whole network. Once created, the user account is active, no matter which access point user tries to connect to.
• Flexibility: Our Wireless System works with broad variety of back-end sources and technologies. It can easily be integrated within a telecoms or ISP systems.
• Dynamic Routing: We use state-of-the-art dynamic routing protocols like OLSR and B.A.T.M.A.N. to provide reliable mesh infrastructure.
• Multipoint connectivity: Whenever you need redundancy or load balancing just add another connection to upstream provider. The whole mesh network will benefit from that. Dynamic Routing will take care to forward traffic to active uplinks until others are being repaired and best of all, no users will be affected.

What we are most glad of is that all those things are no longer on the drawing board. Our software empowers rapidly growing wireless service providers in Asia and Europe.

Management Advertising, Tech