A bit over a week ago, I finished hosting the virtual X.Org Developer Conference (XDC) 2020 with my friend Arkadiusz Hiler (AKA Arek / ivyl). This conference has been livestreamed every single year since 2012, but this was the first time that we went fully-virtual and needed to have hosts / speakers present from their homes.
Of course, the XDC was not the only conference this year that needed to become fully virtual, so we have been lucky-enough to learn from them (thanks to LWN for its article on LPC2020 and Guy Lunardi), and this blog post is my attempt at sharing some of the knowledge we acquired by running XDC 2020.
Over this blog post, I will answer the questions how we selected jitsi over other video-conferencing solutions, how to deploy it, then present 3 different ways to use it for live-streaming your conference. Let’s get to it, shall we?
Why we selected Jitsi? Scaling concerns!
Given that the XDC is about open source graphics, it felt wrong to use proprietary tools for the conference. This pretty much limited our choices to two options: Big Blue Button(BBB), and Jitsi. Luckily, both are excellent!
BBB is a fantastic all-in-one conference tool that is meant for online learning and collaboration. Rooms are controlled by one-or-more moderators who split participants into multiple sub-room which is useful for group work in an eLearning environment, or for break-out rooms in a conference. It excells at keeping bandwidth/cpu usage low as images are shared rather than a full video stream. This also makes it less likely to hit webRTC-related issues. Sounds great for schools where students could be running anything, from some cheap chromebooks to Windows laptops. The recommended server specifications are 8GB or RAM and 8 cores.
Jitsi on the other hand has a simpler UI centered around webRTC. Users join a room, share their webcam or screen, and talk using their microphones. It is also possible to collaboratively work on a shared document with a local etherpad (an open-source web-based UI for collaborative text edition). Most of the work done by Jitsi is happening client-side, which means that bandwidth and CPU requirements are higher than BBB’s, and that browser compatibility is also more problematic (Chromium is fine, Firefox is problematic, Safari is bust), although I am sure the situation will continuously improve. On the other hand, the server requirements are much lower than BBB. The only real resource-heavy feature is the built-in livestreaming which spawns an X-Server, Chromium, screen grabs the output and streams it to youtube using ffmpeg for the encoding.
XDC had an expected attendance of up to 200 participants, which meant that we needed to make sure that whatever solution we would go for had to work nicely with that many attendees. Additionally, we needed to test this solution ahead of time. Using an integrated solution such as BBB presented the risk that too many attendees would try to join and overload the server which would have prevented the conference from happening. Instead, by limiting the audience to just the speakers, testing was relatively easy (find 10 friends to have a chat with on a test instance), and viewers could just watch the stream live on Youtube (AKA, scaling is not my problem). The inconvenient of such method is of course that we need to provide a way for attendees to interact with the speaker and discuss with each other. For this, we just use IRC since this is what the community is used to. See Arek’s blog post to learn more about maximizing the satisfaction of livestream viewers.
So, in the end, the choice to use Jitsi for XDC2020 came down to… being a chicken and not having the time to organize the needed testing, like the Linux Plumber Conference did! However, in hindsight, I would still not have changed anything (more on that later).
How to deploy Jitsi?
If you are going to deploy Jitsi for a conference, I highly suggest using the “cloud” for this! It brings a lot of convenience for testing, duplicating instances, backups, and it keeps the infra cost extremely low! FYI, for XDC 2020, we used 4 instances of Jitsi (Yes, I am a paranoid chicken!), and the overall cost was less than $39. This includes the test instances I set up, up to a month ahead of the conference.
I personnally selected Digital Ocean because I was already a customer of theirs, and it was easy for me to create a $5/month instance that could be scaled to a $80/month instance with 4 dedicated CPUs and 16GB of RAM just for the conference.
When selecting a location for the VM, choose one that is somewhat at the center of the expected participants. For example, if the conference has a majority of Europeans and Americans (North and South), selecting New York City (NYC) or London (LON) will bring the lowest average latency between participants. In case of doubt, set up multiple instances of Jitsi in different data centers so as you can move people to another server in case of an issue. Check out Digital Ocean’s speed test to experience the effect of world-wide routing on your ping and bandwidth, and ask your attendees to verify ahead of time that they will meet the bandwidth requirements.
When installing Jitsi, I strongly suggest you follow the Docker Self-Hosting Guide if you want a painless deployment. I tried the other two recommended methods and always had issues with the bridge which I never managed to fix. Just go with docker, don’t waste your time!
If you want to customize some style/code, I suggest you modify docker-compose.yml to mount-bind some resources such as a replacement watermark for the jitsi instance, or a new css file.
Finally, I suggest you create a systemd service to auto-start Jitsi on boot. Create the following file at /etc/systemd/system/jitsi.service:
[Unit]
Description=Jitsi
Requires=docker.service
After=docker.service
[Service]
Type=simple
WorkingDirectory=/root/docker-jitsi-meet
ExecStart=/usr/bin/docker-compose -f docker-compose.yml -f etherpad.yml -f jibri.yml up
[Install]
WantedBy=multi-user.target
Don’t forget to enable/start the service after that:
# systemctl daemon-reload
# systemctl enable jitsi
# systemctl start jitsi
Jitsi-based live-streamed conference: 3 levels of professionalism
Okay, let’s now get into the gist of this article! Proposing three different ways of hosting a Jitsi-based conference.
Providing a good experience to livestream viewers is important, especially since it also ends up being the recordings for the conference. As suggested by Arek on his XDC2020 blog post, here are important aspects that should drive the quality up:
- No dead air (the stream appears dead);
- Minimize the room for (serious) errors;
- Focus on the audio quality;
- Embrace existing means of communication.
We will use these ideas when proposing solutions.
Level 1: All participants in a Jitsi room
This is the simplest solution by far. All it requires is to deploy a beefy Jitsi instance, create a room, and invite the participants there.
To start live-streaming, simply click on the settings button, press “Start livestreaming”, then copy the livestreaming key you got from Youtube live!
At this point, you will have two pools of attendees: the live ones on Jitsi, and the viewers-only on Youtube. It is up to you to decide how many people will be on one or the other, but in the event of poor performance of the server, it is possible to just kick people out of the room to keep only the most important people there.
Rather than using Jitsi’s and Youtube’s chat, Arek and I suggest having the same chat system for both groups of attendees. Use your community’s favourite mode of text communication (IRC?) there!
To reduce the amount of dead-air which can happen in-between two talks, and to improve the experience of live-viewers that would just tune in the conference during a break, the organizer should share their desktop showing the name of the next talk and when it will start. It could be as simple as a full-screen text editor containing the data, but if you want to do it very well, having a countdown helps reduce confusion with timezones:
Pros:
- Simple to deploy
- Multiple hosts can take care of asking questions or preparing/showing the next-talk slide
Cons:
- Impossible to improve the sound quality of participants for the viewers on Youtube as Jitsi streams directly
- No pre-testing possible for presenters on the live instance, so the switch from one presenter to another can be shaky
- Can’t change rooms without losing the livestream: if a presenter can’t join, tough luck!
Level 2: Only speakers in a room + streaming using OBS
Level 2 is focusing on improving the sound quality of presenters, and the flexibility of presentation. This can be achieved by moving the streaming from the Jitsi instance to a host’s computer. This enables the streaming of almost anything: video, other jitsi instances, …
For any kind of streaming setup, I would recommend using OBS Studio which has pretty much everything you need built-in. Please follow Arek’s blog post to get tips on how to maximize the quality of your setup!
The live-streaming host should focus on video and sound quality only, leaving another host to deal with collecting questions during the talk, then asking them live on Jitsi.
The major drawback of this solution is that the host’s internet connection is becoming critical to streaming the conference. This means that any power or internet cut will lead to a loss of livestreaming. To reduce the power risk, I suggest buying a UPS to power both the modem, the computer, and all the equiments needed for streaming. To reduce the internet cut risk, I suggest setting a router with a failover to a 4G network so as OBS would just re-establish the connection after losing it.
Pros:
- Flexibility to adjust sound, video, and show any waiting screen with a nice transition
- Setting up of the next presenter can happen off-camera during the breaks
- Possibility to tell speakers how much time is left without the stream hearing it
Cons:
- The host needs to take steps to reduce the risk of internet or power issues during the conference
- Presenters might not be able to set up their system during the break, which increases stress level of everyone
Level 3: Only speakers in a room, tripple OBS setup
For our last level, we will focus on reducing the risk that presenters might not be able to set up their system during the break, and allowing the host to take breaks by having 2 Jitsi and 3 OBS instances.
Why create so many instances? Having 3 OBS instances enables two hosts to share the load. When one is live, the other one can take a break and can prepare for the next talk. The 3rd OBS instance is simply used to switch between the two hosts.
Now, you may wonder why would we need 2 Jitsi instances? Remember what I said about being paranoid and a chicken? Well, I really don’t like touching anything system that is live, so having one Jitsi instance per host enable each host to have full control of their Jitsi and OBS instance, with nothing else happening in the background. This thus reduces the chances of crashes or that setting up the next talk would starve the live room \o/!
Again, see Arek’s blog post for the full reasoning behind this, and how to set it up!
Pros:
- Flexibility to adjust sound, video, and show any waiting screen with a nice transition
- Setting up of the next presenter can happen off-camera during the breaks
- Possibility to speak to speakers without the stream hearing anything (good for time-checking presenters)
- Enables hosts to take breaks every other talk
Cons:
- The hosts needs to take steps to reduce the risk of internet or power issues during the conference
Final thoughts?
Hosting a conference using Jitsi is relatively simple! Now that I have done it, I could host a new one in less than an hour if needed, and that is pretty impressive!
Of course, there are a lot of aspects that will elevate the level of professionalism of your conference, but many of these things can be done as time permits. However, the major jump in quality really comes from improving the audio, and going with at least the level 2 architecture for anything but a breakout room is recommended as it enables a lot more control over what goes in the livestream.
No matter what solution you end up with, I highly suggest you document the way attendees and speakers should interact with the conference. For reference, here are the presenter’s guide and the attendee’s guide.
I guess that’s it! I am sure I forgot many things which commenters might remind me of, so expect updates!