Today, we have another episode of our series of interviews with TwinRed team members. Our goal is to show you how an ad network operates from the inside out, let you backstage and also show you who stands behind the TwinRed company. At the end of the day, every business is about the people.  

In this post, we talk to our CTO Denes Kiss who is responsible for everything that is IT related inside the TwinRed team. That includes managing the developers, QA and internal product ownership, handling server provisioning and maintenance, providing support for both clients and internal requests, defining processes to make the workflows even smoother.

So, go get yourself a cup of coffee and learn how IT infrastructure is built in a high-load ad network with a remote team spread around the globe.

Introduction

Hey, Denes. Can you first of all tell us a bit more about your IT career path?

When I was 13 a friend of mine came by with a book about `Coding in Turbo Pascal`, that point was where my whole career started. A few years later I found myself in a high school trying to sell my very first program which was supposed to help the users to learn touch typing. It never happened, but still motivated me to keep coding and learn more about IT. During university I deep dived into the life of Data Centers as a hosting operator where I learned a lot from industry veterans about IT operations. After many private projects of website coding I ended up in the early era of LiveJasmin as a PHP developer. This was the very first opportunity, where I had the chance to deal with really high-loaded systems. I was lucky to be part of various projects inside the company, like creating a tube site, leading the tech side of AWE (affiliate network) and last but not least creating an ad-network from the ground up.

Following these projects I was honored by the management to take the IT lead of the TwinRed project, where I’m acting as a CTO now.

How do you define your main goal here at TwinRed?

My general main goal is to have a smooth operation and as fast delivery as we can.

Managing remote teams 

To begin with, I would love to know how big is the entire IT team that you are supervising?

Our team is about 20 people that includes the developers, QA, Product owners and DevOps.

What are your main IT departments?

As mentioned, we have 4 large areas:

Product owners: When a new feature request lands in our department, they are the ones  who are translating business needs into a more detailed technical description, covering edge cases, preparing UI mockups, etc. Also they are the ones who are adjusting priorities and bundling releases to match business deadlines.

Quality assurance: Once a Developer completes their part, the task is queued for the QA team to test and review. QA makes sure that everything works according to the specifications and requirements which were provided by the Product owners. One of their major responsibilities is to make sure that new features do not break and align with existing ones.

IT Operations(DevOps): The behind the scene guys. They provide the right infrastructure to work with, things like provisioning hardware for new features, keep everything up-to-date, providing access to various systems and of course working closely together with the developers on several tasks.

Developers: They are the magicians. Other than programming, they also create architectural plans and documentations, test their own developments and help in to provide top-notch support.

Following this detailed introduction, I must admit I don’t really like to think of them as well separated departments, we are one team. We usually go beyond the strict limits of each job, for example Developers do extensive testing during development, while QA helps the Product owners in many cases, DevOps is often doing some programming and the list goes on. We have collective goals and we help each other to reach these as efficiently as possible.

 TwinRed branding workshop: Brandon from Ferras, CTO Denes Kiss, Sales Director Anna Blaivas, CEO Peter Rabenseifner

How is your IT team spread across different locations?

We operate in 3 main locations: US (Los Angeles), Hungary and Ukraine

Do you have any daily check-ins for your employees? 

We work in Scrum, which requires daily live sync. However since there is a big time zone difference I cannot keep up with all the colleagues personally on a daily basis. We separated us into 2 virtual teams, one works in the Los Angeles time zone, while the other is based on CET. We have regular weekly calls to keep us in sync, also we heavily use Slack to discuss anything that comes up. 

How do you manage and track the progress and outcomes?

Scrum methods give you a handful of practices that help you to track the progress of each task separately, and provides KPIs for tracking team performance. We use Jira for task tracking and release management.

What is your set of tools for effective remote work?

The most important is to have a simple and effective channel for communication. I always prefer face-to-face conversations if possible, because it is faster, more straightforward and more personal. So video-calls and chat are our weapon here.

Also a comfortable chair and desk is a must have.

Last but not least the key for effective remote work is having faith in your co-workers. You have to trust all your team members and let them work without being supervised closely. Honestly sometimes it is hard for me to step back and just let them work.

What are your main obstacles to running a spread IT team? 

Our biggest obstacle was the time zone difference, but luckily we managed to get over it with proper communication.

 Building and maintaining an IT infrastructure for a high-load ad-network 

Speaking of an ad network we always imply extreme load. Am I correct?

You are absolutely right. We are working with extreme amounts of traffic.

Just to give you an idea, we handle requests in the range of 7 figures per minute.

What could it be compared with?

Wikipedia is the 13th page on the top sites list of Alexa. Based on a site review it has around 300 million page views per day. If you do the math, this translates to around 200K requests per minute.

And an ad network can absolutely not face any down time, as I can imagine. Tell me firsthand what are the consequences of 1-hour down time?

Let’s just say having a full hour of down-time is extremely expensive. We planned our architecture and introduced different practices that ensure this never happens in TwinRed.

What is your monitoring system to prevent such consequences? How do you make sure everything is going smoothly?

Testing, monitoring and redundancy.

We thoroughly test each and every development, before we release it to production. This is a must have.

We monitor everything. It is a common problem that many companies only do the monitoring of the tech side, like checking CPU/memory usage, response times, etc. We make an effort to do business monitoring as well. During a release we always look for changing trends in our traffic, this allows us to spot problems earlier.

Redundancy is also a key to prevent downtimes. Hardware failures can happen anytime and our system is designed in a way that automatic failovers make these failures invisible to our customers.

How do you balance the load?

We utilize many network level load balancers that spread the traffic to many servers. As I already mentioned, the key is the redundancy. Multiple load balancers, multiple network providers, multiple everything.

Thank you, Denes, where can people find you online?

Feel free to add me on LinkedIn https://www.linkedin.com/in/deneskiss/