About Databento

Databento makes it simpler and faster to access institutional-grade financial data. Founded by traders and engineers from among the world’s leading quantitative hedge funds, Databento’s self-service model allows users to instantly pick up live exchange feeds and terabytes of historical data – and only pay for what they use. Databento’s servers are hosted in the colocation facilities of various trading venues for low-latency and high-fidelity data capture, direct from the source.

To learn more about Databento, visit www.databento.com.

Building Private Cloud Infrastructure across Three Colocation Data Centers.

Rick Fowler and Craig Smith are Systems Engineers at Databento. They work on the Infrastructure team, whose mission is to ensure stable operations of the network, compute, and storage systems, as well as to ensure DevOps and InfraOps engineers can elastically spin up network services such (i.e., load balancers, V-Nets, NAT, DHCP, VPN, etc.)

Standing up Databento’s Private Cloud

During Databento’s early seed stage days, Luca Lin, the founding CTO, had originally put together a simple two-track infrastructure to get Databento off the ground. 

As business picked up and started growing fast, Rick Fowler was hired by Luca Lin further to redesign and stand up their version 2 infrastructure with growth, agility, and scalability in mind. They wanted to achieve cloud-like operational efficiency but run their own infrastructure in colocation data centers.

Evaluating the Existing Technology Stack

Rick first examined the existing network toolset to see if it would solve his problem. That initial setup was based on traditional enterprise data center architecture with manual CLI configuration of switches and routers, storing IP address subnet and configuration info in spreadsheets, and it lacked any automation. It was a simple solution for the early days, never meant for scalability. 

Rick is savvy with Ansible, so he initially considered buying Cisco/Juniper routers and firewalls, F5 for load balancing, and some switches and leveraging Ansible for automation. But this solution had three significant downsides for Databento. 1) These traditional networking vendors provide very little abstraction. Even with the most advanced Ansible skills, Rick must design imperative procedural automation, maintain it across multiple vendors, and apply individual logic for each network functionality. It’s very time-consuming and prone to human error. 2) DevOps and InfraOps engineers would not get any self-serviceability; every request would need to go through Rick and Craig, the Network Engineering team introducing delays and risk for configuration drift. 3) Hard to ask for vendor help when multiple products are involved — Lack of centralized support (single throat to choke).

Rick wanted, and Luca required, a general and elegant solution to let them quickly and easily manage the infrastructure and provide self-serviceability to the entire engineering team. They needed to make infrastructure robust and highly scalable – but preserve the startup-spirited agility.

Netris to the Rescue

One of Databento’s investors is Mike Dvorkin, a well-known figure in the network infrastructure space commonly referred to as “the creator of Cisco ACI,” and he suggested Rick check out Netris. 

Netris allows engineers to design, build, and operate fully automated network infrastructures providing the entire engineering team with cloud-like VPC networking abstraction via the intuitive web console, API, and IaC (Infrastructure as Code). Netris leverages SmartNIC accelerated Linux networking running over conventional servers for (Layer 3-7) VPC networking services (Routing, firewalling, load balancing, NAT, DHCP, VPN, etc.) For physical (Layer 1/2) switching, Netris can automate the lifecycle of open-networking switches (Nvidia, Dell, Edge-Core) with Linux OS (Cumulus, SONiC, Ubuntu/SwitchDev), can automate API-driven bare-metal platforms (like Equinix, PhoenixNAP, etc.), or can run anywhere over ethernet switches statically pre-configured with a carved-out range of VLANs.  

Before investing in hardware, how do you evaluate a solution of this scale (3 regions, six border routers, 30+ switches)? 

Rick has used Netris’s on-demand sandbox to learn and evaluate the platform. He reviewed pre-configured examples and walked through step-by-step “learning by doing” guide. Netris Sandbox quickly allowed him to simulate his desired architecture, create the anticipated services, and perform end-to-end testing with realistic end-user traffic flows. Unlike Cisco/Juniper/F5 sales motion, there’s no need to wait for loaner equipment – Netris Sandbox is available immediately at https://netris.io/sandbox.

For switches (Layer 1-2), Rick chose Nvidia Spectrum switches (formerly Mellanox) with 100Gbps ports and Cumulus Linux NOS because he worked with Nvidia/Mellanox switches before at Canonical (the maintainer of Ubuntu Linux) and was impressed with the hardware quality and low switching latency.

Instead of buying Cisco/Juniper routers, firewalls, and F5, Rick chose Netris SoftGate PRO, the 100Gbps version of the VPC gateway combined with the Nvidia ConnectX-6 SmartNIC card running on a Dell server with dual Intel Xeon CPUs.  

Rick says, “I could use Ansible to scratch-build in-house automation and undertake the ongoing maintenance of my automation pipeline, which would take much of my time. Also, my DevOps/InfraOps colleagues may not be able to self-service. Netris came with my desired automation already built-in and with cloud-like VPC Networking abstraction that our entire engineering team found intuitive. When I need automation specific to Databento’s business, I can leverage Netris API through config management like Ansible, Salt, or Terraform, or use Python.”

Netris subscription includes 24/7/365 support which is responsible for the end-to-end operation of the entire system, not just Netris software. Even if there is a hardware issue Netris support team can help to narrow that down and make it easier for you to work it out with your hardware supplier, or if there is an issue with one of the open-source packages, Netris will take care of that too. 

Rick says, “Working with Netris TAC over slack feels like working with colleagues on my team.”

Network Services (Layers: 3-7)

VPC Gateway HW: Dell server, 2x Intel® Xeon® Gold 5315Y Processor, 128GB RAM

VPC Gateway NIC: Nvidia ConnectX-6 SmartNIC 2x 100GbE

OS: Ubuntu Linux

Control plane: Netris-managed FRR + other open-source Linux networking software

VPN: Netris-managed Wireguard

Elastic Load Balancer: Netris

Router Data Plane: Netris

NAT & Firewall Data Plane: Netris

DHCP: Netris-managed ISC – Kea

Ethernet Switch System (Layer 2)

Spine Switches: Nvidia Spectrum MSN3700C 32x QSFP28 100GbE

Leaf Switches: Nvidia Spectrum MSN2410 48x SFP28 25GbE + 8x QSFP28 100GbE

NOS: Nvidia Cumulus Linux

Switch Fabric Management: Netris

Initial Result

Using Netris, with its built-in automation and VPC Networking abstraction, Rick Fowler quickly deployed the main region in NYC. Craig Smith, an experienced Network engineer, has joined Databento and started working with Rick. Although Craig was new to Netris, he found it intuitive and efficient. Shortly Craig and Rick launched their Boston and Chicago data centers. What commonly takes 6-9 months of development of home-grown automation – Rick and Craig achieved in about one week, which provided them with enough time to take care of everything else. Databento’s private cloud launch was a major success. Databento got the flexibility of the cloud but for its own infrastructure – maintained independence from hardware vendors and cloud providers. 

Improving Engineer productivity with Self-Service Private Cloud Infrastructure

Rick and Craig worked with traditional (Cisco/Juniper) networking at previous jobs and used Ansible for automation. Unlike the public cloud, the regular Cisco/Juniper model lacks abstraction, so only network engineers can implement network changes. With that older model, DevOps and InfraOps engineers must create a network change request and, on average, wait 3-10 days to get network engineers to implement the request. 

Netris is designed to amplify network engineers’ productivity as well as enable them to offer safe self-serviceability to the entire engineering team.

Rick and Craig have granted access to the Netris web console to their DevOps and InfraOps engineers, who are experienced in Linux and systems engineering; although they are not network engineers (they never configured any Cisco switch or router), they were able to self-service through Netris console. 

Daily network change implementation productivity has improved by over 90% from days to minutes.

Using the Rest API and IaC (Infrastructure as Code) for Automation

Netris exposes its VPC-like abstraction also over Rest API and IaC (Infrastructure as Code) to allow engineers to automate network infrastructure programmatically. Most network vendors offer little to no abstraction; hence their programmatic interfaces require engineers to handle many implementation details; even when they support Terraform, it’s not as declarative as cloud practitioners would expect; instead, it is simply a mapping of imperative commands to HCL (Hashicorp Configuration Language). Versus Netris, like VPC in the public cloud, provides a high level of abstraction that makes declarative code possible.

Declarative code is like a thermostat: you set the target temperature, and the system automatically kicks on or off to continuously meet the desired state. The thermostat abstracts away temperature sensors and control relays, so the user can think in terms of outcomes (e.g., the desired temperature) instead of wasting time on implementation details.

Most engineers working for Databento had prior automation experience, including at AWS and GCP, so both Network Engineers and DevOps/InfraOps incorporated automation into their deployment process.

Long Term Results

From a Network Engineering perspective, this solution allowed Databento to deploy and operate efficiently multi-region and scalable data center networks quickly and safely. Have holistic software to manage all aspects of the network. Stay hardware agnostic and not afraid of current or future supply chain issues. 

From DevOps and InfraOps perspectives, they benefit from the self-service capability of Databento’s network, consuming it with the operational efficiency of the public cloud. 

Summary

Here’s a summary of how Netris helped Databento to build and operate its own multi-region private cloud infrastructure network.

• Amplified Network Engineers’ productivity through automatic network operations and declarative automation API
• Derisked Databento’s growth from supply chain limiting factors by enabling hardware vendor independence.
• Boosted engineering team’s productivity through self-service VPC network model and Improved collaboration between DevOps, InfraOps, and Network Engineering teams