Having spent nearly two decades building systems that handle everything from student recommendations to financial transactions, I came to understand the best AI models are only as good as the data infrastructure that supports them. While there’s typically plenty of fanfare around large language models and other AI advances happening, the wizardry that actually happens behind the scenes goes unnoticed. It’s here, in the world of expert databases, where systems analyze petabytes of data, enable real-time decisioning, and make intelligent systems intelligent.
Back at Kameleoon, I rebuilt a 100TB data warehouse with ClickHouse and optimized query performance for our analytics platform by orders of magnitude. At RAM-IT, I architected systems that analyzed millions of cross-border transactions leveraging a combination of traditional and expert databases. These projects have taught me that choosing the right database is not strictly a technical decision, rather, it’s a strategic decision that sets the course for a system’s scalability, performance, and reliability in the future. Nowadays, as AI systems are improving, their associated databases should be improving too, but most engineers are still reaching for the same old same-old.
Why Traditional Databases Fall Short in AI-Driven Systems
The One-Size-Fits-All Problem
In the period that I was working on the cross-border transaction system in Central Asia, we were beginning with PostgreSQL for all aspects. The system had to handle thousands of transactions daily with extremely strict consistency needs across different currencies and regulatory environments. What was a straightforward relational issue became a performance disaster when we introduced real-time analytics and machine learning for detection of frauds.
Legacy databases, especially relational ones, are designed with structured, transactional data. They are wonderful at handling ACID properties and complex joins but struggle when they need to handle the unstructured data that our AI models operate on: embeddings, time-series patterns, and graph relationships. Our ML models needed to search for transaction patterns, user behaviors, and network effects simultaneously. PostgreSQL would have otherwise been able to handle each chunk independently, but when we tried concatenating them for real-time inference, query times jumped from milliseconds to seconds.
The issue wasn’t volume, it was variety. Performance bottlenecks within traditional OLTP systems become especially painful when executing ML workloads. Row-storing that is appropriate for transactional updates becomes a liability when you need to scan millions of rows over particular columns for feature engineering. We were building advanced ETL pipelines just to transfer data from our transactional environment to our analytics needs, introducing latency and complexity that ultimately annoyed the user experience.
The Hidden Costs of Database Mismatch
At RAM-IT, before we optimized our database infrastructure, infrastructure costs were spiraling out of control. We were running big PostgreSQL instances trying to accommodate analytical workloads, Redis clusters to cache highly read data, and separate systems for monitoring time-series. The cloud bill every month was astronomical, but more of an issue was the operational overhead in keeping multiple poorly integrated systems up and running.
The result was that we wasted more time futzing with databases and query optimization than putting actual code into place to improve the underlying infrastructure. This is the hidden cost of choosing the wrong database: performance degradation that affects anything from user experience to system integrity, as well as accumulating technical debt in the form of increasingly obtuse code and architecture. Our FoodTech industry was required to provide updates of real-time prices for over 150 client companies, yet our database schema was not capable of handling the required query patterns during peak hours efficiently.
ClickHouse – The Analytics Powerhouse That Changed Everything
Why ClickHouse Transformed Our Operations
ClickHouse migration in Kameleoon was transforming in ways I hadn’t anticipated. We had a 100TB data warehouse that powered A/B testing and analysis of user behavior. Traditional row-based systems were choking on the analytical queries that our data scientists wanted to run. Simple aggregations by time ranges were taking hours instead of minutes, which meant iterative analysis was nearly impossible.
Column-storages made a tangible difference. When you’re calculating conversion rates on hundreds of millions of user sessions, you just need a few columns from each record. ClickHouse’s columnar storage meant that we were processing and scanning orders of magnitude less data for our typical queries. I remember the first time we went live with ClickHouse vividly, our long complicated queries that would run several minutes on the old system now took seconds. Query response improvements were stunning: what previously took 2-3 hours now took less than 10 minutes.
The compression wins were just as impressive. Time-series data, which made up the lion’s share of our warehouse, compressed beautifully in ClickHouse’s columnar layout. We saw storage needs drop by 60-70% while simultaneously improving query performance. This led to a virtuous cycle where we could afford to keep more historical data online, enabling richer analysis and better machine learning models.
Practical Implementation Lessons
Choose ClickHouse when you have analytics-heavy applications with heavy data aggregation, especially time-series data. It is optimal for OLAP queries but not for frequent updates or multicolumn joins across many tables. Best usage is where you can structure your data with a clear time dimension and need to do aggregations on millions or billions of records.
Common pitfalls include trying to use ClickHouse as an OLTP database or using it with high-volume updates. One mistake I’ve seen too frequently is trying to shoehorn a database into a process in which it doesn’t particularly lend itself to the data structure or process. Learning query optimization can be daunting, ClickHouse has its own SQL dialect and optimization process that’s quite dissimilar from traditional databases. Provide a lot of testing and tuning in practice.
ScyllaDB – High-Performance NoSQL That Actually Delivers
Beyond Traditional NoSQL Limitations
Where ultra-low latency access to data was necessary for the trading platform, ScyllaDB’s C++ architecture was the game-changer. The JVM garbage collection stop-the-world pauses in Cassandra were causing variable latency spikes that were unacceptable for transactions in finance, even though Cassandra could maintain throughput. ScyllaDB’s architecture eradicated these pauses without compromising Cassandra compatibility, allowing us to upgrade without rewrites of application code.
In RAM-IT, we used ScyllaDB to handle some of our heaviest workloads. We were handling massive amounts of transactions in real-time, and ScyllaDB’s support for high-volume writes without suffering performance-wise was crucial. Standard NoSQL databases aren’t actually horizontally scalable; that is, you can’t add extra nodes to handle growing traffic without suffering performance. The performance characteristics are more important than benchmarks suggest, stable latency will often be more important than max throughput.
Operational Considerations
One of the things that I found challenging with ScyllaDB was capacity planning. ScyllaDB is distinct from some of the older databases in that it requires more planning ahead on scaling and resource allocation. With per-core architecture, you have to match your hardware configuration to your actual concurrency needs. We discovered the hard way when our initial deployment was under-provisioned in CPU but over-provisioned in memory.
Migration from another NoSQL platform can be surprisingly effortless if you do it well. Cassandra compatibility means less application modification, but you will need to recreate your operational patterns. ScyllaDB monitoring and management APIs are different, and the performance model will likely expose inefficiencies in your existing query patterns masked by less stringent behavior in other platforms.
The Vector Database Revolution – More Than Just Embeddings
Transforming Search and Recommendations
With the evolution of AI systems, it has now become all the more essential to utilize specialized databases that are equipped to handle high-dimensional data. Vector databases, which preserve and handle embeddings of modern machine learning models, are revolutionizing how we think about search, recommendation engines, and semantic analysis. In Profilum, where I had implemented a recommendation system for 50,000 students, we used vector databases to optimize the performance of our EdTech platform.
Traditional search systems struggled with the semantic queries our recommendation engine required. Students were not typing in courses by keyword only, they required recommendations upon patterns of learning, career goals, and skill interdependencies that could not be modeled with traditional relational queries. We turned to vector databases that had the ability to perform fast similarity searches across student profiles and course content. The results were immediate and phenomenal. Our users noticed more precise recommendations, and there was a notable increase in engagement and completion rates.
Integration Challenges and Solutions
The most essential factor in vector databases is how easily they can be integrated with machine learning pipelines. The trade-off is between performance and complexity to have real-time querying features. When choosing a vector database, don’t just consider the size of data but also how it scales and if it can handle the latency requirements of your application.
Hybrid configurations combining vector and mainstream databases are currently in vogue, combining the best of both. In our solution, we archived transactional data in PostgreSQL and used vector databases for similarity queries, creating a system that could both effectively support exact lookups and semantic queries.
Time-Series Databases – The Backbone of Real-Time Systems
Lessons from IoT and Monitoring
At SmartAirkey, we handled massive amounts of sensor data from electronic locks across thousands of installations. Each lock sent constant streams of battery level, usage patterns, connectivity status, and performance. Initially, we tried to store that in PostgreSQL with time-based partitioning, but the write volume overwhelmed the system quite quickly.
General-purpose databases simply aren’t optimized to handle time-series data. The problem isn’t size, it’s the write-intensive workload combined with the need for time-series analytics. Databases are optimized for mixed read-write workloads, while IoT applications are mostly writes with periodic analytical queries. We chose InfluxDB as it was able to handle time-series at scale, and it defeated general-purpose databases when ingesting and querying our sensor data.
Practical Architecture Patterns
If you’re working with a system that generates time-series data in real-time and it has to originate from sensors, devices, or logs, then a time-series database is a blessing. The key challenge is getting your database to leverage its power to the maximum, e.g., through automated retention policies, compression strategy, and monitoring tool integrations. We realized how we needed to design our schema with multiple levels of resolution, high-resolution data for short near-term intervals and aggregate aggregates for longer-term examination.
Graph Databases – Unlocking Relationship Intelligence
Beyond Social Networks
In e-commerce and fintech, where object relationships are dense with information, graph databases are becoming a must. In RAM-IT, we experimented with graph databases so we would get to know more about users’ habits and be able to personalize our cross-border trading platform better. Traditional relational approaches cannot affordably handle multi-hop relationships, queries like querying all three degrees of accounts linked to a suspected transaction become expensive joins that do not scale well.
Graph databases are best suited for handling linked data and traversal queries that are critical for fraud detection and risk analysis. That said, they are not always warranted. Take careful consideration before adopting a graph database to determine whether relationships in data warrant the added complexity and performance issues. In certain situations, standard databases with adequate indexing and denormalization approaches will suffice.
Building a Decision Framework
Strategic Database Selection
The selection of the proper database isn’t purely a technical decision, it’s strategic that hinges on your access patterns to data, not data size. Heavy transactional workloads call for different solutions than queries for analysis, even for access to the same base data. Time-sensitive requirements often gain priority over others; a system with sub-millisecond response needs has different constraints from batch processing scenarios.
Performance vs. complexity tradeoffs become necessary in developing systems that need to evolve. Specialized databases can provide better performance for specific use cases, but they introduce complexity into operations and require expert skills. The difficulty lies in balancing the investment in complexity with the need for performance and team capability and considering scalability, cost, and operational overhead.
The Future of Data Infrastructure
As AI continues to get better, so will the specialized databases below. Vector databases are turning into first-class citizens of data stacks and not special-purpose tools for embeddings. Multi-modal databases that natively support text, images, and structured data together are in the works, and the traditional separation between transactional, analytical, and search systems is breaking down.
The future will demand more diverse, flexible, and sophisticated systems. Building teams that are capable of handling diverse data infrastructure requires technical sophistication and operational diversity. The thriving organizations will be the ones that can balance innovation with stability of operations while cultivating genuine expertise in multiple database technologies. From real-time decisions to focused recommendations, the application of data infrastructure will only grow in importance, and we must be prepared to experiment, adapt, and shift our mindset towards database architecture.
About the author
Dmytro Saiankin is a software architect with over a decade of experience in system development and design, specializing in areas such as e-commerce. As CTO at RAM-IT, he successfully launched a B2B marketplace that generated $2 million in revenue in 2021 and played a pivotal role in architecting a cross-border trading system in Central Asia. Dmytro also led teams of dozens of members across multiple time zones, focusing on process optimization and driving innovation across various industries.
