When migrating to Splunk Cloud, many teams overlook how lookup storage can impact performance and storage quotas. CSV-based lookups have been the default for many years, but KV Store collections provide a more scalable and flexible alternative. This post will explore the key differences between the two, highlight when and why to use one over the other, and offer step-by-step guidance on how to migrate CSV lookups to KV Store collections.
What Are CSV Lookups?
CSV lookups are simple, static, flat files that map fields to values. These files live in the app’s lookup directory and are replicated as part of the knowledge bundle. They are easy to use and widely supported by SPL commands like inputlookup, lookup, and outputlookup.
Although they are easy to manage, CSV lookups present limitations. They do not support data typing, row-level identity, or concurrent writes. Large files can also cause performance degradation, as the whole file must be transferred to the indexers at search time. When stored with the app bundle, they contribute to disk space usage and may cause deploy issues.
What are KV Store Lookups?
KV Store lookups are backed by MongoDB and store records in a collection. Each record has a unique _key and supports typed fields. They are defined in collections.conf and accessed through SPL similarly to CSV lookups.
The key advantage of KV Store is its dynamic nature. Data can be updated without overwriting the entire collection. They scale to millions of records and are stored separately from the app bundle. In Splunk Cloud, this storage separation helps avoid bundle size limits and improves deploy performance.
CSV vs. KV Store: A Feature Comparison
Features | CSV Lookup | KV Store Collection |
|---|---|---|
Storage Location | App bundle (filesystem) | MongoDB-backed service |
Scalability | Limited (~10MB) | High (500MB+ per collection) |
Field Typing | No | Yes |
Unique Record ID | No | _key field per record |
REST API Support | No | Yes |
Concurrent Writes | No | Yes |
App Bundle Impact | App Bundle Impact | Stored outside bundle |
When Should You Use KV Store?
Choosing KV Store is ideal when your lookups are large, dynamic, or frequently updated. If a lookup requires field types or row-level updates, KV Store is a better fit. Teams with multiple users or automated updates will benefit from concurrent write capabilities.
Using KV Store in Splunk Cloud is especially valuable. App bundles must remain under strict size limits. Offloading lookup data to KV Store preserves space and reduces bundle replication delays.
Preparing for Migration Before Moving to Cloud
The ideal time to convert CSV lookups to KV Store is before migrating to Splunk Cloud. This reduces app size and helps validate that existing dashboards and searches work with the new definitions.
To migrate, start by defining the collection in collections.conf. Next, update transforms.conf to define the KV Store lookup. Then use SPL to transfer data.
| inputlookup my_lookup.csv
| outputlookup my_kv_lookup
Repeat this for all relevant lookups. Confirm that dashboards are updated to use the new lookup definitions. Finally, remove the original CSVs from the app bundle.
Migrating After You are Already in Splunk Cloud
Even if migration is complete, you can still move CSV lookups to KV Store.
First, add collections.conf and transforms.conf to your custom app. Deploy the app via the UI or ACS. Afterward, run SPL in the search UI to transfer data. This operation can be performed without CLI access.
| inputlookup original.csv
| outputlookup kv_target
Once copied, update your dashboards and searches. You can retain the original lookup name by redefining it in transforms.conf to point to the KV Store collection.
Conclusion
CSV lookups work well for small, static data. However, KV Store offers better performance, flexibility, and storage efficiency. By migrating to KV Store, especially in Splunk Cloud, you avoid bundle size limitations and future-proof your lookup strategy.
If you haven’t yet migrated, include this conversion in your pre-cloud checklist. If you’re already on Splunk Cloud, take time to clean up your lookups now. You’ll gain a leaner, more scalable, and maintainable solution that aligns with cloud best practices.
To access more Splunk searches, check out Atlas Search Library, which is part of the Atlas Platform. Specifically, Atlas Search Library offers a curated list of optimized searches. These searches empower Splunk users without requiring SPL knowledge. Furthermore, you can create, customize, and maintain your own search library. By doing so, you ensure your users get the most from using Splunk.
