Difference between revisions of "Key-Value Store"

Latest revision as of 21:49, 30 May 2024

Overview

Lianja includes a Key-Value Store (KVS) to support data streaming and IoT applications.

A key-value database is a type of nonrelational database that uses a simple key-value method to store data. A key-value database stores data as a collection of key-value pairs in which a key serves as a unique identifier. Both keys and values can be anything, ranging from simple objects to complex compound objects.

KVS is implemented as a collection of functions built into Lianja or alternatively it can be used with a standard OData/REST API (see below).

It is a high performance object store for storing keys with values that may be objects that are automatically JSON encoded and decoded.

The KVS itself is a B+ tree index which contains keys and values inside the KVS file.

It is a 64-bit file store so it can contain a huge number of indexed Key/Value pairs. Deleted space is reused.

It handles automatic locking supporting concurrent high speed CRUD operations.

KVS can be used for data streaming applications as it is very high performance and, with the kvs_removeLast() function, can be used to implement a stack of data, together with kvs_removeFirst() which can be used to implement a queue of data.

It is planned to enhance KVS later with sharding/partitioning to facilitate high performance clustered indexing.

These KVS functions are available in both LianjaScript and Python as of Lianja 7.0.

LianjaScript usage

// create a new KVS with a key of 10 characters
kvsid = kvs_create("mykvs", 10)
kvs_set(kvsid, 1, "hello world")
kvs_close(kvsid)
kvsid = kvs_open("mykvs", True)
? kvs_get(kvsid, 1)
// add JSON encoded objects
kvs_add(kvsid, 2, json_decode("{'name':'barry', 'lastname':'mavin'}")
obj = kvs_get(kvsid, 2)
? obj
? obj["name"]
? obj.name

Python usage

# create a new KVS with a key of 10 characters
kvsid = kvs_create("mykvs", 10)
kvs_set(kvsid, 1, "hello world")
kvs_close(kvsid)
kvsid = kvs_open("mykvs", True)
print( kvs_get(kvsid, 1) )
# Add a JSON encoded object
kvs_add(kvsid, 2,  {'name':'barry', 'lastname':'mavin'})
obj = kvs_get(kvsid, 2)
# You only need to import json if you are working in the Python console.
import json
print( json.dumps(obj) )
print( obj["name"] )

Using Lianja/KVS in the Cloud

LIanja/KVS is available in the Cloud Server (since Lianja 9.8).

You can also use KVS with Node.js and the Python.

You work with KVS in the cloud using odata format URIs.

You can perform a range of CRUD (Create, Read, Update and Delete) operations.

Creating a key-value pair

OData_Create("/southwind/example.kvs?$key=0", "{'name':'barry', 'lastname': 'mavin'}")

notice that if a $key is not specified the next available key will be generated for you. This is the equivalent of autoinc in a SQL database. This is useful for streaming data.

You can store and retrieve any format of key-value pair. If you are storing binary data base64 encode it and base64 decode it on retrieval.

OData_Create("/southwind/example.kvs", "This is raw textual data")
OData_Read("/southwind/example.kvs?$format=raw&limit=2,1")

Reading Values

use OData_Read("/southwind/example.kvs?$key=xxx")

for queues:

OData_Read("/southwind/example.kvs?$op=removefirst")

for stacks:

OData_Read("/southwind/example.kvs?$op=removelast")

Handling pagination with $key and $limit:

OData_Read("/southwind/example.kvs?$key=region2$limit=0,10")  
OData_Read("/southwind/example.kvs?$key=region2$top=10")

use $size to include the count of key-value items in the KVS file.

OData_Read("/southwind/example.kvs?$size&$limit=0,10")  
OData_Read("/southwind/example.kvs?$size&$top=10")

use $format=raw to retrieve the raw value for a specified $key in the KVS file. This enables the storage and retrieval of large python data structures such as lists and tuples that can be deserialized after retrieval.

Update a key-value pair

OData_Update("/southwind/example.kvs?$key=3", "{'name':'barry3', 'lastname': 'mavin3'}")

Delete a key-value pair

OData_Delete("/southwind/example.kvs?$key=5")

Lianja/KVS Built-in Functions

kvs_add()

Add key/value for the specified key. If the value is an object or array it is automatically JSON encoded.

ok = kvs_add(kvsid, cKey | nKey, objectName | arrayName | cExpr)

In LianjaScript, you can load very long strings or the contents of a file like this:

kvs_add(k, key, "@filetostr:filename.txt")

This is not required in Python as there is no string size limit.

However, to load the contents of a text file in Python, you can use the filetostr() function which is built-in in Lianja/Python.

kvs_add(k, key, filetostr("filename.txt") )

kvs_clear()

Remove all key/value pairs from the store.

kvs_clear(kvsid)

kvs_close()

Close the KVS.

kvs_close(kvsid)

kvs_closeall()

Closes all KVS files that are open and invalidates the ksvids. Returns the number of KVS files closed.

nCount = kvs_closeall()

kvs_contains()

Returns true if key contained in the store.

logical = kvs_contains(kvsid,  cKey | nKey)

kvs_count()

Returns count of number of key/value pairs in the store.

numeric = kvs_count(kvsid)

kvs_create()

Create a new KVS with keys of nKeylen length.

kvsid = kvs_create(cFilename[.kvs], nKeylen [, bEnableLogging])

Note that If there is an active database open, then the KVS file will be created in the database container unless it has a fullpath specified with directory separator characters.

From Lianja v9.8, the cFilename.kvs filename definition can include a database! prefix to create the KVS in the specified database.

kvs_generate()

Generate nCount new key/value pairs. From v7.0.

numeric = kvs_generate(kvsid, nCount)

kvs_get()

Fetch value for the specified key. If the value is an object then cMembername specifies a member (property) to return.

value = kvs_get(kvsid, cKey | nKey [,cDefault [,cMembername] ])

If the value is a JSON encoded object then it will be decoded and an object will be returned. In Python a "dict" is returned so you need to access the members like this:

obj = kvs_get(k, key)
print( obj["name"] )

In LianjaScript you can use that same syntax or dotted object.member notation.

In LianjaScript if the "value" is longer than 64k it will be written into a temporary file which you can delete after JSON decoding it.

value = kvs_get(k, key) // value is @filetostr:filename.txt
if startsWith(value, "@filetostr:")
  filename = value.substr(11)
  object = json_decode_file(filename)
  delete file &filename
endif

This is not required in Python as there is no string size limit.

kvs_getAll()

Fetch all keys/values for the specified key. CKeyPattern may end in * to match wildcard keys. Returns a dynarray of arrays.

object = kvs_getAll(kvsid [, cKey | cKeyPattern ])

kvs_getRange()

Fetch a range of key/value pairs (offset starts at 1). Returns an array of two elements, key and value.

array = kvs_getRange(kvsid, nOffset, nCount)

kvs_info()

Displays information about the specified KVS file.

kvs_info(kvsid)

kvs_isvalid()

Check validity of the given ksvid.

numeric = kvs_isvalid(kvsid)

kvs_keys()

Fetch all keys for the specified key. CKeyPattern may end in * to match wildcard keys.

array = kvs_keys(kvsid, [, cKey | cKeyPattern ])

kvs_list()

List the key/values in the store to the console (useful during development for debugging).

kvs_list(kvsid)

kvs_listLog()

List binary transaction log associated with the store to the console (useful during development for debugging).

kvs_listLog(kvsid)

kvs_loadFromFile()

Load key/value pairs from the specified file. From v7.0.

numeric = kvs_loadFromFile(kvsid, "cFilename.txt")

The contents of the text file may be formatted as:

key=value
key2=value

or

value
value2

The values may be JSON encoded objects:

{'name':'barry', 'lastname':'mavin'}
{'name':'yvonne', 'lastname':'milne'}

kvs_max()

Returns max key.

kvs_max(kvsid)

kvs_min()

Returns min key.

kvs_min(kvsid)

kvs_next()

Position on next key/value pair returns an array of two elements: key and value.

array = kvs_next(kvsid)

kvs_open()

Open an existing KVS. lShared is supported from v7.0, allowing the KVS to be opened shared or exclusive.

kvsid = kvs_open(cFilename.kvs [,lShared])

Note that If there is an active database open, then the KVS file will be opened in the database container unless it has a fullpath specified with directory separator characters.

From Lianja v9.8, the cFilename.kvs filename definition can include a database! prefix to open the KVS in the specified database.

kvs_rebuild()

Rebuild the KVS file from the binary transaction log.

kvs_rebuild(cFilename)

kvs_remove()

Remove the specified key.

kvs_remove(kvsid, cKey | nKey)

kvs_removeFirst()

Use for queue. Returns an array of two elements: key and value.

array = kvs_removeFirst(kvsid)

kvs_removeLast()

Use for stack. Returns an array of two elements: key and value.

array = kvs_removeLast(kvsid)

kvs_removeRange()

Remove a range of key/value pairs (offset starts at 1). From v7.0.

numeric = kvs_removeRange(kvsid, nOffset, nCount)

kvs_rewind()

Position on first key/value pair. Returns an array of two elements: key and value.

array = kvs_rewind(kvsid)

kvs_saveToFile()

Save key/value pairs to the specified file. From v7.0.

numeric = kvs_saveToFile(kvsid, "filename.txt" [,lUpgradeFrom63] )

kvs_set()

Add/update key/value for the specified key. If the value is an object or array, it is automatically JSON encoded.

ok = kvs_set(kvsid, cKey | nKey, objectName | arrayName | cExpr)

In LianjaScript, you can load very long strings or the contents of a file like this:

kvs_set(k, key, "@filetostr:filename.txt")

This is not required in Python as there is no string size limit.

kvs_update()

Update value for the specified key. If the value is an object or array, it is automatically JSON encoded.

ok = kvs_update(kvsid, cKey | nKey, objectName | arrayName | cExpr)

In LianjaScript, you can load very long strings or the contents of a file like this:

kvs_update(k, key, "@filetostr:filename.txt")

This is not required in Python as there is no string size limit.

However, to load the contents of a text file in Python, you can use the filetostr() function which is built-into Lianja/Python.

kvs_update(k, key, filetostr("filename.txt") )

kvs_values()

Fetch all values for the specified key. CKeyPattern may end in * to match wildcard keys.

array = kvs_values(kvsid, [, cKey | cKeyPattern ])