Lianja/KVS (Key-Value Store)

Overview

Lianja includes a Key-Value Store (KVS) to support data streaming and IoT applications.

A key-value database is a type of nonrelational database that uses a simple key-value method to store data. A key-value database stores data as a collection of key-value pairs in which a key serves as a unique identifier. Both keys and values can be anything, ranging from simple objects to complex compound objects.

KVS is implemented as a collection of functions built into Lianja.

It is a high performance object store for storing keys with values that may be objects that are automatically JSON encoded and decoded.

The KVS itself is a B+ tree index which contains keys and values inside the KVS file.

It is a 64-bit file store so it can contain a huge number of indexed Key/Value pairs. Deleted space is reused.

It handles automatic locking supporting concurrent high speed CRUD operations.

KVS can be used for data streaming applications as it is very high performance and, with the kvs_removeLast() function, can be used to implement a stack of data, together with kvs_removeFirst() which can be used to implement a queue of data.

It is planned to enhance KVS later with sharding/partitioning to facilitate high performance clustered indexing.

These KVS functions are available in both LianjaScript and Python as of Lianja 7.0.

LianjaScript usage

// create a new KVS with a key of 10 characters
kvsid = kvs_create("mykvs", 10)
kvs_set(kvsid, 1, "hello world")
kvs_close(kvsid)
kvsid = kvs_open("mykvs", True)
? kvs_get(kvsid, 1)
// add JSON encoded objects
kvs_add(kvsid, 2, json_decode("{'name':'barry', 'lastname':'mavin'}")
obj = kvs_get(kvsid, 2)
? obj
? obj["name"]
? obj.name

Python usage

# create a new KVS with a key of 10 characters
kvsid = kvs_create("mykvs", 10)
kvs_set(kvsid, 1, "hello world")
kvs_close(kvsid)
kvsid = kvs_open("mykvs", True)
print( kvs_get(kvsid, 1) )
# Add a JSON encoded object
kvs_add(kvsid, 2,  {'name':'barry', 'lastname':'mavin'})
obj = kvs_get(kvsid, 2)
# You only need to import json if you are working in the Python console.
import json
print( json.dumps(obj) )
print( obj["name"] )

KVS Built-in Functions

kvs_add()

Add key/value for the specified key. If the value is an object or array it is automatically JSON encoded.

ok = kvs_add(kvsid, cKey | nKey, objectName | arrayName | cExpr)

In LianjaScript, you can load very long strings or the contents of a file like this:

kvs_add(k, key, "@filetostr:filename.txt")

This is not required in Python as there is no string size limit.

However, to load the contents of a text file in Python, you can use the filetostr() function which is built-in in Lianja/Python.

kvs_add(k, key, filetostr(“filename.txt”) )

kvs_clear()

Remove all key/value pairs from the store.

kvs_clear(kvsid)

kvs_close()

Close the KVS.

kvs_close(kvsid)

kvs_closeall()

Closes all KVS files that are open and invalidates the ksvids. Returns the number of KVS files closed.

nCount = kvs_closeall()

kvs_contains()

Returns true if key contained in the store.

logical = kvs_contains(kvsid,  cKey | nKey)

kvs_count()

Returns count of number of key/value pairs in the store.

numeric = kvs_count(kvsid)

kvs_create()

Create a new KVS with keys of nKeylen length.

kvsid = kvs_create(cFilename[.kvs], nKeylen [, bEnableLogging])

Note that If there is an active database open, then the KVS file will be created in the database container unless it has a fullpath specified with directory separator characters.

kvs_generate()

Generate nCount new key/value pairs. From v7.0.

numeric = kvs_generate(kvsid, nCount)

kvs_get()

Fetch value for the specified key. If the value is an object then cMembername specifies a member (property) to return.

value = kvs_get(kvsid, cKey | nKey [,cDefault [,cMembername] ])

If the value is a JSON encoded object then it will be decoded and an object will be returned. In Python a "dict" is returned so you need to access the members like this:

obj = kvs_get(k, key)
print( obj["name"] )

In LianjaScript you can use that same syntax or dotted object.member notation.

In LianjaScript if the "value" is longer than 64k it will be written into a temporary file which you can delete after JSON decoding it.

value = kvs_get(k, key) // value is @filetostr:filename.txt
if startsWith(value, "@filetostr:")
  filename = value.substr(11)
  object = json_decode_file(filename)
  delete file &filename
endif

This is not required in Python as there is no string size limit.

kvs_getAll()

Fetch all keys/values for the specified key. CKeyPattern may end in * to match wildcard keys. Returns a dynarray of arrays.

object = kvs_getAll(kvsid [, cKey | cKeyPattern ])

kvs_getRange()

Fetch a range of key/value pairs (offset starts at 1). Returns an array of two elements, key and value.

array = kvs_getRange(kvsid, nOffset, nCount)

kvs_info()

Displays information about the specified KVS file.

kvs_info(kvsid)

kvs_keys()

Fetch all keys for the specified key. CKeyPattern may end in * to match wildcard keys.

array = kvs_keys(kvsid, [, cKey | cKeyPattern ])

kvs_isvalid()

Check validitity of the given ksvid.

numeric = kvs_isvalid(kvsid)

kvs_list()

List the key/values in the store to the console (useful during development for debugging).

kvs_list(kvsid)

kvs_listLog()

List binary transaction log associated with the store to the console (useful during development for debugging).

kvs_listLog(kvsid)

kvs_loadFromFile()

Load key/value pairs from the specified file. From v7.0.

numeric = kvs_loadFromFile(kvsid, "cFilename.txt")

The contents of the text file may be formatted as:

key=value
key2=value

or

value
value2

The values may be JSON encoded objects:

{'name':'barry', 'lastname':'mavin'}
{'name':'yvonne', 'lastname':'milne'}

kvs_max()

Returns max key.

kvs_max(kvsid)

kvs_min()

Returns min key.

kvs_min(kvsid)

kvs_next()

Position on next key/value pair returns an array of two elements: key and value.

array = kvs_next(kvsid)

kvs_open()

Open an existing KVS. lShared is supported from v7.0, allowing the KVS to be opened shared or exclusive.

kvsid = kvs_open(cFilename.kvs [,lShared])

Note that If there is an active database open, then the KVS file will be opened in the database container unless it has a fullpath specified with directory separator characters.

kvs_rebuild()

Rebuild the KVS file from the binary transaction log.

kvs_rebuild(cFilename)

kvs_remove()

Remove the specified key.

kvs_remove(kvsid, cKey | nKey)

kvs_removeFirst()

Use for queue. Returns an array of two elements: key and value.

array = kvs_removeFirst(kvsid)

kvs_removeLast()

Use for stack. Returns an array of two elements: key and value.

array = kvs_removeLast(kvsid)

kvs_removeRange()

Fetch a range of key/value pairs (offset starts at 1). From v7.0.

numeric = kvs_removeRange(kvsid, nOffset, nCount)

kvs_rewind()

Position on first key/value pair. Returns an array of two elements: key and value.

array = kvs_rewind(kvsid)

kvs_saveToFile()

Save key/value pairs to the specified file. From v7.0.

numeric = kvs_saveToFile(kvsid, "filename.txt" [,lUpgradeFrom63] )

kvs_set()

Add/update key/value for the specified key. If the value is an object or array, it is automatically JSON encoded.

ok = kvs_set(kvsid, cKey | nKey, objectName | arrayName | cExpr)

In LianjaScript, you can load very long strings or the contents of a file like this:

kvs_set(k, key, "@filetostr:filename.txt")

This is not required in Python as there is no string size limit.

kvs_update()

Update value for the specified key. If the value is an object or array, it is automatically JSON encoded.

ok = kvs_update(kvsid, cKey | nKey, objectName | arrayName | cExpr)

In LianjaScript, you can load very long strings or the contents of a file like this:

kvs_update(k, key, "@filetostr:filename.txt")

This is not required in Python as there is no string size limit.

However, to load the contents of a text file in Python, you can use the filetostr() function which is built-into Lianja/Python.

kvs_update(k, key, filetostr(“filename.txt”) )

kvs_values()

Fetch all values for the specified key. CKeyPattern may end in * to match wildcard keys.

array = kvs_values(kvsid, [, cKey | cKeyPattern ])