a toolips-based data-base server

ChiDB is a unique data-base server designed around the AlgebraFrames concept and the .ff file format. Schema is laid using directories and filenames and data is live-read into memory. This is currently in a state of relative infancy, but is primarily being developed for my own use cases and to demonstrate the various server-building capabilities of Toolips.

• get started
  • adding chidb
  • documentation
• chidb setup
  • schema
    • feature files
    • readable data-types
    • editing schema
• querying
  • clients
  • command list
  • query examples
• creating chidb clients
  • existing clients
  • chidb headers
    • header RFC

get started

adding

documentation

setup

In order to use ChiDB, we first need julia. With Julia installed, the package may be added with Pkg:

using Pkg; Pkg.add(url = "https://github.com/ChifiSource/ChiDB.jl")
using ChiDB

To setup a ChiDB server directory, run ChiDB.start and provide an empty directory as path. This will create a new folder called db, which will contain the data-bases core information.

start(path::String, ip::IP4 = "127.0.0.1":8005)

Our admin login will also be printed here; by querying with this new admin login we may create new users.

schema

ChiDB schema is created using two different techniques:

• querying
• or by creating a filesystem of .ff and .ref files.

ChiDB's internal data is primarily stored in the .ff (feature file) format. There are no sub-tables, only reference columns. Both references and .ff feature files are represented by files, and the tables they reside in are represented by directories. In order to create schema, we would simply add new folders with new .ff files for each column to our new data-base directory. Consider the following sample directory structure:

• project directory
  • /db
  • /table1
    • /col1.ff
    • /col2.ff
    • /table2_col1.ref
  • /table2
    • /col1.ff

feature files

Each .ff file's first line will be a readable data-type. For example, col1.ff from above could be

Integer

readable data-types

querying

clients

commands

Commands are issued to the server using

• () indicates an optional argument.
• (table)/column indicates the ability to provide the column if a table is selected, or provide a column and table in the table/column format. For example:

write!(sock, "$(curr_header)vnewt/name|!|1|!|frank\n")

character	name	description	arguments
l	list	lists the columns within a table, and their types, or lists all tables when provided with no argument	(table)
s	select	Selects a table.	table
t	create	creates a new table	tablename
		get-store commands
g	get	gets values using vertical indexing	(table)/column (range)
r	getrow	Gets a full row of data	(table)/column rown
i	index	Gets the index where a certain value occurs in a given table.	(table)/column value
a	store	Stores values, separated by `!;`, into a given table. Will return an argument error if the incorrect shape is provided.	(table) value!;value2
v	set	Sets a singular value in a table.	(table)/column row value
w	setrow	Sets the values of an entire row on a table	(table) row value1!;value2
		column management
j	join	Adds a new column to a frame, creates a reference column when used with a column path from another table.	(table) (reftable)/colname (Type)
k	type	Attempts to cast a given type to a provided column.	(table)/colname Type
e	rename	Renames a given column or table	(table) table_or_colname
		deleters
d	deleteat	Deletes a row from a given table.	(table) row
z	delete	Deletes a table	table
		built-in operations
p	compare	Checks if the provided value is the same as the stored data.	(table)/column rown value
n	in	Checks if the provided value is within the column.	(table)/column value
		server
U	users	Lists current users.
C	newuser	Creates a new user. Will return the new name, password, and dbkey.	user (pwd)
K	setuser	Sets any users login -- must be admin to perform.	user, name (pwd)
D	rmuser	Removes a user by name. Must be admin to perform.	username
L	logout	Disconnects from the server.

query examples

---

creating clients

existing clients

headers

You will likely want an API of some sort to query a ChiDB servers. Every query, including your initial query, will be sent with a two-byte header. This header includes three fields: the opcode (4 bits), the transaction id (4 bits) (composing the first byte) and the second byte (8 bits) is dedicated to the command character -- a single-character reference that requests a query command.

• The opcode returns a success code dependent on the status of the last query. See opcodes for a full list of opcodes.
• The transaction id will be the ID of the transaction that is just issued. This needs to be sent back to the server on each query, and will need to be wrapped into each header we send to the server.

Request header
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16
|    opcode   | transac ID    |          command byte           |

Response header
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|    opcode   | transac ID    |

So, from the API's perspective both the opcode and transaction ID are sent back to the server, meaning we can just send back the first character we get as our response as the header. The second char would then be our selected query command, and from there our arguments are provided directly and separated by |!|. For the initial connection the first character would be nothing, and for S we provide spaces as separators.

using Toolips

sock = Toolips.connect("127.0.0.1":2025)

write!(sock, "nS dbkey admin adminpwd\n")

resp = String(readavailable(sock))
# (opcode)
eightbit_header = UInt8(resp[1])
header = bitstring(eightbit_header)

if header[1:4] == "0001"
    println("query accepted!")
else
    throw("not verified, the query will not work")
end
# list tables:
write!(sock, string(eightbit_header) * "l\n")
resp = String(readavailable(sock))
println(resp)

opcodes

code	status	name	has output
0001	OK	query accept	false
0011	OK	user created	false
0101	OK	password set	false
1000	ERROR	bad packet	false
1100	ERROR	login denied (connection closed)	false
1001	ERROR	bad dbkey (connection closed)	false
1110	ERROR	command error	true
1010	ERROR	argument error	true
1111	ERROR	bad transaction (connection closed)	false