vitastor-cli is a command-line tool for administrative tasks like image management.
It supports the following commands:
- status
- df
- ls
- create
- snap-create
- modify
- dd
- rm
- flatten
- rm-data
- merge-data
- describe
- fix
- alloc-osd
- rm-osd
- osd-tree
- ls-osd
- modify-osd
- pg-list
- create-pool
- modify-pool
- ls-pools
- rm-pool
- ls-user
- modify-user
- rm-user
- serve
- cpubench
Global options:
--config_path FILE Path to Vitastor configuration file
--etcd_address URL Etcd connection address
--iodepth N Send N operations in parallel to each OSD when possible (default 32)
--parallel_osds M Work with M osds in parallel when possible (default 4)
--progress 1|0 Report progress (default 1)
--cas 1|0 Use CAS writes for flatten, merge, rm (default is decide automatically)
--no-color Disable colored output
--json JSON output
status
vitastor-cli status
Show cluster status.
Example output:
cluster:
etcd: 1 / 1 up, 1.8 M database size
mon: 1 up, master stump
osd: 8 / 12 up
data:
raw: 498.5 G used, 301.2 G / 799.7 G available, 399.8 G down
state: 156.6 G clean, 97.6 G misplaced
pools: 2 / 3 active
pgs: 30 active
34 active+has_misplaced
32 offline
io:
client: 0 B/s rd, 0 op/s rd, 0 B/s wr, 0 op/s wr
rebalance: 989.8 M/s, 7.9 K op/s
df
vitastor-cli df
Show pool space statistics.
Example output:
NAME SCHEME PGS TOTAL USED AVAILABLE USED% EFFICIENCY
testpool 2/1 32 100 G 34.2 G 60.7 G 39.23% 100%
size1 1/1 32 199.9 G 10 G 121.5 G 39.23% 100%
kaveri 2/1 32 0 B 10 G 0 B 100% 0%
In the example above, “kaveri” pool has “zero” efficiency because all its OSD are down.
ls
vitastor-cli ls [-l] [-p POOL] [--sort FIELD] [-r] [-n N] [<glob> ...]
List images (only matching <glob> pattern(s) if passed).
Options:
--exact Do not match glob patterns as names, select only exact name matches.
-p|--pool POOL Filter images by pool ID or name
-l|--long Also report allocated size and I/O statistics
--del Also include delete operation statistics
--sort FIELD Sort by specified field (name, size, used_size, <read|write|delete>_<iops|bps|lat|queue>)
-r|--reverse Sort in descending order
-n|--count N Only list first N items
--ids ID1,ID2 Only list images with specified full IDs
--tree Show image snapshot/clone tree
Example output:
NAME POOL SIZE USED READ IOPS QUEUE LAT WRITE IOPS QUEUE LAT FLAGS PARENT
debian9 testpool 20 G 12.3 G 0 B/s 0 0 0 us 0 B/s 0 0 0 us RO
pve/vm-100-disk-0 testpool 20 G 0 B 0 B/s 0 0 0 us 0 B/s 0 0 0 us - debian9
pve/base-101-disk-0 testpool 20 G 0 B 0 B/s 0 0 0 us 0 B/s 0 0 0 us RO debian9
pve/vm-102-disk-0 testpool 32 G 36.4 M 0 B/s 0 0 0 us 0 B/s 0 0 0 us - pve/base-101-disk-0
debian9-test testpool 20 G 36.6 M 0 B/s 0 0 0 us 0 B/s 0 0 0 us - debian9
bench testpool 10 G 10 G 0 B/s 0 0 0 us 0 B/s 0 0 0 us -
bench-kaveri kaveri 10 G 10 G 0 B/s 0 0 0 us 0 B/s 0 0 0 us -
create
vitastor-cli create -s|--size SIZE [OPTIONS] <name>
Create an image. Options:
-s|--size SIZE- New image size in bytes or with a K/M/G/T unit suffix.-p|--pool POOL- Specify pool for the new image (may be omitted if there is only 1 pool).--parent PARENT- Create a copy-on-write image clone based on PARENT (or PARENT@SNAPSHOT). If parent is not a snapshot, it must be a read-only image.--enc-key random- Generate a new random AES-256-XTS encryption key for the new image.--enc-key HEX- Set a specified AES-256-XTS key (64 bytes in hex) for the new image.--enc-key vault:ID- Use an encryption key from an external Vault secret with specified ID.--owner USERNAME- Set image owner. The default owner is the current user taken from the CN of the TLS certificate used to connect to the cluster.--owner-group NAME- Set image owner group. Users in this group get full access to the image.--reader-group NAME- Set image reader group. Users in this group get read-only access to the image.
vitastor-cli create --snapshot <snapshot> [OPTIONS] <image>
vitastor-cli snap-create [OPTIONS] <image>@<snapshot>
Create a snapshot of image <image>. May be used live if only a single writer is active.
Options:
-p|--pool POOL- Move image to pool POOL, leaving the snapshot in the old pool.--enc-key random- Change image encryption key to a new random AES-256-XTS key.--enc-key KEY- Change image encryption key to a specified key, Vault key or to an empty key. By default, the image retains its old encryption key when taking a snapshot.
See also about how to export snapshots.
modify
vitastor-cli modify <name> [--rename <new-name>] [--resize <size>] [--readonly | --readwrite] [-f|--force] [--down-ok]
Rename, resize image or change its readonly status. Images with children can’t be made read-write. If the new size is smaller than the old size, extra data will be purged. You should resize file system in the image, if present, before shrinking it.
--deleted 1|0- Set/clear ‘deleted image’ flag (set automatically during unfinished deletes).-f|--force- Proceed with shrinking or setting readwrite flag even if the image has children.--down-ok- Proceed with shrinking even if some data will be left on unavailable OSDs.--enc-key HEX- Change image encryption key (allowed only with--force).--owner USERNAME- Change image owner.--owner-group NAME- Change image owner group name.--reader-group NAME- Change image reader group name.
dd
vitastor-cli dd [iimg=<image> | if=<file>] [oimg=<image> | of=<file>] [bs=1M] \
[count=N] [seek/oseek=N] [skip/iseek=M] [iodepth=N] [status=progress] \
[conv=nocreat,noerror,nofsync,trunc,nosparse] [iflag=direct] [oflag=direct,append]
Copy data between Vitastor images, files and pipes.
Options can be specified in classic dd style (key=value) or like usual (--key value).
iimg=<image> |
Copy from Vitastor image <image> |
if=<file> |
Copy from file <file> |
oimg=<image> |
Copy to Vitastor image <image> |
of=<file> |
Copy to file <file> |
bs=1M |
Set copy block size |
count=N |
Copy only N input blocks. If N ends in B it counts bytes, not blocks |
seek/oseek=N |
Skip N output blocks. If N ends in B it counts bytes, not blocks |
skip/iseek=N |
Skip N input blocks. If N ends in B it counts bytes, not blocks |
iodepth=N |
Send N reads or writes in parallel (default 4) |
status=LEVEL |
The LEVEL of information to print to stderr: none/noxfer/progress |
size=N |
Specify size for the created output file/image (defaults to input size) |
iflag=direct |
For input files only: use direct I/O |
oflag=direct |
For output files only: use direct I/O |
oflag=append |
For files only: append to output file |
conv=nocreat |
Do not create output file/image |
conv=trunc |
Truncate output file/image |
conv=noerror |
Continue copying after errors |
conv=nofsync |
Do not call fsync before finishing (default behaviour is fsync) |
conv=nosparse |
Write all output blocks including all-zero blocks |
rm
vitastor-cli rm <from> [<to>] [--writers-stopped] [--down-ok]
vitastor-cli rm (--exact|--matching) <glob> ...
Remove layer(s) and rebase all their children accordingly.
In the first form, remove <from> or layers between <from> and its child <to>.
In the second form, remove all images with exact or pattern-matched names.
Options:
--writers-stoppedallows optimised removal in case of a single ‘slim’ read-write child and ‘fat’ removed parent: the child is merged into parent and parent is renamed to child in that case. In other cases parent layers are always merged into children.--exact- remove multiple images with names matching given glob patterns.--matching- remove multiple images with given names--down-ok- continue deletion/merging even if some data will be left on unavailable OSDs.
flatten
vitastor-cli flatten <layer>
Flatten a layer, i.e. merge data and detach it from parents.
rm-data
vitastor-cli rm-data --pool <pool> --inode <inode> [--wait-list] [--min-offset <offset>]
Remove inode data without changing metadata.
--wait-list Retrieve full objects listings before starting to remove objects.
Requires more memory, but allows to show correct removal progress.
--min-offset Purge only data starting with specified offset.
--max-offset Purge only data before specified offset.
--client_wait_up_timeout 16 Timeout for waiting until PGs are up in seconds.
merge-data
vitastor-cli merge-data <from> <to> [--target <target>]
Merge layer data without changing metadata. Merge <from>…<to> to <target>.
<to> must be a child of <from> and <target> may be one of the layers between
<from> and <to>, including <from> and <to>.
describe
vitastor-cli describe [OPTIONS]
Describe unclean object locations in the cluster. Options:
--osds <osds>
Only list objects from primary OSD(s) <osds>.
--object-state <states>
Only list objects in given state(s). State(s) may include:
degraded, misplaced, incomplete, corrupted, inconsistent.
--pool <pool name or number>
Only list objects in the given pool.
--pg <pg number>
Only list objects in the given PG of the pool.
--inode, --min-inode, --max-inode
Restrict listing to specific inode numbers.
--min-offset, --max-offset
Restrict listing to specific offsets inside inodes.
fix
vitastor-cli fix [--objects <objects>] [--bad-osds <osds>] [--part <part>] [--check no]
Fix inconsistent objects in the cluster by deleting some copies.
--objects <objects>
Objects to fix, either in plain text or JSON format. If not specified,
object list will be read from STDIN in one of the same formats.
Plain text format: 0x<inode>:0x<stripe> <any delimiter> 0x<inode>:0x<stripe> ...
JSON format: [{"inode":"0x...","stripe":"0x..."},...]
--bad-osds <osds>
Remove inconsistent copies/parts of objects from these OSDs, effectively
marking them bad and allowing Vitastor to recover objects from other copies.
--part <number>
Only remove EC part <number> (from 0 to pg_size-1), required for extreme
edge cases where one OSD has multiple parts of a EC object.
--check no
Do not recheck that requested objects are actually inconsistent,
delete requested copies/parts anyway.
alloc-osd
vitastor-cli alloc-osd
Allocate a new OSD number and reserve it by creating empty /osd/stats/<n> key.
rm-osd
vitastor-cli rm-osd [--force] [--allow-data-loss] [--dry-run] <osd_id> [osd_id...]
Remove metadata and configuration for specified OSD(s) from etcd.
Refuses to remove OSDs with data without --force and --allow-data-loss.
With --dry-run only checks if deletion is possible without data loss and
redundancy degradation.
osd-tree
vitastor-cli osd-tree [-l|--long]
Show current OSD tree, optionally with I/O statistics if -l is specified.
Example output:
TYPE NAME UP SIZE USED% TAGS WEIGHT BLOCK BITMAP IMM NOOUT
host kaveri
disk nvme0n1p1
osd 3 down 100G 0 % abc,kaveri 1 128k 4k none -
osd 4 down 100G 0 % 1 128k 4k none -
disk nvme1n1p1
osd 5 down 100G 0 % abc,kaveri 1 128k 4k none -
osd 6 down 100G 0 % 1 128k 4k none -
host stump
osd 1 up 100G 37.29 % osdone 1 128k 4k all -
osd 2 up 100G 26.8 % abc 1 128k 4k all -
osd 7 up 100G 21.84 % 1 128k 4k all -
osd 8 up 100G 21.63 % 1 128k 4k all -
osd 9 up 100G 20.69 % 1 128k 4k all -
osd 10 up 100G 21.61 % 1 128k 4k all -
osd 11 up 100G 21.53 % 1 128k 4k all -
osd 12 up 100G 22.4 % 1 128k 4k all -
ls-osd
vitastor-cli osds|ls-osd|osd-ls [-l|--long]
Show current OSDs as list, optionally with I/O statistics if -l is specified.
Example output:
OSD PARENT UP SIZE USED% TAGS WEIGHT BLOCK BITMAP IMM NOOUT
3 kaveri/nvme0n1p1 down 100G 0 % globl,kaveri 1 128k 4k none -
4 kaveri/nvme0n1p1 down 100G 0 % 1 128k 4k none -
5 kaveri/nvme1n1p1 down 100G 0 % globl,kaveri 1 128k 4k none -
6 kaveri/nvme1n1p1 down 100G 0 % 1 128k 4k none -
1 stump up 100G 37.29 % osdone 1 128k 4k all -
2 stump up 100G 26.8 % globl 1 128k 4k all -
7 stump up 100G 21.84 % 1 128k 4k all -
8 stump up 100G 21.63 % 1 128k 4k all -
9 stump up 100G 20.69 % 1 128k 4k all -
10 stump up 100G 21.61 % 1 128k 4k all -
11 stump up 100G 21.53 % 1 128k 4k all -
12 stump up 100G 22.4 % 1 128k 4k all -
modify-osd
vitastor-cli modify-osd [--tags tag1,tag2,...] [--reweight <number>] [--noout true/false] <osd_number>
Set OSD reweight, tags or noout flag. See detail description in OSD config documentation.
pg-list
vitastor-cli pg-list|pg-ls|list-pg|ls-pg|ls-pgs|pgs [OPTIONS] [state1+state2] [^state3] [...]
List PGs with any of listed state filters (^ or ! in the beginning is negation). Options:
--pool <pool name or number> Only list PGs of the given pool.
--min <min pg number> Only list PGs with number >= min.
--max <max pg number> Only list PGs with number <= max.
--osd 1,2,... Only list PGs with some data on specified OSD(s).
Examples:
vitastor-cli pg-list active+degraded
vitastor-cli pg-list ^active
create-pool
vitastor-cli create-pool|pool-create <name> (-s <pg_size>|--ec <N>+<K>) -n <pg_count> [OPTIONS]
Create a pool. Required parameters:
-s R or --pg_size R |
Number of replicas for replicated pools |
--ec N+K |
Number of data (N) and parity (K) chunks for erasure-coded pools |
-n N or --pg_count N |
PG count for the new pool (start with 10*<OSD count>/pg_size rounded to a power of 2) |
Optional parameters:
--pg_minsize <number> |
R or N+K minus number of failures to tolerate without downtime (details) |
--failure_domain host |
Failure domain: host, osd or a level from placement_levels. Default: host |
--root_node <node> |
Put pool only on child OSDs of this placement tree node |
--osd_tags <tag>[,<tag>]... |
Put pool only on OSDs tagged with all specified tags |
--block_size 128k |
Put pool only on OSDs with this data block size |
--bitmap_granularity 4k |
Put pool only on OSDs with this logical sector size |
--immediate_commit none |
Put pool only on OSDs with this or larger immediate_commit (none < small < all) |
--level_placement <rules> |
Use additional failure domain rules (example: “dc=112233”) |
--raw_placement <rules> |
Specify raw PG generation rules (details) |
--local_reads primary |
Local read policy for replicated pools: primary, nearest or random |
--primary_affinity_tags tags |
Prefer to put primary copies on OSDs with all specified tags |
--scrub_interval <time> |
Enable regular scrubbing for this pool. Format: number + unit s/m/h/d/M/y |
--used_for_app fs:<name> |
Mark pool as used for VitastorFS with metadata in image <name> |
--used_for_app s3:<name> |
Mark pool as used for S3 location with name <name> |
--pg_stripe_size <number> |
Increase object grouping stripe |
--max_osd_combinations 10000 |
Maximum number of random combinations for LP solver input |
--creator_group <group> |
User group allowed to create images in this pool |
--wait |
Wait for the new pool to come online |
-f or --force |
Do not check that cluster has enough OSDs to create the pool |
See also Pool configuration for detailed parameter descriptions.
Examples:
vitastor-cli create-pool test_x4 -s 4 -n 32
vitastor-cli create-pool test_ec42 --ec 4+2 -n 32
modify-pool
vitastor-cli modify-pool|pool-modify <id|name> [--name <new_name>] [PARAMETERS...]
Modify an existing pool. Modifiable parameters:
[-s|--pg_size <number>] [--pg_minsize <number>] [-n|--pg_count <count>]
[--failure_domain <level>] [--root_node <node>] [--osd_tags <tags>] [--used_for_app <type>:<name>]
[--max_osd_combinations <number>] [--primary_affinity_tags <tags>] [--scrub_interval <time>]
[--level_placement <rules>] [--raw_placement <rules>] [--creator_group <group>]
Non-modifiable parameters (changing them WILL lead to data loss):
[--block_size <size>] [--bitmap_granularity <size>]
[--immediate_commit <all|small|none>] [--pg_stripe_size <size>]
These, however, can still be modified with -f|–force.
See create-pool for parameter descriptions.
Examples:
vitastor-cli modify-pool pool_A --name pool_B
vitastor-cli modify-pool 2 --pg_size 4 -n 128
rm-pool
vitastor-cli rm-pool|pool-rm [--force] <id|name>
Remove a pool. Refuses to remove pools with images without --force.
ls-pools
vitastor-cli ls-pools|pool-ls|ls-pool|pools [-l] [--detail] [--sort FIELD] [-r] [-n N] [<glob> ...]
List pools (only matching
-l or --long |
Also report I/O statistics |
--detail |
Use list format (not table), show all details |
--sort FIELD |
Sort by specified field (see fields in --json output) |
-r or --reverse |
Sort in descending order |
-n or --count N |
Only list first N items |
ls-user
vitastor-cli ls-users|user-ls|ls-user|list-users [<name> ...]
List users. If names are passed, only list users with those names. User names must match CN (Common Name) of TLS client certificates used to connect to the cluster.
See Users and access rights for details on the Vitastor authorization model.
modify-user
vitastor-cli modify-user --groups <group1,group2,...> <username>
Create or update a user with the given name (matching TLS certificate CN).
Options:
--groups GROUPS- Set user’s groups (comma-separated). Groups are used together with--owner-group/--reader-groupfields of images and--creator_groupof the pool to authorize operations. Pass an empty string to clear the groups.
rm-user
vitastor-cli rm-user|remove-user|delete-user <username>
Remove a user.
serve
vitastor-cli serve
Start an HTTP server that handles vitastor-cli commands over a REST API in JSON format.
Options:
--bind_address ADDR |
Server IP address(es), separated by space. Default is 127.0.0.1. |
--port 8080 |
Server port. Default is 8080. |
--api_cert FILE |
Server TLS certificate. |
--api_pkey FILE |
Private key for --api_cert. |
--cert FILE |
Administrator or client certificate used by the server to access the cluster. |
--pkey FILE |
Private key for --cert. |
When api_cert/api_pkey are set the server operates in HTTPS mode. If client_ca
is also set (in configuration or on the command line), connecting clients are authenticated by their
TLS certificates. When use_perms is enabled, HTTPS with client
authentication is mandatory and per-user permissions are enforced. For regular clients the server only
allows operations on images that the user owns or has group access to; all other operations require
administrator privileges.
The vitastor-cli process itself must be authenticated as an admin user (its own client cert must be signed with admin_ca) to be able to serve all requests.
vitastor-cli serve exposes a full OpenAPI specification at /openapi endpoint.
Start the server and check it out for the information about available API methods.
See also Security in Vitastor for the full picture of how authentication and authorization work.
cpubench
vitastor-cli cpubench [--json]
Run CPU crypto performance tests: AES-256-GCM, AES-256-XTS and xxhash3. This benchmark does not connect to the cluster — it only measures encryption/hashing speed locally.
Specify --json to get the output as machine-readable JSON.
See Encryption performance for example results on modern and older CPUs.