Skip to content

Operations overview ​

This page is for the operators who run a Felis host. It describes what runs on the host, where the data lives, which tasks need doing regularly, and which host operations are available in the panel and which stay on the command line. For installation, see Installation; for host sizing, see Requirements.

Host components ​

The default installation is a single node. The installer installs k3s on the host and runs the control plane, the database, the image registry and the game servers inside it; the game proxy runs directly on the host as a systemd service. For multi-node deployments, see Multi-node and high availability; for the overall design, see Architecture.

ComponentRuns asPurpose
k3sk3s.serviceThe Kubernetes cluster that hosts the containerized components below
felis-apiDeployment felis-api in the felis namespaceThe web panel, the public API, and the internal API the game plugins use
felis-operatorDeployment felis-operatorStarts, stops and watches game servers (MinecraftServer resources)
PostgreSQLDeployment felis-postgresThe control-plane database; its data is in /var/lib/felis/postgres on the host
Image registryDeployment registryThe in-cluster registry for platform images, game images and user-built images
Login space and lobbyMinecraftServer login (Limbo) and lobby (Paper) in the minecraft namespaceWhere players arrive after joining
Game serversMinecraftServer resources in the minecraft namespaceThe servers users create
Build jobsJobs in the felis-build namespaceBuild server images from submissions
Game proxy (Velocity)felis-velocity.serviceAccepts player connections on the game port and forwards them to the servers
Host operations servicefelis-node-control.serviceRuns the host tasks started from the panel, as root
Cloudflare connectorcloudflared-felis.servicePresent only when the access method is Cloudflare Tunnel

The main data locations on the host are listed below. For the size and growth of each, see Storage and disks.

PathContents
/etc/felisConfiguration and secrets: felis.host.toml (felis.toml links to it), felis.pod.toml, secrets.env, offsite.env, the panel TLS certificate and key, host-settings.env, and the tunnel configuration cloudflared.yml
/var/lib/felisThe database cluster (postgres/), database bundles (db-backups/), host task records (node-control/), watchdog state (watchdog/) and support bundles (support/)
/var/lib/rancher/k3sk3s data, the world volumes (storage/), world archives, the image registry and container images
/opt/felisVelocity, the Java runtime and build tools
/usr/local/bin/felisThe felis command, with the panel embedded

The admin kubeconfig /etc/rancher/k3s/k3s.yaml is readable by root only. Run kubectl commands on the host as sudo k3s kubectl.

Services and timers ​

Besides the long-running services above, the installer sets up these scheduled units:

UnitWhen it runsPurpose
felis-watchdog.timerEvery 2 minutes, first 3 minutes after bootHealth checks, alert mail and the external heartbeat; see Watchdog
felis-watchdog-failed.serviceWhen a watchdog run failsReports the watchdog's own failure
felis-db-backup.timerDaily at FELIS_DB_BACKUP_TIME (default 03:30, with up to 15 minutes of random delay); a missed run catches up after bootCreates a database bundle; see Database backups
felis-offsite.timerHourly, present only once the off-site copy is configuredCopies the backups, encrypted, to the bucket; see Off-site copy
felis-update-check.timerDaily around 05:30; a missed run catches up after bootRecords a version check; see Component versions
felis-build-tools.timerDaily at 04:00 and 16:00Copies kaniko, Trivy and its vulnerability databases into the in-cluster registry for image builds
World cleanup CronJob (in k3s)DailyDeletes expired world backups and, when idle-world retention is enabled, archives and deletes idle worlds; see World cleanup

To check the services and timers:

bash
systemctl list-timers 'felis-*'
systemctl status k3s felis-velocity felis-node-control
sudo k3s kubectl -n felis get pods

The Services and timers section of Maintenance & Backups → Status in the panel shows the same information.

While the control plane restarts ​

felis-api runs as a single replica with the Recreate strategy: the old pod stops before the new one starts. When an installer run updates the API, the node restarts or the pod crashes, the API is unavailable until the new pod is ready, normally for a few seconds. During that time:

  • Players already on a server stay there, and game servers keep running.
  • A player leaving the login space or joining a server by its address is admitted when felis-api confirmed their account link within the last 10 minutes; everyone else is told that login verification is temporarily unavailable.
  • The login space retries a new login for up to 60 seconds and tells the player it is retrying, so a short restart only delays the login.
  • Wakes, server stops, /link and panel actions wait for the API.
  • If the game proxy restarts in that window, it routes by the server list cached in /opt/felis/velocity/plugins/felis-link/last-servers.json until a refresh succeeds (every 15 seconds).

During a longer outage the proxy logs server list refresh failed ... keeping current registrations, then still failing: ..., and server list refresh recovered after ... once the API is back. The watchdog sends an alert once deployment/felis-api has been unavailable for 5 minutes.

Routine tasks ​

TaskHow oftenWhere
Check the platform status, open alerts and disk usageDailyThe Platform status bar on the Dashboard; Maintenance & Backups → Status
Confirm that alert mail is delivered: a mail relay is configured and the Owner's address is verifiedOnce after installation, then whenever the mail settings changeAlert mail, Mail relay
Set up the external heartbeat, so that you hear when the whole host goes downOnce after installationExternal heartbeat
Configure the off-site copy and keep its encryption key off the hostOnce after installationOff-site copy, Encryption key
Check that database backups and the off-site copy run on timeWeekly, or rely on watchdog alertsDatabase backups
Review available updates and apply them, inside the maintenance window when one is setMonthly or per releaseUpgrading
Rehearse a host rebuild on a spare machineOnce per releaseHost rebuild
Rotate credentials when staff with host access leaveAs neededRotating credentials
Export audit records before they pass their retentionAccording to the retentionExporting the audit log

Running host operations from the panel ​

The Owner can run the host operations of day-to-day running on the operator console, under Maintenance & Backups (/admin/updates) and Platform settings (/admin/platform). The panel hands each operation to felis-node-control.service, which runs the same felis command or installer step as the command line, as root. felis-api itself stays unprivileged.

Task rules ​

  • One host task runs at a time. Until it finishes, a new request is answered with 409 node_operation_busy (a host task is already running), and the panel shows "Another host task is running. This action becomes available when it finishes."
  • The host service accepts a fixed list of operations and validates every parameter. It offers no general command execution.
  • An operation that interrupts players or the panel asks for confirmation. Restoring a database bundle asks for the bundle's name; rebuilding from the off-site copy asks for the word REBUILD.
  • Every task and every download requires a fresh identity confirmation (a passkey or an email code, valid for 5 minutes) and is recorded in the audit log under platform.host.*; node operations are recorded under platform.node.*.
  • Keys and tokens entered in the panel (bucket keys, the off-site encryption key, the Cloudflare API token) are passed to the executor only. They never appear in the task record, the task log or an API response.
  • The stage, log and result of each task stay readable in the panel; the records are kept in /var/lib/felis/node-control. Losing the panel connection does not stop a task. Each operation has its own time limit, from 1 minute (starting the off-site copy) to 12 hours (the host rebuild).
  • While a database restore, a host rebuild, a credential rotation or a domain change runs, the API refuses to start, maintain or migrate servers, as it does during node operations.

Operation table ​

OperationPanel locationCommand line
Host status: disks, memory, services and timers, the game proxyMaintenance & Backups → Status → Host statusfelis status
Watchdog alerts; holding alert mail for 1, 4 or 24 hours during planned workMaintenance & Backups → Status → Host status → Watchdog alerts: Hold alert mail, Resume alert mail/run/felis/watchdog-quiet-until
Database backup nowMaintenance & Backups → Backups → Control-plane database backup → Back up nowfelis db backup
Verifying, restoring and deleting a database bundleMaintenance & Backups → Backups → Database backups on this host → Verify, Restore, the bin iconfelis db verify, felis db restore, felis db delete, felis migrate up
Daily database backup time and number keptMaintenance & Backups → Backups → Database backups on this host → Daily backup scheduleFELIS_DB_BACKUP_TIME, FELIS_DB_BACKUP_KEEP, installer re-run
World backup now, live or with the servers stopped firstMaintenance & Backups → Backups → Back up worlds nowfelis backup-now
Audit log exportMaintenance & Backups → Backups → Audit log exportfelis db audit-export
Configuring, changing or turning off the off-site copyMaintenance & Backups → Off-site copy → Configure, Change bucket, Turn offFELIS_OFFSITE_*, installer re-run
Copying now; listing the bucket; checking the key; taking over the bucketMaintenance & Backups → Off-site copy → Copy now, List bucket contents, Check key, Take over the bucketsystemctl start felis-offsite.service, felis offsite list, felis offsite check-key, felis offsite take-over
Showing the off-site encryption keyMaintenance & Backups → Off-site copy → Show key/etc/felis/offsite.env
Rebuilding this host from the off-site copyMaintenance & Backups → Off-site copy → Rebuild from the off-site copyfelis restore-host
Health checkMaintenance & Backups → Diagnostics → Health check → Run health checkfelis doctor
Support bundle, downloaded through the browserMaintenance & Backups → Diagnostics → Support bundle → Create support bundlefelis support-bundle
Component versions; the optional maintenance windowMaintenance & Backups → Updates → Component versions, Maintenance window (optional)felis update (read-only check); the maintenance window is set in the panel only
Update Felis to the newer version the daily check found (a newer release on a release build, the newest main commit on a development build)Maintenance & Backups → Updates → Update nowfelis update --apply
Access method: direct, own reverse proxy, Cloudflare TunnelPlatform settings → Access and security → Access and domain → Apply access methodThe access step of felis setup
Checking and changing the root domainPlatform settings → Access and security → Access and domain → Root domain: Check current domain, Change domainfelis domain check, felis domain set
Upload storage: this host or an S3-compatible bucketPlatform settings → Access and security → Upload storage → Apply storages on the summary screen of felis setup
Credential rotationPlatform settings → Access and security → Credential rotation → Rotatefelis rotate-token
Game proxy heap ceilingPlatform settings → Access and security → Game proxy → Heap ceiling, ApplyFELIS_VELOCITY_XMX, installer re-run
System server completionPlatform settings → Access and security → System server completion → Run completionfelis converge (felis converge -user-rcon with Enable RCON on older user servers)
Mail relayPlatform settings → Notifications and retention → Mail deliveryThe email step of felis setup ([smtp] in felis.toml)
External heartbeat URLPlatform settings → Notifications and retention → Monitoring heartbeatFELIS_WATCHDOG_HEARTBEAT_URL, installer re-run
A server's forwarding modeThe server's console page → Forwarding modeThe felis.lolicon.best/forwarding label on the MinecraftServer
Node operations, including the floating address for controller high availabilityPlatform settings → Distributed → Node operationsfelis node …, FELIS_HA_VIP*, installer re-run

WARNING

Restoring the database, rebuilding the host, rotating credentials, changing the domain, and changing the access method or the upload storage all restart the panel or disconnect players. Read the matching page first, for example Restoring the database and Changing the root domain.

Settings changed in the panel that the installer owns (the database backup schedule and the number kept, the game proxy heap ceiling) are stored in /etc/felis/host-settings.env. Later installer runs and felis update take them as their defaults, so the Owner's choice survives upgrades; an environment variable set explicitly for one run still takes precedence. The panel applies such a change by running the installer in reconfigure mode (FELIS_INSTALL_MODE=reconfigure, with FELIS_RECONFIGURE naming the parts), which rewrites only the affected units. For the precedence of settings, see Configuration reference.

The Dashboard shows the Owner a Platform status bar: open alerts, the newest database backup, the off-site copy state and the disk with the least free space, each linking to the matching tab.

Command-line-only operations ​

The panel does not offer the following:

  • Installation and installer re-runs, including the first installation on a new host and rebuilding a lost host with FELIS_RESTORE_FROM_OFFSITE=1. The panel exists only after installation.
  • felis setup, which creates the first Owner. Until then nobody can sign in to the panel.
  • felis break-glass, which restores access when no Owner can sign in and therefore does not depend on the panel; see Emergency access.
  • Version changes with a chosen target, and dependency upgrades: updating to a named release or commit (felis update --version, --ref), reinstalling the same version or downgrading (--force), and upgrading k3s and cloudflared (--all). The panel offers only the update to the newer Felis version the daily check found; see Updating from the panel.
  • Uninstalling; see Uninstall and reinstall.
  • Commands run by service units and the installer, such as felis api, felis operator, felis watchdog, felis ha leader and felis ha migrate-storage.
  • The host operations service itself. When felis-node-control is down, the panel cannot run any host task. Check and restart it on the host:
bash
systemctl status felis-node-control
journalctl -u felis-node-control -n 50 --no-pager
sudo systemctl restart felis-node-control

For the full usage of each command, see Command-line reference.