Running in production
This page covers what you need to run etemenanki-app and katana unattended on a Linux server: the command line, what each exit status means, how the processes react to signals, where their logs go and how to control them, a hardened systemd unit for each program, and an upgrade procedure you can repeat without surprises.
Both programs share the same command-line shape, the same signal handling and the same logging stack, so most of the page applies to both. Where they differ, a table says so. The katana section of this guide covers panel-specific operation; this page covers only what katana has in common with etemenanki-app.
The two programs at a glance
Section titled “The two programs at a glance”| etemenanki-app | katana | |
|---|---|---|
| Config file | -c path, default config.toml in the working directory |
same |
--test success line |
Configuration OK. on standard output |
Configuration OK on standard output |
--test failure line |
configuration invalid: …, written through the logger to standard output |
configuration error: …, written to standard error |
| Graceful stop | SIGINT, SIGTERM |
SIGINT, SIGTERM |
| Reload | automatic, when the config file changes | automatic, when the config file changes |
| Reload signal | none; SIGHUP terminates the process |
none; SIGHUP terminates the process |
| Logs | standard output | standard output |
[log].level changed in a running process |
ignored until the next start | applied at once |
Command line
Section titled “Command line”Both binaries take the same four flags and nothing else. There are no subcommands and no positional arguments.
| Flag | Argument | Default | What it does |
|---|---|---|---|
-c, --config |
path | config.toml |
The TOML file to load. A relative path is resolved against the process’s working directory, not against the binary’s location. |
--test |
none | off | Parse the file and build everything it describes, print the result and exit. Binds no port and opens no connection. |
-h, --help |
none | Print usage and exit with status 0. |
|
-V, --version |
none | Print the name and version, for example etemenanki-app 2.0.0 or katana 3.0.1, and exit with status 0. The version is the program’s own package version, so a kernel release that changes only a library, such as etemenanki 2.0.1 or 2.0.2, still prints etemenanki-app 2.0.0. |
An unknown flag, or -c without a value, is a usage error: the program prints, for example, error: unexpected argument '--bogus' found to standard error and exits with status 2.
What --test checks
Section titled “What --test checks”--test runs the same code a start runs, up to the point where the program would open sockets.
- etemenanki-app parses the file, then builds the DNS resolver, every outbound, every balancer, the routing table (loading geodata) and every inbound (reading TLS certificates and keys). A start does exactly the same, then binds the listeners.
- katana parses the file, builds the outbound pool, then builds each
[[node]]’s panel client and routing table, and checks the local settings of Hysteria 2 nodes. It does not contact the panel, so everything the panel supplies, such as a node’s protocol, port and transport, is checked only when the node starts.
Neither program checks, in --test, anything that depends on the running system:
- that a port is free or that the process may bind it;
- that the process may create a TUN device;
- that upstream servers, DNS servers or the panel are reachable.
Those show up only at start. For example, a TUN inbound tagged tun-in with the device name etm0 passes --test as an unprivileged user, then fails to start with failed to start: inbound tun-in bind tun etm0 failed: Operation not permitted (os error 1).
Exit codes
Section titled “Exit codes”| Status | Meaning |
|---|---|
0 |
--test passed; -h or -V printed; or the process stopped after SIGINT or SIGTERM. |
1 |
--test failed, or the start failed. etemenanki-app fails to start when the file cannot be read or is invalid, or when any inbound cannot bind. katana fails to start when the file cannot be read or is invalid, when the outbound pool cannot be built, when the file has no [[node]], or when not a single node’s panel client and routing table could be built. |
2 |
Command-line usage error. |
killed by SIGHUP |
The process has no SIGHUP handler, so the kernel terminates it. A shell reports status 129; systemd reports code=killed, status=1/HUP. |
At start, etemenanki-app logs the reason as failed to start: …. katana logs one of failed to load config: …, failed to build outbounds: …, config defines no [[node]] entries or no nodes could be started.
Paths and the working directory
Section titled “Paths and the working directory”Every relative path is resolved against the working directory of the process. That covers the -c path and every path inside the file, such as cert_file, key_file, ca_file, geoip, geosite and rule_list_path. (A Unix socket listen path is always absolute: etemenanki-app treats listen as a path only when it starts with /.) A relative path is not resolved against the directory that holds the config file.
[inbound.stream.tls]cert_file = "tls/fullchain.pem" # read from <working directory>/tls/fullchain.pemkey_file = "tls/privkey.pem"Checked from /etc/etemenanki, this file passes. Checked from any other directory with -c /etc/etemenanki/config.toml, it fails:
ERROR etemenanki_app: configuration invalid: No such file or directory (os error 2)The message does not name the file that is missing. When you see it, check -c first, then every relative path in the config.
To avoid the problem, either use absolute paths throughout, or always run the program with the config directory as its working directory. The systemd units below do the second with WorkingDirectory=.
Both programs watch the directory that contains the config file, not the file itself, so that editors which save by writing a new file and renaming it are noticed. Any file event in that directory, including a file being created, written, renamed, deleted or merely opened, makes the program read the config file again. etemenanki-app then compares the bytes with the last version it read, and katana compares the parsed settings, so unrelated files cause no reload. Keep that directory for configuration only, not for logs or other busy files.
The two comparisons differ in one way that matters. etemenanki-app reloads on any byte change, so editing only a comment or whitespace still rebuilds everything and closes every connection; the reload line then reads config reload: no changes. katana applies nothing when the parsed settings are unchanged.
Signals and shutdown
Section titled “Signals and shutdown”Neither program has a reload signal. A reload happens when the config file changes on disk; see Hot reload.
flowchart TB S["start"] --> L["read and parse the config"] L -- "error" --> E1["exit 1"] L --> B["build outbounds, routing, inbounds"] B -- "error" --> E1 B -->|"--test"| E0["exit 0"] B --> N["bind listeners"] N -- "error" --> E1 N --> R["running"] R -- "config file changed" --> R R -- "SIGINT or SIGTERM" --> D["close listeners and connections"] D --> E0 R -- "SIGHUP" --> K["killed by the kernel"]
The diagram shows etemenanki-app. katana follows the same shape, except that it builds and starts each node separately: a node that fails to build is logged and skipped, and the process exits with 1 only when no node could be built.
| Signal | etemenanki-app | katana |
|---|---|---|
SIGINT (Ctrl-C) |
Graceful stop, exit 0 |
Graceful stop, exit 0 |
SIGTERM |
Graceful stop, exit 0 |
Graceful stop, exit 0 |
SIGHUP |
Terminates the process at once, no shutting down line |
same; traffic not yet reported to the panel is lost |
SIGKILL |
Terminates the process at once | same; traffic not yet reported to the panel is lost |
A graceful stop logs shutting down and then:
- etemenanki-app stops accepting, closes every listener and closes every open connection at once. It does not wait for connections to finish. A Unix socket listener removes its socket file. A Hysteria 2 inbound closes its QUIC connections and waits up to about 6 seconds for its UDP port to come free; every other inbound stops immediately.
- katana stops every node: each node closes its listeners and connections, then reports its outstanding traffic to its panel and, on SSPanel, any pending audit results. Each request can take up to the node’s
api.timeout, which is 5 seconds when unset, so a stop can take roughly twice that.
After the first SIGINT or SIGTERM, further SIGINT and SIGTERM signals are absorbed. If a stop hangs, only SIGKILL ends it. systemd sends SIGKILL itself once TimeoutStopSec= (90 seconds by default) has passed.
Logging
Section titled “Logging”Both programs write their logs to standard output, one line per event. Under systemd, the journal collects them. Each line carries a UTC timestamp, the level, the module that logged it and the message:
2026-09-24T20:23:44.726638Z INFO etemenanki_app::instance: inbound socks-in listening on 127.0.0.1:10802026-09-24T20:23:45.424572Z INFO etemenanki_app: shutting downThe lines contain ANSI colour codes even when standard output is not a terminal, which shows up in the journal and in log files as sequences like [2m. Set the environment variable NO_COLOR=1 to turn the colours off. The units below do.
Choosing the level
Section titled “Choosing the level”The filter comes from the first of these that is set:
- The
RUST_LOGenvironment variable, if it is set and parses. ARUST_LOGthat fails to parse is ignored as a whole. An emptyRUST_LOGcounts as set and enables nothing. [log].levelin the config file. If the file cannot be read or parsed, this step is skipped, so the parse error is still logged atinfo.info.
Both take the same directive syntax: a comma-separated list, where a bare level sets the default and target=level overrides it for one module and everything below it. When several directives match, the most specific target wins, whatever its position in the list.
| Value | Effect |
|---|---|
info |
Startup, listener and reload lines, warnings and errors. The default. |
warn |
Warnings and errors only. |
debug |
Adds a line for every connection that ends with an error, and more. Verbose on a busy server. |
info,etemenanki_protocols=debug |
info everywhere, debug for the protocol implementations. |
warn,etemenanki_app::instance=info |
Warnings everywhere, plus the listener and reload lines. |
info,katana=debug |
katana’s own modules at debug, the kernel at info. |
off |
Nothing at all. |
The levels are error, warn, info, debug and trace, plus off. The targets are the crate names with - written as _: etemenanki_app, katana, etemenanki_protocols, etemenanki_environment and etemenanki_concepts, optionally followed by ::module.
[log]level = "info,etemenanki_protocols=debug"RUST_LOG=debug etemenanki-app -c config.toml # overrides [log].level for this runChanging the level at run time
Section titled “Changing the level at run time”| etemenanki-app | katana | |
|---|---|---|
[log].level changed while running |
Not applied. The level stays what it was at start. | Applied at once, without restarting any node. |
| Side effect of the change | Counts as a config change: the reload logs config reload: log changed, restarts every listener and closes every connection. |
None beyond the new level. |
[log].level removed while running |
Not applied | The level becomes info. |
| New value cannot be parsed | Not applied | Logs invalid log level "…": … and keeps the current level. |
Started with RUST_LOG |
RUST_LOG stays in force for the life of the process. |
The first change to [log].level replaces the RUST_LOG filter. |
Running etemenanki-app under systemd
Section titled “Running etemenanki-app under systemd”The unit below runs etemenanki-app as an unprivileged user, gives it only the capabilities it needs, and checks the config before every start. It assumes this layout:
Directory/usr/local/bin/
- etemenanki-app
Directory/etc/etemenanki/ owner root, group etemenanki, mode 0750
- config.toml mode 0640
Directorytls/
- fullchain.pem
- privkey.pem mode 0640
Prepare the host
Section titled “Prepare the host”-
Create a system user with no login shell and no home directory:
Terminal window sudo useradd --system --no-create-home --shell /usr/sbin/nologin etemenanki -
Install the binary:
Terminal window sudo install -m 0755 etemenanki-app /usr/local/bin/etemenanki-app -
Install the configuration so that root owns it and the service user can only read it. The config holds passwords and keys, so it must not be world-readable:
Terminal window sudo install -d -o root -g etemenanki -m 0750 /etc/etemenankisudo install -o root -g etemenanki -m 0640 config.toml /etc/etemenanki/config.tomlPrivate keys under
/etc/etemenanki/tls/need the sameroot:etemenankiownership and mode0640. -
Check it as the service user, from the working directory the service will use:
Terminal window sudo -u etemenanki sh -c 'cd /etc/etemenanki && exec /usr/local/bin/etemenanki-app --test -c config.toml'Configuration OK.
The unit
Section titled “The unit”Save one of these as /etc/systemd/system/etemenanki.service. The second variant is for configs with a TUN inbound: the highlighted lines differ, and it leaves out PrivateDevices=yes.
[Unit]Description=etemenanki-app proxyAfter=network-online.targetWants=network-online.target
[Service]Type=execUser=etemenankiGroup=etemenankiWorkingDirectory=/etc/etemenankiEnvironment=NO_COLOR=1ExecStartPre=/usr/local/bin/etemenanki-app --test -c /etc/etemenanki/config.tomlExecStart=/usr/local/bin/etemenanki-app -c /etc/etemenanki/config.tomlRestart=on-failureRestartSec=5sLimitNOFILE=1048576
# Bind ports below 1024 without running as root.AmbientCapabilities=CAP_NET_BIND_SERVICECapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yesProtectSystem=strictProtectHome=yesPrivateTmp=yesPrivateDevices=yesProtectKernelTunables=yesProtectKernelModules=yesProtectControlGroups=yesRestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINKRestrictNamespaces=yesLockPersonality=yesSystemCallArchitectures=native
[Install]WantedBy=multi-user.target[Unit]Description=etemenanki-app proxyAfter=network-online.targetWants=network-online.target
[Service]Type=execUser=etemenankiGroup=etemenankiWorkingDirectory=/etc/etemenankiEnvironment=NO_COLOR=1ExecStartPre=/usr/local/bin/etemenanki-app --test -c /etc/etemenanki/config.tomlExecStart=/usr/local/bin/etemenanki-app -c /etc/etemenanki/config.tomlRestart=on-failureRestartSec=5sLimitNOFILE=1048576
# CAP_NET_ADMIN creates and configures the TUN device.AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_ADMINCapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_NET_ADMINDeviceAllow=/dev/net/tun rw
NoNewPrivileges=yesProtectSystem=strictProtectHome=yesPrivateTmp=yesProtectKernelTunables=yesProtectKernelModules=yesProtectControlGroups=yesRestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINKRestrictNamespaces=yesLockPersonality=yesSystemCallArchitectures=native
[Install]WantedBy=multi-user.targetThen load and start it:
sudo systemctl daemon-reloadsudo systemctl enable --now etemenankijournalctl -u etemenanki -fThe journal shows one inbound … listening on … line per inbound once the service is up.
What the settings are for
Section titled “What the settings are for”| Setting | Why |
|---|---|
WorkingDirectory=/etc/etemenanki |
Relative paths in the config resolve against it, both for the start and for ExecStartPre=. |
ExecStartPre=… --test … |
Refuses to start on a broken config, and records the error as a failed pre-start step, separate from a bind error. For etemenanki-app this repeats what the start checks anyway. It does not protect a running service: systemctl restart stops the old process before the check runs. Run --test yourself before a restart. |
Restart=on-failure |
Restarts after an exit with status 1 or a crash, but not after systemctl stop, SIGTERM, SIGINT or SIGHUP. When the config is broken, each attempt fails the same way. With RestartSec=5s the default start limit (5 starts within 10 seconds) is never reached, so systemd keeps retrying every 5 seconds until the file is fixed. |
LimitNOFILE=1048576 |
Every client connection holds a descriptor, and usually a second one for its outbound connection. systemd’s default soft limit of 1024 is reached quickly; etemenanki-app then logs accept error, backing off 100ms: Too many open files (os error 24) and retries every 100 ms until descriptors free up. Each TCP or Unix socket inbound holds at most 65,536 live connections; see Limits. |
CAP_NET_BIND_SERVICE |
Needed only for inbound ports below 1024. Without it, the start fails with Permission denied (os error 13). Remove both capability lines if every port is 1024 or above. |
CAP_NET_ADMIN |
Needed only for a TUN inbound, to create the device, assign its addresses and add its routes. Nothing else in etemenanki-app uses it; the WireGuard outbound runs in user space and needs no privileges. |
DeviceAllow=/dev/net/tun rw, no PrivateDevices= |
PrivateDevices=yes hides /dev/net/tun. The TUN variant drops it and allows only that device. |
AF_NETLINK |
The TUN inbound installs its routes over netlink. glibc’s getaddrinfo, which the default system DNS backend calls, also opens netlink sockets to read the host’s addresses. |
ProtectSystem=strict |
The whole file system is read-only for the service. etemenanki-app writes nothing, except Unix socket files for inbounds that listen on a path. For those, add RuntimeDirectory=etemenanki and put the sockets under /run/etemenanki/. |
NO_COLOR=1 |
Keeps colour codes out of the journal. |
There is no ExecReload=. systemctl reload etemenanki therefore fails with an error and changes nothing, which is intended: the program reloads by itself when the file changes.
Changing the configuration
Section titled “Changing the configuration”-
Put the edited file next to the live one, with the same ownership and mode, so the service can read it once it is in place:
Terminal window sudo install -o root -g etemenanki -m 0640 config.toml /etc/etemenanki/config.toml.new -
Check it as the service user:
Terminal window sudo -u etemenanki sh -c 'cd /etc/etemenanki && exec /usr/local/bin/etemenanki-app --test -c config.toml.new' -
Move it into place. The running service notices the change, logs
config reload: …, and switches to the new configuration. A reload closes every open connection:Terminal window sudo mv /etc/etemenanki/config.toml.new /etc/etemenanki/config.toml
If the new file turns out to be invalid after all, the running service logs reload: parse failed, keeping current config: … or reload: build failed, keeping current config: … and carries on with the old one. The next start, however, will fail. Hot reload describes what a reload rebuilds and what happens when a port cannot be bound.
Running katana under systemd
Section titled “Running katana under systemd”One katana process can serve several nodes: repeat [[node]] in one file. Separate processes are useful when nodes should restart independently, for example one process per panel. A systemd template unit runs one process per config file:
Directory/etc/katana/ owner root, group katana, mode 0750
- xboard.toml mode 0640
- sspanel.toml mode 0640
Directory/etc/systemd/system/
[Unit]Description=katana node agent (%i)After=network-online.targetWants=network-online.target
[Service]Type=execUser=katanaGroup=katanaWorkingDirectory=/etc/katanaEnvironment=NO_COLOR=1ExecStartPre=/usr/local/bin/katana --test -c /etc/katana/%i.tomlExecStart=/usr/local/bin/katana -c /etc/katana/%i.tomlRestart=on-failureRestartSec=5sLimitNOFILE=1048576
# Only if a panel assigns a node port below 1024.AmbientCapabilities=CAP_NET_BIND_SERVICECapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yesProtectSystem=strictProtectHome=yesPrivateTmp=yesPrivateDevices=yesProtectKernelTunables=yesProtectKernelModules=yesProtectControlGroups=yesRestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINKRestrictNamespaces=yesLockPersonality=yesSystemCallArchitectures=native
[Install]WantedBy=multi-user.targetThe instance name after @ picks the file: katana@xboard runs /etc/katana/xboard.toml.
sudo useradd --system --no-create-home --shell /usr/sbin/nologin katanasudo systemctl daemon-reloadsudo systemctl enable --now katana@xboard katana@sspaneljournalctl -u 'katana@*' -fWhat differs from etemenanki-app:
--testis stricter than a start. A start skips a node it cannot build;ExecStartPre=refuses the whole file instead, so a typo in one node cannot leave that node silently down.- The panel decides the ports. katana binds whatever port the panel assigns to each node, so you may not know in advance whether
CAP_NET_BIND_SERVICEis needed. katana needs no other capability and writes no files. - Stopping takes a few seconds. Each node reports its last traffic to the panel on the way out. Leave
TimeoutStopSec=at its default, or at least a few timesapi.timeout, so that systemd does not kill the process before the report is sent. - Instances share a directory. Each instance watches
/etc/katana, so editing one file makes every instance re-read its own. An instance whose file did not change applies nothing. Restart=on-failurecovers the process, not each node. A node that was built but cannot come up does not end the process; it retries by itself while the other nodes keep running. Each failed attempt logs one line, such asnode 1: node_info failed: …; retrying in 1s(its first panel request failed) ornode 1: initial start failed: …; retrying in 1s(for example, its port could not be bound). The wait starts at 1 second and doubles after each failure, up to 60 seconds or the node’supdate_periodic, whichever is shorter. Saving an edit to that node’s[[node]]entry retries at once. systemd still sees a running process, so watch the journal forretrying in. Once the cause is fixed, for example the panel answers again or the port is free, the node comes up on its next attempt without a restart. Only a fix to the unit itself, such as a missing capability, needssystemctl restart. The katana section covers how a running node behaves when its panel becomes unreachable later.
Upgrading
Section titled “Upgrading”The same procedure works for both programs. It checks the new binary against the live configuration before anything is touched, keeps the old binary for a rollback, and swaps the file by renaming it. The commands show etemenanki-app. For katana, use /usr/local/bin/katana, the katana user and /etc/katana, run the check once per instance file (for example -c xboard.toml), and expect Configuration OK without a full stop.
-
Put the new binary next to the old one under a different name:
Terminal window sudo install -m 0755 etemenanki-app /usr/local/bin/etemenanki-app.new/usr/local/bin/etemenanki-app.new --version -
Check the live configuration with the new binary, as the service user and from the service’s working directory. A new version may reject a key the old one accepted, and this is where you find out:
Terminal window sudo -u etemenanki sh -c 'cd /etc/etemenanki && exec /usr/local/bin/etemenanki-app.new --test -c config.toml'Stop here if it does not print
Configuration OK.The running service is untouched. -
Keep the current binary for a rollback:
Terminal window sudo cp -p /usr/local/bin/etemenanki-app /usr/local/bin/etemenanki-app.prev -
Swap the files by renaming, not by copying over the running file:
Terminal window sudo mv -f /usr/local/bin/etemenanki-app.new /usr/local/bin/etemenanki-appLinux refuses to open a running executable for writing, so
cponto it fails withText file busy. A rename replaces only the directory entry: the running process keeps the old file, and the next start uses the new one. -
Restart the service. This closes every open connection; clients reconnect on their own:
Terminal window sudo systemctl restart etemenankiFor katana instances:
sudo systemctl restart 'katana@*'. -
Confirm that the service is running and that each inbound is listening again:
Terminal window systemctl status etemenankijournalctl -u etemenanki -n 20
To roll back, rename the old binary back into place and restart:
sudo mv -f /usr/local/bin/etemenanki-app.prev /usr/local/bin/etemenanki-appsudo systemctl restart etemenankiTroubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Fix |
|---|---|---|
No such file or directory (os error 2) from --test or at start |
The -c path, or a relative path inside the config, does not exist from the working directory. |
Check WorkingDirectory=, or use absolute paths. |
Permission denied (os error 13) from --test or at start, without the word bind |
The service user cannot read the config or a file it names. | Give the files group etemenanki (or katana) and mode 0640. |
failed to start: inbound … bind 0.0.0.0:443 failed: Permission denied (os error 13) |
Port below 1024 without CAP_NET_BIND_SERVICE. |
Add the capability, or use a higher port. |
node 1: initial start failed: Permission denied (os error 13); retrying in … (katana) |
The panel assigned a port below 1024 and the unit lacks CAP_NET_BIND_SERVICE. The node keeps retrying the bind, and every attempt fails the same way. |
Add the capability to the unit and restart the instance. The process cannot gain a capability while it runs. |
failed to start: inbound … bind … failed: Address already in use (os error 98) |
Another process holds the port. --test cannot detect this. |
Stop the other process, or change the port. |
failed to start: inbound … bind tun … failed: Operation not permitted (os error 1) |
No CAP_NET_ADMIN. |
Use the TUN variant of the unit. |
accept error, backing off 100ms: Too many open files (os error 24) |
The descriptor limit is too low. | Raise LimitNOFILE=. |
| No log lines at all | [log].level is not a level name, or is off, or RUST_LOG is set to off or to an empty string. |
Set [log].level = "info" and check the unit’s Environment=. |
Sequences like [2m and [0m in the journal |
Colour codes. | Set Environment=NO_COLOR=1. |
| The service stopped and systemd did not restart it | It received SIGHUP, or was stopped on purpose. |
Remove any ExecReload= that sends SIGHUP. |
config hot-reload disabled: … (etemenanki-app) or config watcher disabled (no live reload): … (katana) |
The file watcher could not start, often because the inotify limits are exhausted. The service keeps running without reloads. | Raise fs.inotify.max_user_instances or fs.inotify.max_user_watches, then restart. |
A changed [log].level has no effect in etemenanki-app |
The level is read only at start. | Restart the service. |